Title: Turning your post history into word clouds! Post by: illiki23 on December 09, 2017, 09:49:12 AM Wrote a little script that generates word clouds for a user's last 25 posts.
Each post gets a word cloud but I could easily change it to generate one for each week, month, whatever. This is a tiny part of a hopefull larger project approaching shilling analysis and detection. Later versions may calculate and make use of sentiment. Here are my last 25! If you want me to do one send your user id. Any requests for special formatting or additional features may or may not be granted so feel free to ask. The python script is included at the end of this post so you can do it yourself as well though I just wrote it so like most code it could use some refactoring. https://i.imgur.com/goHCy1i.png Code: import requests Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 09, 2017, 04:46:27 PM No bites? *shrugs*
Probably shouldn't post at 1 in the AM. I would love some examples of known shills and non-shills that I can visualize. These are just word clouds and not very useful and there are a number of other techniques which are more useful. Like plotting shifts of sentiment. BTW there is a really neat Kaggle contest going on where you are given a collection of story snippets and their authors (it was a Halloween contest and focuses on spooky authors like Poe) for training and the goal is to build a classifier that can correctly match unlabeled snippets with their authors. Title: Re: Turning your post history into word clouds! Post by: LoyceV on December 09, 2017, 06:04:29 PM I'll bite, do me please: ID 459836.
Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 09, 2017, 06:19:25 PM I'll bite, do me please: ID 459836. Yes! Rewriting it a bit to make it look nicer. Also parsing dates so I can show word clouds for given data ranges such as a weekly word cloud. I will play around and have yours sometime today. Just wrote the darn script so there is a lot of room for improvement. Lets see what we can find out about you using text visualization. Got another program which detects emotions in text that we used to process novels. Might color code by emotion, or add emoticons indicating the emotion! Unemployed right now so pardon the excitement over little things. Projects keep me going. Title: Re: Turning your post history into word clouds! Post by: LoyceV on December 09, 2017, 07:09:11 PM It's kinda cool to see my own work like this!
Now, some comments:
Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 09, 2017, 07:15:57 PM It's kinda cool to see my own work like this! Now, some comments:
I am totally on it. Was excited to see the intial pieces come together. It also should order them the reverse direction. I will look into the problems, mine did 25 but yeah I think my number of posts per page counter is off. Thanks for the input. The fastest way to develop neat software is iteratively with constant user feedback in my book. Title: Re: Turning your post history into word clouds! Post by: Jet Cash on December 10, 2017, 10:43:11 AM Thanks for posting this - it's really useful on two counts. It will help me to get my head around python. It will help me to check on the quality of my postin.
It's a great idea. Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 10, 2017, 06:43:21 PM This is for user apoorvlathey. Please let me know one thing you would change about the word clouds!
https://i.imgur.com/BqJEFf1.png Title: Re: Turning your post history into word clouds! Post by: Joel_Jantsen on December 10, 2017, 07:03:49 PM Interesting !
The first thing that comes to my mind is,why don't you open-source such projects ? We all can work on it together ! Having said that,this could be transformed into a useful tool that helps us find out shitposters in a way.Looking forward to your ultimate goal.Cheers. Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 10, 2017, 07:22:33 PM Interesting ! The first thing that comes to my mind is,why don't you open-source such projects ? We all can work on it together ! Having said that,this could be transformed into a useful tool that helps us find out shitposters in a way.Looking forward to your ultimate goal.Cheers. I am all about open source. Note the end of the first post! :) Not sure we will ever know the ultimate goal, but the direction I am going in involves the use of data mining and visualization (visual data mining) to model and detect shilling within product review sets, forum threads, and so on. Visualization is a means of doing this, and the techniques I hope to use go well beyond word clouds. Word clouds visualize surface level information and can be rather insightful but there are many great techniques which work at the level of meaning. Currently working with sentiment analysis and emotion detection. When shilling occurs we see certain patterns of sentiment. (posted a toy project a few weeks ago somewhere visualizing shilling) So yeah, shitpost modeling. :) Other than that my goal is to finding constructive ways of meeting signature campaign requirements. I like to meet the requirements but feel bad when not being productive or contributing. Things like this help me constructively contribute while meeting my needs. Title: Re: Turning your post history into word clouds! Post by: Joel_Jantsen on December 10, 2017, 07:28:14 PM I am all about open source. Note the end of the first post! :) I mean,putting the code on Github but anyway,I get your point.Not sure we will ever know the ultimate goal, but the direction I am going in involves the use of data mining and visualization (visual data mining) to model and detect shilling within product review sets, forum threads, and so on. Visualization is a means of doing this, and the techniques I hope to use go well beyond word clouds. Word clouds visualize surface level information and can be rather insightful but there are many great techniques which work at the level of meaning. Currently working with sentiment analysis and emotion detection. When shilling occurs we see certain patterns of sentiment. (posted a toy project a few weeks ago somewhere visualizing shilling) Don't ML make an integral part of it along with data mining ? Honestly,this isn't as easy as it looks like.I'm pretty sure complexity will keep on increasing as the project progresses.So yeah, shitpost modeling. :) Title: Re: Turning your post history into word clouds! Post by: illiki23 on December 10, 2017, 07:45:31 PM I am all about open source. Note the end of the first post! :) I mean,putting the code on Github but anyway,I get your point.Not sure we will ever know the ultimate goal, but the direction I am going in involves the use of data mining and visualization (visual data mining) to model and detect shilling within product review sets, forum threads, and so on. Visualization is a means of doing this, and the techniques I hope to use go well beyond word clouds. Word clouds visualize surface level information and can be rather insightful but there are many great techniques which work at the level of meaning. Currently working with sentiment analysis and emotion detection. When shilling occurs we see certain patterns of sentiment. (posted a toy project a few weeks ago somewhere visualizing shilling) Don't ML make an integral part of it along with data mining ? Honestly,this isn't as easy as it looks like.I'm pretty sure complexity will keep on increasing as the project progresses.So yeah, shitpost modeling. :) Sure I will open a github for this project. Increasing complexity is our friend. As long as we recenter often. Increasing improvement, increasing complexity, and every now and then a shakedown and paradigm change. Machine learning and data mining go hand in hand. I like decision tree ensembles! And don't forget I am doing this for fun. And I do have a graduate degree in this stuff. (data mining with a focus on visualization and natural language processing) :P |