Title: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: bitcointweets on January 13, 2021, 07:06:59 PM Hi guys,
I'm a student at the University of British Columbia studying Computer Engineering. I'm interested in using alt data to predict price movements for Bitcoin. Unfortunately, most of these services are behind a paywall, so I made a free version myself. The tweet data is pulled from the Official Twitter API. I chart tweet frequency (volume of tweets about Bitcoin per minute) and Tweet sentiment as well. You can see the Live chart here: Bitcoin Tweets Sentiment Analysis (https://bitcointweets.com/admin/dashboard) I also made a Bitcoin News Sentiment and volume chart. This one pulls data directly from Google's RSS news feed every 10 minutes to track the release of major events that could affect the price. Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: LeGaulois on January 14, 2021, 01:48:42 PM I have been trying to find out about the AFINN sentiment analysis model, if I understand it correctly it is a list of words containing each its score.
https://github.com/fnielsen/afinn/tree/master/afinn/data It makes me think about Microsoft Or truly a mix of both? The first question I asked myself concerns all the Twitter bots that auto-post a tweet every XX minutes (using IFTTT or Zapier for example). The charts could therefore be easily manipulated, by creating hundreds of accounts, collecting subscribers, and then automating the accounts. It is very easy to create a hundred accounts per day. I believe 1000 accounts could be enough to manipulate the charts. Currently, there are 15.000 tweets/hours, an account can post every 10 minutes (to avoid the Twitter algorithm) 1.000 accounts x 6 tweets/hours= 6.000 tweets. It's about 40% of the 15.000 How the tool filters this? Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: bitcointweets on January 18, 2021, 06:58:57 PM I have been trying to find out about the AFINN sentiment analysis model, if I understand it correctly it is a list of words containing each its score. https://github.com/fnielsen/afinn/tree/master/afinn/data It makes me think about Microsoft Or truly a mix of both? The first question I asked myself concerns all the Twitter bots that auto-post a tweet every XX minutes (using IFTTT or Zapier for example). The charts could therefore be easily manipulated, by creating hundreds of accounts, collecting subscribers, and then automating the accounts. It is very easy to create a hundred accounts per day. I believe 1000 accounts could be enough to manipulate the charts. Currently, there are 15.000 tweets/hours, an account can post every 10 minutes (to avoid the Twitter algorithm) 1.000 accounts x 6 tweets/hours= 6.000 tweets. It's about 40% of the 15.000 How the tool filters this? Good Points. You clearly had an in-depth look at the tool. Yes, that is the correct understanding of AFINN. The tool does take into account volume of tweets along with sentiment plus a few other factors (Twitter account age and activity is used to filter out fresh accounts). Also, Twitter is very good at detecting spam. How do I know? Because I actually tried to create a bot that would tweet out current sentiment and the API was very restrictive on how often it would let me post. There are no explicit rate-limits to posting statuses on twitter, but the algorithm doesn't want you to automate tweets that are too similar to your recent statuses etc. Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: LeGaulois on January 18, 2021, 10:42:27 PM Pretty smart
I see you used the API from developer.twitter.com that's why you found it very limited. If you go through a third-party application it becomes much less restrictive. For example, you could write 100 tweets, spin the text 10 times to have 1000 tweets, upload all of them in a WordPress database that will auto-post on a regular basis, you will see much less restrictions. And yeah, Twitter improved a lot to detect the spams over the years. About the rate-limit... https://i.imgur.com/JgUZWQh.png But as you surely know, it's a case by case depending on the account used Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: tranthidung on January 19, 2021, 03:21:24 PM Thank you for creating that website with visual real-time updated charts.
Some contributions:
Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: bitcointweets on January 20, 2021, 12:40:22 AM Thank you for creating that website with visual real-time updated charts. Some contributions:
Yes, absolutely. These are my thoughts as well, after monitoring the 24hr chart for a few months. I'm currently developing a monthly and yearly chart. It'll be out very soon. Interesting point about volatility depending on time of month, I did not know about this. Yes, that is a very common critique, people do hate the moving lines. I'll make them static. The reason I animated them was because of a UI book I read but everyone hates it. I can expose an API that you can use to pull the data as JSON, that should be very simple as I'm already set up to do that. Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: bitcointweets on January 20, 2021, 12:50:39 AM Pretty smart I see you used the API from developer.twitter.com that's why you found it very limited. If you go through a third-party application it becomes much less restrictive. For example, you could write 100 tweets, spin the text 10 times to have 1000 tweets, upload all of them in a WordPress database that will auto-post on a regular basis, you will see much less restrictions. And yeah, Twitter improved a lot to detect the spams over the years. About the rate-limit... https://i.imgur.com/JgUZWQh.png But as you surely know, it's a case by case depending on the account used Yes, I am currently looking at TWINT etc to get around rate limits. Right now I'm actually using AWS for the backend. I love the scalability of using microservices for everything. The computing, databases, notification system, REST api, all handled through AWS. Title: Re: My side project: a chart that tracks Bitcoin Twitter Sentiment and Tweet Volume Post by: tranthidung on January 20, 2021, 01:47:34 AM I'm currently developing a monthly and yearly chart. It'll be out very soon. That thread is not exactly answer this question, but somewhat and you can handle it by your side I think. You also can visually detect it with Heikin Ashi candle on trading view, 1D chart.Interesting point about volatility depending on time of month, I did not know about this.
Quote Yes, that is a very common critique, people do hate the moving lines. I'll make them static. It is your good choice and will be a good upgrade for your site.Quote I can expose an API that you can use to pull the data as JSON, that should be very simple as I'm already set up to do that. I am not a coder so I can not handle with JSON. It will be good if you dump your data in csv format that is one of standard for machine-readable data.Another note is Twitter is full of farm accounts so do you exclude them from your analysis. Farm accounts can bias your result a lot.
|