New web service: RSS feeds for topics and users' posts

[Vysogota]

= RSS feeds for topics and users' posts =

I would like to announce a new web service that allows you to watch topics and users' activities using an RSS reader.

== Quick guide ==

To subscribe to a specific topic, use following URL in RSS reader: http://100btc.pl/bitcointalk/rss/threads/TID, where TID is an ID of the topic. The ID can be found in url bar in the browser. For example, an URL for the topic "ASICMINER: Entering the Future of ASIC Mining by Inventing It" is https://bitcointalk.org/index.php?topic=99497.0 and its ID is 99497 (please note that ID of a topic always starts with a digit after "topic=" and ends with a digit before a dot). The RSS feed for this topic can be dowloaded from: http://100btc.pl/bitcointalk/rss/threads/99497.

Similary, to subscribe to posts from specific user, use following URL: http://100btc.pl/bitcointalk/rss/users/UID, where UID is an ID of the user. For example, profile of user 'friedcat' is available at https://bitcointalk.org/index.php?action=profile;u=49840 and his ID is 49840. The RSS feed for his posts is available at http://100btc.pl/bitcointalk/rss/users/49840.

== How the service works ==

Known feeds are updated every 10 minutes by cron job and are stored in site cache, so they can be served without interaction with bitcointalk.org server.

When a new topic is retrieved from the site for the first time, the service needs to check the topic existence and its title. To do that it sends a single request to the bitcointalk.org server. If everything is fine, empty feed is generated and is returned to the requestor. After that within a minute the newly generated feed is filled with a content and stored in cache.

==

This is an early version of the service. I am aware of a few things that should be done better. I will fix them in near future but first I would like to know what you think. Any comments and suggestions are welcome.

Just like donations (: 164iKDnK6Ri8HAGbtGMdhhs59FG1f4Vovz.

V.

[greyhawk]

Is this a cron job per feed?

Or is one cron job for all of them at the same time and we will see the forums grind to a halt every ten minutes in the future?

[Vysogota]

This is single cron job. It is run every minute. It checks which channels have not been updated in the last 10 minutes and regenerates them. The script sends queries sequentially, never more than one at a time. It is a kind of a simple balancer.  It can be further improved in order not to update more than one or two channels at a time.

V.

[nimda]

Any guidelines on polling rates?

[Vysogota]

Any guidelines on polling rates?

I have configured my reader to poll the feeds every 10 minutes. This means that the delay in the delivery of messages to my inbox does not exceed 20 minutes.

All the feeds are stored on the server as a static files and are served directly by web server, so performance issues are unlikely. If the service will gain in popularity, the distribution of the traffic across several servers will not be problematic.

V.

[Kluge]

Oh fuck yeah! I'll be stalkin' like a boss, now! 

[gweedo]

Any chance you would release the source code on github?

[Vysogota]

Any chance you would release the source code on github?

At the moment? No.
In the future? Maybe.

Everything depends on the number of users. If it will grow, I will have the motivation to work on this project and it is quite possible that I will release the code. At the moment it is only a prototype, which requires too many design decisions to develop it in a larger group of people.

[Vysogota]

It's been over a week since announcement of the service and it is right time to post statistics.

The total number of downloads per day:

Code:

2013-10-12 438
2013-10-13 403
2013-10-14 475
2013-10-15 673
2013-10-16 788
2013-10-17 813
2013-10-18 839
2013-10-19 759
2013-10-20 810

Number of bytes downloaded per day:

Code:

2013-10-12 14319652
2013-10-13 13646242
2013-10-14 15697403
2013-10-15 23024745
2013-10-16 25667567
2013-10-17 25903330
2013-10-18 28232714
2013-10-19 24446883
2013-10-20 26504304

Most popular threads:

Code:

1 1072 /bitcointalk/rss/threads/99497 ASICMINER: Entering the Future of ASIC Mining by Inventing It
2 449 /bitcointalk/rss/threads/235763 ASICMINER Speculation Thread
3 320 /bitcointalk/rss/threads/289730 [IPVO] [Multiple Exchanges] Neo & Bee - LMB Holdings
4 317 /bitcointalk/rss/threads/294735 Bitburner Fury - Hashrate Protection
5 298 /bitcointalk/rss/threads/309918 [ANN] Bi•Fury | 5+ GH/s USB Miner [FASTEST USB MINER IN THE WORLD]
6 297 /bitcointalk/rss/threads/179769 [Work in progess] Burnins Avalon Chip to mining board service
7 272 /bitcointalk/rss/threads/266828 [BTC-TC] BTC Growth: Capital Growth via Hedge Fund-Style Investing
8 260 /bitcointalk/rss/threads/86854 [NastyFans.org] NASTY MINING | NASTY POOL
9 232 /bitcointalk/rss/threads/280570  [BTC-TC] CryptX introduces the PETA-MINE - 18,000 CHIPS IN SEPTEMBER/UPDATE
10 116 /bitcointalk/rss/threads/310783 [Crypto-stocks] PETA-MINE dividend 7,3% paid
11 104 /bitcointalk/rss/threads/297503 [ActiveMining] The Official Active Mining Discussion Thread [Self-Moderated]

Most popular users:

Code:

1 1178 /bitcointalk/rss/users/49840 friedcat
2 272 /bitcointalk/rss/users/84153 Felipeo
3 270 /bitcointalk/rss/users/142607 cryptx
4 269 /bitcointalk/rss/users/70823 burnin
5 266 /bitcointalk/rss/users/38146 someone42
6 259 /bitcointalk/rss/users/127826 cryptocyprus
7 258 /bitcointalk/rss/users/40149 Deprived
8 257 /bitcointalk/rss/users/132160 DrGregMulhauser
9 92 /bitcointalk/rss/users/1707 kslaughter
10 88 /bitcointalk/rss/users/64492 havelock

If you have any ideas on how to improve the service, please post them in this thread.

Thanks for using the service.

V.

[go1111111]

Feature request: allow having the posts from multiple users in one feed. I don't want to have a separate feed for 20 people. I just want a "interesting stuff on BCTalk" feed.

Nice work overall.

[Vysogota]

Feature request: allow having the posts from multiple users in one feed. I don't want to have a separate feed for 20 people. I just want a "interesting stuff on BCTalk" feed.

Nice work overall.

Sorry for late response. Let me understand your needs correctly. Do you know which users you want to observe? If so you should use some kind of news aggregator like Digg Reader or Feedly (these are cloud services) or TinyRSS (you can host it yourself) or Thunderbird (and I belive that, there are tons of extensions for Firefox and Chrome that might be usefull for you as well). You will get single list of news from bitcointalk.org and you will be able to share it with other users.

Or maybe you don not know what to read and you want advice? In that case I can not help you because I do not know what you are interested in. Perhaps other users have their own reading lists and can share.

V.

[maurya78]

Overall high utility service

Thanks

[go1111111]

Sorry for late response. Let me understand your needs correctly. Do you know which users you want to observe? If so you should use some kind of news aggregator like Digg Reader or Feedly (these are cloud services) or TinyRSS (you can host it yourself) or Thunderbird (and I belive that, there are tons of extensions for Firefox and Chrome that might be usefull for you as well). You will get single list of news from bitcointalk.org and you will be able to share it with other users.

My issue is that I already have an RSS reader (The Old Reader), and I have lots of other RSS feeds in it. So my "view all feeds" feed contains both bitcointalk posts and lots of other stuff that I may not be in the mood for. What I'd like to do is have the bitcoin talk feeds appear in one "feed"/stream, so that I can just click that feed and see what's new on bitcointalk today. But currently I have to read the individual feed for each user.

Are you saying that other RSS readers allow you to create custom feeds composed of multiple 'base' feeds?

[Vysogota]

Are you saying that other RSS readers allow you to create custom feeds composed of multiple 'base' feeds?

Yes. For example Feedly has a nice feature called folders. You can put all your BCT feeds in one folder and read them as a single stream of news.

V.

[Vysogota]

The service is currently shut down due to following error:

2013-12-02 07:14:04.269287 Can't open page https://bitcointalk.org/index.php?action=profile;u=142067;sa=showPosts: Connection refused.

I do not know why this is happening. Perhaps administrators have blocked some IP addresses on the firewall in response to recent attacks and accidentally they have killed an innocent. I want to talk with administrators and clarify the case, but unfortunately I do not know who is right person to this talk. Can you help me?

V.

[Vysogota]

The service is currently shut down due to following error:

2013-12-02 07:14:04.269287 Can't open page https://bitcointalk.org/index.php?action=profile;u=142067;sa=showPosts: Connection refused.

I do not know why this is happening. Perhaps administrators have blocked some IP addresses on the firewall in response to recent attacks and accidentally they have killed an innocent. I want to talk with administrators and clarify the case, but unfortunately I do not know who is right person to this talk. Can you help me?

V.

PM theymos https://bitcointalk.org/index.php?action=profile;u=35

Thanks. Done. Waiting for response.

V.

[Vysogota]

We are online again.

V.

[Voodah]

This is great. I've always wanted something like this.

[Vysogota]

It took me a little more than a month to find some time and check why RSS feeds for the topics stopped working. It turned out that the threads template has been changed, so the parser stopped recognize the posts.

At the moment everything seems to be working.

Sorry for the long response time, but this is a project that I'm doing in my spare time.

V.

[sdersdf2]

Looks like this might have happened again - not seeing some topic threads updated for more than 10 days.

It took me a little more than a month to find some time and check why RSS feeds for the topics stopped working. It turned out that the threads template has been changed, so the parser stopped recognize the posts.

At the moment everything seems to be working.

Sorry for the long response time, but this is a project that I'm doing in my spare time.

V.