Unofficial Bitcoin Talk Forums API

[Logik]

Hi,

It was a long weekend last week here in Australia, and I was too broke from buying Bitcoins to go out, so I embarked on a project to create an unofficial Bitcointalk.org REST API.

I will give a short description of the API, then there are some links you can visit which show the API in action.

The API is JSON only, and currently lets you list categories, boards, topics, and posts. Future work may include adding user profiles also. The API only lets you read from Bitcoin Talk; there is currently no way to post, edit posts and so on. The API is hosted on Google App Engine (GAE). NB: Occasionally an HTTP request to GAE apps will take a long time to respond if it has to spin up a new instance.

• Category / board list
• An individual board listing
• An individual topic listing
• An individual page listing
• Convenience parameter - load a topic and known page number in one request, instead of two. The page is loaded into 'requestedPage'
• Convenience parameter - load a topic and the 'latest' page in one request. The latest page is the last page in a multi-page topic or the first page if there is only one page. The page is loaded into 'requestedPage'

If you browse the API you may discover boards and topics which don't appear to have any content - that's due to my 'lazy scraping' pattern detailed below.

A small note about viewing the API from your web browser: I'm using a little trick I learned from some of my old colleagues while developing APIs - the API checks the "Accept" header to detect if you are requesting the content from your browser, or from JavaScript. If you are in your browser then the API returns pretty printed HTML and turns relevant properties into links, so you can follow them. If you request from JavaScript then non pretty printed, non-linked JSON objects are returned. This lets you browse the API easily without a REST client.
 
The API works by using a screen scraper on the WAP version of the forums, as suggested by theymos in this post. The screen scraper is in its own project, separate from the API itself.

The overall design is to avoid scraping old topics, and instead focus on the newest content. The scraper can’t download the entire forum at once, so it uses a ‘lazy download’ or ‘lazy refresh’ approach to scraping content - if a topic or board is requested and it was scraped less than a certain amount of time ago*, then a task will be added to a task queue for the topic to be re-scraped. That means if nobody is using the API then no tasks go in the queue, and if lots of people are using the API then lots of tasks go in the queue. The task queue will then eventually be filled up by many requests to scrape many boards and topics, and I can adjust the rate of the execution of these tasks based on how much theymos yells at me . If an extremely large topic is encountered (like the Wall Observer thread in Speculation) then the first two pages and the last two pages will be downloaded only, to avoid generating too many requests.

* Current ‘freshness’ of boards is 5 minutes and freshness of topics is 1 minute. If you’re making a client then I recommend simply hard-coding the list of categories and topics since it’s not likely to change that much.

Limitations:

• Fixed number of posts in a page (20, same as these forums)
• Currently JSON only
• There isn't any time/date information on posts - that's because I'm using the WAP version of the forums and the date isn't displayed.
• Similar to the point above, the formatting for the BBCode (quote) feature is lost and it just comes out as plain text like 'Quote: ... <br/></br>

Code is open source at GitHub (Java): BitcoinTalkScraper and BitcoinTalkAPI

Stay tuned this week for news on my unofficial Bitcointalk mobile/tablet forum client.

I am doing this totally for the love of Bitcoin and these forums, and everything is free, but if you are feeling really generous then please tip me at 17SbWcyRoZd7u1tZeJtjzm834a3gAHdf2A. Thanks!

[1715091657]

1715091657

[1715091657]

1715091657

[1715091657]

1715091657

[eldentyrell]

The API is hosted on Google App Engine (GAE).

Thanks for sharing your screen-scraper, but coding to an API is a serious time investment.  That investment is pretty hard to justify if the forum operator does not acknowledge the API or make any effort to avoid breaking it.

[Logik]

Yeah, I agree. I can see your posts from the other thread.. You would have seen that theymos said the WAP page wouldn't change, so until the forum upgrade happens then it should be stable. I can't honestly make any guarantee that my API won't break, since I can't control these forums. However I am making my own client so I wanted to publish my own API so that others can use it, since it doesn't feel right keeping it closed source since it's not my content to begin with!

I'll adjust the scraper when the new forums go in if there is no API available.

[Logik]

Bumping this because of recent interest from another member

[whiskers75]

Thanks so much for making this! Might be able to create something nice with this API, now all I need is login support etc (which you'll hopefully add soon)

0.01 BTC sent (https://blockchain.info/tx/9291f180b4d54ffb453ffda8d57ff47bc597ba2634bef38520039789c550c95e). Plus have a kudos from a fellow coder

[Logik]

Woohoo, my first tip!

Login support was low priority but I can move it to the top since I'd rather code things people actually want/need. 

What do you actually want to be able to do? Off the top of my head there are several major functions

- Create/edit topic
- Create/edit post
- Edit profile/forums settings
- Manage PMs

If you let me know which is the most important I can do that first. But I can start coding the basic login steps (auth to server and store session cookie) now.

 

[🏰 TradeFortress 🏰]

You should make PMs are priority. For example, verification bots "PM CoinLenders-Verify with 28fy28c in the subject".

[Logik]

You're hinting at a tipping bot yeah?

Someone PMed me about that a while ago but it wasn't clear if he was going ahead with it.. I'm surprised nobody is working on it. Do you know something I don't or are you just speculating?

[🏰 TradeFortress 🏰]

Nope, I'm not? I'm just saying a PM api would be useful to me really

Also, what about a firehose API?

[whiskers75]

Woohoo, my first tip!

Login support was low priority but I can move it to the top since I'd rather code things people actually want/need. 

What do you actually want to be able to do? Off the top of my head there are several major functions

- Create/edit topic
- Create/edit post
- Edit profile/forums settings
- Manage PMs

If you let me know which is the most important I can do that first. But I can start coding the basic login steps (auth to server and store session cookie) now.

 

Login, manage PMs.

(Let's work on a tip bot!)

[Logik]

OK manage PMs it is then! Good thing I asked; I assumed managing PMs was the least important.

I'll start work on the auth stuff tonight and update here when I have news.

[theymos]

If possible, you may want to have API users POST the forum directly so that we don't have to deal with abuse/bans through your API.

[whiskers75]

OK manage PMs it is then! Good thing I asked; I assumed managing PMs was the least important.

I'll start work on the auth stuff tonight and update here when I have news.

Thanks! (Well, PM's are important for anybody looking to make a bot, since public topic reply isn't really needed.)

[Logik]

If possible, you may want to have API users POST the forum directly so that we don't have to deal with abuse/bans through your API.

I'm more than happy to oblige but I need you to expand on this a little. Are you talking about the IP banning logic?

[theymos]

I'm more than happy to oblige but I need you to expand on this a little. Are you talking about the IP banning logic?

Yes. If people could post things through your API, your API would be acting as a proxy, preventing IP bans from being effective. Maybe some solution to this could be figured out, but it makes more sense for API users to send such requests to the forum directly. Clients don't need to process any forum HTML when posting -- they just need to POST specific data to specific URLs and sometimes store the returned cookies. You could write a library to help clients figure out what data to send and where to send it, or your API could return this data.

[whiskers75]

theymos - or maybe you could host the API on bitcointalk.org?

[Logik]

I'm more than happy to oblige but I need you to expand on this a little. Are you talking about the IP banning logic?

Yes. If people could post things through your API, your API would be acting as a proxy, preventing IP bans from being effective. Maybe some solution to this could be figured out, but it makes more sense for API users to send such requests to the forum directly. Clients don't need to process any forum HTML when posting -- they just need to POST specific data to specific URLs and sometimes store the returned cookies. You could write a library to help clients figure out what data to send and where to send it, or your API could return this data.

Roger that.

The solution you propose should be totally fine.

One thing I wonder, though, is if it would be sufficient to have the login request be direct, and subsequent requests with that SMF cookie be proxied.

If the user was banned then I'm assuming the IP address would be reported as the one in the original login request?

If not then I can proxy everything but this would be a nice shortcut/work-around/compromise type thing.

While we're on the topic, can you please un-ban 203.206.164.125? I made a joke account while at work (static ip) but one of the mods didn't find it funny, and now I have to tether to my phone to post

[Logik]

theymos - or maybe you could host the API on bitcointalk.org?

That wouldn't actually help this particular issue.

[theymos]

One thing I wonder, though, is if it would be sufficient to have the login request be direct, and subsequent requests with that SMF cookie be proxied.

If the user was banned then I'm assuming the IP address would be reported as the one in the original login request?

No. Bans are based on other logged IPs as well.

While we're on the topic, can you please un-ban 203.206.164.125?

Alright.

[dansmith]

The API is JSON only, and currently lets you list categories, boards, topics, and posts.

I can't find the way to view individual posts.

[Logik]

Sorry, posts are currently only viewable by requesting a topic/page combination. Did you need a way of accessing a post via its BitcoinTalk message id (like when you link to an individual post)? Or would any id do?

[AuctionBot]

Any idea of how you're going to handle the CAPTCHA for sending PMs?

[dansmith]

Sorry, posts are currently only viewable by requesting a topic/page combination. Did you need a way of accessing a post via its BitcoinTalk message id (like when you link to an individual post)? Or would any id do?

Any ID would do.
I want new posts displayed in RSS feed style.
So my script will remember the total amount of posts in a thread. Then when the total changes, the script fetches the last post.

Actually, as I was writing this, I realised that I can achieve this by taking the page and calculating the amount of posts I want to discard by matching the amount of {} pairs. Then I can proceed to fetch the posts I need.

Have you got any tool ready for the end-user?
Ideally I want a tool that integrates with Thunderbird. In such case my script parses your API then converts new posts to RSS and acts as a local RSS server which sends new posts to TB.

[Logik]

OK I can expose the database id for a post. Will do that tonight and post back here.

You're aware of the existing API endpoints built into the forum yeah? They are reasonably limited but there is some support for new posts

For example here is the endpoint for this thread https://bitcointalk.org/index.php?topic=154131&action=.xml - that's in XML. You can RSS an entire board but I'm not sure if you can RSS a particular thread.. for example here is RSS for the main board https://bitcointalk.org/index.php?&action=.xml;board=1;type=rss. Is what you're saying that you need an RSS feed for an individual thread? Someone else actually asked about that the other day on here. Was that you?

Thanks for pointing out about the Captcha; I've never actually encountered that. I assume it triggers when you have sent a certain number of them?

[theymos]

There's a CAPTCHA for PMs when you have less than 10 posts.

[AuctionBot]

Thanks for clearing that up Theymos.

Also, I had no idea about the XML endpoints. It doesn't seem to work on a thread-by-thread basis. It would make scraping a hell of a lot easier.

[AuctionBot]

Looking at this thread: http://www.simplemachines.org/community/index.php?topic=25009.0

It seems you can only get a number of recent posts from particular sub-forums or categories. Shame.

[dansmith]

Logik,
thank you for working on your API.

For example here is the endpoint for this thread https://bitcointalk.org/index.php?topic=154131&action=.xml - that's in XML.

Actually the URL seems like an XML list for the whole Meta board, not this thread.

Is what you're saying that you need an RSS feed for an individual thread? Someone else actually asked about that the other day on here. Was that you?

Yes, I need an RSS feed for a specific thread.
No, I did not ask you about that earlier.

[Logik]

Logik,
thank you for working on your API.

For example here is the endpoint for this thread https://bitcointalk.org/index.php?topic=154131&action=.xml - that's in XML.

Actually the URL seems like an XML list for the whole Meta board, not this thread.

Is what you're saying that you need an RSS feed for an individual thread? Someone else actually asked about that the other day on here. Was that you?

Yes, I need an RSS feed for a specific thread.
No, I did not ask you about that earlier.

Hmm you're right about the link I posted. SMF developers confuse me: they implement a nice API for boards and not for topics..

I added post ids to the end points. Anyway I had an idea about how to implement RSS for threads, I might try it out tomorrow. My idea was to have a bot account subscribe via email to the thread, then have an email hook that receives the email notification and then updates the database which the RSS feed points to. I just don't know how scaleable that would be. I wonder if there is a subscription limit.

[dansmith]

Logik, before you go any further, just wanted to let you know that I've found a nice workaround:

https://bitcointalk.org/index.php?topic=154131;sa=recent;limit=255;action=.xml;type=rss

Will give 255 recent feeds for the board. Then I will add a Thunderbird filter to filter out only the topics I need.
I guess it will work for now. If I set my TB to check the feed, every 30 mins., it is unlikely that more than 255 new posts will be made in that board within 30 mins and so it is unlikely that those 255 posts within 30 mins will drown the post that I need.

The drawback (maybe) is additional strain on the server.

[Logik]

Yeah thats the closest we can get I suppose. I need to make a generic way to 'listen' to threads anyway. I'm going to test out the email approach soon.

[dansmith]

@theymos,
May I suggest that my workaround in #30 be displayed prominently for those who sign up for the forum?
Maybe put it into their first Welcome PM message.

I remember my own frustration with keeping track of multiple threads.
Now I have 20+ threads that I'm watching sitting nicely in my Thunderbird.

[🏰 TradeFortress 🏰]

Is it possible to get a particular message?

[whiskers75]

Is it possible to get a particular message?

ATM, you just have to go through the threads to find it.

(If I'm right, quote me - TradeFortress is ignoring me right now)
Edit: here:

Sorry, posts are currently only viewable by requesting a topic/page combination. Did you need a way of accessing a post via its BitcoinTalk message id (like when you link to an individual post)? Or would any id do?

Quote me!

[dexX7]

Great! Good work!

I'm still wondering, if there is any way to get "recent posts by user" in a more parsable format.

So far I was unable to add wap2 or xml to the list of user posts, but I never found out, if I just used the wrong syntax or if this is not possible. Any hint would be appreciated.

[kikkoman99]

I know this is a very old topic but is there any official documentation for a BCT API since this post or is what is linked in this thread the most up to date? Thanks!

[mprep]

I know this is a very old topic but is there any official documentation for a BCT API since this post or is what is linked in this thread the most up to date? Thanks!

There is no official Bitcointalk API.