Bitcointalk Search Project

[NotATether]

@Bittr.io thanks for the feedback.

Any API I create for this would have to be paid though, and have some sort of credits system in place. Btcointalk hosts a lot of data, and if I do not control how fast people can download it, then I will quickly be overran by accidentally DDoS from various sites, userscripts and maybe even apps if someone decides to make one off of it.

I've got some good news on the irmattig part though - I'm finally preserving that information using the new algorithm.

[TryNinja]

Using Zendriver, we can scrape two or three pages per second, without any selenium-enforced delay.

I implemented an exponential backoff for when the forum server returns 503 errors, starting from one second and multiplying by two for each attempt.

With these changes, the scraper has become 10x faster, and continues to make the mission of organizing the forum's posts more feasible.

Once most of the posts are saved, the server can be queried at much longer intervals.

https://streamable.com/npfr6r

Any reason why you're scraping the forum with headless browsers (zendriver, selenium, etc) instead of using pure http requests? The forum doesn't have any anti-bot measures or heavy javascript, so that's not a problem. Requests >>> browser drivers. Much less memory usage, faster, can scale up to infinite if you use proxies (theymos wouldn't like that, though).

Off-topic: which terminal is that?

[Vod]

@TryNinja - sorry for forgetting your name in my "post archiver" comment, it wasn't intentional. 

OP - what parameters are you collecting in each of your posts?
- Original thread title
- thread author  (do you store by username or usernumber?)
- post title
- post author
- contains quotes
- UTC time/date
- category
- subcategories
- partial url of post

Also, what database engine are you using?  I want to make sure I can use your data when it's complete. 

[NotATether]

An update: I've acquired a copy of all posts up to January 2025 and am now in the process of deleting the removed posts.

Any reason why you're scraping the forum with headless browsers (zendriver, selenium, etc) instead of using pure http requests? The forum doesn't have any anti-bot measures or heavy javascript, so that's not a problem. Requests >>> browser drivers. Much less memory usage, faster, can scale up to infinite if you use proxies (theymos wouldn't like that, though).

Off-topic: which terminal is that?

That's Konsole in dark mode, running the 'tmux' multiplexer.

At first, I was utilizing the Javascript DOM that's built in to Chrome to remove the quote and code tags from all the posts, but eventually I moved away from that because 1) the DOM API is very unreliable and caused frequent crashes and 2) this data could end up being useful for another case case and can simply be filtered out later.

I must admit though, I resorted to getting a list of "alive" topics from LoyceV's website, then ran the topics through your API to get the posts in JSON. It allowed me to get all of the posts really fast without hitting the forum's rate limiter all the time <which turned out to be a really big problem when scraping directly>. Now though, for updates it will be no issue querying the forum itself, even with one IP address.

@TryNinja - sorry for forgetting your name in my "post archiver" comment, it wasn't intentional. 

OP - what parameters are you collecting in each of your posts?
- Original thread title
- thread author  (do you store by username or usernumber?)
- post title
- post author
- contains quotes
- UTC time/date
- category
- subcategories
- partial url of post

Also, what database engine are you using?  I want to make sure I can use your data when it's complete. 

I have all of these parameters actually. It's just a copy of TryNinja's data and I plan to modify my scrapers to use the same names.

It's just JSON data, so not easy to load into a database at all. I plan to put it inside Elasticsearch. Now I will start the cluster and once the empty topics are weeded out I can begin the load procedure.

[LoyceV]

~ now in the process of deleting the removed posts.

Why not keep them for searches? Just mark them as "removed".

[NotATether]

Why not keep them for searches? Just mark them as "removed".

Because if you have a list of topics say from 1 to 5,000,000, in file format, then I can easily handle deleted topics by checking if the file exists on my backend.

If they exist then it's live, otherwise it was nuked at some point.

Also makes for less inodes to deal with, which is good when I have to move them around.

This is not really an archive, I see this project more as a Google, but more precise for this forum.

[NotATether]

I am going to have a proper ANN release very soon - stay tuned.

[NotATether]

All data up to February has been ingested - currently refining the search engine parameters now.

It'll take some time, and the content block in particular have to be parsed carefully, but I'm confident that I can get better results on this thing than the current html soup.



Did you know? Almost 2% of all Bitcointalk posts start with "#proof of authentication".

[LoyceV]

Did you know? Almost 2% of all Bitcointalk posts start with "#proof of authentication".

Years ago, I counted "Twitter" and "Facebook" in posts. Those numbers were in the millions.

[NotATether]

Did you know? Almost 2% of all Bitcointalk posts start with "#proof of authentication".

Years ago, I counted "Twitter" and "Facebook" in posts. Those numbers were in the millions.

I think it really peaked between 2017-2019. There's a giant bell curve at the top where you're supposed to see the date distribution of the posts, but I messed up and sorted by the wrong date key so it's not visible in the image.

Although I bet they are still posted moderately frequently even to this day.

[LoyceV]

I think it really peaked between 2017-2019. There's a giant bell curve at the top where you're supposed to see the date distribution of the posts, but I messed up and sorted by the wrong date key so it's not visible in the image.

All posts peaked around that time. It still feels like a massive waste to have tens of millions of completely pointless posts archived.

Although I bet they are still posted moderately frequently even to this day.

I hadn't look at it for a while, but it looks like it's much less now. The bounty board has much less megathreads, Show unread posts since last visit no longer has bounty threads on top of the list, and the 1000 latest (new) user's first posts mention "Facebook" in less than 1% of the posts. "Twitter" still has 100 matches, and "x.com" 132 (but the post count isn't that high as they all post a list of URLs).
Now that I think about it: even my recent report list is getting very short. Maybe there really are less bad posts

[NotATether]

It still feels like a massive waste to have tens of millions of completely pointless posts archived.

I think so too, but Elasticsearch will chuck them into its "not indexed" tier by itself, as people search for other posts.

The most searched-for posts can be cached somewhere on the server for faster lookup times.

[NotATether]

Talksearch is now generally available.

See the announcement topic for reference.

[NotATether]

Now that Talksearch is public, it doesn't make any sense to have this discussion topic anymore, so I am locking it.