Talksearch.io - Advanced Bitcointalk Search Engine

[NotATether]

PSA: We are not affected by the Amazon Web Services global outage. Our infrastructure is hosted on Google Cloud and on dedicated providers.

Cheers!

[examplens]

I used TalkSearch a little, I would suggest two improvements if possible. It is related to the search result.
For example, what I miss is the sorting of search results by creation date and by the date of the last post in the topic. I'm not sure how the algorithm decides the order of the prints, but it seems that they are thrown randomly.

Also, if possible, separate or at least mark archived or locked threads. It took me a long time to search and check each link, and many were just archived or locked, which was useless to me.

[NotATether]

I used TalkSearch a little, I would suggest two improvements if possible. It is related to the search result.
For example, what I miss is the sorting of search results by creation date and by the date of the last post in the topic. I'm not sure how the algorithm decides the order of the prints, but it seems that they are thrown randomly.

Also, if possible, separate or at least mark archived or locked threads. It took me a long time to search and check each link, and many were just archived or locked, which was useless to me.

I hope to finally add vector embedding search within the next few days. It would require a complete re-index though since I have to add new fields.

Also the second request might be a good idea. I return the entire post data to the frontend anyway so it would be possible to do such a thing.


The bitcointalk scraper is being restarted, in order to use a lower post search threshold which is expected to index posts twice as fast. There may be a minor disruption in indexing during this operation, but it is expected to be immediate.

Update 14:08:00 UTC: as expected, the ML server is causing a few errors with Talksearch that are preventing searches from completing successfully. Please be patient while I fix these.

Update 14:37:00 UTC: All errors have now been fixed. Working on vectorizing all the posts now. I really should've ordered this thing with 64GB memory  

[NotATether]

I'm still working on adding ML features to the search queries, but first I am scanning through the entire post index to identify and trash deleted posts. Talksearch does not currently detect when a post is deleted so I have to do this manually.

[NotATether]

Community: What is the fastest way to scan the forum to check for deleted posts given a set of post & topic ID pairs?

I'm not interested in edited posts, only deleted posts.

To check over 50 million posts, at an average speed of 1 post per 2.64 seconds, it will take 132 million seconds or over four years.

I need to do a one-time scan after I downloaded a post set which may or may not include deleted posts some time ago, and obviously I can't wait that long.

Even if I checked checked 20 posts per page, that would still take 2.5 months assuming no downtime.

I would like to be able to query all this information taking only several days, or at worst a few weeks. These times are way too long for me.

[LoyceV]

Community: What is the fastest way to scan the forum to check for deleted posts given a set of post & topic ID pairs?

First scrape all board lists, that gives you a list of all topics that haven't been deleted yet. Then scrape all topics.

To check over 50 million posts, at an average speed of 1 post per 2.64 seconds, it will take 132 million seconds or over four years.

You're allowed one page request per second, so waiting 2.64 seconds isn't necessary. Maybe you can use "All" for topics with no more than 26 pages to get up to 500 posts at once, but Cloudflare will probably stop you from doing that.

Even if I checked checked 20 posts per page, that would still take 2.5 months assuming no downtime.

It took me several months (years back), but that included scraping non-existing topics because I hadn't thought of scraping the boards first.

I would like to be able to query all this information taking only several days, or at worst a few weeks.

Does it help to prioritize boards? Forget about the altcoin bounty boards, that should take off millions if not tens of millions of posts.

[NotATether]

You're allowed one page request per second, so waiting 2.64 seconds isn't necessary. Maybe you can use "All" for topics with no more than 26 pages to get up to 500 posts at once, but Cloudflare will probably stop you from doing that.

It appears twice as slow because the actual indexer is also running in parallel. I just ran the deletion script on top of it and it's also running at the same speed.

Cloudflare would work, but I would have to embed a token of some kind.

Does it help to prioritize boards? Forget about the altcoin bounty boards, that should take off millions if not tens of millions of posts.

Not really. The deleted posts occur randomly across any board (though I suspect the gambling discussion board has a higher proportion of deleted posts).

[LoyceV]

The deleted posts occur randomly across any board (though I suspect the gambling discussion board has a higher proportion of deleted posts).

Have you considered modlog as a first hint of where to look for deleted posts? I don't think there's a foolproof way to catch all deleted posts. You could also prioritize more recent topics over older ones: there's no need to check this topic from 2010 every few weeks for deleted posts.

[NotATether]

Have you considered modlog as a first hint of where to look for deleted posts? I don't think there's a foolproof way to catch all deleted posts. You could also prioritize more recent topics over older ones: there's no need to check this topic from 2010 every few weeks for deleted posts.

That may not be necessary anymore - theymos suggested I should use the sitemap.xml page. It's a little convoluted, but I managed to hack together a script that will check for deleted/updated posts.

There is one big pass I have to make first, before I can revert to tiny passes from the past day or two.

[LoyceV]

theymos suggested I should use the sitemap.xml page.

Is that https://bitcointalk.org/sitemap.php? I've seen it before, but wasn't sure how to use it (and later on couldn't find it back).

Note:
I can't post this the way I wanted, Bitcointalk turns it into the above:
[url=https://bitcointalk.org/sitemap.php]bitcointalk.org/sitemap.php[/url]?

Even code tags can't post the nobbc-code correctly, the above turns into this:

Code:

Is that https://bitcointalk.org/sitemap.php?

[NotATether]

theymos suggested I should use the sitemap.xml page.

Is that https://bitcointalk.org/sitemap.php? I've seen it before, but wasn't sure how to use it (and later on couldn't find it back).

Yep, that's the one. Basically, he explained to me that the sitemap is organized into many smaller sitemaps, presumably to avoid generating a single giant sitemap. The main sitemap.php contains topic/page boundaries, where p= and o= denote the topic and page inside the topic (such as .20) respectively.


The inner sitemap XML looks like this

Code:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://bitcointalk.org/index.php?topic=178336.703420</loc>
<lastmod>2025-11-18T20:07:14+00:00</lastmod>
<changefreq>hourly</changefreq>
<priority>0.568</priority>
</url>
<url>
<loc>https://bitcointalk.org/index.php?topic=178336.703440</loc>
<lastmod>2025-11-19T02:41:11+00:00</lastmod>
<changefreq>hourly</changefreq>
<priority>0.412</priority>
</url>
<url>
<loc>https://bitcointalk.org/index.php?topic=178336.703460</loc>
<lastmod>2025-11-19T08:24:00+00:00</lastmod>
<changefreq>hourly</changefreq>
<priority>0.418</priority>
</url>
<url>
<loc>https://bitcointalk.org/index.php?topic=178336.703480</loc>
<lastmod>2025-11-19T15:14:25+00:00</lastmod>
<changefreq>always</changefreq>
<priority>0.478</priority>
</url>
<url>
<loc>https://bitcointalk.org/index.php?topic=178336.703500</loc>
<lastmod>2025-11-19T17:59:25+00:00</lastmod>
<changefreq>always</changefreq>
<priority>0.7</priority>
</url>
...

From this data, the most important are the loc and the lastmod. These basically tell you whether a page has been edited or deleted. Deleted posts will cause subsequent pages to appear modified as well.

I guess the other two parameters can be used to gauge how frequently to check for updates, but I don't use those in my current implementation - I am performing a full sweep over the sitemap first.


This process is also causing regular indexing to slow down because I am throttling the indexer speed in order to avoid breaching the rate limit. New posts should now appear instantly though (but only because I am using the recentposts page, nothing to do with the sitemap). But edits will appear to be much slower until I finish crawling through the pages linked by the sitemap.

[NotATether]

Bump (I guess a lot of people are still using forum search. I'm purging deleted posts as fast as I can.)

I'm starting to think it is a bad idea to boost search terms in titles. I'm going to adjust it to consider only post bodies. And make a new internal field for posts containing "effective title" that is only set for the OP, and concatenation they with the post body whenever someone searches for something.

[yhiaali3]

Hello @NotATether
First of all, thank you for the wonderful work and great efforts you have put in so far in developing this advanced Search Engine for BitcoinTalk.

I didn't see the Arabic translation among the translations, so I am honored to contribute to this effort through the Arabic translation I did at the following link:
https://bitcointalk.org/index.php?topic=5567569

If you intend to translate the site into multiple languages, I would also be happy to participate in translating it into Arabic.

[NotATether]

Hello @NotATether
First of all, thank you for the wonderful work and great efforts you have put in so far in developing this advanced Search Engine for BitcoinTalk.

I didn't see the Arabic translation among the translations, so I am honored to contribute to this effort through the Arabic translation I did at the following link:
https://bitcointalk.org/index.php?topic=5567569

If you intend to translate the site into multiple languages, I would also be happy to participate in translating it into Arabic.

Thanks brother.

I wanted to translate it to Arabic myself but I never found the time to.

So this is very helpful.


Sitemap update/delete is still going strong.

[yhiaali3]

Hello @NotATether
First of all, thank you for the wonderful work and great efforts you have put in so far in developing this advanced Search Engine for BitcoinTalk.

I didn't see the Arabic translation among the translations, so I am honored to contribute to this effort through the Arabic translation I did at the following link:
https://bitcointalk.org/index.php?topic=5567569

If you intend to translate the site into multiple languages, I would also be happy to participate in translating it into Arabic.

Thanks brother.

I wanted to translate it to Arabic myself but I never found the time to.

So this is very helpful.


Sitemap update/delete is still going strong.

This is my pleasure, brother NotATether@
I've been following your wonderful efforts so far, and you're still developing it. So, when all the development is complete, just let me know and send me the language file so I can translate it into Arabic immediately.

I also hope you'll update the OP to include the Arabic translation in the main thread.
Thank you.

[NotATether]

Already the mass edited/deleted posts updater using the sitemaps has refreshed 100k threads (a pittance when you take note that there's over 5 million topics, but we're getting there!)

Here are some logs from this operation, which runs simultaneously with the main indexer by the way:

Code:

🔄 Starting real-time scrape and ingest for topic 101198 (mode: update)
📄 Topic: [WTS] MTGOX USD
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1

2025-12-09 01:16:02,987 - core.scraper - INFO - Topic 101198 has 1 pages
2025-12-09 01:16:16,389 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 1 posts | 📋 Queue: 1 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
    ✅ Uploaded 1 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 1 | 📦 Batches: 1

📊 Topic 101198 Summary:
  ✅ Successfully uploaded: 1 posts
  ❌ Failed uploads: 0 posts
  📈 Success rate: 100.0%

⏱️  Waiting 1.0s before next topic...

🔄 Processing topic 16767/20000: 101199
🔄 Starting real-time scrape and ingest for topic 101199 (mode: update)
📄 Topic: "Mintchip is designed to track you" claims anonymous insider
📁 Board path: Bitcoin Forum > Alternate cryptocurrencies > Altcoin Discussion
📖 Pages: 1

2025-12-09 01:16:17,761 - core.scraper - INFO - Topic 101199 has 1 pages
2025-12-09 01:16:31,183 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 3 posts | 📋 Queue: 3 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
    ✅ Uploaded 3 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 3 | 📦 Batches: 1

📊 Topic 101199 Summary:
  ✅ Successfully uploaded: 3 posts
  ❌ Failed uploads: 0 posts
  📈 Success rate: 100.0%

⏱️  Waiting 1.0s before next topic...

🔄 Processing topic 16768/20000: 101200
🔄 Starting real-time scrape and ingest for topic 101200 (mode: update)
📄 Topic: VPN that works in China
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1

2025-12-09 01:16:32,581 - core.scraper - INFO - Topic 101200 has 1 pages
2025-12-09 01:16:46,006 - core.scraper - INFO - Reached detected total pages (1), stopping pagination
📦 Uploading final batch of 5 posts | 📋 Queue: 5 waiting | 📤 Uploaded: 0 | 📦 Batches: 0
    ✅ Uploaded 5 posts | 📋 Queue: 0 waiting | 📤 Uploaded: 5 | 📦 Batches: 1

📊 Topic 101200 Summary:
  ✅ Successfully uploaded: 5 posts
  ❌ Failed uploads: 0 posts
  📈 Success rate: 100.0%

⏱️  Waiting 1.0s before next topic...

🔄 Processing topic 16769/20000: 101201
🔄 Starting real-time scrape and ingest for topic 101201 (mode: update)
📄 Topic: Anyone feel like lending me 120 BTC ?
📁 Board path: Bitcoin Forum > Economy > Marketplace
📖 Pages: 1

2025-12-09 01:16:47,330 - core.scraper - WARNING - Server error (attempt 1), using adaptive delay: 503 Server Error: Service Unavailable for url: https://bitcointalk.org/index.php?topic=101201.0

I also hope you'll update the OP to include the Arabic translation in the main thread.

It will be added soon!

[LoyceV]

Already the mass edited/deleted posts updater using the sitemaps has refreshed 100k threads (a pittance when you take note that there's over 5 million topics, but we're getting there!)

Last time I checked (2 months ago), I counted only 1,415,773 topics (102 MB txt file). Almost 75% is deleted.

[Halab]

Last time I checked (2 months ago), I counted only 1,415,773 topics (102 MB txt file). Almost 75% is deleted.

I began to dare to doubt these figures. But I forgot MindlessElectron, which fights against other spamming bots.
The percentage of topics deleted by human mods must be much lower.

[NotATether]

Already the mass edited/deleted posts updater using the sitemaps has refreshed 100k threads (a pittance when you take note that there's over 5 million topics, but we're getting there!)

Last time I checked (2 months ago), I counted only 1,415,773 topics (102 MB txt file). Almost 75% is deleted.

Right. I forgot that I ran a query on my internal Elasticsearch node some time ago for the number of unique topics, and got similar to this number too.

This significantly cuts down on runtime! I still have to search for and remove any deleted topics post-2020 though (when I loaded your archive into the storage).