Talksearch.io - Advanced Bitcointalk Search Engine

[NotATether]

Talksearch.io

It has always been a dream of mine to create a high-quality search engine for Bitcointalk. After many months of development, I am proud to announce that Talksearch is now generally available. This is an important milestone towards providing users with high-quality search.

Talksearch is a simple search engine that allows you to quickly find and go to any posts on Bitcointalk.

 Features ____________________

- Indexed ALL posts (up to March 2025)

• Deleted and hidden posts in Investigations are not supposed to be indexed. If you see any, report them to me with the post ID and topic ID.

- Superior performance to Bitcointalk's builtin search.
- Tor-friendly, lacking rate limits and captchas.
- Detailed post metadata such as user, date, and topic title.
- Eliminates spammy results by enforcing basic length restrictions.

And will eventually have more accurate results than either.

 Infrastructure ____________________

Talksearch is hosted on Google App Engine, utilizing several Compute Engine servers to run Elasticsearch. It has a capacity to host up to 180GB of posts. Backups are available off-site in case of disaster. None of this is cheap, and it costs around $170/month in total to maintain.

 Roadmap ____________________

- Continuously ingest posts from Bitcointalk (immediate priority)
- Refine search result quality
- Introduce filters for user, date, etc. Done

Posts do not update automatically yet. I am working on that.

I will continuously optimize the quality of results on the index to achieve the best possible results. For now though, you may need to use multiple words to filter relevant posts.

 Donate ____________________

If you wish to support the maintenance of Talksearch, you can send funds to the following addresses:

Bitcoin: bc1q6dphprljdas0xl2cmqn6tlskselx5xtcpcw8kx
Ethereum and ERC-20 tokens: 0xd3CaaE5098b8Bef64A6FD415b0b1B61aE880FFF5
Tron and USDT-TRC20: TGrsWW6knTwcJxkUKvp7SoV5Fjub9KMAeb

Donations will only be used to pay for hosting costs.

 Translations ____________________

• Indonesian by taufik123
  
• German by cygan
  
• Russian by satscraper
  
• Pidgin by Abdulzuruku01
  
• Croatian by katanic97
  
• Urdu by Adiljutt156
  
• Polish by cygan
  
• Turkish by mela65
  
• Portuguese by r_victory
  
• Romanian by GazetaBitcoin
  
• Filipino by Danica22
  
• Spanish by Porfirii
  
  Thanks to all the people who made translations!
  
  Disclaimer - your searches are not private. Talksearch keeps logs of queries in Google Cloud for debugging and troubleshooting purposes. If this is a turn-off for you, use DuckDuckGo instead.

[NotATether]

 Changelog ____________________

Code:

2025-06-16: App v1.0.5 - https://bitcointalk.org/index.php?topic=5536692.msg65488039#msg65488039
2025-06-13: App v1.0.4 - https://bitcointalk.org/index.php?topic=5536692.msg65477942#msg65477942
2025-06-12: App v1.0.3 & Search v1.0.2 - https://bitcointalk.org/index.php?topic=5536692.msg65475066#msg65475066
2025-04-06: App v1.0.2 - https://bitcointalk.org/index.php?topic=5536692.msg65248252#msg65248252
2025-04-03: App & Search v1.0.1 - https://bitcointalk.org/index.php?topic=5536692.msg65238342#msg65238342
2025-03-29: App & Search v1.0.0 - Initial release

[NotATether]

 Operational Expenses ____________________

This section is reserved for screenshots of the Hosting Provider invoices, with sensitive info redacted. The first one for May will be published in a few days.

Additionally, technical specs of the search cluster will be posted here, along with the resource usage and high-level statistics related to the indexed post data.

This information is being published for accounting and transparency purposes as this is being operated as a public service to the forum.

May 2025 invoice - $179.76
Donations March-May 2025 - 2x - $110
Net Cost May 2025: $69.76

Total Running Cost: $69.76

[shahzadafzal]

None of this is cheap, and it costs around $150/month in total to maintain.

$150/month? That’s expensive!

However, it looks good at first glance—definitely faster, and I think the filters will be helpful.

Some basic functionalities would make it even more useful. For example, there’s no option to edit the search text, which is essential for refining searches.

I’ll explore it further.

[NotATether]

Some basic functionalities would make it even more useful. For example, there’s no option to edit the search text, which is essential for refining searches.

You mean like a search bar above the results page?

I can get that added quickly.

Edit: done

[arhipova]

Is Google App Engine a paid host to join ?

[NotATether]

Is Google App Engine a paid host to join ?

It's part of Google Cloud. They offer a free trial of 3 months and $300 in credit, but you must add a debit card to use it at all. (no VCCs allowed).

App Engine is pretty cool - you make it scale down to zero instances if nobody is using it, which hopefully does not happen here, so that you pay for nothing.

It's not really a VM though. You can only make use of it if you know how to code.

[TryNinja]

Talksearch is hosted on Google App Engine, utilizing several Compute Engine servers to run Elasticsearch. It has a capacity to host up to 425GB of posts. Backups are available off-site in case of disaster. None of this is cheap, and it costs around $150/month in total to maintain.

What... are you serious? Why on earth are you paying this much?

For the ninjastic.space database I pay around $26 per month. That gives me more than enough to self host my elasticsearch node and a shit ton of other projects, and searching is very fast (see the new beta.ninjastic.space/search). Believe me, you do *not* need to pay for "several compute engine servers" for your project. That's complete insanity.

[NotATether]

What... are you serious? Why on earth are you paying this much?

For the ninjastic.space database I pay around $26 per month. That gives me more than enough to self host my elasticsearch node and a shit ton of other projects, and searching is very fast (see the new beta.ninjastic.space/search). Believe me, you do *not* need to pay for "several compute engine servers" for your project. That's complete insanity.

Wasn't my choice, I had an option to buy the cluster for $36/month without the extra storage server, but it would only fit half of the posts at 45GB. This was the next smallest config.

Very crazy prices going on at cloud providers.

[TryNinja]

Wasn't my choice, I had an option to buy the cluster for $36/month without the extra storage server, but it would only fit half of the posts at 45GB. This was the next smallest config.

Very crazy prices going on at cloud providers.

Why not move to a VPS or at least your own dedicated server?

I get 1.5 TB of NVME for the price I pay.

If you need space (you do, because you're indexing tons of posts), you're on the wrong service. That's the price huge companies pay because they can, and they usually need the extreme scaling possibility due to the nature of their business.

[KingsDen]

The best search tool I have seen since I joined BTT. I did searches of a few topics I created and the result returned accurately. If there is a means to pin this topic at the menu bar of the forum, I would so much appreciate. Kudos for this exceptional work.

- Eliminates spammy results by enforcing basic length restrictions.

Like short posts are not indence, if yes, to what length?

[DYING_S0UL]

Just a little thought inside my mind! Wouldn't it be better if only one result were shown per topic, with an expandable slider to see the rest of the replies inside that topic? What I mean is, when a search query is made, and it is showing every single reply from a specific topic (not ever grouped or sorted or numbered). Just look at the image. All of the results I was shown belonging to the same topic.

Maybe you could make some changes and make it look more sorted or grouped? Or the topic ID is shown below the username & date?

Btw, the UI is very clean, maybe you can consider adding some new theme. For example pitch black, similar to the new ninjastic.

Sorry, I am having a little trouble choosing the right words to express my thoughts/suggestion/idea. Do let me know if I was confusing.

[KingsDen]

Just a little thought inside my mind! Wouldn't it be better if only one result were shown per topic, with an expandable slider to see the rest of the replies inside that topic? What I mean is, when a search query is made, and it is showing every single reply from a specific topic (not ever grouped or sorted or numbered).

Nice idea but it will lead to complex sorting. I feel that what we have is already good since you might not need to scroll down to see your desired.

Just look at the image. All of the results I was shown belonging to the same topic.

That topic has the major concentration of the search key words.

[NotATether]

Why not move to a VPS or at least your own dedicated server?

I get 1.5 TB of NVME for the price I pay.

If you need space (you do, because you're indexing tons of posts), you're on the wrong service. That's the price huge companies pay because they can, and they usually need the extreme scaling possibility due to the nature of their business.

I actually had a dedicated server rented out for this specifically at one point, with very capable specs (and 33% of the cost). But sysadmin is my pet peeve - I usually cause a lot of downtime during things like updates which would be unacceptable for a production service like this.

I ended up mining some XMR on it while debating whether or not I should use the cloud, until about a week ago when I canceled the box.

As for my main box, well I already have a warning from the hosting provider from last year not to "send abusive port 80 traffic" again (apparently one of my HTTP services was hacked and was being used for DDOS) so I don't dare host any websites on it anymore.

edit: typo

Just a little thought inside my mind! Wouldn't it be better if only one result were shown per topic, with an expandable slider to see the rest of the replies inside that topic? What I mean is, when a search query is made, and it is showing every single reply from a specific topic (not ever grouped or sorted or numbered). Just look at the image. All of the results I was shown belonging to the same topic.

This is a defect in the algorithm I'm using. It weights topics much more than post content which is why you see so many posts from the same thread together even though they may not be relevant themselves.

[Pablo-wood]

Why not move to a VPS or at least your own dedicated server?

I get 1.5 TB of NVME for the price I pay.

If you need space (you do, because you're indexing tons of posts), you're on the wrong service. That's the price huge companies pay because they can, and they usually need the extreme scaling possibility due to the nature of their business.

A person dedicated server should be a better choice but won't it be as expensive as compared to rented dedicated servers?.  Although it might just require a one time fee but the cost might be discouraging. A Virtual private server should just be the best because it's cheaper and more scalable but the issue is with  it's dedicated allocation, it might be another pull back.

Been on the wrong server I agree, even though DS are quite expensive, $150 is much. A range from $36 - $50 should be fair enough

[DYING_S0UL]

Just a little thought inside my mind! Wouldn't it be better if only one result were shown per topic, with an expandable slider to see the rest of the replies inside that topic? What I mean is, when a search query is made, and it is showing every single reply from a specific topic (not ever grouped or sorted or numbered). Just look at the image. All of the results I was shown belonging to the same topic.

This is a defect in the algorithm I'm using. It weights topics much more than post content which is why you see so many posts from the same thread together even though they may not be relevant themselves.

So do you plan on changing the algorithm, maybe in the future or until you figure out an alternative solution?! For now I think it is good enough, not gonna lie. But I guess, at least maybe you can add the main topic ID below the usernames of every results (of course in a serial manner for the same topic). That would greatly help in identifying that this content belongs to the same topic! I assume you understood what I was trying to imply here!

[Mrbluntzy]

Talksearch is a simple search engine that allows you to quickly find and go to any posts on Bitcointalk.

What if I don't remember the whole tittle of the topic that I want to search for and maybe I only remembered a few key words on the topic, when searching on this website with just those few key words, will it show results for other topics that has same key words?
On the forum search, even if I don't remember the topic am looking for correctly, I could just type the few key words that I remembered and it will bring up several topics on that and I will have to scroll and keep nexting until I probably see the right topic am looking for.

[Upgrade00]

The results at the moment look to be only focused on topics, showing those with the key words selected. Is there a plan to index text within the post content too?

On the page number slider, one has to move one at a time forward or backwards, can there be an option to jump to the specific page you want to check out? If it gets too long, it can be trimmed out.

Website looks good after some tries, lightning quick, I don't think I've used any that returns to the homepage as fast. Results load very quickly too.

[NotATether]

The results at the moment look to be only focused on topics, showing those with the key words selected. Is there a plan to index text within the post content too?

Post content is indexed but the results are heavily weighted towards titles at the moment.

On the page number slider, one has to move one at a time forward or backwards, can there be an option to jump to the specific page you want to check out? If it gets too long, it can be trimmed out.

Talksearch will only return 10 pages max, so it would be better if I made them all clickable. This will be done later, as I am on holiday tomorrow.

What if I don't remember the whole tittle of the topic that I want to search for and maybe I only remembered a few key words on the topic, when searching on this website with just those few key words, will it show results for other topics that has same key words?
On the forum search, even if I don't remember the topic am looking for correctly, I could just type the few key words that I remembered and it will bring up several topics on that and I will have to scroll and keep nexting until I probably see the right topic am looking for.

A search engine's #1 job is to help you find stuff, so naturally you can type a few key words that occur in the title and get it returned to you, if you type like 3-4 of them.

The term doesn't have to be an exact match as it searches by word. Also it is not case-sensitive.

One advantage Talksearch has over the Bitcointalk search is that results are not returned in random order.

[Vod]

Congrats OP - although the results are a bit vanilla for me (I like a lot of condensed information), I'm sure you'll modify it over time based on suggestions.    Are you planning on having Theymos fund it or putting paid links in the results?   (The forum has over 1,000 btc donated for this purpose, so regular users should not be supporting this.)

If you are interested in having partners send you traffic, shoot me a line.   

From one developer to another - the rush of seeing something you created being used is better than any man made drug.