Talksearch.io - Advanced Bitcointalk Search Engine

[NotATether]

An update to the enhanced search feature:

A new dataset is being uploaded to Elasticsearch. This dataset is more enriched than the current unprocessed posts and includes even more metadata such as the lock type, scrape time and check time, which will be used along with other parameters to determine in what order should topics be checked for updates and the frequency they will be checked.

An experimental quality score is also included with each post, in an attempt to deprioritize low-quality posts and sig spam from the search results.

In an effort to remove irrelevant data such as quotes from the search results, posts are now divided into chunks, delimited by the presence of a quote or a line separator.

This upload process was started yesterday, and about 5 million records have been indexed so far, out of a total estimated to be around 120 million.



The v2 indices contain the data which Talksearch will use for searching in the future. Also, local language posts are categorized to facilitate for local search.

I continue to work on automatic scraping support. However, the v2 dataset is more recent than the original, and contains posts from up to March 2025.

New translated ANN links will be added shortly.

[BitcoinGirl.Club]

http://talksearch.io/search?q=bitcoingirl

Taking ages to load. Is the search engine working?

[joker_josue]

This upload process was started yesterday, and about 5 million records have been indexed so far, out of a total estimated to be around 120 million.

That's a lot of data.

Is the system cataloging all the words and things like that? How are you processing all the information?

[LoyceV]

An experimental quality score is also included with each post, in an attempt to deprioritize low-quality posts and sig spam from the search results.

I'm curious to see the posts per user sorted by your experimental algorithm. Would that be possible to search for?

about 5 million records have been indexed so far, out of a total estimated to be around 120 million.

Is 120 million the number of posts + edits?

[NotATether]

http://talksearch.io/search?q=bitcoingirl

Taking ages to load. Is the search engine working?

The warm tier of nodes is slower than the hot/content tier when fetching data, but has about 4x more storage capacity.

Currently, v1 (talksearch_bitcointalk in the picture) search is running on warm nodes. It used to be on the hot nodes, where searches were quite fast, but in the process of fixing my cluster, it got moved to warm.

All v2 indices besides English are on the hot nodes, however I'm not particularly satisfied with the amount of low-quality posts present in this index, so I'm considering moving the high-quality Engilsh posts to the hot nodes. Then there would be a checkbox on the site that reads "Only search high-quality posts".

The issue is, I currently don't have a reliable way to measure post quality.

By the way, http:// does not currently work on Talksearch. Use https://. I am thinking about redirecting all traffic to the https:// version anyway.

That's a lot of data.

Is the system cataloging all the words and things like that? How are you processing all the information?

Yes!

In fact I am excited to show you the advanced classifications that are available for the data.

Elasticsearch has a number of field types available for naturally processing JSON, more than just strings, numbers, and booleans. I am talking about things like rank features (common in search applications), vectors, points, geolocation stuff, dates, binary data, and many other stuff:



These are then used by Elasticsearch's query language. It is very advanced and can find stuff much more relevant than e.g. a regex-based search. This is where the true power lies.

Here's a list of things that its query language is able to do:

- It can boost or penalize certain keywords
- It supports semantic search e.g. "satoshi nakamoto identity" is supposed to return topics about Hal Finney or Nick Szabo
- Apply autocorrect and "keyboard shift"
- It can match text that only appears in a certain position in the post
- It can find "More like this" results
- You can attach scripts to create complex searches (but this is slow)
- Feature-based search is fully supported, so you can make queries for posts based on username, board, topic title, date etc in the exact same manner as for post body. (This last part is sorely missing from Google Site Search.)

There's a lot more stuff I didn't list that you can find here. But the exact algorithm I use is proprietary.

I'm curious to see the posts per user sorted by your experimental algorithm. Would that be possible to search for?

That's a great idea, but with the current version of Talksearch this is not possible. It will eventually be available though.

Is 120 million the number of posts + edits?

No, 120 million is an estimate of the average number of chunks there would be if you split posts by quote body, or by line break [ hr ] tags. For example, this post I am writing would be indexed as 4 chunks, as they are separate pieces of information being written. This allows for users to see as specific results as possible from the search results page without having to navigate to the Bitcointalk post.

I assume there are on average a bit less than one quote or line break per post, hence why there's a bit less than 2 chunks on average (quotes etc. plus one).

I only store the latest revision of each post.

[dkbit98]

An update to the enhanced search feature

It is working very slow for me, and I only wrote a simple two word search term.
Maybe you should allow users to narrow down search into specific boards/topics only, that would certainly speed things up significantly.
You can also add search only for specific members to speed things even more.
I compared search using Ninjastic.space with new updated Talkserach with, and Ninjastic gave much better and faster results.

[NotATether]

Searches are temporarily unavailable while I perform some routine maintenance on the nodes.

Thanks for understanding.

Update: maintenance is complete. This should fix the search performance.

[BitcoinGirl.Club]

Update: maintenance is complete. This should fix the search performance.

It's a little faster than the last time I checked.

[NotATether]

It's a little faster than the last time I checked.

The nodes ran out of memory processing the uploaded posts, that's why performance sucked for a while. This was resolved by increasing the size of the hot/content node, and deleting the warm node.

This has cut my storage capacity by about half (I have about 180GB total disk now) and marginally increases the total cost by about $10, but it will keep the node stable for some time.

I will have to slow down the rate at which I'm uploading posts to the cluster, in order to avoid this sort of thing happening again. However, the new data is nowhere near close to 100% uploaded, so i will have to figure out a way to make that faster so that I can make Talksearch use the higher-quality v2 indices.

[SamReomo]

A big congratulations from my side OP, you've really solved one of the biggest issues of Bitcointalk but like others I also think that you're paying way more for it than needed. $150 + $10 a month is a huge amount to pay for a service like that. However, I hope that you may receive some good donations or get some good sponsors so it won't be a burden on your shoulders alone.

[GazetaBitcoin]

Hey NotATether, please be aware that 2 more translations were made for your topic by AOBT. I hope this is good news

Urdu translation, made by Adiljutt156
Polish translation, made by cygan

[PX-Z]

Good project and initiative!
@OP, could you clarify whether the $150 covers only the hosting costs, or does it also include access to Elasticsearch, since I noticed it has paid options too?

I tried accessing the site just yesterday, but it was down at the time. However, it's noticeably faster now, great improvement so far.

[nakamura12]

On the page number slider, one has to move one at a time forward or backwards, can there be an option to jump to the specific page you want to check out? If it gets too long, it can be trimmed out.

Talksearch will only return 10 pages max, so it would be better if I made them all clickable. This will be done later, as I am on holiday tomorrow.

I am sure other forum members who will use your search engine would want the same thing like to jump few messages ahead. In my opinion, it does takes time to find the exact post/topic you are looking for if you can only check one page at a time. I am sure you will make some updates or adjustments in the future (just take your time). Congratulations on your project. The cost is quite a lot so I hope you will receive some donations. Well, you can try some ads like at the top of the search text box or at the very bottom.

[NotATether]

Good project and initiative!
@OP, could you clarify whether the $150 covers only the hosting costs, or does it also include access to Elasticsearch, since I noticed it has paid options too?

I tried accessing the site just yesterday, but it was down at the time. However, it's noticeably faster now, great improvement so far.

It includes Elasticsearch (which is "self-hosted" on Google Cloud, and makes up the majority of the bill).


Edit: It appears that quote splitting over 56 million posts is going to take many days (and not a few days like I imagined at first), and this is with the processor and the document uploader running in parallel, so while we wait, I am going to prioritize adding fine-grained search fields like user, title, date to the website's search parameters.

The old v1 index that's currently being used for search can also have the remaining posts from that collection uploaded in there. That one at least shouldn't take too long, because only new or previously non-existent posts are going to be added there.

[GazetaBitcoin]

Hello again NotATether!

In addition to my previous post, I'd like to let you know that three more translations have been made by AOBT for this thread:

Turkish translation, made by mela65
Portuguese translation, made by r_victory
Romanian translation, made by myself

In this moment, this topic has 10 translations, so from AOBT perspective it is considered as "Done". However, if new translations will be made, I will let you know.

Cheers!

[NotATether]

Hello again NotATether!

In addition to my previous post, I'd like to let you know that three more translations have been made by AOBT for this thread:

Turkish translation, made by mela65
Portuguese translation, made by r_victory
Romanian translation, made by myself

In this moment, this topic has 10 translations, so from AOBT perspective it is considered as "Done". However, if new translations will be made, I will let you know.

Cheers!

Thank you.

My apologies for not adding translations to the OP yet. I am busy fighting my upload scripts to get them sending posts to Elasticsearch at an acceptable rate. For some reason, my usually fast hard disk drive on my server has slowed down massively, even after a reboot, and I'm not exactly sure why.

[LoyceV]

my usually fast hard disk drive on my server has slowed down massively, even after a reboot, and I'm not exactly sure why.

Some (shitty) disks drop in performance after long sustained writes, but since you're using Google Cloud, that shouldn't happen.

[Vod]

For some reason, my usually fast hard disk drive on my server has slowed down massively, even after a reboot, and I'm not exactly sure why.

Are your indexes normally set for reading?   You should disable them while doing massive updates.

[Porfirii]

Hello again NotATether!

In addition to my previous post, I'd like to let you know that three more translations have been made by AOBT for this thread:

Turkish translation, made by mela65
Portuguese translation, made by r_victory
Romanian translation, made by myself

In this moment, this topic has 10 translations, so from AOBT perspective it is considered as "Done". However, if new translations will be made, I will let you know.

Cheers!

Thank you.

My apologies for not adding translations to the OP yet. I am busy fighting my upload scripts to get them sending posts to Elasticsearch at an acceptable rate. For some reason, my usually fast hard disk drive on my server has slowed down massively, even after a reboot, and I'm not exactly sure why.

Hi!

Sorry for the short delay. I've just finished the Spanish translation for this thread. Although we usually mark topics as "Done" when they reach the 10 translations mark, we are going to keep this one as "in progress" for a few more days, just in case other members finish theirs soon too. Gazeta will let you know (ty!!!).

As this tool is still being developed and changes are expected to be made in the OP soon, although we'll be monitoring this thread from time to time, please, let us know when we have to update our translations by posting directly in our thread or, if you prefer, sending a PM to Gazeta or me.

[NotATether]

I don't want to keep you guys waiting, so should I release a mode for Talksearch that queries data from the new index?

There are about 2.6 million chunks so far. A chunk is just a section of a post delimited by someone's quote or by a horizontal line.

The upload is not slow - It's uploading 1000 chunks every couple of seconds, but there are simply millions of them.

This mode will not replace the currennt search engine yet.