Talksearch.io - Advanced Bitcointalk Search Engine

[NotATether]

I am testing a prototype version of an automatic scraper. Can you guys please check after you make some posts today whether you can find your posts in the search engine?

[TypoTonic]

I've been searching about newbie guides for specific topics using the forum search engine and often times it will say that I am searching too quickly. It also goes on a timeout when I input something quite long, it's honestly a bit frustrating. But based on what I have read so far, I think it has something to do with my forum rank. Thankfully I came across this thread, and I immediately had a better experience. This is definitely a great help for me.

[DYING_S0UL]

I've been searching about newbie guides for specific topics using the forum search engine and often times it will say that I am searching too quickly. It also goes on a timeout when I input something quite long, it's honestly a bit frustrating. But based on what I have read so far, I think it has something to do with my forum rank. Thankfully I came across this thread, and I immediately had a better experience. This is definitely a great help for me.

You can also use the google based search function which is avaliable inside the search, and afaik it doesn't have any search rate limit or that posting too quickly thingy. But yeah, I do agree that Talksearch is a great alternative for searching our desired topics and stuff..

This below



Edit:

I am testing a prototype version of an automatic scraper. Can you guys please check after you make some posts today whether you can find your posts in the search engine?

I tried to search my last post! But it didn't show up! Or maybe I didn't searched properly or used relevant keywords.

[NotATether]

I tried to search my last post! But it didn't show up! Or maybe I didn't searched properly or used relevant keywords.

Something is wrong with the search engine algorithm when finding posts. But it is highly likely your post has been indexed. Please share the post ID.

The search engine is not broken per se, but the internal search queries are quite crappy.

[taufik123]

Can you give an example of how to find keywords that are relevant enough for a post that you might be looking for.
Because I can't find my most recent posts, I only have a few old posts and the ones that appear are also in no order from the latest to the oldest.




My post: https://bitcointalk.org/index.php?topic=5545585.msg65662997#msg65662997

[GazetaBitcoin]

Hey NotATehter, good news! A Bangla translation was made by AOBT for your topic ☺️

Here is it: https://bitcointalk.org/index.php?topic=631891.msg65666928#msg65666928

[NotATether]

Hey NotATehter, good news! A Bangla translation was made by AOBT for your topic ☺️

Here is it: https://bitcointalk.org/index.php?topic=631891.msg65666928#msg65666928

Thank you!

Can you give an example of how to find keywords that are relevant enough for a post that you might be looking for.
Because I can't find my most recent posts, I only have a few old posts and the ones that appear are also in no order from the latest to the oldest.

My post: https://bitcointalk.org/index.php?topic=5545585.msg65662997#msg65662997

Your post is not in the index. However, the topic has been indexed. This is the most recent post in the index: https://bitcointalk.org/index.php?topic=5545585.msg65461283#msg65461283

Topics made prior to Feburary 2025 do now have all of their new replies indexed.


One of the challenges involved in searching posts is how to index giant topics like this one. This is where I tap into existing forum archives - the services created by other uses such as Ninjastic.space and loyce.club allows me to get at least a mostly-complete copy of the data set. Usually, this works for the vast majority of long topics. However, the topics in the Gambling Discussion need special care. Because occasionally, there may be dozens of posts made on any given topic per day. Also, the forum's built-in rate limiter will make the indexing quite slow. In this case, I use an incremental updating method that only indexes some of the posts on any given day. Eventually, after a number of days, the topics will finally be in sync.

Checking for edits is a harder challenge though.

Just wanted to share that with you guys. Please consider donating to support the project.

[DYING_S0UL]

I tried to search my last post! But it didn't show up! Or maybe I didn't searched properly or used relevant keywords.

Something is wrong with the search engine algorithm when finding posts. But it is highly likely your post has been indexed. Please share the post ID.

The search engine is not broken per se, but the internal search queries are quite crappy.

Sorry I missed your notification somehow.
I tried to find it but it didn't show up. I'm not sure whether I filled the right data!
This was the post: https://bitcointalk.org/index.php?topic=5450546.msg65659774#msg65659774

[NotATether]

Sorry I missed your notification somehow.
I tried to find it but it didn't show up. I'm not sure whether I filled the right data!
This was the post: https://bitcointalk.org/index.php?topic=5450546.msg65659774#msg65659774

Your post is also not in the index. Moreover, it is getting more difficult for me to scrape posts. It seems that scraping a very long topic such as this one is causing the scraper to encounter 429 errors. These are not like the temporary 503 "back off" errors, and they have completely halted the scraper. Additionally, there is no reason for my scraper to be scraping this particular topic from the beginning, since I already have some older part of it. So I think this is a bug in my scraper.

I will try to see if I can get it to resume with proxies/tor but I am not going to be hammering the forum with concurrent instances anytime soon.

[NotATether]

Final tests are being conducted. If they pass, then real-time indexing will be enabled permanently.

[NotATether]

Real-time indexing seems to be working well. But I am currently filling the index with every single post from February 2025 onwards, because not all of them were indexed the first time.

Once that is done then real-time search will be really fast.

[taufik123]

Real-time indexing seems to be working well. But I am currently filling the index with every single post from February 2025 onwards, because not all of them were indexed the first time.

Once that is done then real-time search will be really fast.

Will it be faster than before?
You work hard enough to make everything run smoothly and quickly.

It takes a lot of effort and money to build all of this, Not impossible to achieve better progress on this tool.
Making it especially useful when Indexing is already running in real-time.

Good job bro

[NotATether]

Will it be faster than before?

Yes, new posts will appear on the website much quicker now. Even most edited posts will be taken care of, because I am scanning for replied-to topics up to a certain date, which leaves some room for edits.

[DYING_S0UL]

Yes, new posts will appear on the website much quicker now. Even most edited posts will be taken care of, because I am scanning for replied-to topics up to a certain date, which leaves some room for edits.

How many times does a post gets re indexed/scanned actually? Or is it a one time thing? In situations where a post is edited multiple times, does it get re indexed every single times when something is changed, or is there a limit after the re indexing stops?? For example, if someone keeps making small changes to his post, adding new infos, removing old datas, will the tool still update it with latest changes?? Sorry i don't know how it works, so I was a little curious, so I asked anyway.

Btw, still now the searches feels slower!

[NotATether]

Yes, new posts will appear on the website much quicker now. Even most edited posts will be taken care of, because I am scanning for replied-to topics up to a certain date, which leaves some room for edits.

How many times does a post gets re indexed/scanned actually? Or is it a one time thing? In situations where a post is edited multiple times, does it get re indexed every single times when something is changed, or is there a limit after the re indexing stops?? For example, if someone keeps making small changes to his post, adding new infos, removing old datas, will the tool still update it with latest changes?? Sorry i don't know how it works, so I was a little curious, so I asked anyway.

Btw, still now the searches feels slower!

The bitcointalk scraper goes through every board like a person would, and just clicks next until it finds topics that are older than X number of days.

Then it opens each topic and reads all of them.

It does this every 5 minutes.

It is currently jammed at the moment due to reading really long topics (like Wall Observer) over and over again, so I'm going to modify it to read only the posts made within the last X days.

[DYING_S0UL]

Yes, new posts will appear on the website much quicker now. Even most edited posts will be taken care of, because I am scanning for replied-to topics up to a certain date, which leaves some room for edits.

How many times does a post gets re indexed/scanned actually? Or is it a one time thing? In situations where a post is edited multiple times, does it get re indexed every single times when something is changed, or is there a limit after the re indexing stops?? For example, if someone keeps making small changes to his post, adding new infos, removing old datas, will the tool still update it with latest changes?? Sorry i don't know how it works, so I was a little curious, so I asked anyway.

Btw, still now the searches feels slower!

The bitcointalk scraper goes through every board like a person would, and just clicks next until it finds topics that are older than X number of days.

Then it opens each topic and reads all of them.

It does this every 5 minutes.

Ahhh I see. I didn't knew it actually worked like that. To be honest, this feels way to much work just for scrapping. I mean going through every single boards, clicking next and reading all those posts, that's basically mimicking an active user but on steroids, LOL . An endless cycle of browsing the forum and scanning every corner of it. But again u have powerful hardware.

[NotATether]

I am aware of an issue occurring with the Elasticsearch service that is impacting the ability to search. I am working on a fix. Sorry for the inconvenience.

Update: This issue has been resolved. It seems to have been caused by the Elasticsearch service running out of memory. This caused the OOM killer to terminate the process.

I can't say I'm happy about this because this means even 32GB of RAM may not be enough for ingesting....

[NotATether]

New content update v1.1 published

This update brings the long-awaited real-time indexing feature to Bitcointalk.

Please note that due to speed constraints, it takes quite a bit of time to index the recent forum topics and as a result, it might take several hours to a few days for your post to be indexed. I am working on making this happen quicker.

I have also added a tracking code to the Talksearch links in the OP (only) so I can measure how many people click on the link from Bitcointalk.

[DYING_S0UL]

snip

Hey there NotATether,
I just wanna let you know that you never updated OP with this translation. It was made several days ago. Hope that helps.

Bangla translation by DYING_S0UL

Hey NotATehter, good news! A Bangla translation was made by AOBT for your topic ☺️

Here is it: https://bitcointalk.org/index.php?topic=631891.msg65666928#msg65666928

[beveryu778]

I'm looking for an option here. But it's not working, TalkSearch can't do it.

The thing is that if someone quotes my post and posts it, then if I type the text of my post and search, I will find the links to all the posts in the search results. For example, if I am posting this, now if I copy the text of my post exactly and search, then my post shows: If someone else quotes my post, then all of them will come up on search results.

Why is this necessary? What this will do is ensure that before reporting a post by quoting it, it will show someone has already reported it. This can help when reporting plagiarism posts, AI posts, malware, and to avoid duplicate reports.