Talksearch.io - Advanced Bitcointalk Search Engine

[NotATether]

Searches seem to be failing. I'm diagnosing the issue.

[Agbe]

Bitcoin: bc1q6dphprljdas0xl2cmqn6tlskselx5xtcpcw8kx

This is a great work and I support the project with $10. I wanted to do more but I am low in balance for the moment. And I love to support the project. BTC sent.

[DPHOR]

Great job boss.. I tried to explore it and it was fine and cool. Do you intend to add dark theme? Sometimes I usually get affected by white screen so most time I do tuned on colors inversion on my phone to control how my screen adept to the environment where I am. Sometimes if I am in a light place my screen light increases to fit the weather or sunlight of where I am. Please I would also want to see that added maybe you might feels is not important that there are people who loves accessing most of the site with dark theme.

[NotATether]

The problem has been identified and searches are now working normally again..

Post-mortem analysis

Logs in the Google Cloud backend indicated that the service unavailability was caused by the Elasticsearch master node becoming completely unavailable as a result of the hard disk capacity of the hot-content tier server overflowing. This was most likely due to a bug in the script I use to bulk import topics into Elasticsearch. After contacting support, they were able to temporarily upscale the hot-content server to a larger capacity within 20 minutes. I am now going through the data and cleaning up any duplicates and other data that might have overflowed the server.

This highlights the importance of using managed resources for production services. If I was running the server myself, it would have probably taken days to recover from this.

edit: Apparently, the warm tier server was not utilized at all, not even for a single document. That's why the content server was overran so quickly. The ingestion process ran into trouble after around 46 million documents. Considering the number of posts on the forum exceeds 65 million, I need to make it utilized more aggressively.

For now, indexing has been halted while I design a better process for storing the the posts.

As a side effect, an option will probably be developed to not search in spam topics such as signature campaigns, bounties, etc when preforming a query. An efficient way to filter local board topics is also desirable but will probably not be done anytime soon.

Great job boss.. I tried to explore it and it was fine and cool. Do you intend to add dark theme? Sometimes I usually get affected by white screen so most time I do tuned on colors inversion on my phone to control how my screen adept to the environment where I am. Sometimes if I am in a light place my screen light increases to fit the weather or sunlight of where I am. Please I would also want to see that added maybe you might feels is not important that there are people who loves accessing most of the site with dark theme.

Dark theme is not a priority right now. The backend needs to be stabilized first.

[Royal Cap]

First of all, thank you for making such a beautiful search system. I searched quite a lot with it and got very good results. However, I think it would be good to add another thing, which is Short by Date. If you had added this, it would have been easier for us to search for any post that has been made recently. Actually, this is my personal feedback.

[NotATether]

First of all, thank you for making such a beautiful search system. I searched quite a lot with it and got very good results. However, I think it would be good to add another thing, which is Short by Date. If you had added this, it would have been easier for us to search for any post that has been made recently. Actually, this is my personal feedback.

I will work on this soon. For now, recent posts from March are not available, but as I configure automatic scraping then they will gradually become available.

[Macabury]

This search engine is fast. This must have cost so much time and effort, thanks for the good job done. I searched a few topics and I saw threads related to my input clustered waiting for me to click on the exact topic I wanted. This is so innovative. Is this meant to be an extension of ninjastic space?

[Vod]

For now, indexing has been halted while I design a better process for storing the the posts.

A good solution for you may be Wasabi.   They do not charge for egress data, but you must keep the data on the server for a minimum of three months. 

Also, Google cloud should have some sort of monitoring for hard drive usage.  I have an alert sent when any disk usage passes 90% for five minutes.

[NotATether]

Also, Google cloud should have some sort of monitoring for hard drive usage.  I have an alert sent when any disk usage passes 90% for five minutes.

They have monitors somewhere, I just haven't figured out how to use them yet.

After development, I am learning Operations the hard way 

This search engine is fast. This must have cost so much time and effort, thanks for the good job done. I searched a few topics and I saw threads related to my input clustered waiting for me to click on the exact topic I wanted. This is so innovative. Is this meant to be an extension of ninjastic space?

No, this is not Ninjastic space.

[joker_josue]

Congratulations on the project. A good search tool for Bitcointalk is really useful.

I will look into this in more detail later and then share any suggestions (if applicable). 

[Synchronice]

First of all, thank you for this service, I appreciate everyone who tries to improve this forum.
Why didn't you host your website on Hetzner? They have one of the cheapest and fastest servers, this is the best thing someone can get for their bucks. $150 per month is a huge cost, I don't think you'll be able to collect that every month.

By the way, can you create a demonstration of what's the difference between your search engine and Bitcointalk's (Google) search engine? At the moment I did some search and Bitcointalk have me more accurate results than Talksearch.io
Do you plan to add more filters? Like sort by ascending/descending, search by an user X and so on?

[NotATether]

Why didn't you host your website on Hetzner? They have one of the cheapest and fastest servers, this is the best thing someone can get for their bucks. $150 per month is a huge cost, I don't think you'll be able to collect that every month.

Because I already own a Hetzner server, and I don't want to get kicked out by the reseller due to a hack or DDoS.

This is what the Google Cloud setup looks like by the way:



So not only are there two content servers, which aren't as big as what you can order with an HDD on a other websites, there's also a server for the internal dashboard, the ingestion server that's going to scrape Bitcointalk, and the server that actually delivers search results (enterprise search, on the right of the screen).

All this allows me to easily experiment with different search algorithms and parameters.

I also get free customer support which allows me to recover from any hardware failure in just hours (literally), like yesterday.

Google App Engine itself is virtually free and that's where I host the website. In my experience, it is much better than attempting to host it directly because I don't have to worry about downtime or attacks like for my other sites.

By the way, can you create a demonstration of what's the difference between your search engine and Bitcointalk's (Google) search engine? At the moment I did some search and Bitcointalk have me more accurate results than Talksearch.io

I haven't gotten a chance to enhance the Talksearch result quality yet.

Do you plan to add more filters? Like sort by ascending/descending, search by an user X and so on?

Yes, of course. Just give me a couple days. There's other stuff I need to take care of first.

[dkbit98]

Nice service and it open everything really fast, but I don't think this is worth paying $150 per month, especially if you want to make this project long term sustainable.
One of my suggestions is to add optional alternative dark theme switch, and maybe add some default search terms, and predefined board locations.
As for donations, you should add all addresses or links in the website footer.  

[libert19]

I searched, 'libert19's first post' — I was expecting that results would show my first post, and it didn't. It may be stupid query, but a simple one, and should deffo work.

Another query I did was, "I was fucked, do not repeat same mistakes as me" which is title of this thread — and results were irrelevant instead of showing thread which I was expecting. Title was literally copy-pasted here!?

[BenCodie]

I searched, 'libert19's first post' — I was expecting that results would show my first post, and it didn't. It may be stupid query, but a simple one, and should deffo work.

This is not an AI or conversational search engine. It works from keywords. The engine is not going to know that "libert19" is the user nor is it going to recognize "first post" as search for the first post, it's going to search libert19, first, and post in the topic title/content. I think you're a bit used to AI I think this is better thought of like a pre-ai search engine.

[JollyGood]

Without a doubt though congratulations are in order, I think the general consensus is clear that $150 per month is far too expensive for any endeavour of this nature.

As for the search results, I searched for "Satoshi" and saw results that had no chronological (date) order nor a way to filter the results allowing me to view them how I wanted (such as new, old, mentioned in subject, mentioned in post).

[KingsDen]

I searched, 'libert19's first post' — I was expecting that results would show my first post, and it didn't. It may be stupid query, but a simple one, and should deffo work.

This is not an AI or conversational search engine. It works from keywords. The engine is not going to know that "libert19" is the user nor is it going to recognize "first post" as search for the first post, it's going to search libert19, first, and post in the topic title/content. I think you're a bit used to AI I think this is better thought of like a pre-ai search engine.

It's kinda funny that libert19 had to think that the forum search function will work in such a way, lol. If so, I will have to search "How much BTC I had the day I became full member in the forum".

[BenCodie]

I searched, 'libert19's first post' — I was expecting that results would show my first post, and it didn't. It may be stupid query, but a simple one, and should deffo work.

This is not an AI or conversational search engine. It works from keywords. The engine is not going to know that "libert19" is the user nor is it going to recognize "first post" as search for the first post, it's going to search libert19, first, and post in the topic title/content. I think you're a bit used to AI I think this is better thought of like a pre-ai search engine.

It's kinda funny that libert19 had to think that the forum search function will work in such a way, lol. If so, I will have to search "How much BTC I had the day I became full member in the forum".

Maybe it's a consequence of replacing search engines with conversational AI, though from what I've witnessed, libert19 commonly makes these kinds of posts (to the point where I can hardly tell if it is extremely sophisticated trolling or natural).

[NotATether]

New content update v1.0.1 published

This update enhances search functionality and includes optimizations to the search algorithm to return more relevant results.

Search features:

• Removed results with duplicate titles - now only the most relevant result from each topic is returned.
• Boosts posts with 2-4 exact query matches.
• Penalizes posts with too many query matches to avoid spam.
• Penalizes posts that are too long using a reciprocal multiplication weight - this will help remove irrelevant altcoin ANN threads from the search results while also avoiding penalizing nullius-style posts.

App features:

• You can now refresh the page by clicking on the Search button on the search results page, previously, it only refreshed if the query was a different one, and it didn't look as nice.

The old versions continue to be cached in the cloud, allowing me to roll back immediately if any bugs are detected. There should not be any, but it's a good failsafe to have.

[libert19]

Maybe it's a consequence of replacing search engines with conversational AI, though from what I've witnessed, libert19 commonly makes these kinds of posts (to the point where I can hardly tell if it is extremely sophisticated trolling or natural).

That was genuine query and I don't troll, may be my posts just come off like that.

Regarding query, even before AI if you search stuff on Google, for example, "Messi's first match" you would find relevant result, so intention was same here.


Meanwhile, I repeated the below query after update above, and I am still not getting expected result.

Another query I did was, "I was fucked, do not repeat same mistakes as me" which is title of this thread — and results were irrelevant instead of showing thread which I was expecting. Title was literally copy-pasted here!?