NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 08:01:35 AM |
|
I am trying to make a search engine for Bitcointalk posts, since Google and the built-in one are so bad.
List all the features you want in a search engine here.
For now, I am scraping topics from the forum using my bot. I made sure to identify the requests as coming from me in my program so that the admins know where this traffic is coming from.
It doesn't look like it's exceeding the threshold of one request per second so that's good.
Private boards are not being scraped. The scraping is being done as a guest.
|
|
|
|
ABCbits
Legendary
Offline
Activity: 2996
Merit: 7864
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 08:46:29 AM |
|
List all the features you want in a search engine here.
How about feature which already available on https://ninjastic.space/search? Aside from that, i would suggest these feature. 1. Sort by relevancy. 2. Showing message that the search keyword may contain typo (such as showing "bitcoin" when someone enter "bitcon").
|
|
|
|
NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 09:26:57 AM |
|
List all the features you want in a search engine here.
How about feature which already available on https://ninjastic.space/search? Aside from that, i would suggest these feature. 1. Sort by relevancy. 2. Showing message that the search keyword may contain typo (such as showing "bitcoin" when someone enter "bitcon"). Ninjastic is showing entire posts so it's impossible to find anything meaningful when you search for a keyword. It needs to show only an excerpt with a link and title like forum search and Google do, it needs to have page numbers for browsing the results by page and most importantly it should not be looking inside quotes for keywords.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17360
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 09:27:46 AM |
|
For now, I am scraping topics from the forum using my bot. If it helps, I can give you a tar.gz copy of my data (note: some posts are missing). I shared it with Ninjastic years ago, and it saves you several months of scraping. Freshly scraping will get you a more recent edit though, and less deleted posts. 1. Sort by relevancy. This would be the one thing I'd like to see, but also no doubt the most difficult one. Ninjastic often gives me a list of hundreds of posts. A good search engine (like Google 10 years ago) would show what I want to see first.
|
|
|
|
ABCbits
Legendary
Offline
Activity: 2996
Merit: 7864
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 10:19:21 AM |
|
Ninjastic is showing entire posts so it's impossible to find anything meaningful when you search for a keyword.
Sorry for not being specific. I mean feature such as "Date Range (UTC)" filter, choosing one or more boards (and optionally with the child board) and sign support (+, -, | and "").
|
|
|
|
mocacinno
Legendary
Offline
Activity: 3500
Merit: 5086
https://merel.mobi => buy facemasks with BTC/LTC
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 10:49:26 AM |
|
if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
If would be nice if i could make a query like `user:Theymos board:Bitcoin\Project_Development +wallet -knots taproot` and i would only see posts made by Theymos in the project developent board that contained the word wallet, did not contain the word knots and hopefully contained the word taproot.
|
|
|
|
NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 12:03:39 PM |
|
For now, I am scraping topics from the forum using my bot. If it helps, I can give you a tar.gz copy of my data (note: some posts are missing). I shared it with Ninjastic years ago, and it saves you several months of scraping. Freshly scraping will get you a more recent edit though, and less deleted posts. Sure, you can send me a copy by PM. if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
If would be nice if i could make a query like `user:Theymos board:Bitcoin\Project_Development +wallet -knots taproot` and i would only see posts made by Theymos in the project developent board that contained the word wallet, did not contain the word knots and hopefully contained the word taproot.
I can't index DT information since it's invisible to guests and it changes too quickly anyway, but I'm already scraping the other stuff like boards, username (of course) etcetera. Even the user IDs are being scraped to help deal with name changes. My bot can also handle anonymous users too.
|
|
|
|
seoincorporation
Legendary
Offline
Activity: 3276
Merit: 3066
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 02:07:18 PM |
|
if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
But this wouldn't be like rebuilding the full forum on a database? I mean, there are 2 ways to do this: 1.- You take all the forum data, and put it together on a database and then your search engine makes calls to that database. But for this, you will have to live update that database or at least have a cron job to add the new data each x time. 2.- Search for the data directly on the site, but for that, you would have to do some kind of hack to the current search engine. If you have other way in mind i would love to know how it work.
|
|
|
|
Vod
Legendary
Offline
Activity: 3822
Merit: 3121
Licking my boob since 1970
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 06:16:47 PM |
|
it should not be looking inside quotes for keywords.
This is what I'm currently having issues with. People break the BB quote code all the time in their posts.
|
https://nastyscam.com - featuring 13 years of OGNasty bitcoin scams https://vod.fan - fast/free image sharing - cleaning it up! (240905) Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
mocacinno
Legendary
Offline
Activity: 3500
Merit: 5086
https://merel.mobi => buy facemasks with BTC/LTC
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 05:34:14 AM |
|
if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
But this wouldn't be like rebuilding the full forum on a database? I mean, there are 2 ways to do this: 1.- You take all the forum data, and put it together on a database and then your search engine makes calls to that database. But for this, you will have to live update that database or at least have a cron job to add the new data each x time. 2.- Search for the data directly on the site, but for that, you would have to do some kind of hack to the current search engine. If you have other way in mind i would love to know how it work. It would be better if there was a way to improve SMF's search function or query the relational database directly, but i don't know if Theymos would give anybody direct access to the database or allow anybody to completely rework smf's soucecode . I don't think he would, and for good reason offcourse... It would require absolute trust in the person building the search engine. But you're right, it would be completely rebuilding bitcointalk's database, like several other members are doing aswell (more or less).. Whenever i see somebody building extensions, offsite tools, proposing changes to SMF, i can't help but wonder how epochtalk is doing, and if epochtalk would solve the problem without requiring browser plugins, offsite tools, scraping,... Don't get me wrong: the current forum software lacks several features, and i'm happy if somebody builds them (even if it's on a different domain, or requires me to install a browser plugin), i just wonder wether we'll ever switch to the new forum software.
|
|
|
|
NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 03:40:58 PM |
|
This thing is going to use Elasticsearch, so if I can figure out how to handle the multi-lingual text, it may possibly support searching in different languages too.
|
|
|
|
TryNinja
Legendary
Offline
Activity: 2954
Merit: 7378
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 04:36:44 PM |
|
Ninjastic is showing entire posts so it's impossible to find anything meaningful when you search for a keyword.
Soon™ ... ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) ![](https://ip.bitcointalk.org/?u=https%3A%2F%2Fwww.talkimg.com%2Fimages%2F2024%2F07%2F17%2F4n6S1.png&t=665&c=yY5kwYzolwIozg)
|
|
|
|
NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 04:57:20 PM |
|
Ninjastic is showing entire posts so it's impossible to find anything meaningful when you search for a keyword.
Soon™ ... ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) ![](https://ip.bitcointalk.org/?u=https%3A%2F%2Fwww.talkimg.com%2Fimages%2F2024%2F07%2F17%2F4n6S1.png&t=665&c=yY5kwYzolwIozg) Personally I am against indexing quotes inside posts, since that runs into the risk of corpus duplication, so gives the quoted parts more weight in the search results - meaning the posts that are quoted the most are also the most likely to be returned at the top (most likely in the form of some person's reply to it). There's also the issue with nested quotes, which is hell to deal with using a database, and even Elasticsearch too. It can lead to infinitely recursive schema/JSON before you can even parse it completely.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17360
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 05:04:35 PM |
|
Personally I am against indexing quotes inside posts, since that runs into the risk of corpus duplication In most cases, I agree. But if the quote comes from an external website, the content can still be relevant to the search query.
|
|
|
|
TryNinja
Legendary
Offline
Activity: 2954
Merit: 7378
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 05:24:50 PM Merited by PowerGlove (1) |
|
Personally I am against indexing quotes inside posts, since that runs into the risk of corpus duplication, so gives the quoted parts more weight in the search results - meaning the posts that are quoted the most are also the most likely to be returned at the top (most likely in the form of some person's reply to it).
I've separated quotes from post content. That way there is: - post content - quotes There's also the issue with nested quotes, which is hell to deal with using a database, and even Elasticsearch too. It can lead to infinitely recursive schema/JSON before you can even parse it completely.
Even if there are nested quotes, they are treated individually and also indexed as their own. Take this post of mine for example. There is the content (everything that is NOT a quote of another user, like this text itself) and both quotes from author NotATether you can see above. You will be able to search only the content, only the quotes, quotes from X user, or both the content and quotes.
|
|
|
|
Vod
Legendary
Offline
Activity: 3822
Merit: 3121
Licking my boob since 1970
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 07:28:12 PM |
|
I can't index DT information since it's invisible to guests and it changes too quickly anyway, but I'm already scraping the other stuff like boards, username (of course) etcetera. Even the user IDs are being scraped to help deal with name changes.
You CAN index DT information by running your parser under your account. See https://bitcointalk.org/captcha_code.php (it no longer works for my account but yours is probably fine) Even if there are nested quotes, they are treated individually and also indexed as their own.
You will be able to search only the content, only the quotes, quotes from X user, or both the content and quotes.
You must have some insane AIish parsing going. Many posts have broken quote html. ![Sad](https://bitcointalk.org/Smileys/default/sad.gif)
|
https://nastyscam.com - featuring 13 years of OGNasty bitcoin scams https://vod.fan - fast/free image sharing - cleaning it up! (240905) Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
TryNinja
Legendary
Offline
Activity: 2954
Merit: 7378
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 07:32:01 PM |
|
You must have some insane AIish parsing going. Many posts have broken quote html. ![Sad](https://bitcointalk.org/Smileys/default/sad.gif) I don't, in this case there isn't much to be done... If it's broken, it's broken. But most posts don't have this problem, so it will be a lot better than what I currently have on ninjastic.space.
|
|
|
|
Vod
Legendary
Offline
Activity: 3822
Merit: 3121
Licking my boob since 1970
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 17, 2024, 08:41:48 PM |
|
so it will be a lot better than what I currently have on ninjastic.space.
I watch your work with interest - I love large datasets and you seem to know the presentation layer well! And then there is LoyceV - he has all the information in text format and is a whiz with queries, but he cannot present the info like you do. Loyce.club can be the fastest to find exactly what one is was looking for, but I work with your website if I only have partial info.
|
https://nastyscam.com - featuring 13 years of OGNasty bitcoin scams https://vod.fan - fast/free image sharing - cleaning it up! (240905) Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
NotATether (OP)
Legendary
Offline
Activity: 1722
Merit: 7248
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 18, 2024, 11:23:55 AM |
|
15,000 out of 1.7 million threads scraped so far, all topics being scraped in numerical order. I can't index DT information since it's invisible to guests and it changes too quickly anyway, but I'm already scraping the other stuff like boards, username (of course) etcetera. Even the user IDs are being scraped to help deal with name changes.
You CAN index DT information by running your parser under your account. See https://bitcointalk.org/captcha_code.php (it no longer works for my account but yours is probably fine) That's a security risk though, since it would require me to also store my password in plain text. Even I use one of the bots (BotATether or Jarvis), the added hassle of dealing with authentication will actually slow down post collection. Currently for each thread I'm launching a new browser - this helps me stay within the rate limits. I might have to do a separate scrape for users specifically, to get everybody's DT information without duplicating stuff. But I don't think that's going to be happening anytime soon.
|
|
|
|
seoincorporation
Legendary
Offline
Activity: 3276
Merit: 3066
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 18, 2024, 02:09:58 PM |
|
if you are scraping and parsing anyway, it would be nice if your search engine was indexing the most common board objects... For example, the username, DT rank, feedback, boards,... That way, you could use keywords, like you can in google (filetype:, site:,...).
But this wouldn't be like rebuilding the full forum on a database? I mean, there are 2 ways to do this: 1.- You take all the forum data, and put it together on a database and then your search engine makes calls to that database. But for this, you will have to live update that database or at least have a cron job to add the new data each x time. 2.- Search for the data directly on the site, but for that, you would have to do some kind of hack to the current search engine. If you have other way in mind i would love to know how it work. It would be better if there was a way to improve SMF's search function or query the relational database directly, but i don't know if Theymos would give anybody direct access to the database or allow anybody to completely rework smf's soucecode . I don't think he would, and for good reason offcourse... It would require absolute trust in the person building the search engine. But you're right, it would be completely rebuilding bitcointalk's database, like several other members are doing aswell (more or less).. If anyone has access to implement a change like this in the forum, that one is our verified hacker PowerGlove ( https://bitcointalk.org/index.php?action=profile;u=3486361), but the fact that the search function doesn't work fine at all must be for a reason. Maybe the forum used to have some kind of attacks from that vector. This project would be easy if the RSS was still active on the forum, but sadly it has been removed: https://bitcointalk.org/index.php?type=rss;action=.xmlaction=.xml is disabled due to slowness. If you use this, write a post in Meta explaining your usage. The problem is the moment to index the new threads, without RSS we have to "Ping" each board to know if there is new content.
|
|
|
|
|