[BETA] NEW ninjastic.space search

[TryNinja]

Some people have said the searching mechanism on ninjastic.space is bad, so I'm looking for ways to improve it.

Today I'm releasing this beta page:

https://search-beta.ninjastic.space



I'll explore new designs for it and take suggestions (let me know what you think). The plan is for this page to become the new ninjastic.space (at least for the searching part).

- You can search everything with a single query.
- You can use special search tags to better filter your results.
- By default, the search will only consider the content from each post itself (ignoring any user quotes, that is, anything inside quotes that have the "Quoted from: USER" header).
- The content that shows up on the results page is the actual content (no user quotes) and the pagination is now numbered (max of 10k results, so 500 pages).
- Results are ordered by relevance (sorting by date will be added really soon).

This allows more complex searches like this one:

All posts from LoyceV, where he quotes TryNinja or theymos, on topic ID 5255508, made from 2019-01-01 to 2021-01-01, with the exact phrase "easy solution":

Code:

author:LoyceV quoted_user:TryNinja|theymos topic_id:5255508 from_date:2019-01-01 to_date:2021-01-01 "easy solution"

P.S: I have messed up separating URLs when indexing the data so that will be fixed in the near future when I reindex everything (takes many hours). Also note that some older posts scraped from Loyce's archive contains no board_id, so they won't be returned on queries where you specify a board_id.


SEARCH TAGS:

Code:

author, topic_id, board_id, from_date, to_date, title, quotes, quoted_user, url, include_quotes

For title and quotes, it will do an exact match search by default. To do keywords, you can use :* (i.e title:*"title with keywords")
For author, topic_id, board_id, and quoted_user, you can specify multiple options ("OR") by using | (i.e author:TryNinja|theymos|"John Doe" board_id:25|12)
For title, quotes, and the post content, you can negate a keyword using a minus - (i.e "i like bitcoin" -doge)


SEARCHING EXAMPLES:

Search by keywords simply by writing the words on the search field:

- Returning all posts with the words "wasabi", "privacy", "feature", but not the word "fees".

Code:

Wasabi privacy feature -fees

Search by exact phrase by writing the phrase between quotes:

- Returning all posts with the EXACT phrase "wasabi privacy feature" from the topic of id 5419000 or 5419001.

Code:

topic_id:5419000|5419001 "Wasabi privacy feature"

Include searching inside user quotes (i.e "quoted from: TryNinja") with the tag include_quotes:true

- Returning all posts with the EXACT phrase "wasabi privacy feature", including the content inside user quotes.

Code:

include_quotes:true "Wasabi privacy feature"

Search an exact specific content found ONLY inside user quotes with the tag quotes.

- Returning all posts with the EXACT phrase "They are lying" AND that have a user quote with the EXACT phrase "privacy matters".

Code:

quotes:"privacy matters" "They are lying"

Search a specific keyword content found ONLY inside user quotes with the tag quotes.

- Returning all posts with the EXACT phrase "They are lying" AND that have a user quote with the keywords "privacy", "matters".

Code:

quotes:*"privacy matters" "They are lying"

Search a post with a specific URL.

- Returning all posts with the exact phrase "Proof of authentication" and with the URL "twitter.com".

Code:

url:"twitter.com" "Proof of authentication"

Search a post with date ranges

- Returning all posts from 2018-06-29 to 2018-06-30 which title has the EXACT phrase "Discussions about Bitcoin".

Code:

title:"Discussions about Bitcoin" from_date:2018-06-29 to_date:2018-06-30

[pakhitheboss]

What you have done is amazing and yet it confuses me as a non technical user. I believe that for a few you made this incredible update. I am nobody yet I feel if someone knows how to do Google search in the correct way they as in the user will get more productive outcome. As you have outcome the issue why not create patch for the forum software.

Pardon me but those quoted example doesn't make any sense to me as they are not related to the forum I believe.

Please can you be more specific and detailed with your features. Their might be a lot of arguments by some one who understands it but as a non technical guy it is a waste for me..

[TryNinja]

What you have done is amazing and yet it confuses me as a non technical user. I believe that for a few you made this incredible update. I am nobody yet I feel if someone knows how to do Google search in the correct way they as in the user will get more productive outcome. As you have outcome the issue why not ask the admin to patch it to the forum software. It should make non technical users enjoy the with in search feature without depending on Google search.

Pardon me but those quoted example doesn't make any sense to me. Please can you be more specific and detailed with your features. Their might be a lot of backlash by some one who understands it but as a non technical guy it is a waste for me..

No offense, but your post seems like a word salad. And looking at the logs, I don't think you even tried the website.

[pakhitheboss]

No offense, but your post seems like a word salad. And looking at the logs, I don't think you even tried the website.

I am not after anyone or anything that makes this forum upgrade. What I am after is that you guys as in the software developers, that I am aware of do neglect a simple think, "make it simple & a newbie friendly"

I faced it when I downloaded your bot for Bitcointalk notification. I am assuming that everyone here is good with it and they are good in understanding the notification bot. I am not and that is my reason to question you new update.

I will check it but please understand their are many who are not that smart like you or have brains to understand what you have created. My humble request is to keep it simpler when you as a software enthusiastic individual come up with a new idea.

[TryNinja]

I am not after anyone or anything that makes this forum upgrade. What I am after is that you guys as in the software developers, that I am aware of do neglect a simple think, "make it simple & a newbie friendly"

I faced it when I downloaded your bot for Bitcointalk notification. I am assuming that everyone here is good with it and they are good in understanding the notification bot. I am not and that is my reason to question you new update.

I will check it but please understand their are many who are not that smart like you or have brains to understand what you have created. My humble request is to keep it simpler when you as a software enthusiastic individual come up with a new idea.

There is literally a single input and a button on the website.

Over there you write what you want to search and add "author:TryNinja" if you want to search posts where the author is TryNinja.
Or title:"Bitcoin" if you want the title to have "Bitcoin".

I might be wrong, but I think that's dead simple.

[joker_josue]

First of all, excellent work. Making a good search tool is not easy.

What kind of tests do you want us to do? Research, layout, a little bit of everything!?



I start by making a suggestion, perhaps in line with what has already been said in previous posts.

Creates small tag buttons. When clicking on them, they immediately place the tag in the search field and the person only needs to enter the value of that tag.
I think this makes it easier to use the search, as we may not remember all the possible tags. In addition to helping to avoid errors when writing tags.



I'll look more into the tool and give other tips.
Excellent job!

[TryNinja]

What kind of tests do you want us to do? Research, layout, a little bit of everything!?

I start by making a suggestion, perhaps in line with what has already been said in previous posts.

Creates small tag buttons. When clicking on them, they immediately place the tag in the search field and the person only needs to enter the value of that tag.
I think this makes it easier to use the search, as we may not remember all the possible tags. In addition to helping to avoid errors when writing tags.

It's specially focused to people who frequently use searching, be it to investigate other users/scams, find lost content, references, etc... "what is missing when you do your searching?"

And of course, how the UI should look like.

I thought about the single input (what we have now), multiple inputs for each filter like we have on ninjastic.space/search (i.e an input for authors, one for content, another for boards...), complex filters with multiple grouped and organized AND/OR conditions, etc.

[PX-Z]

I feel like this is more complex, might be because of using of only single input instead of the per input textbox in ninjastic.space. But with all this search tags for filter, probably this will give more precise results, great job. One to mention that username is case insensitive too.

[uchegod-21]

I feel like this is more complex, might be because of using of only single input instead of the per input textbox in ninjastic.space. But with all this search tags for filter, probably this will give more precise results, great job. One to mention that username is case insensitive too.

I have tried searching some posts by the title of any related keyword I can remember on the title but it returned nothing. No matter how I tried to include the author according to the placeholder tip, it didn't work for me. I didn't include any user ID though.
But whenever I make a search, I should be able to see some returned results and not an empty return. If I see some returned results, I could manually scan through them or be more confident to search again by changing keywords.
If will be fine that every search, results should be returned even if not precised.

[TryNinja]

I have tried searching some posts by the title of any related keyword I can remember on the title but it returned nothing. No matter how I tried to include the author according to the placeholder tip, it didn't work for me. I didn't include any user ID though.

Looking at the logs, that's because you are searching:

Code:

Author: KingsDen "announce yourself"

Notice the space between the author tag and the actual username.

It should be, as per example:

Code:

Author:KingsDen "announce yourself"

There are very few searches going on, so I noticed that. But please, when reporting an issue, post the details. Otherwise there is no way of debuging what you or I did wrong.

[PX-Z]

I have tried searching some posts by the title of any related keyword I can remember on the title but it returned nothing. No matter how I tried to include the author according to the placeholder tip, it didn't work for me.

Really? It works to me though, e.g. "scam casino_name" and other crypto related words on the text box with "title:" or not.

Edit: OP's response is more precise for your problem.

[joker_josue]

It's specially focused to people who frequently use searching, be it to investigate other users/scams, find lost content, references, etc... "what is missing when you do your searching?"

And of course, how the UI should look like.

I thought about the single input (what we have now), multiple inputs for each filter like we have on ninjastic.space/search (i.e an input for authors, one for content, another for boards...), complex filters with multiple grouped and organized AND/OR conditions, etc.

I'm not saying to create filter fields.
What I'm saying is that you create tag buttons, which when you click on them, automatically adds that tag to the search field.

Code:

author, topic_id, board_id, from_date, to_date, title, quotes, quoted_user, url, include_quotes

For example, I click on "Topic ID" and the search field shows "topic_id:" and then I just have to write the topic number.
This way, I don't need to keep typing "topic_id:", when I want to search with this tag.

[NotATether]

IMO, I think this is going to be a problem.

Exact results are pretty much useless as nobody is going to be searching with entire post contents.

Semantic meaning is not taken into context either, just the lexical word, so we are not only getting results for trains here but about a ton of "training" posts from Gambling Discussion.

I'm sure we would love to have google-like search results but we need to figure out the secret sauce first.

[TryNinja]

https://www.talkimg.com/images/2024/07/20/4NKIN.png

IMO, I think this is going to be a problem.

Exact results are pretty much useless as nobody is going to be searching with entire post contents.

Semantic meaning is not taken into context either, just the lexical word, so we are not only getting results for trains here but about a ton of "training" posts from Gambling Discussion.

I'm sure we would love to have google-like search results but we need to figure out the secret sauce first.

And what do you expect it to find when you search "I like trains"? Or any actual example, for that matter.

[NotATether]

And what do you expect it to find when you search "I like trains"? Or any actual example, for that matter.

This is going to sound controversial but I think that meaningless posts should not be indexed, or at least not returned. The algorithm should be prioritizing information over exact matches.

Open DuckDuckGo or any other search engine and search for it. After the videos, images etc you will start seeing results like this:



The mission of a search engine is to help people find what they are looking for. This is almost always stuff like answers, facts, and other kinds information with a semantic load. There is a lot of spam on Bitcointalk that nobody wants to read on the main site. So as much as possible, search engines must avoid returning spammy posts as results because this will discourage people from using them, as they will not find the kind of information they are looking for.

PS. even merit cannot be used to bias the posts because author merit does not apply if the post in question has no semantic load, and post merit allows for merit farms to manipulate search results by meriting spammy posts.

[Eternad]

Is it possible to include a tag on a post that already deleted on search result? Sometimes I  find a deleted post which I’m about to reply but it turns out to be deleted or edited already.

I’m not sure if this is possible but this will be awesome at least for me that use search function to find a topic I need instead of the forum search feature.

[TryNinja]

And what do you expect it to find when you search "I like trains"? Or any actual example, for that matter.

This is going to sound controversial but I think that meaningless posts should not be indexed, or at least not returned. The algorithm should be prioritizing information over exact matches.

Open DuckDuckGo or any other search engine and search for it. After the videos, images etc you will start seeing results like this:

https://www.talkimg.com/images/2024/07/20/4EX7v.png

The mission of a search engine is to help people find what they are looking for. This is almost always stuff like answers, facts, and other kinds information with a semantic load. There is a lot of spam on Bitcointalk that nobody wants to read on the main site. So as much as possible, search engines must avoid returning spammy posts as results because this will discourage people from using them, as they will not find the kind of information they are looking for.

PS. even merit cannot be used to bias the posts because author merit does not apply if the post in question has no semantic load, and post merit allows for merit farms to manipulate search results by meriting spammy posts.

Well, good luck on your journey on recreating Google for BitcoinTalk. You're going to need it.

I don't know how nor want/have the the time to do all of that, so I'm ok with facilitating searching on my archive (which was always a subproject of my telegram bot) the way I'm doing it. I've used it myself many times to find quotes and references, so I can tell you that it's not an useless tool.

What you want is for someone to search "how to create a electrum wallet" and find out all the main OPs and pieces of text which correctly teach how to do that. Without doing AI'ish stuff and/or manually tagging posts I can't see how that would work. But I honestly hope you can make it.

Is it possible to include a tag on a post that already deleted on search result? Sometimes I  find a deleted post which I’m about to reply but it turns out to be deleted or edited already.

Not technically viable right now.

[Vod]

Nice job Ninja; you are obviously the leader in post searching, and I for one would not want to compete with you. 

You have a chance here to implement AI into your search; a third party service could identify what information the user is looking for, instead of what keywords the user is looking for. 

[joker_josue]

The mission of a search engine is to help people find what they are looking for. This is almost always stuff like answers, facts, and other kinds information with a semantic load. There is a lot of spam on Bitcointalk that nobody wants to read on the main site. So as much as possible, search engines must avoid returning spammy posts as results because this will discourage people from using them, as they will not find the kind of information they are looking for.

Do you have any idea how many years it took for search engines to do what you're asking for?
A system of this kind is very complex to build. That's why it remains difficult to destroy Google and its algorithm.


What TryNinja could do is a classification, where users who used the search classified the result as positive (close to what was intended) or negative (spam). And the system began to place the results with the most negative grades at the end of the list. Even so, there would be a danger of making the information biased.

[NotATether]

Do you have any idea how many years it took for search engines to do what you're asking for?
A system of this kind is very complex to build. That's why it remains difficult to destroy Google and its algorithm.

The objective here is obviously not to "destroy" Google, but to make a search system for Bitcointalk that is more useful than Google's site:bitcointalk.org (and the native search).

So yes, I know it's going to be hard, but I don't see it being that complex. For one thing, we have only millions of posts, not billions of pages like google. Second, we won't have as many users, and consequentially people trying to "abuse" the system compared to Google (only a few thousand people, instead of millions of people using the search).

Regarding spam, I don't think anyone's going to shed any tears over spam posts not being indexed. For instance, I plan to (and am currently in the process of) index the entire Politics and Society. However, you can eliminate 95% of the spam by enforcing a minimum character count in posts like in sig campaigns, as well as using a regex to filter out sig/bounty application posts. Google's bias accusations are literally because they tweak the ranking of certain kinds of pages that they are offended by - I have no such plans.

Now yes, I know that if you're trying to conjuncture your own search software, you will run into the limitations you just described.

However, we are very fortunate to have open-source search software written by people who do have years of experience in the field, for example Apache Lucene (which coincidentally I have experience with but in a completely different industry). Because of this, sites can roll up their own search, in case Google's results are insufficient. Example: Hacker News.

The above is not written to discourage anybody from creating their own search algorithms, it just shows the requirements that need to be in place to make an "intelligent" search. There's nothing wrong with the current implementation if you're just using it to search for someone's (or your own) post, and I think that is what TryNinja is trying to make here.

I can see much value there will be for an intelligent search engine if somebody manages to pull it off, especially with other repositories like bitcoin-dev (the old version) in dire need of indexing as well.