Bitcointalk Search Project - Vod's Version

[Vod]

I realize there is another topic about this, but it's about parsing the data - something that is now done on half a dozen individual platforms.    I want this topic to be about the actual search engine and interface.

In 1999 I wrote a PPC search engine with backfill from Google (which was paying $0.05US per search!)   In the market, clicks for certain medical or fashion terms were approaching the mid hundreds of $.  No wonder the market collapsed, but I did sell my patent to a company in Toledo, where I worked for a couple years.  My knowledge of search engines is now obsolete, but I still respect the value certain search engines provide. 

The community can rent a cloud box with GPU/AI hardware to build a natural language search engine for the bitcointalk content.  We don't need the highest level reasoning for search queries, so we can probably get a deepseek type response time for a few hundred a month, depending on how we limit search queries in the community.

Is there anyone that would be interested in working on this?  Do you know how to populating an AI model?    If we can find someone willing to do that, then myself and other archive provers like Loyce and NotATether can provide the data to populate. 

[shield132]

The community can rent a cloud box with GPU/AI hardware to build a natural language search engine for the bitcointalk content.  We don't need the highest level reasoning for search queries, so we can probably get a deepseek type response time for a few hundred a month, depending on how we limit search queries in the community.

It might be possible to raise a few hundred dollars in the first month and second month but I think it's impossible to raise a few hundred dollars for a long time. If theymos invests in this, that's different, he can guarantee to fund long-term because this forum has millions of dollars into Bitcoins.

Is there anyone that would be interested in working on this?  Do you know how to populating an AI model?    If we can find someone willing to do that, then myself and other archive provers like Loyce and NotATether can provide the data to populate. 

As I understand, it will be like DeepSeek. It learns the whole forum, every post and when you ask a question, it will try to answer you from all the information that Bitcointalk carries, right? It sounds interesting but I wonder how it will be able to filter information well, there are many wrong and many right answers.
Btw I like the idea, I can work on UI/UX design.

[NotATether]

I have about half of the forum's posts (as of 2025-01-01), it will take a few weeks for me to fetch the other half.

[Vod]

As I understand, it will be like DeepSeek. It learns the whole forum, every post and when you ask a question, it will try to answer you from all the information that Bitcointalk carries, right? It sounds interesting but I wonder how it will be able to filter information well, there are many wrong and many right answers.

Correct - it has the same model so it will provide the same type of responses DS does, however it's knowledge will be limited to the posts from this forum.   I don't think anyone would use it for crypto research, but it's useful for crypto activities on this forum.  For example: a person would not use it to discuss improved methods of storing key phrases, but could use it to get of list of key phrase discussions that involve a certain offline wallet.   It could produce probability that two users are the same, based on their posts or blockchain activity, or give a loan risk rating of a user.

This is where the project can pay for itself.   We can offer community members so many tokens per day, and charge for excess or non-personal use.  This forum was a good development tool in it's non-greedy days, and many developers still look to it.

Btw I like the idea, I can work on UI/UX design.

The UI already looks just like deepseek.   You will need to figure out what controls will work best - we don't want to complicate it so we don't need to update it.    For example, is it worth it to have a dropdown multi-select showing the categories?  Then you can ask the AI your question, and add "in these categories only".  ?

I have about half of the forum's posts (as of 2025-01-01), it will take a few weeks for me to fetch the other half.

Your new data set contains a copy of the recent forum posts.  Archived copies contain an older data set.  By comparing these datasets, the AI can determine deleted posts, edited posts, etc.    I'm curious about the parameters you are collecting, but I'll discuss that in your thread.  I'd like this thread to stay about the actual search engine/AI, and not the data collection behind it.

[shield132]

Correct - it has the same model so it will provide the same type of responses DS does, however it's knowledge will be limited to the posts from this forum.   I don't think anyone would use it for crypto research, but it's useful for crypto activities on this forum.  For example: a person would not use it to discuss improved methods of storing key phrases, but could use it to get of list of key phrase discussions that involve a certain offline wallet.   It could produce probability that two users are the same, based on their posts or blockchain activity, or give a loan risk rating of a user.

This is where the project can pay for itself.   We can offer community members so many tokens per day, and charge for excess or non-personal use.  This forum was a good development tool in it's non-greedy days, and many developers still look to it.

Why wouldn't a person use it to discuss improved methods of storing data? I think similar searches will be the most important aspect of this AI but I really wonder how will AI model be trained. For example, imagine that I have a question - "Is Bitcoin superior to gold"? On this forum, a similar thread has been opened many times and there are thousands of answers. How will AI differentiate good answers from bad answers and give users the most correct answer?

This is where the project can pay for itself.   We can offer community members so many tokens per day, and charge for excess or non-personal use.  This forum was a good development tool in it's non-greedy days, and many developers still look to it.

That's a very smart solution to let the project pay for itself.

[Vod]

Why wouldn't a person use it to discuss improved methods of storing data? I think similar searches will be the most important aspect of this AI but I really wonder how will AI model be trained. For example, imagine that I have a question - "Is Bitcoin superior to gold"? On this forum, a similar thread has been opened many times and there are thousands of answers. How will AI differentiate good answers from bad answers and give users the most correct answer?

I really have no idea how the machine part of deepseek works - I'm actually taking a beginner course on machine learning so I can understand what this is doing:


I assume it will handle it the same way it handles any question - by responding with the best word one after another until they come up with the solution.  lol   It's kind of like a sculptor who looks at a lump of clay and them removes anything that doesn't belong there - it takes a special skill that I don't even understand.  I think I would train it will all the post data from the forum, and then the matrix would be further refined by the people using the engine:


Another solution I am looking at is https://yacy.net/  It offers a distributed search engine, so multiple people could maintain the engine for speed, reliability and security - same as a blockchain.  It might be as simple as uploading all the posts to our server and having the software distribute it amongst us to crawl and analyze.  It's free, so @LoyceV, if you are interested in working on this with me, LMK.   I can't believe I'm typing this, but your flat files may be useful after all. 

I can also just dump my database to flat files,  so we can have multiple copies.  The search engine can then operate much like the Internet Wayback Machine, showing us multiple versions of edited posts.

[Joel_Jantsen]

Hi Vod, if I understood correctly, you want to create an LLM based on the Bitcointalk forum data you've parsed? The LLM could be trained on this data and users can chat with it just like ChatGpt. Doesn't ChatGpt already scrape the web pages citing sources and have access to all the publicly available data? Unless the data we're talking about is private I guess any of these LLMs that scan the webpages for information should act as a search engine. We can create something just inclined towards Bitcointalk and trained exclusively on forum-related information as a hobby project.

[Vod]

We can create something just inclined towards Bitcointalk and trained exclusively on forum-related information as a hobby project.

Yes, that is what I'm planning - they can chat with it just on bct subjects.   

I don't think any LLM would parse the entire bitcoin forum.  I'm not sure of the methods people use to populate, but I don't think it would be a spider crawl.  Instead, they would contact Google and other agencies for data dumps they could use.    The only way we can get a detailed LLM on bct is to train it ourselves.

I think the first thing to do will be create a small team.  Members will include content creators (parsers), the GUI designer and coders, both general compute and ML.  I would join the team just for my general knowledge of AI, but not as any kind of coder or policy maker.  The goal will be to created an uncensored search engine for this forum.   By uncensored, I mean deleted posts would be included, and not that private areas should be parsed.   Any search engine must respect the public visibility.

Does anyone want to champion this idea?

[NeuroticFish]

Hi Vod, if I understood correctly, you want to create an LLM based on the Bitcointalk forum data you've parsed? The LLM could be trained on this data and users can chat with it just like ChatGpt. Doesn't ChatGpt already scrape the web pages citing sources and have access to all the publicly available data? Unless the data we're talking about is private I guess any of these LLMs that scan the webpages for information should act as a search engine. We can create something just inclined towards Bitcointalk and trained exclusively on forum-related information as a hobby project.

Normally the public AI only uses old data - years or months old. If Vod can feed it with very new data, it's already a huge step forward.

I really have no idea how the machine part of deepseek works - I'm actually taking a beginner course on machine learning so I can understand what this is doing

You may want to also read about LLM hallucinations. I don't know how badly is DeepSeek affected, but I've seen some incredibly bad ones at Gemini and Copilot.

[NotATether]

Normally the public AI only uses old data - years or months old. If Vod can feed it with very new data, it's already a huge step forward.

Grok does just that but with X posts.

It's highly effective for keeping up with current events. Sometimes it is as recent as a few hours ago from now.

This only works because users post the daily news there, though.

[Joel_Jantsen]

Yes, that is what I'm planning - they can chat with it just on bct subjects.   

I don't think any LLM would parse the entire bitcoin forum.  I'm not sure of the methods people use to populate, but I don't think it would be a spider crawl.  Instead, they would contact Google and other agencies for data dumps they could use.    The only way we can get a detailed LLM on bct is to train it ourselves.

I think the first thing to do will be create a small team.  Members will include content creators (parsers), the GUI designer and coders, both general compute and ML.  I would join the team just for my general knowledge of AI, but not as any kind of coder or policy maker.  The goal will be to created an uncensored search engine for this forum.   By uncensored, I mean deleted posts would be included, and not that private areas should be parsed.   Any search engine must respect the public visibility.

Does anyone want to champion this idea?

Got you! The fastest way to get access to the forum's public data would be pinging theymos and seeing if he wants to give the whole data dump from the database. Although, highly unlikely, that would be faster than any of the existing coumminty-created data sources. The LLM part should be pretty easy once the data is available as one would train it using any of the 100 open-source models. You'd need to train the model with updated information every few weeks to keep it ready with the latest information. For instance, what scam projects were flagged last week, if asked, the chatbot wouldn't have access to this information.

I'm not super into Data Science or ML side of things but if we had the data information, I could get this ready within a week or two max. Lot of readily available frameworks that provide plug-and-play interfaces for such bots.

[Vod]

I'm not super into Data Science or ML side of things but if we had the data information, I could get this ready within a week or two max. Lot of readily available frameworks that provide plug-and-play interfaces for such bots.

Tell me about it...  I've spent the last couple days watching youtube videos on the various AI tools available.   It's hard to focus on one because new systems are coming out almost daily now.  I saw a nice open source one that is meant to automatically scan the website and build an LLM off of it.   It would need tweaking, since it cannot parse all off bct and will need historical training - but once it's up to date, the forum provides new posts to guests very easily, and the search engine can be just like Grok.
https://www.youtube.com/watch?v=JWfNLF_g_V0

I think LoyceV can provide you will all the original forum posts.   I'm not going to do any coding because by the time I learn how to do something, I discover there is an AI bot that will do it in seconds.  :/  In fact, I'm pulling away from most parsing projects to instead focus on things I enjoy.  I can be an idea guy if you ever need a different perspective of something.

I assume you have resources available for development?   Most cloud companie give free trials.  If it launches and is popular, you'll obviously need to charge per search, or have it forum sponsored. 

[Floxynice]

If this project can be funded, it will fine because I have struggled with the forum search for long. I once raised this topic about forum search
The forum search isn't responding.

As I understand, it will be like DeepSeek. It learns the whole forum, every post and when you ask a question, it will try to answer you from all the information that Bitcointalk carries, right? It sounds interesting but I wonder how it will be able to filter information well, there are many wrong and many right answers.
Btw I like the idea, I can work on UI/UX design.

Even deepseek sometimes returns wrong answer which could depend on the prompt. My own fear lies on abuse. Some users can start using it as AI assistance to make post in the forum.

[JollyGood]

It is an interesting project, however if you are asking for the community to part with money to cover costs for a server that will be a tough proposition.

Keeping that aside, I notice you have an admirer with ulterior motives. "He" claims to be a "she" and hopes to have both neutral tags removed as he enrols in campaigns. Mine will stay, he is hoping flattering you will equate to you removing the one from you  

The community can rent a cloud box with GPU/AI hardware to build a natural language search engine for the bitcointalk content.  We don't need the highest level reasoning for search queries, so we can probably get a deepseek type response time for a few hundred a month, depending on how we limit search queries in the community.

[Vod]

It is an interesting project, however if you are asking for the community to part with money to cover costs for a server that will be a tough proposition.

I agree with you.   Adding a responsive search to this forum would uncover a lot of past corruption (from the administration and their staff)  so I doubt the whales here will spend a single satoshi making information more available.   As a moderator told me a few years ago about working for Theymos - "Ignorance is bliss".  But since I've started looking into this I've found much better ways than using a GPU for parsing LLMs.  JJ has proven he is talented (and I'm hoping Ninja gets involved as well) so I think they will be able to run a small search on standard shared hosting.  It may take them more resources to train the model, but once trained/indexed it can be run easily on free resources from cloud providers.  These frameworks include a token purchase/use construct, so the developers can make their money back that way for people using the search for non-casual uses.   In other words, this will be a project for the community, and not another tool under admin control.

You'd need to train the model with updated information every few weeks to keep it ready with the latest information. For instance, what scam projects were flagged last week, if asked, the chatbot wouldn't have access to this information.

There is no reason the model cannot be updated in real time; think of it as an index.   The forum only gets a few posts a minute.


Edited to remove an answered question

[Ivystar5]

Nice idea, I'm also taking a beginner course on machine learning and i would have love to add to this project if i have anything to add.

I also made this post below regarding the search engine but guys seem to only be interested in already existing search engine and however Tryninja said he has the solution already but i would love to see yours because it will contain a lot including chat-able AI but specifically bitcoin related discussions.

Hello Guys

How do I find my comment or post on a particular thread? apart from going to my post stats (post history) to find it and click on the link how do I find my comment in a mega thread like the wall observer?

I tried to search my username on the search at the right corner of the forum but it gives me a reverse search that shows all my comments on the board and not a particular thread.

I'm just trying out things and this one thing I have experienced didn't work out so how do I find my post without going back each page of the thread to find it?

[Vod]

Nice idea, I'm also taking a beginner course on machine learning and i would have love to add to this project if i have anything to add.

Well, it make take a beginner to see the value in this project!   Right now we don't have a community leader willing to spend on such projects, so it is going to take a leader to step up, create a project website (I can host) and start attracting talent / monitoring progress.  Great opportunity to also test out your favorite AI assistant  - ask it the best way to get started on a new community project and keep us informed on what it says.

Interest is one of the larger drivers of innovation.   While not as powerful as greed, it does allow open source projects that contribute to and develop the crypto industry. 

This developer is what got me interested in ML:   https://www.youtube.com/@PezzzasWork   Look at the AI battle videos to get your imagination running.      Whether you use that ML to better detect prey, or better detect a duplicate bct account, it's still fun to learn and watch. 

[Ivystar5]

Nice idea, I'm also taking a beginner course on machine learning and i would have love to add to this project if i have anything to add.

Well, it make take a beginner to see the value in this project!   Right now we don't have a community leader willing to spend on such projects, so it is going to take a leader to step up, create a project website (I can host) and start attracting talent / monitoring progress.  Great opportunity to also test out your favorite AI assistant  - ask it the best way to get started on a new community project and keep us informed on what it says.

Interest is one of the larger drivers of innovation.   While not as powerful as greed, it does allow open source projects that contribute to and develop the crypto industry. 

This developer is what got me interested in ML:   https://www.youtube.com/@PezzzasWork   Look at the AI battle videos to get your imagination running.      Whether you use that ML to better detect prey, or better detect a duplicate bct account, it's still fun to learn and watch. 

Thanks buddy
I can see that no one has offered sponsorship yet for the project maybe as time goes by you could get more interest 

Just exploring some knowledge!

[Joel_Jantsen]

Tell me about it...  I've spent the last couple days watching youtube videos on the various AI tools available.   It's hard to focus on one because new systems are coming out almost daily now.  I saw a nice open source one that is meant to automatically scan the website and build an LLM off of it.   It would need tweaking, since it cannot parse all off bct and will need historical training - but once it's up to date, the forum provides new posts to guests very easily, and the search engine can be just like Grok.
https://www.youtube.com/watch?v=JWfNLF_g_V0

Yeah, it's insane how so many of these tools are popped off recently! These are all wrappers over the free LLM's offered by the likes of Facebook/Google/others. We wouldn't go for a wrapper since no one would want to pay them monthly and also it's not so fun! We would need to create a crawler that would either parse the whole bitcointalk or just train one of those FOSS models on data provided by LoyceV/Theymos/Bitcointalk.

I think LoyceV can provide you will all the original forum posts.   I'm not going to do any coding because by the time I learn how to do something, I discover there is an AI bot that will do it in seconds.  :/  In fact, I'm pulling away from most parsing projects to instead focus on things I enjoy.  I can be an idea guy if you ever need a different perspective of something.

To give a different perspective as someone who writes code for 8+ hours a day, AI actually sucks at coding! Fine, ChatGpt/Claude can create a CRUD APP or a React Component with some boilerplate code in seconds but real-world apps are much more complicated than that and even complicated if you're working with event-driven architectures, payment gateways, etc. It's knowledge and reasoning are very limited and at very best you can use AI as an assistant to write repetitive or boilerplate code. So yeah don't hesitate to learn programming!

I assume you have resources available for development?   Most cloud companie give free trials.  If it launches and is popular, you'll obviously need to charge per search, or have it forum sponsored. 

Don't think I can run any of those models on my laptop...I can have a chat with couple of my Data Engineer friends and see what they recommend!

[Silentcursor]

Is there anyone that would be interested in working on this?  Do you know how to populating an AI model?    If we can find someone willing to do that, then myself and other archive provers like Loyce and NotATether can provide the data to populate. 

I have a friend who is capable of doing this, i can reach out to him for this. If that is fine by you.