Storing the bitcointalk forum on a decentralized file system

[SamReomo]

If this ever comes near reality, I would strongly oppose to store the images on-chain or in the main torrent. That could attract the wrong people. Images can still be included via external servers like talkimg, or via external image torrents.

In that case the size of torrent can be pretty small and anyone can easily seed it and others can download it and become its seeders. I believe the one who may create such torrent should take responsibility to create more torrent based backups on weekly or monthly basis so each new post/thread of the forum can be updated and the downloaders can get updated data easily.

It would be even better if the images can be uploaded on external servers because that way the downloads can easily browse the forum's posts/threads and may click on that external links. TalkImg as we all know is a great option for the purpose and I believe Joker_josue will allow such type of uploading of images on his site.

[dkbit98]

If something like Nostr.com can exist for decentralized social media network, and it keep growing despite everything, than I am sure that we could have something similar to work as decentralized forum.
Question is how many people are interested in using forums these days.

In reality there are only a few people holding the coin (any coin - even Bitcoin) who actually care about consensus, community, voting, governance, etc.

True, but there are a lot more bitcoiners who care about real value and decentralized, than there are altcoiners who are doing the same thing.
And big difference is that bitcoin was no premined, there was no pre-sale of coins, and there is no hidden team and owner behind. 

[joker_josue]

I would *love* to make a search engine for the forum though. It has always been one of my pet peeves and I think it would be much more useful than what is essentially a decentralized backup of the forum, given that Google and the built-in search are terrible at finding posts in large topics.

You mean like Ninjastic Search? If you want to build another search engine, I'd be more than happy to get you a downloadable archive with 60-ish million posts (in separate files)

Creating a good search system is very complex and difficult to do. It took Google years to get to the point where we know it today. Of course, a search system could be created in a database, but it would require a lot of work to make everything really better.

It would be even better if the images can be uploaded on external servers because that way the downloads can easily browse the forum's posts/threads and may click on that external links. TalkImg as we all know is a great option for the purpose and I believe Joker_josue will allow such type of uploading of images on his site.

I didn't understand your idea. Can you explain better?

[SamReomo]

I didn't understand your idea. Can you explain better?

The idea is simple, someone who may download hosted images from imgur and other image hosts and then upload those images to TalkIMG and may link those images to the posts/threads of the backup torrent.

Let's say if someone who's going to create a torrent backup of the forum then he/she has to either avoid images or add images to the torrent file which will make the size of the download huge but if that person somehow develops a program where all those images could be uploaded to TalkImg and then either get embedded to the posts/threads they belong to or they may be added to the posts/threads as clickable links.

I know it's easy to say something like that but doing it is going to be a tough job but on our forum there are some members who are good programmers and they have in depth knowledge of web scaping. And those skills are highly needed to turn that idea into reality.

[uchegod-21]

If something like Nostr.com can exist for decentralized social media network, and it keep growing despite everything, than I am sure that we could have something similar to work as decentralized forum.
Question is how many people are interested in using forums these days.

I think I am more interested in having a secured backup of the present forum incase anything happens to the present, than having a decentralized bitcointalk.

[joker_josue]

I didn't understand your idea. Can you explain better?

The idea is simple, someone who may download hosted images from imgur and other image hosts and then upload those images to TalkIMG and may link those images to the posts/threads of the backup torrent.

Let's say if someone who's going to create a torrent backup of the forum then he/she has to either avoid images or add images to the torrent file which will make the size of the download huge but if that person somehow develops a program where all those images could be uploaded to TalkImg and then either get embedded to the posts/threads they belong to or they may be added to the posts/threads as clickable links.

I know it's easy to say something like that but doing it is going to be a tough job but on our forum there are some members who are good programmers and they have in depth knowledge of web scaping. And those skills are highly needed to turn that idea into reality.

Nothing unusual to be able to do. All the system needs to do is load all the images onto the host, and then just list the link.

But in reality, this doesn't even need to be done, as the forum does not host any images, just their links. So, practically this would not be necessary.

[larry_vw_1955]

I think I am more interested in have a secured backup of the present forum incase anything happens to the present, than having a decentralized bitcointalk.

the problem with that is there is no secured backup of the present forum if you just rely on a couple people to do this backup. what happens if they go offline for some reason. then they're not doing backups anymore. how long until people realize that and someone else takes their place. i wouldn't say bitcointalk needs to be decentralized but its data needs to be maybe. but as we have seen, that's easier said than done and we might be stuck with the centralized pay per month plan. i have lost interest in this because it seems like it is a very high cost factor for any type of decentralized storage unless it is the users who are paying for it somehow.

[LoyceV]

Creating a good search system is very complex and difficult to do. It took Google years to get to the point where we know it today.

Actually....... Google was very good at the beginning, so the search part they had covered. The point where they are today is based on earning money, and no longer on providing the best possible search results for the user.
Ninjastic Search works fine for searching, but only if you type the exact right keywords. That's what Google excelled at: showing me what I want to see, even if I don't know exactly what it is. If I place my right hand incorrectly and type "botcpom", Google knows I mean "bitcoin". But that would go far beyond the scope of a forum search engine.

To get back on-topic: a fully decentralized forum would also mean a fully decentralized search system. That's great for privacy, nobody else will know what you're looking for.

The idea is simple, someone who may download hosted images from imgur and other image hosts and then upload those images to TalkIMG and may link those images to the posts/threads of the backup torrent.

From imgur alone I downloaded 801,550 pictures and 146 GB. But there's a bigger problem: a decentralized forum should be immutable. If someone can edit the image links in all posts, you can't be sure what else they've edited.

[Smartvirus]

If something like Nostr.com can exist for decentralized social media network, and it keep growing despite everything, than I am sure that we could have something similar to work as decentralized forum.
Question is how many people are interested in using forums these days.

This would have been some of the best means by which the most values of the Bitcoin or crypto industry would have been achieved with the forum. Given the fact that, there wouldn’t be any threat of getting it offline based on certain allegations that might arise from certain disjointed associations and the need to prove what, like we had in the mixer case. We wouldn’t have been threatened by that and probably wouldn’t have the decision we had on mixers but, how possible would it be.
Bitcoin is decentralized but the forum is centralized and there are known parties to its hosting and governing which make any of the possibilities to these not so feasible.

[nutildah]

I can envision a system where 10-minute batches of posts are "mined" in a new PoW chain that stores the forum database. Of course it would have to come with a "premine" of all the old posts. There are certain sections of the forum that I would omit entirely, including (but not limited to):

- Marketplace
- Politics & Society
- Off-topic
- Archival
- Marketplace (Altcoins)

Cutting out these sections - along with all images - would reduce the initial database size by at least half, I'm guessing. Would be a fun project; if I had all the time in the world I'd probably try to launch it.

[LoyceV]

There are certain sections of the forum that I would omit entirely

For a scalable decentralized forum, I'd expect the user to be able to choose which sections they want to keep. So as long as there's at least one person uploading Off-topic, it will exist. By the time nobody has a copy left, it's clearly not important enough to keep anymore.

[dkbit98]

I think I am more interested in have a secured backup of the present forum incase anything happens to the present, than having a decentralized bitcointalk.

There are plenty of archived posts already with waybackmachine, everything is saved by Ninjastic and I am sure that theymos is making regular backups of everything.
You can also archive anything you want.

Actually....... Google was very good at the beginning, so the search part they had covered. The point where they are today is based on earning money, and no longer on providing the best possible search results for the user.

Hence they removed ''Don't be evil'' moto 
Some people say that g00gle also removed a bunch of history and they altered facts many times.

To get back on-topic: a fully decentralized forum would also mean a fully decentralized search system. That's great for privacy, nobody else will know what you're looking for.

Maybe, but I think this search would be much slower and not that precise in real life.

I can envision a system where 10-minute batches of posts are "mined" in a new PoW chain that stores the forum database. Of course it would have to come with a "premine" of all the old posts. There are certain sections of the forum that I would omit entirely, including (but not limited to):

I don't see any good reason why anyone would do this when there are free alternatives like IPFS.

[joker_josue]

Cutting out these sections - along with all images - would reduce the initial database size by at least half, I'm guessing. Would be a fun project; if I had all the time in the world I'd probably try to launch it.

But the forum does not store images. That is why this aspect does not even arise in this matter.
The issue will be navigation time and content updating. Making a post and it only being available 10 minutes later for everyone, I don't think is ideal.

[PowerGlove]

Actually....... Google was very good at the beginning, so the search part they had covered.

You're completely right about that: Google Search was a much better product 20 years ago... I don't know what has gone wrong with that product team, but they've taken something that used to be eerily good at what it did, and somehow turned it into a completely useless POS.

(I mean, sure, the web was smaller and nicer back then, so search in general was a more tractable problem, but even so, Google really did have something special about their ethos and engineering philosophy back in the day; these days, not so much: they wandered from the path and basically never found their way back onto it... I never choose Google for anything anymore, now I just use their stuff when I'm effectively forced to.)


On-topic: No offense, but this thread reads like a bunch of first-year nursing students talking about how to do a triple bypass.

Figuring out the sequence of 4D chess moves that would eventually map the forum's database onto some kind of custom rendezvous-hashing-type distributed storage layer, while keeping everything running basically as-is (leading up to, during, and after the migration), is not beyond the realm of possibility, but it's pretty much at the tippy top of the complexity totem pole in terms of things that could be done to Bitcointalk. Show me the programmer that estimates they could pull off a task like that, and 999 times out of 1000, I'll show you someone that doesn't really understand the problem.

[d5000]

I can envision a system where 10-minute batches of posts are "mined" in a new PoW chain that stores the forum database.

This is a bit similar to the "sidechain" idea I proposed here.

I think there's the same challenge to solve than in my proposal: The "miners" would have to be rewarded in some way to incentive them to store the correct contents. But how can you be sure that they store the correct content, and not some randomly generated fake posts which would take them probably less time than to scrap the forum?

Perhaps this could be solved in the same way like Optimistic Rollups work. The rewards to the "forum miners" are locked for some time, let's say a week or two. If there is somebody who discovers that the contents added by a "miner" don't exist or differ from the forum contents, they can post a proof. The reward is then slashed and the miner perhaps even penalized in another way (slashing a security deposit, for example), and the fake content is pruned, and an additional "miner" gets the opportunity to store the missing 10-minute window.

Of course this raises additional challenges: there may be miners trying to attack the forum and store the randomly generated fake posts there instead of the "real" posts, to be able to get the reward. Perhaps as an option of last resort a DAO vote (for example via PoS) could he held if there's some doubt about the correct forum content. Just to ensure there's not the slightest incentive to try such attacks.

Figuring out the sequence of 4D chess moves that would eventually map the forum's database onto some kind of custom rendezvous-hashing-type distributed storage layer, while keeping everything running basically as-is (leading up to, during, and after the migration), is not beyond the realm of possibility, but it's pretty much at the tippy top of the complexity totem pole in terms of things that could be done to Bitcointalk.

I think the "Steemit solution" should work without problems, but it would take a lot of preparation time of course. And in general I think it's legit to discuss such ideas, even if some ideas sound impractical. Even theymos has mentioned eventually that he'd prefer a decentralized option for Bitcointalk for the far future.

[PowerGlove]

And in general I think it's legit to discuss such ideas, even if some ideas sound impractical.

Yup, it's worth discussing. My nurses-talking-about-heart-surgery comment wasn't a dig (seriously, I know this stuff is fun to discuss). But... holy shit is this a tough problem. Basically the first sign that you (the proverbial you, not you specifically, of course) wouldn't even be able to make an impression (let alone a serious dent) is if you catch yourself thinking of the solution in terms of existing building blocks. I know that sounds harsh, but, from a programming point of view, if something like a completely from-scratch distributed storage layer is an amount of complexity/work that you'd prefer to avoid (by trying to take advantage of something that looks suitable and has already been built), then you've already demonstrated that (in all likelihood) you don't have the software engineering chops to handle the rest of the complexity involved. I mean, maybe I set the bar unnecessarily high in my previous post, by imagining it as an in-place upgrade with nothing lost along the way (feature-wise, UI-wise, every last bit of the database comprehensively preserved, etc.), but if you relax those constraints and simply say: "Can you make a decentralized forum similar to Bitcointalk that might one day be able to serve as its replacement?", then I'd estimate that to be a much easier lift.

(But, I've been programming since I was an ankle-biter, and all my years doing it have left me with a pretty depressing view: I think programming, as a profession, is in decline, and most of the legends have already left the building, so to speak. So, even in the watered-down version of this problem, I'd still expect most every attempt to end in failure. If not outright failure, then the slow, eventual failure that comes after you basically force a win by hiding the fact that you're dealing with more complexity than you know how to tame.)

[nutildah]

I think there's the same challenge to solve than in my proposal: The "miners" would have to be rewarded in some way to incentive them to store the correct contents. But how can you be sure that they store the correct content, and not some randomly generated fake posts which would take them probably less time than to scrap the forum?

Maybe it would be possible to assign forum contents to be "mined" in a block to a hash, and if the contents don't match the hash, the block is invalid. Of course, posts can be updated, and are continually edited or removed, so there'd have to be a very scientific, precise way in which the hash is calculated. And whichever miner calculates the correct hash first receives the block reward.

I don't see any good reason why anyone would do this when there are free alternatives like IPFS.

Because there's no motivation/incentive for anyone to continually archive the forum, forever. We are currently reliant on the goodwill of TryNinja and LoyceV to properly archive forum posts.

Show me the programmer that estimates they could pull off a task like that, and 999 times out of 1000, I'll show you someone that doesn't really understand the problem.

I don't doubt it 

[larry_vw_1955]

I think the "Steemit solution" should work without problems, but it would take a lot of preparation time of course. And in general I think it's legit to discuss such ideas, even if some ideas sound impractical. Even theymos has mentioned eventually that he'd prefer a decentralized option for Bitcointalk for the far future.

the steem blockchain has blocks just like bitcoin's blockchain does. and the "transactions" in a block can be different types of activities like replies to postings, postings themselves, upvotes, etc. the steem blockchain apparently records them all!

if we only had the source code to the steem blockchain. oh that's right we do. since it's on github. now all we would need to do is write a "bitcointalk" layer on top of it ala "steemit". steemit is a very complex codebase. but all it is is one huge forum. do we know how to write forums?

the steem blockchain stores all the postings, replies, forum activities that happen on steemit. the only thing it doesn't do is store images. people can put images into their postings though. but those get stored on ipfs. apparently.

Because there's no motivation/incentive for anyone to continually archive the forum, forever. We are currently reliant on the goodwill of TryNinja and LoyceV to properly archive forum posts.

that's why you have to turn it into a blockchain. one where there are incentives for being involved. steem has validators. i assume they run full nodes. so all 21 of them are storing the entire steem blockchain. i'm sure loyce would do that.  

[LoyceV]

Some people say that g00gle also removed a bunch of history and they altered facts many times.

Just like I can no longer use it to find plagiarism. It doesn't show any matches nowadays.

I think this search would be much slower and not that precise in real life.

"Less precise" is a feature for searching, and that's what Ninjastic can't do: it only shows exact matches. A good search system shows you what you're looking for.

You're completely right about that: Google Search was a much better product 20 years ago... I don't know what has gone wrong with that product team

Priorities must have changed. The founders own 140 billion dollars (each), and the priorities are share holders. Just like their advertising shifted from useful text ads to completely overdoing it everywhere.

Google really did have something special about their ethos and engineering philosophy back in the day

Money changes people, and companies. They would probably not be as big as they are now if they would have remained a friendly company.

On-topic: No offense, but this thread reads like a bunch of first-year nursing students talking about how to do a triple bypass.

I see where this is going: the Mother of all Patches that decentralizes the forum!

Show me the programmer that estimates they could pull off a task like that, and 999 times out of 1000, I'll show you someone that doesn't really understand the problem.

I can show^* you the programmer (and team), but it comes with a corporate price tag. And that's the problem: it's expensive to create, and can't really be monetized.

^* Not literally though, because of privacy.

But... holy shit is this a tough problem.

That's probably why it doesn't exist yet. And you're only looking at it from a technical perspective. To be useful, it needs moderation to tackle spam.

I'll quote this post here:

I wish that more work was being done on creating decentralized, uncensorable, but also usable forums. I would love nothing more than to be able to shut down bitcointalk.org due to some decentralized solution making it obsolete. Find me a 501(c)(3) nonprofit working on this, and I'll donate to it.

Because there's no motivation/incentive for anyone to continually archive the forum, forever. We are currently reliant on the goodwill of TryNinja and LoyceV to properly archive forum posts.

This was (and is) my incentive:

Ever wanted to see who's lying when a post has been edited or deleted? I may be able to help!

I can't make any promises on the "forever" part. Well, actually I can: I will most definitely not do this forever But I can promise not to abandon it willingly without trying to find someone who takes over if when it comes to that.

that's why you have to turn it into a blockchain. one where there are incentives for being involved.

Incentives need to be paid by someone.

i assume they run full nodes. so all 21 of them are storing the entire steem blockchain. i'm sure loyce would do that.  

I don't have blocks, but I have sha 256 hashes: anyone can archive them to prove editing later on. Read the posts around the linked one for the discussion on data integrity.

[larry_vw_1955]

I'll quote this post here:

I wish that more work was being done on creating decentralized, uncensorable, but also usable forums. I would love nothing more than to be able to shut down bitcointalk.org due to some decentralized solution making it obsolete. Find me a 501(c)(3) nonprofit working on this, and I'll donate to it.

as far as i can tell, you're not going to get something closer to that wishlist than something like steemit. so it's already a solved problem. steemit is a huge forum. it's only weakness is the main interface is a website but that doesn't mean someone couldn't code a software application that could run locally off of the steem blockchain.

steemit already has "subforums" and things.

but i'm not really interested in pursuing this discussion further since in my opinion it's already a solved problem. no need in beating a dead horse...

just for reference, steemit announced on bitcointalk when it first opened up back in 2016 so it shouldn't be any secret: https://bitcointalk.org/index.php?topic=1466593.0

and what did the announcement say?

Welcome to Steemit, decentralized and incentivized social media.

Steemit is a new social media platform built on the Steem blockchain where the community rewards individuals for their posts, comments and votes, co-founded by Ned Scott, CEO, and Daniel Larimer, CTO of Steemit and founder of Bitshares.

Underneath the hood of Steemit.com there is a tradeable cryptocurrency token, called Steem, and it's performing well on the US-based exchange, bittrex.com, and blockchain-based exchange openledger.info. With Steemit.com, posts, comments and votes are immutable.  Attribution is a given. Accreditation is not taken for granted. And community appreciation is an actual reward.