Storing the bitcointalk forum on a decentralized file system

[BlackHatCoiner]

"EThe Bitcointalk Times 18/06/2024 Larry brings the forum into the chain". Here you go, that's the genesis message for if it is ever archived in the blockchain. 

Jokes asides, it takes a lot of money to store it in the blockchain. I just copied and pasted every post above mine, and compressed it. It's 5.6 kilobytes. With 10 sat/vb at low priority, it'll cost 56000 sat just for this single thread (until this 21th post).

It isn't worth the money.

[larry_vw_1955]

"EThe Bitcointalk Times 18/06/2024 Larry brings the forum into the chain". Here you go, that's the genesis message for if it is ever archived in the blockchain. 

maybe bitcoin could have a sidechain that stored the forum. and people could get rewarded with bitcoin when they stored significant parts of it. like mining the forum.  instead of proof of work it would be proof of storing the forum.

I would say that having backups of the forum as torrents is going to be a easier solution and a long-term one. There are some torrent sites that always release the backup of their torrents library and such thing can be done with this forum as well. But doing something like that is going to be costly if it isn't supported by the community who may seed those torrents for the peers, and leechers.

But how big can torrents be allowed to be? Because this forum has to be terabytes in size by this point, after being online for over 13 years. I would guess terabytes but I really am not sure. But lets assume it is terabytes. I don't think too many people would seed something that big. Unless they were getting paid hence my sidechain idea above.

Lets say 100 new threads are posted to the forum everyday and each thread is 5 kilobytes. Then that is 500kb per day. that works out to about 182GB per year. Over 10 years, that's almost 2 TB. it is probably even more than that, in reality though...

[ABCbits]

"EThe Bitcointalk Times 18/06/2024 Larry brings the forum into the chain". Here you go, that's the genesis message for if it is ever archived in the blockchain. 

maybe bitcoin could have a sidechain that stored the forum. and people could get rewarded with bitcoin when they stored significant parts of it. like mining the forum.  instead of proof of work it would be proof of storing the forum.

So who rewards those who store forum data on the sidechain?

I would say that having backups of the forum as torrents is going to be a easier solution and a long-term one. There are some torrent sites that always release the backup of their torrents library and such thing can be done with this forum as well. But doing something like that is going to be costly if it isn't supported by the community who may seed those torrents for the peers, and leechers.

But how big can torrents be allowed to be? Because this forum has to be terabytes in size by this point, after being online for over 13 years. I would guess terabytes but I really am not sure. But lets assume it is terabytes. I don't think too many people would seed something that big. Unless they were getting paid hence my sidechain idea above.

Lets say 100 new threads are posted to the forum everyday and each thread is 5 kilobytes. Then that is 500kb per day. that works out to about 182GB per year. Over 10 years, that's almost 2 TB. it is probably even more than that, in reality though...

I recall the theoretical limit is around few exabytes, although in practice it's limited by your device. And don't forget you can save lots of storage/bandwidth by compression, https://softwarerecs.stackexchange.com/a/49216.

[larry_vw_1955]

So who rewards those who store forum data on the sidechain?

so if the bitcointalk forum was running on a sidechain, people could be required to use a native token to pay to post. the native token would have been bought using bitcoin.

I recall the theoretical limit is around few exabytes, although in practice it's limited by your device. And don't forget you can save lots of storage/bandwidth by compression, https://softwarerecs.stackexchange.com/a/49216.

yes data compression could be useful but it does have costs. the cost is in terms of compressing and then everyone that wants to read it has to decompress it. so it uses energy.

[d5000]

maybe bitcoin could have a sidechain that stored the forum. and people could get rewarded with bitcoin when they stored significant parts of it. like mining the forum.   instead of proof of work it would be proof of storing the forum.

A way how this could be achieved:

- Sidechain has a native token and a Bitcoin bridge (see this thread for some existing mechanisms), let's call the pegged token "bridgeBTC". Mining could be managed via proof-of-work/merged mining, or alternatively a PoS system.
- Forum storage takes place on-chain (like on Steem). Both full nodes (which store the forum) and those who store new parts of the forum (which weren't stored before) via new data transactions get a status similar to "masternodes" in other altcoins.*  Transaction fees however will be probably relatively low.
- The standard client proposes, for each bridgeBTC and native sidechain token transaction, to pay a small part to those "masternodes" creating the transactions that store the forum. This fee is however voluntary and can be canceled.

This is still a quite "fun" solution I don't think will ever materialize. But maybe somebody can even make profit with a semi-centralized "BitcoinTalkCoin".

But how big can torrents be allowed to be?

You could store all subforums separately, then it will be much smaller already. And you could create yearly/monthly snapshots with only the new topics. In general I'm quite fond of the torrent/IPFS idea. It will also show which subforum is really the most popular, and which ones would see almost no interest in storing or even downloading it.


* To know which sections are legitimately "newly stored" could be a quite difficult challenge. It maybe is impossible - at least without the forum operators themselves acting like a sort of "oracle" confirming the new threads are legit. Or there could be PoS voting (a kind of DAO) which new parts are "accepted".

[joker_josue]

But how big can torrents be allowed to be?

You could store all subforums separately, then it will be much smaller already. And you could create yearly/monthly snapshots with only the new topics. In general I'm quite fond of the torrent/IPFS idea. It will also show which subforum is really the most popular, and which ones would see almost no interest in storing or even downloading it.


* To know which sections are legitimately "newly stored" could be a quite difficult challenge. It maybe is impossible - at least without the forum operators themselves acting like a sort of "oracle" confirming the new threads are legit. Or there could be PoS voting (a kind of DAO) which new parts are "accepted".

The way the site's content is stored can be a problem. But the even bigger problem is making the site functional for anyone. All information must be accessible within seconds, for anyone who enters the site. No need for downloads or other extra programs to access.

Furthermore, all information must be 100% online, and it is not possible for someone who has certain content on their PC to go offline, meaning there is no longer access to the information.

This is not simply a question of storage, but of the entire functioning of the forum.

[ABCbits]

So who rewards those who store forum data on the sidechain?

so if the bitcointalk forum was running on a sidechain, people could be required to use a native token to pay to post. the native token would have been bought using bitcoin.

I'm sure it's possible to build such system, but how many people willing to pay for that? https://stacker.news/ has somewhat similar system, but with limited success.

I recall the theoretical limit is around few exabytes, although in practice it's limited by your device. And don't forget you can save lots of storage/bandwidth by compression, https://softwarerecs.stackexchange.com/a/49216.

yes data compression could be useful but it does have costs. the cost is in terms of compressing and then everyone that wants to read it has to decompress it. so it uses energy.

That's true, you can't get everything. But at least you just need to decompress it once.

[mindrust]

Accessibility will be a problem too. Devs could probably code a decentralized forum software using the p2p stuff which the torrent networks use but I am not sure if it will he possible to attach a domain name to that software. So people will have to run a program to access the forum and that means no google seo. Which means nobody will be accessing the forum via google, means the advertisers won’t be happy about it… meaning > no sig camps

Decentralization is not the solution to every problem it seems.

[d5000]

The way the site's content is stored can be a problem. But the even bigger problem is making the site functional for anyone. All information must be accessible within seconds, for anyone who enters the site. No need for downloads or other extra programs to access.

I believe the OP meant the "decentralized storage" as a backup solution, not as an actual proposal to move Bitcointalk entirely to a decentralized platform.

But even if this was the case, there were already models for on-chain stored "forums" or "social media" that worked. I had discussed that here on the Meta subforum in this thread. In my case I have had a bit of experiences with Steem(it), and it worked reasonably well. Of course it has centralization problems but these are not directly related to the fact that it's stored on a blockchain.

In the case Bitcointalk is moved to a Steemit-like platform completely, viewing content and also posting could be provided by independent websites which access the blockchain with a full node. The software needed is a combination of a block explorer (to view the content) and an online wallet (to post yourself). Every forum user would need to create at least one account/address/key pair for the blockchain to post, but that can be hidden (so in fact all what you have to do is to "register" on one of the "node websites"). As Steemit showed, there were basically no differences in reaction time and availability to a centralized forum. The node websites could run ads and sponsor the transaction fees for posting, like it occurred on Steemit basically (although there the model was a bit more complex).

In reality, this concept would be more on-topic here than it was in the other thread I linked above, so I guess the concept could be elaborated deeper here in this thread.

[joker_josue]

But even if this was the case, there were already models for on-chain stored "forums" or "social media" that worked. I had discussed that here on the Meta subforum in this thread. In my case I have had a bit of experiences with Steem(it), and it worked reasonably well. Of course it has centralization problems but these are not directly related to the fact that it's stored on a blockchain.

I don't know if it works that well... The times I tried to access it, I felt very slow and sometimes I couldn't even access it.
But, there always ends up being some centralization point. Someone has to pay for the domain, a server that connects/processes all the nodes, and in this case, even cloudflare.

I'm not saying it's impossible, just that at the moment it's still not very viable for a site as large as the BitcoinTalk.

[nakamura12]

The way the site's content is stored can be a problem. But the even bigger problem is making the site functional for anyone. All information must be accessible within seconds, for anyone who enters the site. No need for downloads or other extra programs to access.

I believe the OP meant the "decentralized storage" as a backup solution, not as an actual proposal to move Bitcointalk entirely to a decentralized platform.

I think you missed the point in my opinion and I don't think that it's all about moving Bitcointalk forum rather it's all about having a backup or copy of the bitcointalk forum if you ask me and store it in another site where you can access it anytime you want by using the search function in the platform if a person wants to check a certain information. If I understand or get the point and that I am not wrong then it should be similar to bitcointalk forum or maybe not except there is no post feature means we cannot post in the site and we can only add new information (if there's a feature to manually add something) if there are new posts/information posted on bitcointalk forum or automatically added on the site/platform which I can say that it is archive site/platform that looks like bitcointalk forum or not or maybe the same as what TryNinja have created https://ninjastic.space/topic/5499856.0

[joker_josue]

The way the site's content is stored can be a problem. But the even bigger problem is making the site functional for anyone. All information must be accessible within seconds, for anyone who enters the site. No need for downloads or other extra programs to access.

I believe the OP meant the "decentralized storage" as a backup solution, not as an actual proposal to move Bitcointalk entirely to a decentralized platform.

I think you missed the point in my opinion and I don't think that it's all about moving Bitcointalk forum rather it's all about having a backup or copy of the bitcointalk forum if you ask me and store it in another site where you can access it anytime you want by using the search function in the platform if a person wants to check a certain information. If I understand or get the point and that I am not wrong then it should be similar to bitcointalk forum or maybe not except there is no post feature means we cannot post in the site and we can only add new information (if there's a feature to manually add something) if there are new posts/information posted on bitcointalk forum or automatically added on the site/platform which I can say that it is archive site/platform that looks like bitcointalk forum or not or maybe the same as what TryNinja have created https://ninjastic.space/topic/5499856.0

But copies of the forum already exist! He gave the example of the TryNinja website, and even LoyceV does the same.

And I think anyone can do the same. I remember seeing that someone had a mirror forum, where it was visually the same but you couldn't post.

[SamReomo]

But how big can torrents be allowed to be? Because this forum has to be terabytes in size by this point, after being online for over 13 years.

Yes, it may have Terabytes in size but it can be easily compressed to very small size because text is highly compressible and that's a positive point. Even the images of the forum can be compressed in batch if someone makes a batch converter for it.

I believe the forum is mostly based on text based posts/threads and those can be highly compressed by anyone who may then upload the torrent by seeding it and then others who want it to be online can also download it and start seeding it.

[d5000]

I don't know if it works that well... The times I tried to access it, I felt very slow and sometimes I couldn't even access it.

Strange, I had used it for some weeks, writing also a couple of posts there (and then lost interest), and never had any problems regarding performance. This was in 2016 or 2017 though, maybe they scaled down the infrastructure, at least on the steemit.com server.

But, there always ends up being some centralization point. Someone has to pay for the domain, a server that connects/processes all the nodes, and in this case, even cloudflare.

There could be several server operators competing here. They could add their own advertisements for example, or finance the server and domain via the mining/minting process. It's likely that one server would become the most popular one (due to duplicate content issues on Google, for example) but as long as other possibilities exist, the risks related to centralization are minimal, because if the operator misbehaves there's another option around the corner.

And for those who want to access directly the content on the chain without wanting to access via a "centralized" domain, there is no technical hurdle to simply write a client where also a content browser / posting interface is included.

But I sorta agree - this is something for the far future, considering not even Epochtalk is ready after 10 years or so ... But I personally think positively about that option

Regarding the backup copies that already exist (TryNinja etc.), I think the problem is that while these copies are stored only by a few people, there's still a relatively high risk they get lost, or corrupted, or whatever. So in general the OP's idea is not bad, I think.

Even the images of the forum can be compressed in batch if someone makes a batch converter for it.

If this ever comes near reality, I would strongly oppose to store the images on-chain or in the main torrent. That could attract the wrong people. Images can still be included via external servers like talkimg, or via external image torrents.

[larry_vw_1955]

But copies of the forum already exist! He gave the example of the TryNinja website,

that's not a very user friendly webiste.

and even LoyceV does the same.

only thing i see on Loyce.club is various links to statistics about the forum but not the actual forum. loyce seems all about statistics. but those are some crazy statistics. people who are members on this forum, loyce is going to have so much data on them that he can pin down exactly where they live at some point!
if you thought bitcoin wasn't anonymous then thank's to loyce's various statistics he can tell you anything about anyone. scary world we live in!

And I think anyone can do the same. I remember seeing that someone had a mirror forum, where it was visually the same but you couldn't post.

that's the one thing i haven't seen. a backup read only copy of the forum.

I believe the OP meant the "decentralized storage" as a backup solution, not as an actual proposal to move Bitcointalk entirely to a decentralized platform.

yes as a backup from which the original forum could be reconstructed if the main site ever went missing in action.

But even if this was the case, there were already models for on-chain stored "forums" or "social media" that worked. I had discussed that here on the Meta subforum in this thread (https://bitcointalk.org/index.php?topic=5481514.msg64087445#msg64087445). In my case I have had a bit of experiences with Steem(it) (https://steemit.com), and it worked reasonably well. Of course it has centralization problems but these are not directly related to the fact that it's stored on a blockchain.

In the case Bitcointalk is moved to a Steemit-like platform completely, viewing content and also posting could be provided by independent websites which access the blockchain with a full node. The software needed is a combination of a block explorer (to view the content) and an online wallet (to post yourself). Every forum user would need to create at least one account/address/key pair for the blockchain to post, but that can be hidden (so in fact all what you have to do is to "register" on one of the "node websites"). As Steemit showed, there were basically no differences in reaction time and availability to a centralized forum. The node websites could run ads and sponsor the transaction fees for posting, like it occurred on Steemit basically (although there the model was a bit more complex).

the should be trying to move bitcointalk to a steemit-like platform so that continuity is transparent and it has some degree of decentraliztion. i guess though some people go by the "if it aint broke then don't fix it" methodology. but that can end up being challenged if something happens like a site gets hacked and usernames/logins get compromised.

i dont think there's much more to discuss at the moment about this topic since i think another thread also touched on this issue. but centralized forums are not ideal. even something like reddit, they are centralized. they will need to move to blockchain eventually or be overtaken.

[NotATether]

But copies of the forum already exist! He gave the example of the TryNinja website,

that's not a very user friendly webiste.

and even LoyceV does the same.

only thing i see on Loyce.club is various links to statistics about the forum but not the actual forum. loyce seems all about statistics. but those are some crazy statistics. people who are members on this forum, loyce is going to have so much data on them that he can pin down exactly where they live at some point!
if you thought bitcoin wasn't anonymous then thank's to loyce's various statistics he can tell you anything about anyone. scary world we live in!

I might be nit-picking here, but I think TryNinja's site is more accessible than LoyceV's site. Obviously loyce.club is essentially an HTML warehouse with historical information, so it is not really a fair comparison.

I would *love* to make a search engine for the forum though. It has always been one of my pet peeves and I think it would be much more useful than what is essentially a decentralized backup of the forum, given that Google and the built-in search are terrible at finding posts in large topics.

[LoyceV]

I think @LoyceV and @TryNinja might able to answer these question, since they own copy of this forum.

A post archive isn't the same as a working forum. Let's say Bitcointalk is gone tomorrow. I have backups of most of the posts, but it's the unedited version. Millions of posts that have been deleted are still in my archive. I know who posted it, but I can't link the forum identity to a real person. So even if I (or someone else) would use the post data to create a new forum on a new domain with all usernames copied, it's going to be impossible to give the real account owner access to their own posts. Some may be able to sign a message, but that's going to be a lot of work and will only reach a small part of the users. And since the forum is gone, most users won't even know there's an effort to create a new domain name for the forum.
If Bitcointalk.org disappears, I expect it to just be the end of this forum.

only thing i see on Loyce.club is various links to statistics about the forum but not the actual forum.

My posts archive is just that: an archive. I have no intention to make a working forum, that's not the point of archiving posts.

people who are members on this forum, loyce is going to have so much data on them that he can pin down exactly where they live at some point!
if you thought bitcoin wasn't anonymous then thank's to loyce's various statistics he can tell you anything about anyone. scary world we live in!

I love the BS accusations! I couldn't care less about where you live, and only archive public data. If you post your location, someone can indeed know. My site loyce.club doesn't block Tor users, so by all means, if you visit it, use Tor.
You can bet there are others archiving every post on Bitcointalk too, without publicly sharing the data. There's no point to worry about it: anything you post on the internet should be considered public knowledge forever. If you don't want that, don't post it!

I might be nit-picking here, but I think TryNinja's site is more accessible than LoyceV's site. Obviously loyce.club is essentially an HTML warehouse with historical information, so it is not really a fair comparison.

TryNinja is much better at presenting the data than I am. All I did was create and publish a static archive before anyone else did, and it's been online for 5 years. I gave TryNinja a copy of my archived old posts, so you can search them there.

I would *love* to make a search engine for the forum though. It has always been one of my pet peeves and I think it would be much more useful than what is essentially a decentralized backup of the forum, given that Google and the built-in search are terrible at finding posts in large topics.

You mean like Ninjastic Search? If you want to build another search engine, I'd be more than happy to get you a downloadable archive with 60-ish million posts (in separate files)

[ABCbits]

But even if this was the case, there were already models for on-chain stored "forums" or "social media" that worked. I had discussed that here on the Meta subforum in this thread (https://bitcointalk.org/index.php?topic=5481514.msg64087445#msg64087445). In my case I have had a bit of experiences with Steem(it) (https://steemit.com), and it worked reasonably well. Of course it has centralization problems but these are not directly related to the fact that it's stored on a blockchain.

In the case Bitcointalk is moved to a Steemit-like platform completely, viewing content and also posting could be provided by independent websites which access the blockchain with a full node. The software needed is a combination of a block explorer (to view the content) and an online wallet (to post yourself). Every forum user would need to create at least one account/address/key pair for the blockchain to post, but that can be hidden (so in fact all what you have to do is to "register" on one of the "node websites"). As Steemit showed, there were basically no differences in reaction time and availability to a centralized forum. The node websites could run ads and sponsor the transaction fees for posting, like it occurred on Steemit basically (although there the model was a bit more complex).

the should be trying to move bitcointalk to a steemit-like platform so that continuity is transparent and it has some degree of decentraliztion. i guess though some people go by the "if it aint broke then don't fix it" methodology. but that can end up being challenged if something happens like a site gets hacked and usernames/logins get compromised.

i dont think there's much more to discuss at the moment about this topic since i think another thread also touched on this issue. but centralized forums are not ideal. even something like reddit, they are centralized. they will need to move to blockchain eventually or be overtaken.

Hive Network/Blog[1] would be better example than Steemit due to past hostile takeover[2].

I think @LoyceV and @TryNinja might able to answer these question, since they own copy of this forum.

A post archive isn't the same as a working forum. Let's say Bitcointalk is gone tomorrow. I have backups of most of the posts, but it's the unedited version. Millions of posts that have been deleted are still in my archive. I know who posted it, but I can't link the forum identity to a real person. So even if I (or someone else) would use the post data to create a new forum on a new domain with all usernames copied, it's going to be impossible to give the real account owner access to their own posts. Some may be able to sign a message, but that's going to be a lot of work and will only reach a small part of the users. And since the forum is gone, most users won't even know there's an effort to create a new domain name for the forum.
If Bitcointalk.org disappears, I expect it to just be the end of this forum.

You're right, although i was simply hoping you or @TryNinja would answer size of your backup or database which contain all forum thread and post.

[1] https://hive.blog/
[2] https://cryptoslate.com/big-exchanges-conduct-a-hostile-takeover-of-steem-blockchain-following-tron-acquisition/

[LoyceV]

You're right, although i was simply hoping you or @TryNinja would answer size of your backup or database which contain all forum thread and post.

My archive directory fills 163 GB and has 15,271,678 HTML files. This would be much more efficient to store in a database, I expect TryNinja to have lower numbers.

[NotATether]

Hive Network/Blog[1] would be better example than Steemit due to past hostile takeover[2].

[1] https://hive.blog/
[2] https://cryptoslate.com/big-exchanges-conduct-a-hostile-takeover-of-steem-blockchain-following-tron-acquisition/

That last quote on the news website hit really hard:

“The depositor may have a ‘balance’ at the exchange, but do not have rights on that blockchain. They exchanged that right for a paper guarantee, all for the purpose of speculation.  If they had cared about the stake, they would have demanded that within their paper exchange.”

In reality there are only a few people holding the coin (any coin - even Bitcoin) who actually care about consensus, community, voting, governance, etc.

The rest of them are "number go up" people who do not care better and would rather these exchanges have all that bargaining power in return for the privilege of storing their money on a centralized (yet "convenient") point of failure.

I would go on but I don't want to take the discussion off-topic.