Storing the bitcointalk forum on a decentralized file system

[larry_vw_1955]

What is the feasibility of storing the entire forum on some decentralized file system. Ideally it would be bitcoin itself through something like ordinals. But alot of people probably would object to that as a waste of blockspace so what other ways could it be done less expensively.

1) how much disc space does the bitcointalk forum hold in total?
2) how much does it grow per month?
3) how often would you update the forum threads and what software would you use to do it?

We can't expect the forum to stay online forever but there are some nice threads with a ton of information in them and that should be saved somehow for future bitcoiners. I know that some people will have a seriously bad reaction to suggesting the entire forum be stored on bitcoin itself but doesn't it make sense that that's where it belongs? Bitcointalk is the de facto forum for bitcoin everything.

[CODE200]

I think that the Internet Archive is doing just fine in terms of being used as a way to save important parts or contents on the Internet, there's probably someone out there that's already doing what you're trying to propose and I think that it would be a really awesome thing if there really is someone doing it and not just some threads on the forum but the whole forum itself and when it happens, it will feel like you're exploring the forum itself when you check out on the preserved forum in the future that this forum is really going to go away.

[Text]

I think using Bitcoin itself might not be the most efficient or feasible solution due to potential objections and block space concerns, there are other decentralized file systems or distributed storage solutions could be a promising avenue that could be explored.

What are your thoughts on using platforms like IPFS or Filecoin for this purpose? They offer decentralized storage without the associated block space concerns of directly using Bitcoin.

[ABCbits]

What is the feasibility of storing the entire forum on some decentralized file system. Ideally it would be bitcoin itself through something like ordinals. But alot of people probably would object to that as a waste of blockspace so what other ways could it be done less expensively.

If you use something like BitTorrent or IPFS protocol, it's feasible although limited by who wants to act as "seeder" or "server". IMO Bitcoin blockchain probably should be used to store BitTorrent or IPFS link.

1) how much disc space does the bitcointalk forum hold in total?
2) how much does it grow per month?

I think @LoyceV and @TryNinja might able to answer these question, since they own copy of this forum.

[tranthidung]

I think that the Internet Archive is doing just fine in terms of being used as a way to save important parts or contents on the Internet

You can use Wayback Machine.

I think @LoyceV and @TryNinja might able to answer these question, since they own copy of this forum.

There are times their scrapers fail to scrap forum posts but it does not occur too often.

We can't expect the forum to stay online forever but there are some nice threads with a ton of information in them and that should be saved somehow for future bitcoiners.

You can archive it by yourself or someone else can do it.

There are some websites which archive historic threads, posts of Bitcoin forum, for instance.

• Satoshi Nakamoto Institute archives forum posts of Satoshi Nakamoto.

[Alone055]

I think that the Internet Archive is doing just fine in terms of being used as a way to save important parts or contents on the Internet

That's not a decentralized system but just a free digital library. We are talking about the distant future here, what makes you think that the forum wouldn't be online but that library would be? The solution shouldn't involve a website that requires its domain or hosting to be paid and updated every year or so because in that case, it could go offline sometime in the future just like how this forum can as OP said.

So, the storage should be decentralized and it should be working without needing any maintenance or something.

[tranthidung]

The solution shouldn't involve a website that requires its domain or hosting to be paid and updated every year or so because in that case, it could go offline sometime in the future just like how this forum can as OP said.

This forum does not update the domain and hosting every year. The last update is in 2019 and the expiration date is 2029 and theymos bought it for 10 years.

https://who.is/whois/bitcointalk.org

Important Dates

Expires On2029-06-24
Registered On2011-06-24
Updated On2019-11-24

Decentralized or centralized forum, the domain must be bought by someone publicly or anonymously. You can buy it for 10 years, 50 years or 100 years if you have money for payment.

[Alone055]

This forum does not update the domain and hosting every year. The last update is in 2019 and the expiration date is 2029 and theymos bought it for 10 years.

I know, I was actually referring to the Internet Archive, the library, because for normal websites or services, their operators or creators buy yearly plans usually because they wouldn't want to pay money in advance since they might have to stop the website or service before that period.

The fact that theymos has paid for the hosting for 10 years in advance makes this forum more unique, and I knew this already.

[larry_vw_1955]

What are your thoughts on using platforms like IPFS or Filecoin for this purpose? They offer decentralized storage without the associated block space concerns of directly using Bitcoin.

using filecoin is not free though. and it only offers temporary storage. if you were using it directly, from what i understood is it's only free to retrieve data for the original uploader. everyone else has to pay for retrieval.

there are services built on top of filecoin but they cost a subscription fee aka similar to web hosting so you stop paying your site goes away. using ipfs directly would have the same issues. no one is going to "pin" all your terabytes of data for free.

so i don't think these 2 technologies would be affective.

The fact that theymos has paid for the hosting for 10 years in advance makes this forum more unique, and I knew this already.

paying for hosting and domain name registration for 10 years are two different things. One of them you can verify. One of them is not public record so I'm not sure how you think you know that.   hosting means paying for the storage space not just the domain name.

[mk4]

How reliable is the Internet Archive really with us trusting it to store all this valuable information? There are times that certain pages simply fail to load in my experience.

I really don't know how much all the Bitcointalk data will cost when using all these decentralized file storage systems so no opinion on that; and I don't know the risks/downsides of IPFS either besides the fact that it's far more tried and tested.

[joker_josue]

We can't expect the forum to stay online forever but there are some nice threads with a ton of information in them and that should be saved somehow for future bitcoiners. I know that some people will have a seriously bad reaction to suggesting the entire forum be stored on bitcoin itself but doesn't it make sense that that's where it belongs? Bitcointalk is the de facto forum for bitcoin everything.

But what's the idea? Is there a backup/copy of the forum, in case it stops working? This is already being done by some users.

Or is it putting the website up and running in a decentralized system?
This would, based on current technology, be very complicated. This required each user to store hundreds of GB of information to visit the site and consult. And even if there was the possibility of visiting the website without downloading files, the speed would be low. I'm not saying low speed when browsing, but rather when updating.

In the end, someone needed to pay for an online server, to serve as a connection point to all parts of the site, spread among the various peers.

I understand the concern, but the site administration has several plans to ensure continuity in the project. Of course we don't know the future, and a lot can happen. But, for now, as long as another operating model is not possible, this model is assured.

[tranthidung]

I understand the concern, but the site administration has several plans to ensure continuity in the project. Of course we don't know the future, and a lot can happen. But, for now, as long as another operating model is not possible, this model is assured.

Two admins and three global moderators who can be promoted to admins if necessary, in human & moderation resources, the forum has enough to do its continuation.

For other risk factors like forum seize by government, by banning mixers proactively, I see theymos and even Cyrus already saw that risk and eliminate it by that 'big decision' which surely does not satisfy all forum members but for forum future, eliminate that big risk is good.

For other risk factors which are insignificant or non-existing now, we will know more in future and more things will be done in forum administration and moderation to adapt to future changes and challenges.

[ABCbits]

using ipfs directly would have the same issues. no one is going to "pin" all your terabytes of data for free.

I hear AI trend use tons of data. So i think we can expect few volunteer to "pin" this forum thread/post for some time.

How reliable is the Internet Archive really with us trusting it to store all this valuable information? There are times that certain pages simply fail to load in my experience.

At very least, they can spare their storage to store this forum (since it's mostly about text) when they store at least 10PB of data[1].

[1] https://blog.archive.org/2012/10/26/10000000000000000-bytes-archived/

[Sandra_hakeem]

At very least, they can spare their storage to store this forum (since it's mostly about text) when they store at least 10PB of data[3].

It's though but, I thought about asking; how about the illustrative diagram/pictures and memes on important threads? I mean.... I know these pictures have already reduced quality - maybe about 100P of vertical resolution which consumes barely 250kb from the 10PB presumably...
* Is that going to be part of the storage problems we'll ever have in the future if we ever adopt a decentralized storage pattern?
* Since the transmission and regular update process will be way too slow, according to this quote, how do we regulate that to not go against our caches?

[...]the speed would be low. I'm not saying low speed when browsing, but rather when updating.

I see theymos and even Cyrus already saw that risk and eliminate it by that 'big decision' which surely does not satisfy all forum members but for forum future, eliminate that big risk is good.

That has only made me realize how much work it takes altogether to be at the top ^{[it's lonely at the top]}

[larry_vw_1955]

How reliable is the Internet Archive really with us trusting it to store all this valuable information?

i don't think it's reliable at all. Not unless you are using their paid services. For example:

https://web.archive.org/web/20240000000000*/bitcointalk.org

for this forum, they can go DAYS without even doing a single snapshot. and then in a single day they might do 2 or 3 snapshots. doesn't make any sense.
what you'll notice if you take a look at the snapshots of this forum is when you start clicking on individual threads, many of them don't even seem to be archived so it's worthless.

There are times that certain pages simply fail to load in my experience.

you are right. i think the internet archive is almost worthless. but they do have a service called Archive It https://archive-it.org/

Archive-It Basic Pricing
• $500 yearly for 128 GB data archived
• $800 yearly for 256 GB data archived

you pay alot to store a little. seems to be the name of that game...

I really don't know how much all the Bitcointalk data will cost when using all these decentralized file storage systems so no opinion on that

it would cost more than any centralized web hosting service most likely.  

; and I don't know the risks/downsides of IPFS either besides the fact that it's far more tried and tested.

using ipfs for a website as dynamic as a forum seems like it could be problematic. ipfs is really best for storing files that don't change. if we think of trying to store each forum thread in it's own file, that file would need to be "updated" possibly hundreds of times. not ideal.

But what's the idea? Is there a backup/copy of the forum, in case it stops working? This is already being done by some users.

which users exactly? and how dedicated are they to doing it? what happens if they stop doing it? i heard that loyce is doing it. but i think that's a pretty big burden to put on one person's shoulders. apparently loyce archives the blockchain too. he must be really busy. what happens if he gets burned out? what happens if all of them get burned out?

Or is it putting the website up and running in a decentralized system?

that would be the end game.

that way things could have real world consequences. merit could actually be not just some imaginary thing but actual bitcoin. and things like AI spam could be stopped by requiring new users to make a small deposit of bitcoin and if they started abusing the forum then their deposit could be siezed for different infractions.

it's not about not trusting the current administration but recognizing they are only human and humans can only last so long. if you want something to last you have to make it decentralized, just like bitcoin...

[joker_josue]

But what's the idea? Is there a backup/copy of the forum, in case it stops working? This is already being done by some users.

which users exactly? and how dedicated are they to doing it? what happens if they stop doing it? i heard that loyce is doing it. but i think that's a pretty big burden to put on one person's shoulders. apparently loyce archives the blockchain too. he must be really busy. what happens if he gets burned out? what happens if all of them get burned out?

Yes, he is the forum administrator. And he has already reported several times that there are several alternative plans, if you find something with him.

Therefore, let's believe that everything is as planned, so that the forum always remains active.

[ABCbits]

At very least, they can spare their storage to store this forum (since it's mostly about text) when they store at least 10PB of data[3].

It's though but, I thought about asking; how about the illustrative diagram/pictures and memes on important threads? I mean.... I know these pictures have already reduced quality - maybe about 100P of vertical resolution which consumes barely 250kb from the 10PB presumably...

Even with images, i'm sure it won't be problem for Internet Archive in terms of storage size.

* Is that going to be part of the storage problems we'll ever have in the future if we ever adopt a decentralized storage pattern?

Aside from what @larry_vw_1955 said, i expect many link to image no longer valid or can't be obtained without blocked by the image provider for requesting too many images.

* Since the transmission and regular update process will be way too slow, according to this quote, how do we regulate that to not go against our caches?

I simply don't know.

[panganib999]

I think that the Internet Archive is doing just fine in terms of being used as a way to save important parts or contents on the Internet, there's probably someone out there that's already doing what you're trying to propose and I think that it would be a really awesome thing if there really is someone doing it and not just some threads on the forum but the whole forum itself and when it happens, it will feel like you're exploring the forum itself when you check out on the preserved forum in the future that this forum is really going to go away.

You're probably talking about the Wayback Machine, accessible in the Internet Archive. If that's so, you'll have to index pages of the internet first on the Internet Archive before you could even access them at a later time, and I do not need to tell you how little the amount of people in here who would go so far as to index pages of the forum just so they can check it at a later time, especially with third-party notifications being a thing in this forum already.

So yeah, the idea's good don't get me wrong, it's just boring and cumbersome, plus the Wayback Machine's facing some of their own shit too as of late, just watch SomeOrdinaryGamers' vid about the Internet Archive facing shut down and shit.

[dkbit98]

What is the feasibility of storing the entire forum on some decentralized file system. Ideally it would be bitcoin itself through something like ordinals.

Using bitcointalk forum with ordinals is a terrible idea and I would never do that, especially not on main chain.
One potential option would be to somehow use bitcointalk on IPFS, but that would make it slower and and I don't know if everything would work correctly.

[SamReomo]

I would say that having backups of the forum as torrents is going to be a easier solution and a long-term one. There are some torrent sites that always release the backup of their torrents library and such thing can be done with this forum as well. But doing something like that is going to be costly if it isn't supported by the community who may seed those torrents for the peers, and leechers.

There are some cloud based seeding sites like Seedbox but those can cost a lot of money on monthly basis. Someone with his own dedicated internet and computer can seed those torrents and if that someone is joined by a team of 10 then it's possible to backup the forum on monthly basis and have it's backup copy available to everyone for long term.