Bitcoin Forum

As we know, block files are saved in smaller files that contain a number of blocks within these blk00000.dat files, the question is, is there a theoretical maximun amount that could be achieved where it stops working? even if it's in practice meaningless such as collision of private keys (2^256 possible so chances are it's insanely unlikely you spawn the same one) so basically, it exists in theory, but not in practice, my question is, can this program just validate an infinite amount of blk files, or there would be a theoretical limit? Assuming hard drive space keeps up with the growth of the blockchain that is. Then as the blockchain grows, the size of hard drives would grow faster, so you could just keep storing these files. My question is more on the software side of things, if it would have problems with so many files, or perhaps even at the hardware level, but we already deal with insane amount of files. An interesting thing to consider would be, maximun amount of files a folder can deal with within the OS. For Windows 10 it's 4,294,967,295 which was surprising, im not sure how many your average Linux distro could deal with, even tho I think there are tricks to split the blockchain in different folders that link with each other I think, or all files must be store within the same folder?
Let's just try to consider all angles and think far into the future and try to detect any walls that could be hit along the way and how to fix them before it even happens.

We really dont need to worry about this.  The way blockchain is designed, it can deal with a crapload of data.  Sure, it can be interesting to theorize about possible limits, but lets be real – we are generations off from hitting any sort of storage or software limitations.  By the time we even approach such a scenario, technology will have advanced far beyond our current capabilities.

That is a problem independent of Bitcoin and it would depend on the filesystem as well as the OS and that is out of our control. For most intent and purposes, we will not run in the problem of individual file size since the blocks are being split into multiple files already. If the problem is with the size of the volume, then I believe that most of the filesystems that we're using already have theocratical compatibility in the order of exabytes. The most naive solution that I can think of is just to symlink the relevant block files into different volumes.

If the individual hard drives were to get bigger, I doubt that OSes wouldn't be supporting different filesystems with larger volume capacities.

Yes. Each and every SHA-256 call will limit the hashed message into 2^64-1 bits at most.

Years ago (DOS / 3.1) that number was smaller it grew with Win 98 and then again in XP and then again. So it's meaningless to worry about. By the time BTC hits that number the OSs we will be running are going to be able to handle much larger amounts of files and file sizes.

Kind of like storage. 4 TB SSDs used to be over $1000 now you can get cheap name brands for under $200 and even the 8TB are under $800

It's not even a concern unless you are going to want to run a copy of bitcoin core that will be released in 2095 on an OS from 2001.

-Dave

I think the theoretical limits for blockchain size do  exists and it is defined by the Bekenstein bound   (https://en.wikipedia.org/wiki/Bekenstein_bound) of Sgr A* (which is black hole in the centre of our galaxy). The  bigger size would be not accommodated by any imaginable device invented (at any time, in the past, present or future) by any civilization   presumably existing in Milky Way.

Just as an FYI
https://x.com/davepl1968/status/1772042158046146792

And

https://www.theverge.com/2024/8/16/24221635/microsoft-fat32-partition-size-limit-windows-11

So even old things might be able to store even more if MSFT makes any other changes as time goes on.

Probably will not matter for BTC since nobody should be using outdated file systems to host their node but still an interesting tidbit.

-Dave

Imagine if file system updates were seen as being bad the same way as Bitcoiners see protocol updates being bad.
We would still be limited to 4GB max file size with FAT32.

Nobody should be using outdated cryptocurrency or for that matter, anything that's outdated.

So does anyone know or has tested if symlinks on linux work? so if you were to run out of max files in a single folder, it would continue from another folder. But Bitcoin Core would need to know where to pick the files from, or that is what symlink would do? it would be as if it was the same folder even tho it's 2 separate folders? Perhaps Bitcoin Core itself should have the option to add several folders with a maximun amount in number of files or MBs. So you would set

Folder 1: C/Bitcoin/blockchain1 - max size 100GB
Folder 2: C:/Bitcoin/blockchain 2 - max size 100GB
etc

This way even older system could keep up. Important since all cpus after 2008 or so are compromised.

Well, if you cannot trust your system, then you cannot trust anything else. Going out of x86 is one option, installing libreboot is another one. Fortunately, some people successfully disabled Intel ME, even on more modern computers: https://puri.sm/posts/deep-dive-into-intel-me-disablement/

You should be probably more worried about using Windows, than about using Intel ME, because it is easier to compromise the former, than the latter. For example, because of IPv6 WiFi vulnerability, we even had a forum news bar, encouraging users to upgrade their Windows.

Bitcoin clients can technically store information related to the blockchain in any format it so chooses. From a technical perspective, a bitcoin client will store information related to the blockchain in some kind of database. The important part of the blockchain is the unspent outputs, and there is a limit to how large this can get due to the limited number of bitcoin and the smallest allowable fraction of a bitcoin (satoshi). Even if the blockchain were to become too large to fit on a single hard drive, it could potentially be stored on multiple hard drives, and the majority of the data would never need to be accessed after it is validated one time.

There is also a technical limit as to how large the blockchain can be as a function of the block number, as each block has a maximum size. The blockchain will grow at a much slower rate due to blocksize limitations than there would be potential issues with hard drive limitations.

No, because you can create and spend zero satoshis. Then, amount is sum-neutral, but signatures and scripts are still verified.

The number of blocks is unlimited. And if they are deterministic, then you can create a very large chain, which will take a very little space. And because the block number is expressed as a "big number", then it is a much larger limit, than the maximum size of the message, which can go through SHA-256.

Obviously the larger the files there are the less efficient a system will operate compared to if it’s only managing small files. It won’t stop working altogether but its effect could be undeniable. No, there’s no actual number of how many blk files are allowed to be created before it stops working so it will just keep adding and adding throughout. I don’t think this is something we have to worry about though since bitcoin was exactly designed to handle large files.

If it helps answer your curiosity, there’s a thing called Pruning. Pruning is basically the removal of old data from bitcoin’s core that have been validated already or are not necessary to validate new ones anymore.