Size of BTC blockchain centuries from now...

[Yurock]

If you have the RAM and hardware, maybe put your datadir inside a ramdrive. At least until it was updated.

This is the only remedy I see as of now. And I'm afraid that transaction history will possibly grow in size faster than average RAM. So, even this method may not be an option in the near future.

[1714930593]

1714930593

[1714930593]

1714930593

[1714930593]

1714930593

[1714930593]

1714930593

[1714930593]

1714930593

[Desolator]

In the next release, they will most likely make the switch to levelDB, which will be significantly faster in disk I/O.

How would that possibly work?  Everyone would download a new version of the blockchain database...from each other? When nobody has it?  Lol, so the original releaser of the fully converted database would be able to put whatever the hell they wanted in it So it'd have to be a very lengthy and difficult conversion process or a flat switch over supporting two databases, which is also not ideal.

[kjj]

In the next release, they will most likely make the switch to levelDB, which will be significantly faster in disk I/O.

How would that possibly work?  Everyone would download a new version of the blockchain database...from each other? When nobody has it?  Lol, so the original releaser of the fully converted database would be able to put whatever the hell they wanted in it So it'd have to be a very lengthy and difficult conversion process or a flat switch over supporting two databases, which is also not ideal.

Erm.  I'm not sure where to start.

The blocks aren't in the database, they are just stuffed into files, and those files will not be changing at all.  What goes in the database is just index information so that blocks can be found quickly when needed.

The network passes blocks around, not files.  There is no need to distribute "new" database files.  New clients that don't have the blocks loaded will continue to load them as usual, by asking the network for them.  I haven't been following the work on leveldb, but I see no reason why the client couldn't include both sets of database libraries so that people upgrading can keep reading their old indexes while they generate the new ones.

And, I promise to leave at least one of my personal nodes running an old version for quite a while, just in case.

[jgarzik]

In the next release, they will most likely make the switch to levelDB, which will be significantly faster in disk I/O.

How would that possibly work?  Everyone would download a new version of the blockchain database...from each other? When nobody has it?  Lol, so the original releaser of the fully converted database would be able to put whatever the hell they wanted in it So it'd have to be a very lengthy and difficult conversion process or a flat switch over supporting two databases, which is also not ideal.

The block format used on the network has not changed.  It is underlying database index on-disk that is changing.

It is trivial to rebuild the database index:  just delete blk*.dat, and download fresh from the network.

The blockchain data itself has not changed, and continues to be self-verifying (integrity protected by hash).

[runeks]

My projections from looking at the following graph tells me we will have 10²⁴ GB hard drives in the year 2120.



Ie. if I just project the line out to the year 2120, it tells me that - at that point in time - hard drives will have a capacity of 1,000 billion billion terabytes.

So to put that in perspective, a 100 byte file on a 10 TB hard drive today will, in the year 2120, have the same relative size as a 10 TB file on a 1,000 billion billion TB hard drive.

So the 100 bytes that what we now think of as a tiny amount of data (about three times the size of an ECC key) will be equivalent to 10 TB in the year 2120.

If the projections hold. Personally, I don't think they will hold. I think widely available storage media will exceed these sizes going over 100 years into the future.

[Desolator]

I'm confused!  "Networks" cannot store data, they're networks.  So someone somewhere on some storage media has the actual block data, right?  I assumed it was everyone and that the most logical storage method would be a database and not a flat file.  I get the indexing part, whether it's were to be in a database with the blocks as a separate table or not, since it allows you to find a block super quickly.  But, if everyone deleted blk*.dat then who on the network are they going to download the new version from?  Nobody would have it, lol.  Unless it's generated by reindexing their local copy of the block chain that you're saying they don't have.  3GB is kinda big for an index file though, so aren't all the blk.dat files the chain data itself?

[luffy]

all the problems and issues are solved as they arrived
Just think why BTC hasn't been introduced 20 years earlier!
I have faith in internet and its evolution

[runeks]

3GB is kinda big for an index file though, so aren't all the blk.dat files the chain data itself?

blk<num>.dat (blk0002.dat, blk0001.dat, etc. in the future) is the actual block chain data. blkindex.dat is the index file.

[jgarzik]

I'm confused!  "Networks" cannot store data, they're networks.  So someone somewhere on some storage media has the actual block data, right?  I assumed it was everyone and that the most logical storage method would be a database and not a flat file.  I get the indexing part, whether it's were to be in a database with the blocks as a separate table or not, since it allows you to find a block super quickly.  But, if everyone deleted blk*.dat then who on the network are they going to download the new version from?  Nobody would have it, lol.  Unless it's generated by reindexing their local copy of the block chain that you're saying they don't have.

The entire network does not upgrade at the exact same moment.

There will be many not-yet-upgraded nodes that will serve blockchain data to those upgrading to a new database index.

3GB is kinda big for an index file though, so aren't all the blk.dat files the chain data itself?

3GB chain data
1GB index

is what I have here.

[Michael_S]

Right now, there is a hard-coded limit of 1 megabyte per block.

If we assume that limit never changes, that gives:
 1MB per block * 6 blocks per hour * 24*365.25*200 = 10,519,200 MB

... or 10.5 terabytes for the maximum size of the entire blockchain over the next 200 years (somebody check my math, I'm really good at dropping zeroes).

I expect that in 200 years 10 terabytes of storage will cost a few pennies.

Now whether or not that 1 megabyte per block limit should go away is hotly debated, and will be debated more and more as transaction volume increases.

By principle I don't understand how this limit can be possibly kept under any circumstances of btc traffic growth. If one block is generated only every 10 min and shall contain *all* the transactions that have occurred during a 10 min period, and if the number of transactions per minute increases dramatically in the future, then the block size has a lower limit that is determined by this number of transactions per 10 minutes, because the blockchain contains all the information of all transactions, which requires non-zero memory space.

Example: Let's assume that in e.g. 20 years from now there are 1 billion btc transactions being carried out every 10 minutes, which is not necessarily completly unrealistic, assuming >20 billion people on earth by that time, plus companies and computers carrying out automized transactions for certain financial purposes all day long.
Let's further assume that one transaction that is recorded in the blockchain occupies at least a few bytes of storage to contain all the transaction information. In lack of detailed knowledge of the bitcoin protocol, I am making the conservative assumption here that 50 bytes of storage are required per transaction (containing input address(es), output address(es), amount(s) etc.).

With these assumptions, the blockchain would grow by 50 GB every 10 minutes, or 7.2 TB per day, or 2630 TB per year.

2630 TB per year looks a lot, but it is certainly something well feasible by certain nodes already nowadays.

However, this number is still conservative! Theroretically, this number can be much higher, in fact infinite, whereas the storage capability of any technology cannot be infinite on a finite planet, given the fact that the number of atoms on earth are finite (ca. 10^50) and assuming that we will never be able to store more bits of information on earth than the number of atoms on earth. Hence, if the transaction volumne really gets too high, one has by principle think of another "format" to safe the bitcoin network state: Instead of storing the limit-less transaction history, one may instead have to store the btc adress associated to each satoshi, which leads to a finite rather than infinite worst-case memory requirement:
In the worst case, every satoshi resides on a different btc address. There are 21e6 x 1e8 = 2.1e15 = 2.1 million billion satoshis in existence. If one bitcoin address has 20 bytes of size (again this is a guess in lack of better knowledge, but should be not too wrong), then such a data base that captures the complete state of the whole bitcoin network at a given point in time would be 2.1e15 x 20 bytes in size, i.e. = 42e15 bytes = 42000 TBytes.

My conclusion: With a storage of ca. 42000 TB one could store the complete bitcoin network state (not transaction history) any time in the future! This storage is well within the reach of technological feasability, even today (you could buy this storage today for less than 4 million USD). Hence, there is no theoretical physical limit that prevents future expansion of the bitcoin system. In the worst case however one may have to transition the system in some far future from a "full transaction history" record to a "current network state" format, or a mixed form like "current network state at time t_0 plus differential trade history after t_0", where different nodes may use different times t_0 depending on their individual storage capabilities.

[misterbigg]

Right now, there is a hard-coded limit of 1 megabyte per block.

Oh my GOD are you kidding me? There's a hard coded limit on the number of transactions? We must fix this immediately *snicker*

    

[solex]

Didn't you see the date on this thread?

[Yurock]

Block size limit is artificial and will be adjusted as needed in the future. However, raising limits opens the network to flood. Increased future transaction fees are supposed keep transaction volume within manageable range. Currently this is my only hope.

[Michael_S]

So does this means that due to today's 1 MB limit per 10 min (i.e. per block), if the number of transactions get so large that more than 1 MB of transaction data is generated every 10 min, then some transactions will never ever make it into the blockchain, if their transaction fees are too low?
Then this means that the queue of transactions that are waiting to get into the blockchain will grow larger and larger all the time, because the rate at which new transactions are created (> 1 MB/10 min) would be higher than the rate at which transactions could be processed (== at maximum 1 MB/10min).

If this is correct, then there would be the following consequences for the future of the bitcoin system, as transaction volume increases:
(1) Only transactions with sufficiently high transaction fees will make it into the blockchain at all.
(2) The Bitcoin clients should get a feature to revert transactions that are not yet in the blockchain, because if I transfer 1 BTC to address xxx with a low fee yyy, and after some days or weeks my transaction still does not show up in the blockchain, then I might want to revert my transaction and try again with a higher fee zzz > yyy.
(3) As a result of this mechanism, there will be a real "fight" to get one's transaction into the blockchain, so in practice this means that transactions will no longer cost "almost nothing", but to the contrary, transaction fees would soon become so high that for example micro-payments and small donations become prohibitive due to the high fees, and it would soon be cheaper again to use PayPal!
(4) As a result of (3), people will move to make bitcoin payments more and more inside "closed sub-systems", i.e. bitcoin banks that do their own bookkeeping on their customers' accounts without loading the blockchain. For example there could be a MtGox subsystem or a blockchain.info subsystem, to name just two. When I transfer BTCs from my MtGox account to another person's MtGox account, I would not use the blockchain, but the MtGox internal bookkeeping would carry out the transaction, just like in today's banking system. This would avoid transaction fees, so people will favor this variant. But this also means that many benefits of bitcoin as we know it today would vanish, and the whole bitcoin system would move towards a de-facto centralized system that is again built on trust into a cartel of a few "subsystems" (bitcoin banks), and these sub-systems may even run fractional reserve banking without their customers knowing about this.
So we would again end up with two kinds of money like in today's fiat money system: First the true bitcoin money (corresponding to today's fiat central bank money), and secondly the account balances of the "subsystems (=bitcoin banks)", that may finally run fractional reserve, like today's bank accounts.


Also, these sub-systems (bitcoin banks) would be vulnerable to government interference etc. So with such an evolution we will reach the point when the bitcoin experiment has finally missed its original goals and has failed. I think such an evolution should be avoided!

If my understanding is essentially correct, then I think the only solution to keep the bitcoin system open and avoid the outlined evolution towards closed sub-systems (="de-facto-bitcoin-banks") would really be to change the way that transactions are stored, as outlined in my last post, such that memory needs for saving the blockchain (be it the complete history or only the current state plus a short incremental history) does not increase infinitely.

Maybe my thoughts are relating to what may happen rather far in the future, but this future might also turn out to be nearer than what we think.

...Or am I missing some fundamental point here and all my worries are pointless?

[justusranvier]

So does this means that due to today's 1 MB limit per 10 min (i.e. per block), if the number of transactions get so large that more than 1 MB of transaction data is generated every 10 min, then some transactions will never ever make it into the blockchain, if their transaction fees are too low?
Then this means that the queue of transactions that are waiting to get into the blockchain will grow larger and larger all the time, because the rate at which new transactions are created (> 1 MB/10 min) would be higher than the rate at which transactions could be processed (== at maximum 1 MB/10min).

If this is correct, then there would be the following consequences for the future of the bitcoin system, as transaction volume increases:
(1) Only transactions with sufficiently high transaction fees will make it into the blockchain at all.
(2) The Bitcoin clients should get a feature to revert transactions that are not yet in the blockchain, because if I transfer 1 BTC to address xxx with a low fee yyy, and after some days or weeks my transaction still does not show up in the blockchain, then I might want to revert my transaction and try again with a higher fee zzz > yyy.
(3) As a result of this mechanism, there will be a real "fight" to get one's transaction into the blockchain, so in practice this means that transactions will no longer cost "almost nothing", but to the contrary, transaction fees would soon become so high that for example micro-payments and small donations become prohibitive due to the high fees, and it would soon be cheaper again to use PayPal!
(4) As a result of (3), people will move to make bitcoin payments more and more inside "closed sub-systems", i.e. bitcoin banks that do their own bookkeeping on their customers' accounts without loading the blockchain. For example there could be a MtGox subsystem or a blockchain.info subsystem, to name just two. When I transfer BTCs from my MtGox account to another person's MtGox account, I would not use the blockchain, but the MtGox internal bookkeeping would carry out the transaction, just like in today's banking system. This would avoid transaction fees, so people will favor this variant. But this also means that many benefits of bitcoin as we know it today would vanish, and the whole bitcoin system would move towards a de-facto centralized system that is again built on trust into a cartel of a few "subsystems" (bitcoin banks), and these sub-systems may even run fractional reserve banking without their customers knowing about this.
So we would again end up with two kinds of money like in today's fiat money system: First the true bitcoin money (corresponding to today's fiat central bank money), and secondly the account balances of the "subsystems (=bitcoin banks)", that may finally run fractional reserve, like today's bank accounts.


Also, these sub-systems (bitcoin banks) would be vulnerable to government interference etc. So with such an evolution we will reach the point when the bitcoin experiment has finally missed its original goals and has failed. I think such an evolution should be avoided!

If my understanding is essentially correct, then I think the only solution to keep the bitcoin system open and avoid the outlined evolution towards closed sub-systems (="de-facto-bitcoin-banks") would really be to change the way that transactions are stored, as outlined in my last post, such that memory needs for saving the blockchain (be it the complete history or only the current state plus a short incremental history) does not increase infinitely.

Maybe my thoughts are relating to what may happen rather far in the future, but this future might also turn out to be nearer than what we think.

...Or am I missing some fundamental point here and all my worries are pointless?

I think this is very close to what will happen if the 1 MB limit is not increased before the demand for transactions catches up to the limit.

With regards to blockchain size, you might want to take a look at this: https://bitcointalk.org/index.php?topic=88208.0;all

[deepceleron]

In 1985 the Cray-2 supercomputer was released, and was the fastest computer in the world for five years. It cost about $16 million in 1985 dollars. The memory was 256M 64k words, more than all previously produced Cray-1s combined - 2GB.  It had 1.9GFLOPS of processing power in it's fastest configuration - approximately equal to an iPad 2, and significantly less than the computer I'm typing on.

I think that technology might keep up with Bitcoin.

[Yurock]

So does this means that due to today's 1 MB limit per 10 min (i.e. per block), if the number of transactions get so large that more than 1 MB of transaction data is generated every 10 min, then some transactions will never ever make it into the blockchain, if their transaction fees are too low?

No. We will not hit today's limit today. As for the near future, I already stated:

Block size limit is artificial and will be adjusted as needed in the future.

(2) The Bitcoin clients should get a feature to revert transactions that are not yet in the blockchain, because if I transfer 1 BTC to address xxx with a low fee yyy, and after some days or weeks my transaction still does not show up in the blockchain, then I might want to revert my transaction and try again with a higher fee zzz > yyy.

Yes. And also a way to add fees to existing transactions. AFAIK, it is already discussed by Bitcoin software developers, so we may see it implemented some time soon. Recipients of bitcoins will also be able to add to the fee.

(3) As a result of this mechanism, there will be a real "fight" to get one's transaction into the blockchain

I would use the word "compete". Competition is an important market force, and Bitcoin is designed to be governed by market forces.

so in practice this means that transactions will no longer cost "almost nothing"

Yes. Transactions cost something to the whole network. Every new transaction has to be propagated through the network and processed and stored (at least for some time) by every full node. Today, a large part of this cost is paid by newly generated coins, I think (but may be wrong). Let those who makes transactions (and thus supposedly needs them) pay most of this cost. Seems fair to me.

transaction fees would soon become so high that for example micro-payments and small donations become prohibitive due to the high fees, and it would soon be cheaper again to use PayPal!

Bitcoin is clearly not going to be generally suitable for payments of few cents. Other than that, I think, fees will not be prohibiting. From charts on blockchain.info, I estimate the average today's fee to be below equivalent of 4 U.S. cents. Multiply it by 5, and it will still seem quite acceptable to me.

(4) As a result of (3), people will move to make bitcoin payments more and more inside "closed sub-systems", i.e. bitcoin banks that do their own bookkeeping on their customers' accounts without loading the blockchain. For example there could be a MtGox subsystem or a blockchain.info subsystem, to name just two. When I transfer BTCs from my MtGox account to another person's MtGox account, I would not use the blockchain, but the MtGox internal bookkeeping would carry out the transaction, just like in today's banking system. This would avoid transaction fees, so people will favor this variant. But this also means that many benefits of bitcoin as we know it today would vanish, and the whole bitcoin system would move towards a de-facto centralized system that is again built on trust into a cartel of a few "subsystems" (bitcoin banks), and these sub-systems may even run fractional reserve banking without their customers knowing about this.

Remember, here you are talking only about micro-payments. Most people who need to transfer an amount like 10 BTC and up will not care to pay 0.01 BTC transaction fee. And fees are roughly per transaction, not per amount transferred. So, in the future, most micro-transactions are expected to be handled by other systems based on Bitcoin as a unit of value. As for larger transactions, Bitcoin is quite capable to handle them. So, only small (I think) part is going to move to systems that handle bitcoins indirectly. And it does not mean centralization. Cryptocurrencies of different kinds are being developed and launched, like Ripple and Open Transactions. They have no problem using Bitcoin as a unit of value. They may be more suitable for instant payments and micro-payments.

Also, these sub-systems (bitcoin banks) would be vulnerable to government interference etc.

Like Bitcoin to fiat exchanges, they may be points of failure. But the possible influence is limited by at least two factors:

• Only a part of the whole Bitcoin economy will see an influence. Larger amounts of bitcoins that reside in normal Bitcoin wallets will not be affected.
• There will be no monopoly. If, for example, you close one large Bitcoin payment processor, others will catch up.

solution to keep the bitcoin system open and avoid the outlined evolution towards closed sub-systems (="de-facto-bitcoin-banks") would really be to change the way that transactions are stored, as outlined in my last post

Your proposal has weak points, but the general idea is worthy, I think. The idea that since total Bitcoin supply is finite, amount of storage required to track every unit may also be finite and within capabilities of future systems.

...Or am I missing some fundamental point here and all my worries are pointless?

There are surely things to worry about, and different solutions to the problems have to be considered.

[misterbigg]

So does this means that due to today's 1 MB limit per 10 min (i.e. per block), if the number of transactions get so large that more than 1 MB of transaction data is generated every 10 min, then some transactions will never ever make it into the blockchain, if their transaction fees are too low?

You're thinking along the right lines. My prediction:

we should see what happens as we run into the soft blocksize limits...what do you predict will happen?

In this order:

1. Most blocks are at or near the 250 kilobyte soft limit.
2. The memory pool of transactions grows due to insufficient space in blocks.
3. Users notice trend of transactions taking longer to confirm, or not confirming at all.
4. Fees increase as users pay more to improve confirmation times.
5. Miners (or mining pools) modify code to select transactions with the highest fees per kilobyte to fit into blocks. They remote the 250 kilobyte soft limit. Some miners disallow free transactions entirely.
6. Transactions clear much more quickly now, but fees decrease.
7. Blocks increase in size until they are at or near the one megabyte hard limit.
8. Fees start increasing. Free transactions rarely confirm at all now.
9. Small transactions are eliminated since they are not economically feasible. SatoshiDice increases betting minimums along with fees. The volume of SatoshiDice transactions decrease.
10. Users at the margins of transaction profitability with respect to fees are pushed off the network.
11. Many people, most non-technical, clamor for the block size limit to be lifted.
12. Fees reach an equilibrium where they remain stable.
13. Spurred by the profitability of Bitcoin transactions, alternate chains appear to capture the users that Bitcoin lost.
14. Pleased with their profitability, miners refuse to accept any hard fork to block size.

[zebedee]

So does this means that due to today's 1 MB limit per 10 min (i.e. per block), if the number of transactions get so large that more than 1 MB of transaction data is generated every 10 min, then some transactions will never ever make it into the blockchain, if their transaction fees are too low?

You're thinking along the right lines. My prediction:

we should see what happens as we run into the soft blocksize limits...what do you predict will happen?

In this order:

1. Most blocks are at or near the 250 kilobyte soft limit.
2. The memory pool of transactions grows due to insufficient space in blocks.
3. Users notice trend of transactions taking longer to confirm, or not confirming at all.
4. Fees increase as users pay more to improve confirmation times.
5. Miners (or mining pools) modify code to select transactions with the highest fees per kilobyte to fit into blocks. They remote the 250 kilobyte soft limit. Some miners disallow free transactions entirely.
6. Transactions clear much more quickly now, but fees decrease.
7. Blocks increase in size until they are at or near the one megabyte hard limit.
8. Fees start increasing. Free transactions rarely confirm at all now.
9. Small transactions are eliminated since they are not economically feasible. SatoshiDice increases betting minimums along with fees. The volume of SatoshiDice transactions decrease.
10. Users at the margins of transaction profitability with respect to fees are pushed off the network.
11. Many people, most non-technical, clamor for the block size limit to be lifted.
12. Fees reach an equilibrium where they remain stable.
13. Spurred by the profitability of Bitcoin transactions, alternate chains appear to capture the users that Bitcoin lost.
14. Pleased with their profitability, miners refuse to accept any hard fork to block size.

I believe your guess is reasonable until point 10.  After that, you're failing to understand the situation isn't stable at all, given all the competing interests, hence your 14th point in particular cannot happen.  The situation will resolve itself in the interests of the majority by the limit being increased.  The miners do not have the final say.

[Michael_S]

[...]

Yurock, after coming across this thread again after 2 months, I think I agree with all your points.

I think that's the way the bitcoin network is going to evolve.

So I would also say that lifting the 1 Mbyte limit is not really needed, because sooner or later anyway the "small" bitcoin traffic (and also retailer payments where we do not want to wait till the next confirmation at the checkpoint) will move to online wallet services. See also my future vision here.