What prevents the blockchain from becoming impossibly large?

[taltamir]

2009-01-03 - bitcoin introduced according to wikipedia
2012-09-21 - blockchain ~3GB - according to https://bitcointalk.org/index.php?topic=63844.msg1209059#msg1209059
2013-02-20 - my bitcoin data folder is 13.4GB (6GB for "blocks" subfolder + 172MB chainstate subfolder + 7.07GB for blkindex.dat and blk0001.dat through blk0003.dat)

At this rate it bitcoin will become unusable due to the chain files becoming too large to store.
Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

[1715271730]

1715271730

[1715271730]

1715271730

[1715271730]

1715271730

[farlack]

We're all going to need 10tb ssd to use bitcoin

[kinghajj]

To enable this, the blockchain uses a merkle tree to organize the transaction records in such a way that client software can locally delete portions of its own database it knows it will never need, such as earlier transaction records of Bitcoins that have changed ownership multiple times.

https://en.bitcoin.it/wiki/Bitcoin#Confirmations

AFAIK, however, this is still unimplemented in the main Bitcoin client. But at least a solution is known.

[taltamir]

To enable this, the blockchain uses a merkle tree to organize the transaction records in such a way that client software can locally delete portions of its own database it knows it will never need, such as earlier transaction records of Bitcoins that have changed ownership multiple times.

https://en.bitcoin.it/wiki/Bitcoin#Confirmations

AFAIK, however, this is still unimplemented in the main Bitcoin client. But at least a solution is known.

That is reassuring...
But if every client out there deletes the same old transactions wouldn't that be a problem?
Does the client not still need to at least initially download and verify every transaction in the chain before it can start pruning?
Would that introduce reliance on central servers who store the data everyone else is deleting?

[Sukrim]

Well, you could easily run one of these "central servers" yourself too.

[gmaxwell]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system— meaning a $100 1TB HDD can store around 20 years of Bitcoin history..  The users of bitcoin would be foolish indeed to change the Bitcoin protocol to permit more size than that until computers and storage are advanced enough that the increased burden wouldn't be problematic.

[grondilu]

But if every client out there deletes the same old transactions wouldn't that be a problem?

It would be a bad thing if an old transaction is totally forgotten, indeed.  It's quite unlikely to happen, though.  Say the probability for one client to forget a given transaction is p.   Then the probability for all clients to forget it, assuming the probabilities are independant, is p^N, where N is the number of clients.   So it is quite unlikely.

Does the client not still need to at least initially download and verify every transaction in the chain before it can start pruning?

Normally, yes.   But I'm not sure it's absolutely necessary.  If a spent transaction has been forgotten by all the network it means all clients have agreed to acknowledge this transaction to be valid and the bitcoins spent.  So you have to trust the network as a whole, as you do when you suppose that a majority of the computing power is honest.

Would that introduce reliance on central servers who store the data everyone else is deleting?

Such servers would not be "central" since anyone could make one at any time.

[Littleshop]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system— meaning a $100 1TB HDD can store around 20 years of Bitcoin history..  The users of bitcoin would be foolish indeed to change the Bitcoin protocol to permit more size than that until computers and storage are advanced enough that the increased burden wouldn't be problematic.

You just worked against your own point.   If a $100 drive can currently hold 20 years of history, allowing a 4x increase would not be problematic.   Also $105 will buy 2x the storage you quoted.  

Allowing a 4x increase in transactions would still allow 10 years of blockchain to be stored on a $105 drive.  This is not a problem.

[foo]

2013-02-20 - my bitcoin data folder is 13.4GB (6GB for "blocks" subfolder + 172MB chainstate subfolder + 7.07GB for blkindex.dat and blk0001.dat through blk0003.dat)

Windows is lying to you. The block files in blocks\ and its parent directory are hardlinked and should only be counted once. You can delete blk0001.dat through blk0003.dat and blkindex.dat.

[ArticMine]

2013-02-20 - my bitcoin data folder is 13.4GB (6GB for "blocks" subfolder + 172MB chainstate subfolder + 7.07GB for blkindex.dat and blk0001.dat through blk0003.dat)

Windows is lying to you. The block files in blocks\ and its parent directory are hardlinked and should only be counted once. You can delete blk0001.dat through blk0003.dat and blkindex.dat.

On GNU/Linux (Ubuntu 12.04) my .bitcoin folder size is 7.7 GB. Anything more is just Microsoft Windows bloat that can be addressed by defenestrating (replacing Windows with GNU/Linux on) the computer in question.

[taltamir]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system— meaning a $100 1TB HDD can store around 20 years of Bitcoin history..  The users of bitcoin would be foolish indeed to change the Bitcoin protocol to permit more size than that until computers and storage are advanced enough that the increased burden wouldn't be problematic.

So, if bitcoin becomes widely successful and we get max grow, in 20 years it could reach 1.04TB chain and in 100 years 5.2TB chain? We are nearing the physical limits of miniaturization (barring subatomic computing)
And if litecoin succeeds too that means double that... then we can add namecoin...

But even if just bitcoin it seems excessive. 6.3GB already hurts on an SSD bootdrive.

It would be a bad thing if an old transaction is totally forgotten, indeed.  It's quite unlikely to happen, though.  Say the probability for one client to forget a given transaction is p.   Then the probability for all clients to forget it, assuming the probabilities are independant, is p^N, where N is the number of clients.   So it is quite unlikely.

This assumes transactions are deleted randomly using a strong RNG rather then based on a ruleset or a weak RNG.

Windows is lying to you. The block files in blocks\ and its parent directory are hardlinked and should only be counted once. You can delete blk0001.dat through blk0003.dat and blkindex.dat.

Thank you for clarifying.
The folder is now 6.32GB but no real change occured in the free space on the drive when deleted. (or if they did it was a few hundred megs at most and I didn't notice).
Are database and chainstate folders also part of it?

On GNU/Linux (Ubuntu 12.04) my .bitcoin folder size is 7.7 GB. Anything more is just Microsoft Windows bloat that can be addressed by defenestrating (replacing Windows with GNU/Linux on) the computer in question.

Unfortunately most games are for windows... so my gaming system is windows, and my gaming system is the one with a powerful enough GPU to mine on.

[notme]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system— meaning a $100 1TB HDD can store around 20 years of Bitcoin history..  The users of bitcoin would be foolish indeed to change the Bitcoin protocol to permit more size than that until computers and storage are advanced enough that the increased burden wouldn't be problematic.

So, if bitcoin becomes widely successful and we get max grow, in 20 years it could reach 1.04TB chain and in 100 years 5.2TB chain? We are nearing the physical limits of miniaturization (barring subatomic computing)
And if litecoin succeeds too that means double that... then we can add namecoin...

But even if just bitcoin it seems excessive. 6.3GB already hurts on an SSD bootdrive.

It would be a bad thing if an old transaction is totally forgotten, indeed.  It's quite unlikely to happen, though.  Say the probability for one client to forget a given transaction is p.   Then the probability for all clients to forget it, assuming the probabilities are independant, is p^N, where N is the number of clients.   So it is quite unlikely.

This assumes transactions are deleted randomly using a strong RNG rather then based on a ruleset or a weak RNG.

Windows is lying to you. The block files in blocks\ and its parent directory are hardlinked and should only be counted once. You can delete blk0001.dat through blk0003.dat and blkindex.dat.

Thank you for clarifying.
The folder is now 6.32GB but no real change occured in the free space on the drive when deleted. (or if they did it was a few hundred megs at most and I didn't notice).
Are database and chainstate folders also part of it?

On GNU/Linux (Ubuntu 12.04) my .bitcoin folder size is 7.7 GB. Anything more is just Microsoft Windows bloat that can be addressed by defenestrating (replacing Windows with GNU/Linux on) the computer in question.

Unfortunately most games are for windows... so my gaming system is windows, and my gaming system is the one with a powerful enough GPU to mine on.

Steam for linux is working nicely.  There are a few small glitches occasionally, but it's technically still in beta.

[taltamir]

Steam isn't the only platform for games... but I might switch even my gaming machine to it .

Back to topic, I was thinking, could clients not compile a snapshot of current ownership of all coins on the 1st of each year and trade those to each other (assembling the snapshot from processing all the transactions rather then from trying to capture what the status is on exactly the 1st). Those snapshops are then chained together in their own merkele tree.
They are repeatedly verified by the network up to a point. And transaction history older then 5 years is discarded (instead relying on the less granular yearly snapshots).
Perhaps monthly is better then yearly, not sure.

Dumb idea or a plausible long term solution?

[kangasbros]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system— meaning a $100 1TB HDD can store around 20 years of Bitcoin history..  The users of bitcoin would be foolish indeed to change the Bitcoin protocol to permit more size than that until computers and storage are advanced enough that the increased burden wouldn't be problematic.

I think 52 GBytes/yr is not that much. It could easily be 512 GBytes/yr, and still not a big deal.

But then again, I definitely think that transactions should not be free. There should be some cost to making a transaction. I don't want to store others transaction spam just for fun.

[HorseRider]

Is there any mechanism in place (or one that is being designed) to prevent this? If so, where can I find more info about the mechanism for preventing data chain bloat?

The maximum growth rate is limited to 52GBytes/yr by the rules embedded in the system

You mean the 1MB block size limit will lead to this? Is there any possibility that the we change this limit? Why would we have to have this limit except for the protecting the bitcoin from DDoS? I start to be really curious about the block size limit recently.

[Piper67]

2013-02-20 - my bitcoin data folder is 13.4GB (6GB for "blocks" subfolder + 172MB chainstate subfolder + 7.07GB for blkindex.dat and blk0001.dat through blk0003.dat)

Windows is lying to you. The block files in blocks\ and its parent directory are hardlinked and should only be counted once. You can delete blk0001.dat through blk0003.dat and blkindex.dat.

On GNU/Linux (Ubuntu 12.04) my .bitcoin folder size is 7.7 GB. Anything more is just Microsoft Windows bloat that can be addressed by defenestrating (replacing Windows with GNU/Linux on) the computer in question.

"Defenestrating!" Hard LOL!

[Zangelbert Bingledack]

It's really inverse defenestration, because it doesn't involve throwing things out of windows but throwing Windows out of things.

[markm]

It's really inverse defenestration, because it doesn't involve throwing things out of windows but throwing Windows out of things.

Think "to throw out windows".

-MarkM-

[Zangelbert Bingledack]

That's a good workaround.

[streblo]

What prevents the blockchain from becoming impossibly large?
It depends on if you denominate "large" in bytes or dollar cost. If the former, then nothing; the blockchain will inexorably become impossibly large (which is good!). If the latter, then the exponentially decreasing price of storage costs will counter blockchain growth.


TL;DR: As long as the time taken for the blockchain size to double is longer than 14 months, the dollar cost to store the blockchain will exponentially decrease.