Proposal for self-pruning Blockchain

[Technologov]

Proposal for self-pruning Blockchain

NOTE: This idea can be applied to Bitcoin, Litecoin, Ethereum or any other blockchain-based technology.

Today, the Bitcoin blockchain is growing exponentially by leaps and bounds, without any way to cut down it’s size and remove unnecessary parts of it.
In a few decades it could grow into a petabyte-size database due to exponentially accelerating adoption and skyrocketing number of transactions, hurting decentralization.

But do we need _really_ need to keep all the data since the blockchain was born ?
Possibly not. Possibly, we can remove outdated parts of the blockchain.

But how ? What is the mechanism ?

I think it could be similar to how VMware or VirtualBox merging multi snapshots made over time into a single snapshot, flattening it.
I.e. all blocks, that older than, say, 4 years, are merged every 4 years.

In terms of Bitcoin network, it means creating a new genesis block every 210,000 blocks, after the next 210,000 blocks were mined, i.e. at 420,000 blocks (8 years), 630,000 blocks (12 years), etc... And that new genesis block will have the final state of the previous 210,000 blocks.
This new genesis-block will be created backwards in time, i.e. between block 0 and 210,000.

The idea, is that every client can do so in a decentralized manner, because this new genesis block will be behind 210,000 new blocks mined since, there will be no way to reverse transactions.
The only unsolved part for now, is what to do with older hashes, and merkle tree of the blockchain ? But maybe community can help here.

New Genesis block will have a number (0, 210000, 420000, etc...) and also timestamp, in seconds, since UNIX-1970.

NOTE: My proposal allows to rewrite the genesis block, which is potentially dangerous, it will have security implications, so this needs to be evaluated at more depth.

UPDATES in BLUE:
1. I read Satoshi whitepaper, and in chapter 7 he describes how to remove "intermediate" steps with zero balance.
2. I assume, that Bitcoin max. block size limit will be lifted, in order to allow Bitcoin to grow beyond Visa (2000/tps).
3. Additional idea: every 210,000 blocks is to clean "dust accounts" (i.e. accounts with balances below a certain value, say, 1 dollar-cent), transferring it to miners. -- (not in genesis block, but regular block, as it will be forwardly-created)

-Alexey Eromenko “Technologov”, 4.Jan.2015

What do you think of it?

[Ibistru]

Sounds like a great idea in principle but I don't know how it would be feasable, because every block includes the hash of the previous block, so if you change one in the past you are going to get a big mess...

[hhanh00]

The only unsolved part for now, is what to do with older hashes, and merkle tree of the blockchain ? But maybe community can help here.

Spoken like a true manager. 

[DannyHamilton]

Since you've gone back and updated your original post (in blue), I'll post the updated responses here in blue:

1. I read Satoshi whitepaper, and in chapter 7 he describes how to remove "intermediate" steps with zero balance.

This is not true. Satoshi never says anything about balances in the entire whitepaper.  The reason for this is because the Bitcoin protocol does not deal with balances at all, it deals with inputs, output, and the associated transactions.  What Section 7 describes is a method by which entire transactions can be pruned out of the blockchain once the transaction outputs have been spent.  This means that when all the outputs created in a block have been spent, then the entire block can be pruned down to only 80 bytes.  Meanwhile, all of the unspent outputs remain in tact (transactionID, offset, and all) so that they can be verified when someone secides to spend them.

2. I assume, that Bitcoin max. block size limit will be lifted, in order to allow Bitcoin to grow beyond Visa (2000/tps).

But, the only reason for increasing the block size is to allow and encourage increased usage.  Obviously there will be no incentive to increase the block size beyond the current technology's ability to contain the blockchain, since that will discourage increased usage.  Therefore, if you assume that the maximum block size limit will be increased, then you can also assume that there won't be a significant need for your disastrous blockchain "self-recycling".

3. Additional idea: every 210,000 blocks is to clean "dust accounts" (i.e. accounts with balances below a certain value, say, 1 dollar-cent), transferring it to miners. -- (not in genesis block, but regular block, as it will be forwardly-created)

There are no "accounts" in Bitcoin at the protocol level, there are transactions, and transaction outputs.  Something that is valued at "1 dollar-cent" might be worth much more (or much less) in the future.  How will the software decide how much the transaction output is worth? If you allow transactions that spend someone else's bitcoins without a valid signature, then how do you keep an attacker from creating false transactions to steal larger amounts?

Today, the Bitcoin blockchain is growing exponentially by leaps and bounds,

This is not true.  Bitcoin blocks are currently limited to 1 megabyte in size.  Blocks occur on average once every 10 minutes.  This means that the Bitcoin blockchain is growing linearly, NOT exponentially.

without any way to cut down it’s size and remove unnecessary parts of it.

This is not true.  It is possible to prune data out of the blockchain. There hasn't been a lot of need for it yet, so there hasn't been much priority on enhancing Bitcoin Core to do so.  Eventually pruning will be encoded into Bitcoin Core.

In a few decades it could grow into a petabyte-size database due to exponentially accelerating adoption and skyrocketing number of transactions, hurting decentralization.

Unless the protocol is modified to increase the maximum block size, it will take nearly 20 years for the blockchain to reach 1 petabyte at 1 megabyte per block and 1 block every ten minutes.  20 years ago, IBM hadn't even released the 170MB Microdrive yet.  Today, you can purchase a half terabyte on a SD card.  Do you really think that a petabyte is going to be considered to be a large amount of storage in another 20 years?

But do we need _really_ need to keep all the data since the blockchain was born ?

No.  You can just run an SPV node if you don't want to store the entire blockchain.  It's explained in the original white paper.  Furthermore, as I already stated, pruning is also possible.

Possibly not. Possibly, we can remove outdated parts of the blockchain.

But how ? What is the mechanism ?

Read the white paper.

I think it could be
- snip -
What do you think of it?

Sounds like a bad idea to me.  Especially since much better ideas already exist and were described in the orignal bitcoin whitepaper.

Why do people insist on coming up with ideas without at least taking the time to read the whitepaper and understanding the basic concepts first?

[dabura667]

So everyone throws away all the old blocks from the most recent "genesis" block.

I want to spend a utxo from a block before that "genesis" block.

How do you verify that the transaction hash I am referring to is valid?

You can't.


So you'd be throwing away all bitcoins confirmed prior to each genesis block.

[Technologov]

>Why do people insist on coming up with ideas without at least taking the time to read the whitepaper and understanding the basic concepts first?

I did read the whitepaper twice. And what ? It is way too short, and lack details.

[dexX7]

rdponticelli commented on 14 Aug 2014:

This pull implements a new mode of operation which automatically removes old block files trying to maintain at most a maximum amount of disk space used by the node. This amount is configured by the user with the -prune switch.

There's also a lightweight sanity check which executes periodically during runtime to make sure the minimum block files required for the node to be operative are present.

This should allow to lower the amount of resources needed to run a node.

See the individual commits, about all the changes introduced.

https://github.com/bitcoin/bitcoin/pull/4701

[DannyHamilton]

>Why do people insist on coming up with ideas without at least taking the time to read the whitepaper and understanding the basic concepts first?

I did read the whitepaper twice. And what ? It is way too short, and lack details.

Apparently you missed Section 7 titled: "Reclaiming Disk Space".

It starts with the sentence:

Once the latest transaction in a coin is buried under enough blocks, the spent transactions before it can be discarded to save disk space.

Satoshi even takes care of the math for you, demonstrating that your "petabytes!" is nothing but a bunch of FUD:

Old blocks can then be compacted by stubbing off branches of the tree. The interior hashes do not need to be stored. A block header with no transactions would be about 80 bytes. If we suppose blocks are generated every 10 minutes, 80 bytes * 6 * 24 * 365 = 4.2MB per year. With computer systems typically selling with 2GB of RAM as of 2008, and Moore's Law predicting current growth of 1.2GB per year, storage should not be a problem even if the block headers must be kept in memory.

[gmaxwell]

The idea, is that every client can do so in a decentralized manner, because this new genesis block will be behind 210,000 new blocks mined since, there will be no way to reverse transactions.
The only unsolved part for now, is what to do with older hashes, and merkle tree of the blockchain ? But maybe community can help here.

Adding to the excellent points DannyHamilton made,  you're completely ignoring the problem of introducing a new node to the network. If the "genesis block" is rewritten and the history forgotten by everyone-- what basis do they have to judge the new system? You haven't provided a security objective/argument at all for that case. (Fortunately, as DannyHamilton points out, the original Bitcoin whitepaper addresses storage and does so in a way that doesn't make it unclear how a new node can be initialized. And Bitcoin core did 99% of the work to implement that storage reduction a couple years ago, in 0.8, it-- as mentioned-- just hasn't been a priority, due to the -- as mentioned-- linear growth maximum).

If you're feeling that the response here is a bit to blunt you should realize that this proposal (and it's ilk) has been repeated here many times since 2011, seemingly by people who missed section 7 of the white paper and seldom proposing something even as good as the design Bitcoin already has. While it's great that you're enthusiastic and have ideas, when you make a public post you're consuming the time of all the readers and people who respond to you. It's helpful if you describe the research you already did when you first propose something in a community that you're new to, since when someone seems to have missed something obvious (like the whole section on the subject in the project's whitepaper) it gives an impression that the poster didn't care for the costs they imposed on others.

[altcoinex]

Its amusing how often someone comes up with the great idea of reducing blockchain size, without searching forums or bitcoins documentation not only that development for pruning is already occurring but that it is briefly outlined in the original whitepaper itself. Also, how rushed people are to suggest the ideas without understanding the fundamentals of security involved, nor the actual data that would need to be maintained in a reduced blockchain (UTXOs)....

[bcearl]

You don't really cut size just by putting all the transactions into a single block.

It is true that you can remove some transactions, but even the number of transactions that CANNOT be removed is growing exponentially.



PS: Of course there is a limit, since there is a finite number of Satoshis. But that limit does not help us much now, and will not help has when Satoshis will be subdivided in the far future.

[bcearl]

This is not true.  Bitcoin blocks are currently limited to 1 megabyte in size.  Blocks occur on average once every 10 minutes.  This means that the Bitcoin blockchain is growing linearly, NOT exponentially.

This is not a valid argument. When Bitcoin's popularity will grow, there will be more transactions and that limit will have to be raised anyway, if you want to keep Bitcoin working.

I agree with everything else you said.

[DannyHamilton]

This is not a valid argument. When Bitcoin's popularity will grow,

You are assuming that Bitcoin's popularity will grow.  It is entirely possible that usage of bitcoin will never be much larger than it is today, or that it will grow so slowly that there won't be any need to increase the block size for many decades, or that it will only become more popular with those that want to transact in VERY high values so that high transaction fees are acceptable, or that off-chain (or alternative chain) solutions will absorb all but the highest value transactions.  There are many possibilities, it isn't a good idea to assume that one particular possibility that you personally like is the one that will actually come about.

there will be more transactions and that limit will have to be raised anyway, if you want to keep Bitcoin working.

This is not necessarily true.  Bitcoin will work just fine without increasing the maximum block size.  Regardless, I suspect that the maximum block size will be increased.  There appears to be a significantly large majority of users that feel that very small transaction fees are a desirable feature even if it means increasing the block size.

[bcearl]

@Danny: Of course nobody knows what the future will bring. But the whole issue of size only makes sense under the assumption that it will.



If block size is limited, small transactions will be pretty much impossible. A miner will always pick a larger transaction over a smaller, because for him it does not matter whether the user is giving a 1% fee or a 10% fee. Only the absolute amount matters. Of course most people don't want this to happen.

[DannyHamilton]

@Danny: Of course nobody knows what the future will bring. But the whole issue of size only makes sense under the assumption that it will.

I'm not sure what you mean by "the whole issue of size only makes sense under the assumption that it will"

If block size is limited, small transactions will be pretty much impossible.

Correct.  That's a very real possibility, and I'm ok with that.  Apparently a lot of other people are not.  Since Bitcoin is a consensus system, changing the block size requires overwhelming acceptance from all the miners AND all the users.  A simple majority isn't enough.  There are quite a few developers and outspoken individuals that seem to think that the overwhelming majority exists.  They better be right or we'll have quite a mess on our hands.

Personally, I find that it's safer to just leave the block size alone, but it will be interesting to see what the future brings.

A miner will always pick a larger transaction over a smaller, because for him it does not matter whether the user is giving a 1% fee or a 10% fee. Only the absolute amount matters.

Correct, a miner should always choose to confirm the transactions that pay the absolute largest ratio of bitcoins to bytes.

Of course most people don't want this to happen.

Well.  Some people don't want it to happen.  Some don't care one way or the other. And then some would prefer that it happen.  Where the majority lies is perhaps debatable, but I suspect you are correct.  The real question is:

"Is that majority overwhelming enough to force a consensus?"

[bcearl]

You argue as if this was some sort of an isolated question. It is not. And I am not talking about opinion polls or anything like this. You just have to imagine what it means if Bitcoin is becoming unattractive for small payments. It means that some users will stop using Bitcoin and look for something else, which will relatively* hurt the value. The “majority”, whoever that is, will have an interest to maximize the value of their assets.




*) Even if the value is still rising and not dropping, it may rise less in case of fewer users than it would rise in case of more users.

[Technologov]

UPDATED (in blue)

[DannyHamilton]

UPDATED responses (in blue)

[hhanh00]

No need to be so finicky about the terminologies.

What the OP suggest is to introduce a snapshot block or blocks in the blockchain. Regular blocks are tx which are essentially delta changes to the UTXO set (he calls it accounts). Once in a while, he wants to flatten and write a snapshot. In the process, if some UTXOs are too small, he would discard them turning them into fees. Of course it's a protocol change but it's not impossible to do.

A couple of things to consider though:
- these snapshots will be huge. Even if you do it while blocks are being processed they will need to be mined and properly rewarded.
- they could be a downtime while these blocks are produced but you could mix snapshot UTXOs & normal tx though.

Not a big problem and it's done in a few applications but at the moment, the blockchain growth is not a big deal. Even if you think the block size grows linearly, the accumulated blockchain only grows quadratically - not exponentially.
Besides trimming spent tx introduces other problems for deterministic wallets.

Btw, it's not a new idea and has been mentioned a zillion times.

[cr1776]

No need to be so finicky about the terminologies.

What the OP suggest is to introduce a snapshot block or blocks in the blockchain. Regular blocks are tx which are essentially delta changes to the UTXO set (he calls it accounts). Once in a while, he wants to flatten and write a snapshot. In the process, if some UTXOs are too small, he would discard them turning them into fees. Of course it's a protocol change but it's not impossible to do.

A couple of things to consider though:
- these snapshots will be huge. Even if you do it while blocks are being processed they will need to be mined and properly rewarded.
- they could be a downtime while these blocks are produced but you could mix snapshot UTXOs & normal tx though.

Not a big problem and it's done in a few applications but at the moment, the blockchain growth is not a big deal. Even if you think the block size grows linearly, the accumulated blockchain only grows quadratically - not exponentially.
Besides trimming spent tx introduces other problems for deterministic wallets.

OP should go ahead and do it then. Once there is some code, it will be much easier to evaluate.  

Terminology is important here because the proposal has to be precise, but if OP is really believes in this, code is even better.

:-)