Block-Size Proposal: "Pruning Blocks"

[briguy37]

The Block-Size Problem

The main concern with raising the block-size limit is that the blockchain will grow at an unsustainable rate that nodes will eventually not be able to support.  Thus, the currently 60GB blockchain will eventually become 1TB, 100TB, and more as adoption grows, and this will be accelerated with an increased block-size.

However, in order for a currency to be useful people MUST be able to spend it.  The artificial 1MB block-size limit puts a cap on this, and so because of the growth we will soon hit this artificial barrier, at which point it will not be possible for everyone who wants to perform a transaction to be able to perform a transaction regardless of the fees paid.  This is good for miners in the short-term because fees will be driven up, but bad for both them and the Bitcoin economy because again, a currency is ONLY useful if you can spend it.

There is no way around keeping the entire transaction history of the world in the blockchain, and there is no way around that it will be massive if world-wide adoption happens.  Satoshi knew this that and believed Moore's law would let technology improve fast enough where storage space for the entire chain would not be an issue for those wanting to run full nodes, and I believe he's correct.  However, if Bitcoin is widely adopted only an elite group of full nodes dedicated to this will be able to handle this amount of data.*

Satoshi knew this, and so in his original paper he foresaw that most clients would not care about the entire transactional history of the world, and would instead only care about the values in each account.  Thus, he mentioned the concept of nodes "pruning" the blockchain.  However, his idea was that this pruning process would be off the blockchain, and so currently to prune the chain you must either download the entire chain and prune it yourself or download the pruned chain from a trusted source.  Since a main design principle of Bitcoin is that it tries to eliminate the need for trust, any node who agrees with this must download the full chain, validate it, and prune it.


"Pruning Blocks" Proposal
Add a special block at regular intervals into the blockchain called the "Pruning Block".  This block will eliminates the need to download or keep the entire chain for partial nodes, but still gives them a very similar level of trust as downloading the full chain.

Pruning Block Requirements

1.  The Pruning Block shall be a special block that occurs every X blocks
  a.  X shall be determined by votes from the community.  For example, to have the pruning block approximately once per week it could start at 6 * 24 * 7 = 1008
2.  The Pruning Block shall contain the regular transaction information so as not to disrupt normal business
3.  The Pruning Block shall contain an appended list of all addresses that contained a balance along with the associated balance as of the previous block.**
  a.  The amended balances shall NOT be included in the calculation of the hash of the block***
  b.  The hash of the amended balances SHALL be included in the block***
  c.  The hash of the amended balances SHALL be included in calculation of the hash of the block***
  d.  The balances shall not take into account the transactions for this pruning block***
4.  The Pruning Block shall only be accepted by clients if it matches all of the regular block requirements plus the aforementioned requirements. 
  a.  The block-size cap shall not consider the appended list of balances in its block-size calculation


Problems
The main problem with this is if someone maliciously inflates the number of addresses to increase the size of the pruning blocks.  For example, they could take 10 bitcoins and split them into 1 billion addresses with 1 satoshi in each as an attack, which would blow the pruning block size from 13.5MB to 27GB.  This would normally be prevented because of transaction fees and block-size caps.  However, if the block-size cap was removed completely and the attacker mined a block and sent the transactions through with zero-fees, this would be theoretically possible.  In the process of doing this they would be hurting themselves, as they would not only be throwing money away, but also devaluing the currency they are significantly invested in with their mining operation (as it takes a lot of power to mine a block nowadays).  However, because of this problem it is recommended that the block-size limit be removed entirely. 


Results on blockchain
If Pruning Blocks were implemented today, the effect on the entire blockchain size would be minimal.  For example, if we assume the current max block-size of 1MB was used for each block, the Pruning Block would be about 14.5MB**.  If the pruning block was set to occur weekly (every 1008 blocks), the growth-rate of the blockchain would be only 1.34% faster than without pruning blocks (1021.5MB/week vs 1008MB/week).  Thus, full nodes would only be affected by this change by a small amount.

However, the effect for partial nodes is very noticeable.  Instead of downloading the full 60GB blockchain and pruning, miners would only need to download blocks back to the last pruning block.  With 1MB block sizes and a pruning block weekly, this would range anywhere from .0145GB to 1.0215GB.  Furthermore, once the next valid pruning block arrives, the node can clear out all the previous blocks.  In this way the storage requirements for the pruned blockchain will be much more manageable and consistent, as they would only grow when Bitcoin adoption/use increases.

More importantly, this solution will remove many of the concerns about raising the artificial block-size limit.  This is because storage/download requirements for nodes will no longer be based on the number of transactions in the history of the world, but instead be based on the number of addresses with a balance and a small subset of recent transactions.  This will lend itself to nodes being much more willing to accept larger and larger transaction blocks, as they only need to keep them until the next pruning block.  For example, 60GB is very low upper-limit for current storage requirement for each node, as that is what full nodes are storing right now, and also what untrusting partial nodes must initially download.  To get this value for partial nodes, the maximum block-size can be set to 50MB, which would allow the current Bitcoin ecosystem to be able to process 50 times more transactions than it supports today, while at the same time putting no extra memory burden on the existing nodes of today.





*If all 7.125 billion people averaged 1 transaction per day at 500 bytes/transaction (the current average), the blockchain size would increase by 3.5625 TB daily

**A bitcoin address is 20 bytes (160 bits), and you can currently have anywhere from 1 satoshi to 21,000,000BTC, which can be represented without loss as 7 bytes, so optimally you can store the current balance of one address with 27 bytes per address, or the balance of all addresses with 27*numAddresses.  Currently there are about 500,000 addresses, which would yield a minimum pruned block size of 13.5MB.  Carrying this forward, if everyone in the world had bitcoin addresses that would be 7.125 billion * 27 bytes = 192.375 GB.  That may seem large, but to put things into perspective, if everyone averaged 1 transaction/day the blockchain would grow by over 18 times that amount daily (3.5625 TB*).

***These requirements are included for performance reasons to make it less expensive when adding new transactions to the pruning block.  Following these requirements gives you the ability to add a transaction without needing to include or recalculate the full balance information of the world to calculate your new block hash.

[shorena]

Lets assume I have just installed my client. How can I trust the "balances" (I assume you want to store the UTXO set, but details...) when I can not calculate them myself from history data and compare it to the data in the "pruning block"? If I need to do this, someone will have to keep all history data for me to download and verify. Once I verified the data I can discard it. This however is already implemented and its called - you guessed it - pruning.

[briguy37]

Lets assume I have just installed my client. How can I trust the "balances" (I assume you want to store the UTXO set, but details...) when I can not calculate them myself from history data and compare it to the data in the "pruning block"? If I need to do this, someone will have to keep all history data for me to download and verify. Once I verified the data I can discard it. This however is already implemented and its called - you guessed it - pruning.

It removes the need to trust a 3rd party and replaces it with the need to trust the blockchain: 

Currently, if you download only a partial chain you must trust a third party for the balance information of the blocks you did not download.

With this, if you download only a partial chain you must trust blockchain technology for the balance information of the blocks you did not download.

[CIYAM]

You provide no technical details and have clearly very little understanding about the technology (from what you have posted).

Yet you think you have come up with some grand idea that others should look at?

No-one is going to waste their time with your ideas unless you actually show that they have the slightest bit of merit (which so far they do not).

Understand how the blockchain works before trying to improve it.

[briguy37]

You provide no technical details and have clearly very little understanding about the technology (from what you have posted).

Yet you think you have come up with some grand idea that others should look at?

No-one is going to waste their time with your ideas unless you actually show they have the slightest bit of merit (which so far they do not).

This is a rough proposal of an idea that I find promising and that I have not heard before.  I am admittedly not an full-time expert in the blockchain, but I am a software engineer by trade and have looked into the technology a great deal.  Although this is my first development contribution to the community I hope that it will be judged based on merit and not by my perceived inexperience.

Here are the merits I see

Merits:
1) Simplify and globalize the pruning process
2) Eliminate the need to download the full blockchain to get an trustworthy & accurate ledger

[CIYAM]

No, I've come up with a rough proposal of an idea that seems promising and I would like to discuss it with the community for them to poke holes in, improve, and see if it actually has merit.

No - you have a silly idea and it was pointed out to you why that is so - yet you ignore that and still think the "community" should pay attention to you.

Please read more about the actual blockchain technology before "trying to improve it" (you'll notice that no Bitcoin dev has taken the slightest bit of interest in your idea although maybe next you'll say that is some sort of conspiracy).

[briguy37]

No, I've come up with a rough proposal of an idea that seems promising and I would like to discuss it with the community for them to poke holes in, improve, and see if it actually has merit.

No - you have a silly idea and it was pointed out to you why that is so - yet you ignore that and still think the "community" should pay attention to you.

Please read more about the actual blockchain technology before "trying to improve it" (you'll notice that no Bitcoin dev has taken the slightest bit of interest in your idea although maybe next you'll say that is some sort of conspiracy).

I'm sorry but them saying pruning already exists does not make this idea silly, and I've already responded why.  If you have a reason why my response to that was inadequate then I am all ears.

[CIYAM]

I'm sorry but them saying pruning already exists does not make this idea silly, and I've already responded why.  If you have a reason why my response to that was inadequate then I am all ears.

If you mean any Bitcoin devs then I'm sorry - none of them have (or likely will) respond to this topic.

Please go and actually find out about the pruning that Bitcoin uses already before proposing something inferior to replace it with.

Try and understand that naive (and sometimes just plain stupid) ideas to "improve Bitcoin" are suggested every day on this forum by people with no software development skills whatsoever (and you have failed to show that you have any).

If the developers were to waste their time having to read and respond to every suggestion such as yours then they would never have time to actually do the work that they are doing (which is why you are not going to get a response from them).

So again - if you really have anything serious to offer then firstly go and study how Bitcoin pruning works before trying to tell others how "it should work" (when you've not even displayed that you can code).

[briguy37]

.. if you really have anything serious to offer then firstly go and study how Bitcoin pruning works before trying to tell others how "it should work" ..

I have studied it, I see major problems in the future if nothing is done, and that is why I'm posting about the topic to begin with.  Here are the relevant facts:

1) Without trust, you must currently download the entire blockchain in order to get a pruned one.  (Note:  I know a lot more about pruning than this and the rest is not relevant. I'm willing to discuss my knowledge in another thread but here is not the place.)
2) As the blockchain grows in total size and pruning software improves, more nodes will switch from full mode to pruning mode.
3) The full blockchain will only ever grow, and will grow proportionately to the number of users using it.

Points 2 & 3 combined are the most worrisome, as less nodes to deliver more data may mean that 1 will become virtually impossible. 

For example, we currently are at a 60GB blockchain with 500k addresses.  Imagine 500 million addresses.  Since the blockchain can only grow, with those levels we can expect very quickly a 60TB blockchain.  If you have a 10MB/s (that's 80Mbps) internet speed, it will take nearly 69.4 days to download the full chain as you prune it along the way, and that's not even counting the 10,000 additional blocks that have been added along the way.  Furthermore, that assumes the full nodes have the bandwidth to support those speeds and that you were able to keep them up 24/7.  More realistically you'll be competing with 10's of other downloading clients per full node since the number will increase proportionately to the total blockchain download time multiplied by the number of users downloading a partial/full node.  In that case, most if not all will cancel the download part of the way through, meaning no one EVER gets the full chain.

Introducing Pruning Blocks into the blockchain will eliminate this problem entirely, and in the process improve download performance for the full chain because only those users desiring to run a full node will actually need to download it.

[CIYAM]

So you refuse to say about anything relevant and want to carry on with some silly stuff that no dev has replied to or will.

Good luck!

[briguy37]

So you refuse to say about anything relevant and want to carry on with some silly stuff that no dev has replied to or will.

Good luck!

I reread all your posts here and you have given exactly 0 technical reasons to back up any of the points you have made thus far (go ahead, read your own posts tell me where I'm wrong).

If you think my points are irrelevant, then please stop being a hypocrite and have a TECHNICAL discussion with me, as I'm starting to get the feeling I'm just being trolled here...

[CIYAM]

If you think my points are irrelevant, then please stop being a hypocrite and have a TECHNICAL discussion with me, as I'm starting to get the feeling I'm just being trolled here...

Okay - well when (even) one of the devs says that your idea is a good one then I'll admit that I should have not rejected your idea.

Fair?

[briguy37]

If you think my points are irrelevant, then please stop being a hypocrite and have a TECHNICAL discussion with me, as I'm starting to get the feeling I'm just being trolled here...

Okay - well when (even) one of the devs says that your idea is a good one then I'll admit that I should have not rejected your idea.

Fair?

Fair.

[achow101]

Lets assume I have just installed my client. How can I trust the "balances" (I assume you want to store the UTXO set, but details...) when I can not calculate them myself from history data and compare it to the data in the "pruning block"? If I need to do this, someone will have to keep all history data for me to download and verify. Once I verified the data I can discard it. This however is already implemented and its called - you guessed it - pruning.

It removes the need to trust a 3rd party and replaces it with the need to trust the blockchain:  

Currently, if you download only a partial chain you must trust a third party for the balance information of the blocks you did not download.

With this, if you download only a partial chain you must trust blockchain technology for the balance information of the blocks you did not download.

That doesn't answer the question. If a new client joins and they start syncing from the latest pruning block, how does the node know that the balances in a pruning block are correct? A miner could mine an invalid pruning block  with incorrect balances but then how would a new client know that that pruning block is invalid? How can it determine this without trusting anyone like how currently full nodes do not trust anyone.

Also, bitcoin doesn't work on balances but rather transaction outputs. The way this would work with less complex changes would be to create new UTXOs with the balances. However, this essentially gives miners the ability to spend other people's bitcoin.

Additionally disk space is becoming a lot cheaper and network bandwidth is also expected to increase. We won't be having to manage a full blockchain with current technology, technology will advance and the blockchain should not outpace technology growth as long as there i is a block size limit.

[hv_]

If you think my points are irrelevant, then please stop being a hypocrite and have a TECHNICAL discussion with me, as I'm starting to get the feeling I'm just being trolled here...

Okay - well when (even) one of the devs says that your idea is a good one then I'll admit that I should have not rejected your idea.

Fair?

Fair.

I think, you are just too early with this, since they is no urge.
Look at how the scaling or the decentralization issue is treated, or is it an issue....hm.

[briguy37]

If a new client joins and they start syncing from the latest pruning block, how does the node know that the balances in a pruning block are correct? A miner could mine an invalid pruning block  with incorrect balances but then how would a new client know that that pruning block is invalid? How can it determine this without trusting anyone like how currently full nodes do not trust anyone.

If a miner submits a pruning block with modified balances, they will be rejected by the rest of the mining community as an invalid block because the submitted balances do not match what they have, and so it will never make it into the actual blockchain to stay.  Thus, similar to accepting payments you should not accept a parsing block until it is 6 or more transactions old (and more is probably preferable) if you are downloading from scratch.

Also, bitcoin doesn't work on balances but rather transaction outputs. The way this would work with less complex changes would be to create new UTXOs with the balances. However, this essentially gives miners the ability to spend other people's bitcoin.

Yes, the pruning block would essentially include a copy of all UTXOs as of the previous block.  However, any pruning block submitted with extra, incorrect, or missing UTXO's should automatically be rejected as an invalid pruning block, so it would not give miners the ability to spend other people's bitcoin (again those would be rejected by other miners).

Additionally disk space is becoming a lot cheaper and network bandwidth is also expected to increase. We won't be having to manage a full blockchain with current technology, technology will advance and the blockchain should not outpace technology growth as long as there i is a block size limit.

I surely hope technology keeps up.  However, if the Blockchain Size continues doubling every year as it has been, it will quickly catch up to Moore's law as that only doubles every 2 years.  Hopefully the economy reaches user-saturation before then at which point blockchain growth will be linear, but if not as you say the block-size limit prevents this from becoming an issue.  The main problem there is that exponentially more people each year will be unable to perform transactions, which is bad for the currency.

[achow101]

If a miner submits a pruning block with modified balances, they will be rejected by the rest of the mining community as an invalid block because the submitted balances do not match what they have, and so it will never make it into the actual blockchain to stay.  Thus, similar to accepting payments you should not accept a parsing block until it is 6 or more transactions old (and more is probably preferable) if you are downloading from scratch.

Confirmations you mean, right? So in this case 6 blocks deep. So what if the miner produces a pruning block for me nodes that has a higher block height than the actual? Then the miner expands on that block? Currently the best blockchain is chosen by having the greatest sum of difficulty, but this breaks that because it cannot be calculated with a partial blockchain. How would the node be able to know that the blockcthat it knows of is incorrect? If it receives the actual blockchain how will it know that is the correct one versus the incorrect one given by a malicious node/miner?

Yes, the pruning block would essentially include a copy of all UTXOs as of the previous block.  However, any pruning block submitted with extra, incorrect, or missing UTXO's should automatically be rejected as an invalid pruning block, so it would not give miners the ability to spend other people's bitcoin (again those would be rejected by other miners).

So then what would happen to the original UTXO that is replaced/removed by the pruning block. What if someone wanted to spend from that? Someone who has synced the bickering and then took it offline prior to the pruning block being mined?

I surely hope technology keeps up.  However, if the Blockchain Size continues doubling every year as it has been, it will quickly catch up to Moore's law as that only doubles every 2 years.  Hopefully the economy reaches user-saturation before then at which point blockchain growth will be linear, but if not as you say the block-size limit prevents this from becoming an issue.  The main problem there is that exponentially more people each year will be unable to perform transactions, which is bad for the currency.

The blockchain cannot grow exponentially forever because the block size limit forces a maximum growth rate. It can only grow at 144 Mb per day right now.

[briguy37]

So what if the miner produces a pruning block for me nodes that has a higher block height than the actual? Then the miner expands on that block? Currently the best blockchain is chosen by having the greatest sum of difficulty, but this breaks that because it cannot be calculated with a partial blockchain. How would the node be able to know that the blockcthat it knows of is incorrect? If it receives the actual blockchain how will it know that is the correct one versus the incorrect one given by a malicious node/miner?

Very good points.  As a core assumption of Bitcoin, a single miner can not replicate the difficulty of the entire network.  Using that knowledge, here is a first-take at such a procedure to download and verify a pruned blockchain:

1) A new miner downloads all unique pruned blockchains visible to them.  It is assumed that one of them will be correct because the Bitcoin network is the most powerful network in the world.
2) If there are no competing chains, great!  If there are competing chains, they must be compared to find out which is the true blockchain and which is/are the counterfeit(s).  As you pointed out it is easier to counterfeit back to a Pruning Block than the blockchain because you do not need to generate all blocks back to the genesis block, you just need to lie with the pruning block you start with.  Thus, we can take additional precautions:
  a) Check the block difficulty level of the blocks.  If it is too low throw the blocks out.  This will limit the people who could take advantage of this attack to only very powerful miners, as you can't fake the difficulty of a block.
  b) Download back to the last 2 pruning blocks instead of just the last 6 regular blocks.  At 1008 blocks between pruning blocks plus 6 confirmations for the pruning block, this means the malicious miner would need to generate at least 1014 near the current difficulty of the actual network instead of 6.  This will limit the risk to extremely powerful miners, as even a miner/pool with 10% of the network would take 2.5 months to do this.
  c) Check that current transactions are getting through in a timely fashion (e.g. monitor for 6+ blocks for anything fishy).  Your node can monitor all published transactions and make sure they are included.  If they are not being included in one chain they you know something is fishy with that.  This will prevent against pre-generating 2000 blocks and then re-playing the blocks back.
  d) Validate that times of the blocks make sense.  This will prevent re-playing a pre-generated block sequence more than once since the dates/times will not line up.
  e) As a final fallback for those not willing to take any risk, download the whole chain.

Further Note:  Because there will always be full nodes (or even partial nodes running against the true chain), they can always warn people and miners can manually switch back to the correct chain if someone does try this attack.  Once on the correct chain, they will stay there, so power will not be moved over to fake chains.

Thus, to do this attack, you must give up your mining reward for 1000+ blocks, and at the end you will be caught by full-nodes and your chain abandoned as news of the fake chain spreads across the world with proof from the full nodes, which will be a very big loss for you.  Thus, it is pretty clear that such an attack would not be in your own self-interest.

So then what would happen to the original UTXO that is replaced/removed by the pruning block. What if someone wanted to spend from that? Someone who has synced the bickering and then took it offline prior to the pruning block being mined?

The pruning block doesn't replace the UTXOs, it only contains copies of them.  They should not be considered separate transactions. 

The blockchain cannot grow exponentially forever because the block size limit forces a maximum growth rate. It can only grow at 144 Mb per day right now.

True, but the amount of transactions people want to make can grow exponentially.  If the limits aren't raised accordingly, exponentially more people will be boxed out, fees will skyrocket, and the Bitcoin adoption rate will decrease.

[shorena]

So what if the miner produces a pruning block for me nodes that has a higher block height than the actual? Then the miner expands on that block? Currently the best blockchain is chosen by having the greatest sum of difficulty, but this breaks that because it cannot be calculated with a partial blockchain. How would the node be able to know that the blockcthat it knows of is incorrect? If it receives the actual blockchain how will it know that is the correct one versus the incorrect one given by a malicious node/miner?

Very good points.  As a core assumption of Bitcoin, a single miner can not replicate the difficulty of the entire network.  Using that knowledge, here is a first-take at such a procedure to download and verify a pruned blockchain:

1) A new miner downloads all unique pruned blockchains visible to them.  It is assumed that one of them will be correct because the Bitcoin network is the most powerful network in the world.

A sybil attack is still possible, you cant just assume it away.

[achow101]

1) A new miner downloads all unique pruned blockchains visible to them.  It is assumed that one of them will be correct because the Bitcoin network is the most powerful network in the world.

That isn't a good assumption to make. Sybil attacks on the bitcoin network are still possible.

2) If there are no competing chains, great!  If there are competing chains, they must be compared to find out which is the true blockchain and which is/are the counterfeit(s).  As you pointed out it is easier to counterfeit back to a Pruning Block than the blockchain because you do not need to generate all blocks back to the genesis block, you just need to lie with the pruning block you start with.  Thus, we can take additional precautions:
  a) Check the block difficulty level of the blocks.  If it is too low throw the blocks out.  This will limit the people who could take advantage of this attack to only very powerful miners, as you can't fake the difficulty of a block.

What constitutes a too low difficulty? How would a new node determine that?

  b) Download back to the last 2 pruning blocks instead of just the last 6 regular blocks.  At 1008 blocks between pruning blocks plus 6 confirmations for the pruning block, this means the malicious miner would need to generate at least 1014 near the current difficulty of the actual network instead of 6.  This will limit the risk to extremely powerful miners, as even a miner/pool with 10% of the network would take 2.5 months to dochecked

While that could work, a malicious miner could generate blocks with timestamps several months in advance and pregenerate them.

  c) Check that current transactions are getting through in a timely fashion (e.g. monitor for 6+ blocks for anything fishy).  Your node can monitor all published transactions and make sure they are included.  If they are not being included in one chain they you know something is fishy with that.  This will prevent against pre-generating 2000 blocks and then re-playing the blocks back.

That could work too prevent 2b from happening but what if the miner pregenerates transactions? The transactions in the blocks are pregenerated and broadcasted to the victim node at the appropriate times?

  d) Validate that times of the blocks make sense.  This will prevent re-playing a pre-generated block sequence more than once since the dates/times will not line up.

IIRC bitcoin core already does that check for new blocks anyways. Lookup the timejacking attack.

  e) As a final fallback for those not willing to take any risk, download the whole chain.

That is always a safe backup.

The pruning block doesn't replace the UTXOs, it only contains copies of them.  They should not be considered separate transactions. 

Well also having those UTXOs will inflate that block size because it has to include all previous UTXOs.

True, but the amount of transactions people want to make can grow exponentially.  If the limits aren't raised accordingly, exponentially more people will be boxed out, fees will skyrocket, and the Bitcoin adoption rate will decrease.

Yes but there will always be a limit so the blockchain will always grow linearly once the maximum is reached. Also there are other scaling solutions than just the block size limit such as lightning and sidechains.