'Blockchain chunking' - can that be done?

[JPage]

Why can't we just 'freeze' and archive the blockchain when it gets to a prescribed amount, say 150GB.  Then, start a new chain with a special genesis block that includes addresses and balances from the UTXO.  We hash that genesis block to get a verification value.  Then, all future transactions rely on that hash value to assure no changes have been made to the values of the genesis block thus assuring all values go back to the very beginning.  

Then we can do 1 minute blocks, or 8MB blocks to increase network capacity.  Each time the blockchain gets to big, we do another freeze operation to reset its size back to 0 with the history saved as a blockchain archive.  

In this way, we don't have to drag around every transaction 7000 transactions / second for the rest of time.  

Can anyone explain why this doesn't work?

[1715628936]

1715628936

[1715628936]

1715628936

[1715628936]

1715628936

[1715628936]

1715628936

[1715628936]

1715628936

[DannyHamilton]

Why can't we just 'freeze' and archive the blockchain

Archive it where?  Who is responsible for maintaining that archive?  Who is going to pay for the archive?  How will we know if the archive has been altered at all?  If there are multiple versions of the archive, how will we determine which one is correct?

[JPage]

Why can't we just 'freeze' and archive the blockchain

Archive it where?  Who is responsible for maintaining that archive?  Who is going to pay for the archive?  How will we know if the archive has been altered at all?  If there are multiple versions of the archive, how will we determine which one is correct?

Special Archive Nodes.  you know if the archive has been altered, because the hash is passed into the new chain.  Any change to the old chain would render that verification hash invalid.  If there are multiple versions of the archive, only the correct one will produce the right hash.  

This way, 'common nodes' would only need a hash value and modest 150GB of local memory.  Some special 'archive nodes' would keep the entire blockchain history and require many TB of memory.  This way common nodes don't have to carry around for all of time everyone of the damn Satoshi Dice transactions.

Because the hash value from the prior retired chain is correct, a new chain can be sure that all derived balances are valid. 

[DannyHamilton]

Special Archive Nodes.  you know if the archive has been altered, because the hash is passed into the new chain.  Any change to the old chain would render that verification hash invalid.  If there are multiple versions of the archive, only the correct one will produce the right hash.  

How is a "Special Archive Node" any different than a regular node today?  Since the integrity of the network depends on the existence of these "Special Archive Nodes", won't it be just as important to have them as it is to have regular nodes today?

Doesn't that mean that we effectively already have this?  Aren't you essentially just renaming "full nodes" as "Special Archive Nodes" and renaming pruning nodes as "Full Nodes"?

[JPage]

Special Archive Nodes.  you know if the archive has been altered, because the hash is passed into the new chain.  Any change to the old chain would render that verification hash invalid.  If there are multiple versions of the archive, only the correct one will produce the right hash.  

How is a "Special Archive Node" any different than a regular node today?  Since the integrity of the network depends on the existence of these "Special Archive Nodes", won't it be just as important to have them as it is to have regular nodes today?

Doesn't that mean that we effectively already have this?  Aren't you essentially just renaming "full nodes" as "Special Archive Nodes" and renaming pruning nodes as "Full Nodes"?

I don't know about the function of 'pruning nodes'.  Maybe we do already effectively have this.   I admit I haven't looked at it very closely.  But I imagine that somehow carrying around microtransactions for all of time is a bad idea when we finally get to 7K/transactions per second.  There has to be a mechanism to throw the old record away and start with a current balance sheet.  Just wondering.

[DannyHamilton]

Special Archive Nodes.  you know if the archive has been altered, because the hash is passed into the new chain.  Any change to the old chain would render that verification hash invalid.  If there are multiple versions of the archive, only the correct one will produce the right hash.  

How is a "Special Archive Node" any different than a regular node today?  Since the integrity of the network depends on the existence of these "Special Archive Nodes", won't it be just as important to have them as it is to have regular nodes today?

Doesn't that mean that we effectively already have this?  Aren't you essentially just renaming "full nodes" as "Special Archive Nodes" and renaming pruning nodes as "Full Nodes"?

I don't know about the function of 'pruning nodes'.  Maybe we do already effectively have this.   I admit I haven't looked at it very closely.  But I imagine that somehow carrying around microtransactions for all of time is a bad idea when we finally get to 7K/transactions per second.  There has to be a mechanism to throw the old record away and start with a current balance sheet.  Just wondering.

Nodes running Bitcoin Core (or any up to date variant) can set a "Prune" size, and they will only store recent blocks up to the indicated size they are willing to hold.

Meanwhile, Full Nodes (nodes that don't turn on Pruning) store the entire historical blockchain so that new nodes can synchronize and get caught up.

Satoshi already pointed out in the white paper that (as long as there are some Full Nodes that store the entire history), most clients can prune from their blockchain all transactions that have had all their outputs spent.  All that is needed for a full node to verify new transactions and blocks is the UTXO and the transactions that supply the UTXO.  The rest is only needed if you want to be able to look at old transactions or if you want to help new nodes get started.

[JPage]

Special Archive Nodes.  you know if the archive has been altered, because the hash is passed into the new chain.  Any change to the old chain would render that verification hash invalid.  If there are multiple versions of the archive, only the correct one will produce the right hash.  

How is a "Special Archive Node" any different than a regular node today?  Since the integrity of the network depends on the existence of these "Special Archive Nodes", won't it be just as important to have them as it is to have regular nodes today?

Doesn't that mean that we effectively already have this?  Aren't you essentially just renaming "full nodes" as "Special Archive Nodes" and renaming pruning nodes as "Full Nodes"?

I don't know about the function of 'pruning nodes'.  Maybe we do already effectively have this.   I admit I haven't looked at it very closely.  But I imagine that somehow carrying around microtransactions for all of time is a bad idea when we finally get to 7K/transactions per second.  There has to be a mechanism to throw the old record away and start with a current balance sheet.  Just wondering.

Nodes running Bitcoin Core (or any up to date variant) can set a "Prune" size, and they will only store recent blocks up to the indicated size they are willing to hold.

Meanwhile, Full Nodes (nodes that don't turn on Pruning) store the entire historical blockchain so that new nodes can synchronize and get caught up.

Satoshi already pointed out in the white paper that (as long as there are some Full Nodes that store the entire history), most clients can prune from their blockchain all transactions that have had all their outputs spent.  All that is needed for a full node to verify new transactions and blocks is the UTXO and the transactions that supply the UTXO.  The rest is only needed if you want to be able to look at old transactions or if you want to help new nodes get started.

Then why does anyone give a damn about blockchain bloat?  It seems like that system would still work fine even when the blockchain is 50,000TB.  I am sure we'd still have 10 full nodes even with a HUGE blockchain. 

[DannyHamilton]

Then why does anyone give a damn about blockchain bloat?  It seems like that system would still work fine even when the blockchain is 50,000TB.  I am sure we'd still have 10 full nodes even with a HUGE blockchain. 

People who are concerned about "blockchain bloat" really don't understand the technical details of bitcoin.  They try to run Bitcoin Core without pruning and then complain that they've run out of disk space.  They complain that it won't be possible for enough people to run a "FULL" node.

Note that they are at least partly correct.  As I mentioned, you'll always need some nodes to store the entire history.  You initially called them "Special Archive Nodes".  They are currently called "Full Nodes".  The point is, those same people would complain about "bloat" in your solution since (in their opinion) not enough people would be able to run "Special Archive Nodes".

The bigger issue with larger block sizes isn't the size of the blockchain, it's the size of the individual blocks themselves.  Before a miner can reliably build a new block on top of a recently solved block in the chain, they need to verify that the recently solved block is valid.  This means they have to take the time to download the entire block, and verify the block header and every transaction in the block.  Those miners or pools with slower internet connections or slower processors will be at a significant disadvantage over those with extremely high speed connections and high end processors.  This disadvantage could result in the higher end miners and pools getting a head start on the next block and solving more blocks than the disadvantaged miners and pools.  Eventually the loss of revenue could result in mining power consolidating in just the few miners or pools with access to the fastest possible internet connections and newest processors.

It also may become very difficult for those on very slow connections to download and verify a single block before the next block is solved.  As such, even lightweight wallets and pruning nodes would continuously fall farther and farther behind on synchronization and never be able to catch up.

The unanswered question is how big is too big when it comes to blocks?  Right now we are operating with 1 MB.  Is 2 MB too big?  How about 10 MB?  100 MB?  1 GB?  1 TB?  Who gets to choose, and how do we get EVERYONE to agree on that choice?

[TransaDox]

People who are concerned about "blockchain bloat" really don't understand the technical details of bitcoin.  They try to run Bitcoin Core without pruning and then complain that they've run out of disk space.  They complain that it won't be possible for enough people to run a "FULL" node.

The fact that pruning exists at all is an admission that block chain size matters. We do indeed complain that it is not possible to run a full node on average systems. Pruning doesn't solve the problem, it just kicks it down the road.

Note that they are at least partly correct.  As I mentioned, you'll always need some nodes to store the entire history. 

You need the network to store the entire history - preferably with redundancy. It is not necessary for all nodes to keep all blocks. The current system is the trivial solution, not the only solution.

The bigger issue with larger block sizes isn't the size of the blockchain, it's the size of the individual blocks themselves.  Before a miner can reliably build a new block on top of a recently solved block in the chain, they need to verify that the recently solved block is valid.  This means they have to take the time to download the entire block, and verify the block header and every transaction in the block. 

Oh. Boo hoo. Those poor miners starving on the streets might have to do a little more work to maximise their profits.  The difficulty will adjust to any more work required or they will just throw more hardware at it. This is a non-argument since they are just doing what miners do. Maybe more importantly, the miners will have to switch from maximising coin collection to maximising fees at some point anyway. This is a bait and switch from full nodes to miners (not the same thing at all) in an attempt to bolster the argument.

 Those miners or pools with slower internet connections or slower processors will be at a significant disadvantage over those with extremely high speed connections and high end processors.  This disadvantage could result in the higher end miners and pools getting a head start on the next block and solving more blocks than the disadvantaged miners and pools.  Eventually the loss of revenue could result in mining power consolidating in just the few miners or pools with access to the fastest possible internet connections and newest processors.

Oh You mean those not in data-centers? This is the argument against centralisation for the current mining regime  regardless of distribution mechanisms.

It also may become very difficult for those on very slow connections to download and verify a single block before the next block is solved.  As such, even lightweight wallets and pruning nodes would continuously fall farther and farther behind on synchronization and never be able to catch up.

Complete speculation. Even an Arduino could process a block in less than 10 minutes.

The unanswered question is how big is too big when it comes to blocks?  Right now we are operating with 1 MB.  Is 2 MB too big?  How about 10 MB?  100 MB?  1 GB?  1 TB?  Who gets to choose, and how do we get EVERYONE to agree on that choice?

No maximum limit. The trade-off between amount of fees (bigger blocks, more fees collected) and the competition to be first on the block chain (hashing efficiency and transmission) will yield an optimum block size depending on the miners' capabilities - no fixed block size! Of course. This destroys the fake scarcity introduced by the economists since all transactions can be catered for.

[Straux]

The thing is, even if one decided to prune, they would still have to download the whole blockchain, because core needs to verify.

If we follow what the OP is proposing, we don't necessarily need to download previous "chunks", as there is a Genesis block to verify from, like a checkpoint.

[Jet Cash]

Thanks for another great explanation Danny. You didn't mention the selfish miners who add a block to the chain without verifying the previous block. This can cause a bit of extra activity if the base block is invalid.

Pruned nodes are for people who want to run a full node, but don't want to dedicate their computer to Bitcoin. Don't forget that Bitcoin is great for people with a mobile lifestyle, and they may not want to add an external hard drive to their notebook computer.

[JPage]

The thing is, even if one decided to prune, they would still have to download the whole blockchain, because core needs to verify.

If we follow what the OP is proposing, we don't necessarily need to download previous "chunks", as there is a Genesis block to verify from, like a checkpoint.

Exactly.  See, this guy gets it.  It's like starting with a premined genesis block where the distribution is from the UTXO of the old chain.  Verification happens because anyone can hash the original state to assure there were no changes - so the data is perfect. 

[Itty Bitty]

How about changing wallets so that they generate 2 public and private addresses at the same time, with the second(ary) public/private address linked to the first. When some pre-determined block is reached, the coins from the first public/private pair are automatically transacted to the second pair. This way all all coins in old blocks are now in newer blocks, and all old blocks can be pruned.

Practically possible? I am certain not. But discuss amongst yourselves anyway if you want.

[Jet Cash]

I'm running a full node on a Core i5 notebook, and I connect using McDonalds free WiFi whick is 30Kb/s at the moment. This is part of my project for people adopting a mobile lifestyle. I don't have any problems, and running core in the background doesn't affect my domain name dealing business, or other activities.

Well I do have one problem - I'm drinking too much coffee.

[jasonv75]

Reaching consensus on this "event" won't be an easy task, if not impossible.
And if you get rid of transaction history periodically, what argument is left to get rid of all transactions all together and only rely on balance amounts?

J.V.

[tsoPANos]

OP, your idea is not new.
There have been many proposals like this.
https://bitcointalk.org/index.php?topic=88208.0
https://www.scribd.com/document/317130737/Bitcoin-On-Chain-Pruning

The downside is that this has the same security assumptions as in SPV wallets.
I think it is worth it though.