Proposal for solving blockchain size issue. Managing a 500GB blockchain

[NyeFe]

The problem with the blockchain, is that its ever increasing, as we all know. Many attempts have been made to solve this issue, though with old fashioned thinking.

Our proposal, though simple for some, is to divide the blockchain into packets (instead of sharing the whole blockchain) by 5% of total online nodes, and having all these packets shared across all nodes online on the network. So i. E. If the blockchain is 30GB, and was shared by 5% to any number of nodes, 20 (+-) copies of the blockchain should be available in the whole blockchain, whilst consuming about 1.5GB per node.

When a packet is less available than other packets, nodes with much more common packets would drop those, to store those less available packets (so to stop them from disappearing from the network)

We've included a formula to further highlight the advantage of this technique.

A = unique copies made
N = online nodes
C = data per node
b = size of data
r = required nodes





As you can see, a size of a 500GB blockchain would only occupy 4GB per node, to replicate itself more than 20(+-) times in any given N (2,500) number of nodes, where each full blockchain would occupy 125(+-) nodes.

[Jet Cash]

As I understand it, the problem is not the size of the blockchain, but the size of the blocks. Pruning allows people who don't want to store the whole blockchain. I guess one could think of a block as a Bitcoin "packet".

[NyeFe]

As I understand it, the problem is not the size of the blockchain, but the size of the blocks. Pruning allows people who don't want to store the whole blockchain. I guess one could think of a block as a Bitcoin "packet".

Thank you for your reply. There's various problems: size of blocks, the increasing blockchain, proning. The later forces a centralisation of bitcoins blockchain because only the necessary amount needed for an application is used.

The purpose of this post is to ensure that the people always have the collected capability and computer resources to manage a blockchain of any arbitrary size.

Furthermore, the use of some words in this post is merely a coincidence. A "packet" is used to describe a full blockchain, which has been divided by 5% of the online nodes (data packet).

[bob123]

To be honestly i dont think a 500GB Blockchain should be a problem in the near future.
SSD's will get pretty cheap.

[rikrikrik]

but that means a few problems, to have a wallet you need a 500GB ssd

and with growth 500GB will be not enough.

sure your idea just sounds like a raid5+0^20

[af_newbie]

To be honestly i dont think a 500GB Blockchain should be a problem in the near future.
SSD's will get pretty cheap.

The growth is not linear. I've seen 15 fold increase in few years.  Speeding up recently.

People who advocate large block size ignore the facts.

Blockchain growth is like human population growth.  Give it resources and means to grow, and the size will explode.

Limiting the growth below a magic number is the key to a long term survival of both.

[Furio]

To be honestly i dont think a 500GB Blockchain should be a problem in the near future.
SSD's will get pretty cheap.

There should be a way to compirimise the data once it has been written to the blockchain, this will be implemented in the coming years IMO

[TierNolan]

There are different node requirements for different types of users.

Validating Node

This node completely validates the block chain.  This means that it need to process every block, in full, but then can throw away the blocks once they have been processed. 

Pruning nodes do this and they keep at least 2 days of blocks (288+).

They also need to keep the active UTXO set.  1-2GB of disk space seems sufficient for a node of this type (assuming 1MB blocks).

You can mine with this node.  It is hard to see how to split this node into pieces.  When a miner receives a transaction, the miner has to be able to check if the inputs are valid.  That means that it needs the entire UTXO set.

The storage required for this node depend on the UTXO set size.  This can be artificially increased by spamming the blockchain with unspendable outputs (and not using OP_RETURN).

With UTXO set commitments, it might be possible to reduce the size of storage on each node.  Blocks could be validated, even without the UTXO set.  However, creating blocks would still likely require it.   This would split validating nodes, in validating nodes and block creation node.  The block creation nodes would have to have all the UTXO set info, in order to actually create the UTXO set commitments.  The validating nodes then become something closer to auditing nodes.

Archive Node

This node keeps the entire block chain.  It doesn't actually need to validate anything.  It just needs to find the longest chain and then download and store the blocks for that chain.

This type of node could be easily partitioned.  As long as at least a few node stores each block block. 

For efficiency, the node should keep blocks in runs of blocks.  That means that it can send many blocks, in order starting from a particular height.  This also makes it easier to say what blocks it has.

Auditing Node

This is a node which checks that blocks are valid.  It doesn't have to be as fast and up to date as a fully validating node.  If it finds a problem, it can create a fraud proof, and then the network will blacklist that block. 

A node like this could store 1% of the UTXO set and check any transaction which tries to spend any of those inputs.  This could work like Bloom filters, and the audit node could ask for the 1% of transactions that spend to/from that portion of the set.

Auditing nodes should concentrate on the newest blocks, but do random tests for older blocks.

SPV Node

This node needs very little resources.  It needs to be able to store the header chain (80 bytes per block, no matter what the block size). 

In the future, it should also be able to process fraud proofs, which blacklist invalid blocks.  This gives it security that nearly matches a full validating node (as long as fully validating and auditing nodes actually send out fraud proofs).

An SPV node on a phone could also do some auditing.  If the phone was on a wi-fi connection and at > 95% charge, auditing would help keep the miner's fully validating nodes honest.  There could a slider indicating what percentage to check.

[Jet Cash]

I could handle a 1Tb blockchain on this computer which cost under £400. The easy way to get more efficiency into blockchain storage and processing would be the SegWit solution together with another option which I have realised recently. That would be to strip out all non-coin transfer messages, and store them elsewhere.

One of the problems with great ideas is that people try to steal resources from them, and use them for additional purposes. One example is WiFi. What a great idea - allow businessmen and others who are away from a cable connection to connect to the net. So now we see parasite projects appearing. One example is the mobile phone charger. You buy a charger unit and connect your phone to it. It then steals electricity from the WiFi supplier, and reduces signal quality for the genuine users.

[whizz94]

I'm not too keen on this, as then no node would contain more than 5% of the future huge blockchain, so no single copy for fast blockchain.info searches would exist.  If a blockchain.info search were launched, it would be [the slowest of] 19 latencies, so certainly slow and ponderous.

How about instead, using BTC as the "gold bullion bars" of online cryptocurrency and keep the majority 95%+ of transactions for goods and services on altcoins and sidechains whose sidechainblockchain fits the size criteria of the thread author?  For example <25GB of regional sidechainblockchain with occasional "international" BTC transactions using a not-too-huge BTC blockchain.  It would help matters if some of the speculator exchanges did not clutter up the BTC blockchain with unproductive speculator trades.

What do people think?

[abs350]

Well you guys have no idea what you're talking about.

Remember the days when the fastest computer was 1Hz and the entire operating system was 10kb ?

You know that was only 30 years ago. (~1980s)

Believe me in 30 years from now 100GB will be the equivalent of 10kb in the 1980s. There will be a radical new way of storing data like maybe DNA, QBits or something else that can hold 10000x the amount of normal storage, like maybe 1000 PB or more on something the size of a SD card.

So when you say that the blockchain will just grow too large... you have no idea what you are talking about.

The technology for storage will grow much faster than the blockchain can. Already in just the last few years storage has got way cheaper, these days you can pick up a 4TB drive for $100 that was not the case back in 2009.

Even if every computer in the world was spamming the blockchain non stop it for the next decade the blockchain would still not get too big.

[hexitor]

Well you guys have no idea what you're talking about.

Remember the days when the fastest computer was 1Hz and the entire operating system was 10kb ?

You know that was only 30 years ago. (~1980s)

Believe me in 30 years from now 100GB will be the equivalent of 10kb in the 1980s. There will be a radical new way of storing data like maybe DNA, QBits or something else that can hold 10000x the amount of normal storage, like maybe 1000 PB or more on something the size of a SD card.

So when you say that the blockchain will just grow too large... you have no idea what you are talking about.

The technology for storage will grow much faster than the blockchain can. Already in just the last few years storage has got way cheaper, these days you can pick up a 4TB drive for $100 that was not the case back in 2009.

Even if every computer in the world was spamming the blockchain non stop it for the next decade the blockchain would still not get too big.

I fully agree with you. In 30 years, so many advances will have been made in the storage field that storage capacity will no longer be a problem. However, be sure that if the storage capacities increase, the blockchain will do the same, and thus will have the same weight has today has 60GB.

[jl777]

There are different node requirements for different types of users.

Validating Node

This node completely validates the block chain.  This means that it need to process every block, in full, but then can throw away the blocks once they have been processed. 

Pruning nodes do this and they keep at least 2 days of blocks (288+).

They also need to keep the active UTXO set.  1-2GB of disk space seems sufficient for a node of this type (assuming 1MB blocks).

You can mine with this node.  It is hard to see how to split this node into pieces.  When a miner receives a transaction, the miner has to be able to check if the inputs are valid.  That means that it needs the entire UTXO set.

The storage required for this node depend on the UTXO set size.  This can be artificially increased by spamming the blockchain with unspendable outputs (and not using OP_RETURN).

With UTXO set commitments, it might be possible to reduce the size of storage on each node.  Blocks could be validated, even without the UTXO set.  However, creating blocks would still likely require it.   This would split validating nodes, in validating nodes and block creation node.  The block creation nodes would have to have all the UTXO set info, in order to actually create the UTXO set commitments.  The validating nodes then become something closer to auditing nodes.

Archive Node

This node keeps the entire block chain.  It doesn't actually need to validate anything.  It just needs to find the longest chain and then download and store the blocks for that chain.

This type of node could be easily partitioned.  As long as at least a few node stores each block block. 

For efficiency, the node should keep blocks in runs of blocks.  That means that it can send many blocks, in order starting from a particular height.  This also makes it easier to say what blocks it has.

Auditing Node

This is a node which checks that blocks are valid.  It doesn't have to be as fast and up to date as a fully validating node.  If it finds a problem, it can create a fraud proof, and then the network will blacklist that block. 

A node like this could store 1% of the UTXO set and check any transaction which tries to spend any of those inputs.  This could work like Bloom filters, and the audit node could ask for the 1% of transactions that spend to/from that portion of the set.

Auditing nodes should concentrate on the newest blocks, but do random tests for older blocks.

SPV Node

This node needs very little resources.  It needs to be able to store the header chain (80 bytes per block, no matter what the block size). 

In the future, it should also be able to process fraud proofs, which blacklist invalid blocks.  This gives it security that nearly matches a full validating node (as long as fully validating and auditing nodes actually send out fraud proofs).

An SPV node on a phone could also do some auditing.  If the phone was on a wi-fi connection and at > 95% charge, auditing would help keep the miner's fully validating nodes honest.  There could a slider indicating what percentage to check.

This is very close to what I have in mind. I think "archival node" can be further simplified to just be any decentralized DHT storage.

iguana processes the blocks into bundles of 2000 blocks worth of read only data. once it is created, it doesnt change and it can also be fully verified. So make a hash of this bundle, push it to the cloud.

Separately, make a list of these hashes and sync the network with this list. Now this also simplifies the initial blockchain sync, as given the list of bundlehashes, you can parallel load the blockchain and get <30 minutes initial sync times.

James

[johnyj]

Well you guys have no idea what you're talking about.

Remember the days when the fastest computer was 1Hz and the entire operating system was 10kb ?

You know that was only 30 years ago. (~1980s)

Believe me in 30 years from now 100GB will be the equivalent of 10kb in the 1980s. There will be a radical new way of storing data like maybe DNA, QBits or something else that can hold 10000x the amount of normal storage, like maybe 1000 PB or more on something the size of a SD card.

So when you say that the blockchain will just grow too large... you have no idea what you are talking about.

The technology for storage will grow much faster than the blockchain can. Already in just the last few years storage has got way cheaper, these days you can pick up a 4TB drive for $100 that was not the case back in 2009.

Even if every computer in the world was spamming the blockchain non stop it for the next decade the blockchain would still not get too big.

I fully agree with you. In 30 years, so many advances will have been made in the storage field that storage capacity will no longer be a problem. However, be sure that if the storage capacities increase, the blockchain will do the same, and thus will have the same weight has today has 60GB.

Exactly, another benefit is that added demand for bandwidth and components from bitcoin is very similar to added demand from high end gaming industry, it will create new jobs and new technology, thus is good for economy in general

[Jet Cash]

The added bandwidth demand comes from Microsoft, Google and other clouds, and from them copying and storing our data for the US government. A bit of extra traffic for a relatively small application like Bitcoin won't make much difference.

[hexitor]

Satoshi was clever when he created the blockchain. I'm sure that he calculated with Moore's law (I think that's the name, but I'm not fully sure) that the blockchain size will always be a small amount of data by taking in account that the hardware's capacity increases too.