Proposed system to reduce Blockchain size

[pawanjain]

To begin with, I don't have deeper technical knowledge about the blockchain as many others here have. Which is why I would like to get opinion for a solution which I think might be possible to reduce the blockchain size.
Again, I am no expert and just expressing my views to get some better reviews and understanding of the blockchain.

Considering that the size of blockchain has already crossed 200GB+ I think the simple solution here would be to archive this data from the Blockchain and store it on some particular locations from where the archival is easy to access.

Let us consider that we have archived all the blocks until 200GB and have stored on some secure place. Also, we have attached a hash/signature to the archive and modified the consensus such that every new block that is generated includes a hash/signature from the archive. So now instead of validating all the previous blocks, the miner will validate only the hashes of archived blockchain, previous block and the current block.

This will save the 200GB of data that a miner has to validate and reduce the size of the blockchain. I know there must be some flaws in this system. Let me know in the comments.

[AB de Royse777]

~snip~

Let us consider that we have archived all the blocks until 200GB and have stored on some secure place. Also, we have attached a hash/signature to the archive and modified the consensus such that every new block that is generated includes a hash/signature from the archive. So now instead of validating all the previous blocks, the miner will validate only the hashes of archived blockchain, previous block and the current block.

~snip~

This might a good useful idea but I have no idea how it can be achieved. I have the same problem like you. I am no expert in the Blockchain technology. :-(

[khaled0111]

What you are suggesting was proposed and discussed before, actually.
And the answer is no, this is not going to work nor solve the blockchain size problem.

First, if we are going to "store" an archived copy of the blockchain in a particular place then it can't be considered decentralized nor trustless system anymore.

Second, each block is cryptographically linked to the previous block. Thus, if we need to validate a block/transaction a full node has to go all the way back to the genesis block.

=> we need full nodes, and we need more of them.

[d5000]

Apart from what khaled0111 already wrote, disk space is not the main bottleneck. 200 GB are not much, it's almost nothing for modern HDDs.

The main bottleneck is the propagation of new blocks and new transactions.

You can already prune your blockchain to less than 1 GB (AFAIK 550 MB was the minimum).

There was a proposal in 2014 called the "Mini-blockchain scheme", which is something very similar to what you're proposing. It has changed some rules to the code so that the fraction of the blockchain which has to be validated by full nodes is much shorter than in the current codebase, and from older blocks only the headers are stored. The proposal, however, hasn't got acceptance in the Bitcoin community, but there are altcoins using the proposed modifications, or variations of it.

Among the problems with this approach are:
- an 51% attack could not only double spend, but replace all complete blocks of the mini-blockchain. A "checkpoint" (issued in the code) would be necessary to make it usable again (see here)
- there is no way to validate most contracts, LN, for example, would be probably impossible.

[vit05]

=> we need full nodes, and we need more of them.

Do we really need more? Is there a statistical study showing which is the ideal and necessary number? Perhaps we need much more of a diversity of locality where there are full nodes than necessarily a much larger number of them.

[khaled0111]

Do we really need more?

My answer: "yes, we do"
Now, you have to believe me as I am the only one who answered, right!

What if someone else joins the conversation and replies: "no, we dont" ?
In this case you need answers from as many members as possible to know who is telling the truth.

The same applies to blockchain. Blockchain is more secure with more full nodes.
Also, full nodes make sure that miners are following the consensus.

Apart from that, it is always advised to run your own full node so you don't have to rely on other nodes.

It is worth mentioning that a full node is helping the network only if it accepts inbound connections.

[Coding Enthusiast]

If the size of the blockchain is the only concern in this discussion then instead of focusing on storing it elsewhere (which sounds more like centralization) focus on bitcoin transactions at byte level and see how you could "compress" them instead. FYI inside a transaction there are lots of wasted bytes that don't need to even exist. For your storage and even communication over P2P network you can come up with a compression technique that reduces the size by a lot.

I have actually been working on this recently and been able to "compress" a transaction up to 15%. That's just a start though, it needs a lot more work and study of other people's work to see what they have done, but you get the idea. With a block this can be more by simply dropping a lot of things and adding a "contract" in your compressor based on some factors such as block height. I expect the final result to be 30% compression.

[pawanjain]

What you are suggesting was proposed and discussed before, actually.
And the answer is no, this is not going to work nor solve the blockchain size problem.

First, if we are going to "store" an archived copy of the blockchain in a particular place then it can't be considered decentralized nor trustless system anymore.

I kind of knew this question would arrive and I was prepared for the same. The thing is we would not store it on one particular location but a group os 'trusted' locations.
Example: If we identify 50 or more trusted locations and store the archival there then it shouldn't be called centralized as the archival is still spread across trusted locations.

Second, each block is cryptographically linked to the previous block. Thus, if we need to validate a block/transaction a full node has to go all the way back to the genesis block.

=> we need full nodes, and we need more of them.

So now instead of validating all the previous blocks, the miner will validate only the hashes of archived blockchain, previous block and the current block.

There is no need to validate all the previous blocks until the genesis block. Only the hash of the archival and the hashes of the previous blocks until the archival would be sufficient.

For example: The blocks until 200GB are archived and from the next block(first block after the archival) it would be considered as second genesis block or genesis block after archival.
So we would need to validate only blocks until second genesis block and not all the blocks until the real genesis blocks.

You can already prune your blockchain to less than 1 GB (AFAIK 550 MB was the minimum).

Yes, pruned blockchain does sort out the issue of storing the whole blockchain on your system however it still has to validate all the blocks until the genesis block.
Whereas my proposed system would eliminate the need to validate all the blocks until the genesis one and will only need to validate blocks until second genesis block/genesis block after archival whatever we name it.

If the size of the blockchain is the only concern in this discussion then instead of focusing on storing it elsewhere (which sounds more like centralization) focus on bitcoin transactions at byte level and see how you could "compress" them instead. FYI inside a transaction there are lots of wasted bytes that don't need to even exist. For your storage and even communication over P2P network you can come up with a compression technique that reduces the size by a lot.

I have actually been working on this recently and been able to "compress" a transaction up to 15%. That's just a start though, it needs a lot more work and study of other people's work to see what they have done, but you get the idea. With a block this can be more by simply dropping a lot of things and adding a "contract" in your compressor based on some factors such as block height. I expect the final result to be 30% compression.

Well that's a nice thing to start with. I haven't thought about it yet henceforth I can't comment on it. It's good that you have reached 15% compression and I hope you achieve more so that we all could get better at it.

[ranochigo]

I kind of knew this question would arrive and I was prepared for the same. The thing is we would not store it on one particular location but a group os 'trusted' locations.
Example: If we identify 50 or more trusted locations and store the archival there then it shouldn't be called centralized as the archival is still spread across trusted locations.

The whole point of Bitcoin is for it to be trustless. Someone still has to manage the files and everyone's definition of who is trustable is different. Who should be the one assigning the trusted people? It's fairly easy for that person to push an agenda through and manipulate the entire system. If I needed to trust someone, I might as well go for Paypal.

There is no need to validate all the previous blocks until the genesis block. Only the hash of the archival and the hashes of the previous blocks until the archival would be sufficient.
For example: The blocks until 200GB are archived and from the next block(first block after the archival) it would be considered as second genesis block or genesis block after archival.
So we would need to validate only blocks until second genesis block and not all the blocks until the real genesis blocks.

How would you get the chainstate without indexing the blockchain? Trusting another person to obtain the chainstate is considered fairly unsafe as the information inside can be manipulated. You need the entire blockchain to ensure the validity of the chain. If not, you might as well just use an SPV client.

Yes, pruned blockchain does sort out the issue of storing the whole blockchain on your system however it still has to validate all the blocks until the genesis block.
Whereas my proposed system would eliminate the need to validate all the blocks until the genesis one and will only need to validate blocks until second genesis block/genesis block after archival whatever we name it.

SPV client would be a better alternative. If you don't validate the blocks fully, you are essentially an SPV client and you don't have to download any blocks at all.

[franky1]

issues:
if you want to be a full node. then take the big boy pants, put them on and be a full node. if your just gonna leach the blockdata from someone else and then strip signatures, prune and filter to be left with pretty much just utxo's your no longer a full node. no longer able to seed the full data to others.
so just give up starting as a full node if next week your going to be a useless litewallet.
make the choice be a full node or a lite wallet.

solutions:
the main concern for full nodes is not the physical space. nor the actual time to download the full sync. but the fact that the main reference software is coded that you cant even see a estimated balance or send out a transaction until the sync is complete. thus making the sync appear slow because your sat there doing nothing twiddling your thumbs waiting for it.

but, you dont actually have to wait. so the solution is simple.
instead of starting as full node leaching off others to then downgrade to spv/litewallet. wasting others resources. do the opposite
get a UTXO set first to get users active and making transactions with a 'independently unverified balance' as a spv/lite wallet which is still safe because if your personal utxo is spent the network wont allow it to relay/confirm in a new block. then. the syncing to get the full archive and verify all transactions becomes a background task, thus no twiddling idle thumbs.

tl:dr
dont start as full and downgrade to spv
start as spv and upgrade to full

[keychainX]

To begin with, I don't have deeper technical knowledge about the blockchain as many others here have. Which is why I would like to get opinion for a solution which I think might be possible to reduce the blockchain size.
Again, I am no expert and just expressing my views to get some better reviews and understanding of the blockchain.

Considering that the size of blockchain has already crossed 200GB+ I think the simple solution here would be to archive this data from the Blockchain and store it on some particular locations from where the archival is easy to access.

Let us consider that we have archived all the blocks until 200GB and have stored on some secure place. Also, we have attached a hash/signature to the archive and modified the consensus such that every new block that is generated includes a hash/signature from the archive. So now instead of validating all the previous blocks, the miner will validate only the hashes of archived blockchain, previous block and the current block.

This will save the 200GB of data that a miner has to validate and reduce the size of the blockchain. I know there must be some flaws in this system. Let me know in the comments.

well if core developers inteoduced schnorr signatures you would save 25% space

/kx

[Wind_FURY]

=> we need full nodes, and we need more of them.

Do we really need more? Is there a statistical study showing which is the ideal and necessary number? Perhaps we need much more of a diversity of locality where there are full nodes than necessarily a much larger number of them.

As a security measure, I believe it's better to have more than it is to have less full nodes in the network. Because if there was a change in the network that causes some people not to run full nodes due to higher costs, the network will centralize by some "number". We want the contrary that, more decentralized = more security.

[NeuroticFish]

If the size of the blockchain is the only concern in this discussion then instead of focusing on storing it elsewhere (which sounds more like centralization) focus on bitcoin transactions at byte level and see how you could "compress" them instead. FYI inside a transaction there are lots of wasted bytes that don't need to even exist. For your storage and even communication over P2P network you can come up with a compression technique that reduces the size by a lot.

I have actually been working on this recently and been able to "compress" a transaction up to 15%. That's just a start though, it needs a lot more work and study of other people's work to see what they have done, but you get the idea. With a block this can be more by simply dropping a lot of things and adding a "contract" in your compressor based on some factors such as block height. I expect the final result to be 30% compression.

I like this. And the beauty of it is that it can be possible without altering the protocol; each and every client stores the data in the way it prefers or can, the useful data is the same.
But I fear that this will not solve much. Size-related discussions are here since the chain data was under 100GB. And I feel like the complaining ones are the newcomers that have to do the initial setup and download everything.

And then Franky's idea becomes even more interesting.

start as spv and upgrade to full

Since many use the Core wallet only because it's the official one and also praised to be the safest one, it could have the option - which could be selected by hand or asked whenever the wallet is more than a couple of blocks out of sync - to run as a SPV wallet (with reduced functionality, compared to the full wallet, obviously)
The only problem here is that it has to be implemented very well to avoid issues like Electrum had and also make it clearly visible what mode is currently running.

[Coding Enthusiast]

well if core developers inteoduced schnorr signatures you would save 25% space

No you won't. This topic is about the size of "blockchain" not size of signatures or transactions. Since block size will remain the same, the blockchain size will remain the same and grow at the same rate as long as new blocks are full.

I like this. And the beauty of it is that it can be possible without altering the protocol; each and every client stores the data in the way it prefers or can, the useful data is the same.

I've been trying to finish it up and release it but other things keep coming up and I sidetrack.

[franky1]

=> we need full nodes, and we need more of them.

Do we really need more? Is there a statistical study showing which is the ideal and necessary number? Perhaps we need much more of a diversity of locality where there are full nodes than necessarily a much larger number of them.

As a security measure, I believe it's better to have more than it is to have less full nodes in the network. Because if there was a change in the network that causes some people not to run full nodes due to higher costs, the network will centralize by some "number". We want the contrary that, more decentralized = more security.

knowing the network is centralised code to the "reference client" it does not matter if its 100 nodes or 100,000 nodes. if there is a bug in the code its not going to be limited to a single user. it will affect everyone running the code.

if you are talking about DISTRIBUTION rather than decentralisation. then having a node connect to 5 other nodes concentrates the bandwidth on 5 nodes. thus faster speed/propagation. but if the node is connected to 100 other nodes than the speed/bandwidth is diluted by 20x making each connection slower and propagation worse. so sometime distribution especially if nodes foolishly then strip/prune/filter data make it so theres more leachers than seeders

having some non-interested basement user that isnt really interested in bitcoin personally but has the naive assumption that they NEED to push their computer and bandwidth to the limit purely assuming it 'helps the network' is not what a decentralised network should be veiwed as.

wats better is for those people who NEED to verify more than a few transactions a day, have more reason to be a full node. and it becomes in their own benefit of needing to verify that would self motivate them to want to invest in self verifying.

there is no need to stifle bitcoin to make average uninterested joe into a full node. its just better to let nature do its thing, better to have 100,000 businesses and regular users be full nodes because their self motivated by the need to verify.. rather than 100,000 randomers thinking they should be full nodes because 'fear of network security'

[Wind_FURY]

=> we need full nodes, and we need more of them.

Do we really need more? Is there a statistical study showing which is the ideal and necessary number? Perhaps we need much more of a diversity of locality where there are full nodes than necessarily a much larger number of them.

As a security measure, I believe it's better to have more than it is to have less full nodes in the network. Because if there was a change in the network that causes some people not to run full nodes due to higher costs, the network will centralize by some "number". We want the contrary that, more decentralized = more security.

knowing the network is centralised code to the "reference client" it does not matter if its 100 nodes or 100,000 nodes. if there is a bug in the code its not going to be limited to a single user. it will affect everyone running the code.

You know well what I was talking about, franky1. Stop bringing your social drama in all of the discussions.

having some non-interested basement user that isnt really interested in bitcoin personally but has the naive assumption that they NEED to push their computer and bandwidth to the limit purely assuming it 'helps the network' is not what a decentralised network should be veiwed as.

Those hat wearing basement dwellers were successful in pushing for Segwit's activation. Remember UASF.

wats better is for those people who NEED to verify more than a few transactions a day, have more reason to be a full node. and it becomes in their own benefit of needing to verify that would self motivate them to want to invest in self verifying.

But it shouldn't be that the ability to run a full node should be taken away from anyone.

there is no need to stifle bitcoin to make average uninterested joe into a full node. its just better to let nature do its thing, better to have 100,000 businesses and regular users be full nodes because their self motivated by the need to verify.. rather than 100,000 randomers thinking they should be full nodes because 'fear of network security'

Scaling out Bitcoin is stifling?

Plus why should it matter? 100,000 nodes are 100,000 nodes from the network's standpoint, whether they are owned by ordinary users or merchants, or miners. They all validate to make sure every transactions and blocks follow the rules.

[bitcratic]

Validate historic transactions and ensure nobody has violated the Bitcoin Protocol. However, there are proposals for more efficient ways to transmit and store the information currently in the block chain.
total amount of information in the block chain can't be reduced without making it impossible for new nodes.

[ranochigo]

As a security measure, I believe it's better to have more than it is to have less full nodes in the network. Because if there was a change in the network that causes some people not to run full nodes due to higher costs, the network will centralize by some "number". We want the contrary that, more decentralized = more security.

I doubt there would be a difference there. At least without the change, the network would be decentralised.

That would likely occur only if there is some form of protocol rule change which somehow results in higher costs for those having nodes. In this case, there would likely be fairly little support from the community and the fork wouldn't even happen in the first place. If it is just a modification to the reference client that doesn't affect Bitcoin at a protocol level, people would be fine running the older nodes which wouldn't matter.

[franky1]

here is a lesson
by only limiting bitcoin to ~3ktx a block =500k a day
means without day traders(spammers) if people only transacted 1tx a day. thats 500k users needing bitcoin daily.
(with day traders/spammers, the numbers of users is less)

forget imagining millions of users wanting to b full nodes. because your not even going to get 500k users wanting to be full nodes

infact if we said 50k transactions were from people doing 5tx a day(10k users) and 450k were from 1tx a day.. psychologically the stats would presume there is only 10k users that NEED to be full nodes because they are handling more than 1 tx day to want to be more involved.

after all who would want to run a full node when they are not even using bitcoin.
..
people need to think about the more they deburden users actually using the blockchain.. the number of people wanting to secure the blockchain diminishes too

[Wind_FURY]

forget imagining millions of users wanting to b full nodes. because your not even going to get 500k users wanting to be full nodes

Maybe not, but it's not a good design-decision for a decentralized network if you have made it to discourage users from running their own full nodes.

Do you want a real decentralized network, or a network centralized towards the miners?