Blockchain is 360GB and growing, can it be consolidated?

[deadlift75]

(Please forgive my lack of specific technical knowledge, my question is likely dumb, so help me understand why)

If the blockchain is a chain of transactions, and its purpose (among others) seems to be to avoid double spending, why can't it be regularly consolidated into a state condition (as in, "at this time, wallet1 has N bitcoins", "at this time, wallet2 has N bitcoins", etc. or however that information would be stored in the same/new blockchain) to save on space?

This consolidation would obviously have to be proven to be consistent with previous transaction history, but once it's done, it would be accepted and added to the blockchain.

Once a state is accepted and consolidated into the blockchain, I assume the entire history of transactions before that state could be discarded for most purposes.

If this consolidation happened, let's say, once a month, and only the record of transactions from the previous month and the previous state was kept, transactions would only need to be checked against this relatively small data set.

Does this make sense? What problems would arise from this that I'm not seeing? Thanks!

[bitmover]

You are basically talking about a pruned node.

It would take about 5GB only, according this post
https://bitcointalk.org/index.php?topic=5213552.0

Prune node
Pros:

• Required about 5 GB by default (a little it more than 5 GB). ^[a]
• You don't have store all blockchain database on your computer
• You can run a prune node to get experience and pratice with Bitcoin core without significant pressure on your computer storage space.

^[a]: In reality, you will prepare at least a little more than 5GB for your prune node with a minimum storage value for blocks (at 2 GB). They are included by:

• Chainstate: 3.5 GB
• Blocks: 2 GB
• Initial setup: 52.1 MB (see the below image)

They are safe to use. The user basically download the whole blockchain, but compress most of the data, so it doesn't occupy so much disk space.

You still need the same bandwidth

[BlackHatCoiner]

Short answer:  Because, that would force you to trust the system.

Long answer:  You may have heard the term “Don't trust, verify” as it's said often in social media; it has a lot of point. Each block with all those transactions is valid, because your node verifies it. Because your node has checked that once you hash the content, you do, indeed, get a valid proof of work. This isn't something insignificant to just pass.

Once a person spends bitcoins, you don't trust the others that they've supposedly verified (and now discarded) the old blocks that contained their transactions. You don't trust anybody; you, instead, verify it yourself since the first block that this very guy has had that many bitcoins.

To give a simplified example:  Say you wanted to run a node when the chain height is 100,000 and the network, by consensus, discards any block after the 1,000th most recent one. Once your node synced, you'd have verified that the transactions in the last 1,000 blocks are valid, but you'd have to trust that the rest nodes verified the other 99,000. You see the problem?

[deadlift75]

Hi, bitmover, thanks! I haven't heard of pruned nodes, I like the term "chainstate".

My question now is why does the network still need full nodes, I mean what's being checked against the full nodes that can't be checked or trusted on the information of the pruned nodes?

[deadlift75]

BlackHatCoiner, very good point about "trust but verify", I agree.

However, what if this consolidation happened every month or year, leaving everybody plenty of time to publicly verify previous states and transactions?

Regarding your example, I see why syncing the node with a network that limits itself to the last 1000 transactions would be a problem, but since past nodes never change, can't a verification on past nodes produce some publicly verifiable hash that can just be checked and everybody can just move on?

[BlackHatCoiner]

My question now is why does the network still need full nodes

For the reason I told you. Due to the need for trust.

However, what if this consolidation happened every month or year, leaving everybody plenty of time to publicly verify previous states and transactions?

There'll never be enough time. Once the consolidation sets sail, every other person who will want to find out the previous transactions by running a node, will have to trust the other nodes.

I see why syncing the node with a network that limits itself to the last 1000 transactions

I said 1,000 blocks, but anyway. Give or take whatever you want, it's a similar analogy.

but since past nodes never change, can't a verification on past nodes produce some publicly verifiable hash that can just be checked and everybody can just move on?

Yeah, of course they can produce a hash. The problem is that you can't verify that hash if you don't have the content of the block. I may give you the following hash:

Code:

000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f

But, if you don't have the whole thing, which is this, how can you confirm that I've, indeed, found a valid proof of work and not written those hexadecimal characters by myself with zero effort?

[deadlift75]

The problem is that you can't verify that hash if you don't have the content of the block. I may give you the following hash:

I see your point. I agree with you that trust would be less self-verifiable and delegated to an extent. The idea is that the hash would be verified against previous transactions up to the previous state, but the problem remains: how to verify the previous state?

Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise". By the same logic, how likely can the entire network be fooled into accepting a wrong but publicly verifiable state for a month or a year?

[n0nce]

How to verify the previous state?

See, without such a consolidation and by instead verifying the whole blockchain, block by block starting from the genesis block, there is no previous state to verify or anything like that. That's why it's really the best way to do things. Also there have been ideas like yours in the past, such as inserting 'sync blocks' into the blockchain periodically, but all these changes even if just slightly, decrease the security, meanwhile the blockchain only grows slowly and storage gets cheaper and cheaper.

The motivation to change anything is really not strong enough. During Black Friday, there were 1TB SSDs for <80€, and it gets cheaper with HDDs. 1TB will last like 10 years-ish, if I remember correctly. I can save up another 80 bucks during 10 years if I'll then need to get a second drive lol.

Also, one thing you got wrong in your initial post:

"at this time, wallet1 has N bitcoins"

Bitcoin is not balance-based, it's UTXO-based. This makes some things better, some things harder.

[DaveF]

Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise". By the same logic, how likely can the entire network be fooled into accepting a wrong but publicly verifiable state for a month or a year?

The entire network does not matter, it's if A node can be fooled into accepting it.
And it's just not worth the risk.

This conversation keeps coming up on the forum about "ways to shrink the blockchain" and the answer is you can't. If you want to verify everything then you need everything.
If you don't want to verify everything, then there are plenty of light wallets that you don't need to download the entire blockchain for.

n0nce even started a thread on how you can run a full node for under $50
https://bitcointalk.org/index.php?topic=5364742.0


-Dave

[HeRetiK]

In addition to what the others already said, keep in mind that Bitcoin is also about choice. With the way it works now, it's completely up to you how much you want to trust and how much additional effort you want to expend to stay trustless:

Want to remain trustless and are willing to keep some extra HD space for easier re-syncing if something goes wrong? Use a full node.

Want to remain trustless but don't want to use more HD space than necessary? Use a pruned node.

Want to spend as few resources as possible while keeping your own keys? Use a SPV wallet.


Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.



On a sidenote, transaction cut-throughs may be interesting to you:
https://bitcointalk.org/index.php?topic=281848.0

Some minor alts are using this approach, unfortunately implementation doesn't appear to be feasible for Bitcoin though.

[deadlift75]

Yeah, I'm pretty sure this was suggested before or some version of it, I was more curious about the objections, so thank you guys for helping me understand. 

[BlackHatCoiner]

Correct me if I'm wrong, but I understand trust here as being statistical, as in, "highly unlikely or computationally too demanding to be otherwise".

No, you should see this in another way. Alright, let's say that Bob who runs his own node receives 1 BTC and the consolidation happens each time a new block is mined, for the 1,000th block. This means that sometime, his node will get rid of the block that contains that transaction and will mark his balance with 1 BTC.

Now, let's say that Alice wants to run her own node and starts receiving the consolidated balances of all the bitcoin addresses. What will prevent Bob from changing his mark that he owns 1 BTC? How can Alice verify that Bob, indeed, owns 1 BTC without trusting his sayings? The answer is:  By verifying every single transaction from the very first block.

Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.

There's no tradeoff in security when it comes to running a node or trusting an SPV server. There's only tradeoff in privacy for the latter.

[HeRetiK]

Each approach comes with a tradeoff of security vs convenience and while SPV wallet suffices for most people it's still important to allow for maximum security and trustlessness, where required.

There's no tradeoff in security when it comes to running a node or trusting an SPV server. There's only tradeoff in privacy for the latter.

Depends on your threat model! For a regular user it might make no difference (unlike e.g. storing your coins in a hot wallet vs using cold storage or a hardware wallet), but if you run a crypto-heavy business like an exchange I'd argue that running your own full node is safer over relying on someone elses.

[PrimeNumber7]

To give a simplified example:  Say you wanted to run a node when the chain height is 100,000 and the network, by consensus, discards any block after the 1,000th most recent one. Once your node synced, you'd have verified that the transactions in the last 1,000 blocks are valid, but you'd have to trust that the rest nodes verified the other 99,000. You see the problem?

Playing devils advocate (and seeing if I can either disprove or validate your point):

If a chainstate was included in block x, and a node that didn’t validate any blocks prior to block x, would need to assume the block x they received is valid. If it is not, the chainstate in block x is invalid. Similarly, block x+1 would be invalid.

One part of validating a block is validating the total proof of work, that is the sum of the POW performed access all blocks to date, and that the current POW is beyond the threshold for the current difficulty. So the cost of producing a fake block x is not trivial and if a node is connected to multiple other random nodes, they should receive the valid blockchain. There are a variety of scenarios in which the total POW for blocks prior to block x could be faked to make it appear that an invalid block x has the highest total POW when it is not a valid blockchain.

Only validating blocks past block x would add uncertainty to the validation process. The chances of receiving an invalid long chain of blocks is low considering the cost of producing invalid blocks, however there are trivial “fixes” to this process that can reduce the chances of accepting an invalid blockchain to nearly zero.

If you are running a pruned node, you will validate the entire blockchain, but will not store more than z blocks on your hard drive. So you will validate all blocks but will not store all blocks. This means that, unless you are being subjected to a Sybil attack, you will not accept an invalid blockchain, but will only store a limited number of blocks. The current implementation of a pruned node is not ideal, for example if a new private key is imported, it will need to download the entire blockchain again in order to know the unspent outputs it can spend, even though it has the UTXO set stored.

[tromp]

A full (as in fully verifying) node needs two things in order to sync to the state of some current header.

1) the UTXO set

2) a proof that this UTXO set results from a valid block history from genesis to the current header

Part 1) only accounts for a small fraction of the 360GB; the vast majority is taken by part 2).

But in principle, part 2) could be replaced by a so-called Succinct Non-interactive Argument of Knowledge with recursive verification, that would be vastly smaller than part 1).

See this book for more details:

https://zcash.github.io/halo2/concepts/proofs.html

This technology is still in its infancy, and formalizing the entire Bitcoin consensus model into a SNARK would be a monumental effort, not to mention constructing the proof (which would have to be redone at regular intervals, e.g. every few months). But verification would be very efficient.

[BlackHatCoiner]

Depends on your threat model!

Oh, so you meant the safety of your physical integrity?

If a chainstate was included in block x, and a node that didn’t validate any blocks prior to block x, would need to assume the block x they received is valid. If it is not, the chainstate in block x is invalid. Similarly, block x+1 would be invalid.

Yes, that's what I said. You have to trust the rest of the nodes.

Only validating blocks past block x would add uncertainty to the validation process.

You mean certainty? Validating blocks prior x or just starting from the genesis block adds certainty that the blocks are valid.

Playing devils advocate (and seeing if I can either disprove or validate your point):

Where's devil's advocate? Judging by your post, I can only assume you agree with me.

[HeRetiK]

Depends on your threat model!

Oh, so you meant the safety of your physical integrity?

That's a great way to put it but not quite what I meant


I was more thinking along the lines of the 3rd party node spoofing incoming transactions, but it looks like that's less of an issue than I initially thought:

First, while the SPV client can not be easily fooled into thinking a transaction is in a block when it is not, the reverse is not true. A full node can simply lie by omission, leading an SPV client to believe a transaction has not occurred. This can be considered a form of Denial of Service.

Obviously the 3rd party node hiding transactions is not ideal either, but less exploitable than the other way round.

[franky1]

pruned nodes are not a trust thing.
all nodes(pruned or full) initially see a complete copy of all blocks and all transactions within.
just a pruned node does not archive them all.

this means a pruned node is not a (torrent buzzword) 'seeder' peer to then give that full information to others. thus peers disconnect from them and search for proper full nodes to get the data

if everyone pruned then no new users can then see the full blockchain to then do a full verification of all transactions.
.

this means the more people that prune. the less archives are decentralised. meaning less peers available to get the full blockchain from.
.
if your a random user that doesnt do much transactions or spend much time with your node running, whereby being a seeder or wanting to be part of the network is not important to you then fine prune. but if you care about decentralisation and want to be a full node then dont prune.

but just be aware that being a pruned node does not make you part of the full decentralised network, you are more of the edge of the network, deemed a (torrent buzzword)'leacher' peer, not a seeder peer. by which you are shrinking/centralising the network by reducing the amount of full node(seeder) peers available, plus also putting those remaining seeder peers under more strain to transmit their full blockchains downstream to random peers by being relied on more as blockchain sources

this can lead to a paradigm where full node archive nodes (seeders) need high bandwidth(unlimited data) internet plans because they are getting more demand, just to service the leachers. leading to another drop off of full nodes due to expense of servicing leachers. or leachers/new peers having a delay in finding seeder peers due to demand because seeder peers limit their connections to not use excessive data. both of which can shrink/centralise the amount of full nodes available, or slowdown the speed of getting blockdata

[DaveF]

To add to what franky1 said, you can also look at people who don't leave their nodes on full time the same way.
You open your client, sync the blockchain, do your transaction(s) and shut it down.
Yes, you can send out old data while your node is syncing, but you are still, more then likely taking in way more data then you are giving out.
Outside of keeping your TX on your own node for privacy, there is really very little reason to run your own node that way.

-Dave

[bitmover]

My question now is why does the network still need full nodes

For the reason I told you. Due to the need for trust.

pruned nodes are not a trust thing.
all nodes(pruned or full) initially see a complete copy of all blocks and all transactions within.
just a pruned node does not archive them all.

I agree with franky1 here. A pruned node is downloading the entire blockchain and verifying each transaction. There is no trust involved. Pruned nodes just doesn't achieve the entire blockchain, and save HD space. which is exactly what the OP is talking about.

Personally, i don't know what is their limitation, but you can even use Bitcoin Core software to run a pruned node.

My question now is why does the network still need full nodes, I mean what's being checked against the full nodes that can't be checked or trusted on the information of the pruned nodes?

Imo, a pruned node is a full node.

Let's look what bitcoin wiki says:

https://en.bitcoin.it/wiki/Full_node
Any computer that connects to the Bitcoin network is called a node. Nodes that fully verify all of the rules of Bitcoin are called full nodes. The most popular software implementation of full nodes is called Bitcoin Core, its latest release can be found on the github page.

Imo, a pruned node fits that description.

Anyway, the network needs full nodes to verify all transactions and all blockdata without any trust.

For example, let's image you are a big exchange (like Binance or Coinbase), or that you are blockexplorer. you need to run a full node to verify each single UTXO, not just the ones that you are interested with.

If someone sends you a coin which was received in 2013, you need to be able to fully verify it was not spent without trusting anyone.

Full nodes are also important for privacy purposes. When you use an spv wallet, this wallet asks for the balance of your addresses to other full nodes. Those nodes may be tracking your requests, and they might link your IP with your addresses.