Blockchain Compression

[Peter Todd]

Including a UTXO hash in the checkpoint makes for a much less secure model than not doing so.

With checkpoints the client only skips validating transactions, it still builds the UTXO set. You've still shown that an vast amount of hashing power work has been spent towards the blocks leading to the checkpoint, and validating that work, as well as the work after the checkpoint, is pretty easy. More importantly if the checkpoint is invalid and the majority of hashing power isn't following it you'll soon notice because your client doesn't see as many blocks as it should.

On the other hand a UTXO hash is trusting the developers to compute the UTXO set properly for you. They could easily leave UTXO's out and you would never know until your node tried to spend one of them and wound up forked from the main network, or with a majority of hashing power, someone found out their coins were now unspendable by developer fiat. Point being fraud on the part of the developers isn't detected automatically or immediately.

UTXO hashes will be acceptable once UTXO proofs are a requirement for a block, and you'll be able to reason about how much work has been done for any given UTXO proof, but as you know we're a long way from finishing that.

[1715335165]

1715335165

[1715335165]

1715335165

[1715335165]

1715335165

[1715335165]

1715335165

[1715335165]

1715335165

[1715335165]

1715335165

[piotr_n]

With checkpoints the client only skips validating transactions, it still builds the UTXO set. You've still shown that an vast amount of hashing power work has been spent towards the blocks leading to the checkpoint, and validating that work, as well as the work after the checkpoint, is pretty easy. More importantly if the checkpoint is invalid and the majority of hashing power isn't following it you'll soon notice because your client doesn't see as many blocks as it should.

But if you are taking a piece of software where some devs have put a checkpoint, why would you trust them in a part of choosing a honest block hash, but not trust that they honestly verified the chain up to that point, difficulty check including?
It just doesn't make any sense to use this software to verify from a scratch a chain that can only be valid if it reaches a block with a specific hash - value fixed inside the same software that is doing (again) the blocks' verification, to reach the same hash - because only then it can work.
It's just a huge overhead.

[piotr_n]

Maybe there should be something like a general solution, to compress it once and for all.
Its all about the most recent content of UTXO - which is not so big, comparing to the so much faster growing blockchain size.
Maybe there should be an automatic checkpoint (like a small-generic) each 2016 blocks or so (with a hash of the UTXO db content included inside the block) from which nodes would be able to start their existence, using the chosen hash as a trusted seed to quickly fetch entire UTXO, instead of re-doing the chain up to that point.
If that works, there would be no more need to store, nor to share, the entire blockchain - it could be stored in a museum, though

[piotr_n]

Maybe actually it should be simpler.
Each miner would have a possibility to mine in a hash of the current utxo db - of course it needs to be sorted, or something, but wrong hash means block is invalid.
And then, by other means, not necessarily ones that need to be sponsored by bitcoin foundation, people would find a way to fetch these snapschot of utxo
as long as you add to the protocol a possibility to mine in hash to utxo and make sure that all the miners get it...
two years at least.
ten since it's my idea

[justusranvier]

Though, again, who am I talking to + the idea of checkpoints is stupid and goes totally against the main purpose of the solution

I don't think there ultimate is a nice, clean, mathematically and cryptographically pure solution to the problem that will work in practise. We'll end up with a messy set of overlapping partial solutions that will provide sufficient security when balanced against usability.

The developers on GitHub could commit malicious checkpoints, but due to the nature of Git there's no way to hide what they've done. Anyone who's been running a full node can notice the discrepancy and spread the word via any number of communication platforms and bring enough scrutiny to the problem that everybody can manually figure out what's going on and apply a fix if necessary.

In the pure theoretical realm of solving the Byzantine Generals' Problem one doesn't normally include the existence of IRC, Twitter, Facebook, and Reddit as part of the solution, but in the real world relying on the ability of users to communicate out of band can sometimes be an acceptable way to resolve an attack. I'd like to see checkpoints distributed via as many mediums as possible, including print media, by as many different entities as possible to make it as difficult as possible for someone to distribute incorrect ones without being detected. We really want everything to work automatically, but it's good to have backup plans to fix things manually when something breaks.

[piotr_n]

OK, agreed.
But since you already have checkpoints, why don't you get advantage of them?

[Mike Hearn]

I'd actually be OK with including a database copy in the downloads, it's a much simpler and easier thing to do than complicated shared UTXO commitment schemes. You can't verify the database by hand, but you can't verify the binaries either and we obviously provide binaries. The same techniques could work fine - a bunch of different people all collaborate to do a threshold-signing of the included database just like they sign the binaries. Then anyone can tell their node to audit the developers just by letting it recalculate the database like anyone can audit the binaries using gitian.

However it seems I'm in the minority on that one. Whenever I've suggested including a database copy in the downloadable binaries lots of other people jumped up to disagree with me. So - SPV it is.

[piotr_n]

I'd actually be OK with including a database copy in the downloads, it's a much simpler and easier thing to do than complicated shared UTXO commitment schemes.

This would only improve the bootstrapping time, but it is nohow a "blockchain compression" solution.

Calculating a reliable check-sum of UTXO snapshot might be a bit time consuming, but the process itself is not a rocket science and the algo should be very simple to implement.
Even if you do the actual hash of all the records, the operation should take far below one minute, on a modern PC.
But if you choose a smarter checksum calculation method, you can speed it up a lot by doing some kind of differential UTXO checksum adjustment while processing each block, which would only require fractions of seconds.
A node would already have the expected checksum value, because a UTXO checksum in a block would actually refer to the UTXO state after the previous block.
Moreover, if you still find calculating of UTXO checksum too much resource unfriendly, you may only allow such checksums to be mined (thus verified) each few thousands blocks.