Ultimate blockchain compression w/ trust-free lite nodes

[jtimon]

Why can the miner not use the block-headers only to verify, that he's on the right chain? Is there a reason all the already spent transactions of the past have to be stored somewhere?

Only reading the headers you can only tell, which one is the longest chain if you receive several of them. But you want the longest VALID chain.
Here longest is the chain with more proof of work.
You have to validate all transactions to be able to know that the chain is following the protocol rules.

[etotheipi]

So this was "merged" with another topic... I'm not really sure what that means or why it happened.  But I'm not finding the most recent posts.   The last one I see is June 18.  Any idea what happened?  Any way to get all the posts since then?

[erk]

I just installed a fresh copy of Bitcoin-Qt v0.85-beta, and it's taken 2 full days so far to download the block chain with 10-20 peers most of the time, and it's still only up to August. This strikes me as something wrong, if I were running a pool or exchange it would be a disaster and I would have lots of unhappy users. Not only does the block chain size need to be looked into, but the replication method. If I were trying to download the same size data using .torrent it would have finished in hours, not days.

[altsay]

I wonder isnt it possible to copy the up-to-date blockchain in the configuration folder and paste it to another computer.

[ShadowOfHarbringer]

I wonder isnt it possible to copy the up-to-date blockchain in the configuration folder and paste it to another computer.

Yep, already done. Several times.

But (obviously) stop Bitcoin client first.

[maaku]

I have posted to the development mailing list the first draft of a BIP for a binary authenticated prefex tree derivative of the one described in this thread. This starts the process of getting it turned into an official BIP standard. The draft is available for viewing here:

https://github.com/maaku/bips/blob/master/drafts/auth-trie.mediawiki

All comments and criticisms are welcome.

[sugarpuff]

Could etotheipi negativeone (I *just* realized what that says!) or someone knowledgeable comment on how the "rolling-root"/"ledger-solution" impacts this proposal (whether it enhances it, makes it unnecessary, or is wrong for reasons XYZ)?

Summary: keep the blockchain down to some finite size by moving unspent transactions out of old blocks and into new ones at the head via a "rolling-root" mechanism.

Relevant link: https://bitcointalk.org/index.php?topic=501039

[Raize]

I have posted to the development mailing list the first draft of a BIP for a binary authenticated prefex tree derivative of the one described in this thread. This starts the process of getting it turned into an official BIP standard. The draft is available for viewing here:

https://github.com/maaku/bips/blob/master/drafts/auth-trie.mediawiki

All comments and criticisms are welcome.

I guess my only comment right now is why doesn't this have a BIP number assigned to it? Due to the hard fork requirements?

[maaku]

Because I don't assign BIP numbers. There's a process to getting a BIP number assigned, which I am still working towards.

[jtimon]

Due to the hard fork requirements?

This would be a softfork, not a hardfork. Meaning that a majority of miners must update, but not all users.

[sugarpuff]

Could somebody here knowledgeable about the details of UTXO please confirm whether my understanding of how it would work in practice is correct?

[maaku]

You never ever under any circumstances want to be mining on top of a chain you have not validated the entire history of.

[doldgigger]

Before I read this I just want to quickly post that I personally, no matter whether justifiably or unjustifiably, I personally feel like this is the most pressing issue when it comes to Bitcoin's successful future and I really hope the core team has planed an order of priorities accordingly.

Why pressing? The blockchain is easy to understand and verify. And the practically required size might be reduced by imposing a fee on old inputs, which would lead to implementors of wallet software implementing per-wallet compression strategies in order to avoid the fee. Having some archival nodes with a full history still wouldn't hurt.

[sugarpuff]

You never ever under any circumstances want to be mining on top of a chain you have not validated the entire history of.

Interesting point. Hope you don't mind if I mention your reply in that other thread as well.

So, what is the takeaway from that then? That new lite-nodes can use UTXO to validate arbitrary queries, but they cannot participate in securing the network until they have all the transactions for the leaf nodes of the entire UTXO tree?

[maaku]

no it's worse than that -- you have to download and process every single block since Genesis. otherwise you may be on an invalid chain and not even know it.

[sugarpuff]

no it's worse than that -- you have to download and process every single block since Genesis. otherwise you may be on an invalid chain and not even know it.

Can you explain why that is?

If what you're saying is true, then the rolling root is still needed for Bitcoin's future feasibility and safety: http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/#comment-1442133143

[maaku]

Sure I can, now that I'm at my computer instead of my phone

The miner needs to verify the entire block chain history because otherwise he has no way of knowing if he is actually on a valid chain or not. This has nothing to do with UTXO commitments, rolling root, or any other proposal. It's a basic, fundamental requirement of any consensus system: if the miners themselves operate in SPV mode (which you advocate), then anyone -- no matter their hashrate! -- can trick the network into mining an invalid chain. The attacker does so by mining a fork with invalid history and temporarily (by luck or 51%) overcoming the honest network. New miners coming online, or miners tricked into reseting their state then switch to the invalid chain This completely invalidates the SPV assumption and makes it unsafe for anybody to use the network.

"Rolling root" doesn't even make sense in the context of bitcoin, as has been explained multiple times by multiple people in your own thread. Let's not bring that discussion here. If your goal is to minimize the amount of data required to bring new non-mining nodes online, then that is what UTXO commitments does.

[sugarpuff]

The miner needs to verify the entire block chain history because otherwise he has no way of knowing if he is actually on a valid chain or not. This has nothing to do with UTXO commitments, rolling root, or any other proposal. It's a basic, fundamental requirement of any consensus system: if the miners themselves operate in SPV mode (which you advocate)

Full stop. I have never advocated that. If you believe that, then you never understood the rolling root proposal.

I agree it's best to keep the threads separate. If you have questions about how rolling root works feel free to ask in the other thread.

Sticking to UTXO though, I don't believe you answered my question. UTXO is not SPV either, so it is still not clear to me why you say they don't know whether they're on a valid chain or not. They downloaded the headers from genesis (SPV), but in addition to that they downloaded the entire UTXO meta chain which they can then use to verify any txn and build the merkle/btree or w/e the latest data structure is.

[doldgigger]

no it's worse than that -- you have to download and process every single block since Genesis. otherwise you may be on an invalid chain and not even know it.

People might speed up that process by relying on checkpoint hashes built into the client. Since most cryptocurrency client software is developed using git (which also comes with a cryptographically secured history), detecting manipulations there is also practical in most cases.

[maaku]

Checkpoints are a temporary hack that will go away soon, we hope.