Why do you need to download 7 years of chain block

[kushti]

In the first place, there is no rationale to store 7 years of history for synced nodes. But how a new node could get into network if all the fullnodes have thrown away all the blocks(except last few ones to handle possible rollbacks)? Bitcoin has no answer for that.

I've developed a proposal to fix the issue and solve some important questions towards better scalability. The paper is just uploaded http://arxiv.org/abs/1603.07926 . I would be happy to get feedback.

[1714824180]

1714824180

[1714824180]

1714824180

[1714824180]

1714824180

[TransaDox]

In the first place, there is no rationale to store 7 years of history for synced nodes. But how a new node could get into network if all the fullnodes have thrown away all the blocks(except last few ones to handle possible rollbacks)? Bitcoin has no answer for that.

I've developed a proposal to fix the issue and solve some important questions towards better scalability. The paper is just uploaded http://arxiv.org/abs/1603.07926 . I would be happy to get feedback.

Great to see people are working on this. I was beginning to think the "technology is cheap" brigade were winning. It was a bit much for me to take in in one go so I'll need to read it several times but there was once thing that struck a chord with me - the linked list. I think this is a key point to enable breaking the blockchain into resolvable chunks to enable both forward and backward discovery. You only need to find one chunk to be able to construct the whole chain by pregressively downloading the prev/next in the list..

I too have been working in that direction except using a variant of the bitorrent DHT. There are a couple of proposals whereby the DHT conveniently uses  merkle trees! and an arbitrary data field (1K) which I would increase to 1MB (current block size). The protocol would additionally have the prev/next  (linked list) of the blocks so the blockchain would be stored in the DHT and cached throughout the network. This would also work with the current chains without modification because you could just dump the entire chain and let the client use it.

I'm not sure about replication, as yet. The one thing that is lacking is a mechanism to ensure that at least one of each is available at all times so a slight modification of the rarity calculation might be needed.

I hope to see more research in this area.

[gmaxwell]

In the first place, there is no rationale to store 7 years of history for synced nodes. But how a new node could get into network if all the fullnodes have thrown away all the blocks(except last few ones to handle possible rollbacks)? Bitcoin has no answer for that.

I've developed a proposal to fix the issue and solve some important questions towards better scalability. The paper is just uploaded http://arxiv.org/abs/1603.07926 . I would be happy to get feedback.

This paper is a bit of a disappointment, sad to say.  First-- it makes a number of false claims about the capabilities of the existing Bitcoin system, e.g.

"To be rational it is needed to switch to an implementation of the Bitcoin protocol which does not hold the unnecessary data (such an implementation of a full node does not exist to the best of our knowledge)."

The whole reason that Bitcoin has state snapshots, as described in the paper, was because they were introduced to enable strong pruning (a more limited version is described in section 7 of the Bitcoin whitepaper).  Bitcoin has supported operating in a pruned fashion for _years_.


The primary suggestion the paper is making is to bootstrap nodes from state snapshots. The paper describes this as "trustless" but it is _not_, rather it makes the same strong assumption of hashpower honesty that SPV (lite wallet nodes do), implicitly trusting the history of the longest chain to be faithful.

This is not a new idea, its expression spans at least back to 2011:

https://bitcointalk.org/index.php?topic=21995.0

and

https://en.bitcoin.it/wiki/User:DiThi/MTUT

(as is using random tests for data in the mining process,-- though the scheme you suggest is fully delegatable, so long as the miner doesn't mind giving temporary custody of his income to the centralized party holding the world's only copy of the blockchain. )

The state commitment proposals face a couple challenges, one is that if you are willing to strongly trust the hashpower-- then why not use the vastly more efficient SPV mode?  The other is that the initial efforts to implement continual state commitments found that the overhead (primarily IO) of maintaining them multiplied the full node operating costs at runtime by several fold.  The costs have made what would otherwise be another option for client security less attractive, supporting a more niche security model would be fine if it were inexpensive.

Alternative designs like mimblewimble try to make bootstrapping more efficient without adding a new form of strong trust towards miners.

[italianMiner72]

ok, so I installed bitcoin core latest version and it downloaded over 80G of chain block. Let's assume, just for the fun of it, there are 1000 bitcoin core users out there. That's ~8T of wasted disk space. and considering bitcoin will live another 7 years and it will grow of couse, that's like ~20T of disk space (1000 users remember?) for what? couldn't be a centerlaize, maybe mirrored, location that the client will ask for the chain block from there? Why do we need to download it?

turn your point of view...
in a next futures, nearest than 7 years, 20Tb will be like today 20gb or maybe 20Mb..
just look here
http://arstechnica.com/gadgets/2016/08/seagate-unveils-60tb-ssd-the-worlds-largest-hard-drive/
give it time

[TransaDox]

ok, so I installed bitcoin core latest version and it downloaded over 80G of chain block. Let's assume, just for the fun of it, there are 1000 bitcoin core users out there. That's ~8T of wasted disk space. and considering bitcoin will live another 7 years and it will grow of couse, that's like ~20T of disk space (1000 users remember?) for what? couldn't be a centerlaize, maybe mirrored, location that the client will ask for the chain block from there? Why do we need to download it?

turn your point of view...
in a next futures, nearest than 7 years, 20Tb will be like today 20gb or maybe 20Mb..
just look here
http://arstechnica.com/gadgets/2016/08/seagate-unveils-60tb-ssd-the-worlds-largest-hard-drive/
give it time

Platitudes like this are speculative, unhelpful and infuriating.

Now we are being told we need two drives because the block chain gets corrupted. The current blockchain storage isn't fit for purpose and you are in denial.

[manselr]

Will it ever be possible to enable pruning mode without having to download the entire chain the first time?
Isn't 1GB enough to guarantee safety? I mean who (attacker) is ever going to manage to go "1gb back in time?"
Of course full nodes is deal but I think people would use Core wallet more if you could enable pruned without download entire chain first time.

[Syke]

Will it ever be possible to enable pruning mode without having to download the entire chain the first time?

That depends on what you want to do. If you just want to verify a specific transaction, like when you receive a payment, you really only need to download all blocks that include all previous inputs. Then, any inputs that have been spent can be safely pruned.

[damnMscollec]

It reminds me when i first download the qt wallet, it said I need to download the blockchain for many days. LOL

The answer of why: the wallet needs to have full blockchain file for sync, we need to sync the wallet to see more transactions details, but you can still send bitcoin if the wallet is not sync

[kushti]

In the first place, there is no rationale to store 7 years of history for synced nodes. But how a new node could get into network if all the fullnodes have thrown away all the blocks(except last few ones to handle possible rollbacks)? Bitcoin has no answer for that.

I've developed a proposal to fix the issue and solve some important questions towards better scalability. The paper is just uploaded http://arxiv.org/abs/1603.07926 . I would be happy to get feedback.

This paper is a bit of a disappointment, sad to say.  First-- it makes a number of false claims about the capabilities of the existing Bitcoin system, e.g.

"To be rational it is needed to switch to an implementation of the Bitcoin protocol which does not hold the unnecessary data (such an implementation of a full node does not exist to the best of our knowledge)."

The whole reason that Bitcoin has state snapshots, as described in the paper, was because they were introduced to enable strong pruning (a more limited version is described in section 7 of the Bitcoin whitepaper).  Bitcoin has supported operating in a pruned fashion for _years_.


The primary suggestion the paper is making is to bootstrap nodes from state snapshots. The paper describes this as "trustless" but it is _not_, rather it makes the same strong assumption of hashpower honesty that SPV (lite wallet nodes do), implicitly trusting the history of the longest chain to be faithful.

This is not a new idea, its expression spans at least back to 2011:

https://bitcointalk.org/index.php?topic=21995.0

and

https://en.bitcoin.it/wiki/User:DiThi/MTUT

(as is using random tests for data in the mining process,-- though the scheme you suggest is fully delegatable, so long as the miner doesn't mind giving temporary custody of his income to the centralized party holding the world's only copy of the blockchain. )

The state commitment proposals face a couple challenges, one is that if you are willing to strongly trust the hashpower-- then why not use the vastly more efficient SPV mode?  The other is that the initial efforts to implement continual state commitments found that the overhead (primarily IO) of maintaining them multiplied the full node operating costs at runtime by several fold.  The costs have made what would otherwise be another option for client security less attractive, supporting a more niche security model would be fine if it were inexpensive.

Alternative designs like mimblewimble try to make bootstrapping more efficient without adding a new form of strong trust towards miners.

Dear Greg, it seems you've missed a part of the protocol. Let me provide a quick description of Rollerchain, more precisely, modifications from the Bitcoin.

1. We authenticate the state and include root value into a blockheader. It is an old idea yes, and already implemented in Ethereum (and some other coins) as mentioned in the paper.

2. We then modify a Proof-of-Work function. A miner chooses "k" state snapshot versions out of last "n" a (sufficiently large) network stores collectively. In order to generate a block miner needs to provide proofs of possession for the state snapshots. On a new block arrival a miner updates k+1 states, not one, so full blocks (since minimal value in "k") are also needed.

Thus miners store a distributed database of last "n" full blocks AND state snapshots getting rewards for that activity. A new node could download not just a last snapshot, but from "n" blocks ago (or from somewhere in between). So it is not needed to "strongly trust the hashpower", but "hashpower" and also up to "n" last full blocks("n" considered to be >> than a deepest fork possible, say, 5000-10000 for Bitcoin) . And this is not about SPV, but about getting fullnode-going-from-genesis security(with probability of divergence going down exponentially with "n", see Theorem 2). Full blocks not needed in order to mine could be removed by all the nodes.

If you have concerns about this scheme please let me know, and I will try to formalize/address them.

Authenticated state root in a block header could be useful anyway for better SPVs. On efficient implementation, we're now working on that along with benchmarking. Let's see how it goes.


P.S. However, as PoW function modification is needed it is unlikely Bitcoin can go this way(miners will not throw away their ASICs).

[Amitabh S]

I think Rollerchain (RC) is similar in motivation to SPV but different under the hood.

(If I understood correctly, haven't spent too much time on it)

Assume round 0 is "now" , -1 is last block, -2 the one before that, etc.

Let U_i be UTXO set at round i

SPV stores Delta(U_{-n}, U_{-n+1}), Delta(U_{-n+1}, U_{-n+2}) .. Delta(U_{-1}, U_0)

RC stores: (U_{-n}, Delta(U_{-n}, U_{-n+1}), Delta(U_{-n+1}, U_{-n+2}) .. Delta(U_{-1}, U_0))

Thus, SPV is equivalent (at least in storage complexity) with RC having the additional initial state.
In RC, once a new block (delta) is mined, the state U_{-n} is updated to U_{-n+1} and U_{-n} is discarded.

The difference I think is that SPV does not enforce nodes to store n blocks and its more or less a risk if every node prunes their chain to just 10 blocks (acting "rationally").

RC claims to enforce within the protocol itself that the n blocks be stored.

[awesome31312]

I understand "why" we have to download it. But oh man, it's awful! I wish I knew I was in for eighty gigs about three years ago, when I got into Armory.

Back then it was only around 30 gigs, doable in a day. There has got to be an easier way. Let's hope that when the blockchain size hits something like 20 T, internet services will also have kept up and evolved at the same rate.

[kushti]

The difference I think is that SPV does not enforce nodes to store n blocks and its more or less a risk if every node prunes their chain to just 10 blocks (acting "rationally").

Even for a fullnode, Bitcoin protocol does not enforce nodes to store (any number) of full blocks, so if a new node is able to download blocks since genesis that is the artifact of altruistic behaviour. RC works under stronger assumptions about rationality of network participants. Please note nodes in RC could save more full blocks than required by the protocol, even from genesis (as in Bitcoin), but that's not really needed (as well as in Bitcoin).

[severaldetails]

I must say that I do not mine and that I do not know much about mining, but I have a question:
Can't the blockchain be imported from a usb medium?
I mean, if I download it and put it on a usb stick (if that is possible), can't I give it to my neighbour so that he does not have to download the blockchain?

[BeginnerCoin]

You don't have to unless you download the main client. It supports the network to be a node, and you can even self-mine from your wallet (with little to no effect, though.)

At least some clients/miners need to maintain the entire blockchain at all times to reach consensus. Every single transaction matters because it affects the entire history.

[AmDD]

I must say that I do not mine and that I do not know much about mining, but I have a question:
Can't the blockchain be imported from a usb medium?
I mean, if I download it and put it on a usb stick (if that is possible), can't I give it to my neighbour so that he does not have to download the blockchain?

Yes, this is possible. Your neighbor's client will read the blockchain and start where it left off.

[Hazir]

I know that Bitcoin Core can run in Prune Mode, effectively cutting disk space you will need for storing blockchain data.
But you will still need to download whole blockchain before that. Is there even remote possibility that we won't need to download whole blockchain in the future by using some sort of 'pruning'?

[TheUltraElite]

It really hurts to download the 7 years of blockchain on slow and turtle-speed internet connections like in my country. However as an alternative you can use Multibit-HD wallet which is very much compact ans useful since id does not need to download the entire 20GB+ blockchain.

Otherwise the need for the entire block ledger is there since that what is needed to understand the distribution of coins and for new transactions.

[gmaxwell]

Dear Greg, it seems you've missed a part of the protocol. Let me provide a quick description of Rollerchain, more precisely, modifications from the Bitcoin.

1. We authenticate the state and include root value into a blockheader. It is an old idea yes, and already implemented in Ethereum (and some other coins) as mentioned in the paper.

2. We then modify a Proof-of-Work function. A miner chooses "k" state snapshot versions out of last "n" a (sufficiently large) network stores collectively. In order to generate a block miner needs to provide proofs of possession for the state snapshots. On a new block arrival a miner updates k+1 states, not one, so full blocks (since minimal value in "k") are also needed.

Thus miners store a distributed database of last "n" full blocks AND state snapshots getting rewards for that activity. A new node could download not just a last snapshot, but from "n" blocks ago (or from somewhere in between). So it is not needed to "strongly trust the hashpower", but "hashpower" and also up to "n" last full blocks("n" considered to be >> than a deepest fork possible, say, 5000-10000 for Bitcoin) . And this is not about SPV, but about getting fullnode-going-from-genesis security(with probability of divergence going down exponentially with "n", see Theorem 2). Full blocks not needed in order to mine could be removed by all the nodes.

If you have concerns about this scheme please let me know, and I will try to formalize/address them.

Authenticated state root in a block header could be useful anyway for better SPVs. On efficient implementation, we're now working on that along with benchmarking. Let's see how it goes.


P.S. However, as PoW function modification is needed it is unlikely Bitcoin can go this way(miners will not throw away their ASICs).

I did not miss this.   But unless I am misunderstanding you,  you appear to be conflating two orthogonal changes.

One is the change to the POW which requires the miner to show possession of state snapshot data.  This does nothing to assure users that the data is correct.  It ~may~ increase the odds there are multiple copies of it (though unless I misunderstand it-- your design is trivially delegatable so long as the miners are willing to trust the party storing the data until their coins mature, which users of mining pools already do almost ubiquitously).

The other is that you begin nodes from a state snapshot, not at the tip, but at some point somewhat back. This is also taught in in the citations I provided above.  This also does not prove that the state is correct, in fact it provides no evidence at all that the state is correct except under a strong assumption.

Instead, participants hope that the state is correct without checking under a strong assumption that the majority hashpower would not extend a chain with invalid state updates. This is the SPV security assumption.

It is different and weaker than the normal Bitcoin behavior, trusting a majority hashpower for the soundness of state updates and not merely their sequencing. In practice it is not sufficient to simply take such a strong assumption and not question it's validity-- normally the SPV hashpower assumption is justified by the widespread and economically significant use of full nodes which will not be deceived by dishonest miners; but with full nodes also making a SPV assumption for blocks away from the tip, this argument becomes circular.

The SPV assumption is especially concerning because we have discovered that miners have begun bypassing validation themselves in order to mitigate the negative effects of larger amounts of transaction data on propagation. State sampling proof of work has been argued as a potential improvement for this particular issue, but proposals so far have worsened the generation of progress in the mining function (as the last miner to create a block has a state update advantage) and they still leave plenty of opportunity to optimize by shortcutting the parts of validation that the state updates do not expose.


Perhaps it would be revealing to answer a question:  You are a new node joining the network. I am part of an attacking consortium which has, for several months, had a super majority hashpower at my disposal.  Can I cause you to accept a chain extending the chain from no further than a few months back, otherwise compatible with a 'honest chain' (if it existed)-- meaning containing all the same transactions and ownership except as mentioned-- where I instead have a million extra BTC reassigned from the earliest blocks?

[kushti]

One is the change to the POW which requires the miner to show possession of state snapshot data.  This does nothing to assure users that the data is correct.  It ~may~ increase the odds there are multiple copies of it (though unless I misunderstand it-- your design is trivially delegatable so long as the miners are willing to trust the party storing the data until their coins mature, which users of mining pools already do almost ubiquitously).

If you see a newly generated block, that means "k" state snapshots in the past (for different heights) are stored by the network, and the guarantee is pretty strong. On delegation, to reach line 15 in the RollerPoW function with a new input (so anything changed in <a_t, a_S, ctr> with fixed <s, a_t>)  you need to recalculate a ticket so you can't fix a_t and outsourcing of ticket calculation doesn't make any sense.

The other is that you begin nodes from a state snapshot, not at the tip, but at some point somewhat back. This is also taught in in the citations I provided above.  This also does not prove that the state is correct, in fact it provides no evidence at all that the state is correct except under a strong assumption.

Which assumption?

Perhaps it would be revealing to answer a question:  You are a new node joining the network. I am part of an attacking consortium which has, for several months, had a super majority hashpower at my disposal.  Can I cause you to accept a chain extending the chain from no further than a few months back, otherwise compatible with a 'honest chain' (if it existed)-- meaning containing all the same transactions and ownership except as mentioned-- where I instead have a million extra BTC reassigned from the earliest blocks?

If "a super majority hashpower" is committed to adversarial behaviour, we got common prefix property (and chain quality also) from the GKL model broken. Light verification is also broken in this case(it assumes common prefix property being hold, see Theorem 2). Full verification is also broken though. Even more, how can you define a desirable bootstrapping procedure outcome if common prefix property isn't held? If it is held, bootstrapping outcome could be simply defined like "a new node is getting chain C such as set (C |_| chains of honest participants) satisfies common prefix property". I'm going to update the paper with a definition like that.

[royalfestus]

ok, so I installed bitcoin core latest version and it downloaded over 80G of chain block. Let's assume, just for the fun of it, there are 1000 bitcoin core users out there. That's ~8T of wasted disk space. and considering bitcoin will live another 7 years and it will grow of couse, that's like ~20T of disk space (1000 users remember?) for what? couldn't be a centerlaize, maybe mirrored, location that the client will ask for the chain block from there? Why do we need to download it?

The entire purpose of downloading the entire blockchain is to independently verify it and allow for the independent verification of every single block and transaction that occurs later. The only thing that a full node needs to "trust" is the genesis block, the very first block of the blockchain. This is actually hard coded into the software so that there is a defined starting point. After that, the node downloads all of the blocks from its peers. It will check each transaction in the block, and then check the block to ensure that it is valid. Because each block is chained together and transactions are also chained together, it is not possible to start from any point in the blockchain other than the beginning and still maintain trustlessness.

It answered my question of y download the block chain