A rolling root to solve Bitcoin's scalability problem?

[sugarpuff]

EDIT June 18, 2014: To avoid confusion: this thread is now about a solution to the tremendous (and increasing) financial and temporal costs of bringing new nodes online, not disk space.

...to handle the number of transactions that visa handles in 3 months the bitcoin system will require 14 Terabytes of storage space.

To assume that people are going to be willing to buy 14 terabytes of disk space every 3 months (even if it is just $40 in 2020) is a leap.

References:

- http://stormcloudsgathering.com/bitcoin-what-youre-not-being-told
- https://en.bitcoin.it/wiki/Talk:Scalability#Disk_space

So... I don't get it. Why do we need to hold onto the root?

Can't this problem be easily fixed by letting go of the root?

"What about unspent transactions at block 20?" I hear you cry.

Copy whatever data is necessary to move them automatically to the head in a new transaction. Am I missing something?

Edit March 5, 2014: thanks to Realpra for pointing out that this was called the "ledger-solution". See his reply below, also quoted here:

This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

Edit2 March 5, 2014: Something not previously noted here is that bitcoind can (and might already in some ways do) prune its locally stored blockchain, but only after it downloaded the entire thing once. Thus, this issue is more of a problem (I think) for new full-nodes, which currently must download the entire blockchain before they can prune the entire thing.

[1714930674]

1714930674

[1714930674]

1714930674

[maaku]

How else would you bring new nodes online?

[sugarpuff]

How else would you bring new nodes online?

Download the internet-agreed upon new "rolling-genesis-block-snapshot".

[maaku]

How do you reach "internet-agreement"?

[DeathAndTaxes]

How do you reach "internet-agreement"?

Better yet how do the new nodes verify that the agreement has been reached and that agreement is valid.

To OP, the reality is most nodes in the future will be SPV.  The storage requirements are maybe 1% of that of a full node.   Eventually even most "full nodes" will be storing only a pruned copy of the db.  Today that means a 80%+ reduction in storage requirements and that will only improve in time.   Some nodes (maybe call them archive nodes to distinguish from full nodes) will need a full copy of the blockchain and for them the cost will be worth it.  If you are a major exchange and are making $1B a month in profit (and you would if Bitcoin is so huge it has higher tx volume them VISA) do you think it might be worth it to pay $1,000 a year in additional storage cost to add some drives to the SAN?  I think it might be.

[sugarpuff]

How do you reach "internet-agreement"?

One way would be to do what you already do (checkpoints), although those are controversial.

I haven't given it too much thought, but it might be possible to do something like have a hard coded rolling window, and have nodes mark the new root with some flag ("currentRoot = true").

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.

I'm making these answers up on the fly btw. I'm sure better ones could be given with more thought.

Still don't get what the problem is...

[DeathAndTaxes]

Still don't get what the problem is...

Well you are abstracting away the entire problem.  It is like saying how can we get into space and your answer is you keep going up until you are there.

Rereading your OP my guess is you don't understand how Bitcoin works (what is a transaction?  how does it work?  what are outputs?  what is the UXTO?  how can we verify that any given output is valid?).

If I "sent" you coins and you didn't have the block which contained the transaction which contained that output how would you validate the input is valid and not just some nonsense I sent you because I know you have no way of validating?

[sugarpuff]

If I "sent" you coins and you didn't have the block which contained the transaction which contained that output how would you validate the input is valid and not just some nonsense I sent you because I know you have no way of validating?

That just wouldn't be possible. It would be discarded like all other invalid transactions.

You wouldn't be able to send to or from an outdated address (it would be considered invalid). You'd use the new ones that were copied as new transactions to the head. It's possible I am misunderstanding (I've only read the original paper a couple times, and perused the wiki, but I'm not working with the code itself).

Sidenote: consensus about an ancient block should be extremely high. There should be no disagreement about it.

[kjj]

If I "sent" you coins and you didn't have the block which contained the transaction which contained that output how would you validate the input is valid and not just some nonsense I sent you because I know you have no way of validating?

That just wouldn't be possible. It would be discarded like all other invalid transactions.

You wouldn't be able to send to or from an outdated address (it would be considered invalid). You'd use the new ones that were copied as new transactions to the head. It's possible I am misunderstanding (I've only read the original paper a couple times, and perused the wiki, but I'm not working with the code itself).

Sidenote: consensus about an ancient block should be extremely high. There should be no disagreement about it.

Discarded by whom?  "Considered invalid" by whom?  Which ancient block?

Bitcoin is an imperfect solution to a very hard problem.  You can't just handwave it all away like Dilbert's boss.

http://dilbert.com/strips/comic/1994-10-17/

[DeathAndTaxes]

Only the private key holder of the tx output can create a new tx.  There is no mechanism to "copy the outputs as a new transactions to the head" and then verify off that.

[sugarpuff]

Discarded by whom?  "Considered invalid" by whom?

By all full-nodes.

Which ancient block?

The one that's at the back end of the rolling window.

Bitcoin is an imperfect solution to a very hard problem.  You can't just handwave it all away like Dilbert's boss.

I agree and I'm doing my best to give concrete answers to these questions that are not handwavy.

[sugarpuff]

Only the private key holder of the tx output can create a new tx.  There is no mechanism to "copy the outputs that were coped as new transactions to the head".

The new protocol dictates that your node create this new transaction for you. Edit: I suspect that might not even be necessary, since the balance wouldn't change. It would just be an alias for the old txn.. not entirely sure ATM.

If this method works, and disk space becomes a real problem, it might just have to work this way.

(BTW, I'd love to give ya'll faster replies, but this forum has imposed a bunch of restrictions on this account because it hasn't posted much. I can currently post a maximum of 1 reply per 6 minutes, and I don't know if that timer resets each time I try.. Annoying.)

[grau]

The restrictions are there for a good reason, to encourage learning before teaching.

The problem you are about to tackle is hard. I recommend this thread to get an insight of its complexity.

https://bitcointalk.org/index.php?topic=88208.0

[sugarpuff]

The restrictions are there for a good reason, to encourage learning before teaching.

The problem you are about to tackle is hard. I recommend this thread to get an insight of its complexity.

https://bitcointalk.org/index.php?topic=88208.0

Thanks, I'd had a look at that thread already, but a second, closer read-through is probably warranted (especially since etotheipi has revised his idea a few times).

Still looking for confirmation on this or a "you're suggestion won't work for reason(s) XYZ."

[maaku]

Your suggestion won't work because there's no way to resolve conflict over which purported branch is valid except by reference to the historical block chain data.

[DannyHamilton]

Still looking for confirmation on this or a "you're suggestion won't work for reason(s) XYZ."

Explaining why your solution won't work in a way that you can understand will first require you to have a much better understanding of the problem that bitcoin solves, the way that is solves it, and the technical details of exactly how a transaction works.

Of course, once you have all that knowledge, the "reason(s) XYZ" will become obvious to you, and you'll be here with the rest of us trying to explain to the newbie what you've already tried to explain to dozens of other newbies that thought they were the first to come up with this solution.

I don't have time to take you through all the education you need at the moment, I'm getting sleepy.  If you don't have a satisfying grasp of the problem with your solution in the next day or so, I'll try to return and walk you through the steps necessary for you to understand the technical details of bitcoin.

Note:  At this point you have 5 "Hero Members" all trying to tell you that the idea has been suggested before, the idea has been analyzed, and that the idea won't work. While the moniker "Hero Member" is mostly meaningless (It just means that the member has been here a long time and is verbose in their posting frequency), it seems statistically likely that at least one person that has been here that long and has participated that frequently would have gained some useful knowledge by now.  The fact that there are 5 that are all saying the same thing, should cause you to start thinking along the lines of "there appears to be something here that I need to understand better" rather than "there appears to be something here that they all need to understand better".  It's not impossible for 5 experienced people to all fall into the same faulty line of thought and assumption, but the odds are that you should give some consideration to the possibility that there's something significant that you're misunderstanding.

[Realpra]

This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

[sugarpuff]

This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

Thank you sir! *That* is what I was looking for. I will do as you suggested and search for those terms.

If you're feeling especially kind, could you summarize the current status of the "ledger-solution"/"rolling-root proposal"? Did you already summarize it by saying, "The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin"?

And does that mean that it actually is a potential solution (as vetted by people with brains</ducks!>)?

[sugarpuff]

Realpra, sorry, I tried, but my google-fu is failing me. The closest thing I was able to find is this post of yours mentioning it. If you could provide some history/links it'd sure be appreciated!

[sugarpuff]

Your suggestion won't work because there's no way to resolve conflict over which purported branch is valid except by reference to the historical block chain data.

Unless I'm misunderstanding you, I think I addressed that in an earlier reply:

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.