Bitcoin Forum

...to handle the number of transactions that visa handles in 3 months the bitcoin system will require 14 Terabytes of storage space.

To assume that people are going to be willing to buy 14 terabytes of disk space every 3 months (even if it is just $40 in 2020) is a leap.

This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

How else would you bring new nodes online?

How do you reach "internet-agreement"?

How do you reach "internet-agreement"?

Still don't get what the problem is...

If I "sent" you coins and you didn't have the block which contained the transaction which contained that output how would you validate the input is valid and not just some nonsense I sent you because I know you have no way of validating?

That just wouldn't be possible. It would be discarded like all other invalid transactions.

You wouldn't be able to send to or from an outdated address (it would be considered invalid). You'd use the new ones that were copied as new transactions to the head. It's possible I am misunderstanding (I've only read the original paper a couple times, and perused the wiki, but I'm not working with the code itself).

Sidenote: consensus about an ancient block should be extremely high. There should be no disagreement about it.

Discarded by whom?  "Considered invalid" by whom?

Which ancient block?

Bitcoin is an imperfect solution to a very hard problem.  You can't just handwave it all away like Dilbert's boss.

Only the private key holder of the tx output can create a new tx.  There is no mechanism to "copy the outputs that were coped as new transactions to the head".

The restrictions are there for a good reason, to encourage learning before teaching.

The problem you are about to tackle is hard. I recommend this thread to get an insight of its complexity.

https://bitcointalk.org/index.php?topic=88208.0

Still looking for confirmation on this or a "you're suggestion won't work for reason(s) XYZ."

This was debated before, what you call  "rolling root" we called "ledger-solution"/"ledger-block".

Same idea; every say 1 year you would take all the 20 years or older transactions and move their unspent outputs to the latest "ledger block". Amounts in the ledger block would be much like coinbase transactions/miners fees.
Barring minor minor data growth from address fragmentation and perhaps block headers, this puts a quite final limit to the blockchain size.

Another idea (of my own) was "swarm clients" that individually would validate only parts of the blockchain, but as a whole would validate the whole thing - all zero trust etc..
This would allow for almost the smallest devices to participate in block validation forever.

You can google these terms, and I do mean google, the forum search is shite.

The main reason nothing has happened yet is laziness and people too busy getting rich off of Bitcoin - and no you can't lump me in with that group I have been hard at work helping Bitcoin for a while now.

Your suggestion won't work because there's no way to resolve conflict over which purported branch is valid except by reference to the historical block chain data.

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.

If you are a major exchange and are making $1B a month in profit (and you would if Bitcoin is so huge it has higher tx volume them VISA) do you think it might be worth it to pay $1,000 a year in additional storage cost to add some drives to the SAN?  I think it might be.

With that much data, what do you think would be a reasonable estimate of the disk I/O?

True VISA scale (i.e 5,000 tps on blockchain)?

Figure 5,000 transactions per second.  Average tx is 400 bytes that is maybe 2MB/s (16 Mbps) sustained to write to the disk.  Average block would be 600 * 5,000 * 400 / 1024^3 = 1.2 GB.  So pretty much nothing.  Now to validate txs requires reading the outputs from the UXTO so maybe double that in terms of random reads.  Still we are talking a tiny fraction of the IO that even high end disks (to say nothing of SANs) can handle today.

Everyone thinks of disk first because it is the most obvious but in my analysis of the resource requirements I see it as the least critical bottleneck for full nodes.  If I had to rank them it would be node bandwidth (especially for residential connections), then memory, then computing power, and way back in last place would be disk capacity and speed.

Edit2 March 5, 2014: Something not mentioned here is that bitcoind can (and might already in some ways do) prune its locally stored blockchain, but only after it downloaded the entire thing once. Thus, this issue is more of a problem (I think) for new full-nodes, which currently must download the entire blockchain before they can prune the entire thing.

1:31:31 PM sipa: sugarpuff: now, about dealing with ways to improve performance, IF you are giving up zero trust
1:31:35 PM sipa: sugarpuff: see it this ways
1:31:47 PM sipa: sugarpuff: internally, every full node maintains a database of all unspent transaction outputs
1:32:13 PM sipa: sugarpuff: this is a high-efficiency, compact database with just outputs (not signatures for example) and not those that have already been spent
1:32:31 PM sipa: sugarpuff: every block you can see as sort of an authenticated patch against that database
1:32:45 PM sipa: (add outputs of trenasactions in the block, remove the outputs spent by transactions in it)
1:33:08 PM sipa: if we can somehow agree on a snapshot of that database at some point in time, and you trust it, we can just copy the database
1:33:13 PM sipa: instead of copying all blocks before it
1:34:01 PM sugarpuff: sipa: right, and that sounds like going into checkpoint territory. the rolling-root proposal is similar but not entirely the same
1:34:27 PM sugarpuff: but old blocks should have almost unanimous agreement in the network
1:34:36 PM sipa: sugarpuff: not done yet :)
1:35:07 PM sipa: sugarpuff: there are proposals to make such a snapshot of that database essentially for every block, and put a hash of that snapshot in the block itself
1:35:14 PM sipa: sugarpuff: and have this validated by miners too
1:35:43 PM sugarpuff: sipa: ah, yeah, that's a good idea. is that what UXTO is about?
1:35:56 PM sipa: the UTXO set is just what we call that database
1:36:10 PM sugarpuff: ah, lol, i just got the acronym
1:36:11 PM sipa: (Unspent TransaXtion Outputs)
1:36:14 PM sugarpuff: :)
1:36:29 PM sipa: now, this has a very significant impact on the network
1:36:41 PM sipa: as validating and maintaining these database snapshots is not cheap
1:37:16 PM sipa: (there are ways to make it pretty efficient, by using a merkle tree for that entire database, but the impact is not neglectable)
1:37:48 PM sipa: now, you can have a node that just asks a peer, "give me your current UTXO set", and it can validate that at least the majority of hashing power agrees with it
1:38:02 PM sipa: without needing to downloading any old blocks
1:38:36 PM sugarpuff: sipa: just by asking enough nodes for it and assuming the majority is right?
1:38:56 PM sipa: sugarpuff: no no, not the majority of your peers
1:39:06 PM sipa: sugarpuff: you can actually just validate the entire old blockchain, but only the headers
1:39:14 PM sipa: sugarpuff: which is very cheap (80 bytes per block)
1:39:18 PM sipa: sugarpuff: like SPV nodes do now
1:39:31 PM sipa: sugarpuff: but the headers would be committing to the hash of these UTXO set snapshots
1:39:42 PM sipa: so you can verify that the chain agrees with the snapshot
1:39:50 PM sipa: which implies the majority of miners do
1:40:31 PM sugarpuff: sipa: i'd like to do more reading on this and not use up more of your time, do you have some recommended links about this topic? Think SPV + UTXO thread is a good start?
1:40:54 PM sipa: several mailing list posts, forum posts, ...
1:41:00 PM sugarpuff: too broad...
1:41:05 PM sugarpuff: i could spend a year doing that
1:41:18 PM sugarpuff: i love the bitcoin paper, i've read it like 3.5 times now
1:41:18 PM sipa: there's no nice centralized knowledge repository of all these experimental ideas :)
1:41:28 PM sugarpuff: ah that's unfortunate
1:42:15 PM sugarpuff: sipa: ok, well thanks, i think what you said here is enough to get me started. i'll go do some more research on it. right now there are too many gaps in my knowledge to give you useful replies on this
1:42:23 PM sipa: :)
1:42:23 PM sipa: yw

Unless I'm misunderstanding you, I think I addressed that in an earlier reply:

New nodes could ask others for the current root then. If nodes lie about it, there are one of two possible scenarios:

- They give a root that is too old. This can be mitigated by asking multiple nodes and picking the majority answer, combined with simply downloading the rest of the blockchain. If it got unlucky and the majority lied, it will eventually catch up and discard the outdated root on its own anyway because of the rolling window that's hard coded.

- They give a root that's too young. This can be mitigated, again, by asking multiple nodes. If it also gets unlucky with its chosen majority, it should be able to figure this out because it's assumed that not all nodes are evil, and it will eventually stumble across a trustworthy one, which would make it have a longer blockchain than the ones that the lying nodes gave it, and it would automatically adopt that one.

I thought this was obvious, but perhaps not:


Downloading the rest of the block chain requires someone storing the entire block chain, no? You can't download something which everyone has deleted.

But besides that you have an even bigger problem: it costs me, or any network attacker approximately nothing to make sure that every single node you connect to is a compromised node I control. This is the problem which bitcoin seeks to solve! And the answer which Satoshi came up with, and the only answer which we have at this time is to validate the entire chain from start to finish and trust the longest valid fork.

This is an invalid assumption. Your ISP could simply block packets from honest nodes, for example.

I just want to point this out, to make sure that no one overlooked it.

Lots of problems get really, really easy once you give up security.

Trusted systems have wildly different properties than trustless systems.  If you don't mind moving from a trustless to a trusted system, then what you are proposing can work*.

Even the posts telling you it won't work are recycled.

The fact is that ancient blocks, at any given point in time, are already highly trusted by the network. If they weren't, Bitcoin would simply not work.
There is already "trust" in the Bitcoin network, for these ancient block

Every node verifies the history, this is how you can be sure that claims like "satoshi premined a billion bitcoins and is secretly waiting to spend them" cannot be true. We do not trust, and cannot trust— because it would be trust founded on nothing.

Only a small number of archival nodes need to store the full chain going back to the genesis block. These nodes can be used to bootstrap new fully validating nodes from scratch but are otherwise unnecessary.

I think you should stop using the word trustless while you have multiple bitcoin developers telling you that you do not understand it.

Except your own Wiki says the opposite:

The way my mind works is that I do not accept BS from anyone. It doesn't matter how large an angry mob gets if what it is saying is complete nonsense.
I've addressed every objection to this proposal, and have received confirmation on it now from two people who seem to know something about Bitcoin: Realpra, and a friend of mine whom I know is intelligent (works on SocialVPN).

The wiki is a public document that anyone can edit. The scalability page is almost entirely the work of a single person who has removed views conflicting with his own onto other pages (including my own, for that matter).  You should be a sophisticated consumer of wiki content and not believe everything you read.

You haven't really addressed _any_ of the objections here, because apparently you do not understand them. I spent quite a bit of time on IRC trying to help you understand but I have failed you. I'm sorry that I'm unable to help further.

Except your own Wiki says the opposite:


From: https://en.bitcoin.it/wiki/Scalability#Storage

Notice how nowhere in that section does it say anything close to "Every node verifies the history.", and instead refers to "archival nodes". It also says "Only a small number" of them are needed.



The way my mind works is that I do not accept BS from anyone. It doesn't matter how large an angry mob gets if what it is saying is complete nonsense.

I've addressed every objection to this proposal, and have received confirmation on it now from two people who seem to know something about Bitcoin: Realpra, and a friend of mine whom I know is intelligent (works on SocialVPN). And while not a vote of recognition, one of the Bitcoin is Vulnerable authors didn't say that the idea wouldn't work when I shared it with him, but said "i'd feel nostalgic if i had to part with the genesis block." :P

Edit: I called it 99.9% trustless, not 100%.

OK, if this is how you guys discuss ideas, no thank you, I've had enough. I'll reply to constructive comments that actually address the merits of the proposal only.

On IRC I agreed that it's not 100% trustless, and frankly you weren't very helpful, just like you're not being helpful here. sipa was the one who actually gave meaningful help and explanations. (None of which, btw, showed that this proposal "lacks security" or anything of the sort.)

So far, this is what has happened here (as far as I can tell):

Me: blah blah blah
Someone: that won't work because of X
Me: That's incorrect because of Y
Realpra: Sugarpuff, your proposal had this other name (indication of agreement).
gmaxwell: "You haven't really addressed _any_ of the objections here"
DannyHamilton: ad hominem attack directed at sugarpuff

OK, if this is how you guys discuss ideas, no thank you, I've had enough. I'll reply to constructive comments that actually address the merits of the proposal only.

I really don't understand why you are discussing this.  Clearly you've come up with an idea that no one ever has before.  Now you just need to code it up and the whole world will switch to your fork.

However even if the cited section was implemented it would still be trustless.   Full nodes would still maintain a pruned copy of the blockchain going all the way back to the genesis block.  An unpruned copy (the archive node) is not required to validation transactions or create new blocks.   Nobody needs to trust the archive nodes.  

Your "solution" (and I use the term lightly) would break that trustless relationship between nodes.

you've refused to do that basic research*.

But for the sake of a tiny increase in trust, you get a network where new full nodes can be brought online without having to download several hundred terabytes of data. EDIT: that still retain their ability to verify all the way back to the root, on their own time.

You see, as I've mentioned, I have read the Bitcoin paper. Several times. I understand it well.

The Bitcoin paper points out this interesting property of the blockchain: that the longer it gets, the harder it gets to fake. Impossibly harder.

The relationship indeed becomes no longer 100% trustless as I've acknowledged multiple times.

If Bitcoin gets to that point, you will come back to this proposal, and you will see it in a different light all of a sudden.

False. (https://bitcointalk.org/index.php?topic=501039.msg5530547#msg5530547)

No you don't or you wouldn't make mistakes like saying "transactions which are spent" or the one you just repeated below.

You also would have realized the SPV clients exist.

Your "solution" doesn't work but even if it did would be inferior to SPV which is something Satoshi came up with before the blockhain even started.

This is a concern in a situation where an SPV client is subjected to a double-spend attack by somebody who controls its network connection. For example, suppose you are at a wi-fi cafe and are paying for something using your smartphone -- the cafe owner controls your network connection. Satoshi acknowledges this implicitly when he writes that "the verification is reliable as long as honest nodes control the network" -- to be completely pedantic, this means that the verification is reliable as long as honest nodes control the part of the network that the SPV client is able to communicate with. In an attack-by-ISP scenario this may not be a sufficiently strong security property. The attacker would not need to overpower "the rest of the network" because the client is unable to communicate with it.

This isn't in a vacuum that property only applies because for a given tx, the tx, and everything that is needed to validate that tx is also deeper in the blockchain.  You can't throw that away and then assume the same dynamic is in play.

Any system which is not 100% trustless is inferior to the system we have now.  Why would you want to make the protocol worse.

You haven't proposed anything other than drop all method of verifiy a transaction

, throw it into a new block at the head of the chain and hope nobody abuses that.

Pruning is very likely to keep the storage requirements modest.  It is likely Bitcoin will never need a ledger snapshot system but if it did it would require more thought that what you propose.  Still the true bottlenecks to transaction volume are node bandwidth and memory.

a) SPV clients have copy of all block headers

b) there is no such thing as "balances" at an address.  Bitcoin works on the concept of inputs and outputs.

d) "The transactions containing them can simply be moved automatically according to some strict rules that I don't care enough to come up with"  That is why you have no proposal.  "Um yeah we can do this stuff but it will require some stuff but I don't really fill like saying the stuff but if you don't agree with me you are jerks and just trolling.  We can fill in the stuff later and stuff."

What language do you speak then? Special forum language?

Here is an address: 18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs (https://blockchain.info/address/18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs)

Click on that link and you will see on the resulting page a label that says Final Balance, indicating what I think is a "balance", at an "address".

People insist on retaining a full copy of blockchain locally, supposedly for the purpose of protecting the interest of every Bitcoin user, nevertheless, if the anonymization of coins through Coinjoin/CoinSwap/Stealth payment...becomes widespread in the future, those who participate in such txs will have absolutely no interest in you keeping a record of them and couldn't appreciate you more for pruning them.

Those anonymizing techniques are interesting specifically because the records of transfer are not kept on-chain.

Well, this farce has gone on long enough.  By now it should be perfectly clear to anyone stumbling across this thread that you are a moron with no interest in how bitcoin actually works.

Unspent outputs are only a transactionID and an output number.   In other words , any spending of a Bitcoin requires this sha256 hash and a number referring to the output and the script therein.

Bitcoins dont exist. Inputs and outputs do.

We define an electronic coin as a chain of digital signatures.

Find the lowest target in the blockchain (remember difficulty can decrease due to a split or other reasons).  If that is less than 100,000 (selected above) then choose the candidate purge point to start at current-100,000 blocks.  If it is more than 100,000 blocks away from the current block, candidate purge point starts at that location.

At the request of "HoboJerk" (http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/#comment-1442376730), I've revived this thread.

OK, I decided it's about time I dive into the details of UTXO to see whether or not you are correct. After spending about an hour reading and thinking about it, I think you might be correct.

Here is my current understanding, please let me know whether I've understood it correctly (you too @HoboJerk):

For a new node to boot up from scratch and begin securing the network, approximately the following needs to happen:

1. Download entire BTC blockchain headers.
2. Download entire UTXO blockchain headers. EDIT: download the entire UTXO meta chain.
3. Begin merged mining on UTXO and BTC blockchains.
4. For each new transaction the "almost-full node" receives, query other nodes for the block in which the payer's bitcoins were previously located, along with the hashes to verify the merkle root in the BTC blockchain. Verify the root can be found.
5. Query other nodes for the transactions and merkle root hashes that result in the merkle root hash of the most recent block in the UTXO blockchain. Verify that previous txn the coins belong to exists in the current data comprising the most recent merkle root of the UTXO blockchain.
6. If the transaction passes the above verifications, store all received data so far in order to build the Merkle hash tree that represents all the UTXOs.
7. Continue mining blocks as per above to secure the network and build new blocks, slowly transforming this light node into a full node.

Is that correct? If that's how it works, then you are both correct UTXO secures the network and brings new nodes online fast (and is actually better, I think, than my rolling-root suggestion).

Copying a reply I posted there (http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/#comment-1442800545) which I am copying here because if my now (possibly complete) understanding of UTXO is correct, then UTXO is a superior solution to my rolling-root idea:

Actually, I'm not sure about that. What's the incentive for nodes to mine on that meta-chain? There doesn't seem to be one, and without a compelling incentive a denial-of-service attack might be possible against UTXO. Since mining on the meta-chain decreases the rewards a miner can expect (if there were to mine only on BTC), they wouldn't mine on it, and that might make it easier for someone to get 51% control over it... I'm not sure how that would play out.

I see that the UTXO thread has some discussion about this issue, but it's too late in the day for me to spend more time on this (work calls!): https://bitcointalk.org/index.php?topic=88208.msg971762#msg971762

Interesting point. Hope you don't mind if I mention your reply in that other thread as well.

So, what is the takeaway from that then? That new lite-nodes can use UTXO to validate arbitrary queries, but they cannot participate in securing the network until they have all the transactions for the leaf nodes of the entire UTXO tree?

OP, It seems like the bottom line is you would need massive changes to bitcoin (with hard fork) to do what you want to do....

the scalability problem is quite manageable so that's why it doesn't make sense.

Every client keeps some kind of a record with a balance for each address

It's time for a hard fork anyway:

http://hackingdistributed.com/2014/06/13/time-for-a-hard-bitcoin-fork/
http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/


There are opinions, and there is data. The latter impresses me more and is linked in the first post.

Then fork it and implement the changes - unless that is what you've been working on since March and are ready to release some alpha or beta code, which would be cool.

Code will give some very good data as to how it works, potential issues etc.  Discussing it here will do little, but since it is open source, implementing will do a lot.

Sounds like "bitfreak!" might have implemented something not quite the same as a rolling root... I'm not sure why he's getting rid of unspent transactions after 1 week though (sounds like a bad design choice, but maybe I misunderstood when I skimmed it).

Yes, and that doesn't prevent the MITM attack that I mentioned.


What language do you speak then? Special forum language?

Here is an address: 18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs (https://blockchain.info/address/18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs)

Click on that link and you will see on the resulting page a label that says Final Balance, indicating what I think is a "balance", at an "address".


... Maybe I will take the time, or maybe I won't. It doesn't much matter in the end does it? If you don't see the potential usefulness of the idea at this point, I don't think you'll see it even if I write the code for it, so why waste my time?

Could you show me the lines from the blockchain (not the website) which show the 'balance' for this address you have selected as I can't find them and I have been searching for hours, thanks.

(reminder of your selected address 18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs (https://blockchain.info/address/18uvwkMJsg9cxFEd1QDFgQpoeXWmmSnqSs))

Now, to answer e4xit:


The balance is calculated from the inputs and outputs to that address. You would need to know how to do arithmetic to figure it out (or have your computer or friend do it for you).

bitfreak: your paper is on my reading list and I have already read part of it. I'd like to get back to you soon on it with my impressions. Your paper will be the ... 3rd or 4th alternative consensus/blockchain proposal that I've reviewed, and it looks promising. Here are some preliminary comments since I suspect that my full review won't come for a while due to deadlines:

- One of the few important things that you're missing is the recent 51% solution to mining pools (http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/). Add that to your protocol/software and you'll go a long way to have a altcoin/altchain of major significance.

- You use "we" frequently in your writings but it sounds like you mean to say "I". If you have multiple people working on this you should make that clear by mentioning their names.

- Your other thread seems to indicate that you're already mining and giving away coins? There are good ways and bad ways of doing this, and depending on which method you choose it can make or break your coin. More fair = more chances of success for the coin (and you). I recommend doing a *lot* of reading on this and seeing how other coins approached the task (and what the end result was). And if you're too lazy to do that, just do whatever Ethereum is doing (Vitalik is a bright guy).

Now, to answer e4xit:


The balance is calculated from the inputs and outputs to that address. You would need to know how to do arithmetic to figure it out (or have your computer or friend do it for you).

I thought you might answer something like "do arithmetic"... I hate to say it but I think that I must agree with some of the other members who have posted here when they say that I think you lack quite a lot of understanding regarding bitcoin: the protocol.

NB. I'm not saying that I think that you don't understand bitcoin: the idea.

- One of the few important things that you're missing is the recent 51% solution to mining pools (http://hackingdistributed.com/2014/06/18/how-to-disincentivize-large-bitcoin-mining-pools/). Add that to your protocol/software and you'll go a long way to have a altcoin/altchain of major significance.

- You use "we" frequently in your writings but it sounds like you mean to say "I". If you have multiple people working on this you should make that clear by mentioning their names.

- Your other thread seems to indicate that you're already mining and giving away coins?

Well I'll keep that in mind but we already have a lot on our plate and I don't feel like pools pose a significant threat. People tend to leave pools when they become too large anyway.

There were several people involved in the development of the scheme and I have two developers working on the implementation. Some of them would prefer to remain anonymous so I refrain from mentioning names when possible.

No, that is a test thread where the client must run in testnet mode. Any coins mined in those tests will be lost, we've already had to reset the blockchain one or two times due to changes we made in the protocol.