Gold collapsing. Bitcoin UP.

[smooth]

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

Again we are going to disagree a bit about what a "full node" means. To be full full node, you have to be able to serve blocks to peers who need them. When people talk about running full nodes to support the network, that is one form of support they are providing. Relaying is another. A pruned node only does the latter, not the former.

Agreed. But let me give you a practical consequence of what has happened.

I look at this situation and think instead of converting my current full nodes to pruned  nodes, I am thinking of how i will add at least a dozen or more new pruned nodes on those little compute sticks from Intel to every open, free wifi connection I can get my hands on.

Yes I'm just making a point about terminology more than anything else. Previously we just had full nodes and clients. Now we have another category of nodes to consider. I don't think its bad that people can run nodes on smaller hardware.

[1714207905]

1714207905

[1714207905]

1714207905

[1714207905]

1714207905

[thezerg]

So it seems like there are two approaches:
  
The "strong" validation model allows anyone at any time (T) to validation the entire history.
The "weak" validation model allows anyone to validate T to T-N back in history.  We assume that at any time T1 that is currently unavailable there were people validating it.  

I like the strong model but in practice I think the weak one works fine and could be very useful for a micropayment sidechain.  To explain, consider this thought experiment:  Tomorrow somebody reveals that they've been mining all along and releases a competing blockchain starting from the genesis block that is longer than the current one and gives them all the coins.  Do you REALLY think all the people involved in Bitcoin would just be like "OK" we'll use your chain?  No way.  We'd issue a patch to the client that prefers "our" chain and probably create some hopefully temporary and unfortunately-centralized solution where Gavin or someone periodically stamps what he thinks is the valid chain.  The point is that the fact that the chain is globally accepted today, yesterday and last year actually makes it stronger than the math.


Here's an alternate idea:

create a hash of the UTXO sorted set (ok a Merkle tree of hashes would be better) and periodically issue it in a dummy txn.  Today, that can be done via the extra bytes.  Now anyone can use your published, hashed UTXO set (presumably its shared P2P), if they trust you.  

If the above is useful, next we create a soft-fork new op code that identifies this as a hash of the UTXO and miners don't accept block with an invalid UTXO.  Miners must put this UTXO hash in every (block number%31 ==0).  Clients don't use the most recent UTXO hash but the prior one.

So now we can have a very small client that is also trustless (well, a single miner would have to find 32 blocks in a row or 51% the network to fool a client).  Small clients could use even older UTXO sets if they are less trusting of the diversity of the miner network.  

The only problem with the above is that today 51%ing the network only lets you get all the newly minted coins and deny the ability of others to spend.  This is a pretty cool feature

But with the above a 51% miner could "mint" new coins (at least from the perspective of the small clients) and remove existing ones by creating a fake UTXO set.  You'd have to remove existing UTXOs or the # of coins would not match the coinbase issue rate.  But if a 51%er ever did this, the non-small clients would see it instantly and so there would presumably be a huge PSA.  So this idea is acceptable within the "weak" validation model...

[cypherdoc]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security).  

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well.

Right, and now we're back to one of Smooth's points that a network full of nodes that don't personally verify every transaction in each block back to the genesis block (e.g., by relying on UTXO commitments instead) doesn't offer the same level of trustlessness than one that does.

Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding  any blocks. But I do like the idea of adding a  hash of the UTXO set into each block to ensure it's integrity.

[cypherdoc]

Here's an idea for some young entrepreneur that will greatly expand the number of  network nodes.

Instead of using a USB adapter to power these small compute sticks, use an AC adapter to power them. Load in the pruned 1.3GB block chain and connect them to a nearby open wifi in some obscure outlet. I know I have access to lots of open and closed wifi systems.

[cypherdoc]

Imo, the volatility we're starting to get to the upside, OKcoin last week and BTC-e  today,   is a bullish sign. 

[cypherdoc]

Here's an idea for some young entrepreneur that will greatly expand the number of  network nodes.

Instead of using a USB adapter to power these small compute sticks, use an AC adapter to power them. Load in the pruned 1.3GB block chain and connect them to a nearby open wifi in some obscure outlet. I know I have access to lots of open and closed wifi systems.

Oh yeah, don't forget to build in a  miner. 

[smooth]

Here's an idea for some young entrepreneur that will greatly expand the number of  network nodes.

Instead of using a USB adapter to power these small compute sticks, use an AC adapter to power them. Load in the pruned 1.3GB block chain and connect them to a nearby open wifi in some obscure outlet. I know I have access to lots of open and closed wifi systems.

Oh yeah, don't forget to build in a  miner. 

Add solar cells and a mesh network.

[rocks]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

You still have to download the full chain and verify the full chain in order to be able to trust that your state is correct. When the blockchain reaches many petabytes that will be an issue.

Hashing, verifying and committing the UTXO set to blocks enables a node to only download headers and the current UTXO set, and still operate in a trustless manner. That is a massive savings in bandwidth, and one that will be needed.

The current update simply reduces the amount of storage and node requires to operate, it does not reduce the amount of bandwidth a node needs to join the P2P network. Even with the update, you still have to download the full blockchain.

[smooth]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

You still have to download the full chain and verify the full chain in order to be able to trust that your state is correct. When the blockchain reaches many petabytes that will be an issue.

Hashing, verifying and committing the UTXO set to blocks enables a node to only download headers and the current UTXO set, and still operate in a trustless manner.

People throw this word trustless around a lot. It doesn't mean what you think it means. If you are relying on a committed UTXO then you are trusting whoever created and verified that UTXO (meaning the miners for the most part) to have done so faithfully. That may be a reasonable or even entirely necessary trade off to avoid needing to process petabytes of data, but you can't wish away that it is indeed a trade off.

Justus is correct that there could conceivably be a zero knowledge proof that the UTXO is the result of only valid bitcoin transactions which could be trustless. (Maybe, but only if there isn't a trusted setup, not even a distributed one.) Just committing any old UTXO doesn't do that.

[sidhujag]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

You still have to download the full chain and verify the full chain in order to be able to trust that your state is correct. When the blockchain reaches many petabytes that will be an issue.

Hashing, verifying and committing the UTXO set to blocks enables a node to only download headers and the current UTXO set, and still operate in a trustless manner. That is a massive savings in bandwidth, and one that will be needed.

The current update simply reduces the amount of storage and node requires to operate, it does not reduce the amount of bandwidth a node needs to join the P2P network. Even with the update, you still have to download the full blockchain.

Yup and maybe as Smooth says its not practical to create lite wallets which are entrusted by the foundation or some other globally known trustful entity because it may cause havoc if that node(s) was hacked. However in order to do some kind of SPV to save bandwidth when syncing (something we will need) we have to give up some sort of trust somewhere.. so problems still there.. and that is the major problem I have with the bloat, storage is a non issue imo.

[smooth]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

You still have to download the full chain and verify the full chain in order to be able to trust that your state is correct. When the blockchain reaches many petabytes that will be an issue.

Hashing, verifying and committing the UTXO set to blocks enables a node to only download headers and the current UTXO set, and still operate in a trustless manner. That is a massive savings in bandwidth, and one that will be needed.

The current update simply reduces the amount of storage and node requires to operate, it does not reduce the amount of bandwidth a node needs to join the P2P network. Even with the update, you still have to download the full blockchain.

Yup and maybe as Smooth says its not practical to create lite wallets which are entrusted by the foundation or some other globally known trustful entity because it may cause havoc if that node(s) was hacked. However in order to do some kind of SPV to save bandwidth when syncing (something we will need) we have to give up some sort of trust somewhere.. so problems still there.. and that is the major problem I have with the bloat, storage is a non issue imo.

I didn't say it's not practical to relax any of these trustlessness assumptions, just different. Peter R explained more or less the same thing a bit more clearly, I think.

You don't need to trust a central party to have light wallets though, just the collective good behavior of the miners and at least one of the nodes you connect to has to be honest (but you don't have to know ahead of time which one).

[marcus_of_augustus]

Open questions;
i) UTXO hash in blockchain is vulnerable to re-orgs, so it needs to be >120 or 288 blocks deep or w/e your 'trust' level is ...

ii) Also some UTXO could discard/prune long unspent dust/spam or other 'bad' TX so not all UTXO sets will be created equal ...

[thezerg]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security).  

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well.

Right, and now we're back to one of Smooth's points that a network full of nodes that don't personally verify every transaction in each block back to the genesis block (e.g., by relying on UTXO commitments instead) doesn't offer the same level of trustlessness than one that does.

Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding  any blocks. But I do like the idea of adding a  hash of the UTXO set into each block to ensure it's integrity.

LOL I should read the whole thread rather then skimming it.  You guys were already talking about the UTXO hash!   But its easy to miss stuff when multiple conversations are interleaved and you are only interested in one of them :-).  

Note if you make it a UTXO merkle tree whose root is stored in the block (not a flat hash) then a light client could verify that certain UTXOs exist without downloading the entire UTXO set.

EDIT:  And what do you think of the idea of just doing it (as data in a txn), and then if people use it we can incorporate it into every block so that it is trustless?

[brg444]

Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding  any blocks. But I do like the idea of adding a  hash of the UTXO set into each block to ensure it's integrity.

You don't seem to quite get it.

From who do you suppose you will "download" the entire block chain initially.?

I don't care if you stick "validating" nodes on every USB key you can get your hands on. This does not improve the number of COMPLETE copies of the transaction log from which nodes can sync on the network.

Did you bother reading the last link I posted? To quote

Finally (for the purposes of this piece), there is the immutable transaction log. The log demonstrates incontrovertibly the complete heredity of each and every satoshi, from coinbase to unspent output. This globally-shared, append-only log cannot be disentangled from that which is Bitcoin. Nodes that run with "pruned" blockchains will still always depend on the existence of nodes with unpruned blockchains from which to sync. Perhaps in some idyllic (for the fiat shitgnomes) world, the entire Bitcoin network would switch over to this "headers first" synchronization method, bypassing the all-important verification of each transaction in each block.

So much for inclusion and "nodes" propagation when only people who can afford petabytes of storage can run the only nodes that matters and that are foundation to the Bitcoin network

[STT]

"centralized blockchain", at a very fundamental level does not understand what Bitcoin is.

The dollar is a digital currency pretty much, so what he appears to be suggesting is the status quo and this technology is pointless.    A problem with education is its biased to government, partly from the funding and many other cross overs.    Bitcoin fail or suceeds, it was never pointless and it is very much something new and never seen in this world before.  Whether we need it Im not sure yet

[cypherdoc]

Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding  any blocks. But I do like the idea of adding a  hash of the UTXO set into each block to ensure it's integrity.

You don't seem to quite get it.

From who do you suppose you will "download" the entire block chain initially.?

for one, you'll be able to get it from me.

and you don't seem to get that if we increase the # of nodes by orders of magnitude larger than what we have now, even if just pruned nodes, that Bitcoin's exchange value will likely increase along with it making it much easier financially for those of us to continue committing to providing full blockchain server nodes.  it also will help decentralize mining.

[solex]

Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding  any blocks. But I do like the idea of adding a  hash of the UTXO set into each block to ensure it's integrity.

You don't seem to quite get it.

From who do you suppose you will "download" the entire block chain initially.?

I don't care if you stick "validating" nodes on every USB key you can get your hands on. This does not improve the number of COMPLETE copies of the transaction log from which nodes can sync on the network.

Did you bother reading the last link I posted? To quote

Finally (for the purposes of this piece), there is the immutable transaction log. The log demonstrates incontrovertibly the complete heredity of each and every satoshi, from coinbase to unspent output. This globally-shared, append-only log cannot be disentangled from that which is Bitcoin. Nodes that run with "pruned" blockchains will still always depend on the existence of nodes with unpruned blockchains from which to sync. Perhaps in some idyllic (for the fiat shitgnomes) world, the entire Bitcoin network would switch over to this "headers first" synchronization method, bypassing the all-important verification of each transaction in each block.

So much for inclusion and "nodes" propagation when only people who can afford petabytes of storage can run the only nodes that matters and that are foundation to the Bitcoin network

You are taking the situation to its extremes.

The reality is that in a decent sized Bitcoin economy there will be thousands of businesses who will always run full nodes. Individuals who want to use pruning *should* have the option of storing a percentage of the full blockchain. Even if it is 5% or 1%, as mentioned earlier, the whole blockchain can be rebuilt from segments held by many nodes with this variation of pruning.

Also, storage is not really the problem, in 10 years time petabyte storage should be available to consumers, let alone to high-end users. 3D (even 5D) glass storage has huge potential.
Bandwidth is the real limiting factor, and fortunately there are major efficiencies to be had there.

[justusranvier]

You don't seem to quite get it.

From who do you suppose you will "download" the entire block chain initially.?

So we have people who want a service and they aren't sure how they are going to get it. What a tough problem.

If only there was this technology that could help people cooperate to get what they want more effectively than direct barter. Something that people could exchange but not consume directly, in order to help them keep track of whose turn it is to consume the products and services being produced. Wouldn't that be a great invention?

You know what would be even better? What if anyone who wanted to provide a service was free to do so, and the people who wanted to consume the service were free to choose any provider which which they could come to a mutual agreement, or no provider at all?

I wonder how good of a job such a mechanism might do at making sure that the number of suppliers was sufficient to meet the demand for the service? Could that improved barter trading thing maybe play role? Perhaps there's some kind of signal that might be passed through it that could be useful.

It feels like we're so close to having all the technology we need to solve problems of how people are going to be able to get what they want via mutual trade. Perhaps in some far off future utopia a more advanced civilization just might finally crack this Enigma.

[rocks]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

You still have to download the full chain and verify the full chain in order to be able to trust that your state is correct. When the blockchain reaches many petabytes that will be an issue.

Hashing, verifying and committing the UTXO set to blocks enables a node to only download headers and the current UTXO set, and still operate in a trustless manner.

People throw this word trustless around a lot. It doesn't mean what you think it means. If you are relying on a committed UTXO then you are trusting whoever created and verified that UTXO (meaning the miners for the most part) to have done so faithfully. That may be a reasonable or even entirely necessary trade off to avoid needing to process petabytes of data, but you can't wish away that it is indeed a trade off.

Justus is correct that there could conceivably be a zero knowledge proof that the UTXO is the result of only valid bitcoin transactions which could be trustless. (Maybe, but only if there isn't a trusted setup, not even a distributed one.) Just committing any old UTXO doesn't do that.

As stated in my original post, you'd have to trust the P2P network to reject blocks with invalid UTXO hashes. So yes you are relying on that processes work correctly, which requires more trust than we have today (i.e. from fully trustless to trusting the P2P network) and does introduce a potential attack vector.

However given how visible the blockchain is, I find it difficult to imagine a scenario where someone effectively pulls off the attack. It would be flagged immediately and all honest miners would reject it and go to a valid chain.

I think attacking a UTXO hash will prove to be as plausible as a 51% attack, which is not likely.

[rocks]

Open questions;
i) UTXO hash in blockchain is vulnerable to re-orgs, so it needs to be >120 or 288 blocks deep or w/e your 'trust' level is ...

ii) Also some UTXO could discard/prune long unspent dust/spam or other 'bad' TX so not all UTXO sets will be created equal ...

The way I've envisioned it is the UTXO hash becomes a part every block's structure (just as the coinbase transaction) and it serves as a marker for the UTXO set as of the current block. This enables nodes to optionally ask for: a) all prior blocks, or b) the UTXO set only.

If there is a re-org, then the new chain's blocks simply have their own UTXO hash in each block and provide a valid reference for the UTXO set on that branch. Each separate chain would have their own UTXO hash that is valid at every block. The cost is adding 256 bits to each block, but this doesn't even have to be a part of the header structure and instead is part of the block itself.

During a re-org a node could ask for either: a) all new blocks since the branch or b) the UTXO set of the latest block on the new chain. Both would be valid options to function on the new branch.

What I like about this is it allows each node to make optimal decisions that alleviate bandwidth pressure on the network. Every time a node restarts after x number of days, it could quickly determine which is the fastest method to catch-up, a) download the newest block or b) download the UTXO set, whichever is smaller. This would allow nodes to minimize their download requests.