Gold collapsing. Bitcoin UP.

[cypherdoc]

Not normally a fan of Kashmir Hill but this was well written:

http://fusion.net/story/125475/ai-weiwei-jacob-appelbaum-and-laura-poitras/

[1713583322]

1713583322

[1713583322]

1713583322

[1713583322]

1713583322

[1713583322]

1713583322

[sidhujag]

Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.

There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run (and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits.

In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain.

Kinda like bittorrent ?

Not exactly. BitTorrent distributes the entire file to all peers and in fact it actively tries to distribute pieces with a low count more quickly. In this case, not every node would have every block by design. It's a bit more dangerous what is being proposed. An attacker can prevent new nodes from syncing by finding all of the nodes that have a particular piece of the blockchain (whichever piece is rarest) and attacking them.

Still, as long as there is one good copy of the blockchain in the world, it can always be sent out again. So I don't think we need to worry too much. Some people will still run nodes without pruning.

Isn't the pruning approach Satoshi alluded to in his whitepaper #7 (https://bitcoin.org/bitcoin.pdf) the better approach? The hash of the block remains so full verification is still possible after pruning.. blocks are still around just the spent tx's are gone... so you can reindex to get them again if you wish.

No, unless you can verify the flow of transactions yourself you are ultimately trusting the miners. They put the transaction in the block; the hash proves that. But how do you know they were not cheating when they did? Also, you don't know that they didn't prune something they shouldn't. Maybe you got a payment, and the miner doesn't want you to see it (or the government told him to remove it), so they prune it (retaining only the hash, but the block is still valid).

Satoshi's pruning is basically the same in terms of trust as SPV.

In the back of my mind im still intertwining the concepts of bandwidth/time required for sync versus storage requirements of the blockchain. The bandwidth/time was solved by SPV and perhaps ultimately for desktop users be useful to have a lite wallet download which would be in SPV node and have a default trust of a bitcoin foundation node or something configurable OOTB.. so avg joe can simply click on it and run, but with the fine line that its only for new users who don't already have a wallet.

The storage which isn't really that big of a problem but solves being able to store on smaller flash disks is now solved by the pruning of old blocks of spent outputs.

Correct?

The current pruning scheme that is implemented compromises nothing in terms of trust. You receive the entire blockchain and verify it completely, but you only keep some N recent blocks (or maybe specified amount of storage? i'm not sure). It uses the same bandwidth as before, but less storage.

This doesn't really address where those old blocks will be stored, so that is being worked on as discussed in the last several posts.

If you want to reduce bandwidth you are going to end up with various weaker models in terms of trust because since you aren't receiving the entire chain, you obviously can't verify everything yourself. You end up needing to trust somebody somehow.

SPV is fine for end users, and it has pretty much the minimum possible bandwidth usage. It does have some privacy compromises because you have to tell the nodes which addresses you are interested in (or at least a superset of that).

Satoshi's pruning is ultimately similar to SPV in terms of trust (you are trusting the miners) but has intermediate bandwidth usage (less than full trustless verification and more than SPV) and unlike SPV does allow mining.

If you are a new user why would you want to reverify transactions you don't care about?

[smooth]

If you are a new user why would you want to reverify transactions you don't care about?

1. You might want to know that transactions aren't violating the rules that would compromise the integrity of the system (and therefore the value of your own coins).

2. How do you know the coins you receive are real? In theory you could trace back only the coins you receive in an SPV-line manner, but as coins get split and combined this seems in practice to grow to much or all of the chain before long.

3. Maybe you in fact don't care and would find a higher trust level acceptable.

[sidhujag]

If you are a new user why would you want to reverify transactions you don't care about?

1. You might want to know that transactions aren't violating the rules that would compromise the integrity of the system (and therefore the value of your own coins).

2. How do you know the coins you receive are real? In theory you could trace back only the coins you receive in an SPV-line manner, but as coins get split and combined this seems in practice to grow to much or all of the chain before long.

3. Maybe you in fact don't care and would find a higher trust level acceptable.

A new user wouldn't have any coins yet.... they would first need an address. So #1 doesn't really apply they are trusting the node they are connected to (blockchain foundation or something)... the people are care about security would fully verify. Those that import their pvt key would reverify too.

You trust the foundation in the same way that you must trust that they won't commit bad code that may compromise the value of their coins aswell.

[smooth]

If you are a new user why would you want to reverify transactions you don't care about?

1. You might want to know that transactions aren't violating the rules that would compromise the integrity of the system (and therefore the value of your own coins).

2. How do you know the coins you receive are real? In theory you could trace back only the coins you receive in an SPV-line manner, but as coins get split and combined this seems in practice to grow to much or all of the chain before long.

3. Maybe you in fact don't care and would find a higher trust level acceptable.

A new user wouldn't have any coins yet.... they would first need an address. So #1 doesn't really apply they are trusting the node they are connected to (blockchain foundation or something)... the people are care about security would fully verify. Those that import their pvt key would reverify too.

You trust the foundation in the same way that you must trust that they won't commit bad code that may compromise the value of their coins aswell.

I think you don't understand the concept of trustlessness. The design of bitcoin doesn't trust a foundation. Yes, in practice we have to trust the developers today because the system isn't fully built yet, although maybe that's not true. If there were no more commits ever, it still might continue to work indefinitely in some manner (obviously blocks size limit would be an issue, so if it did continue to function it would be some kind of gold-like backing asset, not transactional).

Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.

[cypherdoc]

If you are a new user why would you want to reverify transactions you don't care about?

1. You might want to know that transactions aren't violating the rules that would compromise the integrity of the system (and therefore the value of your own coins).

2. How do you know the coins you receive are real? In theory you could trace back only the coins you receive in an SPV-line manner, but as coins get split and combined this seems in practice to grow to much or all of the chain before long.

3. Maybe you in fact don't care and would find a higher trust level acceptable.

A new user wouldn't have any coins yet.... they would first need an address. So #1 doesn't really apply they are trusting the node they are connected to (blockchain foundation or something)... the people are care about security would fully verify. Those that import their pvt key would reverify too.

You trust the foundation in the same way that you must trust that they won't commit bad code that may compromise the value of their coins aswell.

I think you don't understand the concept of trustlessness. The design of bitcoin doesn't trust a foundation. Yes, in practice we have to trust the developers today because the system isn't fully built yet, although maybe that's not true. If there were no more commits ever, it still might continue to work indefinitely in some manner (obviously blocks size limit would be an issue, so if it did continue to function it would be some kind of gold-like backing asset, not transactional).

Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.

You keep saying this. Why?

[Peter R]

…
Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.

You keep saying this. Why?

Because trustlessness is the foundation upon which Bitcoin is built.

My node could leave the network for a few months, and, upon rejoining, determine the state of the ledger without requiring any new information beyond what is encoded in the candidate blockchains.  To do this, my node considers all valid chains and selects as the "best chain" the one with the highest cumulative work.

If it was not possible to verify every transaction (including the pruned ones) then how would my node know that nothing "funny" occurred while it was not connected?  Maybe some coins got moved without valid signatures?  [That being said, I'm still supportive of pruning--it's just that it must always remain possible for nodes to verify all transactions right back to the genesis block].  

Remember, this trustlessness is one property that separates a PoW system from a PoS system.  For example, a Nxt node needs new information upon rejoining the network (what they call economic clustering) in addition to what's encoded in the candidate blockchains.  I think this "not-quite-trustless-by-design" property of PoS is what Vitalik et al. refer to as "weak subjectivity."

[smooth]

That being said, I'm still supportive of pruning--it's just that it must always remain possible for nodes to verify all transactions right back to the genesis block

Agree, in case that was not clear from my earlier comments.

[cypherdoc]

…
Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.

You keep saying this. Why?

Because trustlessness is the foundation upon which Bitcoin is built.

My node could leave the network for a few months, and, upon rejoining, determine the state of the ledger without requiring any new information beyond what is encoded in the candidate blockchains.  To do this, my node considers all valid chains and selects as the "best chain" the one with the highest cumulative work.

If it was not possible to verify every transaction (including the pruned ones) then how would my node know that nothing "funny" occurred while it was not connected?  Maybe some coins got moved without valid signatures?  [That being said, I'm still supportive of pruning--it's just that it must always remain possible for nodes to verify all transactions right back to the genesis block].  

Remember, this trustlessness is one property that separates a PoW system from a PoS system.  For example, a Nxt node needs new information upon rejoining the network (what they call economic clustering) in addition to what's encoded in the candidate blockchains.  I think this "not-quite-trustless-by-design" property of PoS is what Vitalik et al. refer to as "weak subjectivity."

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

[Peter R]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). 

[cypherdoc]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). 

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

[justusranvier]

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security).

I'd like to think it's within the realm of possibility to create a ZKP that a given set of headers is only composed of the hashes of valid Bitcoin transactions.

If such a proof existed, it would be safe for the entire network to discard the old history.

[smooth]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). 

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Yes I agree. I would feel a bit better if there were an incentive to do it, but that isn't hard to imagine. Something like a small payment per block served should be enough to incentivize people to hold it (and also to share it when needed).

[rocks]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). 

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

It does reduce the trustless aspect, but only slightly. With this you'd have to trust that the P2P network rejected any attempts to insert an invalid UTXO hash (which has the effect of being able to arbitrarily change unspent outputs). But to attack this, you essentially would have to take over the P2P network. Plus given how visible the blockchain is, I'd say any attempts to do this would be noticed and rejected in real time.

As the total blockchain gets very very large, I think we are going to need something like this. A UTXO hash would enable full nodes to operate without ever downloading past blocks, you'd only need headers (which is small) and the UTXO set. Otherwise we are forced to have new nodes download petabytes of data they'll never keep. Network BW is still a limited resource, so this aspect of bitcoin will become more and more of an issue.

[cypherdoc]

A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

[cypherdoc]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). 

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well.

[smooth]

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

Again we are going to disagree a bit about what a "full node" means. To be full full node, you have to be able to serve blocks to peers who need them. When people talk about running full nodes to support the network, that is one form of support they are providing. Relaying is another. A pruned node only does the latter, not the former.

[Peter R]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security).  

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well.

Right, and now we're back to one of Smooth's points that a network full of nodes that don't personally verify every transaction in each block back to the genesis block (e.g., by relying on UTXO commitments instead) doesn't offer the same level of trustlessness than one that does.

[cypherdoc]

Also, the block chain of a full node is never accessed except to serve blocks  to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.

Again we are going to disagree a bit about what a "full node" means. To be full full node, you have to be able to serve blocks to peers who need them. When people talk about running full nodes to support the network, that is one form of support they are providing. Relaying is another. A pruned node only does the latter, not the former.

Agreed. But let me give you a practical consequence of what has happened.

I look at this situation and think instead of converting my current full nodes to pruned  nodes, I am thinking of how i will add at least a dozen or more new pruned nodes on those little compute sticks from Intel to every open, free wifi connection I can get my hands on.

[smooth]

I guess I don't understand. You've verified everything up to the time you exit the network. You have an  up to date UTXO set and the last 2016 blocks or so. When you return, you can  download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?

Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security).  

I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia.

Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node.

Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well.

Right, and now we're back to one of Smooth's points that a network full of nodes that don't personally verify every transaction in each block back to the genesis block (e.g., by relying on UTXO commitments instead) doesn't offer the same level of trustlessness than one that does.

Yes there are some interesting divergences between what it is reasonable to do as an individual (especially when the rest of the network does not take shortcuts) and what you want the network to do as a whole. I smell a bit of a free rider problem here.