looks to me like the $DJI is starting to get dragged down by the $DJT:  also, RUT has broken down below the support line of the ascending wedge, depending on how you draw it:  |
|
|
|
Open questions;
i) UTXO hash in blockchain is vulnerable to re-orgs, so it needs to be >120 or 288 blocks deep or w/e your 'trust' level is ...
ii) Also some UTXO could discard/prune long unspent dust/spam or other 'bad' TX so not all UTXO sets will be created equal ...
The way I've envisioned it is the UTXO hash becomes a part every block's structure (just as the coinbase transaction) and it serves as a marker for the UTXO set as of the current block. This enables nodes to optionally ask for: a) all prior blocks, or b) the UTXO set only. If there is a re-org, then the new chain's blocks simply have their own UTXO hash in each block and provide a valid reference for the UTXO set on that branch. Each separate chain would have their own UTXO hash that is valid at every block. The cost is adding 256 bits to each block, but this doesn't even have to be a part of the header structure and instead is part of the block itself. During a re-org a node could ask for either: a) all new blocks since the branch or b) the UTXO set of the latest block on the new chain. Both would be valid options to function on the new branch. What I like about this is it allows each node to make optimal decisions that alleviate bandwidth pressure on the network. Every time a node restarts after x number of days, it could quickly determine which is the fastest method to catch-up, a) download the newest block or b) download the UTXO set, whichever is smaller. This would allow nodes to minimize their download requests. Wouldn't the UTXO hash have to be in the header since otherwise it could get pruned? |
|
|
|
Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding any blocks. But I do like the idea of adding a hash of the UTXO set into each block to ensure it's integrity.
You don't seem to quite get it. From who do you suppose you will "download" the entire block chain initially.? for one, you'll be able to get it from me. and you don't seem to get that if we increase the # of nodes by orders of magnitude larger than what we have now, even if just pruned nodes, that Bitcoin's exchange value will likely increase along with it making it much easier financially for those of us to continue committing to providing full blockchain server nodes. it also will help decentralize mining. |
|
|
|
Here's an idea for some young entrepreneur that will greatly expand the number of network nodes.
Instead of using a USB adapter to power these small compute sticks, use an AC adapter to power them. Load in the pruned 1.3GB block chain and connect them to a nearby open wifi in some obscure outlet. I know I have access to lots of open and closed wifi systems.
Oh yeah, don't forget to build in a miner. |
|
|
|
|
Imo, the volatility we're starting to get to the upside, OKcoin last week and BTC-e today, is a bullish sign.
|
|
|
|
|
Here's an idea for some young entrepreneur that will greatly expand the number of network nodes.
Instead of using a USB adapter to power these small compute sticks, use an AC adapter to power them. Load in the pruned 1.3GB block chain and connect them to a nearby open wifi in some obscure outlet. I know I have access to lots of open and closed wifi systems.
|
|
|
|
I guess I don't understand. You've verified everything up to the time you exit the network. You have an up to date UTXO set and the last 2016 blocks or so. When you return, you can download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?
Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia. Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node. Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well. Right, and now we're back to one of Smooth's points that a network full of nodes that don't personally verify every transaction in each block back to the genesis block (e.g., by relying on UTXO commitments instead) doesn't offer the same level of trustlessness than one that does. Actually, I wasn't agreeing with his whole concept of just depending on headers and a hash of the UTXO set for a full node. I still think it's important for every new full node to download the entire block chain initially to build a valid UTXO set before discarding any blocks. But I do like the idea of adding a hash of the UTXO set into each block to ensure it's integrity. |
|
|
|
Also, the block chain of a full node is never accessed except to serve blocks to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.
Again we are going to disagree a bit about what a "full node" means. To be full full node, you have to be able to serve blocks to peers who need them. When people talk about running full nodes to support the network, that is one form of support they are providing. Relaying is another. A pruned node only does the latter, not the former. Agreed. But let me give you a practical consequence of what has happened. I look at this situation and think instead of converting my current full nodes to pruned nodes, I am thinking of how i will add at least a dozen or more new pruned nodes on those little compute sticks from Intel to every open, free wifi connection I can get my hands on. |
|
|
|
I guess I don't understand. You've verified everything up to the time you exit the network. You have an up to date UTXO set and the last 2016 blocks or so. When you return, you can download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?
Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia. Personally I like the concept of adding a hash of the current UTXO set to each block. This would enable a new node to only download headers and the UTXO set as of the last block, and that would be enough to start operating as a full node. Yes, thanks for bringing this up; I was gonna say. I think that makes great sense as well. |
|
|
|
|
A couple things. Iirc, the UTXO set was not part of the original Satoshi design; which is why he proposed a slightly different means of trimming down the blockchain. It was added only later by the current core dev team.
Also, the block chain of a full node is never accessed except to serve blocks to new nodes coming online or for reorgs; which is why the new proposal makes you save at least the last 288 blocks. So once you've verified your download, you can prune away yet still function completely as a full node would otherwise do.
|
|
|
|
I guess I don't understand. You've verified everything up to the time you exit the network. You have an up to date UTXO set and the last 2016 blocks or so. When you return, you can download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not?
Yes, that's fine provided there are nodes in the network that can serve you the blocks you haven't verified yourself. A brand new node of course needs to verify every transaction in every block all the way back to the genesis block (to keep the current level of security). I'm very comfortable with that fact. There will always be enough of us that will hold copies of the full blockchain out of purity or shear paranoia. |
|
|
|
…
Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.
You keep saying this. Why? Because trustlessness is the foundation upon which Bitcoin is built. My node could leave the network for a few months, and, upon rejoining, determine the state of the ledger without requiring any new information beyond what is encoded in the candidate blockchains. To do this, my node considers all valid chains and selects as the "best chain" the one with the highest cumulative work. If it was not possible to verify every transaction (including the pruned ones) then how would my node know that nothing "funny" occurred while it was not connected? Maybe some coins got moved without valid signatures? [That being said, I'm still supportive of pruning--it's just that it must always remain possible for nodes to verify all transactions right back to the genesis block]. Remember, this trustlessness is one property that separates a PoW system from a PoS system. For example, a Nxt node needs new information upon rejoining the network (what they call economic clustering) in addition to what's encoded in the candidate blockchains. I think this "not-quite-trustless-by-design" property of PoS is what Vitalik et al. refer to as "weak subjectivity." I guess I don't understand. You've verified everything up to the time you exit the network. You have an up to date UTXO set and the last 2016 blocks or so. When you return, you can download all the blocks from the longest candidate chain and verify them working forward from the UTXO set , can you not? |
|
|
|
If you are a new user why would you want to reverify transactions you don't care about?
1. You might want to know that transactions aren't violating the rules that would compromise the integrity of the system (and therefore the value of your own coins). 2. How do you know the coins you receive are real? In theory you could trace back only the coins you receive in an SPV-line manner, but as coins get split and combined this seems in practice to grow to much or all of the chain before long. 3. Maybe you in fact don't care and would find a higher trust level acceptable. A new user wouldn't have any coins yet.... they would first need an address. So #1 doesn't really apply they are trusting the node they are connected to (blockchain foundation or something)... the people are care about security would fully verify. Those that import their pvt key would reverify too. You trust the foundation in the same way that you must trust that they won't commit bad code that may compromise the value of their coins aswell. I think you don't understand the concept of trustlessness. The design of bitcoin doesn't trust a foundation. Yes, in practice we have to trust the developers today because the system isn't fully built yet, although maybe that's not true. If there were no more commits ever, it still might continue to work indefinitely in some manner (obviously blocks size limit would be an issue, so if it did continue to function it would be some kind of gold-like backing asset, not transactional). Bitcoin without pruning is trustless by design (though maybe not 100% in practice). You can verify everything yourself. With pruning it is not trustless even by design.You keep saying this. Why? |
|
|
|
Did you see the latest commit on pruning? Allows you to specify max disk usage and it prunes old blocks.. validates by reindexing which downloads all chain then prunes. You check it out? I wonder what the point is if youhave todownload the full chain anyway why prune? The main bottleneck is the lenghty sync time not storage space.
I also noticed that it has logic to stop sending blocks requested that have been pruned. So there is assumption that there are full nodes out there to give you the block that others have pruned.
There's also a plan to let you keep certain block ranges and network protocol addition to advertise which parts a node has. That way, full block data can be kept in a distributed fashion while nodes can still run ( and contribute more meaningfully than just as a relay and utxo set provider) even with disk space limits. In fact with that it would be theoretically possible to run the network (fully trustable) without any node having storage space for the whole blockchain. Yes, keeping whole block ranges (randomly determined before pruning) is an important feature of pruning which will hopefully be included in due course. So what do all these pruned nodes do with the blocks they mine going forward? Immediately delete them after they get accepted into the longest chain? |
|
|
|
starting to roll:  |
|
|
|
cool thanks, but I thought unspent outputs would be derived from transactions which would no longer exist in the "backend" node. How does it know the unspent balance without tallying up all of the transactions it was involved in?
The way pruning works is the node still receives all transactions as before. The ones that get spent, it throws away. So the ones left are the ones it keeps in its database. Those are the unspent ones. This requires much less storage. UTXO's have always worked this way; throwing away the spent ones. The UTXO set is very small compared to blocks because it only contains info on unspent data and doesn't contain any signatures or data, thus, this is not what results in less storage. it is the discarding of old blocks. in the case of saving the last 2016 blocks (minimum 288), 351874 of them as of now. |
|
|
|
My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.
In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.
So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space. The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do. In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model. With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode.. No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners). Ahh i see so essentialy those that dont want to sync fully can download a lite wallet which gets blocks via spv mode via a trusted node( ssl rpc).. so pruning doesnt really come into play here? If you prune then you wont be able to get your tx if trusted node pruned that block. So spv solves the blockchain sync time and pruning solves what? Being able to store on small flash disks running to relay transactions? SPV solves being able to create and receive transactions without the bandwidth requirements of a node. Pruning solves being able to run a node that validates and relays transactions (and can even mine!) without storing the entire blockchain. They can even serve SPV clients, because they have all of the unspent transactions, and that's all SPV clients would ever want. These nodes just can't sync new nodes though (at least not by themselves) since they don't have all the old blocks to share. By unspent I assume you mean the ones in mempool. But how would one import their wallet key into an spv thin client backed by a pruned node and get correct balance? It needs tx that have been pruned? i think you are mixing up these concepts. think of 3 types of nodes: 1. SPV clients: only store headers, no blockchain. has to trust other mining peers by measuring their POW. subject to Sybil attacks. 2. full nodes: full blockchain with all historical tx history. heavy data storage demands which is what primarily drives up the cost of running a full node. 3. Pruned full node: in between #1 & #2. highly cost effective as it trims the blockchain down to 1.3GB. initially downloads entire blockchain to verify tx history and build UTXO set and then deletes older blocks up to 2016 blocks back (i hear it can vary depending on a flag with the minimum being 288). pruned nodes won't be able to serve up the blockchain to new nodes but it can validate, verify, relay and mine. they will actually improve mining decentralization. |
|
|
|
My understanding is you download the whole blockchain first for verificatio, then delete all but around the last 200 or so blocks. It works off the UTXO set.
In the case of the compute stick, you could use a separate SSD to initialize the blockchain, then disconnect it. Sticking these things in every nook and cranny could be a real problem for those against Bitcoin.
So for basically to allow for p2p tx and block relaying around the globe thru smaller mediums. The main use case I woulda thought is to reduce time for new nodes to sync up.. Since we know the price of hd is falling faster then the need for that space. The pruning stuff does not reduce sync time at all, though the recent headers-first and syncing improvements do. In order to prune syncing you have to relax the idea of independent verification and rely on someone else telling you that the part of the chain you are seeing is valid (as with SPV nodes). That's a major change to the Bitcoin trust model. With spv you only get headers and maybe coinbase tx correct? So you wont be able to import your pvt key and get your txs in spv mode.. No you tell the server the addresses you are interested in and it gives you those tx. However, since you don't see the other tx you can't be sure that the tx you receive actually represent a valid chain of custody back to a coinbase (unless you trust the miners). Ahh i see so essentialy those that dont want to sync fully can download a lite wallet which gets blocks via spv mode via a trusted node( ssl rpc).. so pruning doesnt really come into play here? If you prune then you wont be able to get your tx if trusted node pruned that block. So spv solves the blockchain sync time and pruning solves what? Being able to store on small flash disks running to relay transactions? SPV solves being able to create and receive transactions without the bandwidth requirements of a node. Pruning solves being able to run a node that validates and relays transactions (and can even mine!) without storing the entire blockchain. They can even serve SPV clients, because they have all of the unspent transactions, and that's all SPV clients would ever want. These nodes just can't sync new nodes though (at least not by themselves) since they don't have all the old blocks to share. actually, pruned nodes can't interact with SPV clients since they won't have the blocks. SPV clients use a bloom filter looking for the addresses and transactions they are interested in to the peer. The peer then only includes the relevant transactions in blocks it returns, together with their Merkle paths to prove that these transactions were indeed part of returned block. |
|
|
|
|
Show Posts
