Bitcoin 20MB Fork

[DeathAndTaxes]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

You can't.  Not everyone will be running a pruned node.  Of course you already knew the answer.

[1713978254]

1713978254

[1713978254]

1713978254

[1713978254]

1713978254

[justusranvier]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

Suppose every node prunes a 99% of the blockchain, and keeps a random 1% of the blocks.

A new node bootstraps by contacting 100 random nodes.

[DeathAndTaxes]

That's very nice indeed, but I'd rather have the value of my stash rely on correctly aligned incentives than on the goodwill of a few hobbyist derps.

If you have a 'stash' then you can protect it by aligning your own incentives.  If you run an unpruned full node then the network can not be compromised and full nodes can still boostrap in a trustless manner.  If your 'stash' is sufficient value then you have a very strong incentive to pay the minimal cost of keeping your node unpruned.

[inBitweTrust]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

Suppose every node prunes a 99% of the blockchain, and keeps a random 1% of the blocks.

A new node bootstraps by contacting 100 random nodes.

Exactly, This is how it should be done. The Blockchain should be sharded and sidechains should be used for other purposes like smart contracts or notary databases where not everyone needs to download if they don't need it.

If you have a 'stash' then you can protect it by aligning your own incentives.  If you run an unpruned full node then the network can not be compromised and full nodes can still boostrap in a trustless manner.  If your 'stash' is sufficient value then you have a very strong incentive to pay the minimal cost of keeping your node unpruned.

Full unpruned nodes will be maintained by decentralist ideologues like myself, payment processors and exchanges, those with a sufficient stake to protect the ecosystem as D+T explained, and other services that provide blockchain explorers or researchers. These sources will always remain sufficient to keep bitcoin decentralized with or without sharding the blockchain.

[DeathAndTaxes]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

Suppose every node prunes a 99% of the blockchain, and keeps a random 1% of the blocks.

A new node bootstraps by contacting 100 random nodes.

This could even become part of the core protocol in time by putting historical data in a DHT but keeping headers and complete utxo as full local copies.  All nodes contain full blockheaders.  This makes attempting to rewrite history impossible.  The headers are too important to store in a DHT but the transactions can be.  All nodes don't need all the copies of SPENT transactions just full headers and the UTXO set.  Of course users also have a vested interest in ensuring a copy of their txns remains so they would probably store that locally.

Future bitcoin node could only hold the following
Full blockheaders (~4MB per year)
Full blocks for the most recent x blocks (can improve efficiency in the event of reorg)
Current UTXO set
Users own transaction history
Some user defined portion of the full transaction set (could be 0% to 100%)

With Bitcoin's current architecture it would make sense to break up the transactions by blocks which would allow a single node to supply a full block.  This isn't a requirement but based on how nodes currently bootstrap it would be the easiest to implement.

Full node boostrapping today
a) contact peers, get best chain
b) download block headers for chain(s) (multiple if peers disagree on best chain)
c) validate block headers for chain(s) (if multiple chains exist then the longest valid chain is best chain)
d) contact multiple peers (each who have 100% of blocks) and requests blocks according to best chain
e) if none of peers contain the full chain contact additional peers to find one with a full copy of chain
f) use blocks received to validate and build UTXO
Node is now bootstrapped

DHT node boostrapping
a) contact peers, get best chain
b) download block headers for chain(s) (multiple if peers disagree on best chain)
c) validate block headers for chain(s) (if multiple chains exist then the longest valid chain is best chain)
d) contact multiple peers and request their DHT identifier (a value which identifies what subset of the blocks the peer has locally)
e) Request needed blocks from the main chain from peers based on their inventory
e) if peers in aggregate do not contain full set of blocks then contact additional peers to find those which contain the missing inventory
f) use blocks received to validate and build UTXO
g) compute DHT identifier based on random value and user chosen DHT share (0% to 100%)*
h) remove blocks outside of inventory computed from DHT share
Node is now bootstrapped

[tvbcof]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

To be honest, I see the technical challenges of pruning to be not terribly high and the 'purity' argument of being able to trace value back to it's full history to be more philosophical than practical.  I'd love to be able to believe otherwise since this is a distinct mark against how I would like Bitcoin to work for other mainly unrelated reasons.

A checkpoint could be marked with high reliability and it could serve as a basis for bootstraping.  No real rocket science here I don't believe.

I think that a more perpetually supportable solution would have involved a POW-backed re-bases.  I'd have been inclined to apportion some percentage of coinbase and transaction fees to whoever comes up with the best re-base on, say, a quarterly basis.

But, as always, these major design items are water under the bridge.  There is no practical way for Bitcoin to adopt them.  We've got what we've got to a large degree and we just have to work within the confines of the system as it is.

One thing that everyone should remember is that the voice in the back of one's head warning against believing 'magic' should be listened to.  All 'bitcoins' need to be accounted for individually.  In the theoretical best-case scenario, right after an 'ideal' re-base, this would mean having an entry for every item in the UTXO.  Bitcoin, being in it's infancy, has a highly compact UTXO with relatively little 'fragmentation'.  With use this is likely to expand by many orders of magnitude and 'pruning' can do nothing about that.

That's my read of the technicals.  If someone thinks this is wrong and want's to make a correction, please do.

[danielpbarron]

It doesn't matter what miners want. If you had truly been reading the logs, you would have known that.

Yes it matters what miners want.

I guess you don't follow hyperlinks, or know how to use them for that matter. Moving on...

I think I have read most of what has been written on the subject, including the 2 last years of trilema (and some older articles) + months of logs of b-a.

I have not seen where or how the old bitcoin would survive :
- under a difficulty that matches a target for which it has 5% of the required hashrate
- under the risk of an attacker committed to write empty blocks on the chain as long as it is possible

I do not say :
- that the gigablockchain is a good idea,
- or that a majority of miners will switch to it.

What I say is that IF miners switch, with 95% of the hashrate or more, THEN the old chain is dead. And I don't see any arguments that could explain how it could be any different from what MP himself wrote here.

If you've read so much of the log and trilema then why aren't you in the WoT? Get on that.

Even if the miners were dumb enough to broadcast the new version and triggered the fork, that doesn't mean they will stay on the fork. They will soon find that it is not possible to sell their USGavincoins for the price they were expecting. And seeing as how mining is a barely profitable venture (due to adjusting difficulty), they will be forced to switch to the original chain or shut down entirely.

[davout]

You can't.  Not everyone will be running a pruned node.  Of course you already knew the answer.

And we're back at this, ahem, "trusted" archive nodes business...

If your 'stash' is sufficient value then you have a very strong incentive to pay the minimal cost of keeping your node unpruned.

So I get to waste bandwidth on every derp that figures keeping blocks around is unnecessary?
And I'm supposed to keep doing that when blocks become increasingly large?

A new node bootstraps by contacting 100 random nodes.

Directly?

Full unpruned nodes will be maintained by decentralist ideologues like myself

Special, as in Ralph Wiggum special.

[NewLiberty]

Pruning should arrive in Bitcoin Core 0.11, allowing people to run a full node with only 1 GB of storage space.

Please answer this simple question: "how does one bootstrap a full node once everybody's pruning?".

Suppose every node prunes a 99% of the blockchain, and keeps a random 1% of the blocks.

A new node bootstraps by contacting 100 random nodes.

More formally constructed the optimal balance is a hybrid DHT.  All nodes contain full blockheaders.  This makes attempting to rewrite history impossible.  The headers are too important to store in a DHT but the transactions can be.  All nodes don't need all the copies of SPENT transactions just full headers and the UTXO set.  Of course users also have a vested interest in ensuring a copy of their txns remains so they would probably store that locally.

Future bitcoin node could only hold the following
Full blockheaders (~4MB per year)
Full blocks for the most recent x blocks (can improve efficiency in the event of reorg)
Current UTXO set
Users own transaction history
Some user defined portion of the full transaction set (could be 0% to 100%)

This presents a drastic improvement in the bitcoin network cost and decentralization if:
Bitcoin Network Cost = Data Size * decentralization

[DeathAndTaxes]

This presents a drastic improvement in the bitcoin network cost and decentralization if:
Bitcoin Network Cost = Data Size * decentralization

Necessity is the mother of all invention.  Simple is better until it is not good enough.  We started from a system where 100% of nodes were full nodes and they downloaded the entire chain from a peer.  SPV clients were added.  Header first syncing was added.  Downloading the chain from multiple peers were added.  Pruned nodes will be added in the future.  Each step provides enhanced scalability at the expense of complexity.  A DHT store is just a progression along that sequence.

Right now I download the full headers and then download blocks in parallel from multiple peers.  Today all of those peers have all the blocks but they don't need to.  I only download a subset of the blocks from each peer.  So the "extra" blocks they have but I don't download is unused.

I should point out the example provided is simplistic to avoid it being a multipage post.  Also some people give DHT concept a bad name by proposing them incorrectly.  The UTXO and blockheaders should never be part of the DHT.  They are simply too critical and they are used for forward validation.  Not having elements needed for forward locally is a bad design choice.  It would mean that inter node traffic required to validate future transactions and blocks would be excessive.  You are just trading bandwidth for storage and honestly bandwidth is more critical.  Also it would make sense for all nodes to keep the tip of the chain to better handle reorgs and syncing up nodes who have been recently offline.  How much of the tip should be kept is something which would need modeling.  Getting it wrong doesn't compromise security but it can reduce efficiency.

Here is a simple way to look at it.  Most people assume the full blockchain is used in real time to validate transactions and new blocks.  That is incorrect and it leads to mental blocks in seeing how the data could be better distributed.  If I need the full blockchain and I don't have it then how can I validate.  What really happens is that the full blockchain is used to CREATE the UTXO.  Once your node has the UTXO it never uses those historical blocks locally again.  When a new block comes in it is validated and then used to update the UTXO.  The new block is never used locally again.   The UTXO is used to validate transactions and new blocks.   You need a local copy of the UTXO.  You don't need a full copy of the historical chain to create the UTXO but you don't need to keep it.  If nobody has the full chain then that is a problem but a DHT can ensure that a full copy exists even if you only have a portion of it.

 

[justusranvier]

So I get to waste bandwidth on every derp that figures keeping blocks around is unnecessary?

I can tell that you didn't read my proposal that I've posted several times, because the answer is embarrassingly obvious.

[DeathAndTaxes]

You can't.  Not everyone will be running a pruned node.  Of course you already knew the answer.

And we're back at this, ahem, "trusted" archive nodes business...

That is false.  Your node today doesn't 'trust' other nodes.  It downloads blocks and INDEPENDENTLY verifies them.  As of v0.10 you node will bootstrap by getting headers first and contacting multiple peers in parallel to download the chain.  If you only get x% of the chain from each peer then why does each peer need to have 100% of the chain.  Even if there was only a single node which had the full blockchain they can't modify the chain.  There attack would be limited to a denial of service (refuse to give you blocks when you ask).   Since you love to make false claims I will put it in black and white that I am not advocating a single node contain the full chain.  A DHT is a perfect data structure when nodes do not need the full dataset, and can't change the dataset.  That exactly describes the historical blockchain.

There is no reduction in security.  There is an increase in complexity but headers first and parallel block sync is already more complex.  The first client was very simple it contacted one peer and asked them for the full chain until it had a copy.  That worked when the network was very small but wouldn't work today.  The complexity of the interpeer communication will continue to increase over time even if the block size is never raised.

[davout]

There is no reduction in security.

It becomes much easier to pretend some blocks don't exist to selected targets, and get them to accept selectively rewritten history for just long enough to rip them off.

[NewLiberty]

There is no reduction in security.

It becomes much easier to pretend some blocks don't exist to selected targets, and get them to accept selectively rewritten history for just long enough to rip them off.

This security risk is balanced by the correspondingly increased number of nodes, given an equivolent Bitcoin Network Cost, where:
Bitcoin Network Cost = Data Size * Decentralization.

So the net result is no decrease in security.

We can expect a marginal increase in security as the node maintenance cost becomes less 'chunky'.

[DeathAndTaxes]

There is no reduction in security.

It becomes much easier to pretend some blocks don't exist to selected targets, and get them to accept selectively rewritten history for just long enough to rip them off.

All nodes maintain a full copy of the blockheaders.   If you can prevent a node from obtaining a block then you can denial of service the validation of the chain but you can't trick a node into thinking a block doesn't exist.  An attacker can't fake the content because that would require a preimage of the blockhash.  If one peer refuses to provide a block, you can ask another peer.  If all of them also refuse you can connect to new peers.  If you are unable to find a single peer to provide the requested block then you can't validate the chain but the attack is limited to a denial of service.  If a full node hasn't received all the blocks then it hasn't synced and it can't validate transactions.  It should not provide unvalidated information to the user.

With the blockheaders of the longest chain a node knows the full set of blocks that exist.  It may not know the contents yet but it already knows they exist.

[davout]

This security risk is balanced by the correspondingly increased number of nodes, given an equivolent Bitcoin Network Cost, where:
Bitcoin Network Cost = Data Size * Decentralization.

You can't arrive to meaningful conclusions if you base your reasoning on undemonstrated equalities.

All nodes maintain a full copy of the blockheaders.   If you can prevent a node from obtaining a block then you can denial of service the validation of the chain but you can't trick a node into thinking a block doesn't exist.  An attacker can't fake the content because that would require a preimage of the blockhash.  If an attacker has the valid block his only option is to not provide that block but that exists today with full nodes.  If all your peers refuse to provide you a requested block you can't validate the chain.  This however is limited to a denial of service.   If a full node hasn't received all the blocks then it hasn't synced and it can't validate transactions anyways.  If it is then the node is flawed and insecure.   

With the blockheaders of the longest chain a node knows the full set of blocks that exist.  It may not know the contents yet but it already knows they exist.

Yes, I was reasoning in the context of a "hey, i'm getting back online, what's the network been up to?" scenario.

[traincarswreck]

So...

Have we reached our consensus yet?

And do we know yet what is "Ideal Money"?

(euro/greek talk http://ec.europa.eu/avservices/ebs/live.cfm?page=2)

But if, for example, all of the countries of the world would base the value for their national currencies on the value of the british currency then this situation would appear singular and unstable, while it was not so singular for a lot of countries to base their currency value on gold.

[DeathAndTaxes]

Yes, I was reasoning in the context of a "hey, i'm getting back online, what's the network been up to?" scenario.

It is true that if you do not know the longest chain then you have no security.  That is the foundation on which all other layers of security depend.  Running a full node will not make you any more secure if you can not connect to at least one honest node.  All nodes (including full nodes) must know the chaintip of the longest chain in order to sync in a trustless manner.

[Cryddit]

No. He is talking about the 970-980 kB blocks which not rare these days. While the average is around 300kB, that means we are at 1/3 capacity. And I personally think 1/3 is where we should seriously consider our options for scalability. The way people are split on the subject right now it is even more urgent to start getting a consensus.

But on the other hand there is no hurry and no crisis. The extra transactions can just go through altcoins if Bitcoin chooses the 1MB block + higher fees 'solution'.

They aren't rare, but they are not the norm.  Most aren't near full.
F2Pool solved a few that were close recently, above 900K.  With a good number of freebies.

https://blockchain.info/block/0000000000000000175b44859017a5148c48ecba7a67f14012232e9bb6b47a73

https://blockchain.info/block/00000000000000000f9597aed448ce8429c550a65f896b66760381d0c364901e

and then there is this 7K block in between
https://blockchain.info/block/000000000000000014efb22561313ebe3c27780808b5d8939ebc1a850badf9da

There are a lot of blocks with <200K, so we'd get some scalability of more Tx/s with a minimum block size too, but that would not be a good thing to do.

The more merchant involvement there is, the more we need to be thinking about the spikes associated with merchant business.   We need to think about what happens at 8 AM Pacific Time on Black Friday - when the local credit card network in the places where I shopped couldn't handle the traffic and were knocked offline for an hour and then backlogged for the next six.

The more trader business there is, the more we need to be thinking about the spikes associated with exchange businesses.  We need to think about what happened in the markets the next day after Jim Leeson took down Barents, when the mortgage meltdown chain reaction set in after Lehman Bros. defaulted and made $trillions evaporate overnight, when Madoff announced that his $billions were all part of a Ponzi scheme, and when Cyprus announced its plan to just TAKE whatever money people happened to have trusted banks with.  When people finally take that 'Beware False Profits' message to heart, odds are they'll want to trade into Bitcoin.  It'd be a real shame if they can't.

And we also need to consider the enormous follow-on effects of government decisions that change the rules more broadly.  We need to think about what happens at 8 AM GMT on the International Currency Exchange the day after Brussels announces that Europe will either form a fiscal union (ie, have a uniform tax law and a uniform policy for allocation of the funds so collected) or end its attempt to have a common Euro fiat currency without one.  'Cause that's going to happen - what they've got now isn't a stable configuration in the long term.

I think there's all kinds of reasons why, when we need bigger volume, we're going to need enormous amounts of it very suddenly.  The call for bigger volume is sometimes a result of predictable but large events, and sometimes a response to statistical 'Black Swan' events that result in a scramble to reach a new balancing point far from the old one.

We're seeing spikes to 1MB now, even without driving events we can point at. So, at the moment when 1MB definitively isn't enough, I fear that 5MB won't be enough either.  Transaction volume in the real world is incredibly spiky in response to such events, and Bitcoin is starting to have closer and closer interactions with the 'real world.'

Cryddit

[davout]

All nodes (including full nodes) must know the chaintip of the longest chain in order to sync in a trustless manner.

That's backwards.