Idea on the "Blocks are [not] full problem"

[rme]

Problem discussed in this thread:
https://bitcointalk.org/index.php?topic=339505.0

Miners create 250KB or less blocks because they broadcast faster and they have less chances of been orphaned, they include few transactions in them

Here it comes my idea about this issue:

Miners include few transactions in blocks because more transactions equals more probability of orphaned block so lets equal all blocks:

Miners should craft a block normally, so lets imagine they generate a 250KB block. Before they send it to other node they have to concatenate junk bytes (random (?)) to the block data, so all blocks are 1MB.

When a node sees this block, they broadcast it and when they finish they delete this junk bytes and they only the block.

The junk data never gets into the blockchain

Pros:
- All blocks "are" 1MB in terms of relaying them.
- We avoid other more tecnical mecanisms
Cons:
- Bitcoin QT needs some bandwith more because now all blocks are 1MB.


If one day we need to rise the 1MB block limit this process will be the same but all blocks will require to be 10MB (for example). We only need to concatenate junk to them.


How to perform this hard fork?
Bitcoin core developers can release an update that includes this fix but only enforcing it when the blockchain reaches the block 277000 (30 days later) so we give some time for people and miners to update their software.

[gmaxwell]

Bandwidth isn't the only consideration— checksig can have a pretty substantial impact too.

But conserving bandwidth— and keeping the cost of operating a node down— is part of the reason to not have 1MB blocks constantly. Taking a 4x bandwidth hike for everyone is a pretty substantial cost. I suppose it's better than the blocks actually being that size, but still…

Miners include few transactions in blocks because more transactions equals more probability of orphaned block so lets equal all blocks

Citation needed. 250 is the default in the reference software. A simpler hypothesis than miners performing some complex orphaning cost benefit analysis is that they don't care to change the defaults.

[grue]

this solves the block propagation problem, but it still doesn't solve the problem of TXs eating up a full node's hard drive space.

[justusranvier]

250 is the default in the reference software. A simpler hypothesis than miners performing some complex orphaning cost benefit analysis is that they don't care to change the defaults.

An easy way to test that hypothesis would be to change the default to 1 MB and see how many miners suddenly start creating larger blocks.

[niniyo]

250 is the default in the reference software. A simpler hypothesis than miners performing some complex orphaning cost benefit analysis is that they don't care to change the defaults.

An easy way to test that hypothesis would be to change the default to 1 MB and see how many miners suddenly start creating larger blocks.

Exactly.  Are the core devs treating this issue seriously?  In my opinion it's the biggest threat to Bitcoin's future.  The blockchain transactions-per-second needs to get MUCH higher in order to support the future that we all want.

If bitcoin becomes a low-TPS system with high fees that is only viable for moving millions of dollars, and all other transactions move off blockchain, then I think Bitcoin will collapse since it loses its main feature that is appealing (ie. peer-to-peer payments).

[grue]

Exactly.  Are the core devs treating this issue seriously?  In my opinion it's the biggest threat to Bitcoin's future.  The blockchain transactions-per-second needs to get MUCH higher in order to support the future that we all want.

If bitcoin becomes a low-TPS system with high fees that is only viable for moving millions of dollars, and all other transactions move off blockchain, then I think Bitcoin will collapse since it loses its main feature that is appealing (ie. peer-to-peer payments).

that said, it's important to note that bitcoin is not for micropayments.

250 is the default in the reference software. A simpler hypothesis than miners performing some complex orphaning cost benefit analysis is that they don't care to change the defaults.

An easy way to test that hypothesis would be to change the default to 1 MB and see how many miners suddenly start creating larger blocks.

for a pool admin, it's trivial to change the "soft" block size in bitcoin-qt

[niniyo]

Exactly.  Are the core devs treating this issue seriously?  In my opinion it's the biggest threat to Bitcoin's future.  The blockchain transactions-per-second needs to get MUCH higher in order to support the future that we all want.

If bitcoin becomes a low-TPS system with high fees that is only viable for moving millions of dollars, and all other transactions move off blockchain, then I think Bitcoin will collapse since it loses its main feature that is appealing (ie. peer-to-peer payments).

that said, it's important to note that bitcoin is not for micropayments.

What counts as a micropayment?  Fair enough if you can't pay a few cents through the blockchain, but I think anyone should be able to pay for a meal or a coffee via the blockchain, so transaction fees would have to stay below ~5-10 cents to make this viable.

[justusranvier]

Exactly.  Are the core devs treating this issue seriously?

Some are, some aren't.

[Mike Hearn]

An easy way to test that hypothesis would be to change the default to 1 MB and see how many miners suddenly start creating larger blocks.

Indeed. I already suggested this a few days ago (see pull reqs). However 0.8.6 is going to go out with a relatively small bump of 50 or 100kb in the default size, to see how many miners follow.

[DeathAndTaxes]

It is an interesting idea.  It essentially eliminates any "orphan cost" due to bandwidth related propogation delays.  As mentioned some complex tx can be computationally intensive but bandwidth & latency are likely are more critical resources.  Something would probably need to be done to handle significantly older blocks in a more efficient manner.  For example Bitcoin blockchain is ~270,000 blocks, if the padding was enforced for all block messages to bootstrap a new node would require ~270 GB, compared to ~12 GB actual so older blocks need to be handled without padding.

Still it is an interesting concept and if not used in Bitcoin it could be refined and used in altcoins.  This could be combined with another improvement, replacing tx with tx hashes in block message, to generate better results at reduced cost.

Block message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction list (variable - average 600 bytes ea)

Proposed tx less block message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction hash list (variable - 32 bytes ea)

The average tx size appears to be ~600 bytes, tx hash is 32 bytes so this reduces the new block message by >90%.  Now for bootstrapping nodes the first format is more useful but for up to date nodes they should have block txs in their memory pool.  It is also in a miners best interest to ensure their peers have all those tx to reduce the propogation time.  If a node is missing some tx they can simply request those tx by tx id (hash) from peers.  This method can be used by itself to reduce the orphan cost by itself or it could be combined with the concept in the OP to both reduce the bandwidth requirement and eliminate the additional tx orphan cost. 

One needs to consider how much padding would be required as there is no fixed relationship between size of all txs in a block and the size of all tx hashes in a block.  Since the goal is just uniformity some lower bound could be considered.   An average tx size of <300 bytes is dubious (would require a block of all single input, single output txs) so in additional to max tx size a max tx count limit could be enforced.  If the limit was 1MB and 3,000 tx this would make the max block message size 94 KB (4+4+80+9+3000*32).  Block message could be padding enforced to 100 KB (for current block size limit).

Block-txless message format
magic # (4 bytes)
block size (4 bytes)
blockheader (80 bytes)
tx count (1 to 9 bytes)
transaction hash list (variable - 32 bytes ea)
Randomized padding (100KB - rest of message size)

How would this work in practice.  Well lets look at a recent large block:
https://blockchain.info/block-index/444118/0000000000000002166ff15ec0ce6427f21c2f7ce55676d280b0677fe04c1f2a
Tx Count: 1368
Size with current block message: 882 KB
Size if padded to 1MB: 1024 KB (16% overhead compared to current)
Size using txless block message: 43KB
Size using padded txless block message: 100KB (89% reduction compared to current)


Pros:
For blocks over 100KB the new format even with padding is still smaller than the "full block" format without padding so there is no bandwidth costs to miners decrease.
Simplifies tx inclusion economics.
Can be implemented as a non forking change by creating a second new block message type and legacy nodes can still request block in older "full" format.
Doesn't increase on disk storage requirements.

Cons:
More complex broad propagation logic, older blocks need to be handled in legacy manner to avoid massively increasing bootstrap requirements.  
Hard enforcement at protocol level requires a hard fork (which may not be possible to obtain the consensus needed).
With a non-forking expansion, enforcement is only at the client level.  Miners could locate other miners and communicate directly in more efficient block message (i.e. block hash only with no padding) to gain advantage on other miners.

[gmaxwell]

If bitcoin becomes a low-TPS system with high fees that is only viable for moving millions of dollars, and all other transactions move off blockchain, then I think Bitcoin will collapse since it loses its main feature that is appealing (ie. peer-to-peer payments).

If Bitcoin blocks become gigabytes in size such that only powerful entities bother running validation nodes, then I think Bitcoin will collapse since it loses its main feature that is appealing (truselessness).

There is careful balancing which is required and hysteria ("biggest threat", "MUCH higher") is not helpful.

At either extreme— where it's too costly to transact, or where it's too costly to validate Bitcoin loses value.  If Bitcoin becomes defacto centralized due to being too costly to validate then it will be _impossible_ to build anything more decentralized on top of Bitcoin, and this is arguably a greater risk both because it can gradually end up in that state and because the risk of it being too costly to transact can at least be answered by alternative payment networks (which are mandatory in any case to achieve things like strong privacy, instant payments, efficient very low value transactions, and offline transaction).

[justusranvier]

If Bitcoin blocks become gigabytes in size such that only powerful entities bother running validation nodes, then I think Bitcoin will collapse since it loses its main feature that is appealing (truselessness).

A transaction rate which produces 144 1 GB blocks per day is achievable on a home internet connection which I can get for less than $100/month, assuming transaction messages are suitably optimized as mentioned by D&T.

By the time the network is processing that many transactions, the cost of such a connection will be even lower.

The answer is to optimize block propagation into something that resembles efficient use of bandwidth so that we don't have to fear success.

[DeathAndTaxes]

If Bitcoin blocks become gigabytes in size such that only powerful entities bother running validation nodes, then I think Bitcoin will collapse since it loses its main feature that is appealing (truselessness).

A transaction rate which produces 144 1 GB blocks per day is achievable on a home internet connection which I can get for less than $100/month, assuming transaction messages are suitably optimized as mentioned by D&T.

That is an incorrect assumption.  The point of replacing full tx with tx hashes in the block message is that you will already have received those txs.  So while it reduces the bandwidth requirements for timely relaying of the block message it doesn't reduce the overall bandwidth requirements.  Also remember Bitcoin is a p2p network so for the network to be robust each peer needs to have multiple connections to other peers.  While a node can get by with only eight connections to reduce the risk of an isolation attack a healthy connection would be far more.

So if we consider that there are 144 GB of blocks and you are connected to 16 peers.  Either they will relay to you or you will end up relaying to them.  That means 144 GB daily of synchronous bandwidth for tx messages plus the reduced overhead of block message is still say another 10% so maybe 158 GB daily or 4,700 GB monthly.   While in theory a 15 Mbps connection could handle that, try to find an ISP which won't cap, throttle, or simply bad you for using 5 TB per month.

Of course even this example excludes the additional load on the average node in order to accommodate bootstrapping new nodes, or nodes which are out of sync.  The network would be very very unhealthy IMHO at even 10% of that on residential connections.

[justusranvier]

While in theory a 15 Mbps connection could handle that, try to find an ISP which won't cap, throttle, or simply bad you for using 5 TB per month.

Time Warner Cable has been pretty good about that so far. OTOH I've been paying for the highest-tier package and not obviously using it for Bittorrent.

Several other countries have it even better.

[niniyo]

If bitcoin becomes a low-TPS system with high fees that is only viable for moving millions of dollars, and all other transactions move off blockchain, then I think Bitcoin will collapse since it loses its main feature that is appealing (ie. peer-to-peer payments).

If Bitcoin blocks become gigabytes in size such that only powerful entities bother running validation nodes, then I think Bitcoin will collapse since it loses its main feature that is appealing (truselessness).

There is careful balancing which is required and hysteria ("biggest threat", "MUCH higher") is not helpful.

At either extreme— where it's too costly to transact, or where it's too costly to validate Bitcoin loses value.  If Bitcoin becomes defacto centralized due to being too costly to validate then it will be _impossible_ to build anything more decentralized on top of Bitcoin, and this is arguably a greater risk both because it can gradually end up in that state and because the risk of it being too costly to transact can at least be answered by alternative payment networks (which are mandatory in any case to achieve things like strong privacy, instant payments, efficient very low value transactions, and offline transaction).

I wasn't trying to be extreme but I did mean what I said.  The current transaction rate appears to be below 1 TPS, so obviously that needs to go much higher.  I'm not sure what figure people quote for Visa but it's in the thousands.  My fear is that if bitcoin gets stuck at 1 TPS, then there isn't a future for it.

If the problem isn't related to orphan costs, and it simply due to miners running defaults, then that's great and I look forward to seeing block sizes go up as the new releases come out.

It's not exactly clear to me where you stand, gmaxwell - is there some approximate TPS figure that you imagine might be possible in the future?  Is there a TPS limit that you would deem to be unacceptable for bitcoin success?  How long do you see 1 TPS being adequate before transactions are unaffordable, given the growth in transactions and bitcoin price?

I know you are an influential developer so it would be nice to know where you stand on this.

Thanks

[gmaxwell]

I wasn't trying to be extreme but I did mean what I said.  The current transaction rate appears to be below 1 TPS, so obviously that needs to go much higher.  I'm not sure what figure people quote for Visa but it's in the thousands.

Visa is an erroneous comparison. Bitcoin is a currency with a payment network. Visa is a payment network. I expect that someday people will use Visa to pay people with Bitcoins (though thats not a way of using Bitcoins that I'd prefer).

It's not exactly clear to me where you stand, gmaxwell

I don't appreciate the partisan "where you stand" language, which appears to be asking me to draw a bright line.  This is a complicated matter engineering and it involves nuanced trade-offs and understanding. Not bright line politics.

I support the vision of Bitcoin as I originally learned about it, which is a trust-less decentralized system intended to make it easy for people to transact without interference or mediation by trusted parties. Bitcoin is a system where we minimize the interaction of failure prone men in making decisions— with the best of intentions— which are pushed on others without their consent— it's a system made somewhat immune to even majority whim by the widely distributed hard enforcement of system invariants which are purposefully difficult to change, especially if the changes are controversial.

To the extent that trustlessness and scaling aren't in conflict, I say— and I think everyone with honest intentions would agree: great! lets have all the things!  I believe that as technology improves the level of conflict free scaling improvements will increase.

I am, however, also concerned that even at today's scale we have operating cost and education problems which are endangering the system's security assumptions. I believe these are perfectly solvable problems.

Generally, to the extent that trustlessness and scaling come into direct conflict I believe that Bitcoin should tend to prefer trustlessness. There are a couple of reasons I believe this:

Scaling with trustlessness should improve in time just from the march of technological progress. Trustlessness seems harder to regain (e.g. p2pool didn't replace all the centralized pools, though I believe if it had come first we'd probably have no centeralized pools today or at least they'd be substantially different) than scaling is to add.

Scaling can always be achieved through alternative payment mechanisms, but I know of no similar way to improve trustlessness through overlay systems. Alternative payment networks for Bitcoin must exist in order to support things like instantaneous (semi-)irreversibility, improved privacy, offline payments, etc. Bitcoin will never be as scalable as a less decenteralized payment system. E.g. If we had to choose between Bitcoin having a dozen miner supernodes that do all the validation which we effectively must trust, or having to run low value bitcoin transactions through distributed 5-party chaum banks, which exist by the hundreds around the world... I'd prefer the latter. This is doubly true because if Bitcoin is too bloated it will harm diversity in these alternative payment systems.

Finally, trustlessness should be preferred because it is the unique value that Bitcoin (and its clones) provide. Visa does the scaling game better than we do, for fundamental reasons. Many prior ecash systems with ample funding and brilliant technology failed, largely because their trusted center provided a chokepoint.

But all thats assuming that there is a conflict and I think our job in Bitcoin is to advance the technology and minimize the conflict, foremost.  And then we don't have to answer the hard questions like who picks who wins and who loses, it's better to have everyone win. Past that, it's a balancing act— one where either extreme diminishes Bitcoin's value to the world.

[DeathAndTaxes]

I still think the solution to the orphan problem is something which involves transmitting the blocks before finding the nonce. It's only the delays which take place after finding the nonce which cause orphans.

The vast majority (as in ~100%) of blocks are never solved.  A particular blockheader only has ~4 billion possible nonces which even a low end rig runs through in a fraction of a second.  Difficulty is currently 700 million which means for every solved block roughly 700 million block headers are constructed and no solution is found.  

[justusranvier]

I still think the solution to the orphan problem is something which involves transmitting the blocks before finding the nonce. It's only the delays which take place after finding the nonce which cause orphans.

The vast majority (as in ~100%) of blocks are never solved.  A particular blockheader only has ~4 billion possible nonces which even a low end rig runs through in a fraction of a second.  Difficulty is currently 700 million which means for every solved block roughly 700 million block headers are constructed and no solution is found.  

If there are changes in the pipeline which require a hard fork anyway, might as well fix that problem too by increasing the size of the nonce field.

[DeathAndTaxes]

Bitcoin is decentralized, but it isn't, and can't be, completely trustless.

There is a difference between counterparty trust and a trusted third party.  You may have to trust your counterparty but Bitcoin wasn't desolved to remove that trust element, Bitcoin was designed to remove the need for a trusted third party.


Alice <-----> Bob
Alice and/or Bob need to trust each other but they don't need to trust anyone else.

Alice <------ Bank ------> Bob
Both Alice and Bob can't simply trust each other they now must trust the "trusted" third party. 


Satoshi talks about removing the trusted third party in the introduction of the whitepaper. 
http://bitcoin.org/bitcoin.pdf

[DeathAndTaxes]

Nodes should already have the txs which is why I indicated the block message can be reduced to a list of hashes for the txs included in the block and that reduces the block message by 90%+.   Even if you assumed that no miner would need to change their tx set per block do you really see a system where each miner pre-broadcasts their exact tx set in advance is going to scale?  That would mean every node receives one tx set from every active miner before every block.   We should be advocating as many independent miners as possible.  So if there are 100 miners then each node on the network needs to receives, store, and relay 100 tx sets for every block?  If there are someday 10,000 miners it is 10,000 tx sets for every block?