Blocks are [not] full. What's the plan?

[Carlor]

Why not a dynamic blocksize?

I think that an automatic dinamic block size is difficulty to design and implement and could lead to disparities among peers. The community should be in charge of deciding what the max block size should look like through planned increases.

Best regards,
Ilpirata79

Badly typed on my iPad

I understand your point, but I think if it is possible it would be the best solution. In fact, a dynamic blocksize would be something like "planned increase" (and decrease).
What I would like, that - if implemented after careful testing - it gives the chance for not having to hardfork the system again. This would make bitcoin highly stable.

[DeathAndTaxes]

So a couple points of clarification.  The idea of including tx hashes in the block MESSAGE wouldn't change the size of the actual block.  It would simply be a new message type.  Remember bootstraping nodes also require older blocks and they won't have the txs either so the existing block message would be useful in cases like that.

Simplified version:
on newest block when latency is important = use header + tx hash message.
on older blocks when latency isn't an issue = use header + full tx message.

As for the block limit well that is another more complex issue.  Honestly I don't see it very important right now.  The number of full blocks since the genesis block is exactly 0.0%.  Even today with a backlog of txs the average block size is ~200KB and the largest block is ~600KB.  Raising the limit doesn't change the "orphan economics". 

[Cryddit]

If you're gonna be elbows-deep in that code anyway, there's a major privacy upgrade you can do with block formats.

Is there any chance of just storing blocks as list of txid's, list of destroyed txouts, list of new txouts, without telling which txins/txouts go with which tx?

After all, if we're assuming that the clients have already *seen* the individual tx, they can check them anyway.  And look at the list of txid's to find out which ones they're missing, get those, and check them.

This doesn't make privacy absolute in any sense; it just makes it so that in order to trace tx and do data mining you have to be listening in realtime to get individual transactions instead of looking at the blockchain after the fact.  But that would still a big improvement.

[niothor]

Briefly hitting 100 k transactions/day.
Now , if there are some selfish miners who include just a few tx on their blocks and we continue to grow we might hit the limits and the backlog will start to grow.
I'm grabbing some popcorn and waiting for the decisions on the block size.

[BurtW]

If you're gonna be elbows-deep in that code anyway, there's a major privacy upgrade you can do with block formats.

Is there any chance of just storing blocks as list of txid's, list of destroyed txouts, list of new txouts, without telling which txins/txouts go with which tx?

After all, if we're assuming that the clients have already *seen* the individual tx, they can check them anyway.  And look at the list of txid's to find out which ones they're missing, get those, and check them.

This doesn't make privacy absolute in any sense; it just makes it so that in order to trace tx and do data mining you have to be listening in realtime to get individual transactions instead of looking at the blockchain after the fact.  But that would still a big improvement.

I think this is a great idea.  One question:  the fees would then be the difference between these two large sums?  Would that work?

[DeathAndTaxes]

Is there any chance of just storing blocks as list of txid's, list of destroyed txouts, list of new txouts, without telling which txins/txouts go with which tx?

No or as a miner I could simply modify your txs keeping your tx inputs and making the outputs to me.  The security model of Bitcoin involves three layers.

1) Senders digitally sign the entire* tx to ensure it is immutable.
2) All nodes (not just miners) verify all txs and blocks are valid.
3) Miners place tx in blocks to create a consensus history.

*well a simplified form

[BurtW]

Is there any chance of just storing blocks as list of txid's, list of destroyed txouts, list of new txouts, without telling which txins/txouts go with which tx?

No or as a miner I could simply modify your txs keeping your tx inputs and making the outputs to me.  The security model of Bitcoin involves three layers.

1) Senders digitally sign the entire* tx to ensure it is immutable.
2) All nodes (not just miners) verify all txs and blocks are valid.
3) Miners place tx in blocks to create a consensus history.

*well a simplified form

Point and match

[Kouye]

Well larger blocks are never going to be faster or as fast as smaller blocks.  The goal is to reduce the latency time per kB.  The faster a block can be broadcast the lower the "orphan cost" per tx.   Still larger blocks will always have a higher orphan rate but they also have higher gross revenue.
...snip...

This looks promising.

As the adoption rises, the number of tx processed by the network per day will need to be increased greatly.
I'm coming back here with the idea of dynamic minimum block size... Couldn't we just index that size (or min tx count/block) requirement on the current difficulty?

Miners could then reject small blocks that don't meet the requirement (so basically, miners trying to send small blocks to have less latency would get their blocks orphaned, which looks like a good way to reduce orphan cost?).

Combined with your proposition to reduce block size dramatically, and keeping in mind that the network will also become faster and faster, that would allow us to ensure have a better chance that the tx throughput remains consistent with the global need, progressively?

[DeathAndTaxes]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

[Carlor]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

I think the issue has to be solved asap. When it becomes a bottleneck it is already too late. Because it will have an impact on bitcoin, it has to be tested for a longer period.
But afaik it's already very high on the dev's priority list.

[Peter Todd]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

Yeah, as ugly as it is, everyone knowledgeable about the issue agrees that if the limit is to be raised, it has to be done in such a way that there is still a limit of some kind that is not under miner control to avoid creating new incentives to centralize hashing power, or for that matters furthering existing incentives to centralize. You know how they say in cryptography that attacks only get worse, not better? What we've seen in Bitcoin suggests that clever ways for large miners to abuse their position to get even larger seem to be found a lot more often than the other way around.

[DeathAndTaxes]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

Yeah, as ugly as it is, everyone knowledgeable about the issue agrees that if the limit is to be raised, it has to be done in such a way that there is still a limit of some kind that is not under miner control to avoid creating new incentives to centralize hashing power, or for that matters furthering existing incentives to centralize. You know how they say in cryptography that attacks only get worse, not better? What we've seen in Bitcoin suggests that clever ways for large miners to abuse their position to get even larger seem to be found a lot more often than the other way around.

Agreed.  Often is protocol design there is a saying "don't let perfect be the enemy of good".  Is going from one static artificial limit to another static artificial limit perfect?  Of course not.  However it is simple, easy to model, and well understood.  It won't be the be all end all solution which "solves" the issue to 2140 and beyond but it does buy us some time in a low risk manner.

The one thing I actually like about the "orphan cost" is that it acts to limit blocksizes below the cap.  The cap is 1MB today but nobody (as in not a single block in the history of Bitcoin) is 1MB.  So the cap is really acting just as a safety mechanism to prevent malicious activity.  The real "economic cap" is well below the 1MB limit.  That won't change if/when the cap is raised so the thinking should be what cap still protects the Bitcoin network from centralization and blockchain bloating attacks.   With current technology I think 10MB is viable. 

If someone wants to solo mine from a home connection they may have to pay for upgraded internet service but it isn't outside of the realm of possibilities like a 1GB block would present.

[DeathAndTaxes]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

I think the issue has to be solved asap. When it becomes a bottleneck it is already too late. Because it will have an impact on bitcoin, it has to be tested for a longer period.
But afaik it's already very high on the dev's priority list.

Well it is important to remember that raising the block size won't "solve" the problem because it isn't so much a problem as a compromise.  The reason why I say we have time is because miners are currently using all the space "available".  If the memory pool was empty of tx older than the last block and block sizes were rising day after day I might feel differently however IMHO if the block limit was 10MB right now blocks wouldn't be any larger.

[Carlor]

Personally (and I really believe block size is a completely different issue) I think the best option would be to raise the block limit once it becomes a bottleneck to a higher static limit.  The reason is that Bitcoin is very hard to undo and going from 1MB static to 10 MB static is a simple and well understood change.  Other systems while they may be more future proof are more complex and need more time for discussion, analysis, and testing.  I project the 1MB limit will become an issue within a year (or 18 months on the outside) and I wouldn't be confident with any radical change in the block system in a short period of time.

I think the issue has to be solved asap. When it becomes a bottleneck it is already too late. Because it will have an impact on bitcoin, it has to be tested for a longer period.
But afaik it's already very high on the dev's priority list.

Well it is important to remember that raising the block size won't "solve" the problem because it isn't so much a problem as a compromise.  The reason why I say we have time is because miners are currently using all the space "available".  If the memory pool was empty of tx older than the last block and block sizes were rising day after day I might feel differently however IMHO if the block limit was 10MB right now blocks wouldn't be any larger.

Yep, you're right on this.
But I think it will become an issue rather sooner than later. And imho it is urgent, to have a working plan for this phase. I can buy more and more stuff with bitcoin here in Germany, which is great, but it's still fragile and shops depend on the quick transactions.

[solex]

Ghash.io and ASICminer both still observing the 256KB default. Anyone know if they will increase this soon?

[Peter Todd]

Agreed.  Often is protocol design there is a saying "don't let perfect be the enemy of good".  Is going from one static artificial limit to another static artificial limit perfect?  Of course not.  However it is simple, easy to model, and well understood.  It won't be the be all end all solution which "solves" the issue to 2140 and beyond but it does buy us some time in a low risk manner.

Yup, and a bump lets us learn about what changes as the size is increased without risking unpleasant surprises; frankly even 10x is risky.

The one thing I actually like about the "orphan cost" is that it acts to limit blocksizes below the cap.  The cap is 1MB today but nobody (as in not a single block in the history of Bitcoin) is 1MB.  So the cap is really acting just as a safety mechanism to prevent malicious activity.  The real "economic cap" is well below the 1MB limit.  That won't change if/when the cap is raised so the thinking should be what cap still protects the Bitcoin network from centralization and blockchain bloating attacks.   With current technology I think 10MB is viable.

Viable? Based on what?

Notably orphan cost doesn't work the way you think it does - under certain conditions, conditions while not yet fully present are likely to be present in the future, it is in your benefit as a miner to avoid propagating blocks you mine more widely than a certain threshold. It's kinda like the selfish miner attack, but even more fundamental in that you gain through your inaction rather than explicit actions.

Anyway, this just points to how we need more people working on rigorous analysis of this stuff, rather than empty debate based on nothing... Pity really that the people behind the selfish miner paper are so short-sighted in how they handle PR.

[Cryddit]

We wouldn't be having this problem if blocks that don't have at least a quarter of the transactions currently in the mempool would, like tx without fees, not propagate across the network.

[gmaxwell]

We wouldn't be having this problem if blocks that don't have at least a quarter of the transactions currently in the mempool would, like tx without fees, not propagate across the network.

Then someone starts maximum rate dust flooding transactions and people are forced to include them. Good job. Worse— they send floods of orthogonal transactions to different nodes on the network and now can freely trigger forking by effectively partitioning block propagation.

[DeathAndTaxes]

We wouldn't be having this problem if blocks that don't have at least a quarter of the transactions currently in the mempool would, like tx without fees, not propagate across the network.

Actually the issue is more with free tx (or tx with unconfirmed inputs, or other tx which weird issues).  Very few paying tx with confirmed inputs are being "left behind".  Also gmaxwell point is spot on.  If you force miners to include tx you open up whole new attack vectors which don't exist now.  Even if your rule was implemented right now miners would already be "compliant" as far more than 25% of waiting paying tx are being included.

[ilpirata79]

and we can easily support more TPS than PayPal.

Off course, but that comes at a cost. I think Paypal is a good target because it's about internet payments that does not require very rapid confirmations. We could aim for a little more than paypal  and choose, for instance, a max block size that allows 1.2 or 1.5 of the paypal tps (so 12 or 15 mB).

On the other hand, I think that we should just forget about being able to address the same number and types of transactions visa and mastercard do. Such kinds of transactions (super-market, bar, etc.) have to be made off-chain. First because they require rapid confirmations. Second, because it is useless to register everything into the blockchain (I don't want to have even the coffee I buy registered into the blockchain. In fact, the less transactions inside the block chain, the better).

Best regards,
ilpirata79