Block size limit automatic adjustment

[MoonShadow]

I'm not smart enough to figure this out. I wish Satoshi would weigh in on this issue. I suspect he may have already envisioned the outcome.

Austrian economic theory says that no one is smart enough, because no one can have all of the information.  The best that we can do is take a guess, and I would guess that it isn't going to be a real problem.  Certainly not in my lifetime.

[RHorning]

I'm not smart enough to figure this out. I wish Satoshi would weigh in on this issue. I suspect he may have already envisioned the outcome.

Austrian economic theory says that no one is smart enough, because no one can have all of the information.  The best that we can do is take a guess, and I would guess that it isn't going to be a real problem.  Certainly not in my lifetime.

The main issue raised by this thread does seem like something of concern for the long-term health of this project, as there does seem to be the very real possibility that the current limit is not going to be sufficient for "ordinary" transactions at some point in the future.  Using examples for data processing rates and transaction rates for other payment processing systems like PayPal, this current limit is not only going to be insufficient but woefully insufficient.  I realize we aren't anywhere near those demand levels, but it still is an issue to think about.

There are also plenty of examples where decisions of software architecture including the use of constants or other features in software architecture have profound real-world impact simply because the software design team has been short sighted and didn't anticipate the future very effectively.  Examples of this include the Y2K bugs, the Unix 2038 date overflow bug (remains to be seen how it will be completely solved), and perhaps most similar to this current situation is the IPv4 address space issue.  There are other instances where a coded constant of some kind also can come up and bite end-users in unexpected ways... one of the reasons computer software developers call these kind figures "magic numbers".  When you have some very intelligent people who are complaining about an issue of this nature as having some significant impact, it is at least something which needs some attention.

The specifics on how to avoid this problem is the point of this thread, and a strong suggestion that "rules" ought to be incorporated into the network in terms of how to somehow allow this hard coded limit.

The moral of this story is that the non-generating clients operate on the network at the pleasure of the miners. The miners are effectively in control of the "health" of the network and the current block size limits reflect that. So for example block http://blockexplorer.com/b/92037 is about 200455 bytes long and mostly contains spam. Normal blocks max out at 50k. This shows that at least one generator has chosen to waive the current fees scheme. I think that letting miners effectively decide their own fees scheme will be seen to be the least bad option.

I will say in regard to the control of the network by the miners, that is mostly true but not 100% of the time.  Blocks sent out by miners can also be rejected by "the vast masses of clients" who simply refuse to recognize a block.  Perhaps some other miner that fits within the rules set up by the network will do something that another miner doesn't take into consideration, and that particular block is simply going to be rejected.  With the rules as currently established, a miner who chooses to create a very large block is simply going to have that block ignored by the current network.  Essentially this is "proof" that the miners don't have absolute authority here.  Miners also work at the pleasure of the network as a whole, and have "constitutional limits" imposed upon them by the networking rules.  This particular issue with the maximum block size is one of the few rules that is outside of the control of a single miner.  Other kinds of similar rules could be adopted by a significant portion of the clients that may exclude certain miners or even groups of miners providing a check to a sort of "tyranny of the miners".

I'm not going to speculate about how such rules might be established or what other potential rules might be, other than suggesting that the block limit rule is one such rule and that needs to be somewhat reconsidered, certainly as a fixed size.  The long-term consequence is that without being changed, transaction fees may potentially escalate to absurd levels as more people trying to get the network to incorporate a particular transaction becomes a sort of "fees arms race", particularly when miners simply would be unable to get blocks of a larger size incorporated into the network.

The opposite situation is a voluntary self-limiting feature on miners who simply choose not to grow blocks to large sizes.  As long as somebody somewhere is allowed to have an arbitrarily large block, it will deal with the transactions with a low fee or perhaps no fee at all, even if it will take awhile to get those blocks incorporated into the network.

[MoonShadow]

I'm not smart enough to figure this out. I wish Satoshi would weigh in on this issue. I suspect he may have already envisioned the outcome.

Austrian economic theory says that no one is smart enough, because no one can have all of the information.  The best that we can do is take a guess, and I would guess that it isn't going to be a real problem.  Certainly not in my lifetime.

The main issue raised by this thread does seem like something of concern for the long-term health of this project, as there does seem to be the very real possibility that the current limit is not going to be sufficient for "ordinary" transactions at some point in the future.

I was responding specificly to his concern about transaction fees, not the block size limit.  The hard limit is a real concern, but also one that I imagine has been well considered by others before us.  Did anyone bother to search the archives before diving into this thread?  The block limit exists to prevent spamming from packing the blocks, not support transaction fees.  I think that the recently instituted priority rule does the job at least as well as the hard block limit, but it's not enough on it's own. 

[Alex Beckenham]

Has anyone had any further thoughts on dynamic block size limit, in light of the recent slow down of free transactions recently?

Is the block size limit still a concern?

[MoonShadow]

Has anyone had any further thoughts on dynamic block size limit, in light of the recent slow down of free transactions recently?

Is the block size limit still a concern?

It never really was a real problem except for future scalability.

[Alex Beckenham]

Is the block size limit still a concern?

It never really was a real problem except for future scalability.

Doesn't that by itself make it a real problem?

[MoonShadow]

Is the block size limit still a concern?

It never really was a real problem except for future scalability.

Doesn't that by itself make it a real problem?

Well, yes.  But the talk recently is about how maintaining the transaction fees in order to maintain the hasing power of the network, not scalability.  This particular thread was mostly about scalability, and in such a case the max block limit can be raised or removed.  It's only present as a backstop against the possibility of there being some presently unknown exploit that would permit limitless transaction spamming of the blockchain, and not as a means to support the transaction fees.  The short answer to the scalability issue is that it can be easily removed long before the network traffic is high enough that the max block size becomes an actual scalability issue.

[Alex Beckenham]

Okay great. My understanding from reading this thread was that it was not so 'easily' removed; Having to get every client updated at basically the same time.

Good to read otherwise.

[MoonShadow]

Okay great. My understanding from reading this thread was that it was not so 'easily' removed; Having to get every client updated at basically the same time.

Good to read otherwise.

No, not at the same time.  Just before hitting that limit were to become a regular event.  We could change that rule in the next vanilla client release, so long as everyone agreed that we should, and voiced consent by downloading the new client.  At the current rate we still have months, if not years, before hitting that limit regularly.  As far as I know, we have never come close to it.

If we wait until every other block is hitting that limit; however, implementing such a rule change is going to be problematic.

[caveden]

Okay great. My understanding from reading this thread was that it was not so 'easily' removed;

It could be relatively easy now or even in some months from now. But once bitcoin goes mainstream, doing such a change will not be that easy, mainly due to the coordination effort. And the more popular bitcoin gets, the harder such change becomes, that's why I still think it should be done just once. And to do it just once, a self-adjusting rule should be created... just raising a constant is bad for several reasons (multiple backward incompatible changes, gives more space for spammer-miners to abuse on, risk of dropping the fee value and by consequence the difficulty factor in the future etc)

[Mike Hearn]

Block size limits are only relevant to miners, as it's they who decide whether to "accept" a block by building up on it or not. Most users will end up on lightweight clients which don't need to check the block size. So as long as mining consolidates around professionals who communicate and keep up, there probably won't be a "doomsday" scenario.

[caveden]

Block size limits are only relevant to miners, as it's they who decide whether to "accept" a block by building up on it or not. Most users will end up on lightweight clients which don't need to check the block size. So as long as mining consolidates around professionals who communicate and keep up, there probably won't be a "doomsday" scenario.

Like the professional ISPs who have waited until the last minute to migrate from IPv4 to IPv6 - not to mention that many have not yet migrated? Or like the also professional SMTP servers who have never implemented some stronger authentication system in e-mail transfers?

I'm not saying it's impossible, neither that there will be a "doomsday" due to this. I'm just saying it's a problem that will need to be fixed someday, and the earlier it's done, the easier it is. If it's done too much later, it may provoke avoidable problems like long chain splits, people not understanding why their bitcoin is not working anymore etc.

And it's not an issue only to miners, every full client performs block validations. I don't think miners will be the only ones running full clients. Those who serve the lightweight clients for ex., they will need to be full clients.

[Mike Hearn]

Or like the also professional SMTP servers who have never implemented some stronger authentication system in e-mail transfers?

Virtually all large, professionally run SMTP networks do authenticate their mail, as far as I know. We track how much mail is authenticated coming into the Google network (ie: gmail consumer/business editions), and it's pretty high. There's a long tail of home-run SMTP servers that will never be upgraded but they also don't represent a whole lot of users.

And it's not an issue only to miners, every full client performs block validations. I don't think miners will be the only ones running full clients. Those who serve the lightweight clients for ex., they will need to be full clients.

Yeah, we might want to change that :-) I'll ask Gavin about it next time I see him on IRC. I think non-miners don't need to check the block size even if they are full nodes as an attempt to explode the size of the chain maliciously will be overridden by genuine miners pretty quickly. There's a risk of temporarily flooding the broadcast network with a gigantic block but it's not an easy attack to pull off. Nodes can prune blocks on side chains after a while so the increase in storage required would be temporary. If anyone ever did it, a patch to forcibly delete a block from storage would be written pretty fast.

[ribuck]

What's the current failure mode? What happens if the existing Bitcoin client encounters an over-long block?

[theymos]

What's the current failure mode? What happens if the existing Bitcoin client encounters an over-long block?

It is considered invalid and rejected.

[MoonShadow]

Okay great. My understanding from reading this thread was that it was not so 'easily' removed;

It could be relatively easy now or even in some months from now. But once bitcoin goes mainstream, doing such a change will not be that easy, mainly due to the coordination effort. And the more popular bitcoin gets, the harder such change becomes, that's why I still think it should be done just once. And to do it just once, a self-adjusting rule should be created... just raising a constant is bad for several reasons (multiple backward incompatible changes, gives more space for spammer-miners to abuse on, risk of dropping the fee value and by consequence the difficulty factor in the future etc)

I have already laid out my plan for a self adjusting limit, based on the average number of blocks a fee paying transaction spends in the average queue, but with some fudge factor to allow for different miners having slightly different averages across the network.  To account for fee rates, the average could be weighted by the amount of the fee in such a manner that a fee larger than the minimum gets affects the average more than the average transaction fee.

The space available for free transactions could be set static, or as a percentage of the moving blocksize.  The blocksize could be adjusted when the difficulty is adjusted, and the winning block first after the change encodes the calculated max limit into that block in some fashion.  If the network accepts that block, then that max limit becomes a fixed limit for another 2015 blocks.

[ribuck]

Here's an idea that might be dismissed as stupid-simple, but sometimes stupid-simple ideas work really well.

How about: The maximum block size equals the higher of: (a) the current hard-coded maximum block size, and (b) 'difficulty' bytes.

[MoonShadow]

Here's an idea that might be dismissed as stupid-simple, but sometimes stupid-simple ideas work really well.

How about: The maximum block size equals the higher of: (a) the current hard-coded maximum block size, and (b) 'difficulty' bytes.

That's awesome.  And I agree it's sledgehammer simple.  I like that rule better than mine.  The free section, and the tiered fee schedule, would have to become percentages of that number; also sledgehammer simple.  No fudge factors.  Elegant.

EDIT:  Rule (b) might have to be some agreed upon multiple of difficulty, however.  If the blocksize does not naturally increase until difficulty is over one million, I'm afraid that we really would have some scalability issues.  Can anyone think of a metric that can be used for that multiple, or must it be fixed?

EDIT #2:  And Rule (a) should be reduced by half at least.

[Mike Hearn]

There are actually several block size limits. Practically speaking, the client holds incoming messages in RAM. Gigabyte sized blocks would require a gigabyte of RAM to receive. Any block size limit would have to ensure the max message size is also adjusted to take it into account, at least until blocks are distributed as header+tx hash lists.

We can probably just set it to a gigabyte max for non-miners and forget about it for a while. That's approximately what it'd take to keep up with VISA - might as well aim high, right? :-) Miners can be more distinguishing as long as they're responsive.

Automatic adjustments based on difficulty assume difficulty will scale with traffic. I'm not convinced that relationship will hold. If there's going to be an automatic formula (median size of recent blocks * 1.1) seems like as good as any, and is also simple.

[sandos]

Here's an idea that might be dismissed as stupid-simple, but sometimes stupid-simple ideas work really well.

How about: The maximum block size equals the higher of: (a) the current hard-coded maximum block size, and (b) 'difficulty' bytes.

I was just thinking that block-size should depend on some measurement of how much it taxes the network, just like difficulty is measuring how fast blocks can be found and corrects that rate.

I have a hard time seeing how to objectively measure block-size impact (which is global!) in a similar way as mining?