How a floating blocksize limit inevitably leads towards centralization

[Jouke]

And what about having nodes build in some sorts of max block size relay delay, so they can punish miners that in their eyes produce too large blocks?

[MoonShadow]

there are things that can be done to disincentive miners from attempting to break that limit when it suits them.  Did you not understand what I was trying to communicate concerning adding a 'soft limit' propagation rule to the clients?  Truly harmless to the network and the client, and only somewhat damaging to the miner, unless a significant enough of a minority of clients participate in this rule, and the miner with an overlarge block ends up with an orphaned block as a direct result.  It doesn't have to work every time, just enough for the miners to notice.

It is also truly harmless, that is, utterly uneffective in influencing them in any way, to the big miners.

Sure it might impact miners who are too small to have direct dedicated high bandwidth pipes to the top 51% miners, but the top 51% miners will not even notice if every one of you irrelevant non-mining nodes all stick your heads in the sand at every block on the chain. You just don't matter to the big boys, the most you could do would be help the big boys to shake the small players out of the game faster.

-MarkM-

<sigh>

What if those big miners also had that same soft limit rule?  The mining pools of BTCGuild, 50BTC and Slush still collectively represent over 50% of the mining power of the bitcoin network, the rise of ASICs notwithstanding.  While it's true that small miners can leverage their bandwidth best by directly connecting to these three mining pools.  So, do these three mining pools have an economic incentive to disregard the community will and exceed the soft limit?  On the one hand, yes; for they stand to gain more transaction fees just as you guys presume.  On the other hand, no, for they depend upon their membership for that hashing power; and it's that hashing power that creates the other incentives to begin with.  All three pools depend upon their good reputations with their membership in order to dominate.  If even a minority of their membership disagreed with the owners of the pools exceeding the soft limits without the community's overall consent, the pools will suffer hashrate loss.  Thus, all three of these pools actually have a greater incentive to participate in the convention set by the community, and resist any players that would not abide.  They are, in fact, most likely among nodes to impose such a propagation rule upon their own systems; partly because of they control a large percentage of the mining power.

In fact, I would here propose that, at the same time that we are raising the soft limit to 500KB, we also ask these major mining pools to include my soft limit propagation rule.  Just these three alone would make such a rule quite effective.

[markm]

And what about having nodes build in some sorts of max block size relay delay, so they can punish miners that in their eyes produce too large blocks?

there are things that can be done to disincentive miners from attempting to break that limit when it suits them.  Did you not understand what I was trying to communicate concerning adding a 'soft limit' propagation rule to the clients?  Truly harmless to the network and the client, and only somewhat damaging to the miner, unless a significant enough of a minority of clients participate in this rule, and the miner with an overlarge block ends up with an orphaned block as a direct result.  It doesn't have to work every time, just enough for the miners to notice.

It is also truly harmless, that is, utterly uneffective in influencing them in any way, to the big miners.

Sure it might impact miners who are too small to have direct dedicated high bandwidth pipes to the top 51% miners, but the top 51% miners will not even notice if every one of you irrelevant non-mining nodes all stick your heads in the sand at every block on the chain. You just don't matter to the big boys, the most you could do would be help the big boys to shake the small players out of the game faster.

-MarkM-

[Nagato]

Only someone who assumes that only a hard limit is actually the only limiting factor.  Again, we've already bumped up against the 250KB soft limit on several occasions; and amazingly, none of the miners have chosen to comment out that line of code.  Obviously there is some kind of cost to the miners for doing so, or a presumption of a cost, that would exceed the benefits to the miner for doing so.  One such cost is simply the work involved in commenting out that line of code and recompiling their mining programs over a relatively few number of tiny transaction fees.  That condition is bound to change, certainly.  This does not mean that miners will start to ignore the soft limit.  It is a community convention that, while not presently enforceble by the running protocol, is still enforceable by the community.  If some miner actually did this prior to the community deciding to raise that soft limit, the offender would prompt some kind of response.  What kind of response, I won't hazard a guess; but those miners certainly don't desire to prompt the community to include rules more strict than what I'm proposing.  It's not in the interest of the miners to invoke a response.

I don't really buy the community argument. I have seen blocks above 250KB btw.
If you look at the total fees for blocks hitting 250KB, it is almost never above 1BTC, and these blocks still include many free transactions, so it is safe to assume that most if not all the transactions that were left out were free ones. Why bother commenting out that line of code when it brings no additional benefit to you?

You will probably not see miners accepting blocks above 250KB until 250KB blocks filled with only paying TX appear.

[MoonShadow]

And what about having nodes build in some sorts of max block size relay delay, so they can punish miners that in their eyes produce too large blocks?

That's very similar to what I proposed earlier, except the relay delay would be infinity.

[MoonShadow]

Only someone who assumes that only a hard limit is actually the only limiting factor.  Again, we've already bumped up against the 250KB soft limit on several occasions; and amazingly, none of the miners have chosen to comment out that line of code.  Obviously there is some kind of cost to the miners for doing so, or a presumption of a cost, that would exceed the benefits to the miner for doing so.  One such cost is simply the work involved in commenting out that line of code and recompiling their mining programs over a relatively few number of tiny transaction fees.  That condition is bound to change, certainly.  This does not mean that miners will start to ignore the soft limit.  It is a community convention that, while not presently enforceble by the running protocol, is still enforceable by the community.  If some miner actually did this prior to the community deciding to raise that soft limit, the offender would prompt some kind of response.  What kind of response, I won't hazard a guess; but those miners certainly don't desire to prompt the community to include rules more strict than what I'm proposing.  It's not in the interest of the miners to invoke a response.

I don't really buy the community argument. I have seen blocks above 250KB btw.
If you look at the total fees for blocks hitting 250KB, it is almost never above 1BTC, and these blocks still include many free transactions, so it is safe to assume that most if not all the transactions that were left out were free ones. Why bother commenting out that line of code when it brings no additional benefit to you?

You will probably not see miners accepting blocks above 250KB until 250KB blocks filled with only paying TX appear.

It's been a while, but IIRC, the soft limit only counts fee paying transactions.  I certainly could be wrong about that, but if not, a block with an actual size over 250KB is possible without violating the soft limit.  Any such propagation rule would have to consider the blocksize in the same manner, and perhaps also add a fudge factor.

[Jouke]

And what about having nodes build in some sorts of max block size relay delay, so they can punish miners that in their eyes produce too large blocks?

That's very similar to what I proposed earlier, except the relay delay would be infinity.

Yes I know, but I guess that could bring you in a situation that you won't be aware of any blocks in the blockchain because nodes around you won't relay you the blocks.

[Jouke]

Sure it might impact miners who are too small to have direct dedicated high bandwidth pipes to the top 51% miners, but the top 51% miners will not even notice if every one of you irrelevant non-mining nodes all stick your heads in the sand at every block on the chain. You just don't matter to the big boys, the most you could do would be help the big boys to shake the small players out of the game faster.

Indeed, if 51% of the miners decide to work together we will have a problem.

[Rampion]

Please don't forget that what attracted most of us the users is the decentralized nature of bitcoin. We want a secure p2p network that serves as a store of value.

I feel your pain - managing the blockchain with increasing transactions per second will be a pain - but the top priority should always be avoiding centralization.

[markm]

If I recall correctly, pretty much the only, or at least the core, argument in favour of allowing any free transactions at all is as a propaganda / public-relations scheme to suck in more users. Basically a free loss-leader.

Aren't we getting to a point where anyone who wants to blow wealth on giving out free loss-leaders can do so themselves, simply by paying the damn fee themselves or lowering the prices of what they are selling by the amount of the fees involved in the actual selling of the thing?

For example, merchant terminals could add an input from their own float account to the transaction before presenting it to the buyer for signing of the buyer's input, or publish separately a transaction with fee that takes one or more recent payments of customers as input. (This latter though would make verifying blocks harder, since verifiers would not be able to tell whether a free transaction has sneaked into the block simply by checking whether the transaction contains a fee. So maybe free transactions are not yet being rejected by verifiers simply because verifiers are currently too simplistic to recognise cases in which the fee is being provided by a dependent transaction? We could however, make each block able to be verified as to its lack of free-riders by requiring any such method of providing its fee be in the same block.)

-MarkM-

[misterbigg]

Again, we've already bumped up against the 250KB soft limit on several occasions; and amazingly, none of the miners have chosen to comment out that line of code.

It's been said a few times in this thread that there is little incentive to change the code as long as the block subsidy (25 BTC reward) is orders of magnitude larger than the total of transaction fees in the block.

I'm pretty sure that the 250KB limit has never been broken to date.

But it has. If you will go back to one of the earlier posts in this thread you will see that there is at least one mining pool with customized software that does not have the 250kb limit. You can also scan through the block chain and find plenty of blocks larger than 250kb. There are also a good number that have no transactions at all. The presumption is that these came from a miner whose only interest was in collecting the subsidy. As long as the subsidy is orders of magnitude larger than the total of transaction fees, this type of behavior will persist (as has been stated numerous times already).

Show me why you believe that this is so, not just your assumptions about what the future holds using overly simplified mental models.

Was this earlier post insufficient?

If we want to cap the time of downloading overhead the latest block to say 1%, we need to be able to download the MAX_BLOCKSIZE within 6 seconds on average so that we can spend 99% time hashing.

At 1MB, you would need a ~1.7Mbps  connection to keep downloading time to 6s.
At 10MB, 17Mbps
At 100MB, 170Mbps

and you start to see why even 100MB block size would render 90% of the world population unable to participate in mining.
Even at 10MB, it requires investing in a relatively high speed connection.

It seems that all of your relevant points have been answered at least once so now I'm just being redundant, and no longer adding value to the conversation.

[misterbigg]

You will probably not see miners accepting blocks above 250KB until 250KB blocks filled with only paying TX appear.

Yep. I mentioned that in my prediction for what will happen to the soft limit.

[markm]

Re sucking in more users, if it is precisely an increase in the number of users that threatens us with all these problems in the first place, maybe it is time that we stopped being so eager to suck in more users. We are working nicely so far with the users we already have. If more want in that is going to cost us a lot, so using freebies that cost the whole population of full nodes, including miners, a lot, precisely to attract more new users faster, does not seem that great an idea.

-MarkM-

[jayson3]

It sounds like the core problem is a bottleneck in getting all of the transactions mined within the time desired and within bandwidth constraints of typical miners.  Faster transactions via large blocks may result in a few mega-bandwidth miners, but limiting bandwidth via small blocks slows transactions and reduces usefulness of bitcoin.

I know this is radical, but perhaps the answer is parallel processing.

a) Sort all of the existing and future bitcoins into two (or more) parallel and independent sub-chains using some existing sort-able attribute.
b) Clients operate on both sub-chains, but this is completely hidden from users.  A bitcoin is still a bitcoin.
c) When Bob send coins to Alice, Bob's client attempts to select a bundle of coins from his wallent that are all from the same sub-chain.  If not possible, it will use coins from both chains to make the payment.
d) Now that there are two chains, two miners can work in parallel, each on a different sub-chain.  Assuming the process in (c) is usually successful, Bob's coins to Alice will be a transaction on just one sub-chain.  Overall, this makes the blocks on each subchain much smaller, avoiding the bottleneck without increasing bandwidth.

notes: 
-coins never change sub-chains.  These are separate chains masked by the client.
-new coins would be equally distributed among the chains.  21 million overall cap remains.
-more than 2 subchains could be used.
-clients would have to split and re-assemble payments via multiple sub-chains when a pure bundle cannot be made. micro-payments could usually be from a single sub-chain

I am sure there are good reasons why this cant work, and perhaps its a coding nightmare, but maybe it will stimulate other lines of thought

[hazek]

I don't like that solution either. It's not linked to actual usage in any way and could lead to issues. Simply doubling it based on time is a totally arbitrary decision.

All decisions about rules in Bitcoin are arbitrary, I really don't understand why you keep throwing that around as if it's a reason not to consider one.

[Nagato]

One of the reasons why high bandwidth is required is because you need it in bursts every 10mins(on average).

Sending blocks with only hashes of TX is a start.

Any other optimisations which reduce the download size within that 6s period either by pre-downloading known information or by downloading unimportant data(data not needed to begin mining next block) later once mining has commenced will help to drastically reduce bandwidth requirements.

After all the only real discovery in a new block is the nonce value.

[markm]

I don't like that solution either. It's not linked to actual usage in any way and could lead to issues. Simply doubling it based on time is a totally arbitrary decision.

All decisions about rules in Bitcoin are arbitrary, I really don't understand why you keep throwing that around as if it's a reason not to consider one.

It is not even arbitrary in itself, once you already accept halving the block subsidy. It is simply an attempt to give the miners more space to sell to make up for lowering their subsidy.

Now if you want to argue that changing the subsidy over time is arbitrary, leading to the limit of 21,000,000 coins being arbitrary, well yes you've got me there. But, the actual total number of coins is irrelevant as no matter how many coins "100% of all bitcoins" happens to be it remains "100% of all bitcoins" so how many that is in various units of measure other than "fractions of the whole" it happens to be is kind of irrelevant (and making it look different cosmetically like calling it millis or macros or megas or picos or whatever is purely a user interface design issue).

-MarkM-

[hazek]

I don't like that solution either. It's not linked to actual usage in any way and could lead to issues. Simply doubling it based on time is a totally arbitrary decision.

All decisions about rules in Bitcoin are arbitrary, I really don't understand why you keep throwing that around as if it's a reason not to consider one.

It is not even arbitrary in itself, once you already accept halving the block subsidy. It is simply an attempt to give the miners more space to sell to make up for lowering their subsidy.

I like that reasoning.

[cabin]

Could we float the blocksize based on 'network centralization'? If the last x blocks were all mined by the same few large entities then the blocksize decreases which leads to transaction fees rising, bandwidth costs decreasing, and hopefully more small miners joining in.

[markm]

Could we float the blocksize based on 'network centralization'? If the last x blocks were all mined by the same few large entities then the blocksize decreases which leads to transaction fees rising, bandwidth costs decreasing, and hopefully more small miners joining in.

Only if all the folk we are worried about clearly identify blocks they make as having been made by them.

Absent voluntary identifying of themselves who made a block is unknown, they are pseudonymous just like any other bitcoin-address.

And even if they do vokuntarily identify blocks they make that they want us to know were made by them that would not force them not to make any other blocks they don't tell us were theirs.

-MarkM-