Bitcoin Forum

My concern is that there is little room for error with geometric growth.  Lets say that things are happily humming along with bandwidth and block size both increasing by 50% per year.  Then a decade goes by where bandwidth only increases by 30% per year.  In that decade block size grew to 5767% while bandwith grew to 1379%.  So now peoples connections are only 24% as capable of handling the blockchain.

Not a big deal?  Well, except that we can expect the power of nodes to follow some sort of curve ("exponential" in the vernacular) such that most nodes are barely above the threshold to be viable.  Meaning that this event would mean that the majority of nodes would shut down, likely permanently.

    // Size limits
    if (block.vtx.empty() || block.vtx.size() > MAX_BLOCK_SIZE || ::GetSerializeSize(block, SER_NETWORK, PROTOCOL_VERSION) > MAX_BLOCK_SIZE)
        return state.DoS(100, error("CheckBlock() : size limits failed"),
                         REJECT_INVALID, "bad-blk-length");

(e.g. by using transaction hashes and/or IBLT), once implemented, will certainly keep the new block message size growth rate much lower than the bandwidth growth rate.  

Not a big deal?  Well, except that we can expect the power of nodes to follow some sort of curve ("exponential" in the vernacular) such that most nodes are barely above the threshold to be viable.  Meaning that this event would mean that the majority of nodes would shut down, likely permanently.

Keep in mind these techniques don't reduce the amount of data that needs to be sent (except, at most, by a factor of two). They reduce the amount of latency critical data. Keeping up with the blockchain still requires transferring and verifying all the data.

Right. There is a decentralization trade-off at the margin.  But this isn't scaleless-- there is _some_ level, even some level of growth which presents little to no hazard even way down the margin.   The a a soft stewardship goal (not a system rule, since it can't be) the commitment should be that the system should be run so that it fits into an acceptable portion of common residential broadband, so that the system does not become dependant on centralized entities. As some have pointed out, being decenteralized is Bitcoin's major (and perhaps only) strong competitive advantage compared to traditional currencies and payment systems. How to meet that goal best is debatable in the specifics.

At the moment there are a bunch of silly low hanging fruit that make running a node more costly than it needs to be; we're even at a case where some people developing on Bitcoin core have told me they've stopped running a node at home. It's hard to reason about the wisdom of these things while the system is still being held back by some warts we've long known how to correct and are in the process of correcting.

It doesn't make sense to guess at this.  Any guess is bound to be wrong.
If after picking the low hanging fruit, there is still an issue here (and there may be).
It ought not be resolved by a guess when there is data within the block chain that would be useful for making a determination on max block size.
In the same way that difficulty adjustment is sensitive to data within the block chain, so also this could be.

I don't know what the right answer is anymore than Gavin does, but making an estimation would not be the best way to solve this in any case.

.....
One example of a better way would be to use a sliding window of x number of blocks 100+ deep and basing max allowed size on some percentage over the average while dropping anomalous outliers from that calculation.  Using some method that is sensitive to the reality as it may exist in the unpredictable future give some assurance that we won't just be changing this whenever circumstances change.
Do it right, do it once.

There isn't a way to predict what networks will look like in the future, other than to use the data of the future to do just that.

Is this 50% per year intended to be a hardcoded rule like the block reward?

That's not how I interpreted Gavin's report. It sounded more like a goal that the developers thought was attainable.

That said, 50% per year does seem aggressive. At some point, the opportunity cost of including more transactions is going to exceed the tx fee value, certainly as long as the block reward exists, so the blocksize cannot increase indefinitely. And so what if there is little room in the blockchain? Not every single tiny transaction needs to be recorded indefinitely. Since the (I expect) cost of increasing the block size is increased centralization, shouldn't the developers be hesitant to make such a commitment without allowing for discretion?

I also wonder what the best approach will be, way out in the future, when the block reward is near zero. Can there be an equilibrium transaction fee if the difficulty is allowed to continue to fall? A simple, kludgy solution might be to fix the difficulty at some level, allowing blockrate to depend on the accumulated bounty of transaction fees.

Though I'm sure some new kind of proof of work/stake approach could best solve this problem and make the network more secure and cheaper.

It also may be contrary to the eventual goal of usage driven mining, where transaction fees ultimately overtake block reward in value.  This proposal may drive TX fees to zero forever.  Block chain is a somewhat scarce resource, just as total # of coins.  Adding an arbitrary 50% yearly inflation changes things detrimentally.

If this forks as currently proposed, I'll be selling all my BTC on Gavin's fork and mining on the other.  I suspect I will not be the only one.

I'm sending a follow-up blog post to a couple of economists to review, to make sure my economic reasoning is correct, but I don't believe that even an infinite blocksize would drive fees to zero forever.

Commodity prices never drop to zero, no matter how abundant they are (assuming a reasonably free market-- government can, of course supply "free" goods, but the results are never pretty). The suppliers of the commodities have to make a profit, or they'll find something else to do.

That has very little to do with whether or not transaction fees will be enough to secure the network in the future. I think both the "DON'T RAISE BLOCKSIZE OR THE WORLD WILL END!" and "MUST RAISE THE BLOCKSIZE OR THE WORLD WILL END!" factions confuse those two issues.

I don't think adjusting the block size up or down or keeping it the same will have any effect on whether or not transaction fees will be enough to secure the network as the block subsidy goes to zero (and, as I said, I'll ask professional economists what they think).

Okey dokey. You can join the people still mining on we-prefer-50-BTC-per-block fork (if you can find them... I think they gave up really quickly after the 50 to 25 BTC subsidy decrease).

One example of a better way would be to use a sliding window of x number of blocks 100+ deep and basing max allowed size on some percentage over the average while dropping anomalous outliers from that calculation.  Using some method that is sensitive to the reality as it may exist in the unpredictable future give some assurance that we won't just be changing this whenever circumstances change.
Do it right, do it once.

There isn't a way to predict what networks will look like in the future, other than to use the data of the future to do just that.  Where we are guessing we ought acknowledge that.

Commodity prices never drop to zero, no matter how abundant they are (assuming a reasonably free market-- government can, of course supply "free" goods, but the results are never pretty). The suppliers of the commodities have to make a profit, or they'll find something else to do.

Not only that, but there will always be non-infinite bandwidth and storage available to users, while anyone can create transaction spam essentially for free. So minimum fees remain necessary for non-priority transactions.

Okey dokey. You can join the people still mining on we-prefer-50-BTC-per-block fork (if you can find them... I think they gave up really quickly after the 50 to 25 BTC subsidy decrease).

Check your assumptions.

1) We don't know what the future networks will look like.

2) Commodity prices do go to zero for periods of time.  Sometimes they rot in silos, and cost money to dispose of them (negative worth).

What would happen if the blocksize were increased to 1 GB tomorrow? Pretty much nothing. Miners will always be able to create blocks less than the maximum blocksize.
What would happen if the blocksize were decreased to 1 KB tomorrow? Bitcoin would come grinding to a halt.

Too small blocksize = death to bitcoin.
Too big blocksize = non-issue.

I'd rather see the blocksize too big than too small.

IBLT makes it an issue because there would no longer be a risk/reward tradeoff on tx fees vs propagation delay in building the largest possible blocks. As a result the miner is incentivized to always build the largest possible block to collect maximum tx fees with no propagation risk.

What would happen if the blocksize were increased to 1 GB tomorrow? Pretty much nothing. Miners will always be able to create blocks less than the maximum blocksize.
What would happen if the blocksize were decreased to 1 KB tomorrow? Bitcoin would come grinding to a halt.

Too small blocksize = death to bitcoin.
Too big blocksize = non-issue.

I'd rather see the blocksize too big than too small.

No comment on this?

There isn't a way to predict what networks will look like in the future, other than to use the data of the future to do just that.  Where we are guessing we ought acknowledge that.

The last mile is typically the speed bottleneck in communication networks; its bandwidth limits the bandwidth of data that can be delivered to the customer. This is because retail telecommunication networks have the topology of "trees", with relatively few high capacity "trunk" communication channels branching out to feed many final mile "leaves". The final mile links, as the most numerous and thus most expensive part of the system, are the most difficult to upgrade to new technology. For example, telephone trunklines that carry phone calls between switching centers are made of modern optical fiber, but the last mile twisted pair telephone wiring that provides service to customer premises has not changed much in 100 years.

Fiber-optic communication is a method of transmitting information from one place to another by sending pulses of light through an optical fiber. The light forms an electromagnetic carrier wave that is modulated to carry information. First developed in the 1970s, fiber-optic communication systems have revolutionized the telecommunications industry and have played a major role in the advent of the Information Age. Because of its advantages over electrical transmission, optical fibers have largely replaced copper wire communications in core networks in the developed world. Optical fiber is used by many telecommunications companies to transmit telephone signals, Internet communication, and cable television signals. Researchers at Bell Labs have reached internet speeds of over 100 petabits per second using fiber-optic communication.

why not linear growth, like  +n MB per block halving, or quadratic like +n MB per n'th block halving?

To answer your question of What would also happen if the block size were increased to 1 GB tomorrow is the introduction of new attack vectors, which if exploited would require intervention to resolve by miners, and development.

Like what? What "new" attack vectors? It is already quite cheap to attack the current 1 MB blocksize. What would it cost to attack a 1 GB blocksize vs the current 1 MB blocksize?

The cost is not that significant.  Heck, the whole BTC market cap is not that significant.

If there were 6 GB block size bloat per hour?
A financial attack could do this independently.
Miners could do this free-ish.
Small miners would fail, as would all hobby miners.

Full nodes would become centralized, increased 51% risks, etc.
These are just the obvious.  No more decentralisation for Bitcoin.

Note that a typical transaction is 500 bytes, so the typical transaction fee for low-priority transactions is 0.1 mBTC (0.0001 BTC), regardless of the number of bitcoins sent.

To spam the 1 MB blocksize takes roughly .2 BTC per block, or 1.2 BTC per hour. That's only $500 per hour.

To spam a 1 GB blocksize takes roughly 200 BTC per block, or 1200 BTC per hour. That's $500,000 per hour!

A 1 GB blocksize is far more costly to attack. We could increase the block size to 1GB now and nothing would happen because there aren't that many transactions to fill such blocks.

Why would we choose linear growth when the trend is exponential growth?

Because exponential growth is unsustainable

it is bound to cap at some point in the near future.

Not inherently. It depends on the rate of growth and what is growing. For example, a 1% per year addition to bandwidth is exceedingly conservative, based on historical evidence.

Define 'near future'. Is that 5 years, 10 years, 40? And what makes you say that? It's easy to make a general unsupported statement. Don't be intellectually lazy. Show the basis for your reasoning, please.

Of course one can say, let's put it 50% per year until the bandwidth stops growing that fast,
and then we fork again. But this only postpones the problem.  Trying to predict now  exactly when this happens, and to  program for it now, seems futile.

Okey dokey.  My latest straw-man proposal is 40% per year growth for 20 years. That seems like a reasonable compromise based on current conditions and trends.

You seem to be looking hard for reasons not to grow the block size-- for example, yes, CPU clock speed growth has stopped. But number of cores put onto a chip continues to grow, so Moore's Law continues.  (and the reference implementation already uses as many cores as you have to validate transactions)

I'm not making predictions on constants that we don't know; but when speaking
about exponential growth it  is not even necessary.  Want to know how fast the exponent
growth? Take your 50% growth, and just out of curiosity  see for which n your (1.5)^n exceeds
the number of atoms in the universe. Gives some idea.

The point being, with an exponent it's too easy to overshoot.

I had a current cost of $252/hour to spam a 1MB/block. --> $252.000/hour to spam a 1GB block.
With 1 MB block, if an attacker spam the blocks, the users have no way to counter the attack raising the fees they pay. To move the cost  for the attacker to $252.000/hour they should pay 10$ per transaction.
With 1 GB block, if the attacker spam the blocks, the users just need to move from 1 cent to 2 cent and the cost of the attack move from $252K to $504K per hours. At 4 cents per transaction it become $1M per hour.

Remember the cost of spamming the blocks go directly in the pockets of miners. So they can reinvest the money in better bandwidth and storage and move to 2GB blocks, doubling the cost for the attacker.
At $1M per hour it is $100M in four days and $750M in a month and $8.760 trillion per year (more than ten times the miners income today)

10: 769
...
20: 44337

A miner can put as many transactions as they like in a block with no fees.

Okey dokey.  My latest straw-man proposal is 40% per year growth for 20 years. That seems like a reasonable compromise based on current conditions and trends.

Why do we fear hard forks?  They are a highly useful tool/technique.  We do not know for sure what the future will bring.  When the unexpected comes then we must trust those on the spot to handle the issue.

so if the bandwith growth happens to  stop in 10 years

Theory and practice diverge here.
A miner can put as many transactions as they like in a block with no fees.
The cost is then replicated across every full node which must store it in perpetuity.

And the rest of the miners are free to ignore a block like that. You have yet to convince me there's a problem. Miners can fill blocks with worthless free transactions today.

Then maybe the problem isn't a large maximum blocksize, but the allowance of unlimited feeless transactions. They are not free for the network to store in perpetuity, so why not eliminate them?

Eliminate free transactions and eliminate the maximum blocksize. Problem solved forever.

Free transactions are nice and we can keep it that way if people don’t abuse them.

And the rest of the miners are free to ignore a block like that. You have yet to convince me there's a problem. Miners can fill blocks with worthless free transactions today.

Then maybe the problem isn't a large maximum blocksize, but the allowance of unlimited feeless transactions. They are not free for the network to store in perpetuity, so why not eliminate them?

Eliminate free transactions and eliminate the maximum blocksize. Problem solved forever.

I'll let Satoshi speak to that: 
Lets not break what needn't be broken in order to facilitate micropayments (the reason for this proposal in the first place, yes?)

How would you enforce those non-free transactions? If the transaction spammer is the miner then he can include any fee whatsoever because he pays himself. The only cost for him is that the coins are frozen for about 100 decaminutes.

You're thinking of the current network. We're talking about a hard fork. A hard fork could require all transactions to have some sort of minimum fee included.

Again, what is the point of minimum fee if the transaction isn't propagated? Miner-spammer includes the fee payable to himself. What's the point of that exercise?

Why would it? Why on earth would it???

You're thinking of the current network. We're talking about a hard fork. A hard fork could require all transactions to have some sort of minimum fee included.

While (I think we'd all agree  that) predicting technology decades ahead is hard,
 it is not impossible that a group of specialists,  after a thorough discussion, could
get the prediction about right.

This transmission experiment involved sending the equivalent of 400 DVDs per second over 7,000 kilometers, roughly the distance between Paris and Chicago.

Either we've gone through the looking glass, or else the goal is that Bitcoin should fail and some alt coin take its place?
Why hard fork Bitcoin to enable microtransactions if only to make them too expensive, as well as add risk and cost and also remove functionality in the process?

Gavin's 2nd proposal also seems worse than the first by the arbitrariness factor.  x20 size first year, for years two through ten x1.4, then stop.  

I linked you to the report of Bell Labs achieving 10Gbps over copper wire. Here is the link to them achieving 100 petabits per second over fiber in 2009:

http://www.alcatel-lucent.com/press/2009/001797


These are demonstrated capacities for these two mediums. The only limiting factors for achieving such rates for individual consumers are physical and economic considerations for building out the infrastructure. Nonetheless the technologies for achieving exponential increase in bandwidth over current offerings is proven. Achieving these rates in practice on a scale coinciding with historical exponential growth of 50% annually, which does take into consideration economic and physical realities, seems well within reason. I'm sure telecommunications specialists would agree.

Designing something to work and designing to not fail are entirely different endeavors and someone qualified for one is not necessarily qualified to even evaluate the other.

Then... who knows? Every prediction of "this will surely be enough technology" has turned out to be wrong so far.

Sure, we were also able to get x.25 and x.75 telecom to run over barbed wire, in the lab.  (There are places in the world that still use these protocols, some of which would deeply benefit from bitcoin in their area.)
The logistical challenges of implementation is not what you find in the lab.  
This stuff has to go out in environments where someone backs up their truck into a cross country line so they can cut it and drive off with a few miles of copper to sell as scrap.  We live in the world, not in the lab.

We're in luck then, because one advantage of fiber lines over copper is they're not good used for anything other than telecom :)

I'm no telecommunications specialist, but do have an electronics engineering background. Raise some issue with fundamental wave transmission and maybe I can weigh in. My understanding is it's easier to install fiber lines, for example, because there is no concern over electromagnetic interference. Indeed, the fiber lines I witnessed being installed a week ago were being strung right from power poles.

However, is such theoretical discussion even necessary? We have people being offered 2Gbps bandwidth (http://www.theverge.com/2013/4/15/4226428/sony-so-net-2gbps-download-internet-tokyo-japan) over fiber not in theory but in practice in Japan, today.

That's already orders of magnitude over our starting bandwidth numbers. I agree with Gavin that demand for more bandwidth is inevitable. It's obvious all networks are converging - telephone, television, radio, internet. We'll eventually send all our data over the internet, as we largely do now, but in ever increasing bandwidth usage. To imagine progress in technology will somehow stop for no apparent reason, when history is chock full of people underestimating what technological capacity we actually experience is not only shortsighted, it borders unbelievable.

Perhaps few disagree that Bitcoin can be improved by a plan for block size maximum adjustment.  My issues with the proposals are less what it achieves (a good thing) but what it doesn't (preventing this from happening in the future).

There are myriad external realities that we can not know about.  The development of the telecom technology is perhaps less the issue than what the world has in store for us in the coming decades.  I don't know, and no one else does either, but that shouldn't stop us from striving to achieve what has not been done before.

Undersea cables are cut accidentally, and by hostile actions, economic meltdowns and military conflicts halt or destroy deployments, plagues, natural disasters etc, OR new developments can accelerate everything, robots might do this all for us.  We can't know by guessing today what the right numbers will be.  We could be high or low.  I am just hoping that some more serious thought goes into avoiding the need to guess or extrapolate (an educated guess but still a guess).  We do not have a crisis today other than some pending narrow business concerns (some of which are on the board of TBF and possibly suggested that Gavin "do something").  I am also thankful that he is doing so.  This is an effort that deserves attention (even with the other mitigating efforts already in development).  Gavin is a forward thinking man, and is serving his role well.  We should be all glad that he is not alone in this, and that no one person has the power to make such decisions arbitrarily for others.

The difficulty adjustment algorithm works without knowing the future.  We should similarly look for a way that can also work for many generations, come what may, and save Bitcoin from as many future hard forks as we can. 

This is our duty, to our future, by virtue of us being here at this time.

It would seem that there could be a simple mathematical progressive increase/decrease, which is based on the factual block chain needs and realities of the time that can work forever into the future.

Here is an example that can come close to Gavin's first proposal of 50% increase per year.

If average block size of last 2 weeks is 60-75% of the maximum, increase maximum 1%, if >75% increase 2%
If average block size of last 2 weeks is 25-40% of the maximum decrease maximum 1%, if <29% decrease 2%

Something like this, would have no external dependencies, would adjust based on what future events may come, and won't expire or need to be changed.

These percentage numbers are ones that I picked arbitrarily.  They are complete guesses and so I don't like them anymore than any other number.  This is just to create a model of the sort of thing that would be better than extrapolating.  To do even better, we can do a regression analysis of previous blocks to see where we would be now and tune it further from there.

This may be manipulable:  miners with good bandwidth can start filling the blocks to capacity, to increase the max and push miners with smaller bandwidth out of competition.

I am just hoping that some more serious thought goes into avoiding the need to guess or extrapolate (an educated guess but still a guess).

It is offered as an example of the sort of thing that can work, rather than a finished product.

I happen to think we can do better than Gavin's idea.  I like the idea of trying to come up with a solution that works with the blockchain and adapts over time instead of relying on Gavin, or NL or whomever.  The answer should be in the blockchain.

It would seem that there could be a simple mathematical progressive increase/decrease, which is based on the factual block chain needs and realities of the time that can work forever into the future.

This can be easily gamed by stuffing transactions into the blockchain, shutting out smaller players prematurely.

Decreasing the block limit (note, not required block size) in the future would not be a hard fork, it would be a soft fork.

My 2uBTC on this issue:
...
I'd welcome comments / criticism of why having such a feedback mechanism is a good or bad idea.

This is the problem.

People don't seem to realize Gavin's proposal may be the best we can do. I happen to think it is. If anyone had a better idea we'd have heard it by now. We, the entire community, have brooded on this issue for months if not years now. Here is a spoiler alert: nobody can predict the future.

My 2uBTC on this issue:
Instead of guessing the costs of the network via extrapolation, code in a constant-cost negative feedback mechanism.  For example, similar to difficulty adjustments, if mean non-coinbase block reward > 1 BTC, increase max size.  If mean block reward < 1 BTC, decrease max size (floor of 1MB).

Here's why I think this is a long term solution.  With Bitcoin, "costs" and "value" have a very interesting relationship; currently with mining, the costs to run the network are determined by the exchange value of a bitcoin.  Long term, the block size constrains both the cost and value of the network.  By "long term", I mean 100 years from now.  Long term, there's no more coinbase reward.  So miners compete for transaction fees.  Limited block size causes transactors to compete for space in the block, driving up the fees.  An unlimited block size would, without other costs, tend to drive fees to near-zero, and then there's not enough incentive for miners to bother, and the security of the system is compromised.  That's the death spiral idea anyway, which may not actually happen, but it's a legitimate risk, and should be avoided.  The value and utility of bitcoin today has a lot to do with the probability that it will have value in 100 years.

Max block sizes doubling every two years makes them pretty much unlimited.  Capping after 20 years is also a big guess. That also extrapolates Moore's law for potentially longer than the law keeps going.  Gigabit ethernet is what, 15 years old?  And that's what every PC has now, I've never seen 10G over copper ethernet.  Reliance on everything else becoming awesome is a very fragile strategy.

An issue I have with exponentially increasing block size, or static block size, is there's no feedback, and can't respond to changes in the system.  The block size in many ways determines the value of the network. All else being equal, a network that can handle more transactions per day is more useful and more valuable.

I think that similar to the current system of mining costs determined by bitcoin value, block propagation, verification and storage should be determined by how much people are willing to pay.  If transaction fees are high, block space is scarce, and will expand.  If transaction fees are low, block space is too cheap, and the max block size will expand.

This fixes a cost independent of the mining coinbase reward, allowing for sustainable, predictable mining revenue.  The issue is we would have to come up with a number.  What should it cost to run the bitcoin network?  1% of M0 per year?  That would be 210,000 coins per year in transaction fees to miners.  That would be about 3BTC per block.

0.5% M0 annually would be 1.5BTC per block, and so on.  This would be a ceiling cost; it could cost less, if people didn't make too many transactions, or most things happened off-blockchain, and the blocks tended back towards the 1MB floor.  It would effectively put a ceiling on the maintenance cost of the network, however; if blocks were receiving 6BTC in fees, the size would double at the next difficulty adjustment, which would tend to push total fees down.

If you wanted to get fancy you could have hysteresis and non-linearity and stuff like that but if it were up to me I'd keep it really simple and say that max block size is a linear function of the previous epoch block rewards.

This can be "gamed" in 2 ways.  It can be gamed to a limited extent by miners who want to push up the max block size.  They can pay a bunch of fees to themselves and push up that average.  I can't think of a clean way to get rid of that, but hopefully that's OK; isn't it the miners who want smaller blocks anyway?  If miners are competing for larger blocks, why would the non-mining users complain?  The only issue is one miner who wants larger blocks, and everyone else wants smaller ones.  Maybe use median instead of mean to chop out malicious miners or fat-finger giant transaction fees.

It can also be gamed the other way.  Your transaction fee is 0, but you have some off-channel account with my mining group which includes all your txs for a flat monthly rate.  This also seems unlikely; if it were more expensive that way, transactors would stop using the off-channel method and just go to the open market for transaction inclusion.  If it were cheaper, why would the miner forgo that revenue?

So if I ran this whole Bitcoin thing (which would defeat the point... :), that's what I would do.  The question is how much it should cost.  1BTC per block sounds OK, it's nice round number.  That's 50K BTC per year for the miners.

I'd welcome comments / criticism of why having such a feedback mechanism is a good or bad idea.

My 2uBTC on this issue:
Instead of guessing the costs of the network via extrapolation, code in a constant-cost negative feedback mechanism.  For example, similar to difficulty adjustments, if mean non-coinbase block reward > 1 BTC, increase max size.  If mean block reward < 1 BTC, decrease max size (floor of 1MB).

I happen to think we can do better than Gavin's idea.  I like the idea of trying to come up with a solution that works with the blockchain and adapts over time instead of relying on Gavin, or NL or whomever.  The answer should be in the blockchain.

a miner can include into his block a transaction with an arbitrary large fee (which he gets back of course), throwing the mean off the chart.

The answer cannot be in the blockchain, because the problem being addressed (resource usage rising too quickly so only people willing to spend tens of thousands of dollars can participate as fully validating nodes) is outside the blockchain.

You will go down the same path as the proof-of-stake folks, coming up with ever more complicated on-blockchain solutions to a problem that fundamentally involves something that is happening outside the blockchain. In this case, real-world CPU and bandwidth growth. In the POS case, proof that some kind of real-world effort was performed.

What about the following modification:

a1) fold a modified p2ppool protocol into the mainline protocol
a2) require that transactions mined into the mainline blockchain have to be seen in the majority of p2ppool blocks
a3) p2ppool then has an additional function of proof-of-propagation: at least 50% of miners have seen the tx
a4) we can then individually adjust the fees and incentives individually for:
a4.1) permanent storage of transactions (in the mainline blockchain)
a4.2) propagation of transactions (in the p2ppool blockchain, which is ephemeral)

Right now the problem is that miners receive all fees due, both for permanent storage and for network propagation.

Another idea in the similar vein:

b1) make mining a moral equivalent of a second-price auction: the mining fees of block X accrue to the miner of block X+1
b2) possibly even replace 1 above with a higher, constant natural number N.
Late edit:
b3) reduce the coinbase maturity requirement by N
End of edit.

Both proposals above aim to incentivize and enforce propagation of the transactions on the network and discourage self-mining of non-public transactions and self-dealing on the mining fee market.

What about the following modification:

a1) fold a modified p2ppool protocol into the mainline protocol
a2) require that transactions mined into the mainline blockchain have to be seen in the majority of p2ppool blocks
a3) p2ppool then has an additional function of proof-of-propagation: at least 50% of miners have seen the tx
a4) we can then individually adjust the fees and incentives individually for:
a4.1) permanent storage of transactions (in the mainline blockchain)
a4.2) propagation of transactions (in the p2ppool blockchain, which is ephemeral)

Right now the problem is that miners receive all fees due, both for permanent storage and for network propagation.

Another idea in the similar vein:

b1) make mining a moral equivalent of a second-price auction: the mining fees of block X accrue to the miner of block X+1
b2) possibly even replace 1 above with a higher, constant natural number N.
Late edit:
b3) reduce the coinbase maturity requirement by N
Later edit:
b4) since nowadays the fees are very low compared to subsidy, (b3) would imply a temporary gap of global mining income. Subsidy of block X accrues to the miner of X, fees of block X accrue to the miner of block X+N.
End of edits.

Both proposals above aim to incentivize and enforce propagation of the transactions on the network and discourage self-mining of non-public transactions and self-dealing on the mining fee market.

I'd welcome comments / criticism of why having such a feedback mechanism is a good or bad idea.

.. more about this:
there's actually the opposite kind of manipulation (or rather attack) possible:
empty blocks. Right now they exist but don't hurt anyone; here they would push
the max block size down, hurting the network.

There may be a way of using the coinbase fee also in this calculation, but treated differently.  The coinbase fee primarily serves the emission and distribution functions, but also stimulates adoption in the early years.  It might be used as a way of amplifying the metric in the early years (when lack of adoption is a significant existential risk, and percentage growth is presumably higher) and then let this effect subside in later years by some form of multiplying by (Coinbase)^-1/2

Currently the cost per transaction, with the coinbase included, is often higher than the transacted amount.  Such transactions would not occur without the coinbase, so a way to accomodate for what this proposal would mean (because we would be unlikely to have any meaninful MAX_BLOCK_SIZE increases so long as coinbase transactions are the funding source for the network.

If I miner did somehow push the fees and blocksize up, that miner could then publish large blocks in an attempt to spam / DoS the network.  That's the only real threat, and it could be very costly and slow for the miner to accomplish.  Unlike the difficulty adjustment, which is bounded at 0.25X to 4X, the max block size adjustment could have a much tighter bound, like 10%, so that it would take months to double the max block size.

I think this is simple and straightforward enough that miners, developers, and bitcoin users can read it, understand it, and be OK with it.  I also think that it's safe long-term, and doesn't require human intervention once set up, regardless of how computer technology evolves (or fails to) over the next few decades.

By including the coinbase fee (or maybe a square or other root of it) we would come closer to Gavin's increase in the early years and move steadily toward a fee supported mining within the next 20 years or so while increasing the MAX_BLOCK_SIZE.

Did you read my "blocksize economics" blog post?

I don't understand why you think MAX_BLOCK_SIZE necessarily has anything to do with "supporting mining" (aka securing the network).

What stops this from happening:

Big miners accept off-blockchain payments from big merchants and exchanges that want their transactions confirmed. They are included in very small blocks with zero fees.  The blocksize stays at 1MB forever.

Did you read my "blocksize economics" blog post?

I don't understand why you think MAX_BLOCK_SIZE necessarily has anything to do with "supporting mining" (aka securing the network).

What stops this from happening:

Big miners accept off-blockchain payments from big merchants and exchanges that want their transactions confirmed. They are included in very small blocks with zero fees.  The blocksize stays at 1MB forever.

Lets look at incentives:

Big miners: have cozy agreements with Big Merchants. Those agreements keep the little guys out.

Big Merchants: same thing. The need to get an agreement with a miner to get your transactions accepted is a barrier to entry for the Little Guys.

Consider the existence of a central authority, employed by a member organization with the charter of interfacing with governments.  The Chiefs then take the role of arbitrarily deciding on the supply and adjusting as the organization's economic advisers suggest, we then have progressed towards replicating the Federal Reserve Bank.

KISS:

1. Since technology allows increase to 20 MB per block today, make an increase to this size as soon as consensus and logistics allow.

2. Continue to evaluate the situation ...

I completely disagree with this. Believe it or not it's actually not that easy for the Fed to adjust monetary policy. I mean all things considered, it's exceptionally easy, but they still have to get their board to go along and sell the public on what they're doing. That's a task made harder as they try more extraordinary things (like now) and the public becomes more astute to the way money works and its importance (like now), and that's a center driven design.

Bitcoin is designed from the ground up to be the opposite. It's extraordinarily hard to implement changes affecting the whole without consent from the whole. I sincerely believe after a certain point of adoption it will be impossible to make changes to Bitcoin, even ones not so controversial; if there isn't a do or die mandate behind the action (like a break in SHA256) I don't see consensus from millions and millions of independent thinkers coming easily. Somebody's going to think differently for some reason, even if it appears irrational. People call this ossifying of the protocol.

Think how hard this 1MB issue is. There was a time when Satoshi simply told everyone to start running a protocol change without question. He knew there was a severe bug allowing excess coins, but people simply upgraded and now the fork ignoring that block is locked in.

Bitcoin isn't the first to come up with decentralization. That was actually the idea behind America. Instead of power reigning down from monarchs it would be vested within all the individuals. However, the founders even then recognized authority by committee wasn't always ideal. It would be a clear disadvantage if attacked since the battle might be lost before it was decided what to do. That's why the president has full authority to respond militarily in case of attack.

It sounds like you're objecting for reasons more ideological than practical. While that's admirable and understandable I hope you also recognize that's not automatically best given the circumstances.

i agree with this idea
gavin is correct !

I understand you believe that Bitcoin is doomed to fail ...

... because of insufficient central authority

In any case, even if you were right and such a thing were needed, that should not stop people from offering better ideas to those who are claiming to have authority.

So you are about as wrong as anyone can possibly be, to suggest that just because someone claims authority, that they should make decisions and everyone blindly follow when they see clearly better solutions available.  
Why?
Just for the sake of establishing authorities?

I never said I believe Bitcoin is doomed to fail, although my questioning it's viability may strengthen with developments. That should be everyone's position, because Bitcoin is an experiment. Those who think Bitcoin is guaranteed to succeed in serving the world are misunderstanding the situation. This doesn't mean it can't succeed at that, only that it's not guaranteed (how could it be?).


My position isn't Bitcoin needs central authority. My position is Bitcoin needs a viable solution. If you read the post (https://bitcointalk.org/index.php?topic=815712.msg9247444#msg9247444) I made above you'll see I asked whether the majority community could be convinced to accept Gavin's proposal or one more like what you're crafting. My position was one of adopting a viable solution.


Who has claimed any authority? Where? All I see is people putting forth their suggestions.


Like I said above, it seems you're arguing from a position of ideology. You seem to see resolving the block size issue as divided between those who tend toward centralization and those who demand absolute decentralization, even to the point of seeing people establishing positions they haven't. That is the reason I question Bitcoin's viability. It's because people have their own thoughts about how things should work, or how things can work, and even if there is a solution which can work (I'm not saying which) it may not be possible to get everyone to agree, because it's not possible to do a Vulcan mind meld and have everyone understand everyone else's thoughts, conclusions, and informing information. People think differently (and with differing abilities). In the absence of some deciding force (usually a leader or authority as you call it) the result may be no clear decision whatsoever.

I'm simply seeking something which can work, something a majority can agree upon, nothing more.

Gavin's proposal is fit for immediate purposes, but it just kicks the can down the road for future readjustment.  This is problematic in that it will then re-require this leader/authority/deciding force.

Rather than accepting an extrapolation which is guaranteed to be wrong, ...

The answer cannot be in the blockchain, because the problem being addressed (resource usage rising too quickly so only people willing to spend tens of thousands of dollars can participate as fully validating nodes) is outside the blockchain.

This is one of those instances I'm talking about regarding people thinking differently.

You and I seem to fundamentally think differently here, and who is to say who is right? I believe whatever hard fork change we make, if we make one, it will be locked in quite probably forever more. It won't be subject to adjustment. Whatever it is future users will have to work with, sort of like we simply have to work with 1MB if we can't adequately change it. This is due to an ossifying of the protocol, again, as I mentioned above (https://bitcointalk.org/index.php?topic=815712.msg9257470#msg9257470).


Define "wrong".

IMHO, this is the most salient point on this whole thread.  Sometimes you just have to think deeply and clearly to see the truth.  

The core dev team are more than just experts in computers, they are also experts in human relations.  It's why Gavin and others have so much respect in the community, and certainly why they have my respect.  Bitcoin is all about free will and voluntary participation.  Every aspect of bitcoin must be in philosophical alignment with those concepts.  I'm not worried, I believe the consensus around the principles that bitcoin embodies will continue to grow.  This thread has been a great discussion.  Some of alternatives to Gavin's plan just don't work on the philosophical level.  Therefore, they cannot function on a technical level.

Necessitating future adjustment.  A change that does not resolve the fundamental problem, and addresses only the immediate perceptions of today.  

In the same way that a fixed 1MB is "wrong".  

Now define "right". Is it simply a block size which grows/shrinks dynamically with real world bandwidth over time? What if usage demand is far higher? What if the BTC exchange rate experiences unending volatility due to uncertainty about usage capacity (ie its user base)?

In other words does not needing future bandwidth adjustment automatically mean "right"?

Not according to these people (http://keepbitcoinfree.org/). They think 1MB is the right answer for "many more years" and until we know it's "safe" to change. Can't you see any answer given is subjective? With that being the case doesn't it make sense to adopt a solution which can fit the most common perception of Bitcoin's promise, which certainly includes global usage, and can win popular support?

Correct.  If you fix something and it never needs fixing again, and just keeps working from them on, you have fixed in a right way.

I do agree that any feedback mechanism such as we are seeking with this line of discussion holds the potential for creating a perverse incentive.

Automatic adjustment based on the environment of the future at least has a possibility of being right.  

We can play "what if" about all sorts of things, but at least the ones you mention here are not going to take additional hard forks to accommodate if we use any either of the methods described in this thread.  

At best they are saying "not now".  

No one is even filling up the current 1MB blocks with self-dealing transactions as it is.  Remind me again why it was lowered?

A maximum block size which is too small will naturally lead to more off-chain activity; folks/entities will not be denied the ability to transact.

A maximum block size which is too big thwarts participation by bandwidth-starved nodes.  So?

I propose we set MAX_BLOCK_SIZE to the maximum functional value possible today and walk away trusting the future to the caretakers then.  If any idiot/malicious bad actors try to take advantage of it and attack then Bitcoin was vulnerable to that already anyways. 

No one is even filling up the current 1MB blocks with self-dealing transactions as it is.  Remind me again why it was lowered?

Would it be reasonable to reject blocks with too few transactions in them if the pool of transactions waiting is above some threshold?

I have an idea. Why not ask everyone in Bitcoin what they think we should do, then just do all of them! Or, we can just debate each idea until it no longer matters since the year will be 2150.

Please tell me if you agree an ossifying of the protocol - the fact it will become increasingly hard, probably impossible to make changes as adoption grows - is what we'll likely see.

At the risk of putting words into his mouth (for which I apologize if I'm wrong), gavin sees it as a technical anti-DOS measure: to prevent miners from DOSing voting enthusiasts out of the network.

But that's a very costly attack, yet it doesn't accomplish anything.

At worst harder, but not impossible.

If the ISO can finally manage to crank out C++11, despite the contentious issues and compromises that were ultimately required (and C++14 just two months ago too!), pretty much anything is possible IMO.

LOL are you not following this thread? What easy way forward do you see emerging for the block size issue?

That's for a programming language not a protocol. Also see Andreas Antonopoulos's comment (http://www.reddit.com/r/Bitcoin/comments/2jw5pm/im_gavin_andresen_chief_scientist_at_the_bitcoin/clfnorn) on ossification considering hardware, which I also agree with.

Consensus does not imply unanimous appeal. Although threads such as these help to flesh out different ideas and opinions, they do very little towards determining where consensus lies. This thread could represent a vocal majority just as easily as it could a small but vocal minority (on either side).

I had a complete reply typed out for all your points but my browser closed before I sent it  :'(

Ah well, I'm not re-typing it. The gist is I'm aware of the above, but don't think that's the case. I tend to think those in this part of the forum on this thread have sentiments which are not isolated. If we can gain consensus here we have a good chance in the wider community; if not then who knows, but it would become ever harder with passing time.

Asside: did you preview your post at any point? It's in your draft history (https://bitcointalk.org/index.php?action=drafts) if so...

Consensus does not imply unanimous appeal. Although threads such as these help to flesh out different ideas and opinions, they do very little towards determining where consensus lies. This thread could represent a vocal majority just as easily as it could a small but vocal minority (on either side).

This is exactly where a more formal governance model (as I mentioned) could help. It too would surely be imperfect, but just about anything would be better than determining consensus based on who writes the most posts, thoughtful though they may be.

If, for example, I knew that there was little support for gavin's version, I for one would be much more willing to compromise. But I simply don't know....

I'm having trouble imagining a use case where embedded hardware with difficult-to-update software would connect to the P2P network, much less having anything to do with handling the blockchain, but my imagination isn't all that great. I also have trouble in general with any device whose purpose is highly security related that isn't software upgradeable. (Such things do exist today, and they're equally ill-advised.)

Fears over lack of consensus are rhetoric to encourage people to accept shoddy work out of a mistaken sense of urgency.

Please tell me if you agree an ossifying of the protocol - the fact it will become increasingly hard, probably impossible to make changes as adoption grows - is what we'll likely see.

This I know, but tend to think it's not the case. Anyone in this part of the forum, on this thread, probably doesn't have some obscure minority point of view. Whatever the basis for their reasoning it's probably not isolated. I think if we can get some sort of consensus on a thread like this we can too in the wider community. If we can't it would be harder, maybe not impossible, but harder depending how people dug into their positions. The longer the wait the harder. If this were mid 2010 there would likely be zero problem. High profile devs (like Satoshi/Gavin) would simply explain the necessary way forward and the small community would move along. If we're talking 2019 I don't see that happening so easily, or at all actually.

I'd be in favor of more structure for progress, but you won't convince everybody. There will be purists that cry centralization.

There is always a long tail of technology out into the marketplace. Just because our community is at the cutting edge of technology doesn't mean everyone is. For example, I was surprised to learn of a story in the community I came from (Hacker News) about a very successful business that still ran BASIC (http://en.wikipedia.org/wiki/BASIC). This was used for order fulfillment, accounting, you name it. The business was profitable in the millions if I recall, but completely reliant on their workhorse infrastructure. It wasn't cutting edge, but it worked, and that's all that mattered. A similar story exists in the banking industry.

More structure can cut both ways. If done well (big if), it can reduce centralization by better distributing "votes." But you're right that you can't convince everybody.

It's a free attack for a miner, and can arbitrarily kick anyone off the network (even if temporarily) who doesn't have sufficient bandwidth or ultimately enough disk space.

It's not free. The larger the block, the higher chance it has to be orphaned. No miner is going to inflate his blocks to reduce his chance to win a block race.

I don't think adjusting the block size up or down or keeping it the same will have any effect on whether or not transaction fees will be enough to secure the network as the block subsidy goes to zero (and, as I said, I'll ask professional economists what they think).

Yes... and the orphan risk rate also decreases with bandwidth availability.
I continue to maintain that market forces can rightsize MAX_BLOCK_SIZE if an algorithm with a feedback mechanism can be introduced, and that doing so introduces both less centralization risk than an arbitrary patch, and less risk of future manual arbitrary adjustments.
Fix it right, fix it once.

Yeah, something that looks at block sizes over a past period of time to determine the next max block size for a certain period would be ok for example.

I continue to maintain that market forces can rightsize MAX_BLOCK_SIZE if an algorithm with a feedback mechanism can be introduced, and that doing so introduces both less centralization risk than an arbitrary patch, and less risk of future manual arbitrary adjustments.
Fix it right, fix it once.

I think you are confusing MAX_BLOCKSIZE with the floating, whatever-the market-demands blocksize.

MAX_BLOCKSIZE is, in my mind, purely a safety valve-- a "just in case" upper limit to make sure it doesn't grow faster than affordable hardware and software can support.

Ideally, we never bump into it. If we go with my proposal (increase to 20MB now, then double ten times over the next twenty years) I think it is reasonably likely the market-determined size will never bump into MAX_BLOCKSIZE.

I think it is very unlikely that in 20 years we will need to support more Bitcoin transactions than all of the cash, credit card and international wire transactions that happen in the world today (and that is the scale of transactions that a pretty-good year-2035 home computer and network connection should be able to support).

Yes, I think Bitcoin can surpass this.  There are other problems with the way that scalability is limited than the block size to get there, this is just one.
And we agree on the purpose of the block size limit.  Just not on how to set it.

You don't know what the future market will look like.  You don't know what bandwidth or storage will be available.  Neither do I or anyone else.

When you respond to me saying patronizing things like "there are other problems with the way scalability is limited," I have trouble not thinking you are either confused or insane. Or just lazy, and did not read my "Scalability Roadmap" blog post.

It is certainly true that nobody can predict the future with 100% accuracy. We might get hit by an asteroid before I finish this sentence. (whew! didn't!)

But extrapolating current trends seems to me to be the best we can do-- we are just as likely to be too conservative as too aggressive in our assumptions.

Are you simply unaware of the other ways scalability is limited and only focused on this one?
We can go into it if you like.
I was looking to keep this discussion on the more narrow issue.

Start a new thread.  HAVE you read my Scalability Roadmap blog post?

It is certainly true that nobody can predict the future with 100% accuracy.

NewLiberty, I have a quick question for you which will hopefully clarify your position in my mind.

Excluding DOS flooding or other malicious actors, do you believe it would ever be a beneficial thing to have the blocksize limit hit?

I don't know why there's so much discussion about the max block size when the real issue should be how are you going to increase adoption (more people willing to do more txs and pay more fees to keep the network secure) so that Bitcoin is sustainable without a much lower or zero block subsidy in the future.

Primarily the limit is a safeguard, a backstop.  It is not meant to be a constraint on legitimate commerce.
It also serves to facilitate adoption and decentralization by keeping the ability to participate affordable.
Backstops are beneficial when hit, if they are protecting grandma from getting hit with the ball.

Thank  you.

Would you agree, if it were possible (although it is not), that the blocksize limit should somehow be automatically tied to "the bandwidth and disk space an average enthusiast can afford"?

Yes.  Though you should also recognize that block size and bandwidth use are not so tightly tied.
Network and disk compression can compress data, the block size is measured after decompression.

Fair enough- block size may not be the best parameter to tweak to maintain the stated "bandwidth and disk space" goal, but it is a technically simple parameter to tweak.

Do you believe that there exists somewhere in the blockchain a metric, let's call it X, which would serve as a good predictor of "the bandwidth and disk space an average enthusiast can afford"?

I think this is the same question, though you may disagree: Do you believe that this metric X has a causal relationship with "the bandwidth and disk space an average enthusiast can afford"?

The Socratic inquiry is a bit pedantic, don't you think?
Skip to your point please.

Very well.

No metric that can be gleaned from the blockchain has a causal relationship with "the bandwidth and disk space an average enthusiast can afford", and therefore any such predictor has a high danger of being either too restrictive or not restrictive enough.

Using Nielsen's Law also has a danger of being inaccurate, however given that it has at least been historically accurate, I find this danger much lower.

Do you disagree? (let's leave ossification out of this just for the moment, if you don't mind)

Commodity prices never drop to zero, no matter how abundant they are (assuming a reasonably free market-- government can, of course supply "free" goods, but the results are never pretty). The suppliers of the commodities have to make a profit, or they'll find something else to do.

Primarily the limit is a safeguard, a backstop.  It is not meant to be a constraint on legitimate commerce.
It also serves to facilitate adoption and decentralization by keeping the ability to participate affordable.
Backstops are beneficial when hit, if they are protecting grandma from getting hit with the ball.

Yes, I disagree.  
Both Block size and transaction fee may be better tools than Nielsen's law, the combination may be even more so.  Dismissing inquiry on the matter, is a missed opportunity.

NewLiberty, you seem to be ignoring me (https://bitcointalk.org/index.php?topic=815712.msg9295024#msg9295024).

Your sticking point, in my mind, is less about solving this issue than it is you feel people are not taking adequate time to find an input based solution to "fix it right".

As I said before my goal isn't to be right. It's to find a solution which can pass the community so we're not stuck. Ideally it also meets Bitcoin's promises of decentralization and global service. I made a bullet point list (https://bitcointalk.org/index.php?topic=815712.msg9281473#msg9281473) outlining my thinking on the two proposals, but please note I didn't refer to any specific plan from you. I said any input based solution, which implies any taking accurate measurements too - lack of consideration in uncovering such isn't relevant. I fundamentally think that approach wouldn't work as well for reasons I outlined.

Would you make a bullet point list of your likes and dislikes on the two proposed paths so we can at least see in a more granular way where our beliefs differ?

Finally good news, even Satoshi told the 1MB limit is temporary.

But easier would be square maxblocksize every block reward halving, to make Bitcoin simple...

50 BTC = 1 MB
25 BTC = 2 MB
12.5 BTC = 4 MB
6.25 BTC = 16 MB
3.125 BTC = 256 MB

and so on.

At this point it might be advisable to relax the presentation with some charts based on actual data ;).

Unfortunately I'm using quite old data (missing some blocks so the longest chain ends at block 210060), as you can see by this query result.


So here we go: some histograms, click images for slightly larger versions.

https://i.imgur.com/Vd9dgl.png (https://i.imgur.com/Vd9dg.png)

https://i.imgur.com/kfCUBl.png (https://i.imgur.com/kfCUB.png)

https://i.imgur.com/yqkELl.png (https://i.imgur.com/yqkEL.png)

Observations and clarifications/notes:

• I'm looking at overlapping sequences, so a block that takes 127 minutes to calculate would result in multiple sequences being counted
• The case of a 3-block sequence taking at least 127 minutes to find happened 759 out of 210,060 times (0.3613%)
• The case of a 4-block sequence taking at least 127 minutes to find happened 1,551 out of 210,060 times (0.7383%)
• 135,421 blocks (out of 210,060) have been solved in less than 10 minutes (64.47%)
  
• There can be negative block times because miners clocks can be unsynced.
• The block that took longest to calculate was block #2 (7719 minutes). It might've been block #1, but we don't know how long that took.
• I put a confusing date on the upper right corner, data is from 2012-11-29
• first and last "bins" include the rest of the data (for example last bin contains the number of blocks that took 127 minutes or more to find)
• Surprisingly to me the "bin" with the most blocks is the 1-2 minutes bin (1:00 to 1:59.999) (bar labeled "1" in the charts")

queries I used are here: http://pastebin.com/tPg1RQtG

Does this stuff look like it could be correct to you guys?

EDIT: some corrections

NewLiberty, we can continue back and forth trying to sway one another and who knows how that will turn out. How about the following compromise:

We implement Gavin's plan - go to 20MB blocks and 50% annual increases thereafter. That is the default. However, we add a voting component. We make it possible to restrain the increase by say 1/2 if enough blocks contain some flag in the block header. It could also be used to increase scheduled increase by 1/2 if the model is too conservative for computing growth. There was a header variable mentioned before I think in the block size debate, the first time around.

I think this is the best of both worlds. It provides a measure of predictability, and simplicity, while allowing the community to bend capacity more inline with the growth of the time if needed. What do you think?

My (ideal) goals, in particular, would be to (1) never kick out grandma, and (2) never prevent a minor from including a legitimate transaction. (edited to add: those are in priority order)

If we wanted to be brutally pedantic on ourselves we could kick around the definitions of who grandma might be, and what makes a transaction legitimate, but I agree with the sentiment entirely.

I dunno know; here I am watching for blocks at or near the limit of 1MB and along comes ... it just seems strange to me https://blockchain.info/block-height/326639 -- apparently the miner couldn't be bothered to include even just one other transaction except the coinbase transaction in the block?  Could the pool have been empty from his point of view?

This is a common optimisation virtually all crappy pools use shortly after a new block since their software can't scale to get miners to work on the new block full of transactions quickly, they just broadcast a blank sheet for the first lot of work after a block change. Most pools blame bitcoind for being so slow to accept a block and generate a new template, and this is actually quite slow, but it's obviously more than just this.

I'd rather not implement a grandma-cap on bitcoin's growth. Grandma doesn't need to run a full node. She can use an SPV or other thin client.

We share these design goals, and priority.  They aren't comprehensive for me though.

I'd add (at least)
3) provide conditions conducive for mining when the transaction fees are supporting the network  
4) avoid future changes on the same issue.
And of course avoid introducing new unmitigated risks (more of a criteria than a goal).

I'd rather not implement a grandma-cap on bitcoin's growth. Grandma doesn't need to run a full node. She can use an SPV or other thin client.

lol @ grandma-cap for Bitcoin
We agree, in this case "grandma" is substituting for "bitcoin enthusiast" WRT bandwidth availability, I think.
I'm guessing he was thinking that our enthusiast might be living with grandma, or maybe is grandma, IDK?

Actually I picked the term up from NewLiberty's post (https://bitcointalk.org/index.php?topic=815712.msg9303401#msg9303401), but yes that's what I was assuming it meant. Should the term "grandma-cap" make it into the BIP?

It seems to me that (1) and (2) could both be implemented with either the static (Gavin's) method or some reactive method, although the I suspect the reactive method can do (1) more safely/conservatively. If a reactive method can do (2) safely enough (I suspect it could), I'd prefer it. A reactive method seems much more likely to meet (4).

If I understand you correctly, (3) takes us back to an artificial cap on block size to prevent a perceived, as Gavin put it, "Transaction Fee Death Spiral." I've already made my rant on that subject (https://bitcointalk.org/index.php?topic=815712.msg9292869#msg9292869); no need to repeat it.

I'm of the opinion that reaching consensus on (3) is more important, and possibly more difficult, than any static-vs-reactive consensus. (3) is an economic question, whereas static-vs-reactive is closer to an implementation detail.

Much mining can be done with a single node.  Costs are discrete between nodes and mining[,] and [those costs are] asymmetric.
...
People look at hashrate to determine network health and not so much at node population and distribution, but both are essential.

If costs for node maintenance overwhelm the expected rewards at <x hashrate / y nodes then we lose all mining under that hashrate irrespective of other costs, and we lose nodes per hashrate.

In later years, there is a very long chain, and most coins transacted will be the most recent.
...
We don't know how it will play out, its uncharted territory.  #3 is more about not creating a perverse incentive to unbalance this that doesn't materialize until the distant future than about encouraging compensation through artificial constrain on supply.

3) provide conditions conducive for node maintenance and mining when the transaction fees are supporting the network by avoiding perverse incentives.

lol @ grandma-cap for Bitcoin
We agree, in this case "grandma" is substituting for "bitcoin enthusiast" WRT bandwidth availability, I think.
I'm guessing he was thinking that our enthusiast might be living with grandma, or maybe is grandma, IDK?

This is a common optimisation virtually all crappy pools use shortly after a new block since their software can't scale to get miners to work on the new block full of transactions quickly, they just broadcast a blank sheet for the first lot of work after a block change. Most pools blame bitcoind for being so slow to accept a block and generate a new template, and this is actually quite slow, but it's obviously more than just this (since I don't ever include transaction-free blocks in my own pool software).

Some people want everyone to run a full node. I'm suggesting that's not a good idea. We should not limit bitcoin growth such that everyone can run a full node. Not everyone needs to run a full node to benefit from bitcoin.

I don't recall Gavin ever proposed what you are suggesting here.  1st round was 50% per year, 2nd proposal was 20MB + 40% per year, yes?

I'm less a fan of voting than you might imagine.  
It is mostly useful when there are two bad choices rather than one good one, and a choice is forced.  I maintain hope for a good solution yet.  To give us an easy consensus.

The idea has a lot of negatives.  Possibly its fixable.
Thank you for bringing forward the suggestion.

Heck, jump to 20MB and grow it at 40% for 20 years; that's fine with me *but* be prepared to alter that if there be a need.

That's fine by me. My last proposal does this. What does everyone think? I say we start building some idea of what can pass community consensus. We may need to leave NewLiberty searching for the legendary ideal solution.

Since the monumental debate which occurred in February 2013 (which you will remember), I have despaired for 18 months that nothing will get done until the limit is actually hit, throughput is crippled for days, and world's press laughs at Bitcoin for years afterward.

Which is why I even argued for a 3MB cap, just to buy some time.

https://bitcointalk.org/index.php?topic=709970.msg8129058#msg8129058

That 40% for 20 years is more than fine by me :-)

1) What is the maximum value for MAX_BLOCKSIZE functionally possible given the APIs being used?

Breaking blocks into multiple messages would be a significant code change to go above 32 MB.

So who are we kidding with this?  Are we doing the block segment code now or later?  Bump it to 32MB now to buy us time to do the block segment code.

2. That said a slow ~40% growth rate gives us time to improve the clients to scale nicely. Again I give full support to this.

I do remember. I'm really hoping it doesn't take running into the limit to provide impetus to take action. Not only would we likely get negative press as you mention, but it would highlight the issue to people completely oblivious to it. If we can't get people with good knowledge of the subject to agree how would we fare after adding even more noise to the signal?


I think half-measures only increase the likelihood we can't get a comprehensive solution.


Thanks for your feedback and for being IMO reasonable :)

It will be nice to have this 40% solution in pocket as the minimum quality, temporary patch, while a fix may be devised that would not need future adjustment.

Um, is that it?  How do we know if we've reached consensus?  When will the version with the increased MAX_BLOCKSIZE be available?

The people at keepbitcoinfree.org don't want to change the 1MB now at all. They think, for Tor and other consideration, it's necessary, but I agree with Syke that not everyone needs to be able to run a full node.

1) What is the maximum value for MAX_BLOCKSIZE functionally possible given the APIs being used?

2) What is the maximum value which has been tested successfully?  Have any sizes been tested that fail?

3) Why not just set it to that value once right now (or soon) to the value which works and leave it at that?
       3.1) What advantage is there to delaying the jump to maximum tested value?

No miner is consistently filling up even the tiny 1MB blocks possible now.  We see no evidence of self-dealing transactions.  What are we afraid of?

Heck, jump to 20MB and grow it at 40% for 20 years; that's fine with me *but* be prepared to alter that if there be a need.  How will we know we need to jump it up faster?  A few blocks at the current maximum is hardly a reason to panic but when the pool of transactions waiting to be blocked starts to grow without any apparent limit then we've waited too long.

But clearly some blocks are already full right up to the 1MB limit.  I've been doing transactional systems for 30+ years; the serious trouble will start when the average over reasonable periods of time, e.g. an hour or so but not more than a day, begins to approach ~70%.

http://en.wikipedia.org/wiki/Little's_law

Per https://blockchain.info/charts/n-transactions?showDataPoints=true&timespan=1year&show_header=true&daysAverageString=1&scale=0&format=csv&address=, Nov. 28, 2013 had the most transactions in a day, i.e. 102010.  From https://blockchain.info/block-height/271850 to https://blockchain.info/block-height/272030, i.e. 180 blocks that day, one wonders what the block size distribution looked like.  Gosh, it would be useful to have the size of the pool of waiting transactions at that time.

Per https://blockchain.info/charts/n-transactions-per-block?showDataPoints=false&timespan=1year&show_header=true&daysAverageString=1&scale=0&format=csv&address=, we had an average of 560 transactions per block (only the 8th highest day so far).  Feb. 27, 2014 had the highest average transactions per block of 618 so far.

April 3, 2014 had the highest average block size at 0.365623MB.  Arg, a day is too long.  I just bet the hourly average peaks around 70% of 1MB.

Does *anyone* have a record of the pool or waiting transactions?  That's our key.  When there are ~2,000 transactions in the queue waiting then we would expect a full 1MB block to be coming out next.  When there are ~4,000 transactions in the queue waiting then we would expect two full 1MB blocks to be coming out next.  In this state, transactions can expect to take ~20 minutes to confirm.  ~6,000 waiting -> 30 minute confirmation times.  And so on.

7t/s * 60s/m = 420t/m, 420t/m * 10m/block = 4200t/block.  That does not match observations:  Observations reveal only about 2000t/block.  2000t/block * 1block/10m = 200t/m, 200t/m * 1m/60s ~= 3.3t/s.  Who thinks we can squeeze 4200t/block?  3.3t/s * 86400s/d = 285,120t/d.  Trouble is closer than we thought.  70% * 285.120t/d = 199,584t/d.

Gentlemen, I've seen this too many times before; when the workload grows to somewhere north of 200,000t/d we *will* begin to see the pool of waiting transactions grow to tens of thousands and confirmation times will be well over an hour.

Increase the MAX_BLOCKSIZE as soon as is reasonable.  20MB, 32MB, whatever.  Then enhance the code to segment blocks to exceed the API limit after that.

Gavin, if you're still reading in my humble opinion we need a few things to move forward.

You're still the closest thing this community has to Satoshi/leadership so I think impetus comes from you. IMO you should update your Scalability Roadmap (http://bitcoinfoundation.org/2014/10/a-scalability-roadmap/) to reflect 40% annual increases noting that seems to be able to garner consensus. Maybe a second update includes a step-by-step process to rolling out the update (fork), so people know what to expect. I think starting to talk about things in a matter-of-fact way will engender confidence and expectation of what's to come. Thanks again for your hard work.

For the rest of us I think it's helpful to be supportive, in any way we can, of pushing forward this update. Notice I've updated my signature. If we can explain to those wondering whether it's the best option that it certainly is by implementing something, while covering the most possible bases people may go along.

My personal opinion is 20MB /40% is a kick-ass combo.  Between that, the headers-first downloading, the O(1) miner backbone system, and (hopefully) sidechains, we are looking at the most important set of technical improvements since bitcoin started.

My sentiments exactly.  I have actually found this whole thread to be quite heartening.

NewLiberty - you have done a good job tirelessly advocating for a certain approach.  It almost seems as if every even-numbered post is yours, and every odd-numbered post is from a different poster who steps up to the plate to explain to you the flaws in your position.  You seem like a perfectionist, which in general isn't a bad thing at all.  But action is needed now, even if we don't have some perfect 'forever' solution. Fortunately we don't need a 100% consensus to move forward, just an undefined super-majority.  I'd guess we have it around the 20MB / 40% concept.

My personal opinion is 20MB /40% is a kick-ass combo.  Between that, the headers-first downloading, the O(1) miner backbone system, and (hopefully) sidechains, we are looking at the most important set of technical improvements since bitcoin started.

-snip-
Does *anyone* have a record of the pool or waiting transactions?  That's our key.  When there are ~2,000 transactions in the queue waiting then we would expect a full 1MB block to be coming out next.  When there are ~4,000 transactions in the queue waiting then we would expect two full 1MB blocks to be coming out next.  In this state, transactions can expect to take ~20 minutes to confirm.  ~6,000 waiting -> 30 minute confirmation times.  And so on.
-snip-

 bitcoind getrawmempool false | wc -l

 bitcoind getrawmempool true | grep fee | grep 0.00000000 | wc -l

What did you see and where did you see it.
200Kt/d may be several years away, yes?
https://blockchain.info/charts/n-transactions?timespan=all&showDataPoints=false&daysAverageString=1&show_header=true&scale=0&address=
(If you are fond of extrapolating this might tell you something)

Or 200kt/d may be much sooner.
So flexible limits are probably better than "whatever" right?

I don't have historical data. But I just setup a rrdtool database to track the number of transactions on my full node. The stats for the last 24 hours are shown here [1], the pic is updated every 30 minutes and I will add more for 30 and 360 days once the database has enough data. As you can see from the little data that's allready there (collecting ~1 hour now) we are already closer to 4000 transactions waiting than to 2000.
The raw data is gathered every minute with the following command

and is not filtered in any way that is not inherent to bitcoind.


shows that right now 2792 of 3685 TX are without fee. I might make another database to improve the stats.

[1] http://213.165.91.169/pic/mempool24h.png

Nice work shorena!  I sent you a small tip to the address in your profile.  I checked out all the other stats on you full node, overall it is an awesome testament to Bitcoin's openness.

-snp-
Inspirational!

Starting from your spark I found https://blockchain.info/tx/e30a4add629882d360bc87ecc529733a9824d557690d1e5769453954ea4a1056.  It appears to be the oldest transaction waiting at this moment.  It was 31:34 old at the time of block #327136.  Block #326954 was the first block that could have added it to the block chain; 182 blocks ago.  One wonders how old the oldest transaction is that includes a fee.

Maybe someone that operates a pool can shed some light on this.

[2] https://www.blocktrail.com/BTC/block-all/1 - you can sort by pool by clicking on its name, but I have not found one that has exclusivly big blocks.

A significant number of pools simply run the default bitcoind client and transaction rules. Some have their own rules claiming their rules are "anti-spam" (see the many threads about luke-jr's custom patches to the bitcoind client in the gentoo packaging, and probably any pool running his pool software does so too). Most of the alleged anti-spam selection criteria are (presumably an ideological objection) aimed at gambling sites.

From the wiki:


To spam the 1 MB blocksize takes roughly .2 BTC per block, or 1.2 BTC per hour. That's only $500 per hour.

To spam a 1 GB blocksize takes roughly 200 BTC per block, or 1200 BTC per hour. That's $500,000 per hour!

A 1 GB blocksize is far more costly to attack. We could increase the blocksize to 1 GB now and nothing would happen because there aren't that many transactions to fill such blocks.

What is the reason to stop at 20MB?  That just seems to be pushing things further into the future to make the decision.

It does mean that the 32MB message size limit wouldn't be a problem though.

Under Gavin's 2nd proposal, (starting at 20MB and +40% per year) >32MB is 2 years out.