Blocks are [not] full. What's the plan?

[PenAndPaper]

concern of a "downward spiral of transaction fees"?

If blocksize isn't scarce at all why not include every transaction that pays at least 1e-8 BTC which you've already received and validated?  If it's purely orphaning that influences your decision why won't all your spending go to bandwidth (neutrino links to other miners and such ) and none to POW? etc.

Ok i got the first part of your answer but for the second part i'm not so sure. You mean that bigger blocks will likely cause more orphans?
Anyway thanks for clearing it out.

[Kouye]

Ok i got the first part of your answer but for the second part i'm not so sure. You mean that bigger blocks will likely cause more orphans?
Anyway thanks for clearing it out.

I think the point is:

- The more transactions a block contains, the bigger it is.
- Bigger blocks take longer to propagate on the network and get accepted.
- If 2 smaller blocks get confirmed before the bigger one gets a child, all TX included in the bigger block don't get confirmed, and this is the "orphan cost".

There is no motivation today for miners to include more TX, as it makes it very risky.
They play small and secure.

The question now, is : "how do we make sure bigger blocks get bigger rewards overall".

Or maybe I didn't get it right, which is also a likely hypothesis.

[niothor]

Ok i got the first part of your answer but for the second part i'm not so sure. You mean that bigger blocks will likely cause more orphans?
Anyway thanks for clearing it out.

I think the point is:

- The more transactions a block contains, the bigger it is.
- Bigger blocks take longer to propagate on the network and get accepted.
- If 2 smaller blocks get confirmed before the bigger one gets a child, all TX included in the bigger block don't get confirmed, and this is the "orphan cost".

There is no motivation today for miners to include more TX, as it makes it very risky.
They play small and secure.

The question now, is : "how do we make sure bigger blocks get bigger rewards overall".

Or maybe I didn't get it right, which is also a likely hypothesis.

So , let me make sure I understand this...
Because the rewards from the tx can compare to the block reward.. there is no incentive for miners to create larger blocks , so even if we raise the limits to 10 Mb , there won't be much difference , unless a caring miner and a lucky one gets some blocks done.

[Cryddit]

That's roughly it.  Miners want to produce small blocks to minimize the time they take to propagate over the network.

Otherwise somebody who came up with a smaller block, later than theirs, can have it propagate and reach a majority of miners before their block does, which results in orphaning their block and sacrificing their block reward.

The "orphan cost" will cut itself in half when the mining rewards get cut in half -- but we can't wait that long because Bitcoin is currently in a mode of intermittent service failure and that undercuts the value of all coins.

[ilpirata79]

Centralized systems requiring trust are

You are presenting a false choice. The choice is not "centralized systems" OR "everything in Bitcoin". There are many options to build non-centralized ways of settling payment on a decentralized Bitcoin— ilpirata79 linked to one such approach. A lack of care into the operating costs of Bitcoin nodes may result in Bitcoin itself becoming effectively centralized, which is an outcome which would make building a non-centeralized payment method on top of Bitcoin pointless or impossible.

The fear of bitcoin becoming centralized because of "heavy" nodes is imho completely unnecessary. The traffic and storage technology can be maintained by small groups, it is not like there will be "one big government node" in every country. It is more like having a node in every town.
Any system that requires a system on top of bitcoin for payments is a threat for bitcoin and it is not necessary. The basic, existing system of bitcoin allows to replace visa if it is adapted to a bigger volume.
Anything else will make Bitcoin complicated and I know I am repeating this, but one huge factor of bitcoin is, that I don't need an intermediary.
Keep Bitcoin simple, complicating systems unnecessarily has never been wise.

A node in every town is risky because it is easy to take down. Who is going to administer such nodes?

The idea is that the max block size is going to increase, but slowly. Maybe a x10 multiplication of the soft and hard block sizes could be a good start (maybe even risky).

At the same time, to be on the safe side, it is good to develop safe and cheap distributed tecniques to handle micro transactions in order to avoid the unnecessary inflation of the block chain. Like the one I posted and that invite you to read.

It is not an unnecessary complication. Other crypto currencies may arise that handle things better because are better designed and  more sofisticated/complicated.

Best regards,
ilpirata79

[d'aniel]

A node in every town is risky because it is easy to take down. Who is going to administer such nodes?

The idea is that the max block size is going to increase, but slowly. Maybe a x10 multiplication of the soft and hard block sizes could be a good start (maybe even risky).

At the same time, to be on the safe side, it is good to develop safe and cheap distributed tecniques to handle micro transactions in order to avoid the unnecessary inflation of the block chain. Like the one I posted and that invite you to read.

It is not an unnecessary complication. Other crypto currencies may arise that handle things better because are better designed and  more sofisticated/complicated.

Best regards,
ilpirata79

Sharding the operation of a full node amongst mutually trusting participants seems like an easy enough way to scale.  It's not the ideal, and there appear to be more clever ways, but its certainly better than any centralized outcome, if that actually turns out to be the only alternative.  So I don't really believe centralization of fully verifying nodes is a realistic long-term fear, except in the case of ridiculously high loads and very unreasonable expectations of the blockchain.

Otherwise I agree, and expect this is pretty much how things will play out: keep kicking the block size limit can down the road until the problem solves itself (it just doesn't need to be raised any more), and build micropayment channel networks for microtransactions, where the point that micro becomes macro is defined by reality, and not what we wish it to be (not that we can't push the envelope with clever scaling solutions).

[solex]

concern of a "downward spiral of transaction fees"?

If blocksize isn't scarce at all why not include every transaction that pays at least 1e-8 BTC which you've already received and validated?  If it's purely orphaning that influences your decision why won't all your spending go to bandwidth (neutrino links to other miners and such ) and none to POW? etc.

You gotta limit blocksize at some level. Otherwise it's probably just too easy to release ridiculously large blocks which fork the blockchain.

That's what I used to think too. However, the system is self-stabilizing, ridiculously large blocks which fork the blockchain will find themselves orphaned. If unlimited size is too worrying then the simplest safety net would be max block size = 10x median block size during previous 2016 blocks or previous difficulty period.

[Carlor]

concern of a "downward spiral of transaction fees"?

If blocksize isn't scarce at all why not include every transaction that pays at least 1e-8 BTC which you've already received and validated?  If it's purely orphaning that influences your decision why won't all your spending go to bandwidth (neutrino links to other miners and such ) and none to POW? etc.

You gotta limit blocksize at some level. Otherwise it's probably just too easy to release ridiculously large blocks which fork the blockchain.

That's what I used to think too. However, the system is self-stabilizing, ridiculously large blocks which fork the blockchain will find themselves orphaned. If unlimited size is too worrying then the simplest safety net would be max block size = 10x median block size during previous 2016 blocks or previous difficulty period.

Exactly!

[DeathAndTaxes]

But the cost to transfer even 10000 megs in a few seconds is low compared to the cost to compute the block.

For a major pool server maybe but the cost is pretty much static so it makes super massive pools and operations far easier to ammortize that cost.

The issue with large blocks is the verification. But most of that can take place on CPUs during the 10 minutes or so that the ASICs are working on the hash.

Not really.  A CPU can perform roughly 15,000 ECDSA verifications per second and that is operating single core (no multithreading).  At PayPal transaciton volume = 50 tps, a block would contain 30,000 tps.  Even today that would take a single core no more than 2 seconds to verify.  With a multi CPU server with 8 to 16 cores total you could perform that in a fraction of a second.

As far as transaction fees only being 1e-8 BTC... Hopefully that's where we're going.

Fees buy security.  Right now the subsidy distorts that relationship.  Even at high transaction volume 1 satoshi in fees isn't going to generate significant revenue for miners so the hashrate would collapse and the network would be far easier to attack.   

[johnyj]

Bitcoins value just rises, because people believe it might become an alternative currency.

The value rises because it is a superior store of value and has international transaction ability. Chinese investors highly value its international transaction ability since they can't freely exchange USD with CNY.

Although many shops accept bitcoin payment, as long as you can exchange it for fiat money, you will always get a fiat loan to spend and wait for the bitcoin price to appreciate many folds and repay the loan with a tiny fraction of your original coins, this is a no-brainer (see my signature why bitcoin will appreciate forever)

And, any ponzi scheme or bubble theory does not apply to money itself, a perfect example is fiat money, although it has no intrinsic value, everyone are still joining this ponzi scheme day by day

[Carlor]

The value rises because it is a superior store of value and has international transaction ability. Chinese investors highly value its international transaction ability since they can't freely exchange USD with CNY.

It rises because it is a store of value (it is not superior, if I looked out for a safe store of value I would definitely not choose bitcoin) and because it's transaction abilities, right.  And these transaction abilities are for example, cheap and fast transactions with a simple system without intermediaries.
Everything else could be also done with computer game money (e.g.), which are not a working currency.

That's exactly my point, as long as the current system gets adapted to more and cheap transactions, everybody can use bitcoin for what they want it to use. You can use it as a good value storage, you can use it to buy your breakfast and you can use it to send money to your parents in Australia, etc. etc.
But if you start to castrate bitcoin more and more, it will lose all these abilities, in long term also the value storage function.

[Altoidnerd]

The value rises because it is a superior store of value and has international transaction ability. Chinese investors highly value its international transaction ability since they can't freely exchange USD with CNY.

[...] (it is not superior, if I looked out for a safe store of value I would definitely not choose bitcoin) [...]

What would you choose instead?

There is no answer right now.

[zvs]

But the cost to transfer even 10000 megs in a few seconds is low compared to the cost to compute the block.

For a major pool server maybe but the cost is pretty much static so it makes super massive pools and operations far easier to ammortize that cost.

You can transfer 10 gigs out of Amazon for $1.20. Incoming transfers are free. You pay according to how much you transfer out, not as a static payment.

I must have missed something here?

First off, 10 gigs is nothing.  Most major pools are probably on unmetered bandwidth or 100TB limits.

I was sending out about 1.5TB upstream every week, just from running a bitcoind node.. that'd be (whoops) *$180 from Amazon, I guess (in a week)

[Saturn7]

Difficulty adjustment already provides a mechanism to adjust a variable value with consensus. Why not just treat block size the same?
For example if the average size of the last 2016 blocks were 80% full then the block size would double.

[Saturn7]

Difficulty adjustment already provides a mechanism to adjust a variable value with consensus. Why not just treat block size the same?
For example if the average size of the last 2016 blocks in 80% full then the block size would double.

In the last 2016 blocks, or in the 2016 blocks which make up the previous difficulty calculation? (I think the latter would probably be a better choice.)

What, if anything, is the mechanism to shrink the blocks back down again? (Halve if the average size of the last 2016 blocks is 20% full, with a hard minimum of 1 meg?)

I suspect this might be vulnerable to blockchain-forking attacks which near-simultaneously release very differently sized blocks, but it's hard to say without a full specification.

Depending on your answer to the second question, it also might increase the incentives for miners to release blocks with as few transactions as possible.

It also generally makes the design of mining software more complicated and thus more vulnerable to attack. Being able to statically allocate the size of a block is a definite advantage, though I don't know off hand how the reference implementation handles this. I'd say some hard maximum is necessary, even if it's ridiculously huge. But then what's the advantage of not just setting the maximum at whatever that hard maximum is?

In the end this might be viable, but I'd want a lot more details.

I would say the 2016 blocks which make up the previous difficulty calculation.

I don't think it should shrink, there may be periods where blocks are not fully utilised but if that became an ongoing trend it would only mean people stopped using bitcoin.

I would say there are less risks in slowly growing the block size over time then just not having a limit at all (even if there was a large hypothetical limit). We also need to consider network propagation time. If out of the blue we had a 1 gigabyte block would all the clients globally have this data in ~10 minutes (about 6 minutes when the network hash rate grows)?

[Cryddit]

Is there any way to compensate miners for creating full blocks? 

I mean, if the issue is that smaller blocks are more profitable due to less broadcast latency, shouldn't larger blocks get a premium to compensate for the loss of profit? 

Right now it looks like the 25BTC per block is built in - but if a block bigger than 750Kbytes paid 26BTC and a block less than 500Kbytes paid 24BTC (or whatever ratios turned out to be needed) .... 

Awww, crap, if we did that we'd get Sybil attacks where miners had to spam the network with  a bunch of "padding" transactions to make every block bigger than 500/750 Kbytes.  You'd have to keep the premium perfectly balanced with the size cost (including halving it when block reward went down) to keep it from being worth anybody's time to game it.  Is there a balancing mechanism like there is for difficulty?  Based on the previous 2016 blocks, is there a statistical measure we can do to determine the size cost?  And if there is, can it be done without providing a motive and method to game that?

What's the expected statistical distribution of block size given the assumption that all possible tx are included?  Can we base a premium on deviation from that statistical distribution? 

[Cryddit]

In the end I don't see any obvious way to handle this other than increasing TX fees enough to compensate for "orphan costs;"  where does that leave tx fees? 

[Cryddit]

Oh.  From Gavin's post above, it leaves tx fees at 3.3 MilliBitcoins per Kilobyte.  For as long as the mining award is at 25BTC anyway.  Would people find that acceptable?

[DeathAndTaxes]

Oh.  From Gavin's post above, it leaves tx fees at 3.3 MilliBitcoins per Kilobyte.  For as long as the mining award is at 25BTC anyway.  Would people find that acceptable?

In the short term probably not.  The reality is the lowest cost, higher short term revenue would simply be to solo mine empty block and leave all txs even paying ones.   However miners (or pool operators) have shown some longer term thinking and HAVE built larger blocks.   However I think it does illustrate that if the min fee of 0.1 mBTC doesn't cover the orphan cost it is downright silly to expect miners to build massive blocks full of free txs and simply kill their own revenue so other people can avoid paying ~$0.10.

The longer term factors are in Bitcoins favor.  Overtime bandwidth gets cheaper it roughly follows Moore's law.  At the last mile it lags behind but for pool servers in a datacenter it is much cheaper.  You also have the block subsidy declining in half in 3 years which reduces the distortion from the "first 25 BTC are free" effect.   Combined that means even if NOTHING else changes the orphan costs should fall by a factor of 8x in the next four years. 

I already pointed out one potential solution to reducing the block broadcast size/time/latency and thus orphan cost by 90% by creating a message type which contains block header + list of tx hashes instead of block  header + list of full txs.   Combine with block subsidy decline and bandwidth improvements over time getting oprhan costs down to 0.05 mBTC or less should be possible.

[DeathAndTaxes]

Difficulty adjustment already provides a mechanism to adjust a variable value with consensus. Why not just treat block size the same?
For example if the average size of the last 2016 blocks in 80% full then the block size would double.

In the last 2016 blocks, or in the 2016 blocks which make up the previous difficulty calculation? (I think the latter would probably be a better choice.)

What, if anything, is the mechanism to shrink the blocks back down again? (Halve if the average size of the last 2016 blocks is 20% full, with a hard minimum of 1 meg?)

I suspect this might be vulnerable to blockchain-forking attacks which near-simultaneously release very differently sized blocks, but it's hard to say without a full specification.

Depending on your answer to the second question, it also might increase the incentives for miners to release blocks with as few transactions as possible.

It also generally makes the design of mining software more complicated and thus more vulnerable to attack. Being able to statically allocate the size of a block is a definite advantage, though I don't know off hand how the reference implementation handles this. I'd say some hard maximum is necessary, even if it's ridiculously huge. But then what's the advantage of not just setting the maximum at whatever that hard maximum is?

In the end this might be viable, but I'd want a lot more details.

I would say the 2016 blocks which make up the previous difficulty calculation.

I don't think it should shrink, there may be periods where blocks are not fully utilised but if that became an ongoing trend it would only mean people stopped using bitcoin.

That definitely solves a lot of problems.

Except that ISN'T the problem.   If miners can make a positive return by producing larger blocks ... they will.  They want money, more money is always better.   However tonight you could flip the network to 1 GB blocks and ACTUAL block sizes aren't going to change at all.

Miners actually are including most paying txs.  Take a look at the memory pool, remove tx seen after the last block and 90%+ of the remaining tx are:
* no fee txs.
* txs with unconfirmed outputs.
* double spends or other tx problems.

Implementing child pays parent will improve the second category, the third category probably should just be excluded.  That leaves essentially free txs.   Miners aren't going to produce 20 GB blocks of free txs just because users want something for nothing.  That is never going to happen so as the correct titles says "Blocks are NOT full.  What is the plan?".


The good news is there is something which can be done to make blocks larger and reduce confirmation delays:
1) The default bitcoind has a rule where fees double as blocks get larger.  It looks like most major pools have stripped that out otherwise we wouldn't see blocks >500 KB however that rule should probably go.  It no longer really serves any purpose.  The nice thing is it is a client side change so it requires no protocol change or fork.

2) Education.  Until recently a company as big and old as MtGox was creating free txs.  For something as timely as exchange cashouts that is just stupid.  Sorry it is.  They should know better.  If you want to send your friend Bob some coins and want to be cheap that is one thing but major commercial enterprises should really be favoring reliability over trying to save pennies.

3) Child pays parent.  Currently the way bitcoind prioritizes txs it does not include paying tx with unconfirmed outputs in the next block.  So someone sends you coins with no fee, or possibly a bunch of dust spam and you try to resend them and include a fee and it looks like miners are screwing you over.   The tx selection algorithm needs to be improved so if tx B has as an input an output for tx A and tx A has no fee and is unconfirmed but tx B has a fee the miners will include BOTH A & B in the block.  This would also allow users to fix stuck txs and allow merchants to get confirmations on payments faster by respending them.

3) Improve block message format.  Currently all block messages are the same old blocks back to the genesis block and the newest found block are transmitted as header + list of txs.  For new blocks, nodes that are up to date likely already have most (and probably all) txs so a simple change to create a block header + tx hash message would reduce the bandwidth requires by ~90%.  A 1Mb "header + hash only" block message would be smaller than a 100 KB full tx block message is now.