Blocks are [not] full. What's the plan?

[yaffare]

blocks are full NOW

there are already other threads about this

https://bitcointalk.org/index.php?topic=339257.0
https://bitcointalk.org/index.php?topic=338999.0
https://bitcointalk.org/index.php?topic=338793.0
https://bitcointalk.org/index.php?topic=338452.0

most pools create 250kb blocks only

please do something!!!

> release bitcoin 0.8.6, remove stupid -blockmaxsize
> transactions > 0.01 BTC are processed before transactions < 0.01 BTC
   tell people to not send transactions < 0.01 BTC, bitcoin is not ready for micro payments yet
> transactions not confirmed after 12 hours are removed from mempools automatically
> after 12 hours there must be a cancel/overwrite mechanism

at least tell people/merchants what to do if they have never confirming transactions

massive unconfirmed transactions are coming!!!

I know you devs are busy selling coins, but you owe the community solving this problem at least, before buying your ferrari.

[DeathAndTaxes]

Blocks are not full. Pools can already create 1MB blocks they simply don't.  Devs can't "solve" that.  Complain to your favorite pool to increase the block size they target.

[Carlor]

Devs can solve that.
People didn't want it to change...
And this is what might destroy bitcoin.

[cr1776]

Just two points, I don't think that < 0.01 bitcoins is a micropayment. At $600-$700 per bitcoin, that is $6-7. Perhaps < 0.0001 might be a micropayment though (6-7 cents), but even that perhaps might be an order of magnitude too high. :-)

Regarding block being filed, D&T makes a good point. 

[BurtW]

The title of this thread is wrong and misleading.  Please change it.  I thought it was something interesting and new and it is not.  Thanks.

[Dabs]

Include a higher than required transaction fee, and your transaction will go through.

[oakpacific]

I think maybe pools should advertise a "guaranteed to process given available space" fee level? Lots of people may prefer to pay quite a bit more than being kept in the unknown, which is infuriating.

[DeathAndTaxes]

I think maybe pools should advertise a "guaranteed to process given available space" fee level? Lots of people may prefer to pay quite a bit more than being kept in the unknown, which is infuriating.

The min fee (0.1 mBTC per kB) should be fine.  Comparing the memory pool to recent blocks almost all paying tx are included in the next block.   The issue is more people creating tx w/ unconfirmed inputs and people creating tx w/ no fee (and any fee < 0.1 mBTC is considered no fee).  Still pools DO need to start making blocks large.  150KB (0.3 tps) isn't going to cut it.  Bitcoin growth is essentially halted until pools start expanding block size.

[oakpacific]

I think maybe pools should advertise a "guaranteed to process given available space" fee level? Lots of people may prefer to pay quite a bit more than being kept in the unknown, which is infuriating.

The min fee (0.1 mBTC per kB) should be fine.  Comparing the memory pool to recent blocks almost all paying tx are included in the next block.   The issue is more people creating tx w/ unconfirmed inputs and people creating tx w/ no fee (and any fee < 0.1 mBTC is considered no fee).  Still pools DO need to start making blocks large.  150KB (0.3 tps) isn't going to cut it.  Bitcoin growth is essentially halted until pools start expanding block size.

I figure that creating fee tiers may help improving the user experience for many, and help motivating the miners to create large blocks as well(the about 7 U.S cents min fee is embarrassingly low, in the sense that you don't even dare to call it a service you paid for and I can't figure out what people are up to when they choose to pay lower), but such policies are guaranteed to create an uproar in a community where many takes everything free for granted.

[Dabs]

Are solo miners affected? I mean, those who use bitcoind and their own ASIC devices, at the usual or standard set up.

If, for example, I had maybe 100 TH/s of mining equipment, and I decide to go do solo or p2pool ...

[DeathAndTaxes]

Are solo miners affected? I mean, those who use bitcoind and their own ASIC devices, at the usual or standard set up.

If, for example, I had maybe 100 TH/s of mining equipment, and I decide to go do solo or p2pool ...

What do you mean affected? 

You can create blocks of any size up to 1MB.  Of course pools can do as well.  They have simply CHOSEN to make much much smaller blocks.   It looks like a few regular miners are keeping blocks <100KB.  Most seem to cap at 250KB.  The average for prior 30 days is ~150KB or a staggering 0.3 transactions per second.

[Foxpup]

The min fee (0.1 mBTC per kB) should be fine.  Comparing the memory pool to recent blocks almost all paying tx are included in the next block.   The issue is more people creating tx w/ unconfirmed inputs and people creating tx w/ no fee (and any fee < 0.1 mBTC is considered no fee).  Still pools DO need to start making blocks large.  150KB (0.3 tps) isn't going to cut it.  Bitcoin growth is essentially halted until pools start expanding block size.

You misunderstand how fees work with regard to large blocks. The minimum fee is not fine. Under the Satoshi client's fee rules, only the first 250kB of a block is available for transactions paying the minimum fee (which is why we're seeing so many 250kB blocks). The Satohsi client will create larger blocks if and only if there are transactions paying more than double the minimum fee. This is due to the fact that large blocks take longer to propagate, resulting in an increased risk of the block being orphaned. Higher fees must be paid to compensate for this increased risk.

[DeathAndTaxes]

Right NOW, the reality is 0.1 mBTC IS fine.  There isn't a massive backlog of paying tx. The backlog is on free tx.  The average block size isn't >250KB it is ~150KB.  That isn't to say it won't be an issue in the future but right now the min fee is fine.  

Still the larger fee for larger blocks is a huge landmine and beyond stupid.  It makes absolutely no sense.  Users have no idea what size blocks miners are targeting.  If a user pays double but a miner is targeting 100KB block the higher fee is just useless.  The larger fees for larger blocks nonsense should just be scrapped.  It might have made sense in the early history when average block was 20KB and there was a risk of a spam attack creating 1MB bloat blocks but those days are gone now.

[binaryFate]

The min fee (0.1 mBTC per kB) should be fine.  Comparing the memory pool to recent blocks almost all paying tx are included in the next block.   The issue is more people creating tx w/ unconfirmed inputs and people creating tx w/ no fee (and any fee < 0.1 mBTC is considered no fee).  Still pools DO need to start making blocks large.  150KB (0.3 tps) isn't going to cut it.  Bitcoin growth is essentially halted until pools start expanding block size.

You misunderstand how fees work with regard to large blocks. The minimum fee is not fine. Under the Satoshi client's fee rules, only the first 250kB of a block is available for transactions paying the minimum fee (which is why we're seeing so many 250kB blocks). The Satohsi client will create larger blocks if and only if there are transactions paying more than double the minimum fee. This is due to the fact that large blocks take longer to propagate, resulting in an increased risk of the block being orphaned. Higher fees must be paid to compensate for this increased risk.

That seems ingenuous to me, to believe miners would use the satoshi client "as it is".

When you say "Higher fees must be paid to compensate for this increased risk" (of taking a longer time to propagate), the idea is true, but your approach is not quite logic. It's not that there is a threshold above which you take a risk. Actually it starts from the first KB, everything you add in terms of size increase your chance to get an orphan block because your freshly mined block will take more time to propagate, and this is a continous, linear trade-off. Not something that appears after 250KB.

You can actually measure the cost (in terms of network propagation) of including more transactions in a block, and thus define a transaction fee per KB that matches perfectly the real cost in terms of propagation. There was an interesting discussion going on recently on the bitcoin-dev mailing list, in short Michael Gronager shown that a fee of about 0.0004 per KB is "fair" in this regard.

[Dabs]

Are solo miners affected? I mean, those who use bitcoind and their own ASIC devices, at the usual or standard set up.

If, for example, I had maybe 100 TH/s of mining equipment, and I decide to go do solo or p2pool ...

What do you mean affected? 

You can create blocks of any size up to 1MB.  Of course pools can do as well.  They have simply CHOSEN to make much much smaller blocks.   It looks like a few regular miners are keeping blocks <100KB.  Most seem to cap at 250KB.  The average for prior 30 days is ~150KB or a staggering 0.3 transactions per second.

Oh okay. So it's only the major pools are choosing to make small blocks. If I were a solo miner, I could set up and be customized to a larger block size (one that won't fork the chain.)

[DeathAndTaxes]

Are solo miners affected? I mean, those who use bitcoind and their own ASIC devices, at the usual or standard set up.

If, for example, I had maybe 100 TH/s of mining equipment, and I decide to go do solo or p2pool ...

What do you mean affected?  

You can create blocks of any size up to 1MB.  Of course pools can do as well.  They have simply CHOSEN to make much much smaller blocks.   It looks like a few regular miners are keeping blocks <100KB.  Most seem to cap at 250KB.  The average for prior 30 days is ~150KB or a staggering 0.3 transactions per second.

Oh okay. So it's only the major pools are choosing to make small blocks. If I were a solo miner, I could set up and be customized to a larger block size (one that won't fork the chain.)

Larger blocks up to 1MB yes.  Creating blocks greater than 1MB no.

[lindatess]

... Looks like a new client isn't going to fix this.

I guess it's time to rethink whether bitcoin will ever go mainstream.

Paying extra transaction fees doesn't make anything go faster. It merely transfers your position on the waiting list, driving up transaction costs for everyone.

Remember, it's the people, it's the miners that are doing this...

[fcmatt]

Just how would a pool go about putting more transactions into the blocks they find? I assume they have to modify the source code, recompile, and then deploy? Are the changes trivial for the average pool operator?

And on top of this there is a greater chance of an orphan block resulting in the loss of 25 BTC?

It sounds like the pools will not change until the reward is much less then it is today. It simply does not justify the risk to add more transactions and lose out on the reward. Is this a correct assumption(s)?

[Foxpup]

That seems ingenuous to me, to believe miners would use the satoshi client "as it is".

The fact is that many miners are producing 250kB blocks, and this is the behaviour of the Satoshi client. If many miners are using clients that function substantially differently, we would not be seeing so many 250kB blocks. Whatever modifications they have made, they still seem to be following the same fee rules.

When you say "Higher fees must be paid to compensate for this increased risk" (of taking a longer time to propagate), the idea is true, but your approach is not quite logic. It's not that there is a threshold above which you take a risk. Actually it starts from the first KB, everything you add in terms of size increase your chance to get an orphan block because your freshly mined block will take more time to propagate, and this is a continous, linear trade-off. Not something that appears after 250KB.

True, there's nothing special about the 250kB limit other than that's the threshold imposed by the Satoshi client. I never said it was a good idea, and in fact I think it's particularly absurd that the minimum fee suddenly jumps to more than double once the threshold is hit and then increases gradually from there, but that's for the miners to decide. If they don't like it, they're free to modify the fee rules in any way they choose.

Just how would a pool go about putting more transactions into the blocks they find? I assume they have to modify the source code, recompile, and then deploy?

Not necessarily. A number of parameters regarding the fee rules can be set from bitcoin.conf. In any case, modifying the code and recompiling is not at all difficult.

And on top of this there is a greater chance of an orphan block resulting in the loss of 25 BTC?

It sounds like the pools will not change until the reward is much less then it is today. It simply does not justify the risk to add more transactions and lose out on the reward. Is this a correct assumption(s)?

It appears that way, yes. I expect that as the block subsidy drops, it will eventually stop making sense to scale transaction fees with the block size.

[DeathAndTaxes]

Just how would a pool go about putting more transactions into the blocks they find? I assume they have to modify the source code, recompile, and then deploy? Are the changes trivial for the average pool operator?

My understanding (pool ops feel free to correct me) is that pools are already running highly customized version of bitcoind.  Honestly if you can't compile bitcoind then you probably don't have the technical skills to run a pool.

Still up to 250KB you wouldn't even need to compile bitcoind simply modify one line in the bitcoin.conf file.  A LOT of blocks are much smaller than 250KB.  So it becomes a no brainer to focus on that first.  The average block in past 30 days is ~150KB.  Eyeballing it, it looks like 80% of blocks are <250KB.  A good 25% of blocks aren't even 100KB and at least 10% aren't even 50KB.  If all blocks were 250KB we wouldn't even have a backlog (except for no-fee txs).

There are some exceptions.  BTC Guild does sometimes push out much larger blocks (350KB to 400KB).  Not sure what conditions trigger that.  Maybe they only do it when the backlog gets real bad.  https://blockchain.info/block-index/441049

And on top of this there is a greater chance of an orphan block resulting in the loss of 25 BTC?   It simply does not justify the risk to add more transactions and lose out on the reward. Is this a correct assumption(s)?

There is a cost but fees do offset that.  Users shouldn't expect pools to make larger blocks full of free tx but paid tx offset the marginal cost of larger blocks.  One estimate put the marginal cost (due to increased orphan risk) at ~0.04 mBTC per KB.  Still at some point pool ops have to decide if 1% or so lower orphan losses worth the bad PR of Bitcoin "choking" at ~0.3 tps.  I mean 0.3 tps.   The 1 MB limit is 4 to 6 tps.  PayPal is 50 tps.  VISA is 5,000 tps.   We are at 0.3 tps, and the network seems to be straining.   If pools are unwilling to push the envelope to at least 300KB to 500KB blocks well we might as well pack it up because this experiment is going nowhere.

For the record I do think they will adapt and I do think the protocol will eventually handle block broadcasting in a more intelligent manner so this becomes less of an issue over time.  The falling subsidy will also improve the economics of larger blocks.  It also isn't all on the pools.  Users need to accept you either PAY or you get a confirmation eventually where eventually can be days or possibly weeks.   It is a "charity" option and you get space when space is available (which probably means a block on Tuesday at 2:48 in the morning). The days of pay no fee and see it in the next block are over.  So doing that over and over and getting mad is just silly.