Blocks are [not] full. What's the plan?

[DeathAndTaxes]

In the short term probably not.  The reality is the lowest cost, higher short term revenue would simply be to solo mine empty block and leave all txs even paying ones.   However miners (or pool operators) have shown some longer term thinking and HAVE built larger blocks.   However I think it does illustrate that if the min fee of 0.1 mBTC doesn't cover the orphan cost it is downright silly to expect miners to build massive blocks full of free txs and simply kill their own revenue so other people can avoid paying ~$0.10.

I would think the fact that miners are already willing to do so would be evidence that it's not downright silly.

If miners don't want to include free transactions, that's fine. But when miners collude to limit the ability of others to include free transactions, that's not fine.

Miners aren't colluding but they do heavily limit the amount of space the devote to free txs.  Some pools like Eligus make the LARGEST blocks but include zero (yes zero) free txs.   Excluding brand new tx, ones with unconfirmed outputs, and double spends/problems at any given time 90% of the memory pool is free txs.

The fact that miners haven't gone cold turkey and universally killed all free txs doesn't mean it is a reliable method of making a payment.  The free tx volume is growing and the amount of free txs in a block are declining the result of those two is the backlog everyone complains about.   It is silly to think miners are going to change.  It makes no sense for them to do so.  Many will include some free tx because it provides a release valve on the network but while blocks get bigger I wouldn't expect the amount of free tx in blocks to get bigger.  If you want timely processing include a fee it is really that simple.  If you don't then it could be days, weeks, potentially months before your tx is processed.  You are asking for charity and charity isn't always reliable.

[DeathAndTaxes]

If miners can make a positive return by producing larger blocks ... they will.

How? By forking the blockchain?

Eventually that might happen if the limit isn't raised. But hopefully that's not going to be required.

They want money, more money is always better.

And they'll get more money by not colluding to artificially limit supply. Otherwise there will either be a fork, or there will be an altcoin, or there will be an anti-trust lawsuit, or something will give.

(That is, assuming you're correct in the first place. It's not actually true that all miners care only about money.)

However tonight you could flip the network to 1 GB blocks and ACTUAL block sizes aren't going to change much.

They'll change though. Eligius will soon start creating larger blocks.

Nobody is making 1MB blocks.  Nobody.  Not a single block in the history of Bitcoin is 1MB.  So the "limit" isn't a limit.  Miners are choosing their own parameters which result in blocks less than 1 MB.

It would be like the fastest car in the world is 180 mph and there is too much traffic so the government decides to raise the speed limit from 500 mph to 2,000 mph think that is going to have any material change.   The limit will be raised eventually I have no doubt but right now the constraint on tx volume isn't the 1 MB limit.  That is pretty obvious when there are no 1 MB blocks.  The constraint is on the economics of mining.   Eligus for example makes some of the largest blocks on the network.  Routinely over 500 KB but they also include no free transactions.  Eligus couldn't make a 1MB block right this second even if they WANTED TO because there aren't 1MB worth of paying tx waiting for a block.  So how would raising the limit to 2MB, 5MB, 10MB change anything to that equation?

Case in point here is a recent Eligus block.
https://blockchain.info/block-index/444092/00000000000000029fd11f8e23b450749807f78ab4aa789b764cd10ea7062e59
780KB
1280 txs. 

It couldn't be any larger because it includes 100% of the tx which met Eligus inclusion requirements. 

[justusranvier]

or there will be an anti-trust lawsuit

Good luck with that.

[DeathAndTaxes]

Do we even know that a block of exactly 1 megabyte would be accepted by a majority of the miners?

Yes.  That is what testnet is for.  You are running testnet to conduct your own research before "fixing" Bitcoin.

Like I said the limit will be raised.  It is just a matter of how, when, and to what level.   Still if the limit was 2MB right now we still wouldn't have 1MB blocks.  Miners are chosing to self enforce a lower limit.

Actually even that is simplistic because most miners are enforcing tx rules not a preset block size.  The intersection of current tx volume and the rules set by miners is producing blocks <250KB on average.  Going from 1MB to 2MB or 20MB isn't going to change that tx selection behavior.  Most waiting tx (if we look at seen longer than 1 block prior) are free txs.  Miners are willing to make larger blocks but not larger blocks full of free txs.

[rme]

Here it comes my idea about this issue:

Miners include few transactions in blocks because more transactions equals more probability of orphaned block so lets equal all blocks:

Miners should craft a block normally, so lets imagine they generate a 250KB block. Before they send it to other node they have to concatenate junk bytes (random (?)) to the block data, so all blocks are 1MB.

When a node sees this block, they broadcast it and when they finish they delete this junk bytes and they only the block.


Pros:
- All blocks "are" 1MB in terms of relaying them.
- We avoid other more tecnical mecanisms
Cons:
- Bitcoin QT needs some bandwith more because now all blocks are 1MB.


If one day we need to rise the 1MB block limit this process will be the same but all blocks will require to be 10MB (for example). We only need to concatenate junk to them.


How to perform this hard fork?
Bitcoin core developers can release an update that includes this fix but only enforcing it when the blockchain reaches the block 277000 (30 days later) so we give some time for people and miners to update their software.

[niniyo]

I would like to see some statement from the developers that actually affirms their commitment to the goal of keeping transactions costs low and transaction volume high.  At the rate we are going, we're going to be limited to only a few transactions per second, and as the competition for block space increases, bitcoin will turn into a system only for higher value transactions.

If the devs are happy to let it go in that direction, I'd like to know now so I can sell my bitcoins.  There's no future in bitcoin if we throw away all the key features.

Prediction:  By 2016, either you will be able to pay for your coffee via the blockchain with affordable fees, or bitcoins will be worth less than $1000 and investment in bitcoin ventures will dry up.

[zvs]

What's the limit that Eligius has set? (*) How did Eligius arrive at that limit? (**) Apparently you know the answers to these questions, since you can make the bold statement that "if the limit was 2MB right now we still wouldn't have 1MB blocks."

how exactly was that a bold statement?

eligius has a limit of 1MB.  they arrived at that limit because that's the limit set in the bitcoin protocol, probably.  it sure wasn't because they were colluding with other pools

there has never been more than 900KB worth of transactions waiting that have paid the standard fee (0.0001 per 1kb), hence the lack of a 2MB block even if it wouldn't be rejected by the rest of the network

[Cryddit]

Crap.

This issue might be addressed by making sure all blocks propagate equally slowly.  I read in another thread where someone is proposing forcing all blocks to be exactly 1M in size so they propagate at the same speed, but that's a waste of bandwidth.

No matter how you slice it though, failure of transactions to go through in a timely way looks like a service failure to actual users of bitcoin.

And it matters.  If bitcoin doesn't scale smoothly up past 100 tx/second with minimal tx expenses, then it will die within weeks after that limit starts to inconvenience people.  Investors will drop it like a rock and crash the market the minute they see that this isn't a system which can replace VISA, MasterCard, and Western Union while enabling micropayments and lowering costs by at least a factor of five to compensate for basic distrust of a new system.  That's what they thought they were buying into, after all. 

So, an order of magnitude faster block propagation gets us what?  Closer to the current theoretical max of less than 8 transactions per second, and still an active deterrent for miners to add tx to blocks unless  they pay at least $0.30 in fees.  All blocks propagating at the same speed gets us what?  More orphan blocks (which in the long run doesn't hurt miners because just as many valid blocks per hour will be found after difficulty adjustments), no marginal cost to miners for including more transactions in blocks, but still limited to under 8 transactions per second. 

That's just not enough. 

The protocol needs a fundamental design change for bitcoin to continue to exist at this valuation.  The current valuation represents investor expectations that the current protocol cannot fulfill.

[Luke-Jr]

Bitcoin transactions "go through" instantly, just like credit cards.
It's just confirmation that takes an hour or more (credit cards take 6+ months!).

[Luke-Jr]

What's your current blockmaxsize for Eligius? If the protocol limit were raised to 2,000,000 bytes, would you raise your blockmaxsize?

Eligius's blockmaxsize will likely always be the largest possible on the network (minus breathing room to avoid potential bugs).
That's not to say the blocks will always get that big - that depends on priorities, transaction fees, spam filters, etc...

[Lloydie]

I just tried to send BTC6.5 with a 0.0001 fee (per software recommendation) and it took 2.5 hours! Yes, this issue is impacting users like me now.

[Luke-Jr]

I just tried to send BTC6.5 with a 0.0001 fee (per software recommendation) and it took 2.5 hours! Yes, this issue is impacting users like me now.

Sends are almost always instant with Bitcoin.

Confirmation may have taken 2.5 hours, but that's still relatively fast.
Outside of Bitcoin, it's typically 6+ months with expensive credit card fees or at best half a day with even more expensive wiring fees.

[Lloydie]

I don't disagree and I'm still super bullish bitcoin, but I think discus fish pool are doing block sizes of 48 kB.  Would be good if someone did something clever about that.   

[Lloydie]

Note: the equivalent Ltc transfer took less than 4 minutes. (I know it's meant to be faster, but still...)

[Luke-Jr]

Note: the equivalent Ltc transfer took less than 4 minutes. (I know it's meant to be faster, but still...)

Obviously an unused network is going to find room for your transaction with lower fees (although is it really "equivalent" in that case?).
Reality is that all things equal, Litecoin is not any faster, though!

[Lloydie]

But it's not equal and bitcoin's transactions have slowed considerably. 

[Lloydie]

When max block is 1MB and pools are using less than 200kB and in Discus Fish's case 48kB, this is something that doesn't need to happen, I'm guessing, as I'm no tech wizard unlike you.

[Luke-Jr]

Rumour has it the Chinese pools can't get a decent internet connection in China and are too cheap to bother setting up a remote block-making node, so they just make tiny blocks and push the workload onto the other miners.
I don't have first-hand knowledge if this is true or not, so research it before acting on it...
Feel free to suggest they get in touch with me if it turns out to just be a technical problem of some sort (these pools aren't participating in the regular pool-operator communciations networks, so I'm not sure how to reach them off-hand).

[go1111111]

Here it comes my idea about this issue:

Miners include few transactions in blocks because more transactions equals more probability of orphaned block so lets equal all blocks:

Miners should craft a block normally, so lets imagine they generate a 250KB block. Before they send it to other node they have to concatenate junk bytes (random (?)) to the block data, so all blocks are 1MB.

When a node sees this block, they broadcast it and when they finish they delete this junk bytes and they only the block.


Pros:
- All blocks "are" 1MB in terms of relaying them.
- We avoid other more tecnical mecanisms
Cons:
- Bitcoin QT needs some bandwith more because now all blocks are 1MB.


If one day we need to rise the 1MB block limit this process will be the same but all blocks will require to be 10MB (for example). We only need to concatenate junk to them.


How to perform this hard fork?
Bitcoin core developers can release an update that includes this fix but only enforcing it when the blockchain reaches the block 277000 (30 days later) so we give some time for people and miners to update their software.

This is exactly what I was about to post. It seems like an elegant solution with good incentives (if someone includes a reasonable fee with their transaction, the miner would rather have it in their block than some junk).

I haven't thought that deeply about this, but it may not be necessary to have the minimum block size be equal to the maximum block size. Could the minimum size of the next block be calculated by every node based on some property of the N previous blocks? The intuition is that we'd want the minimum size to be maybe 10% larger than the block size that we predict we'll need to include all transactions that pay a reasonable fee. So perhaps if this code were working right now, the min block size would be 250KB instead of 1MB, but the max block size would still be 1 MB.

[jl2012]

This is exactly what I was about to post. It seems like an elegant solution with good incentives (if someone includes a reasonable fee with their transaction, the miner would rather have it in their block than some junk).

I haven't thought that deeply about this, but it may not be necessary to have the minimum block size be equal to the maximum block size. Could the minimum size of the next block be calculated by every node based on some property of the N previous blocks? The intuition is that we'd want the minimum size to be maybe 10% larger than the block size that we predict we'll need to include all transactions that pay a reasonable fee. So perhaps if this code were working right now, the min block size would be 250KB instead of 1MB, but the max block size would still be 1 MB.

This is TOTALLY useless. If the content of the junk is dynamic but deterministic (e.g. repeatedly hashing the last block), miners don't need to transfer the junk because everyone know the content. If the content is unspecified, all miners will fill it with 0s. So, again, they don't need to transfer the junk because everyone know the content. If you require different junk for different block, all miners will simply fill it with the current block height. If you require "random" junk, you must have a public algorithm to determine "randomness" so it's no longer random and miners will make it deterministic again. No miner will break this consensus because everyone want to save bandwidth