Blocks are [not] full. What's the plan?

[Dabs]

The days of pay no fee and see it in the next block are over.  So doing that over and over and getting mad is just silly.

I hope not. I have a medium priority transaction that's not yet confirmed for the past 12 hours, although I think it will confirm by the end of the day. I'll let you guys know. I sent a whole bitcoin a day after I got it, thinking it would confirm. The Satoshi Client (bitcoin-qt) did not request for a fee, so I did not include one.

But, that also tells me, I should include transaction fees next time if I need it to confirm within the hour.

[Gavin Andresen]

DeathAndTaxes: I'm normally impressed with your posts, but I think you've got some details wrong.

First, RE: the orphan cost of transactions:  Decker/Wattenhofer measured 80ms for a 1K bigger block. The math to compute orphan cost is:

Code:

orphan cost = (block_reward+fees) * (1 - e^(-(1/target block time)*delta_time)

Plugging in 25 XBT block reward, 600 target time, 0.08 delta time, and assuming no fees (to make the math easier):

Code:

orphan cost = 25 * (1 - e^(-(1/600)*0.08) = 0.0033

So 3.3 millies per kilobyte is the orphan cost.

Even if we assume Decker/Wattenhofer are off by a factor of two (we have made some improvements since they measured block propagation; better measurements welcome), default transaction fees (1 to 5 millies per kilobyte) are in the right ballpark to minimize orphan costs. the .1 default transaction fee does not come close to covering the orphan cost (edited: thanks foxpup).

It should be fairly easy to get about another factor of about 10-20 reduction in orphan costs. And as I said in another thread, if EVERYBODY produces larger blocks then EVERYBODY bears the increased orphan cost, and the result is better for everybody . There is a fixed number of new bitcoins to be earned, regardless of the orphan rate; everybody's share of that fixed number will be the same if everybody has a slightly higher orphan block rate. But everybody will earn more fees, and their bitcoins will be worth more because bitcoins will be more useful.

[Foxpup]

Even if we assume Decker/Wattenhofer are off by a factor of two (we have made some improvements since they measured block propagation; better measurements welcome), default transaction fees (1 to 5 millies per kilobyte) are in the right ballpark to minimize orphan costs.

You seem to have misplaced a decimal point.

[Gavin Andresen]

Yes, I did misplace the decimal point.  I'm really good at that.

[Dabs]

Received Time    2013-11-19 04:13:16
Included In Blocks    270581 (2013-11-20 08:53:32 +1,720 minutes)

My transaction confirmed in about 28 hours.

So I can still send no fee transactions, you just got to wait a little bit longer. Add a 0.0001 fee, and it confirms a lot sooner.

[niothor]

Oh god !!!!! another misleading title...
Was there ever a full 1mb block?

[Come-from-Beyond]

Right NOW, the reality is 0.1 mBTC IS fine.

2 days ago I received a payment for my work. 1 input, 1 output, 0.1 mBTC fee and the transaction stuck for a lot of hours. This is NOT fine. I think I should force all my customers to use BTC-e USD redeemable codes. That delay cost me >4000 USD due to BTC price crash.

[Peter Todd]

DeathAndTaxes: I'm normally impressed with your posts, but I think you've got some details wrong.

First, RE: the orphan cost of transactions:  Decker/Wattenhofer measured 80ms for a 1K bigger block. The math to compute orphan cost is:

Code:

orphan cost = (block_reward+fees) * (1 - e^(-(1/target block time)*delta_time)

Plugging in 25 XBT block reward, 600 target time, 0.08 delta time, and assuming no fees (to make the math easier):

Code:

orphan cost = 25 * (1 - e^(-(1/600)*0.08) = 0.0033

So 3.3 millies per kilobyte is the orphan cost.

Even if we assume Decker/Wattenhofer are off by a factor of two (we have made some improvements since they measured block propagation; better measurements welcome),

Actually the only measurement you need is the orphan rate and the block interval:

http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03274.html

Note that's actual block interval, not steady state, an oversight Michael Gronage corrected for me in his reply. Of course this doesn't give you detailed breakdowns of latency and bandwidth, but given the consolidation of hashing power the measurements Decker et al. made on network wide latency/bandwidth make a tonne of assumptions about topology that if not already inaccurate, won't be in the future.

[Yogafan00000]

All I know is I'm trying to do business in Bitcoins and I've got a wallet full of "0/unconfirmed" and no good answers as to what to do about it.

We ain't ready for mainstream yet.

[BurtW]

Oh god !!!!! another misleading title...
Was there ever a full 1mb block?

+1 Change the f*&$%ng title of this thread!

[DeathAndTaxes]

Right NOW, the reality is 0.1 mBTC IS fine.

2 days ago I received a payment for my work. 1 input, 1 output, 0.1 mBTC fee and the transaction stuck for a lot of hours. This is NOT fine. I think I should force all my customers to use BTC-e USD redeemable codes. That delay cost me >4000 USD due to BTC price crash.

Some factors to consider.
How many blocks were you "passed over"? Measuring in hours is kinda misleading.  A fee can't make a block come faster.   If in a 2 hour interval only 5 blocks were found that is a lot different than if 15 blocks were found.  

Another thing to consider is was the input confirmed or unconfirmed?  Often (not always but anecdotal it seems to be very common) when there is ranting about paying tx being "stuck" for hours it is because one or more of the inputs are unconfirmed and they didn't have fees.  If your inputs are unconfirmed and have no fee then for all intents and purposes your paying tx is a free one (at least until "parent pays child" is implemented).

I shouldn't share this semi-obvious fact but I pay a fee of 0.1001 mBTC on every single tx.  Since miners prioritize by fees and 99.9% of tx are either no fee or min fee (0.1 mBTC) it almost guarantees my tx will be prioritized the highest in the mempool and thus included in the VERY NEXT block for just 0.001 mBTC more than the default fee.

[Come-from-Beyond]

I shouldn't share this semi-obvious fact but I pay a fee of 0.1001 mBTC on every single tx.  Since miners prioritize by fees and 99.9% of tx are either no fee or min fee it almost guarantees my tx will be in the VERY NEXT block for just 0.001 mBTC more.

Good advice.

[gmaxwell]

Prioritize by fee in the stock code, at least, is made robust against that kind of gamesmanship by treating any fee below a minimum threshold as zero for the purpose of prioritizing the transactions.

[niothor]

Oh god !!!!! another misleading title...
Was there ever a full 1mb block?

+1 Change the f*&$%ng title of this thread!

And our prayers have been heard!!!!!!

[DeathAndTaxes]

DeathAndTaxes: I'm normally impressed with your posts, but I think you've got some details wrong.

Thanks and I am happy to be corrected.  

First, RE: the orphan cost of transactions:  Decker/Wattenhofer measured 80ms for a 1K bigger block.
...
So 3.3 millies per kilobyte is the orphan cost.

I didn't know about this source.  I recall (but can't seem to find a cite) with a much lower estimated cost although it may have simply been an error on my part ~0.4 mBTC vs ~4 mBTC.

It should be fairly easy to get about another factor of about 10-20 reduction in orphan costs.

Correct me if I am wrong but a rather simple block broadcast improvement would be to not include the tx just include their hashes.  Since a well connected miner aware of a given tx X should already have relayed that tx X to his peers there is little need to include the full tx in the block message.  Instead block message could just consist of the header and a list of tx hashes.  If average tx size is 400 bytes and the SHA-256 hash is 32 bytes that alone could cut the orphan cost of a tx by 90% to ~0.3 mBTC.  In reality since the tx hash is just being compared to the UXTO a more involved modification would be for the block message to include a truncated hash of the tx.  This wouldn't represent a security risk as the actual merkle tree would involve the full SHA-256 hash.  Using say the first 64 bits of the SHA-256 hash would still make collisions essentially highly unlikely events and would reduce the orphan cost by another 4x to <0.1 mBTC.

And as I said in another thread, if EVERYBODY produces larger blocks then EVERYBODY bears the increased orphan cost, and the result is better for everybody . There is a fixed number of new bitcoins to be earned, regardless of the orphan rate; everybody's share of that fixed number will be the same if everybody has a slightly higher orphan block rate. But everybody will earn more fees, and their bitcoins will be worth more because bitcoins will be more useful.

Agreed.  Hopefully pools and major solo operators can see that their long term profitability is based MORE on the growth of Bitcoin than the short term orphan cost.   Hopefully large pools are aware they don't need universal support.  If 70% of hashpower agrees to produce larger blocks (say 500KB avg) for the good of the network.  It cuts the orphan cost by 70%.   Still I think the orphan cost does highlight the reality that miners are going to be increasingly reluctant to devote more space to free tx.  This is something users will have to come to grips with.   Including a fee and waiting hours or days is unacceptable (although it is often for non-fee reasons) however no fee tx should be considered charity by users and if someone does you an act of charity in an hour, or day, or week well you got what you paid for.  The default behavior of most clients should probably be changed to include the "min fee" on ALL txs not just low priority ones.  If users want to they could change this but they should be warned "Including no tx fee may result in delayed confirmation times".  Enforcement for relaying at node level should still only be on low priority. 

It is somewhat ironic that this is more of an issue due to the higher exchange rate.  The min fee on low priority tx was lowered due to rising exchange rate.  Today 3.3 mBTC is ~$1.50.  Ouch.  However if Bitcoins were worth less it would be less of a cost.  Since Bitcoin is often used as a proxy for USD a 5 mBTC fee (which more than covers orphan costs) is more viable at a lower exchange rate.

[DeathAndTaxes]

Prioritize by fee in the stock code, at least, is made robust against that kind of gamesmanship by treating any fee below a minimum threshold as zero for the purpose of prioritizing the transactions.

0.1001 mBTC is ABOVE the min threshold of 0.1 mBTC. I am going to shut up now as I am quickly going to cut my own throat.

[gmaxwell]

0.1001 mBTC is ABOVE the min threshold of 0.1 mBTC. I am going to shut up now as I am quickly going to cut my own throat.

The threshold is 1 mBTC: main.cpp:int64_t CTransaction::nMinTxFee = 10000;  // Override with -mintxfee

[DeathAndTaxes]

0.1001 mBTC is ABOVE the min threshold of 0.1 mBTC. I am going to shut up now as I am quickly going to cut my own throat.

The threshold is 1 mBTC: main.cpp:int64_t CTransaction::nMinTxFee = 10000;  // Override with -mintxfee

On edit:  to avoid a long rambling semi-off topic subthread.  Lets just correct the math error here.

10,000 sat = 0.1 mBTC not 1 mBTC.

Deleting the rest of my posts.

[fcmatt]

Just how would a pool go about putting more transactions into the blocks they find? I assume they have to modify the source code, recompile, and then deploy? Are the changes trivial for the average pool operator?

My understanding (pool ops feel free to correct me) is that pools are already running highly customized version of bitcoind.  Honestly if you can't compile bitcoind then you probably don't have the technical skills to run a pool.

Still up to 250KB you wouldn't even need to compile bitcoind simply modify one line in the bitcoin.conf file.  A LOT of blocks are much smaller than 250KB.  So it becomes a no brainer to focus on that first.  The average block in past 30 days is ~150KB.  Eyeballing it, it looks like 80% of blocks are <250KB.  A good 25% of blocks aren't even 100KB and at least 10% aren't even 50KB.  If all blocks were 250KB we wouldn't even have a backlog (except for no-fee txs).

There are some exceptions.  BTC Guild does sometimes push out much larger blocks (350KB to 400KB).  Not sure what conditions trigger that.  Maybe they only do it when the backlog gets real bad.  https://blockchain.info/block-index/441049

And on top of this there is a greater chance of an orphan block resulting in the loss of 25 BTC?   It simply does not justify the risk to add more transactions and lose out on the reward. Is this a correct assumption(s)?

There is a cost but fees do offset that.  Users shouldn't expect pools to make larger blocks full of free tx but paid tx offset the marginal cost of larger blocks.  One estimate put the marginal cost (due to increased orphan risk) at ~0.04 mBTC per KB.  Still at some point pool ops have to decide if 1% or so lower orphan losses worth the bad PR of Bitcoin "choking" at ~0.3 tps.  I mean 0.3 tps.   The 1 MB limit is 4 to 6 tps.  PayPal is 50 tps.  VISA is 5,000 tps.   We are at 0.3 tps, and the network seems to be straining.   If pools are unwilling to push the envelope to at least 300KB to 500KB blocks well we might as well pack it up because this experiment is going nowhere.

For the record I do think they will adapt and I do think the protocol will eventually handle block broadcasting in a more intelligent manner so this becomes less of an issue over time.  The falling subsidy will also improve the economics of larger blocks.  It also isn't all on the pools.  Users need to accept you either PAY or you get a confirmation eventually where eventually can be days or possibly weeks.   It is a "charity" option and you get space when space is available (which probably means a block on Tuesday at 2:48 in the morning). The days of pay no fee and see it in the next block are over.  So doing that over and over and getting mad is just silly.

Posting below everything due to time constraints.

I have a feeling almost every large pool operator can compile bitcoin. No worries there. I also 'think' they do not make as many changes as we might think.

In the earlier days they did because mining was becoming popular and the load on the servers kept going up. So RPC changes were made and what not.
But now a lot of those changes are already in the code by default if my understanding is correct. Each time a new release comes out they have to make
fewer changes to the source before compiling.  And it always seemed like it was a small group of people suggesting and coding these changes. Not the operators
themselves. They just paid for it and/or gave a tip.

What does worry them, I think, is making a mistake in changing the source and running a binary that is failing to generate new blocks in some fashion for the pool causing a loss for the miners and a loss of hash rate as users possibly leave. It is simply not worth the risk so by running stock they truly minimize their risk. So until the risk can be minimized for the 'average' pool operator or the default source forces them to ALL change i think what we see today is the status quo.

I run a litecoin pool myself. And I can tell you I do not make a change lightly. I don't run the newest litecoin release. My main worry is scaling the server up to handle more hash rate and keeping things running non stop. I was always interested in getting more fees from blocks but not at the risk of an orphan. Just how long should I wait when litecoin finds a block every 2.5 minutes? That is what goes through my mind.

[DeathAndTaxes]

The min fee is 0.1 mBTC

The minimum fee is 0. The minimum fee which isn't regarded as a zero-fee transaction is 1 mBTC.

There is no reason for the threshold to not be set at the same value as the min fee to relay low priority txs.  No wonder why we have all kinds of issues with people having tx unconfirmed.  So the min fee for low priority tx to get relayed is 0.1 mBTC but then they are treated as zero fee by miners for inclusion in a block.  NOBODY (and I mean nobody I just checked the last 15,000 transactions) pays 1 mBTC per KB.  That is an asinine $0.60 per KB. The line of code is utterly useless.  It might as well be 10 BTC.   Every major pool is overriding that as they DO sort by fee.