I know this has been brought up before, but confirmation times are getting weird

[chriswilmer]

Guys... everything that everyone has said in this thread so far has ALWAYS BEEN TRUE (not paying transaction fees leads to long confirmation times, sure). None of that explains why the confirmation times are spiking NOW.

[DeathAndTaxes]

That's it, right here.  The block only permits a limited amount of zero fee transactions to be included, and any additional transactions must meet the criteria for the fee schedule to be included into a block. 

That is not correct.   A miner could make every block 1MB ~2,400 tx only include the free ones and exclude every paying tx if they wanted.  Not saying they would (it would be stupid) or they should but they could.  Saying the "block" prevents them is a cop out.

Miners decide what tx to include in a block.  The only thing which can't be included in a block is an invalid tx, or a sum of tx greater than 1MB in size.

[DeathAndTaxes]

Guys... everything that everyone has said in this thread so far has ALWAYS BEEN TRUE (not paying transaction fees leads to long confirmation times, sure). None of that explains why the confirmation times are spiking NOW.

It is very simple.

Tx volume is higher than average block size.  It is THAT SIMPLE.

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 300 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 400 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 600 tx every 600 seconds what is going to happen to the tx backlog?


If it is still too complicated lets try a non bitcoin metaphor.

If you have a pool (mem pool) and you are adding 2 gallons a minute to the pool (unconfirmed txs) and at the same time have the drain open and water is draining out of the pool at 1 gallon per minute (tx being placed into blocks) will the water in the pool rise or lower?  What is required to prevent the water level in the pool from rising?

[chriswilmer]

Guys... everything that everyone has said in this thread so far has ALWAYS BEEN TRUE (not paying transaction fees leads to long confirmation times, sure). None of that explains why the confirmation times are spiking NOW.

It is very simple.

Tx volume is higher than average block size.  It is THAT SIMPLE.

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 300 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 400 tx every 600 seconds what is going to happen to the tx backlog?

If users are creating an average of 400 tx every 600 seconds and miners are making blocks with an average of 600 tx every 600 seconds what is going to happen to the tx backlog?


If it is still too complicated lets try a non bitcoin metaphor.

If you have a pool (mem pool) and you are adding 2 gallons a minute to the pool (unconfirmed txs) and at the same time have the drain open and water is draining out of the pool at 1 gallon per minute (tx being placed into blocks) will the water in the pool rise or lower?  What is required to prevent the water level in the pool from rising?

I feel like we are talking past each other...

Are you saying that you think the tx volume has suddenly crossed a threshold ~4 days ago? I am not asking about why confirmation times go up and down in general, but why they have gone up a lot very recently.

(I noticed for example, that Elgius is de-prioritizing transactions that reuse addresses, this is new, and perhaps that is a cause for this)

[DeathAndTaxes]

Well threshold would imply some sort of cental block planning agency.   Various miners have various different block parameters.  Tx volume is higher than the block size being created by the various parameters currently used by various miners.   Some miners target larger blocks, some target smaller ones, a couple seem to include nothing but the coinbase tx.   Collectively all miners have a certain tx throughput which is less than the tx volume and the limit imposed by the 1MB cap.

Miners can either expand the # of tx they include in blocks
OR
Users can collectively reduce tx volume
OR
The backlog will grow

For the time being a way to put yourself at the front of the line is to pay a tx fee but that won't reduce/eliminate the backlog it will just ensure your tx has a higher chance of going first.

[BurtW]

(I noticed for example, that Elgius is de-prioritizing transactions that reuse addresses, this is new, and perhaps that is a cause for this)

This will cause the transactions that reuse addresses to be delayed.  Is there a way to get a statistic on how many transactions fall into this category (reused therefore being delayed by some of the pools)?

[chriswilmer]

Well threshold would imply some sort of cental block planning agency.   Various miners have various different block parameters.  Tx volume is higher than the block size being created by the various parameters currently used by various miners.   Some miners target larger blocks, some target smaller ones, a couple seem to include nothing but the coinbase tx.   Collectively all miners have a certain tx throughput which is less than the tx volume and the limit imposed by the 1MB cap.

Miners can either expand the # of tx they include in blocks
OR
Users can collectively reduce tx volume
OR
The backlog will grow

For the time being a way to put yourself at the front of the line is to pay a tx fee but that won't reduce/eliminate the backlog it will just ensure your tx has a higher chance of going first.

I don't understand why our communication is breaking down so much. I can tell that we're both trying to tell each other something, but it isn't working.

I completely understand the general mechanics of transaction backlogs and I very much appreciate your explaining them in detail. The point of my starting this thread was to wonder what event/dynamic/new-mining-pool/whatever occurred on Nov. 3, 2013 such that, since that date, the confirmation times have gone up significantly.

My apologies in advance if I am somehow misunderstanding you.

[chriswilmer]

(I noticed for example, that Elgius is de-prioritizing transactions that reuse addresses, this is new, and perhaps that is a cause for this)

This will cause the transactions that reuse addresses to be delayed.  Is there a way to get a statistic on how many transactions fall into this category (reused therefore being delayed by some of the pools)?

That would be great to find out! Apparently BTC Guild is thinking about doing this too.

[BurtW]

I don't understand why our communication is breaking down so much. I can tell that we're both trying to tell each other something, but it isn't working.

I completely understand the general mechanics of transaction backlogs and I very much appreciate your explaining them in detail. The point of my starting this thread was to wonder what event/dynamic/new-mining-pool/whatever occurred on Nov. 3, 2013 such that, since that date, the confirmation times have gone up significantly.

My apologies in advance if I am somehow misunderstanding you.

Could it be as simple as the total number of transactions per day crossing a threshold where the backlog starts to happen, given a flat number of transactions per block?

Notice that about that time the "Number of transactions excluding popular addresses" jumped up and crossed over the 50,000 transactions per day line.

6 * 24 * 350 is about 50K transactions per day.

[chriswilmer]

I don't understand why our communication is breaking down so much. I can tell that we're both trying to tell each other something, but it isn't working.

I completely understand the general mechanics of transaction backlogs and I very much appreciate your explaining them in detail. The point of my starting this thread was to wonder what event/dynamic/new-mining-pool/whatever occurred on Nov. 3, 2013 such that, since that date, the confirmation times have gone up significantly.

My apologies in advance if I am somehow misunderstanding you.

Could it be as simple as the total number of transactions per day crossing a threshold where the backlog starts to happen, given a flat number of transactions per block.

Notice that about that time the "Number of transactions excluding popular addresses" jumped up and crossed over the 50,000 transactions per day line.

6 * 24 * 350 is about 50K transactions per day.

Yeah, that's a fine hypothesis, but why would popular/non-popular matter? If you look at the total number of transactions over the past year, it has been pretty constant (well, constant is relative, but the important point is that it has been higher in the past).

https://blockchain.info/charts/n-transactions?timespan=1year&showDataPoints=false&daysAverageString=7&show_header=true&scale=0&address=

[BurtW]

I believe in that graph you are looking at transactions processed per day, which, of course, will show a constant number given the saturation condition we are talking about.

Assume the number of "transactions to the popular addresses" is constant then a jump in transactions to the other "not popular" addresses shows that now we have more transactions being requested than are being processed by the system.

This will cause a backlog - at about the time the confirmation time increased.

[phelix]

Code:

            // Free transaction area
            if (nNewBlockSize < 27000)
                nMinFee = 0;

By default the client reserves 27kb for free high priority transactions.

Code:

/** The maximum allowed size for a serialized block, in bytes (network rule) */
static const unsigned int MAX_BLOCK_SIZE = 1000000;
/** The maximum size for mined blocks */
static const unsigned int MAX_BLOCK_SIZE_GEN = MAX_BLOCK_SIZE/2;

Code:

    // Raise the price as the block approaches full
    if (nBlockSize != 1 && nNewBlockSize >= MAX_BLOCK_SIZE_GEN/2)
    {
        if (nNewBlockSize >= MAX_BLOCK_SIZE_GEN)
            return MAX_MONEY;
        nMinFee *= MAX_BLOCK_SIZE_GEN / (MAX_BLOCK_SIZE_GEN - nNewBlockSize);
    }

Also it will raise the minFee once blocksize becomes larger than 250kb.

Of course miners can change this.

[MoonShadow]

That's it, right here.  The block only permits a limited amount of zero fee transactions to be included, and any additional transactions must meet the criteria for the fee schedule to be included into a block. 

That is not correct.   A miner could make every block 1MB ~2,400 tx only include the free ones and exclude every paying tx if they wanted.  Not saying they would (it would be stupid) or they should but they could.  Saying the "block" prevents them is a cop out.

Miners decide what tx to include in a block.  The only thing which can't be included in a block is an invalid tx, or a sum of tx greater than 1MB in size.

I was generallizing.  I know that miners can choose their own rules, but the default set of rules say that there is a limit to the free transactions.  Changing those rules requires programming skill and the desire to do so.

[phillipsjk]

Anyone knows if Eligius is still doing zero-fee transactions?

As far as I know, they have always required a fee of 4096 Satohies.

[JTrain_51]

I also have noticed a problem when I was using the bitcoin atm machine i went there and my confirmation came instantly but the next day it took my 10 minutes to get my 1 confirmation from block chain

[speeder]

I also have noticed a problem when I was using the bitcoin atm machine i went there and my confirmation came instantly but the next day it took my 10 minutes to get my 1 confirmation from block chain

Block chain usually takes 10 minutes at most to confirm a good transaction.

You were just lucky on the first day (making your transaction just before a block) and unlucky on the second (making your transaction just AFTER a block)

[Foxpup]

Code:

            // Free transaction area
            if (nNewBlockSize < 27000)
                nMinFee = 0;

By default the client reserves 27kb for free high priority transactions.

Code:

/** The maximum allowed size for a serialized block, in bytes (network rule) */
static const unsigned int MAX_BLOCK_SIZE = 1000000;
/** The maximum size for mined blocks */
static const unsigned int MAX_BLOCK_SIZE_GEN = MAX_BLOCK_SIZE/2;

Code:

    // Raise the price as the block approaches full
    if (nBlockSize != 1 && nNewBlockSize >= MAX_BLOCK_SIZE_GEN/2)
    {
        if (nNewBlockSize >= MAX_BLOCK_SIZE_GEN)
            return MAX_MONEY;
        nMinFee *= MAX_BLOCK_SIZE_GEN / (MAX_BLOCK_SIZE_GEN - nNewBlockSize);
    }

Also it will raise the minFee once blocksize becomes larger than 250kb.

Of course miners can change this.

^^ This. Miners will not mine large blocks unless you pay more than double the normal transaction fee. Much more than double, if you want very large blocks. People have often asked, "will paying a larger transaction fee really lead to faster confirmation times?" and the answer is now yes. Will not paying a fee (even where free transactions are "allowed") lead to longer confirmation times? Also yes.

People, pay the fucking transaction fees. If your transaction is taking a long time to confirm and you paid no fee, or only the minimum required fee, it's your own damn fault.

[chriswilmer]

Code:

            // Free transaction area
            if (nNewBlockSize < 27000)
                nMinFee = 0;

By default the client reserves 27kb for free high priority transactions.

Code:

/** The maximum allowed size for a serialized block, in bytes (network rule) */
static const unsigned int MAX_BLOCK_SIZE = 1000000;
/** The maximum size for mined blocks */
static const unsigned int MAX_BLOCK_SIZE_GEN = MAX_BLOCK_SIZE/2;

Code:

    // Raise the price as the block approaches full
    if (nBlockSize != 1 && nNewBlockSize >= MAX_BLOCK_SIZE_GEN/2)
    {
        if (nNewBlockSize >= MAX_BLOCK_SIZE_GEN)
            return MAX_MONEY;
        nMinFee *= MAX_BLOCK_SIZE_GEN / (MAX_BLOCK_SIZE_GEN - nNewBlockSize);
    }

Also it will raise the minFee once blocksize becomes larger than 250kb.

Of course miners can change this.

^^ This. Miners will not mine large blocks unless you pay more than double the normal transaction fee. Much more than double, if you want very large blocks. People have often asked, "will paying a larger transaction fee really lead to faster confirmation times?" and the answer is now yes. Will not paying a fee (even where free transactions are "allowed") lead to longer confirmation times? Also yes.

People, pay the fucking transaction fees. If your transaction is taking a long time to confirm and you paid no fee, or only the minimum required fee, it's your own damn fault.

Maybe I should change the title of this thread... you are addressing something that I feel is quite off-topic. I just wanted to have a discussion over what dynamics changed after Nov. 3, 2013 -- not a general discussion about transaction fees and confirmation times.

So far the hypotheses are that Elgius changed how they prioritize transactions based on address reuse and that we hit recently hit a threshold of transactions broadcast per day (due to the growing adoption of Bitcoin).

[Foxpup]

Maybe I should change the title of this thread... you are addressing something that I feel is quite off-topic. I just wanted to have a discussion over what dynamics changed after Nov. 3, 2013 -- not a general discussion about transaction fees and confirmation times.

So far the hypotheses are that Elgius changed how they prioritize transactions based on address reuse and that we hit recently hit a threshold of transactions broadcast per day (due to the growing adoption of Bitcoin).

This is what I'm talking about. I didn't bother mentioning it because it's already been explained several times in this thread. I was merely elaborating on why, with a sufficiently large number of transactions, many of those transactions will not get into a block, even though the block size is nowhere near the limit and the transactions pay the minimum required fee. The minimum fee is not sufficient if a block is more than a quarter full. If nobody ever pays more than the minimum fee, blocks will never be more than a quarter full, leading people to ask "why is there a huge backlog of fee-paying transactions when the blocks are nowhere near full?" And now you know. And knowing is half the battle.

[DeathAndTaxes]

Maybe I should change the title of this thread... you are addressing something that I feel is quite off-topic. I just wanted to have a discussion over what dynamics changed after Nov. 3, 2013 -- not a general discussion about transaction fees and confirmation times.

So far the hypotheses are that Elgius changed how they prioritize transactions based on address reuse and that we hit recently hit a threshold of transactions broadcast per day (due to the growing adoption of Bitcoin).

This is what I'm talking about. I didn't bother mentioning it because it's already been explained several times in this thread. I was merely elaborating on why, with a sufficiently large number of transactions, many of those transactions will not get into a block, even though the block size is nowhere near the limit and the transactions pay the minimum required fee. The minimum fee is not sufficient if a block is more than a quarter full. If nobody ever pays more than the minimum fee, blocks will never be more than a quarter full, leading people to ask "why is there a huge backlog of fee-paying transactions when the blocks are nowhere near full?" And now you know. And knowing is half the battle.

That is a flawed understanding of the fee rules.   High priority txs have no required fee, none.  As in 0 satsohis.  This applies regardless if a block is 1MB in size.  

The min mandatory fees FOR LOW PRIORITY TXS are designed as a denial of service prevention mechanism.  They are only enforced at the client level.   Even if miners follow those they have absolutely no effect on high priority txs.  

Lastly the rules aren't enforced at the protocol level.  Miners an build any valid block they want by using custom bitcoind.  This is hardly beyond the capabilities of a major pool (all are using custom nodes already).  It is possible to have a block contain NOTHING but no fee txs (as in ~2,400 of them) and the block is still valid.