Bitcoin Block confirmations, Why do they lag at times?

[logfiles]

I understand the average bitcoin block confirmation time is around 10 minutes, however there is something that i would love to understand.
Earlier on this week i observed that at certain times the next block would be confirmed in over 1 hour while at other times 3 blocks would be confirmed in 10 minutes.

During the times when the next block would be confirmed in over 1 hour, i could still see over 40K unconfirmed transactions. What causes this lag at times?
I presume at least some transaction among the 40K unconfirmed transactions have already optimal transaction fees. Why not just confirm those transactions with optimal fees in that over-one-hour-wait?

Thanks in advance for any useful info 

[hosseinimr93]

Each block has a proof of work problem.
Miners compete with each other to find the solution of this mathematical problem. (the solution is called hash).

Sometimes, a miner can find the solution (the hash) in few seconds. And sometimes, it takes more than an hour until a miner finds the solution.

Why we say the confirmation time is 10 minutes?
 
That's only in theory.

After every 2016 blocks the difficulty is adjusted. In simple words, when blocks are mined in more than 10 minutes, that mathematical problem becomes easier and when blocks are mined in lower than 10 minutes, that mathematical problem becomes more difficult. This adjustment is done every 2016 blocks.
When total hash power of miners increases, difficulty needs to increase. When total hash power of miners decreases, the difficulty needs to decrease.
These adjustments are done, so the average confirmation time is close to 10 minutes as much as possible. Even the average confirmation time is not exactly 10 minutes. Because the total hash power is always changing.

[pooya87]

it is not exactly a "mathematical problem", but looks like it. you want to compute hash of a data that you keep changing until you get a hash that looks a certain way. that data is the block header (fixed 80 bytes) and the "certain way" that is should look is that it should have starting zeros or if i wanted to be more accurate we want the hash result converted to an integer be smaller than a target.

this whole process is completely random and based on luck. you may hash a certain block header and get the desired result right away (aka get lucky) so the block is found in a second. or you can change it trillions of times and still not find the answer (aka get unlucky).

to understand how PoW works you can use an arbitrary byte array and hash it to see how long it takes to reach an answer. to simplify i am going to use a 3 byte long array and skip converting to int and only focus on starting zeros:
starting data is the data value we start hashing with, each round we perform a single SHA256 and if the result had 2 zeros at the beginning we stop and report the result (we also skip reversing hashes).
starting data = { 0, 0, 0 } -> { 0, 118, 222 } after 30,431 hashes. 000091907ff1256d5738c9da2aa5056111913025e941e598374cad4d022ca80f
starting data = { 35, 4, 89 } -> { 36, 6, 219 } after 42,549 hashes. 000061353b12eb3e9bfaa8d17fa130fb10be6480e6204af7227b94d80e1b056f
starting data = { 40, 130, 200 } -> { 40, 131, 254 } after 111 hashes. 00005e7eb7c51ee31963d05160bf2d91e8e7f7573e2b7acf7c29365eaa237461
as you see in the end we got lucky and chose a value that was so close to the result that we found it after 111 hashes instead of 30k, 40k. that's the same in reality with a block header.

[LoyceV]

You can easily test this for yourself with 6 dices: Throw all dices once every 10 minutes. Now imagine each time you roll a 6, a block is found.
On average, you'll find one block every 10 minutes, but once in a while you'll find more than one, while at other moments you find nothing. Occasionally, it can take an hour or more before you throw a 6!

[o_e_l_e_o]

The replies above explain why mining is essentially the luck of a random process, but I wanted to clear up a misconception you have based on these two sentences you wrote:

I presume at least some transaction among the 40K unconfirmed transactions have already optimal transaction fees. Why not just confirm those transactions with optimal fees in that over-one-hour-wait?

The fees of the unconfirmed transactions are completely irrelevant when considering when the next block might be found. Having a transaction with a higher fee doesn't make a block any more or less likely to be found, it simply means that transaction is more likely to be included in the next block when it is found.

There is nothing for a miner to be gained by not broadcasting a valid block once they have found one. All that will happen is someone else will broadcast a valid block instead, and they will lose both the transaction fees and the freshly generated bitcoin of the block reward. Even broadcasting an empty block with no transactions in it will still earn a miner 12.5 BTC (soon to be 6.25 BTC after the halving). Indeed, if a miner found a valid block, but then wanted to include a higher fee transaction in it before broadcasting it, they would have to start the process again - changing any of the transactions within the block will change the resulting hash, and almost certainly invalidate it.

The reason that even high fee transactions are waiting over an hour is simply because there has been no block found to include them in. It wouldn't matter if you paid a fee of 1 BTC - your transaction cannot be confirmed until a block has been found.

[mikeywith]

I would like to add a bit to the above answers, this is not just about "luck", it has a lot to do with numbers as well.

in other words, the difficulty is adjusted for every epoch based on the time it took to mine the previous 2016 blocks (actually 2015 blocks but doesn't really matter here).

Taking the current epoch as an example, the current difficulty is about 13.9T, and for miners to find a block every 10 mins for whatever left of the current epoch the hashrate must not change, which is never the case and that's why we need the difficulty adjustment in the first place.

Think of it this way:

 Diff / Hashrate = Time

* The difficulty formula is way more complicated, this is just to give you a clear simple explanation of how things work.


You want time to be 10, the hashrate is 1.39, you set diff to 13.9 to get

13.9 / 1.39 = 10

what happens when the hashrate changes? remember the diff is CONSTANT for every 2016 blocks, the hashrate is VARIABLE.

Now assuming the mining "luck" remains constant at 100%, and bitcoin price drops 30-40% in price, a lot of mining gears (let's assume 30%) will go offline, and the problem starts here

Diff (constant) = 13.9
Hashrate = 0.973 ( after a 30% drop)

 Diff / Hashrate = Time

13.9/0.973 = 14 mins

We don't usually see a 30% drop in difficulty and by looking at the previous epoch, we had about 15% drop in difficulty and it was massive, which means on average those 2016 blocks took 11.5 mins to mine, but that's just the average, it's very possible that at certain points in time the hashrate drops 10-30% and remains there for a while, and thus blocks will take longer to be found even with luck staying constant at 100% but with a little bit of bad luck - blocks will take even longer to be mined, I think it's easy to observe that during a bull market when people keep on adding gears online, you see blocks being found more often in less than 10 mins, and the opposite is true.

Without a block found, it does not matter how many transactions are there or who do they belong to, even if the guy who owns the largest mining farm wants to send some bitcoin and miners fail to find a block for 2 hours, he MUST wait for that block to be mined, you should also keep in mind that transactions with higher fees don't have any technical advantages over transactions with lower fees, they are just "economically" superior because miners are financially incentivized.

[Lauda]

While these answers have substances, they are all wrong i.e. do not explain what is going on. The time between blocks follows the Exponential distribution, the number of blocks in an interval follows the Poisson distribution (Poisson distribution can be derived from an Exponential or Binomial distribution) — with the expected time being 10 minutes, and the variance being 100 minutes.

During the times when the next block would be confirmed in over 1 hour, i could still see over 40K unconfirmed transactions. What causes this lag at times?

1 hour blocks have very little to do with the hashrate unless there is a temporary collapse in the hashrate (very unlikely and infrequent, but 1 hour blocks are not that infrequent in comparison). They have some influence, as in a strong decrease in the hashrate at the "right time" could cause the 100 minute variation to be even higher (but this happens very infrequently).

End thread.

[nullius]

Bitcoin mining is a Poisson process.  Lauda hit the nail on the head by identifying the mathematically precise distributions, for those who want to learn more about how this really works.

The popular expectation that blocks occur on some sort of a schedule is a form of classic Gambler’s Fallacy.  Each calculation of a hash is (pseudorandomly) independent of all other calculations.

Think of it roughly this way:  When an hour passes between blocks, the network hashrate as a whole had a losing streak.  When two blocks occur within one minute of each other, that counts as something tantamount to a winning streak.  Whereas the time since the last block neither increases nor decreases the probability that any particular hash will be lucky next blockhash.  As LoyceV said, you can get a very basic conceptual feel for how this works by playing with some dice.  (I was going to suggest flipping a coin, and counting your H/T streaks.)  And indeed:

1 hour blocks have very little to do with the hashrate unless there is a temporary collapse in the hashrate (very unlikely and infrequent, but 1 hour blocks are not that infrequent in comparison).

The overall distribution of times between blocks, and distribution of blocks in any given interval, will follow the distributions stated by Lauda; however, to expect that we are “due” for a block because it’s been 10 minutes since the last one, or to feel like it’s “too long” since the last block because an hour has passed, is no different than selling a gambling script that essentially tries to measure winning/losing streaks.  Except in very rare circumstances of sudden, significant decrease in hashrate, confirmations do not “lag”.  They simply follow a distribution that is unexpected by those who have been told the technically incorrect information that “confirmation takes 10 minutes”.

(Thanks to o_e_l_e_o for explaining another popular misconception.)

[mikeywith]

While these answers have substances, they are all wrong i.e. do not explain what is going on. The time between blocks follows the Exponential distribution, the number of blocks in an interval follows the Poisson distribution (Poisson distribution can be derived from an Exponential or Binomial distribution) — with the expected time being 10 minutes, and the variance being 100 minutes.

I consider this answer to be the most accurate, non of the above informations is wrong but they simply don't really answer the question (including my answer)

Lauda brought up a very interesting point to say the least, I have read about Poisson distribution in the white paper some time ago, but never thought of using exponential distribution to "estimate" time between blocks, after reading this I decided to make a little excel sheet.

Using the Cumulative distribution function for the exponential distribution where λ=1/600 as explained by Felix Weis




I am not completely new to probabilities and statistics, but after seeing these results I have to admit that I am surprised! I was not expecting to see such numbers.

Fun Facts:

- Every 621 blocks or 4.3 days one block is expected to take between 55-60 mins.

- 63.21% of blocks are solved in less than 10 mins.

- Every 4595 blocks or a month one block is expected to take 75 to 80 mins.

- Only 36.79% of blocks are actually solved in exactly 10 mins.

- Every 4.7 years a block can take 2 hours to be solved.


* Somebody might want to double-check these numbers.

[fillippone]

I'd like to rephrase the answering my own very layman (and I think a little bit catchy) terms: The number of bitcoin blocks in a given time interval follows a Poisson process, hence the expected time to have a Bitcoin bock mined is, in every moment, exactly 10 minutes".

This is always true, be the last block been mined 1 second ago, or 2 hours ago.
Obviously I am not taking in consideration variations in hash rate. Hashate swings happens all the time and can vary the above answer considerably, but here I am referring to the average, theoretical case of bitcoin mined with unchanged projected difficulty due to constant hashrate.

[buwaytress]

Most layperson to respond here probably but for the sake of comprehending what matters, is it not enough for me as a regular user to understand that there is an average time of 10 mins per block every 14 days? And the variance in that is what counts into difficulty adjustment to keep ensuring that that the 14-day period is always 10 mins per block? Or is this timeframe even larger?

[o_e_l_e_o]

is it not enough for me as a regular user to understand that there is an average time of 10 mins per block every 14 days?

Essentially yes, although you can remove the words "every 14 days" from the end of that sentence. The targeted mean rate of bitcoin blocks is one every 10 minutes.

And the variance in that is what counts into difficulty adjustment to keep ensuring that that the 14-day period is always 10 mins per block?

The difficulty readjusts every 2016 blocks. At exactly 10 minutes per block this would be exactly 14 days, but in reality will be longer or shorter than 14 days if the mean rate is slower or faster respectively.

If the mean rate of blocks over the last 2016 blocks is faster than 1 per 10 minutes, then at the next difficulty readjustment the difficulty will increase, and the mean rate of block production will slow down. The converse is also true, and if the mean rate of block production over the last 2016 blocks is slower than 1 per 10 minutes, then at the next difficulty readjustment the difficultly will decrease, and the mean rate of block production will increase. The aim is always to bring the mean rate back to 1 per 10 minutes.

Note that when considering difficulty, it is all about considering the mean rate of production over that difficulty period. As pointed out above, block times of greater than an hour are not uncommon, and usually have nothing to do with the hashrate. A prolonged block time or two won't affect anything directly, but they will serve to increase the mean block time when it comes to the difficulty readjustment. However, short block times of only a few minutes or less are similarly not uncommon. All these outlier block times tend to negate each other. It is the average that matters.

[20kevin20]

Say the last mined block is Block no. 628,000. Are the miners:
 1. Trying to solve automatically and specifically the following "mathematical problem" aka no. 628,001 or
 2. Trying to simply solve a random one and whoever finds a solution counts as block no. 628,001?

My logical thinking says the first one is the correct answer. Anyways, here's why I was asking:

I thought for a minute about the idea of having a fixed block time of exactly 10 minutes which I suppose is achievable through coding and I was wondering: is there any advantage in having a precise 10min. block time?

From my point of view, if 1. from above applies, I have to assume that it'd mean all miners have to try solving a block and stop mining as soon as one of the miners sends a signal that they've successfully solved it. As soon as the 10 min timeframe passes, the block would be broadcasted.

If 2. from above applies, I guess it would be a bad idea: miners could solve more than only one block within the 10min timeframe and there would be a congestion in the block broadcasting as the code only allows one block to be published every exactly 10 minutes.

But would there be an advantage in having a precise block time or would it actually make things worse?

[LoyceV]

Say the last mined block is Block no. 628,000. Are the miners:
 1. Trying to solve automatically and specifically the following "mathematical problem" aka no. 628,001 or
 2. Trying to simply solve a random one and whoever finds a solution counts as block no. 628,001?

It's more or less #2: whoever finds the solution creates block 628,001.

I thought for a minute about the idea of having a fixed block time of exactly 10 minutes which I suppose is achievable through coding and I was wondering: is there any advantage in having a precise 10min. block time?

You could do that, in a centralized database. With Proof of Work, the next block is found at a random time. If all miners would find a block after exactly 10 minutes, who gets the block reward?

But would there be an advantage in having a precise block time or would it actually make things worse?

It wouldn't be Bitcoin

[fillippone]

If 2. from above applies, I guess it would be a bad idea: miners could solve more than only one block within the 10min timeframe and there would be a congestion in the block broadcasting as the code only allows one block to be published every exactly 10 minutes.

You cannot do that.
One of the input of problem 628,002 is a part of the solution of problem 628,001.
This is why it is called a blockchain.

When problem 628,001 the solution is propagated (a few seconds) to the rest of the network, and almost everyone sops working on that problem and start working on  the following block.
If there are too many solution found, then in the following epoch a difficulty adjustment will try to adjust the difficulty of those problems to be solved in the correct tie again.
You can have a burst of blocks found in a very rapid sequence, but on average the time is 10 minutes.

Hash power in bitcoin is so big, it's very difficult to observe difficulty swings (we just had a double digit diff adjustment, recently, but it was an exceptional one).
In lower hashrate shoitcoins (BCH, or BSV) you have hyge difficulty swings and the miners actually gamer difficulty adjustments to maximise their profits.

[BrewMaster]

Say the last mined block is Block no. 628,000. Are the miners:
 1. Trying to solve automatically and specifically the following "mathematical problem" aka no. 628,001 or
 2. Trying to simply solve a random one and whoever finds a solution counts as block no. 628,001?

It's more or less #2: whoever finds the solution creates block 628,001.

i don't think we can explain mining with either 1 or 2. it is somewhere in between. what miners "solve" is partly random and partly pre-defined and similar to each other.
for example in this case when 2 miners are competing for block 628,001 they both have same exact:
- previous block hash (32 bytes)
- target (4 bytes)
- same block height number (which has to be placed in coinbase scriptsig)
with block version and time also being nearly the same. so they are both hashing 80 bytes, 36 bytes of which is exactly the same and 44 bytes changing.

it should be the same for any other algorithm too since it is a blockchain and to be a chain there has to be a link to the previous one and we'll end up using a partially similar input.

[mikeywith]

I thought for a minute about the idea of having a fixed block time of exactly 10 minutes which I suppose is achievable through coding and I was wondering: is there any advantage in having a precise 10min. block time?

From my point of view, if 1. from above applies, I have to assume that it'd mean all miners have to try solving a block and stop mining as soon as one of the miners sends a signal that they've successfully solved it. As soon as the 10 min timeframe passes, the block would be broadcasted.

This minus the bolded part seems right, think of it this way, miners don't really just solve a "random block", they solve the next block randomly, every miner "assuming they are synced with others" have the last block's hash N and working towards N+1 in your example 628,001, the "solution" to that block is "random" but they are indeed are working on the same block height, however, once someone finds it they STOP mining that block and move forward to the next block N+1+1, but they don't WAIT 10 minutes, as soon as they find the solution they will propagate the block and start working again.

You should understand that there is no 10 min rule in the bitcoin code, nobody is stopping miners from finding 1000 blocks in 1000 seconds except statistics and probability, you can't tell for certain that miners won't solve that many blocks in such a short period of time, and if they do it, nobody will stop them, it's just a matter of CAN they do it, which is statistically difficult.

[fillippone]

I thought for a minute about the idea of having a fixed block time of exactly 10 minutes which I suppose is achievable through coding and I was wondering: is there any advantage in having a precise 10min. block time?

From my point of view, if 1. from above applies, I have to assume that it'd mean all miners have to try solving a block and stop mining as soon as one of the miners sends a signal that they've successfully solved it. As soon as the 10 min timeframe passes, the block would be broadcasted.

This minus the bolded part seems right, think of it this way, miners don't really just solve a "random block", they solve the next block randomly, every miner "assuming they are synced with others" have the last block's hash N and working towards N+1 in your example 628,001, the "solution" to that block is "random" but they are indeed are working on the same block height, however, once someone finds it they STOP mining that block and move forward to the next block N+1+1, but they don't WAIT 10 minutes, as soon as they find the solution they will propagate the block and start working again.

You should understand that there is no 10 min rule in the bitcoin code, nobody is stopping miners from finding 1000 blocks in 1000 seconds except statistics and probability, you can't tell for certain that miners won't solve that many blocks in such a short period of time, and if they do it, nobody will stop them, it's just a matter of CAN they do it, which is statistically difficult.

Also, it’s worth remembering not all miners are working on the same problem.
Actually every miner works on a new block that can, and probably will, be different from the one another miner is working on. Every miner has their own strategy to build the “best block”, and for sure there are variation between different miners about that.
So all the miner can start working on problem N+1+1 only after block N+1 has been broadcasted also because they don’t know which transaction are included in the first one (let’s ignore selfish mining or other practices, for sake of simplicity).

[pooya87]

So all the miner can start working on problem N+1+1 only after block N+1 has been broadcasted also because they don’t know which transaction are included in the first one (let’s ignore selfish mining or other practices, for sake of simplicity).

it is not about which transactions were mined already, if that were true then every miner would have mined empty blocks in the future as others were working on old ones. the only reason why everyone has to wait for N+1st block to be mined before they work on N+2nd is because the N+2nd block has to have a link to N+1 inside of it and it can't do that until the hash of N+1 is found.

[fillippone]

So all the miner can start working on problem N+1+1 only after block N+1 has been broadcasted also because they don’t know which transaction are included in the first one (let’s ignore selfish mining or other practices, for sake of simplicity).

it is not about which transactions were mined already, if that were true then every miner would have mined empty blocks in the future as others were working on old ones. the only reason why everyone has to wait for N+1st block to be mined before they work on N+2nd is because the N+2nd block has to have a link to N+1 inside of it and it can't do that until the hash of N+1 is found.

Of course.
I didn’t make myself clear.
In the end the hash of a block is function of the transaction included in the said block.
What I meant is each miner is working on a different problems, and different problems leads to different (valid)solutions.