Bitcoin Empty Blocks

[DougM]

That's the other puzzling thing, how is it possible that a block could be successfully mined within seconds when it should take on average 10 minutes?

This is an average time range, it could be faster or slower depending on some factors such as the one given by @ranochigo; luck, cause finding a hash is based on randomness. The difficulty level is adjusted approximately every 2 weeks or 2016 blocks to maintain the average block time of 10 minutes.

I did some quick checks based on the earliest blocks (0  to 119,964 contained in blk00000.dat just because I had them handy).  Comparing the time deltas based on sequential blocks from the beginning of (blockchain) time the delta ranges from -119 to  7,719 minutes, but sure enough the average is ~10 minutes!  That is surprising because the difficulty didn't change until the end of 2009, but the first dat file covers from Jan 03 2009 (Genesis block 0) to April 24 2011 (119,964) so I guess maybe it compensated later if the average was off initially?

Maybe so...I checked the metrics from blocks 1 to 38,091 (Feb 4, 2010) around the time of the first difficulty change from 1 to 2:  Min -65 mins, Max 7,719 min (FYI delta 0 to 1 was an anomaly next largest was 1509 min)  and the average was close to 15 minutes versus target 10 minutes. This means the average after that must have been less than 10 minutes for a while to compensate.

Interestingly the timestamps for follow on blocks can be earlier in time...that was a bit of surprise to me at first, but I learned I can't assume sequential blocks are in temporal order.  For example:

https://www.blockchain.com/btc/block/32649  -->  2010-01-02 04:49
https://www.blockchain.com/btc/block/32650  -->  2010-01-02 02:50

[1715217525]

1715217525

[1715217525]

1715217525

[1715217525]

1715217525

[barrysty1e]

When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.

perhaps if you use a commodore64..

doublesha of the raw transaction data per transaction in block,
store in list, then hash the list,
then compare this hash with the one provided in the blockheader,
then hash the blockheader and ensure its below target,
we're talking milliseconds here..

heres an example from dash (x11 is a bit more harsh than sha256d, only enforcing the point)..

Code:

2020-08-24 23:37:04 received: block (452 bytes) peer=0
2020-08-24 23:37:04 received block 4829e011bc551411c903f35b1f436f92c2a021d89e3e9b2a12be1ffd632cb736 peer=0
2020-08-24 23:37:04   - Load block from disk: 0.00ms [0.01s]
2020-08-24 23:37:04     - Sanity checks: 0.02ms [0.01s]
2020-08-24 23:37:04     - Fork checks: 0.03ms [0.01s]
2020-08-24 23:37:04       - Connect 2 transactions: 0.01ms (0.007ms/tx, 0.015ms/txin) [0.01s]
2020-08-24 23:37:04     - Verify 1 txins: 0.03ms (0.026ms/txin) [0.02s]
2020-08-24 23:37:04       - IS filter: 0.01ms [0.01s]
2020-08-24 23:37:04       - GetBlockSubsidy: 0.01ms [0.00s]
2020-08-24 23:37:04       - IsBlockValueValid: 0.02ms [0.01s]
2020-08-24 23:37:04       - IsBlockPayeeValid: 1.63ms [0.62s]
2020-08-24 23:37:04         - Loop: 0.01ms [0.07s]
2020-08-24 23:37:04         - quorumBlockProcessor: 0.04ms [0.41s]
2020-08-24 23:37:04         - deterministicMNManager: 3.06ms [1.14s]
2020-08-24 23:37:04           - GetTxPayload: 0.00ms [0.00s]
2020-08-24 23:37:04             - BuildNewListFromBlock: 1.40ms [0.56s]
2020-08-24 23:37:04             - CSimplifiedMNList: 3.39ms [1.19s]
2020-08-24 23:37:04             - CalcMerkleRoot: 3.05ms [1.08s]
2020-08-24 23:37:04           - CalcCbTxMerkleRootMNList: 8.81ms [3.20s]
2020-08-24 23:37:04             - GetMinedAndActiveCommitmentsUntilBlock: 0.19ms [0.07s]
2020-08-24 23:37:04             - GetMinedCommitment: 0.01ms [0.09s]
2020-08-24 23:37:04             - Loop: 0.01ms [0.01s]
2020-08-24 23:37:04             - ComputeMerkleRoot: 0.01ms [0.00s]
2020-08-24 23:37:04           - CalcCbTxMerkleRootQuorums: 0.24ms [0.18s]
2020-08-24 23:37:04         - CheckCbTxMerkleRoots: 9.09ms [3.39s]
2020-08-24 23:37:04       - ProcessSpecialTxsInBlock: 12.22ms [5.01s]
2020-08-24 23:37:04     - Index writing: 0.07ms [0.03s]
2020-08-24 23:37:04     - Callbacks: 0.01ms [0.00s]
2020-08-24 23:37:04   - Connect total: 14.15ms [5.76s]
2020-08-24 23:37:04   - Flush: 0.09ms [0.04s]
2020-08-24 23:37:04   - Writing chainstate: 0.01ms [0.00s]

[barrysty1e]

Interestingly the timestamps for follow on blocks can be earlier in time...that was a bit of surprise to me at first, but I learned I can't assume sequential blocks are in temporal order.  For example:

https://www.blockchain.com/btc/block/32649  -->  2010-01-02 04:49
https://www.blockchain.com/btc/block/32650  -->  2010-01-02 02:50

Still reigns true now (https://github.com/bitcoin/bitcoin/blob/0.20/src/chain.h#L18-L30), unsure of the exact rational behind it..

[CardCoins]

When it comes to validating a block, depending on the composition of transactions, it can take upwards of a few seconds (even more in some pathological/adversarial scenarios) to complete.

perhaps if you use a commodore64..

doublesha of the raw transaction data per transaction in block,
store in list, then hash the list,
then compare this hash with the one provided in the blockheader,
then hash the blockheader and ensure its below target,
we're talking milliseconds here..

heres an example from dash (x11 is a bit more harsh than sha256d, only enforcing the point)..

Code:

2020-08-24 23:37:04 received: block (452 bytes) peer=0
2020-08-24 23:37:04 received block 4829e011bc551411c903f35b1f436f92c2a021d89e3e9b2a12be1ffd632cb736 peer=0
2020-08-24 23:37:04   - Load block from disk: 0.00ms [0.01s]
2020-08-24 23:37:04     - Sanity checks: 0.02ms [0.01s]
2020-08-24 23:37:04     - Fork checks: 0.03ms [0.01s]
2020-08-24 23:37:04       - Connect 2 transactions: 0.01ms (0.007ms/tx, 0.015ms/txin) [0.01s]
2020-08-24 23:37:04     - Verify 1 txins: 0.03ms (0.026ms/txin) [0.02s]
2020-08-24 23:37:04       - IS filter: 0.01ms [0.01s]
2020-08-24 23:37:04       - GetBlockSubsidy: 0.01ms [0.00s]
2020-08-24 23:37:04       - IsBlockValueValid: 0.02ms [0.01s]
2020-08-24 23:37:04       - IsBlockPayeeValid: 1.63ms [0.62s]
2020-08-24 23:37:04         - Loop: 0.01ms [0.07s]
2020-08-24 23:37:04         - quorumBlockProcessor: 0.04ms [0.41s]
2020-08-24 23:37:04         - deterministicMNManager: 3.06ms [1.14s]
2020-08-24 23:37:04           - GetTxPayload: 0.00ms [0.00s]
2020-08-24 23:37:04             - BuildNewListFromBlock: 1.40ms [0.56s]
2020-08-24 23:37:04             - CSimplifiedMNList: 3.39ms [1.19s]
2020-08-24 23:37:04             - CalcMerkleRoot: 3.05ms [1.08s]
2020-08-24 23:37:04           - CalcCbTxMerkleRootMNList: 8.81ms [3.20s]
2020-08-24 23:37:04             - GetMinedAndActiveCommitmentsUntilBlock: 0.19ms [0.07s]
2020-08-24 23:37:04             - GetMinedCommitment: 0.01ms [0.09s]
2020-08-24 23:37:04             - Loop: 0.01ms [0.01s]
2020-08-24 23:37:04             - ComputeMerkleRoot: 0.01ms [0.00s]
2020-08-24 23:37:04           - CalcCbTxMerkleRootQuorums: 0.24ms [0.18s]
2020-08-24 23:37:04         - CheckCbTxMerkleRoots: 9.09ms [3.39s]
2020-08-24 23:37:04       - ProcessSpecialTxsInBlock: 12.22ms [5.01s]
2020-08-24 23:37:04     - Index writing: 0.07ms [0.03s]
2020-08-24 23:37:04     - Callbacks: 0.01ms [0.00s]
2020-08-24 23:37:04   - Connect total: 14.15ms [5.76s]
2020-08-24 23:37:04   - Flush: 0.09ms [0.04s]
2020-08-24 23:37:04   - Writing chainstate: 0.01ms [0.00s]

In perfect circumstances block validation is quite fast, that's correct!

There are adversarial cases where block validation can take quite a bit longer then the block you have presented here. In fact, one of the most significant improvements that segwit introduced was designed to partially address this exact issue, as the signature hashing algorithm was O(n^2). Now it is O(n): https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki#motivation

While segwit addresses one of these complexities, there are still others (albeit much less vicious): https://bitslog.com/2017/04/17/new-quadratic-delays-in-bitcoin-scripts/

Even if you don't assume the block is intentionally adversarial, there are lots of other factors which can effect validation time, including how much of the UTXO set the validator has cached, and how much of the block's content is in that cache. The libraries and hardware they use is also an important factor. Sipa has a great stackexchange post which details some of this stuff here, he suggests blocks can take multiple seconds to validate: https://bitcoin.stackexchange.com/questions/50349/how-long-does-block-validation-take/50407#50407

[Picard78]

There are 11 empty blocks in 966 blocks or 1.13% of the blocks are empty from Block 645,198 to 644,232 = 966.

645,198
645,098
645,090
645,027
644,937
644,928
644,644
644,543
644,498
644,394
644,232

Is it possible to download Block Header Data for analysis in a tool like Excel or MS Access?  I would like to analyze the average Block Size for the past 50,000 blocks.  Average Block Fee in satoshis for the past 100,000 blocks etc...   Also I am very curious how many Empty Blocks there are for the past 50,000 blocks.

Thank you
 

[Picard78]

Andreas Antonopoulos discusses the reason for the Empty Blocks in the below video from 0 - 3:15 min.  According to him "it takes about 15 - 20 seconds to validate" a block.  He says this at the start of the video.

https://www.youtube.com/watch?v=dizF2S63RXY

[NotFuzzyWarm]

One has to wonder about the date of that video.
Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.

[ranochigo]

One has to wonder about the date of that video.
Per-Kano it takes around 120-150ms to fully validate blocks. Being the operator of a pool that only uses fully validated blocks and has NEVER mined an empty one, he knows what it takes.

Indeed. I remember Kano being an opponent of SPV mining and some drama between him and F2Pool. BIP152 would've improved the propagation timing further and the effects of SPV mining would've diminished from the past.

I can't comment too much on how Kano and Antpool handles their mining code nor the efficacy of their code because I can't have access to either. He does have sound arguments against SPV mining and miners are mining less empty blocks in the years after that incident.

Blockchair has database dumps here but you need to parse it to have anything useful :https://blockchair.com/dumps

I just remembered,  LoyceV has everything: https://bitcointalk.org/index.php?topic=5246271.0

[NotFuzzyWarm]

Indeed. I remember Kano being an opponent of SPV mining and some drama between him and F2Pool.

As in f2pool and Antpool mining 4 empty blocks in a row in 2015(?) that were off-chain thanks to their jumping the gun to be 1st out the gate to start a new (un-validated) block by using results from their previous also empty and un-validated block?

Turns out that someone else had won the Orphan race for that 1st empty & un-validated block that Antpool & f2 had based their chain on.... Oops. Thanks to the back-end monitoring that Kano has he was very quickly alerted to the issue and contacted Antpool ops about it who then quickly relayed the info to f2pool who promptly switched tracks back to the valid current chain and of course invalidating the off-chain blocks they found.

Kano.is uses a modified version of ckpool as its front-end in conjunction with KDB which also directly monitors the network via bitcoind. Most of the code is out there in gits. Mind you his changes to ckpool are not there despite him being one of the major developers of it (the 'k' stands for Kano) thanks to a certain someone kicking him off the ckpool github due to irreconcilable differences in their views on proper coding/testing & pool financial safety. Anywho, the core code is there for anyone to peruse & use but Kano's mods to ckpool and updates to his KDB remain private. Since he does not distribute his changes that does not violate the GPL .

[Picard78]

I just remembered,  LoyceV has everything: https://bitcointalk.org/index.php?topic=5246271.0

Thank you ranochigo.  This is exactly what I was looking for.

Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks there were:

- 184 empty blocks which is ~ 0.4% of 45,343 blocks.

- Average block size of 1,116,072 bytes    

- Average block fees USD $3,145.  Average Block Coinbase reward USD $90,806.  Average profit from fees ~3.5% per block.

- Maximum USD $ / KB is $23 on 2020-02-01 and 2020-07-28.   (Note: Fees were over $100 / KB for many blocks in Dec 2017)

[Picard78]

Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks:

                   Blocks Mined  Percent   Empty Blocks
1   Unknown   8,448   18.63%     11
2   F2Pool   7,797   17.20%     56
3   BTC.com   5,765   12.71%     33
4   AntPool   5,078   11.20%     23
5   Poolin   4,475   9.87%        9
6   ViaBTC   2,818   6.21%        6
7   SlushPool   2,004   4.42%       10
8   Huobi   1,842   4.06%       14

Hmmm... this was formatted properly before I posted it.

[Picard78]

Since block 600,000 (Oct 19, 2019) or in the last 45,343 blocks:

The average block mining time is 9.892152041 min

Very nice.

[DarkDays]

It's perfectly normal and is usually a result of SPV mining.

When a block is discovered, time is taken for the block to propagate and for the miners to verify the entire block. This can take a few seconds and those time would be wasted if the ASICs and the miners were just to idle and wait for the pool to validate the block. Enter SPV mining, SPV mining takes the block header of the previous block and create a new block header without including any transactions so as to prevent any conflicting transactions.

After the block is fully validated, the pool goes ahead to do everything normally and compile the unconfirmed transactions from the mempool. These kinds of blocks are usually observed when the blocks are mined in quick succession.

But wouldn't this create many empty blocks eventually after the decades of transactions? I know right now the size limited by that single reward it gives out to the miners. But can't a sidechain be implemented for the rewards itself? I do count myself among those few members that think that empty blocks are indeed a waste of resources. All it would take to stop empty blocks from being mined is simply not allowing miners to solve a block's hash without downloading the entire content for that block. I think a simple line of code could even solve this. No?

I personally think the block reward should be stripped from empty or stuffed blocks. It's just causing inflation for now real benefit.

[pooya87]

But wouldn't this create many empty blocks eventually after the decades of transactions?

no, because we are talking about finding the hash within a ~second and the chances of that happening is very small that it has happened very rarely in the past 646k blocks.

All it would take to stop empty blocks from being mined is simply not allowing miners to solve a block's hash without downloading the entire content for that block. I think a simple line of code could even solve this. No?

it is not that simple and there is already a much better solution: incentive. when the price is high and block reward is low and the total fee is a meaningful amount of money, it means if the miner finds an empty block they lose all that extra income.

I personally think the block reward should be stripped from empty or stuffed blocks. It's just causing inflation for now real benefit.

the bitcoin design is that you get paid for the "work" you have done and you do pretty much the same amount of work for an empty block or for a full block so you get paid for it all the same, meaning the reward can not change.

[wanted sliter]

https://www.blockchain.com/btc/block/644927
https://www.blockchain.com/btc/block/644928
Observing two adjacent blocks 644927 and 644928, we can see them mined in the same minute. Block 644927 has 612 transactions and the next block 644928 has only 1 transaction.
Since the mining speed is too fast, transactions cannot be sent in this block, so only 1 transaction can be made.