Empty blocks

[jonnybravo0311]

There's been quite a bit of discussion on this topic throughout the forum.  Some feel empty blocks are valuable.  Others feel they are of no value.

I thought it might be fun to write a small program to see just how many of these blocks exist in the blockchain.  For those who don't know, an empty block is a block that is mined containing only the coinbase transaction that awards the new coins.  To be honest, I was quite frankly surprised by the number there were, especially since people seemed to think they were rare occurrences.  Sure, I expected there to be quite a few at the beginning of the blockchain when BTC was pretty much an unknown and only a select few were mining it.  However, I was not expecting there to be so many empty blocks as there are.

As of block 360189 there are 85295 empty blocks on the chain.  For the math challenged, 23.68% of all blocks are empty.  8 of the last 189 blocks are empty.  25 of the last 1000 are empty.

Maybe I'll get ambitious and write the output to a spreadsheet so I can get some good data points out of this to see trends.

Where do you stand on this debate?

EDIT: I've uploaded the data from my last run.

https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0.  Has blocks up to 364144

Archive contains two files:

empty_blocks.csv has the raw data in the following format:

Block Height, NumTx, BlockTime (UTC), BlockSize (Bytes), WhoMined (if I could figure it out, BTC address of coinbase transaction if I couldn't, or unknown if no BTC address could be deciphered from coinbase transaction).

stats.csv has data on pools, how many blocks they've mined, how many were empty and percentage of empty to total.

Enjoy.

[chalkboard17]

Empty blocks aren't useless as they confirm previous transactions and secure them further against attacks. Though, they are much less useful than normal blocks. I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners. 25 out of 1000 last which makes 2.5% seem much more right to work on.
I think miners shouldn't mine empty blocks because they would lose tx fee and make people await more time for their tx to be included in the chain. Why do some pools do that? Is there an advantage?

[os2sam]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

[TheRealSteve]

Why do some pools do that? Is there an advantage?

Time.  Get the details of the last block mined, immediately start mining for the next - at this point 'empty' - block (and your chance at the block subsidy).  Then while things are busily mining away, build new headers that do include transactions, and at the next opportune moment start mining on that. It's just a small sliver of time, but against an otherwise equal opponent, collecting transactions by default would be a losing strategy in the long run.

[jonnybravo0311]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners.

I'm sure the earlier blocks had far more empties.  Like I wrote, maybe I'll get ambitious and plot this in a spreadsheet.  I mostly wrote the code for fun just to see how many blocks are empty.

Why do some pools do that? Is there an advantage?

Time.  Get the details of the last block mined, immediately start mining for the next - at this point 'empty' - block (and your chance at the block subsidy).  Then while things are busily mining away, build new headers that do include transactions, and at the next opportune moment start mining on that. It's just a small sliver of time, but against an otherwise equal opponent, collecting transactions by default would be a losing strategy in the long run.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

[alani123]

Weren't 0 transaction blocks dominating the testnet recently because of a pool? Other than that, here are some stats by blockr

https://btc.blockr.io/trivia/block

Blocks with 1 transaction: 85294  Blocks with 2 transactions: 12438

Edit: 1 transaction in a block essentially means that it's empty.

[jonnybravo0311]

Weren't 0 transaction blocks dominating the testnet recently because of a pool? Other than that, here are some stats by blockr

https://btc.blockr.io/trivia/block

Blocks with 1 transaction: 85294  Blocks with 2 transactions: 12438

Edit: 1 transaction in a block essentially means that it's empty.

That's a pretty cool site... didn't know it existed, or I wouldn't have bothered writing my own code to get the results .  At least we both match up (I included the genesis block whereas they didn't).

[TheRealSteve]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:
Please see update post

Note: This post previously contained an image based on incorrect data.  See: https://bitcointalk.org/index.php?topic=1085800.msg11578412#msg11578412

The old image, for reference, can be found at: https://i.imgur.com/RlhKM97.png

Comments below up to kano's post identifying a problem are regarding this old image.

Why do some pools do that? Is there an advantage?

Time.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

Disagreement is healthy

[os2sam]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:

What is the "dataset"?

Doesn't look like anyone is purposely mining empty blocks on a regular basis.  So, that's a good thing.
Thanks,
Sam

[TheRealSteve]

What is the "dataset"?

Last 10,197 blocks (at the time I ran it), data sourced from blocktrail.com .  They have a paginated API (200 results per page, so 51 requests) that at least has info on the block height, timestamp, number of transactions, size of the block, apparent pool that mined it and some other info.

[achow101]

Can we see the code that creates this?

[TheRealSteve]

Can we see the code that creates this?

I could give you the pseudocode for the poller - all the rest is just LibreOffice Base + Calc bits - but that's not all that interesting either.  Basically, take this URL: https://api.blocktrail.com/v1/btc/all-blocks?sort_dir=desc&limit=200&api_key=YOUR_API_KEY_HERE&page=, concatenate with a page number that increments until either you find the page number of interest (or in my case, the block of interest).  Download the concatenated URL.  Parse result with whatever JSON library you're happy with.  I only read out height, is_orphan, block_time and miningpool_slug originally, added transactions and byte_size for this thread.
They offer libraries for Python, PHP and NodeJS, so.. pretty easy to get started with.

You'll need an API key, of course, but that's as easy as signing up, going to the relevant page, and off you go.
( Do keep quota and throttles in mind - they're pretty lax right now, but be a good API citizen and all that. This includes not sharing the actual polled data, as per their T&C. )

[Mikestang]

There's been quite a bit of discussion on this topic throughout the forum.  Some feel empty blocks are valuable.  Others feel they are of no value.

I thought it might be fun to write a small program to see just how many of these blocks exist in the blockchain.  For those who don't know, an empty block is a block that is mined containing only the coinbase transaction that awards the new coins.  To be honest, I was quite frankly surprised by the number there were, especially since people seemed to think they were rare occurrences.  Sure, I expected there to be quite a few at the beginning of the blockchain when BTC was pretty much an unknown and only a select few were mining it.  However, I was not expecting there to be so many empty blocks as there are.

As of block 360189 there are 85295 empty blocks on the chain.  For the math challenged, 23.68% of all blocks are empty.  8 of the last 189 blocks are empty.  25 of the last 1000 are empty.

Maybe I'll get ambitious and write the output to a spreadsheet so I can get some good data points out of this to see trends.

Where do you stand on this debate?

Thank you for the research, the numbers are enlightening.  I tend to fall on the side of the discussion that holds that blocks should contain transactions, since that is their ultimate purpose.

[kano]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:
https://i.imgur.com/RlhKM97.png

Why do some pools do that? Is there an advantage?

Time.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

Disagreement is healthy

Your calculations must be wrong.
None of the blocks on my pool kano.is have 0 non-coinbase transactions.
One recently only had about 10 since it was a few seconds after another one on the internet.

[TheRealSteve]

Thank you for the research, the numbers are enlightening.  I tend to fall on the side of the discussion that holds that blocks should contain transactions, since that is their ultimate purpose.

Oh I absolutely agree with that.  I thought you were referring to disagreement on whether or not the empty-block-first mining strategy was actually advantageous.  I seem to recall some reddit chatter about the impact being negligible compared to other factor and minor debate about whether there was in fact an actual advantage

[TheRealSteve]

Your calculations must be wrong.
None of the blocks on my pool kano.is have 0 non-coinbase transactions.
One recently only had about 10 since it was a few seconds after another one on the internet.

Probably - let me check what it's doing.

[TheRealSteve]

Probably - let me check what it's doing.

Stared myself blind on the code, couldn't find any issues.  Turns out it's in the API.  E.g. "height":360206,"block_time":"2015-06-09T22:17:46+0000",[...],"byte_size":230144,"confirmations":7,"transactions":1
That's not right: https://www.blocktrail.com/BTC/block/00000000000000000f757987312e482d3edd6a019d9eea5eb62e0b4a13e51e05

I'll drop them a line about this later.  In the mean time, it looks like the byte_size is correct, so I'll redo the stats image.

[jonnybravo0311]

OK... this has probably been answered before, so forgive me for asking it.  Where do I find out which pool mined a block?  I can't see anything useful in the output of the core API, and the "relayed_by" value provided by the blockchain.info API doesn't appear to be what I'm looking for, either.  I figure it's probably in the output script for the coinbase transaction.  Can somebody give me a quick rundown of how to decode that (other than passing it to bitcoin-cli)... like some pseudocode, or a link with some info?

For example, looking at the block TheRealSteve used in his last response, I can clearly see that on blockchain.info it shows it was submitted by Eligius.  Looking at the coinbase output script it shows a bunch of garbled crap with Eligius buried in it.  Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

[TheRealSteve]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

[kano]

Just looked up kano.is smallest block:

29 transactions.
https://blockchain.info/block-index/878311/00000000000000000880afbc23f0176372837c3edc9251763a4268edac280299

---

When trying to find empty blocks, of course the transaction count is what matters.

If the data source is no good, the next best bet is the block reward.
If it's only 25BTC then it is very likely to be an empty block (I'd doubt there are few if any recent blocks that have only free transactions)

The block size doesn't help since there are 2 pools with very large coinbase transactions. p2pool is one of them.