Empty blocks

[kano]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

You don't have to "decode" it.
You just look for the string that the pool always puts there.
e.g. for kano.is it always contains "/Kano"
Of course that only works for pools that do that

[TheRealSteve]

When trying to find empty blocks, of course the transaction count is what matters.

If the data source is no good, the next best bet is the block reward.
If it's only 25BTC then it is very likely to be an empty block (I'd doubt there are few if any recent blocks that have only free transactions)

The block size doesn't help since there are 2 pools with very large coinbase transactions. p2pool is one of them.

Gotcha.  I've got a ticket open with Blocktrail, so hopefully I can just re-run that with their API later.  If not, I'm sure there's alternatives

[jonnybravo0311]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

You don't have to "decode" it.
You just look for the string that the pool always puts there.
e.g. for kano.is it always contains "/Kano"
Of course that only works for pools that do that

Maybe I'm missing something painfully obvious, but when I look at the output of getrawtransaction or decoderawtransaction, this is what I see:

Code:

{
"txid" : "661b8707036fddb8ea2c178a57988b4ed9cc9d9bc800d7aebe5dd58b813a9694",
"version" : 1,
"locktime" : 0,
"vin" : [
{
"coinbase" : "03117f051e4d696e656420627920416e74506f6f6c20626a35180f42fa2055776c0001bc2a0000bf480500",
"sequence" : 4294967295
}
],
"vout" : [
{
"value" : 25.26360849,
"n" : 0,
"scriptPubKey" : {
"asm" : "OP_DUP OP_HASH160 9524440a5b54cca9c46ef277c34739e9b521856d OP_EQUALVERIFY OP_CHECKSIG",
"hex" : "76a9149524440a5b54cca9c46ef277c34739e9b521856d88ac",
"reqSigs" : 1,
"type" : "pubkeyhash",
"addresses" : [
"1Ebb8NfVmKMoGuMJCAEbVMv2dX8GnzgxSa"
]
}
}
]
}

That "coinbase" value is what I need to decode.  This transaction was an empty block mined by AntPool (I know this because I looked it up on blockchain.info).  This is what it looks like there:

Code:

CoinBase
03117f051e4d696e656420627920416e74506f6f6c20626a35180f42fa2055776c0001bc2a0000bf480500 
(decoded) Mined by AntPool bj5B� Uwl
�*�H

So, how do I decode the coinbase hash to get back the string I can parse?

EDIT: I figured it out... it's just a hex to ascii conversion.  Duh.

[jonnybravo0311]

So I got a bit ambitious... and wrote some code to drop the information into a CSV.
Block number, date, decoded script (if available)

As of block 360324 there are 85298 empty blocks.

If anyone wants to look at it, I've uploaded it here: https://www.dropbox.com/s/tisp9mj6a369dds/empty_blocks.csv?dl=0

Strangely, I found a whole lot of coinbase transactions that the core API wouldn't parse (53,612 of them to be precise).  I have no idea why the core API wouldn't parse it, but here's an example (block 337104):

Code:

getblockhash 337104
00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691

Code:

getblock 00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691

{
"hash" : "00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691",
"confirmations" : 23233,
"size" : 256,
"height" : 337104,
"version" : 2,
"merkleroot" : "57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2",
"tx" : [
"57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2"
],
"time" : 1420197655,
"nonce" : 3357110901,
"bits" : "181b0dca",
"difficulty" : 40640955016.57649231,
"chainwork" : "00000000000000000000000000000000000000000003beb652b37a2f4b948166",
"previousblockhash" : "00000000000000001654d3aa9abac3e815391b9b5eca90e15767cc1ee6ce1dc2",
"nextblockhash" : "00000000000000001686943c32233916b9d9485230bf63fc0f9144c94be4c1af"
}

Code:

getrawtransaction 57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2 1
No information available about transaction (code -5)

Yet, if I look at that same transaction on blockchain.info I can see the block was mined by Discus Fish:
 �$七彩神仙鱼��mmvs��̮���M���,���w, z1FaW���f
Z�rMined by andy518038

Anyone have any idea?

[jonnybravo0311]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

[kano]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

My block detection code I run at home, not on the pool, in one of the many checks I do to find our blocks, I simply convert the ascii to hex and grep the coinbase from bitcoin (each new block that appears at home)
Obvious, but easer than having to deal with removing characters below ' ' or above '~' and the problems they can cause in scripts and output

blockchain has this (incomplete) list of coinbase strings they use:
https://github.com/blockchain/Blockchain-Known-Pools

I had one many years ago (last time I updated it was Sep-2012) ... but half the pools in it don't exist any more

[jonnybravo0311]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

My block detection code I run at home, not on the pool, in one of the many checks I do to find our blocks, I simply convert the ascii to hex and grep the coinbase from bitcoin (each new block that appears at home)
Obvious, but easer than having to deal with removing characters below ' ' or above '~' and the problems they can cause in scripts and output

blockchain has this (incomplete) list of coinbase strings they use:
https://github.com/blockchain/Blockchain-Known-Pools

I had one many years ago (last time I updated it was Sep-2012) ... but half the pools in it don't exist any more

I like that idea... no need to bother with the bad actors, just convert the ascii strings you're trying to grep from the coinbase into hex and compare that.  I was doing it the other way around and the random characters that don't convert nicely were playing havoc.  Thanks for the suggestion!

Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

[kano]

...
Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

I used to identify it last, due to it having so many coinbase addresses, and the 'other' pool that sometimes had many was already identified by name.
I'm pretty sure the last address in the p2pool coinbase that shows up as invalid on blockchain is also a way to identify p2pool.

[jonnybravo0311]

...
Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

I used to identify it last, due to it having so many coinbase addresses, and the 'other' pool that sometimes had many was already identified by name.
I'm pretty sure the last address in the p2pool coinbase that shows up as invalid on blockchain is also a way to identify p2pool.

Yeah... the last address in p2pool's coinbase transaction is always unable to be decoded.

Looks like I'll have to use something other than /P2SH/ since block 360476 has /P2SH/ in the coinbase and it's not a p2pool block.  Guess I'll do more checks in p2pool's case.

[TheRealSteve]

Gotcha.  I've got a ticket open with Blocktrail, so hopefully I can just re-run that with their API later.

They found the bug on their end and fixed it earlier today, so here's an updated table:


For the three biggest 'offenders', I did an additional check for block time deltas.  A very short time between blocks can mean that A. there were no transactions in the mempool to include, B. there were no attractive transactions to include, C. they included no transactions for other reasons.

The 245 second one for F2Pool was an outlier.  Still, given the great number of transactions existing in the mempool and being added all the time, 20-30 seconds for a coinbase-only block seems unnecessary.

Here's some other semi-relevant graphs...
AntPool and BTC China adjusting their max block size after the recent (and on-going) debate:

See also: http://www.reddit.com/r/Bitcoin/comments/39f59k/aftermath_of_may_29_stress_test_btc_china_antpool/

Discus Fish / F2Pool's possible strategy on 'empty blocks + fill with transactions later' (note 100KB lines):

[jonnybravo0311]

OK... I rebuilt everything and ran it again.  For any blocks that I couldn't identify the pool, I listed the payout address.

As of block 360520, there are a total of 85307 empty blocks.  The raw data is here: https://www.dropbox.com/s/tisp9mj6a369dds/empty_blocks.csv?dl=0

[kano]

Probably what needs to be added would be a running avg of txn per block to show when there were basically no blocks txns early on.
Edit: fixed typo

[jonnybravo0311]

Probably what needs to be added would be a running avg of txn per block to show when there were basically no blocks early on.

I didn't track every block, just the empty ones.  However tracking them all and including number of transactions per block is pretty easily accomplished.

[jonnybravo0311]

So I tweaked things a bit.  Now the file will contain every block, how many transactions were in the block, when the block was found and who found it (if I could discern that, otherwise it shows the payout address from the coinbase transaction).  It's currently executing and I'll upload the file when it's done.

EDIT:

It's done and uploaded.  360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

[kano]

Heh, now for another of those things you might not have known about bitcoin
The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.
So you will find blocks in the blockchain with DateTime out of order.
That's sometimes referred to as roll-n-time in pool/mining software.

[jonnybravo0311]

Heh, now for another of those things you might not have known about bitcoin
The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.
So you will find blocks in the blockchain with DateTime out of order.
That's sometimes referred to as roll-n-time in pool/mining software.

Actually that one I did know.  Unfortunately I don't have actual block times so I had to make due with what info was available in the block header

Just a cursory glance at the file reveals some pretty big "empty block" offenders... AntPool and f2pool are the leaders in that metric.

[TheRealSteve]

The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.

There's still some pretty egregious differences between the time in the block and what's probably the actual time.  The actual actual time can't be determined, but for my graph above I ended up using the time that blocktrail saw the block announced.  Much smaller deviations there.

Just a cursory glance at the file reveals some pretty big "empty block" offenders... AntPool and f2pool are the leaders in that metric.

Still fairly small, percentage-wise.  I think it was around 2% of all blocks in that last ~10,000 blocks.  2% more than I'd like to see, but not entirely horrible,

[kano]

The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.

There's still some pretty egregious differences between the time in the block and what's probably the actual time.  The actual actual time can't be determined, but for my graph above I ended up using the time that blocktrail saw the block announced.  Much smaller deviations there.
...

... and this directly relates to my issue with Empty Blocks.

If the pool is putting out work close to the actual actual time, then the case for putting out empty blocks would depend on, over a large sample, showing what that delay is: making a full block vs an empty block.
That delay should be very small - and thus invalidate doing it.
If it's not, then that's a software problem.

Next there is the issue of how do you decide to put out an empty block early?
If you decide based on the arrival of the block before it has been validated, then that means that the block you are switching to may not be valid.

However, there's something that no doubt no one has bothered to look at (I do mean Eligius clearly has never bothered to look at) and that is to verify something else to prove their theory about why they say it's OK to send out empty blocks.

If their pool is putting out empty blocks AFTER other pools are putting out full blocks, then they have no argument at all for doing it.

I've checked this. They do put out most of their block change information slower.
(Yeah that really does mean no one should be mining at Eligius)

[jonnybravo0311]

Tweaked the code a bit more to run some stats.  Ran it again today.  The vast majority of the empty blocks are from the beginning, which is to be expected.  Here's the data broken down by pool:

There's a bit of inaccuracy in it, since I know that -ck has recorded 67 blocks found in his solo pool, but I only detected 54... not sure if the coinbase wasn't always signed as it is today, or what the root cause of the discrepancy is.

Anyway, from the data and ignoring the unknowns, AntPool is by far the worst offender with 4.62% empty blocks.  Next up is KnC with 2.91% and they are virtually tied with discus fish at 2.9%.  That's what I've got.  Not sure how useful it is, but I had fun putting it together

[TheRealSteve]

Well, technically, the worst offender in your list is 'unknown'   Earlier you said that where you couldn't determine the pool, you used the coinbase address (presumably if only one found) instead.  Have you tried separating those out? That's how I got the 1KTNE and 1Ar2gR ones in my table.. blocktrail doesn't separate those out (with a few exceptions).