Empty blocks

[jonnybravo0311]

There's been quite a bit of discussion on this topic throughout the forum.  Some feel empty blocks are valuable.  Others feel they are of no value.

I thought it might be fun to write a small program to see just how many of these blocks exist in the blockchain.  For those who don't know, an empty block is a block that is mined containing only the coinbase transaction that awards the new coins.  To be honest, I was quite frankly surprised by the number there were, especially since people seemed to think they were rare occurrences.  Sure, I expected there to be quite a few at the beginning of the blockchain when BTC was pretty much an unknown and only a select few were mining it.  However, I was not expecting there to be so many empty blocks as there are.

As of block 360189 there are 85295 empty blocks on the chain.  For the math challenged, 23.68% of all blocks are empty.  8 of the last 189 blocks are empty.  25 of the last 1000 are empty.

Maybe I'll get ambitious and write the output to a spreadsheet so I can get some good data points out of this to see trends.

Where do you stand on this debate?

EDIT: I've uploaded the data from my last run.

https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0.  Has blocks up to 364144

Archive contains two files:

empty_blocks.csv has the raw data in the following format:

Block Height, NumTx, BlockTime (UTC), BlockSize (Bytes), WhoMined (if I could figure it out, BTC address of coinbase transaction if I couldn't, or unknown if no BTC address could be deciphered from coinbase transaction).

stats.csv has data on pools, how many blocks they've mined, how many were empty and percentage of empty to total.

Enjoy.

[1715201569]

1715201569

[1715201569]

1715201569

[chalkboard17]

Empty blocks aren't useless as they confirm previous transactions and secure them further against attacks. Though, they are much less useful than normal blocks. I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners. 25 out of 1000 last which makes 2.5% seem much more right to work on.
I think miners shouldn't mine empty blocks because they would lose tx fee and make people await more time for their tx to be included in the chain. Why do some pools do that? Is there an advantage?

[os2sam]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

[TheRealSteve]

Why do some pools do that? Is there an advantage?

Time.  Get the details of the last block mined, immediately start mining for the next - at this point 'empty' - block (and your chance at the block subsidy).  Then while things are busily mining away, build new headers that do include transactions, and at the next opportune moment start mining on that. It's just a small sliver of time, but against an otherwise equal opponent, collecting transactions by default would be a losing strategy in the long run.

[jonnybravo0311]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners.

I'm sure the earlier blocks had far more empties.  Like I wrote, maybe I'll get ambitious and plot this in a spreadsheet.  I mostly wrote the code for fun just to see how many blocks are empty.

Why do some pools do that? Is there an advantage?

Time.  Get the details of the last block mined, immediately start mining for the next - at this point 'empty' - block (and your chance at the block subsidy).  Then while things are busily mining away, build new headers that do include transactions, and at the next opportune moment start mining on that. It's just a small sliver of time, but against an otherwise equal opponent, collecting transactions by default would be a losing strategy in the long run.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

[alani123]

Weren't 0 transaction blocks dominating the testnet recently because of a pool? Other than that, here are some stats by blockr

https://btc.blockr.io/trivia/block

Blocks with 1 transaction: 85294  Blocks with 2 transactions: 12438

Edit: 1 transaction in a block essentially means that it's empty.

[jonnybravo0311]

Weren't 0 transaction blocks dominating the testnet recently because of a pool? Other than that, here are some stats by blockr

https://btc.blockr.io/trivia/block

Blocks with 1 transaction: 85294  Blocks with 2 transactions: 12438

Edit: 1 transaction in a block essentially means that it's empty.

That's a pretty cool site... didn't know it existed, or I wouldn't have bothered writing my own code to get the results .  At least we both match up (I included the genesis block whereas they didn't).

[TheRealSteve]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:
Please see update post

Note: This post previously contained an image based on incorrect data.  See: https://bitcointalk.org/index.php?topic=1085800.msg11578412#msg11578412

The old image, for reference, can be found at: https://i.imgur.com/RlhKM97.png

Comments below up to kano's post identifying a problem are regarding this old image.

Why do some pools do that? Is there an advantage?

Time.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

Disagreement is healthy

[os2sam]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:

What is the "dataset"?

Doesn't look like anyone is purposely mining empty blocks on a regular basis.  So, that's a good thing.
Thanks,
Sam

[TheRealSteve]

What is the "dataset"?

Last 10,197 blocks (at the time I ran it), data sourced from blocktrail.com .  They have a paginated API (200 results per page, so 51 requests) that at least has info on the block height, timestamp, number of transactions, size of the block, apparent pool that mined it and some other info.

[achow101]

Can we see the code that creates this?

[TheRealSteve]

Can we see the code that creates this?

I could give you the pseudocode for the poller - all the rest is just LibreOffice Base + Calc bits - but that's not all that interesting either.  Basically, take this URL: https://api.blocktrail.com/v1/btc/all-blocks?sort_dir=desc&limit=200&api_key=YOUR_API_KEY_HERE&page=, concatenate with a page number that increments until either you find the page number of interest (or in my case, the block of interest).  Download the concatenated URL.  Parse result with whatever JSON library you're happy with.  I only read out height, is_orphan, block_time and miningpool_slug originally, added transactions and byte_size for this thread.
They offer libraries for Python, PHP and NodeJS, so.. pretty easy to get started with.

You'll need an API key, of course, but that's as easy as signing up, going to the relevant page, and off you go.
( Do keep quota and throttles in mind - they're pretty lax right now, but be a good API citizen and all that. This includes not sharing the actual polled data, as per their T&C. )

[Mikestang]

There's been quite a bit of discussion on this topic throughout the forum.  Some feel empty blocks are valuable.  Others feel they are of no value.

I thought it might be fun to write a small program to see just how many of these blocks exist in the blockchain.  For those who don't know, an empty block is a block that is mined containing only the coinbase transaction that awards the new coins.  To be honest, I was quite frankly surprised by the number there were, especially since people seemed to think they were rare occurrences.  Sure, I expected there to be quite a few at the beginning of the blockchain when BTC was pretty much an unknown and only a select few were mining it.  However, I was not expecting there to be so many empty blocks as there are.

As of block 360189 there are 85295 empty blocks on the chain.  For the math challenged, 23.68% of all blocks are empty.  8 of the last 189 blocks are empty.  25 of the last 1000 are empty.

Maybe I'll get ambitious and write the output to a spreadsheet so I can get some good data points out of this to see trends.

Where do you stand on this debate?

Thank you for the research, the numbers are enlightening.  I tend to fall on the side of the discussion that holds that blocks should contain transactions, since that is their ultimate purpose.

[kano]

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?

Not from the data I pulled since the core API doesn't provide it - I ran this against my local node.  I don't know how happy blockchain.info would be if I pummeled them with 86k requests .

I ran my parser against a dataset - wasn't really designed for this - but here's what I've got:
https://i.imgur.com/RlhKM97.png

Why do some pools do that? Is there an advantage?

Time.

This is where there's plenty of disagreement between pool software writers.  On one side of the fence you've got luke-jr and wizkid who support the empty block theory.  On the other you've got ck/kano who do not support it at all.

Disagreement is healthy

Your calculations must be wrong.
None of the blocks on my pool kano.is have 0 non-coinbase transactions.
One recently only had about 10 since it was a few seconds after another one on the internet.

[TheRealSteve]

Thank you for the research, the numbers are enlightening.  I tend to fall on the side of the discussion that holds that blocks should contain transactions, since that is their ultimate purpose.

Oh I absolutely agree with that.  I thought you were referring to disagreement on whether or not the empty-block-first mining strategy was actually advantageous.  I seem to recall some reddit chatter about the impact being negligible compared to other factor and minor debate about whether there was in fact an actual advantage

[TheRealSteve]

Your calculations must be wrong.
None of the blocks on my pool kano.is have 0 non-coinbase transactions.
One recently only had about 10 since it was a few seconds after another one on the internet.

Probably - let me check what it's doing.

[TheRealSteve]

Probably - let me check what it's doing.

Stared myself blind on the code, couldn't find any issues.  Turns out it's in the API.  E.g. "height":360206,"block_time":"2015-06-09T22:17:46+0000",[...],"byte_size":230144,"confirmations":7,"transactions":1
That's not right: https://www.blocktrail.com/BTC/block/00000000000000000f757987312e482d3edd6a019d9eea5eb62e0b4a13e51e05

I'll drop them a line about this later.  In the mean time, it looks like the byte_size is correct, so I'll redo the stats image.

[jonnybravo0311]

OK... this has probably been answered before, so forgive me for asking it.  Where do I find out which pool mined a block?  I can't see anything useful in the output of the core API, and the "relayed_by" value provided by the blockchain.info API doesn't appear to be what I'm looking for, either.  I figure it's probably in the output script for the coinbase transaction.  Can somebody give me a quick rundown of how to decode that (other than passing it to bitcoin-cli)... like some pseudocode, or a link with some info?

For example, looking at the block TheRealSteve used in his last response, I can clearly see that on blockchain.info it shows it was submitted by Eligius.  Looking at the coinbase output script it shows a bunch of garbled crap with Eligius buried in it.  Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

[TheRealSteve]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

[kano]

Just looked up kano.is smallest block:

29 transactions.
https://blockchain.info/block-index/878311/00000000000000000880afbc23f0176372837c3edc9251763a4268edac280299

---

When trying to find empty blocks, of course the transaction count is what matters.

If the data source is no good, the next best bet is the block reward.
If it's only 25BTC then it is very likely to be an empty block (I'd doubt there are few if any recent blocks that have only free transactions)

The block size doesn't help since there are 2 pools with very large coinbase transactions. p2pool is one of them.

[kano]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

You don't have to "decode" it.
You just look for the string that the pool always puts there.
e.g. for kano.is it always contains "/Kano"
Of course that only works for pools that do that

[TheRealSteve]

When trying to find empty blocks, of course the transaction count is what matters.

If the data source is no good, the next best bet is the block reward.
If it's only 25BTC then it is very likely to be an empty block (I'd doubt there are few if any recent blocks that have only free transactions)

The block size doesn't help since there are 2 pools with very large coinbase transactions. p2pool is one of them.

Gotcha.  I've got a ticket open with Blocktrail, so hopefully I can just re-run that with their API later.  If not, I'm sure there's alternatives

[jonnybravo0311]

Is there a nice way to extract the info from there, or is it just a messy thing like, "if decoded script contains eligius, then the block was found by Eligius.  If decoded script contains antpool, then it was found by antpool, etc"?

It's a messy thing.  There's actually a write-up about it here: Don't believe everything you read on blockchain.info about block sources

Blocktrail also does the messy thing, as does whomined.com .  You can ready whomined's approach here: http://whomined.com/#how-it-works.

I'd imagine there's libraries that'll decode the scriptSig without going through Bitcoin Core.

You don't have to "decode" it.
You just look for the string that the pool always puts there.
e.g. for kano.is it always contains "/Kano"
Of course that only works for pools that do that

Maybe I'm missing something painfully obvious, but when I look at the output of getrawtransaction or decoderawtransaction, this is what I see:

Code:

{
"txid" : "661b8707036fddb8ea2c178a57988b4ed9cc9d9bc800d7aebe5dd58b813a9694",
"version" : 1,
"locktime" : 0,
"vin" : [
{
"coinbase" : "03117f051e4d696e656420627920416e74506f6f6c20626a35180f42fa2055776c0001bc2a0000bf480500",
"sequence" : 4294967295
}
],
"vout" : [
{
"value" : 25.26360849,
"n" : 0,
"scriptPubKey" : {
"asm" : "OP_DUP OP_HASH160 9524440a5b54cca9c46ef277c34739e9b521856d OP_EQUALVERIFY OP_CHECKSIG",
"hex" : "76a9149524440a5b54cca9c46ef277c34739e9b521856d88ac",
"reqSigs" : 1,
"type" : "pubkeyhash",
"addresses" : [
"1Ebb8NfVmKMoGuMJCAEbVMv2dX8GnzgxSa"
]
}
}
]
}

That "coinbase" value is what I need to decode.  This transaction was an empty block mined by AntPool (I know this because I looked it up on blockchain.info).  This is what it looks like there:

Code:

CoinBase
03117f051e4d696e656420627920416e74506f6f6c20626a35180f42fa2055776c0001bc2a0000bf480500 
(decoded) Mined by AntPool bj5B� Uwl
�*�H

So, how do I decode the coinbase hash to get back the string I can parse?

EDIT: I figured it out... it's just a hex to ascii conversion.  Duh.

[jonnybravo0311]

So I got a bit ambitious... and wrote some code to drop the information into a CSV.
Block number, date, decoded script (if available)

As of block 360324 there are 85298 empty blocks.

If anyone wants to look at it, I've uploaded it here: https://www.dropbox.com/s/tisp9mj6a369dds/empty_blocks.csv?dl=0

Strangely, I found a whole lot of coinbase transactions that the core API wouldn't parse (53,612 of them to be precise).  I have no idea why the core API wouldn't parse it, but here's an example (block 337104):

Code:

getblockhash 337104
00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691

Code:

getblock 00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691

{
"hash" : "00000000000000000cc790ec61cd20d394f9d0f0c3e34adb1005d2d909b3f691",
"confirmations" : 23233,
"size" : 256,
"height" : 337104,
"version" : 2,
"merkleroot" : "57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2",
"tx" : [
"57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2"
],
"time" : 1420197655,
"nonce" : 3357110901,
"bits" : "181b0dca",
"difficulty" : 40640955016.57649231,
"chainwork" : "00000000000000000000000000000000000000000003beb652b37a2f4b948166",
"previousblockhash" : "00000000000000001654d3aa9abac3e815391b9b5eca90e15767cc1ee6ce1dc2",
"nextblockhash" : "00000000000000001686943c32233916b9d9485230bf63fc0f9144c94be4c1af"
}

Code:

getrawtransaction 57224de458d9356a471099c5ab9f5e68f395ec5f2822915bdf3c6b296c1428e2 1
No information available about transaction (code -5)

Yet, if I look at that same transaction on blockchain.info I can see the block was mined by Discus Fish:
 �$七彩神仙鱼��mmvs��̮���M���,���w, z1FaW���f
Z�rMined by andy518038

Anyone have any idea?

[jonnybravo0311]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

[kano]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

My block detection code I run at home, not on the pool, in one of the many checks I do to find our blocks, I simply convert the ascii to hex and grep the coinbase from bitcoin (each new block that appears at home)
Obvious, but easer than having to deal with removing characters below ' ' or above '~' and the problems they can cause in scripts and output

blockchain has this (incomplete) list of coinbase strings they use:
https://github.com/blockchain/Blockchain-Known-Pools

I had one many years ago (last time I updated it was Sep-2012) ... but half the pools in it don't exist any more

[jonnybravo0311]

I got my answer from grue:

you need -txindex to view transactions that don't belong in your wallet.

So, I'm currently rebuilding the block index using txindex and once that's done, I'll rerun the code so I'll have all of the data.

My block detection code I run at home, not on the pool, in one of the many checks I do to find our blocks, I simply convert the ascii to hex and grep the coinbase from bitcoin (each new block that appears at home)
Obvious, but easer than having to deal with removing characters below ' ' or above '~' and the problems they can cause in scripts and output

blockchain has this (incomplete) list of coinbase strings they use:
https://github.com/blockchain/Blockchain-Known-Pools

I had one many years ago (last time I updated it was Sep-2012) ... but half the pools in it don't exist any more

I like that idea... no need to bother with the bad actors, just convert the ascii strings you're trying to grep from the coinbase into hex and compare that.  I was doing it the other way around and the random characters that don't convert nicely were playing havoc.  Thanks for the suggestion!

Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

[kano]

...
Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

I used to identify it last, due to it having so many coinbase addresses, and the 'other' pool that sometimes had many was already identified by name.
I'm pretty sure the last address in the p2pool coinbase that shows up as invalid on blockchain is also a way to identify p2pool.

[jonnybravo0311]

...
Now for a stupid question... and I only say it's stupid because I've been mining on p2pool for well over a year and don't know the answer myself ... is p2pool the only pool to use "/P2SH/" in the coinbase?  I swear I saw that in the coinbase of another... maybe I'm wrong and that really is the p2pool identifier.  Guess I'll find out when I run my code and do some checks.

I used to identify it last, due to it having so many coinbase addresses, and the 'other' pool that sometimes had many was already identified by name.
I'm pretty sure the last address in the p2pool coinbase that shows up as invalid on blockchain is also a way to identify p2pool.

Yeah... the last address in p2pool's coinbase transaction is always unable to be decoded.

Looks like I'll have to use something other than /P2SH/ since block 360476 has /P2SH/ in the coinbase and it's not a p2pool block.  Guess I'll do more checks in p2pool's case.

[TheRealSteve]

Gotcha.  I've got a ticket open with Blocktrail, so hopefully I can just re-run that with their API later.

They found the bug on their end and fixed it earlier today, so here's an updated table:


For the three biggest 'offenders', I did an additional check for block time deltas.  A very short time between blocks can mean that A. there were no transactions in the mempool to include, B. there were no attractive transactions to include, C. they included no transactions for other reasons.

The 245 second one for F2Pool was an outlier.  Still, given the great number of transactions existing in the mempool and being added all the time, 20-30 seconds for a coinbase-only block seems unnecessary.

Here's some other semi-relevant graphs...
AntPool and BTC China adjusting their max block size after the recent (and on-going) debate:

See also: http://www.reddit.com/r/Bitcoin/comments/39f59k/aftermath_of_may_29_stress_test_btc_china_antpool/

Discus Fish / F2Pool's possible strategy on 'empty blocks + fill with transactions later' (note 100KB lines):

[jonnybravo0311]

OK... I rebuilt everything and ran it again.  For any blocks that I couldn't identify the pool, I listed the payout address.

As of block 360520, there are a total of 85307 empty blocks.  The raw data is here: https://www.dropbox.com/s/tisp9mj6a369dds/empty_blocks.csv?dl=0

[kano]

Probably what needs to be added would be a running avg of txn per block to show when there were basically no blocks txns early on.
Edit: fixed typo

[jonnybravo0311]

Probably what needs to be added would be a running avg of txn per block to show when there were basically no blocks early on.

I didn't track every block, just the empty ones.  However tracking them all and including number of transactions per block is pretty easily accomplished.

[jonnybravo0311]

So I tweaked things a bit.  Now the file will contain every block, how many transactions were in the block, when the block was found and who found it (if I could discern that, otherwise it shows the payout address from the coinbase transaction).  It's currently executing and I'll upload the file when it's done.

EDIT:

It's done and uploaded.  360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

[kano]

Heh, now for another of those things you might not have known about bitcoin
The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.
So you will find blocks in the blockchain with DateTime out of order.
That's sometimes referred to as roll-n-time in pool/mining software.

[jonnybravo0311]

Heh, now for another of those things you might not have known about bitcoin
The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.
So you will find blocks in the blockchain with DateTime out of order.
That's sometimes referred to as roll-n-time in pool/mining software.

Actually that one I did know.  Unfortunately I don't have actual block times so I had to make due with what info was available in the block header

Just a cursory glance at the file reveals some pretty big "empty block" offenders... AntPool and f2pool are the leaders in that metric.

[TheRealSteve]

The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.

There's still some pretty egregious differences between the time in the block and what's probably the actual time.  The actual actual time can't be determined, but for my graph above I ended up using the time that blocktrail saw the block announced.  Much smaller deviations there.

Just a cursory glance at the file reveals some pretty big "empty block" offenders... AntPool and f2pool are the leaders in that metric.

Still fairly small, percentage-wise.  I think it was around 2% of all blocks in that last ~10,000 blocks.  2% more than I'd like to see, but not entirely horrible,

[kano]

The DateTime in the block header is not when the block was found.
It's close, but, far from exact. It's within about 1.5 hours.
Some pools used to push this number a long way ahead - close to 1/1.5 hours.

There's still some pretty egregious differences between the time in the block and what's probably the actual time.  The actual actual time can't be determined, but for my graph above I ended up using the time that blocktrail saw the block announced.  Much smaller deviations there.
...

... and this directly relates to my issue with Empty Blocks.

If the pool is putting out work close to the actual actual time, then the case for putting out empty blocks would depend on, over a large sample, showing what that delay is: making a full block vs an empty block.
That delay should be very small - and thus invalidate doing it.
If it's not, then that's a software problem.

Next there is the issue of how do you decide to put out an empty block early?
If you decide based on the arrival of the block before it has been validated, then that means that the block you are switching to may not be valid.

However, there's something that no doubt no one has bothered to look at (I do mean Eligius clearly has never bothered to look at) and that is to verify something else to prove their theory about why they say it's OK to send out empty blocks.

If their pool is putting out empty blocks AFTER other pools are putting out full blocks, then they have no argument at all for doing it.

I've checked this. They do put out most of their block change information slower.
(Yeah that really does mean no one should be mining at Eligius)

[jonnybravo0311]

Tweaked the code a bit more to run some stats.  Ran it again today.  The vast majority of the empty blocks are from the beginning, which is to be expected.  Here's the data broken down by pool:

There's a bit of inaccuracy in it, since I know that -ck has recorded 67 blocks found in his solo pool, but I only detected 54... not sure if the coinbase wasn't always signed as it is today, or what the root cause of the discrepancy is.

Anyway, from the data and ignoring the unknowns, AntPool is by far the worst offender with 4.62% empty blocks.  Next up is KnC with 2.91% and they are virtually tied with discus fish at 2.9%.  That's what I've got.  Not sure how useful it is, but I had fun putting it together

[TheRealSteve]

Well, technically, the worst offender in your list is 'unknown'   Earlier you said that where you couldn't determine the pool, you used the coinbase address (presumably if only one found) instead.  Have you tried separating those out? That's how I got the 1KTNE and 1Ar2gR ones in my table.. blocktrail doesn't separate those out (with a few exceptions).

[jonnybravo0311]

Well, technically, the worst offender in your list is 'unknown'  Earlier you said that where you couldn't determine the pool, you used the coinbase address (presumably if only one found) instead.  Have you tried separating those out? That's how I got the 1KTNE and 1Ar2gR ones in my table.. blocktrail doesn't separate those out (with a few exceptions).

I just got lazy there, to be honest.  Any time I couldn't figure out what the pool was, I just lumped it into "unknown" for the stats.  I could certainly break it down to show every address, but figured it was overkill.  I also just figured out why I didn't identify some of the ckpool solo blocks.  The first blocks coinbase script was simply "ckpool/".  I was looking for "/solo.ckpool.org/".  Doesn't matter though since the ckpool code doesn't allow for empty blocks.

Steve, if you want, I'll tweak things again to pump out all of the addresses.  The vast majority of the empty blocks by "unknown" were from the beginning of the blockchain.  Pretty much every block up to 50,000 is empty.  As a matter of fact, I think the first block I saw with more than 1 transaction was 49,076 (but that was just quickly scrolling through the data and I might have missed an earlier block or two).  It's not until the mid 60k range that we really start to see blocks with more than the coinbase transaction start to outweigh the ones with only the coinbase.

Also, there were a number of blocks found where the coinbase was paid to multiple addresses, but the first address in the list couldn't be decoded.  For example, check out block 157606.

[-ck]

There's a bit of inaccuracy in it, since I know that -ck has recorded 67 blocks found in his solo pool, but I only detected 54... not sure if the coinbase wasn't always signed as it is today, or what the root cause of the discrepancy is.

A private entity had a unique solo instance that was signed avahuobi (but still said ckpool).

[kano]

Tweaked the code a bit more to run some stats.  Ran it again today.  The vast majority of the empty blocks are from the beginning, which is to be expected.  Here's the data broken down by pool:
https://i.imgur.com/QkD1UeJ.png
There's a bit of inaccuracy in it, since I know that -ck has recorded 67 blocks found in his solo pool, but I only detected 54... not sure if the coinbase wasn't always signed as it is today, or what the root cause of the discrepancy is.

Anyway, from the data and ignoring the unknowns, AntPool is by far the worst offender with 4.62% empty blocks.  Next up is KnC with 2.91% and they are virtually tied with discus fish at 2.9%.  That's what I've got.  Not sure how useful it is, but I had fun putting it together

As far as I know, AntPool based their code off Eloipool code.
You'll probably find most of the pools with more than 0.5% empty have done that or simply use Eloipool as it is.

[TheRealSteve]

I could certainly break it down to show every address, but figured it was overkill.

I think if you kept every single one - certainly since the beginning - that would be quite a few  Was just wondering if you had that information already and just chose to lump them together under 'Unknown' after the fact.

Also, there were a number of blocks found where the coinbase was paid to multiple addresses, but the first address in the list couldn't be decoded.  For example, check out block 157606.

Odd - then again, early blocks.. who knows if something was broken there.

A private entity had a unique solo instance that was signed avahuobi (but still said ckpool).

I think I saw a few that had 'ckpool' in the scriptSig, without it being detected as kano on blocktrail, or solo via the scriptSig.  I ended up lumping those together under CKPool/other.

Examples:
ckpool/Kano: Block 360242 / https://www.blocktrail.com/BTC/tx/f8f596c2ef4fe49c2ebe0727ceefa017a43bd2081b0571cfd52229e746afe189
ckpool/solo.ckpool.org: Block 360687 / https://www.blocktrail.com/BTC/tx/c618ff74a23b343e84c0185d24695045358495c49cd030c6e974e99c19632a69
ckpool/mined by devastra/: Block 360419 / https://www.blocktrail.com/BTC/tx/b78ba888133ac4c496759490b14199903678eed2a0746f929232e7b6eb7555f4
ckpool/NiceHashSolo: Block 360224 / https://www.blocktrail.com/BTC/tx/778724d38cfec7dc8782621cd8d0d9f09e4a7c1a1f82a66ddbdda698ef3b9433
ckpool: Block 353625 / https://www.blocktrail.com/BTC/tx/c9b538f11484781f7b40b59c026e8a2763d878bbac25396dc413d782feb4d35b
ckpool: Block 352116 / https://www.blocktrail.com/BTC/tx/13f836bcbc1c6dc952bef00dea5de38096bb85a3f148faff6f62c285d7441d1d

[jonnybravo0311]

I could certainly break it down to show every address, but figured it was overkill.

I think if you kept every single one - certainly since the beginning - that would be quite a few  Was just wondering if you had that information already and just chose to lump them together under 'Unknown' after the fact.

Also, there were a number of blocks found where the coinbase was paid to multiple addresses, but the first address in the list couldn't be decoded.  For example, check out block 157606.

Odd - then again, early blocks.. who knows if something was broken there.

A private entity had a unique solo instance that was signed avahuobi (but still said ckpool).

I think I saw a few that had 'ckpool' in the scriptSig, without it being detected as kano on blocktrail, or solo via the scriptSig.  I ended up lumping those together under CKPool/other.

Examples:
ckpool/Kano: Block 360242 / https://www.blocktrail.com/BTC/tx/f8f596c2ef4fe49c2ebe0727ceefa017a43bd2081b0571cfd52229e746afe189
ckpool/solo.ckpool.org: Block 360687 / https://www.blocktrail.com/BTC/tx/c618ff74a23b343e84c0185d24695045358495c49cd030c6e974e99c19632a69
ckpool/mined by devastra/: Block 360419 / https://www.blocktrail.com/BTC/tx/b78ba888133ac4c496759490b14199903678eed2a0746f929232e7b6eb7555f4
ckpool/NiceHashSolo: Block 360224 / https://www.blocktrail.com/BTC/tx/778724d38cfec7dc8782621cd8d0d9f09e4a7c1a1f82a66ddbdda698ef3b9433
ckpool: Block 353625 / https://www.blocktrail.com/BTC/tx/c9b538f11484781f7b40b59c026e8a2763d878bbac25396dc413d782feb4d35b
ckpool: Block 352116 / https://www.blocktrail.com/BTC/tx/13f836bcbc1c6dc952bef00dea5de38096bb85a3f148faff6f62c285d7441d1d

I've got the data for every block... I got lazy with lumping things together

[kano]

...
Examples:
ckpool/Kano: Block 360242 / https://www.blocktrail.com/BTC/tx/f8f596c2ef4fe49c2ebe0727ceefa017a43bd2081b0571cfd52229e746afe189
...

Well that was my pool's last block found ... and it says /Kano in the coinbase

[TheRealSteve]

Well that was my pool's last block found ... and it says /Kano in the coinbase

Yeah, that - and the solo one - were two examples that did get recognized  Should have been clearer about that.  It's the next four that caught my attention

[kano]

Well running ckpool on it's own (without ckdb) is as easy as, for almost anyone without too much expertise to do.
Of course there are other issues - but they are the same for anyone running any pool.
There are a few pools that have used it and probably then changed the line of code that says ckpool after finding 1 or more blocks.
I know of at least 4 - even BW.com has used it for quite a few blocks - but not sure if they now only use ckpool or what else they use.

[TheRealSteve]

default string leftovers?  reasonable assumption

[kano]

Why the Eligius empty block excuse is a scam:
https://bitcointalk.org/index.php?topic=789369.msg11638137#msg11638137

[Lauda]

I don't think that this discussion went in the right way. In general I believe that people don't know if empty blocks are good or bad (neither do I). Instead of discussing the total amount of empty blocks someone should try to explain this in a single post that we can relate to in the future.
I'm not really interested in the exact number, however looking at the picture we can clearly see that AntPool has the highest percentage.

Empty blocks aren't useless as they confirm previous transactions and secure them further against attacks. Though, they are much less useful than normal blocks. I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners. 25 out of 1000 last which makes 2.5% seem much more right to work on.
I think miners shouldn't mine empty blocks because they would lose tx fee and make people await more time for their tx to be included in the chain. Why do some pools do that? Is there an advantage?

Can someone confirm this or elaborate the usefulness of empty blocks?

[jonnybravo0311]

I don't think that this discussion went in the right way. In general I believe that people don't know if empty blocks are good or bad (neither do I). Instead of discussing the total amount of empty blocks someone should try to explain this in a single post that we can relate to in the future.
I'm not really interested in the exact number, however looking at the picture we can clearly see that AntPool has the highest percentage.

Empty blocks aren't useless as they confirm previous transactions and secure them further against attacks. Though, they are much less useful than normal blocks. I am assuming the 23% number is skewed by 2009 and 2010 blocks which were mosly empty due to not having actual transactions rather than being empty on purpose by miners. 25 out of 1000 last which makes 2.5% seem much more right to work on.
I think miners shouldn't mine empty blocks because they would lose tx fee and make people await more time for their tx to be included in the chain. Why do some pools do that? Is there an advantage?

Can someone confirm this or elaborate the usefulness of empty blocks?

It's a debate that has supporters on both sides of the argument.  I am pretty firmly entrenched on the side that believes empty blocks are a waste if they are added simply to add a block to the chain.  Now, if there really are no transactions to be included - which was the case quite a bit at the beginning of the block chain - then, by all means if you find a valid solution add the block, because as chalkboard17 points out, even an empty block provides value in securing the blockchain.

Some pools are coded in such a way as to provide empty blocks to their miners until the pool can properly provide transactions.  Whether this is a limitation of the language in which the pool is written, the inability of the pool's coder(s) to properly include transaction, or some other factor, it's wrong.  Kano has provided very clear evidence that the claims of "we do it this way because it's faster" that the empty block guys stand on is indeed patently false.

Since there are almost always plenty of transactions to be included in a block, then I believe it is up to the pool to properly include those transactions in any work it provides to miners.  Creating a block just for the sake of creating a block because your pool can't push work out quickly enough is an excuse for poorly coded software and not a valid reason.  Why should all those transactions have to wait another ~10 minutes to be included on the blockchain?

[kano]

So I've continue this Eligius vs my pool https://kano.is/ comparison
... and I'll repeat: feel free to try this yourself, point 2 cgminers: one at kano.is and the other at eligius to see the block change notifications

I am of course still getting faster block change information from kano.is than from Eligius - well OVER 90%
No wonder the eligius pool luck is so bad.
The block change information from kano.is is a block of transactions that are available after the block change.
The block change information you get from Eligius is an empty block ... but still slower.

I've got over 5.5 days non-stop of stats and I will add, that for 12 hours on my pool a few days ago the stats went crap again for my pool - regularly 10-15s slower than eligius for around 12 hours (that's why I say over 90% and not over 98% as it usually is)
When I've seen a few blocks more than a few seconds slower than Eligius I KNOW something is wrong and do something about it,
since I also know eligius sux slow at block changes and if my pool is slower that means something must be wrong.
That particular 12 hours was a side effect of me not seeing a particular problem ... that I wont miss seeing again

There was an interesting 1.5 hours today (6 hours ago), where Eligius sent no block changes ... mining was still running along fine ... on the old block for 1 hour after it changed (it was a slow network block of about 0.5 hrs - plus 1 hr of mining on it stale from eligius)

Maybe the fact that they don't have someone full time looking after the pool means things go to crap and no one cares

Now there's yet another issue about all this.
Any pool mining using the standard bitcoin uses an RPC function called getblocktemplate()
Here's the actual commit:
https://github.com/bitcoin/bitcoin/commit/3390014fd0d91b0148425e794ac01c10b646a682
Yeah not much of a code change, but of course which name is on it?
Luke-Jr
The same guy who bypasses using it on block changes coz he says it's too slow
Gotta love the ignorant lack of change control on that one
I will also add that the code could be written to provide a block of transactions faster on a block change ... but that is clearly beyond his mcdonalds burger flipping skills where he used to work.

--

Edit: sigh - and on Eligius today they say how they've finished changing stuff and no one should have noticed anything.
Of course no one noticed anything, coz they hid the fact that part of that was an hour of mining stale blocks i.e. the whole pool was mining rubbish for an hour (as stated above) that could only find orphans

Update of my stats for the last 150 blocks:
mining on Eligius has been 1,345 seconds (more than 22 minutes) of stale mining when compared to kano.is
with still >90% of eligius block changes slower than kano.is (making up that 22 minutes)

[kano]

Another update

Last 326 blocks:
95% of Eligius block changes were slower than https://kano.is/
( only 5 of the 326 block changes were slower on https://kano.is/ )

So, if you were mining at Eligius, almost half an hour of mining on stale work in the last 326 blocks.

Clearly their empty blocks are not relevant to the argument they hide behind.

[TheRealSteve]

or at the very least has negligible impact considering other delays

For something slightly more generic on the topic: https://tradeblock.com/blog/bitcoin-network-capacity-analysis-part-6-data-propagation

[kano]

... actually
If I get time to, some time soon, I may start up a thread about that.
Specifically comparing pool block change times.

Even big pools like AntPool use Eloipool and also merge mine.
I'd not be surprised if they have crap average block change times also.
I doubt any pools in the past have even bothered to look into this - or if they have, may have got bad results they didn't want to report about their pools.

An expectation of >1% orphans seems quite high ... when the block average is expected to be 600s per block (or less when diff is rising)

I would not be surprised at all if my pool was one of the fastest, on average, to send out block changes.
Will be interesting to see how quickly other pools - not using eloipool - handle block changes - eloipool is slow so it may be a bit pointless comparing it running on other pools.

Although it's only been 254 blocks, the number of orphans at https://kano.is/ is only 0.8% (2) - and my ckdb reports ALL block shares submitted to the pool before it knows about a block change, no matter what bitcoin says about them - there are no 'hidden orphans'.
We've only had 2 so far, one back when the pool started, and one since then that was an example of what other pools may not show, where the block was late for the network, but not late for the pool - yes sometimes network blocks can take a few seconds to get to the pool

ckpool work handling by -ck is fast, but I also monitor the information relevant to block changes and act on it
(yeah my pool even wakes me up in one of the rare cases that causes block delays)

[Mikestang]

I doubt any pools in the past have even bothered to look into this - or if they have, may have got bad results they didn't want to report about their pools.

This would be a very interesting study to compare block change times across several pools, the information could be very useful.  Not only might it help people decide where to send their hash (one reason anyway), but it might also motivate other devs to improve their pool code.

[el kaka22]

Today, there is a stress test on bitcoin, and lots of transactions have been made (>7k unconfirmed as of the time of writing), however there are still empty block: https://blockchain.info/block/0000000000000000088390da860e3624e9c6f3162dcce3f2f166bb371e018de7 (height=363266). So I don't think empty blocks should exist as they make people waiting for confirmations wait even longer!

[Peter R]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

[jonnybravo0311]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

[Peter R]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

[jonnybravo0311]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

I suppose it could be... but since I've already started the program execution and it takes quite a while to complete (done 326,000 blocks so far), you'd have to wait a while longer for me to make the code changes and run it again .

[Peter R]

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

I suppose it could be... but since I've already started the program execution and it takes quite a while to complete (done 326,000 blocks so far), you'd have to wait a while longer for me to make the code changes and run it again .

Haha no problem.  The current file format will give me plenty to get started on.  Thanks again!

[jonnybravo0311]

So it just finished... here are the numbers:

As of block 364125, there are a total of 85,422 empty blocks.

AntPool: 5.16% of their blocks are empty, latest one was today.
Discus Fish: 2.88% of their blocks are empty
KnC: 2.88% of their blocks are empty
Eligius: 2.26% of their blocks are empty

EDIT: I made the changes to the code to include the block size... re-running it now.  I'll post it and provide the link when it's done.

Format of the data is:
Block Height, NumTx, BlockTime (UTC), BlockSize (Bytes), WhoMined (if I could figure it out, BTC address of coinbase transaction if I couldn't, or unknown if no BTC address could be deciphered from coinbase transaction).

EDIT 2: Just finished running it again with the addition of block size per your request.  I've uploaded a zip file here: https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0.  Has blocks up to 364144

Archive contains two files:

empty_blocks.csv has the raw data in the format described above
stats.csv has data on pools, how many blocks they've mined, how many were empty and percentage of empty to total.

Enjoy.

[Peter R]

Enjoy.

Thanks so much!.  Here's some preliminary results, if you're interested: https://bitcointalk.org/index.php?topic=68655.msg11809512#msg11809512

[achow101]

Is your code for this open source or available for people to download and run themselves? If so, where can I get a copy.

[jonnybravo0311]

Is your code for this open source or available for people to download and run themselves? If so, where can I get a copy.

I didn't publish it... it's just something I wrote up for myself.  To be honest, it's not the cleanest Java code I've ever written.  For example, rather than handle Exceptions, I just throw them.  Everything is in one file - multiple defined private classes and inner classes.  Hardcoded bitcoin daemon connection parameters.  No comments.  No build system (like maven, etc).

Thanks so much!.  Here's some preliminary results, if you're interested: https://bitcointalk.org/index.php?topic=68655.msg11809512#msg11809512

I'll check it out.  Glad to help

[unamis76]

Pretty interesting data... There seem to be more "blank" blocks than I thought. This is definitely not good, in my opinion...

[NeuroticFish]

Pretty interesting data... There seem to be more "blank" blocks than I thought. This is definitely not good, in my opinion...

I agree, especially since the last attack/stress test.
Somebody smarter than me should maybe think if this can be done in the network somehow - if the number of unconfirmed transactions is 10k, the pool / finder includes only 1, or 100 transactions, making the block 1k or 100k, while maximum is 1MB is unfair - the network should force to include more transactions, close to max block size.

While this is not fixed, discussions about bigger block size are .. just not needed.

[jonnybravo0311]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

[p3yot33at3r]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

It's not just the Chinese either. I'm not sure what's worse, the pools that are knowingly throwing out these empty blocks in search of greed - or the miners that mine with them......in search of greed......

[kano]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

It's not just the Chinese either. I'm not sure what's worse, the pools that are knowingly throwing out these empty blocks in search of greed - or the miners that mine with them......in search of greed......

Well ... it's even more confusing when Luke-jr/Eloipool is saying that people mining the spam (and thus getting rid of it) are doing the wrong thing ... lulz.
https://bitcointalk.org/index.php?topic=441465.msg11826511#msg11826511

Although the spam has caused a lot of problems on the network (tell me about it - I've missed sleep thanks to it ...) it's also meant higher transaction fees:
Last 5 https://kano.is/ block reward txn fees % have ~been 2% 1% 1% 2% and 1% ... normally it averages 0.4%

--

Oh and I ran a 40 block https://kano.is/ vs eligius block change test again last night.

https://kano.is/ was only slower with 4 blocks in that 40 range ... of course the reason was they were Eligius blocks.

https://kano.is/ was faster with the other 36 blocks, and the total average was 1.4% faster for the 40 blocks with an average of 8 seconds faster per block change!

I still have no idea why anyone mines there, I guess they don't compare pools properly or see the '0%' fee comment, but don't realise that there's a high average expected loss due to mining on eligius (vs my pool) that more than makes up for that.

[p3yot33at3r]

It is a little worrying. With the attacks currently taking place & these pools that are not pulling their weight or contributing to the network, it would be nice if the mining community tried to increase awareness of the harm they are doing. It seems to me that these pools are biting the hand that feeds them, leeching even. I've read so many complaints from miners about extended tx confirmation times & fees, yet they continue to mine at the very pools that are making the situation worse - am I right?

If so, what does anyone suggest?

[TheRealSteve]

There's only one thing to suggest: go mine at a pool that aligns more with your ideals.  Problem is that the ideals of many people are trumped by their greed.
All else being equal, would you rather mine at a pool that for a given period pays out BTC1.0 while playing nice, or at a pool that pays out BTC1.01 but they mine coinbase-only blocks more often than strictly necessary?  What if it's BTC1.05?

[kano]

I've had a PM question about what the numbers I've posted mean so ...

When a pool gives out work to the miners, it gives work that can find a block built on the pool's current known best block.

The pool changes it's current best block either when one of it's pool miners finds a new one, or, most often, when someone else on the bitcoin network finds and distributes a 'better' block, around the network.

In simplest terms, what this means is that: most miners will have a small time frame, each time the block changes, from when someone finds the block, distributes it on the network, and then the pool gets the new block, then the pool sends out new work, for the new block, to the pool's miners.

There's 3 parts in that:
1) Some miner somewhere on the network finds a block (and sends it to their pool if they are mining to a pool)
2) The block is distributed on the network and your pool gets it
3) Your pool processes the new block then sends out new work to your pool's miners

If someone on your pool finds the block, then 1) and 2) are simply how long it takes for the miner to send their block winning share to the pool.
Otherwise:
Step 1) is out of the pool's control so you can't certainly know exactly when that happens
Step 2) is dependent upon how well the pool is connected to the the rest of the network and specifically, wherever the block was found.
e.g. if you are mining at PoolA and PoolB finds the block, it depends on how long it takes for that new block message to get from PoolB to PoolA

No matter who finds the block,
Step 3) is dependent on how quickly the pool can process the block and send out new work to the pool's miners.
It will also depend on the size of the block and other factors like the block processing complexity and of course how fast the pool software is and how long it takes to send work to the miners

Ignoring when the pool itself finds a block, the time the pool takes to do step 2) and step 3) represents wasted mining time.
Step 3) is completely 100% wasted time
Step 2) is either wasted mining time, or if a block is found during this time, will represent how likely the pool's chance is of winning an orphan race against the other block

So .........

What is my 8 seconds I posted above?

If you have 2 cgminers each pointing to different pools, then each cgminer will report on the screen each time it gets work that is a block change.
The difference between a block change time on the 2 miners represents wasted mining time.
It is of course Step 2) + Step 3)
It should be small and usually each pool should randomly be faster than the other pool
If one pool is averaging better than another pool, it is clearly doing one or both of:
a) better connected to the rest of the bitcoin network (Step 2)
b) faster at processing blocks and sending out new work (Step 3)

The average time difference represents, on average, the amount of extra wasted time mining on the slower pool.
The faster pool is also wasting some time mining each block change, but the slower pool is wasting, on average, that much more per block change.

Eligius' claim is that their empty blocks make the total of Step 2) + Step 3) faster so miners are wasting less time mining stale work.

But what point is there to that when even with that 'faster' processing, they are averaging slower than a pool that doesn't cheat the network by mining empty blocks ...

[NeuroticFish]

Wow, long text, but interesting!

I see 2 options:

#1. More "advertising" to increase awareness between miners about this problem, maybe some will switch pool.
#2. Implement into the network something to force them to play fair.


I don't know if #2 is actually possible, but that would be the way to go. I find this more important than increasing the block size.

[p3yot33at3r]

Wow, long text, but interesting!

I see 2 options:

#1. More "advertising" to increase awareness between miners about this problem, maybe some will switch pool.
#2. Implement into the network something to force them to play fair.


I don't know if #2 is actually possible, but that would be the way to go. I find this more important than increasing the block size.

Both good suggestions, I was thinking along similar lines. Maybe a sticky or some kind of bold statement from organofcorti in his pools thread, or any respected community members for that matter would help? I'm sure that #2 would be possible somehow, but it would take time to implement I think.....?

EDIT: Some kind of list showing which pools are contributing to the network & which ones aren't - name & shame kinda thing......

[TheRealSteve]

Maybe a sticky or some kind of bold statement from organofcorti in his pools thread, or any respected community members for that matter would help?

Various people quite often state that they would prefer to see people mine at smaller pools (for reasons besides any concerns about coinbase-only blocks).  In the end, well, see my previous post.  I'm happy to mine at Slush's pool, kano.is and BTCDig with the smidgens of hash rate I've got, but then I don't worry about that marginal increase in income or whether I go without a payout for weeks on end..I'm not running anything profitable anyway, even with solar (get more money feeding back into the utility network)

organofcorti posts mostly just verifiable (with a bit of effort) statistics - I think that actively discouraging certain pools on his blog or even his pools thread (which looks like it'll be replaced if the guy volunteering gets his thread off the ground) would just introduce partiality that would both hurt objective interpretations and credibility of his stuff.  As it is, some might think the BTC20 donation 'coincidentally' made after a post that showed the pool involved as being the biggest is of potential concern. ( I'm not, as long as he keeps posting statistics, and the funds are probably welcome for covering expenses and justifying what is mostly a hobby. )

To skip an item...

EDIT: Some kind of list showing which pools are contributing to the network & which ones aren't - name & shame kinda thing......

That certainly is something that organofcorti could do.  This thread started out with some statistics of coinbase-only blocks vs non-, how that relates to previous block time, and in another thread how it relates to the previous block's size, etc.  Presented in a simple table it again just becomes verifiable statistics and people can make up their own mind; and do remember that miners who aren't including any regular transactions are still contributing to the network by building the block chain, and there are differing opinions on whether coinbase-only blocks are actually a bad thing, a good thing, or one of several shades of grey in between.

I'm sure that #2 would be possible somehow, but it would take time to implement I think.....?

It's possible, I think would require another consensus round (considering you'd have to deny certain blocks as being invalid after some time), and a whole lot of discussion.. which is where most of the time would be (I think that was discussed in this thread as well?  It was discussed somewhere anyway).  The required code itself is somewhat trivial once the rules have been fleshed out.  Good luck with that, though... the max block size debate hasn't even been properly settled yet

[drakoin]

The chain had tens of thousands of empty blocks in the beginning.

What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...

How many empty blocks are there this year - and who mined them?

[TheRealSteve]

The chain had tens of thousands of empty blocks in the beginning.
What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...
How many empty blocks are there this year - and who mined them?

Partial answer to that is already in this thread ( https://bitcointalk.org/index.php?topic=1085800.msg11593801#msg11593801 ), or you could divine it from jonny's dataset ( https://bitcointalk.org/index.php?topic=1085800.msg11584786#msg11584786 ).
Updated for blocks in 2015 up to block 365,401:

Edit: updated to show 2-transaction blocks, only because that stood out for one particular pool.

Keep in mind that this data on its own isn't very interesting without also looking at e.g. block times and sizes.

[drakoin]

The chain had tens of thousands of empty blocks in the beginning.

What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...

How many empty blocks are there this year

546

I used your
https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0
and deleted all blocks until before block 336861 (2015-01-01 00:02:52),
and then did a =COUNTIF(B1:B27284, 1)

546 / 27284 = 2.0%

- and who mined them?

[drakoin]

The chain had tens of thousands of empty blocks in the beginning.
What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...
How many empty blocks are there this year - and who mined them?

Partial answer to that is already in this thread ( https://bitcointalk.org/index.php?topic=1085800.msg11593801#msg11593801 ), or you could divine it from jonny's dataset ( https://bitcointalk.org/index.php?topic=1085800.msg11584786#msg11584786 ).
Updated for blocks in 2015 up to block 365,401:

Keep in mind that this data on its own isn't very interesting without also looking at e.g. block times and sizes.

Ah thanks, very good.

I wanted to see who are the freeriders who are
devaluing the PoW of those who are including transactions

(which -as far as I understand it- was Satoshi's concept,
to actually include transactions into mined blocks ...)

Keep in mind that this data on its own isn't very interesting without also looking at e.g. block times and sizes.

I don't believe that the mempool was empty
when they mined those 71+272+53+150 blocks.

megabigpower,
antpool,
kncminer, and
discussfish

are so far above the average with their number of
coinbase-only blocks that it might be safe to say
that they a sabotaging the Bitcoin PoW deliberately?

[-ck]

megabigpower,
antpool,
kncminer, and
discussfish

are so far above the average with their number of
coinbase-only blocks that it might be safe to say
that they a sabotaging the Bitcoin PoW deliberately?

They're not trying to sabotage PoW deliberately. It's just that most of them use the same software that takes the same shortcut to cope with its lack of scalability by building transaction free blocks first every time and what you're seeing is the larger the pool, the more that scalability problem becomes pronounced thereby mining more transaction free blocks.

[TheRealSteve]

It's not so much about whether the mempool was empty (not sure there if there's been a time recently where that is even true, but I don't collect that data), but about how these pools mine.  You should probably read through this thread and a few related ones, also over on reddit.  The tl;dr is that depending on how the pool is set up, seeing coinbase-only blocks mined is to be expected as part of a short term revenue-maximizing strategy.  Should take a page out of ckpool's book
Now if they consistently decided to mine coinbase-only blocks, that'd be more troubling.  I'm very slightly curious what's up with BTCChina's relatively large number of 2-transaction blocks, though
( There's also a few bumps at other powers of 2 - mostly from megabigpower, which I already knew about; https://i.imgur.com/76kZa1W.png  - but 2 exactly stands out. )

[p3yot33at3r]

megabigpower,
antpool,
kncminer, and
discussfish

are so far above the average with their number of
coinbase-only blocks that it might be safe to say
that they a sabotaging the Bitcoin PoW deliberately?

They're not trying to sabotage PoW deliberately. It's just that most of them use the same software that takes the same shortcut to cope with its lack of scalability by building transaction free blocks first every time and what you're seeing is the larger the pool, the more that scalability problem becomes pronounced thereby mining more transaction free blocks.

Not sure how I missed this info - very interesting. For the record, as well as the benefit of those who are unaware:

Is it safe to say that the worst offenders are all using the same software?

What software is it?

Who built/is responsible for it?

Do you know if they will address the issue?

[organofcorti]

This is interesting. Data has been kernel smoothed against blocks solved, but plotted against time.

[-ck]

This is interesting. Data has been kernel smoothed against blocks solved, but plotted against time.

Cool. Look for a correlation between percentage of empty blocks and hashrate.

[organofcorti]

This is interesting. Data has been kernel smoothed against blocks solved, but plotted against time.

Cool. Look for a correlation between percentage of empty blocks and hashrate.

Why's that?

[valkir]

Why would mbp do 1 transactions block??

[-ck]

Cool. Look for a correlation between percentage of empty blocks and hashrate.

Why's that?

It's just that most of them use the same software that takes the same shortcut to cope with its lack of scalability by building transaction free blocks first every time and what you're seeing is the larger the pool, the more that scalability problem becomes pronounced thereby mining more transaction free blocks.

Strictly speaking, with stratum mining the "load" is proportional to the number of miners, not the absolute hashrate but hashrate is a surrogate marker of number of miners.

[organofcorti]

Cool. Look for a correlation between percentage of empty blocks and hashrate.

Why's that?

It's just that most of them use the same software that takes the same shortcut to cope with its lack of scalability by building transaction free blocks first every time and what you're seeing is the larger the pool, the more that scalability problem becomes pronounced thereby mining more transaction free blocks.

Strictly speaking, with stratum mining the "load" is proportional to the number of miners, not the absolute hashrate but hashrate is a surrogate marker of number of miners.

Ah, gotcha. So we're comparing blocks per unit time and empty blocks, by block maker.

[kano]

You left the best pool of all out of your graph

[organofcorti]

You left the best pool of all out of your graph

Hrmm, either of the best pools you mean? 

I had to leave some out just to make room. Sorry. I was actually tempted to leave out all the pools that had no 1 tx blocks, but then you  wouldn't be able to see how many were making them and how many weren't.

[organofcorti]

This is interesting. Data has been kernel smoothed against blocks solved, but plotted against time.

Cool. Look for a correlation between percentage of empty blocks and hashrate.

Here it is, showing a correlation between one tx blocks per week and blocks per week - although you'd seem similar for the relationship between orphan blocks per week and blocks per week too.

I'm not sure that helps much.

[jonnybravo0311]

To see how well the network coped with the recent "stress tests" I ran the numbers again, from block 364145 until the most recent one: 365138.  The following screenshot is pretty telling:



The most recent block in the list, found by f2pool was indeed an empty block.  There are around 20,000 unconfirmed transactions.

[sloopy]

F2Pool will continue to mine empty blocks, spv mine, or anything else which they think gives them even the slightest advantage.
They are making the coin while it can be made no matter what it takes to do so.
Greed, she is evil and she will not be comforting when Mrs. Karma comes to play.

Don't get me wrong, I don't think either of the other ones (antpool) are any better. They don't say peep about it, and just keep doing it.

None of them care about bitcoin in any way, unless it serves their wallet.

Unless someone puts together a simple, coherent explanation the regular Joe's will not get it, and that's the ones who would care. So many wouldn't...

...and don't get me wrong. I would thoroughly love seeing some of these choices backfire in a big way.

[p3yot33at3r]

F2Pool will continue to mine empty blocks, spv mine, or anything else which they think gives them even the slightest advantage.
They are making the coin while it can be made no matter what it takes to do so.
Greed, she is evil and she will not be comforting when Mrs. Karma comes to play.

Don't get me wrong, I don't think either of the other ones (antpool) are any better. They don't say peep about it, and just keep doing it.

None of them care about bitcoin in any way, unless it serves their wallet.

Unless someone puts together a simple, coherent explanation the regular Joe's will not get it, and that's the ones who would care. So many wouldn't...

...and don't get me wrong. I would thoroughly love seeing some of these choices backfire in a big way.

This! Well said sloopy. It's the averge joe/noob that needs to be made aware of what is going on.

[jonnybravo0311]

Unfortunately, the average Joe probably won't care one way or the other.  He's going to get his BTC payout for mining on f2pool, AntPool, whatever pool, and will continue to mine there unless those payments fail to get to his wallet.  Furthermore, the people on the side of the argument that even empty blocks secure the blockchain aren't helping matters any.  It's almost too bad that there's any BTC awarded for finding a block.  Those pools would change their tunes very quickly if they got nothing from finding a block.  You can bet they'd stuff every transaction they could into a block.

[-ck]

It's almost too bad that there's any BTC awarded for finding a block.  Those pools would change their tunes very quickly if they got nothing from finding a block.  You can bet they'd stuff every transaction they could into a block.

As the block reward halves and transactions become a more significant proportion of the income they will need a different approach but for the time being the transaction rewards add only a little, though it's getting more every day. In a year when the reward is only 12.5 BTC, then the current transaction rewards of .25+ will regularly amount to 2%. It can only but rise with time, but not soon enough for the existing pools to give a shit about their empty blocks if people keep mining there.

[Mikestang]

if people keep mining there.

That's the key, the only way to get pools to care is to stop mining to them.

Really, pools that do care [about the bitcoin network] already don't mine empty blocks.  The ones that are only in it for a quick buck stick out like a sore thumb.

[sloopy]

I always try to stay on topic, but if I have the opportunity I am blasting what these pools are doing.

I have sold some S3s to a couple of acquaintances who have really been getting in the coin scene but I told them if I catch them supporting pools who do not use common sense for the community I will break in their house and steal them back!
Heh, obviously I would not go to that extreme, but the average 'Joe' who only cares about his / her own btc income simply hasn't been taught the error of their ways.

Of course it is easy to be uncaring behind a keyboard. If people take the time to read about what has transpired with all of the scams in bitcoinland since inception they will see a clear picture of the human psyche.
Not to get too far off topic here, but this is why we avoid religion and politics at meetings and parties, because people like to hide behind the facade of helping themselves. At what cost to others, and are you truly doing such by mining at Discus Fish or Antpool?
I am all for making a profit. I am all against doing something which will cause harm to the network. It is something akin to the great pyramids, or other natural wonders which has been achieved by the human race and you should each hold yourself to a higher standard.

When I was a little child I remember imagining my life was a movie the world was watching, every minute of every day. Then years later there was a real movie made called Truman? I think it was, Truman. As a child I simply thought about what an interesting thing it would be to pop in on different people and see what they were really like. As I grew and saw the movie it brought out many of the bad things which could possibly come of it, but real life, well it is much worse than either experience could have ever let me think about. My imagination has been outplayed many times by real life and seeing the choices people make.

I am far from a saint, and would love to make more coin, fiat, gold, silver, you name it.
I also want that movie of my life to replay and show mostly good things. I wish they had all been great choices but eh, I am human. Making the same choice day after day after day, after seeing proof of harming other people, other people's creations, or knowingly causing any sort of true harm for a bit of greed simply isn't worth it.

Back to politics and religion, I don't pretend to be a programmer, know what happens in the next world, or who the first President of the World will be, but I know that I don't want to go to bed tonight with the thought I made someone else hungry because I manipulated a system for a few bucks.

 I think F2pool and Antpool are black holes which need to be closed and I just hope the network and all of us are around to see it happen.
I also think if the people posting in this thread talk about these two subjects enough they can get through to some average joes. Anything is possible. You guys are smart enough to make a 'legal' way to attract their miners!

[achow101]

I have sold some S3s to a couple of acquaintances who have really been getting in the coin scene but I told them if I catch them supporting pools who do not use common sense for the community I will break in their house and steal them back!
Heh, obviously I would not go to that extreme, but the average 'Joe' who only cares about his / her own btc income simply hasn't been taught the error of their ways.

What pool do you want them to mine with then? Every pool that I know of mines empty blocks. It is simply part of mining. Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does. They lose time and money by creating only full blocks. For receiving an extra 0.1-0.2 BTC every so often, it doesn't make much sense to most pool operators and miners.

[kano]

I have sold some S3s to a couple of acquaintances who have really been getting in the coin scene but I told them if I catch them supporting pools who do not use common sense for the community I will break in their house and steal them back!
Heh, obviously I would not go to that extreme, but the average 'Joe' who only cares about his / her own btc income simply hasn't been taught the error of their ways.

What pool do you want them to mine with then? Every pool that I know of mines empty blocks. It is simply part of mining. Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does. They lose time and money by creating only full blocks. For receiving an extra 0.1-0.2 BTC every so often, it doesn't make much sense to most pool operators and miners.

Well firstly, not every pool.
See the statistics posted.

Secondly, where did you come up with

Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does.

Have you verified this to be relevant?

It's a simple test to run 2 cgminers and compare 2 pools and see if it is relevant or not ...

As I have stated many times, my pool averages block changes FASTER than Eligius but also NEVER skips getting available transactions from the mempool ... ... ...

[-ck]

What pool do you want them to mine with then? Every pool that I know of mines empty blocks. It is simply part of mining. Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does. They lose time and money by creating only full blocks. For receiving an extra 0.1-0.2 BTC every so often, it doesn't make much sense to most pool operators and miners.

This is exactly the kind of misinformation spread by the pools responsible that we have proven is bullshit on this thread. It is NOT every pool mining empty blocks as has been stated over and over again on this thread. Nor do they HAVE to mine empty blocks to be fast on block changes. It is their choice of software and how they configure it at fault.

[sloopy]

I have sold some S3s to a couple of acquaintances who have really been getting in the coin scene but I told them if I catch them supporting pools who do not use common sense for the community I will break in their house and steal them back!
Heh, obviously I would not go to that extreme, but the average 'Joe' who only cares about his / her own btc income simply hasn't been taught the error of their ways.

What pool do you want them to mine with then? Every pool that I know of mines empty blocks. It is simply part of mining. Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does. They lose time and money by creating only full blocks. For receiving an extra 0.1-0.2 BTC every so often, it doesn't make much sense to most pool operators and miners.

Man you gotta take a long look at what they are doing and what they plainly state they are doing.
I assure you I am not here bitching about it for my health. I sincerely prefer lurking and learning much more than posting and comparing.
When I see something which feels like an injustice I take a second look, and a third, and I start asking people who have earned a level of trust regarding the topic, and I read more. If you look at the threads from the Bitcoin Discussion section, the Development & Technical Discussion threads, along with the Discus Fish / F2pool thread it all adds up. Especially when the spokesperson / one of the pool operators or owners states very clearly what their intentions are. My understanding of his intent is he flat does not care about anything but his profit. Not the network, not miners (except to make himself profit) and certainly not the individuals who see this network as bigger than what profit we are making today. I said that is my interpretation but I quoted a few posts up exactly what was said regarding SPV mining and that attitude carries through to 0 or single transaction blocks, whichever way you wish to refer to it. Sure there were great reasons early in the network's life to have MT blocks, but man we have been living with a backlog of transactions and they (MT blocks) are still being submitted.

I am not a programmer and probably as close to an average Joe or closer than most of the people who have posted in this thread. There have been posts by some extremely bright and talented individuals with not only the knowledge, the skills, the experience but the right type of neutral outlook on the entire scene. They have presented information which is based in fact you can check for yourself.

It is a decision / decisions to use the software which is setup to cause the scenarios we are discussing in order to make more profit at the expense of anyone and everyone. I applaud you for taking time to consider this discussion, but I sincerely request you take a moment to consider what anyone, or even me in particular would have to gain by posting anything negative about F2pool / Discus Fish. I used them as a backup pool to my normal PPLNS pools with a tiny amount of hashrate. It is through reading the aforementioned areas of this forum, following events unfolding over the past few weeks along with spending months of reading and watching videos from any bitcoin source of information I can find on the net.

If you take a few minutes and look at bitcoin.org and read through some of the conversations here and other places I am sure you will see these actions have negatively impacted the network. Once I saw the motive, I suspected, once I saw the action that morning of the 4th I saw the crime, and once I straight up asked in their thread and they told me well it did not take much more for me to understand that these people do not care if their actions negatively impact the network. They care if it profits them today.

I read F2pool lost 5% of their hash over the fork so you may think they would pay attention to such a loss, no. They have a system in play and the SPV mining will never stop because it leads to a higher success rate which in turn attracts more miners who are not familiar with their practices. 0 - 1 transaction blocks also lead to a higher success rate due to propagation times. The world isn't fair, but there is a right and wrong.

At some point you have to ask yourself the most simple question... if spv mining and 0-1 transaction blocks lead to a higher success rate without having a negative impact on the network and its users / miners why wouldn't every pool do it?

[p3yot33at3r]

HINT.

Answer - is it:

a) Greed?
b) Uncaring disregard for the Bitcoin Eco-system?
c) Shoddy coding?
d) Pure greed?
e) All of the above?

Answers on the offending pools threads please 

[achow101]

I have sold some S3s to a couple of acquaintances who have really been getting in the coin scene but I told them if I catch them supporting pools who do not use common sense for the community I will break in their house and steal them back!
Heh, obviously I would not go to that extreme, but the average 'Joe' who only cares about his / her own btc income simply hasn't been taught the error of their ways.

What pool do you want them to mine with then? Every pool that I know of mines empty blocks. It is simply part of mining. Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does. They lose time and money by creating only full blocks. For receiving an extra 0.1-0.2 BTC every so often, it doesn't make much sense to most pool operators and miners.

Well firstly, not every pool.
See the statistics posted.

Well, I stand corrected. I didn't quite understand the data since I didn't see any headings, but after examining it a little more, it seems that your pool and a few others have never found an empty block, which I find kind of surprising.

Secondly, where did you come up with

Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does.

Have you verified this to be relevant?

I used logic, I have not verified it. Logically, more CPU cycles to process transactions will take more time than not spending those cpu cycles to process transactions. However, I will test this and see if my theory is correct.

It's a simple test to run 2 cgminers and compare 2 pools and see if it is relevant or not ...

As I have stated many times, my pool averages block changes FASTER than Eligius but also NEVER skips getting available transactions from the mempool ... ... ...

I do not think that such a test would be an accurate test. Since both pools use different software and are run on different servers and configurations, there are too many variables that could change those times. Instead, I will perform a test on a regtest network where I will create a mempool of 0, 10000, 50000, and 100000 transactions and compare the times it takes for the getwork and getblocktemplate commands to complete. This creates a controlled test environment and reduces other factors that could affect results.

At some point you have to ask yourself the most simple question... if spv mining and 0-1 transaction blocks lead to a higher success rate without having a negative impact on the network and its users / miners why wouldn't every pool do it?

Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. It doesn't particularly mess with confirmation times and while it would be nice if those blocks included transactions, it doesn't matter (at least for me) that those transactions wait another ten minutes like they should have

[kano]

...

Secondly, where did you come up with

Gathering transactions from the mepool to fill a block takes time, time when another pool that doesn't do that can claim the block reward before your pool does.

Have you verified this to be relevant?

I used logic, I have not verified it. Logically, more CPU cycles to process transactions will take more time than not spending those cpu cycles to process transactions. However, I will test this and see if my theory is correct.

It's a simple test to run 2 cgminers and compare 2 pools and see if it is relevant or not ...

As I have stated many times, my pool averages block changes FASTER than Eligius but also NEVER skips getting available transactions from the mempool ... ... ...

I do not think that such a test would be an accurate test. Since both pools use different software and are run on different servers and configurations, there are too many variables that could change those times. Instead, I will perform a test on a regtest network where I will create a mempool of 0, 10000, 50000, and 100000 transactions and compare the times it takes for the getwork and getblocktemplate commands to complete. This creates a controlled test environment and reduces other factors that could affect results.
...

And yet my test checks the ONLY reason to do it.

If a pool is sending you slower block changes, then the argument to use it is completely nullified.

The reason to do it, is to reduce the amount of time spent mining on stale work.

If what ever the pool does means a certain pool is slower at getting you new work, then ... well it seems blatantly obvious to me ...

Lets look at it this way:
If my pool can get new work extremely quickly ... most of the time ... and your RPi takes 12 days to process a block change ... who cares about your test?

It is quite obvious that doing more work takes more time.
What is also obvious is that the way we process block changes without doing something bad for bitcoin, is faster than that another pool that does the "faster" way this bad for bitcoin.

My point is the more obvious that there are all sorts of things that do affect the pool producing block changes.
While we are not doing one that is specifically bad for bitcoin, we still win.

[kano]

...
Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. It doesn't particularly mess with confirmation times and while it would be nice if those blocks included transactions, it doesn't matter (at least for me) that those transactions wait another ten minutes like they should have

Um, you completely misunderstood something there.
It doesn't matter if 2 blocks are found 1 second apart or 1 hour apart.
They are still counted in determining the future difficulty that determines future transaction confirm times.
If every 2nd block was found as such a pair, then that means the network can only confirm half as many transactions over a given time period.

[achow101]

And yet my test checks the ONLY reason to do it.

If a pool is sending you slower block changes, then the argument to use it is completely nullified.

The reason to do it, is to reduce the amount of time spent mining on stale work.

If what ever the pool does means a certain pool is slower at getting you new work, then ... well it seems blatantly obvious to me ...

Lets look at it this way:
If my pool can get new work extremely quickly ... most of the time ... and your RPi takes 12 days to process a block change ... who cares about your test?

It is quite obvious that doing more work takes more time.
What is also obvious is that the way we process block changes without doing something bad for bitcoin, is faster than that another pool that does the "faster" way this bad for bitcoin.

My point is the more obvious that there are all sorts of things that do affect the pool producing block changes.
While we are not doing one that is specifically bad for bitcoin, we still win.

I'm not trying to compare two specific implementations. My point is that miners will choose to continue to allow empty blocks to be mined because they in general allow them to make more money. Some implementations might be worse than others, for instance, your ckpool software is obviously working better or on better hardware than Eligius's software.

Consider this:
I set up a pool using the exact same software, server config, etc, except I allow empty blocks to be mined. Since I am not using extra cpu time to gather mempool transactions, logically, I will have faster block changes and more blocks than you will.

Another thing, you seem to be comparing between you and Eligius. Both only represent a small percentage of the hash power. What about comparing between your and f2pool or antpool? Are their block changes faster than your or not? Have you tried?

...
Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. It doesn't particularly mess with confirmation times and while it would be nice if those blocks included transactions, it doesn't matter (at least for me) that those transactions wait another ten minutes like they should have

Um, you completely misunderstood something there.
It doesn't matter if 2 blocks are found 1 second apart or 1 hour apart.
They are still counted in determining the future difficulty that determines future transaction confirm times.
If every 2nd block was found as such a pair, then that means the network can only confirm half as many transactions over a given time period.

Right. I forgot about the difficulty retarget. However, those blocks found with extreme time differences (1 sec or 1 hour) essentially cancel each other out in regards to the retarget.

[sloopy]

At some point you have to ask yourself the most simple question... if spv mining and 0-1 transaction blocks lead to a higher success rate without having a negative impact on the network and its users / miners why wouldn't every pool do it?

Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. It doesn't particularly mess with confirmation times and while it would be nice if those blocks included transactions, it doesn't matter (at least for me) that those transactions wait another ten minutes like they should have

I believe you will find as with SPV mining 0-1 block transactions do indeed cause issues when they are the goal. If it were something which happened as a byproduct, as with the early days of mining then it is not having an impact. When pools make a conscious decision and not only this one, but choose to have a mindset geared in this manner it speaks volumes regarding their largest motivation.


Please, give me your opinion from your own research should Elgius have better payouts to their miners?

All things equal and 0-1 transaction blocks, etc but as a pool that still want profits. F2pool is pps and motivated to propagate faster as well, but  for their individual profit and if they did not pay pps (again in your opinion) would they have the share of the network they have today?

I do not see any pool which doesn't care about profit or deny caring as any capitalistic enterprise should. IT is what is 'supposed' to keep miners on task.
I see a huge difference in the way pools choose to get there and the payouts to miners.
I also see a huge line in the sand between pools who choose to mine 0-1 transactions and SPV mine versus the ones who do not choose to do either.

Through all of the reading I have done there have been various PPS and PPLNS pools but most of the one's which are no more seemed to share a common history. PPS go broke and cannot pay the miners. There are scams, and over-promising, bad operators, but overall you start seeing it well before they close. PPLNS the same, but variance hits PPS much harder doesn't it? I mean much harder, right? I know you know this but I am talking it out for myself, but the PPS must pay for every share no matter what, and a PPS pool with 1 - 2 PH is almost immediately laughed out of the community starting up these days. If they throw in some other things like the first miner to pop a block gets the 25, or do PPS in combination with DVG I think it is like MMpool, but overall a straight up PPS pool is no more except at F2pool, and now Antpool has the option.
 
Is it a coincidence those are also two of the largest pools in the network, mining the ways they are? Why did Antpool suddenly employ the PPS option? You can just click and select a 2.5% PPS for your entire account. They are not doing such to lose money. They could have left the price of S5s at $340 and still be making a great profit but the price was jacked to 430ish with freight to the US. They are not making decisions to lose money. They also work together as stated in the F2pool thread by the same owner / operator who said they will not stop. Antpool notified F2pool fairly quick, or was it the other way?

I am not claiming some ultra conspiracy exists. I am stating there are pools whose ultimate priority is to make a profit today no matter how it impacts the overall network, any pools outside their circle of trust, and of course any user by making decisions to cut out time.  

When there is a backlog of transactions, the mempool is loaded heavier than it has ever been in the past and for most people the goal is to grow the network and gain adoption the only reason not to include those transactions is to get the block out faster. I think we agree on this, but differ on the the motivation being morally acceptable in our view of what bitcoin is supposed to be.

Being a wishful idealist I want my cake and to eat it with ice cream and pie, which I mean I want profits, and I want a protected network with my pools paying me well and making choices which guarantee the security of a network decades from now.
I think F2pool, Antpool, and a few others care the most about lining their pockets, about how much they can make between now and the halving, and doing it any way possible. These things show intent, and they brag about it.  

As you mentioned SPV mining has cause forking issue but MT blocks do not have a negative impact. I still have the same question of if so, why isn't every pool making that choice?
 




 

[kano]

And yet my test checks the ONLY reason to do it.

If a pool is sending you slower block changes, then the argument to use it is completely nullified.

The reason to do it, is to reduce the amount of time spent mining on stale work.

If what ever the pool does means a certain pool is slower at getting you new work, then ... well it seems blatantly obvious to me ...

Lets look at it this way:
If my pool can get new work extremely quickly ... most of the time ... and your RPi takes 12 days to process a block change ... who cares about your test?

It is quite obvious that doing more work takes more time.
What is also obvious is that the way we process block changes without doing something bad for bitcoin, is faster than that another pool that does the "faster" way this bad for bitcoin.

My point is the more obvious that there are all sorts of things that do affect the pool producing block changes.
While we are not doing one that is specifically bad for bitcoin, we still win.

I'm not trying to compare two specific implementations. My point is that miners will choose to continue to allow empty blocks to be mined because they in general allow them to make more money. Some implementations might be worse than others, for instance, your ckpool software is obviously working better or on better hardware than Eligius's software.

Consider this:
I set up a pool using the exact same software, server config, etc, except I allow empty blocks to be mined. Since I am not using extra cpu time to gather mempool transactions, logically, I will have faster block changes and more blocks than you will.

Consider this, you have a powerful server, a drastically well tuned bitcoind, a very good protected internet connection ... and that overhead suddenly seems really not worth caring about ...
Your test is also considering something else that doesn't affect me

Another thing, you seem to be comparing between you and Eligius. Both only represent a small percentage of the hash power. What about comparing between your and f2pool or antpool? Are their block changes faster than your or not? Have you tried?

Yes but there is something going on there that concerns me even more ...
1 in every few they find, the antpool block shows up on the network ~8s later than when they find it ...
This throws off my stats comparison and I don't know why they are doing this ...
Also there are different random time ranges when you get different results ...

...
Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. It doesn't particularly mess with confirmation times and while it would be nice if those blocks included transactions, it doesn't matter (at least for me) that those transactions wait another ten minutes like they should have

Um, you completely misunderstood something there.
It doesn't matter if 2 blocks are found 1 second apart or 1 hour apart.
They are still counted in determining the future difficulty that determines future transaction confirm times.
If every 2nd block was found as such a pair, then that means the network can only confirm half as many transactions over a given time period.

Right. I forgot about the difficulty retarget. However, those blocks found with extreme time differences (1 sec or 1 hour) essentially cancel each other out in regards to the retarget.

I've no idea what you mean by cancel each other out?
Network difficulty is the factor in determining the average block time.
The time difference between 2 blocks does not affect the speed of the block after it.

Empty blocks simply mean that less transactions per time period can be processed.
They are counted like every other block on the network for determining diff changes.

[achow101]

I believe you will find as with SPV mining 0-1 block transactions do indeed cause issues when they are the goal. If it were something which happened as a byproduct, as with the early days of mining then it is not having an impact. When pools make a conscious decision and not only this one, but choose to have a mindset geared in this manner it speaks volumes regarding their largest motivation.

I haven't seen any evidence of miners intentionally mining empty blocks. To me, they just seem to be a byproduct of their mining operations which probably aren't all that optimized so mining empty blocks still allows them to be competitive. Of course, if they are mining empty blocks intentionally, then they are also losing some Bitcoin from fees (not enough to have a large impact) and hurting the network.

Being a wishful idealist I want my cake and to eat it with ice cream and pie, which I mean I want profits, and I want a protected network with my pools paying me well and making choices which guarantee the security of a network decades from now.
I think F2pool, Antpool, and a few others care the most about lining their pockets, about how much they can make between now and the halving, and doing it any way possible. These things show intent, and they brag about it.  

As you mentioned SPV mining has cause forking issue but MT blocks do not have a negative impact. I still have the same question of if so, why isn't every pool making that choice?

Some pools have better conditions than others. As kano stated

Consider this, you have a powerful server, a drastically well tuned bitcoind, a very good protected internet connection ... and that overhead suddenly seems really not worth caring about ...

Those pools with those conditions and other mindsets as ck and kano have (seems to want to help the network, not just profit) choose not to mine empty blocks because at the very least, there is no incentive. They can make more by including more transactions because of fees. And since they have conditions where the overhead doesn't matter, why not mine full blocks and fully verify blocks?

Also, for Chinese miners and pools, that overhead does matter. They might have a powerful server, probably not a well tuned bitcoind or they have poorly written custom software, and they most likely have a shitty internet connection due to the Great Firewall. This makes them find every possible way to make a profit, which includes having to SPV mine and mine empty blocks.

I've no idea what you mean by cancel each other out?
Network difficulty is the factor in determining the average block time.
The time difference between 2 blocks does not affect the speed of the block after it.

Empty blocks simply mean that less transactions per time period can be processed.
They are counted like every other block on the network for determining diff changes.

I mean that the average of two blocks, one found a few seconds after the previous and the other twenty minutes after the last block, is still around 10 minutes. The affect of those blocks on the difficulty retarget is negligible.

[jonnybravo0311]

I haven't seen any evidence of miners intentionally mining empty blocks. To me, they just seem to be a byproduct of their mining operations which probably aren't all that optimized so mining empty blocks still allows them to be competitive. Of course, if they are mining empty blocks intentionally, then they are also losing some Bitcoin from fees (not enough to have a large impact) and hurting the network.

You won't see it, either.  A miner doesn't control whether or not the block is empty, the pool does.  One could argue that as a solo miner, I could configure my bitcoind to do something silly like "do not include any transactions that have a fee lower than 100BTC".  In that case, if I ever happened to find a solo block, then it would very likely be empty.

Also, for Chinese miners and pools, that overhead does matter. They might have a powerful server, probably not a well tuned bitcoind or they have poorly written custom software, and they most likely have a shitty internet connection due to the Great Firewall. This makes them find every possible way to make a profit, which includes having to SPV mine and mine empty blocks.

If they have a poorly tuned bitcoind, poorly written custom software, etc. shouldn't they address those problems?  They don't.  Instead they remain lazy.  Bitmain has proven time and again they can't write software worth a damn, and they just don't do anything about it.  Even with kano including the Bitmain drivers in mainline cgminer, Bitmain still uses its own crap fork.  There are pools and pool software out there proven to work faster and be better for the health of the ecosystem.  If these players are so worried about their profit, don't you think they'd do everything they can to maximize it?  SPV mining and empty blocks is not the answer, yet these pools and their operators fall back on it just the same.

[drakoin]

Sorry for waking up this thread.

I am thrilled to see all the data analysis since my last post here. Especially the time evolution looks like a real signal. Well done. Very interesting, thanks a lot to organofcorti, and jonnybravo0311; and everyone who asked the right questions.


Catching up with my reading ... I might have one more such question:

...

... if spv mining and 0-1 transaction blocks lead to a higher success rate without having a negative impact on the network and its users / miners why wouldn't every pool do it?

Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. ...

If that ("within seconds of the previous block") is empirically found, let's close the case.

But ...

... empty blocks negatively impact the network. They are typically found within seconds of the previous block ...

It should be non-difficult, to aggregate the block generation times of all those 1-tx-blocks ... per mining pool. By giving arithmetic mean, and standard variation of the 1-tx-block-generation-times PER POOL. Together with the number of such 1-tx blocks per pool, it could give an estimate of the statistical significance of the two hypotheses "just a mining software effect" versus "deliberate attempt at freeriding".

[eleuthria]

It doesn't matter if the blocks were found within seconds of each other unless you're looking at pre-2013 blocks.  I doubt there has never been a point in the last 2 years where there were 0 legitimate unconfirmed transactions on the network.

[achow101]

It doesn't matter if the blocks were found within seconds of each other unless you're looking at pre-2013 blocks.  I doubt there has never been a point in the last 2 years where there were 0 legitimate unconfirmed transactions on the network.

Perhaps if you read the other stuff I posted about how those blocks found within seconds of the previous you would know why that can happen. I'm not going to repeat myself.

Sorry for waking up this thread.

I am thrilled to see all the data analysis since my last post here. Especially the time evolution looks like a real signal. Well done. Very interesting, thanks a lot to organofcorti, and jonnybravo0311; and everyone who asked the right questions.


Catching up with my reading ... I might have one more such question:

...

... if spv mining and 0-1 transaction blocks lead to a higher success rate without having a negative impact on the network and its users / miners why wouldn't every pool do it?

Because it negatively impacts the network. As we have seen with SPV mining, there are issues that can occur and cause blockchain forks and weaken Bitcoin. However, I personally don't think empty blocks negatively impact the network. They are typically found within seconds of the previous block, so for transactions, it really is just like a block was found with two block rewards. ...

If that ("within seconds of the previous block") is empirically found, let's close the case.

But ...

... empty blocks negatively impact the network. They are typically found within seconds of the previous block ...

It should be non-difficult, to aggregate the block generation times of all those 1-tx-blocks ... per mining pool. By giving arithmetic mean, and standard variation of the 1-tx-block-generation-times PER POOL. Together with the number of such 1-tx blocks per pool, it could give an estimate of the statistical significance of the two hypotheses "just a mining software effect" versus "deliberate attempt at freeriding".

You can run the numbers, let us know what you find.

[p3yot33at3r]

Bumping for awareness & educational purposes.....

[TracerX]

Bumping for awareness & educational purposes.....

I've built a little tool to help visualize these blocks at bitblk.com.  F2Pool loves to drop empty BTC blocks, but it slaughters the Litecoin network.  I'm amazed more people aren't aware of this.

I'll be adding a bit of copy that describes the issue for users that may be unaware--if we keep talking about it maybe we can affect change!

Cheers.

[Nexious]

Bumping for awareness & educational purposes.....

I've built a little tool to help visualize these blocks at bitblk.com.  F2Pool loves to drop empty BTC blocks, but it slaughters the Litecoin network.  I'm amazed more people aren't aware of this.

I'll be adding a bit of copy that describes the issue for users that may be unaware--if we keep talking about it maybe we can affect change!

Cheers.

Handy little website TracerX thanks adding to my bookmarks

[TracerX]

Bumping for awareness & educational purposes.....

I've built a little tool to help visualize these blocks at bitblk.com.  F2Pool loves to drop empty BTC blocks, but it slaughters the Litecoin network.  I'm amazed more people aren't aware of this.

I'll be adding a bit of copy that describes the issue for users that may be unaware--if we keep talking about it maybe we can affect change!

Cheers.

Handy little website TracerX thanks adding to my bookmarks

Cool man, thanks.  I built it for my mobile so I could obsess over my pool without having to log in or sit there and try to count them one at a time!

[Sir_lagsalot]

Cool. Is it possible for you to open-source this program? I have a few ideas for it.

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Are you saying that some pools can opt out of mining empty blocks? I thought you don't know what's coming until you finish mining the block :-/

I guess you learn something new everday day!

[kano]

Cool. Is it possible for you to open-source this program? I have a few ideas for it.

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Are you saying that some pools can opt out of mining empty blocks? I thought you don't know what's coming until you finish mining the block :-/

I guess you learn something new everday day!

The contents of the 'block to be' are decided by the pool before it sends out the work to the miners.

It's rare if ever that there are no transaction available on the network.

Many pools send out work to their miners that includes no new transactions other that the necessary coinbase transaction required to build a block.
That's what all the "Empty Block" stats are about.

My pool https://kano.is/ doesn't send out work containing empty blocks
(unless there are no transaction available in the pool's bitcoind - which is probably never)

[os2sam]

Cool. Is it possible for you to open-source this program? I have a few ideas for it.

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Are you saying that some pools can opt out of mining empty blocks? I thought you don't know what's coming until you finish mining the block :-/

I guess you learn something new everday day!

As Kano explained, pools can choose whether they mine empty blocks or not.  Evidently it used to be common for some pools to mine either empty or very small blocks.

My old message was asking what pools were still doing that on a regular basis.  I think the answer was none except for the initial work immediately after a block find.

[multicurrency]

I have been up and down and in and out of the bitcoin codebase and have pondered many, many design characteristics of bitcoin, some seem absolutely unreasonable on first thought, especially given how easy they would be to change with a few lines of code. However, for each, after much thought and reasoning, I have realized the subtle brilliance of some of these seeming mistakes.

From my experiences, I am confident that the presence of empty blocks is a design feature that was considered and their presence is desirable. That we haven't figured out the logic behind their existence does not mean that they are a bad thing, it just means that we haven't thought hard enough about why they strengthen the protocol.

I haven't thought about this issue, so I can't give a good reason why they are permitted.

[achow101]

I have been up and down and in and out of the bitcoin codebase and have pondered many, many design characteristics of bitcoin, some seem absolutely unreasonable on first thought, especially given how easy they would be to change with a few lines of code. However, for each, after much thought and reasoning, I have realized the subtle brilliance of some of these seeming mistakes.

From my experiences, I am confident that the presence of empty blocks is a design feature that was considered and their presence is desirable. That we haven't figured out the logic behind their existence does not mean that they are a bad thing, it just means that we haven't thought hard enough about why they strengthen the protocol.

I haven't thought about this issue, so I can't give a good reason why they are permitted.

Because without empty blocks, times when there are no transactions (like in the beginning of Bitcoin) requires empty blocks. Otherwise the network could never grow and it would never have worked. The code that allows them must stay in in case of situations that may arise due to people not spending for some reason. It allows mining to continue at such times.

[sloopy]

This is where your morals as a miner and ultimately a human being who has an impact on the Bitcoin Ecosystem and by proxy the world and you must make a decision to support pools who mine empty blocks when there are plenty of transactions waiting to be handled.

These pools who choose to spv mine and mine 0 transaction blocks do not deserve anyone's loyalty because of the choices they make. The people who make the decisions for these pools do not care about the future of bitcoin outside of what it makes for them tomorrow. They do not care if in 10, 20, or 50 years bitcoin is worthless, they do not use their profits to help the miners, they do not give their time to help the ecosystem, they only care about them, and supporting such is complete greed and a lack of a moral compass.

Now when I say morals I am not talking about something as complicated as a religious discussion. I do not mean a philosophical expression which one could contemplate for weeks, I am talking about the difference between black and white, good and bad,  dirty or clean, being a thief, or doing the best someone can to treat others as they want to be treated.

This is about as simple as any conversation or debate could be and anyone denying that mining an empty block when there are valid transactions waiting to be processed does not have reality in their sights.

F2pool will tell you they are just a bunch of geeks trying to get by but what they will not tell you is that they collude with a couple of other of the largest pools which exist to provide each other with block confirmations in China before the rest of the world has an opportunity.

What you should ask yourself is have you shown yourself you cannot make just as much if not more coin by mining at a pool which does not mine zero transaction blocks. People could make just as much money and help the network grow while securing it for the future by mining at pools who do not spv mine and mine zero transaction blocks.

There are many things allowed by code. Just because the code doesn't stop you from doing something doesn't mean it is something you should do. We all know the people contributing to bitcoin core have immense amounts of things to address and they should do so with the interest of the network security and longevity at heart.

We have already had forks because of the spv mining and we see transactions skipped by the largest pools every single day. Does anyone think this will translate to helping the network grow? If so, please explain how not processing transactions will help bitcoin? The more people who sit waiting for a transaction to process then the more people we have thinking that bitcoin isn't all good and well. The more miners who grow frustrated seeing those same pools grow larger begin to question bitcoin as well. Eventually things like this can be a breaking point and this is where morals can come into play.

Imagine you have provided all of the miners on the bitcoin network a fantastic sum of money to do nothing but continue mining wherever they choose, as long as it helps the network grow and provides the security to do so. Removing the economic reason for mining and place all decision making on the individual miner to choose the pool they mine at not based on how much coin they make, but how much they think that particular pool will help bitcoin grow and trust the pool owners / operators to make the best decisions for the bitcoin network.
What do you think the hashrate charts would look like then? Anything similar to today's distribution?

Again, consider these things yourself, consider the choices you have made and continue to make regarding a variety of things in life. Isn't it always better in the long run to do the right thing?
If it wasn't right no one would care. If I hadn't seen these things first-hand, if I hadn't been exposed to the pure greed which drives one of the biggest pools I wouldn't care. If I hadn't been shown the empty blocks and began to do my own research, began to ask my own questions and also have a natural curiosity to question the status quo I literally would never post about it because it wouldn't matter.

Please don't take my word for it. Look at these things on your own and if you care that bitcoin puts its best foot forward to the world then start by showing them that bitcoin is based on people who make decisions to better the bitcoin community for the long haul. Allowing F2pool and Antpool to continue mining with these tactics shows the corporate world this is a simple extension of their banking system for the taking, and that is exactly what is happening. As more control flows to these central locations even if the power was never used (imagine a company having power over bitcoin and not abusing it), even if those companies didn't choose to perform further acts which harm the community while filling their pockets they have the opportunity to do so.

Maybe a different viewpoint will help some people, and if you are driven by pure greed then at least be a person who stands up for what they believe in and sincerely open your eyes to long term growth. Where do we go at this pace? More new miners join those pools because they are the largest and those pools continue to get larger. You may get daily payouts (albeit much smaller payouts than other pools), but that is short sighted, you can still make just as much if not much more mining at a pool which is not setup to spit out zero transaction blocks.

Someone mining for the act of purely making more coin can even see that as those pools grow larger and gather more power they position themselves to control the network. Consider BIP101. It doesn't matter if you supported it or not, but the fact that F2pool and Antpool could push the network to BIP101 if they choose to do so is not at all what any true supporter of bitcoin would ever choose, much less someone mining for pure profit. Because they originally were quoted as saying they would support a revision of BIP101 but then did not do so, and regardless of the exact semantics why on earth would anyone want to see such an immense amount of control over the complete bitcoin universe in the hands of a small group of people who have shown consistently they only care about themselves.

What choices do we always see greedy corporations make once they become a monopoly? They make choices that make them more money. ...And who supplies the additional income for them above what they are making now? The very customers who provided them with the ability to become that monopoly. We do not live in a fair world and I think most miners know this. Regardless of the fact that some of us (myself included) may enjoy the occasional immature joke, or even act immature when calling out someone on the forum, we know right from wrong and when removing ourselves mentally from the situation enough to think clearly can see what is happening.
Those of us old enough to remember other monopolies, or the people who see the ones standing today as the local cable company and AT&T work together to make sure you pay out the wazoo every month for less than half or worse of the internet speeds and caps of the rest of the world, well, I know that if I'd understood this twenty five years ago I would not have continued to give those companies my money. The majority of miners are doing the same thing today. Do we need governments telling us where to mine? Do we need the governments who have proven repeatedly not to have our best interests at heart telling us where to point our hash. Because if / when the government gets involved the only monopoly is theirs. They may let F2pool keep the name, but they certainly will no longer have control. But we are a long way from anything like that. We have many years of the monopoly of the few before we get to that extreme, and we have the opportunity to make a choice.

You may quietly continue pointing your hash at a pool where you think you make more money for whatever reason regardless of how it impacts the network and there is nothing anyone can do about it. Actually, you are part of the majority by continuing to do so. You may quietly kick back and only care about today, and you may even have yourself convinced that your hashrate doesn't matter. Hell they already have so many PH of the network it doesn't matter what you choose, you may be heading out the door to go out on Friday night and spend some of your coin and you don't have time for what you see is a trivial thing that has no bearing on you.
... Just like casting a vote, it matters more than you will understand until you no longer have a choice.

Take a few minutes, say 20 minutes and read some of the opinions of the people on this forum who are posting about these scenarios. JohnyBravo doesn't have a dog in this fight in the manner of pool versus pool, but look at his opinions. I've followed JB's posts on a couple of different forums and firmly believe he has a good handle on the bitcoin landscape from the home miner perspective. He has nothing to lose or gain by making the statements he does regarding zero transaction mining. Why would he display his research and opinion, or spend his time delving into things such as this if it weren't fact.
I would not care what any pool did if it didn't harm the future of this gorgeous creation called bitcoin. I know I am pretty much small potatoes as miners go but through everything I've learned about bitcoin is to some people it is their personal money making machine and nothing more. They care about today and into tomorrow. They care about keeping the cash train rolling, but they do not have the goal of changing the financial world forever and leveling the playing field, they do not care about the people paying hundreds of millions of dollars in remittance fees, the people who have to spend 5, 7, or even 10 to 20 % of their minimum wage paycheck just to send money back home so their family can eat while they try to save up enough money to move their family to their new country.

There will only be a few people who make real money with bitcoin and it will not be the miners. I hate to break it to the people who think they will be millionaires from mining coin but unfortunately those of us who weren't around a few years ago are catching the tail end of what could be a beautiful thing.  This is because the more control those pools gain, the less people will be convinced this is a secure system. See, it doesn't matter if the actual miners are decentralized if all of the hashrate is going to 2 or 3 pools who work in collusion with each other. They have the ability to do all sorts of evil deeds with it, and you haven't began to understand the mindsets involved if you do not think they will use that power to gain even more power. It is as obvious as the sky being blue and if miners do not wake up and begin making choices to turn this car around none of the other small stuff is hardly worth consideration. It doesn't matter what BIPs are implemented, it doesn't matter if the core programmers make any changes, and it doesn't matter what the price of bitcoin does because the same small group of people will control it.

You and I both are watching them murder bitcoin in broad daylight. They have the power and are given more everyday, so they will make the rules. Back doors into back rooms where the deals are truly made by people who will soon if not already control what used to be the network belonging to the people of the world. Are you going to continue sitting idly by while they control you? You are being manipulated. When kids read about bitcoin in history books will they read about how it was destroyed by greed, or will they read about how the miners made the right choice and changed the way the world works forever? I know one thing they will read, and that is that there were groups of miners who were not on the payroll of the monopoly. I hope to see your name next to mine along with thousands of others who finally said, enough is enough and took a stand.

If driven by pure greed then calculate your earnings and compare them to what you could have earned at kano.is over the past year.
It isn't that I agree with every choice made by that particular pool, but I know the operator has bitcoin survival and proliferation as his top priority. I do not believe he is driven by corporate investors, and I know he is smart, and has good morals.
You will be surprised at the earnings you are missing. There are other good pools as well. There are certainly other pools who do not subscribe to a quick buck above anything else mentality.

If you are driven by your conscience and involved for a variety of reasons you already know the right choice to make, you just need to execute it and show the world where your heart and mind were during the bitcoin revolution.

Either way, wake up and wipe the sleep from your eyes, it is time for you to make a move.

[p3yot33at3r]

Bravo again sloopy 

[sloopy]

Bravo again sloopy 

Thank you, you are kind.

I know most people will say it is too long, too much to read, and many or most will skip on by, but if maybe a few people will read and contemplate some of these things we can have an impact. Even one miner is better than none, and I know there are many great people mining at those pools who just haven't been exposed to an opportunity where the facts have been presented as they are in this thread.

I think one of the best things is that people could make fantastic amounts of money using their hashpower if they used it wisely. Making those pools compete more for your hashpower is what will drive bitcoin into the future.

Every single miner should consider the power they have, and the possibilities are huge by simply changing the pool you mine at for a short amount of time, IE a couple of months just to see,  as it is a vote to say no, you want bitcoin to be the best it can be and it will be controlled by no one and everyone but never the small few.

I do appreciate the kind words, and please help pass the word so more people consider their choice in where they mine. That single choice has the power to make this one of the most fantastic times to ever be alive. Lets make sure we do not give up our right to mine where we want and see bitcoin evolve into the infrastructure of the most beautiful financial system this universe will ever see.

[727miner]

Well said Sloopy! I agree with your points and that's why I choose to mine at CK's solo pool as well as Kano's pool. I feel that CK & Kano are truly looking out for and helping the little guys have a chance against these Mega pools. I hope others consider their choices and do the same. Peace 

[opentoe]

Greed will be the demise of Bitcoin. It is interesting though.

[thebonenz]

Thanks for this thread it opened my eyes a lot! and now I feel dirty...dirty but informed about my poor previous decisions haha

[enthus]

They relay and verify previous transaction, they are not useless I do not think.

[opentoe]

Bravo again sloopy 

Thank you, you are kind.

I know most people will say it is too long, too much to read, and many or most will skip on by, but if maybe a few people will read and contemplate some of these things we can have an impact. Even one miner is better than none, and I know there are many great people mining at those pools who just haven't been exposed to an opportunity where the facts have been presented as they are in this thread.

I think one of the best things is that people could make fantastic amounts of money using their hashpower if they used it wisely. Making those pools compete more for your hashpower is what will drive bitcoin into the future.

Every single miner should consider the power they have, and the possibilities are huge by simply changing the pool you mine at for a short amount of time, IE a couple of months just to see,  as it is a vote to say no, you want bitcoin to be the best it can be and it will be controlled by no one and everyone but never the small few.

I do appreciate the kind words, and please help pass the word so more people consider their choice in where they mine. That single choice has the power to make this one of the most fantastic times to ever be alive. Lets make sure we do not give up our right to mine where we want and see bitcoin evolve into the infrastructure of the most beautiful financial system this universe will ever see.

I started to mine bitcoin to gain a little profit. Pre KNC days. I think I've tried every pool that was available.  The most profitable pool I used was BTCGuild. The uptime was probably %99 or better. When the Guild disappeared I went on Ghash (ya know, that pool/company that was going to take over the Bitcoin network) but eventually dropped on Slush's for a while. Wasn't having a good time there so if I can remember I think I hopped on Bitminter. Then Eligius somewhere in there. I'm not talking about just a few days on each pool, I'm talking at least a month or more on each pool. I ended up on Antpool. Always got constant payouts and according to the small hashing power I pointed there it was correct. Then read "bad" things about Antpool and it wasn't good to mine on these Chinese pools because once again they are getting too large and going to take over the world. Well, if you already live in the US you are aware that our economy sucks and basically import everything from China. Our companies and jobs also went there too. Ok, so I pointed 10TH to CKPool and ran for about a 2 months. With all the re-starts or firmware issues, who knows, I could never get the same payouts on that pool as I did with Antpool. Even when I switched back to Antpool after my try with CKPool, my payouts were more. It is nice telling people to mine on certain pools because technical issues that probably %90 of miners have no idea of.  I can't write as elaborate as user sloppy, but what is the little guy like me who's not making any real money switch to a pool that gets less payouts then a hated pool? Everyone is mining to make a profit and they stay with the pool at which makes them the most profit. Hence, greed comes into play and all I want to do is get as much bitcoin in return to pay my electric bill, and maybe some "dust" after that.

[Monnt]

Do pools value empty blocks, or do they try to not mine them? What really is the point of empty blocks, I thought bitcoin has transactions every single block?

[jonnybravo0311]

It depends on the pool.  Some pools use code trickery to try and get work out to their miners as fast as possible.  The technique is to publish an empty block as quickly as possible, then to update it with transactions afterwards.  Well, it can happen that a miner solves the empty block.  Many pools take this approach, and the bigger the pool, the more often you'll see empty blocks.  Other pools fill up the block with as many transactions as possible an then publish the work out to miners.  These pools will never submit empty blocks because their miners are never given an empty block on which to work.

There is nothing in the bitcoin protocol that forces a block to contain transactions other than the coinbase one.  You could feasibly setup a pool that would only ever mine empty blocks.  The downside to mining empty blocks is that plenty of transactions out there will now have to wait until somebody else includes them in their blocks.  With the largest pools out there mining empty blocks, it's quite possible that you'd see a whole string of empty blocks in a row on the blockchain all the while new transactions are waiting to be confirmed.

[thirdchance57]

Here is the question I am pondering and I did not see a comment answering this, but would empty blocks allow future access into the BTC system in order to scam the chain somehow?

[achow101]

Here is the question I am pondering and I did not see a comment answering this, but would empty blocks allow future access into the BTC system in order to scam the chain somehow?

How? What do you mean to scam the chain?

The answer to whatever scenario you think of is probably not. There is nothing special about empty blocks. They don't have any special properties, they are just blocks that only have the coinbase transaction. In general they are basically a nuisance, they don't really add anything to the blockchain but they aren't detrimental and harming it.

[nepaluz]

This whole empty blocks cum spv mining is simply "too much ado about nothing"!

The way I see it is there will be a time when mining empty blocks will neither be in the interests of the pool nor the miner, assuming they are separate entities and otherwise, therefore the empty blocks will die a natural death. Before that threshhold is crossed, it can not be reasonably assumed that empty block mining is or can be detrimental to the bitcoin ecosystem, infact, it would be foolish to even think so and downright silly to assert so.

I should add that the only time I've noticed transaction confirmation delays is when the network is under attack and not when empty blocks have peaked. It would also be interesting to establish what percentage of network-wide transaction confirmations is attributed to the pools being pillored for empty block / spv mining against those that do not regularly mine empty blocks.

The only pseudo-substantive argument that I have seen put forward against empty block mining boils down to a veiled effort to lure miners to certain pools, no more.

[jonnybravo0311]

Nepaluz, unfortunately you're probably correct about there being a tipping point where pools will realize that mining empty blocks is not beneficial for them.  Right now, the generation of 25BTC far outweighs any transaction fees included in a block.  Honestly, I don't understand why AntPool would ever mine an empty block, since they keep the transaction fees for themselves.  It would behoove them to stuff as many large-fee transactions as they could into their blocks.  F2Pool charges 4%, so until the transaction fees are greater than 1BTC per block, or 0.5BTC per block after the halving, they've got absolutely no incentive to put transactions in their blocks.

Where I will disagree with you, however, is in your assertion that SPV mining is not detrimental.  SPV mining has previously caused forks in the blockchain.  The very process itself is unstable, using only header information instead of verifying the actual blocks.

[nepaluz]

Johnybravo, I do not agree with you on the specific issue of the process of SPV mining being unstable in itself, I rather think it is the people who code it that flagrantly implement it - and that is a nod with particular reference to the mentioned forking.

On top of that, forking is not exclusively caused by SPV mining but more importantly there's no harm to "rest of us" when a fork of this nature happens. Infact, forks do happen and the protocol knows about it, thus it can not, by definition, be detrimental.

[ShrykeZ]

Cool. Is it possible for you to open-source this program? I have a few ideas for it.

Can you tell what pool(s) are currently mining 0 transaction blocks consistently?
Thanks,
Sam

Are you saying that some pools can opt out of mining empty blocks? I thought you don't know what's coming until you finish mining the block :-/

I guess you learn something new everday day!

The contents of the 'block to be' are decided by the pool before it sends out the work to the miners.

It's rare if ever that there are no transaction available on the network.

Many pools send out work to their miners that includes no new transactions other that the necessary coinbase transaction required to build a block.
That's what all the "Empty Block" stats are about.

My pool https://kano.is/ doesn't send out work containing empty blocks
(unless there are no transaction available in the pool's bitcoind - which is probably never)

This is why people should switch from the larger usually Chinese based pools over to pools like Kano's or others that operate similarly.

[kano]

Johnybravo, I do not agree with you on the specific issue of the process of SPV mining being unstable in itself, I rather think it is the people who code it that flagrantly implement it - and that is a nod with particular reference to the mentioned forking.

On top of that, forking is not exclusively caused by SPV mining but more importantly there's no harm to "rest of us" when a fork of this nature happens. Infact, forks do happen and the protocol knows about it, thus it can not, by definition, be detrimental.

Well it only takes one miner to produce a block with 1 bad transaction in it, and that can push ~65% of the mining power off the bitcoin blockchain.
Not detrimental?

The law covers the issue of dealing with someone killing someone else.
However, killing someone else is usually considered detrimental ...... by definition.

[nepaluz]

Ofcourse it does, and mangoes are prunes though just like not all blockchain forking is due to SPV mining.

[kano]

Ofcourse it does, and mangoes are prunes though not all blockchain forking is due to SPV mining.

At least reread what you wrote here:
https://bitcointalk.org/index.php?topic=1085800.msg13205770#msg13205770
Yes you can't delete it coz I quoted it also.

[nepaluz]

Is it possible to delete messages? Well I'll spell it out for you then, not all prunes are mangoes.

[jonnybravo0311]

I don't think it was ever claimed that forking was exclusively caused by SPV mining.  You made that statement.  However, what is true is that SPV mining can and has caused forks.

Just for fun, I re-ran things again to see where we are.  As of block 387696:

AntPool 7.61% of their blocks are empty
BW.com 4.14%
f2pool 3.55%
KnC 3.23%
Eligius 2.3%

[nepaluz]

You indeed never claimed such, and your numbers should tell a good story ... the one you refuse to see and tell, so I'll ask for some pointed stats.
1. What is the percentage of forks / empty blocks?
2. What is the percentage of forks / all blocks?
3. What is the percentage of forks / empty blocks in a timespan of the highest concentration of forks (in which timespan at least 2000 blocks have been mined)?
4. What is the percentage of forks / all blocks in a timespan of the highest concentration of forks (in which timespan at least 2000 blocks have been mined)?

The answer to all of those is bound to be in fractions of a percentage; extrapolations will without doubt yield similar results, meaning the likelihood of forks happening due to empty block mining is statistically negligible, and that will be true for a very long time.

So if you base your argument of empty block mining being detrimental due to the evil that is forking, which said forking's occurence is statistically negligible, you may have an agenda but you definitely have no point.

[jonnybravo0311]

I welcome you to perform those calculations.  I'm simply providing the data.  I've also never made the conclusion you came to:

base your argument of empty block mining being detrimental due to the evil that is forking, which said forking's occurence is statistically negligible, you may have an agenda but you definitely have no point.

There is nothing in this thread stating empty block mining is detrimental due to forking.  In fact, I've never even tied the two together.

I'm not sure what agenda you think I may have, other than to report the numbers.

[philipma1957]

We had the largest block count day in years on the 10th of dec

 215 blocks.

we had more then 20

 1 and 2 transaction blocks.

The  purpose of BTC is to move payments via transactions.


a 10+ percent of no transaction blocks is not good.

It clogs up the payment system.   If the 20 tiny transaction blocks all had 1000 transactions we would have moved  20,000 transactions.

We would not have this happening

blockchain info skipped auto payment of fees pulled excess coin out of account sent it back causing an un confirmed .0101 amount

from a fully confirmed original balance of .202

https://blockchain.info/tx/92a7cba0eac341534e79362b2d9c2dba42ee3479d69bd347c036f5c778186c6d

thus 1 minute later when blockchain info made this payment with a fee they did won't confirm this .151 transfer

so .202 fully confirmed btc is now locked up as unconfirmed due to 2 issues. first blockchain info client forget to pay a fee and needlessly casued an unconfirmed .0101

second dozens of empty blocks passed by these two transactions which if you do both of them you do get .0001 btc

So the entire point of mining which is moving coins via transactions is lost here.  Why some pools just want block rewards.

Well we will see how long the .202 stays tied up and pools continue to mine empty blocks to pass by the fees to get rewards.

More then 30 hours now.

https://blockchain.info/tx/3989f4a2d7cdbcd146e5b9df949c514fd5920e66e9486d4c18bf1681ab7ba3b7

[Rizky Aditya]

I don't really understand the issue here. Does the miner earn less if he/she finds/mines an empty block? I'm pretty sure that he/she would earn the same amount.

[kano]

I don't really understand the issue here. Does the miner earn less if he/she finds/mines an empty block? I'm pretty sure that he/she would earn the same amount.

No.

[leowonderful]

lol antpool mines yet another empty block https://blockexplorer.com/block/000000000000000001a7ce5d59bd2b32422884114adeadbdb5427ffb3836e47e

[-ck]

lol antpool mines yet another empty block https://blockexplorer.com/block/000000000000000001a7ce5d59bd2b32422884114adeadbdb5427ffb3836e47e

No need to point out each and every one of them. It's a regular occurrence.

[jonnybravo0311]

So... with the backlog of transactions currently plaguing the network, one would think that the pools would be constantly pushing full blocks onto the chain.  Yeah.  Here's how it is really stacking up since last Monday (block 399383 to block 400871):

AntPool - 44/341 blocks mined are empty (12.9%)
bw.com - 8 of 103 blocks mined are empty (7.77%)
f2pool - 4 of 382 blocks mined are empty (1.05%)
KnC - 3 of 58 blocks mined are empty (5.17%)
Eligius - 2 of 9 blocks mined are empty (22.22%)
Slush - 1 of 69 blocks mined are empty (1.45%)

If we take an average of 1500 transactions per block, which is low by the way, then had AntPool had even half as few empty blocks there would be no transaction backlog at all.  Everybody's transactions would be confirmed.  There wouldn't be pages and pages of questions in the forums asking why a transaction hasn't been confirmed after days of waiting.

For the love of all that's holy - STOP MINING ON POOLS THAT PRODUCE EMPTY BLOCKS!!!

[os2sam]

Thanks for the update.

[valkir]

They want bigger block so they do empty block.  So idiot.

[TracerX]

So... with the backlog of transactions currently plaguing the network, one would think that the pools would be constantly pushing full blocks onto the chain.  Yeah.  Here's how it is really stacking up since last Monday (block 399383 to block 400871):

AntPool - 44/341 blocks mined are empty (12.9%)
bw.com - 8 of 103 blocks mined are empty (7.77%)
f2pool - 4 of 382 blocks mined are empty (1.05%)
KnC - 3 of 58 blocks mined are empty (5.17%)
Eligius - 2 of 9 blocks mined are empty (22.22%)
Slush - 1 of 69 blocks mined are empty (1.45%)

If we take an average of 1500 transactions per block, which is low by the way, then had AntPool had even half as few empty blocks there would be no transaction backlog at all.  Everybody's transactions would be confirmed.  There wouldn't be pages and pages of questions in the forums asking why a transaction hasn't been confirmed after days of waiting.

For the love of all that's holy - STOP MINING ON POOLS THAT PRODUCE EMPTY BLOCKS!!!

F2Pool has really cleaned up it's act, at least with BTC mining.  They had a higher than 1% weekly average for a long time--I suppose I'll start a thirty/60/90 day stat. 

[jonnybravo0311]

Yes, they've certainly gotten better about submitting empty blocks.  Unfortunately, they still SPV mine and they are still submitting empty blocks, albeit far less frequently.

[watashi-kokoto]

they are still submitting empty blocks, albeit far less frequently.

empty blocks result from bursts

[Lauda]

empty blocks result from bursts

No? They're the result of SPV mining. This means that the miners don't verify the block nor transactions within it but start immediately mining a new block (referencing only the header). They have to mine it without any transactions since they don't know what was included in the last block. A empty block will propagate faster through the network and a full block would most likely get orphaned (if both are solved in similar times). This is why some of the miners are doing this.

Yes, they've certainly gotten better about submitting empty blocks.

They need to completely stop doing this.

They want bigger block so they do empty block.

Miners can impose their own limits anyways.

[Mikestang]

empty blocks result from bursts

Uh, what is a "burst"?  Whatever it is, it's not why empty blocks happen.  Empty blocks come from bad/lazy coders looking for a shortcut.

A empty block will propagate faster through the network and a full block would most likely get orphaned (if both are solved in similar times).

Not true at all, kano.is has about the fastest network block propagation and never mines empty blocks.

[organofcorti]

empty blocks result from bursts

Uh, what is a "burst"?  Whatever it is, it's not why empty blocks happen.  Empty blocks come from bad/lazy coders looking for a shortcut.

A empty block will propagate faster through the network and a full block would most likely get orphaned (if both are solved in similar times).

Not true at all, kano.is has about the fastest network block propagation and never mines empty blocks.

Are you sure about that last part?

[Lauda]

Not true at all, kano.is has about the fastest network block propagation and never mines empty blocks.

If that were true, why would miners bother mining empty blocks in the first place? They lose out on fees by doing so.

[jonnybravo0311]

On AntPool miners don't get the fees anyway, the pool does.  Of course one wonders why anyone mines on that pool at all...

[Mikestang]

Are you sure about that last part?

Yes.

Not true at all, kano.is has about the fastest network block propagation and never mines empty blocks.

If that were true, why would miners bother mining empty blocks in the first place? They lose out on fees by doing so.

It's not "if" it were true because it is true.  Read Kano's posts about block changes, they do it faster than the guys who mine empty blocks.  The empty block miners do it because they are lazy coders who don't care about bitcoin or the network, all they see is $$$.

[organofcorti]

Are you sure about that last part?

Yes.

I think you meant block change response rather than block propagation.

Block propagation is different to a pool's response to block changes. Block propagation is related to how connected a pool is to the network, block change response seems to be a software issue.

I've read that Kano's response to block changes is very quick, but I haven't read that Kano's block propagation times are that much better than any one elses. If I'm wrong, please post a link.

[Mikestang]

Off the top of my head I could only come up with this post by ck:
https://bitcointalk.org/index.php?topic=789369.msg13822317#msg13822317

15 mins ago we got another block

  okay I see it on blocktrail but not blockchaininfo

The block explorers seem to really struggle around the time the orphans hit. Sometimes they may even need manual intervention to keep working it seems... blockchain is showing Eligius as having won that block yet if you click on their block it then says orphan. Regardless, it's our block as all nodes have synced up to ours and confirmed it numerous times since. Our extensive propagation network (and numerous ckpool improvements to distant nodes) may well have paid off on saving that one.

[Lauda]

It's not "if" it were true because it is true.  Read Kano's posts about block changes, they do it faster than the guys who mine empty blocks.  The empty block miners do it because they are lazy coders who don't care about bitcoin or the network, all they see is $$$.

Okay exclude Kano then. Take Antpool and F2Pool for example. Wouldn't e.g. Antpool's empty block propagate faster than F2Pool's full block? After all, Antpool has the highest percentage of empty blocks.

[jonnybravo0311]

Take a look here: https://poolbench.antminer.link

That site gives a good breakdown of how the pools propagate blocks.

[TracerX]

Take a look here: https://poolbench.antminer.link

That site gives a good breakdown of how the pools propagate blocks.

Interesting data, thanks.

[David Rabahy]

Take a look here: https://poolbench.antminer.link

That site gives a good breakdown of how the pools propagate blocks.

Please help me understand this data.

[Mikestang]

Okay exclude Kano then. Take Antpool and F2Pool for example. Wouldn't e.g. Antpool's empty block propagate faster than F2Pool's full block? After all, Antpool has the highest percentage of empty blocks.

I don't think a block being full or empty has anything to do with its propagation speed, i.e. I don't think there is any valid reason for producing empty blocks.

[jonnybravo0311]

Take a look here: https://poolbench.antminer.link

That site gives a good breakdown of how the pools propagate blocks.

Please help me understand this data.

I didn't write it, so I can only provide insight based on my own observation and running my own pool.

I presume the owner of that site has a number of cgminer processes running, each connecting to a pool (i.e. f2pool, Ant, kano, bravo, etc).  As and when each cgminer instance reports a block notification from the pool it is connected to, he writes the information to a data store.  He then correlates the data and presents it as you see on the web page.

The summary at the top shows over the past 100 blocks how each pool he's connected to fared.  The first column is the pool, the second is the average time difference between when that pool notified the block was found vs the first pool to provide notification.  The third column is the average of how the pool did subtracting out the best (i.e. how it compared to the number 1 pool).

The data below shows the values for each individual block.

[Lauda]

I don't think a block being full or empty has anything to do with its propagation speed

So propagating a 10kB block and a 1 GB block will take the same amount of time? This doesn't make sense.

i.e. I don't think there is any valid reason for producing empty blocks.

I do agree that they should not be doing this.

[jonnybravo0311]

If we agree that all pools are equally well connected to the network, then obviously broadcasting an empty block (~200 bytes) vs a full block (~1,000,000 bytes) will be faster.

[Mikestang]

If we agree that all pools are equally well connected to the network, then obviously broadcasting an empty block (~200 bytes) vs a full block (~1,000,000 bytes) will be faster.

I don't think it's that obvious.  When does a node "timestamp" the block it's receiving, when it first starts seeing data, or when the data is 100% received?  I think it's when it first sees data coming in, therefore rendering the size of the block a moot point with respect to propagation rate.  We're not comparing 10k to 10,000,000k, we're comparing 200k to 800k and with today's technology those values are essentially the same.

[achow101]

If we agree that all pools are equally well connected to the network, then obviously broadcasting an empty block (~200 bytes) vs a full block (~1,000,000 bytes) will be faster.

I don't think it's that obvious.  When does a node "timestamp" the block it's receiving, when it first starts seeing data, or when the data is 100% received?  I think it's when it first sees data coming in, therefore rendering the size of the block a moot point with respect to propagation rate.  We're not comparing 10k to 10,000,000k, we're comparing 200k to 800k and with today's technology those values are essentially the same.

It doesn't matter when the block is time-stamped by the node, only after it is added to its databases which only happens after it had fully verified the block and thus has had to receive it in full.

[jonnybravo0311]

You just stated the quintessential difference between SPV mining and fully verified block mining.  In SPV mining, the pool starts work on a new block based upon only the block header of the previous one.  It has no knowledge the the previous block's transactions when it does so and is why it starts work on an empty block.  Eventually when it does in fact verify the full block, only then can it include transactions into the new block it's already mining.  Without verification, there is no guarantee the new block they are building is even valid... and it's why the chain forked last year.  Because without actually validating the block, they built upon a version that should have been rejected and submitted a new block on top of it.

So, without validating, the SPV mining pool can (and does quite frequently) find a solution to the bogus empty block they created initially, and it gets added to the chain.

The real difference that needs to be evaluated is how long it takes to actually verify the block before creating a new one on which to work.  If I remember correctly, kano's shown that difference to be a few hundred milliseconds.

Given that information, a pool like AntPool will start work on a new block faster than a pool like kano's (by a few hundred milliseconds) and if it finds a solution the block it produces will propagate across the network faster than a full block from kano.  That, more than anything else, is why the big pools refuse to change, no matter how much damage they are doing in the process (huge transaction backlogs and potential forking of the blockchain).

[-ck]

Take a look here: https://poolbench.antminer.link

That site gives a good breakdown of how the pools propagate blocks.

That's not propagating blocks. It's just the stratum work templates which amounts to no more than the headers going to one VPS in the west coast of the USA. It's a marker of how quickly the pools' miners get block changes and has nothing to do with how quickly blocks propagate across the network, but is the combined sum of how quickly the pool itself sees the block changes and sends out stratum templates to miners. Also the code on that page is buggy and occasionally starts producing wrong values (like showing blocks appear from some pools up to 30 seconds before the block even exists) so use it as a guideline but don't place too much value on it when things look extreme. SPV pools don't verify headers when they send out their stratum templates so although their work templates are more often sent out slightly faster than other pools, they can't guarantee that they're not working on a nonsense fork since only real node pools verify block changes completely. A pool using that site as its reference may be tempted to start SPV mining to make its values look better but ultimately that then means they've moved to unverified headers for block changes and header only work also cannot contain any transactions so they'll be guaranteed to be empty blocks.

[jonnybravo0311]

Thanks for the clarification to my statement -ck.  I realize now re-reading it after your reply that I should have stated it shows how fast pools get new work out to miners, and not how fast the pool propagates the blocks it receives.  Also, as you correctly point out - it's just how fast that particular server receives the updates from the pools to which it is connected.

So, take the information provided with a grain of salt.  I do, however, find it an interesting look into things - seeing the vast discrepancies between the top and bottom.  For example, if we ditch the SPV pools, taking a look at the list shows that (for the last 100 blocks anyway) the pools getting work to their miners the fastest are slush (his node in China), ck.solo, kano, bravo with a difference of less than 0.8 seconds between them.

[philipma1957]

One good thing about the 1/2 ing is block to fee ratio may change by 2x.

Ie a 25.5 block with fees now will be 12.5 block with fees in July after the 1/2 ing.

So pools like f2pool and antpool may want to end spv mining.

Also if we do not change from 1mb blocks to 2mb blocks. The new norm may be 12.9 or 13.1 or even 13.5

This will place a lot of pressure on spv pools to stop that method of mining.

It is possible that empty block. Mining will become a bad financial play for current spv miners.

[VRobb]

Let us hope so, philipma, since money is a big motivator!

[Mikestang]

Unfortunately money is the only motivator to the spv mining pools.  I hope that it becomes financially necessary for them to stop mining empty blocks, it will be a great thing for the network once they do, and it would give a more accurate portrayal of how the network is able to handle tons of transactions (and hopefully put an end to the excessive delays waiting for tx to go through).

[coinits]

empty blocks result from bursts

Uh, what is a "burst"?  Whatever it is, it's not why empty blocks happen.  Empty blocks come from bad/lazy coders looking for a shortcut.

A empty block will propagate faster through the network and a full block would most likely get orphaned (if both are solved in similar times).

Not true at all, kano.is has about the fastest network block propagation and never mines empty blocks.

BURST is a coin and it is certainly not its fault for empty blocks 

[iMish]

Unfortunately money is the only motivator to the spv mining pools.  I hope that it becomes financially necessary for them to stop mining empty blocks, it will be a great thing for the network once they do, and it would give a more accurate portrayal of how the network is able to handle tons of transactions (and hopefully put an end to the excessive delays waiting for tx to go through).

hello,

as of today, the average fee per block is of 2 BTC so mining empty blocks is about 16% less profitable. but is still happens that some pools mine empty blocks.

can anyone explain this?

thanks

[TracerX]

Unfortunately money is the only motivator to the spv mining pools.  I hope that it becomes financially necessary for them to stop mining empty blocks, it will be a great thing for the network once they do, and it would give a more accurate portrayal of how the network is able to handle tons of transactions (and hopefully put an end to the excessive delays waiting for tx to go through).

hello,

as of today, the average fee per block is of 2 BTC so mining empty blocks is about 16% less profitable. but is still happens that some pools mine empty blocks.

can anyone explain this?

thanks

I think the accepted reason is that BITMAIN is using a covert version of the ASICBOOST system to boost their profits.  IIRC, the empty blocks are an effect of that.  I may not be up to speed, however--if someone has a differing opinion I would love to hear it.