Empty blocks

[Peter R]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

[jonnybravo0311]

360629 blocks, 85311 are empty.  I zipped it up because with every block the file was about 22MB.  If anyone wants to check it out, you can find it here: https://www.dropbox.com/s/gduh8oafyqzi57t/empty_blocks.csv.zip?dl=0

The format is:
Height,NumTx,DateTime,WhoMined

Enjoy.

Great work!

I'd love to do some statistical hypothesis testing on the data in this file, but the link seems to be broken.  Any chance you could repost the raw data?  

I took the file down a while ago... let me run it again and get you the most up-to-date data.  I'll post a link as soon as it's done.

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

I suppose it could be... but since I've already started the program execution and it takes quite a while to complete (done 326,000 blocks so far), you'd have to wait a while longer for me to make the code changes and run it again .

[Peter R]

Thanks!

Edit: it would be helpful if there was also a "blocksize (MB)" column, if that's easy to add!

I suppose it could be... but since I've already started the program execution and it takes quite a while to complete (done 326,000 blocks so far), you'd have to wait a while longer for me to make the code changes and run it again .

Haha no problem.  The current file format will give me plenty to get started on.  Thanks again!

[jonnybravo0311]

So it just finished... here are the numbers:

As of block 364125, there are a total of 85,422 empty blocks.

AntPool: 5.16% of their blocks are empty, latest one was today.
Discus Fish: 2.88% of their blocks are empty
KnC: 2.88% of their blocks are empty
Eligius: 2.26% of their blocks are empty

EDIT: I made the changes to the code to include the block size... re-running it now.  I'll post it and provide the link when it's done.

Format of the data is:
Block Height, NumTx, BlockTime (UTC), BlockSize (Bytes), WhoMined (if I could figure it out, BTC address of coinbase transaction if I couldn't, or unknown if no BTC address could be deciphered from coinbase transaction).

EDIT 2: Just finished running it again with the addition of block size per your request.  I've uploaded a zip file here: https://www.dropbox.com/s/2rt64pt6hyhr41b/btc_stats.zip?dl=0.  Has blocks up to 364144

Archive contains two files:

empty_blocks.csv has the raw data in the format described above
stats.csv has data on pools, how many blocks they've mined, how many were empty and percentage of empty to total.

Enjoy.

[Peter R]

Enjoy.

Thanks so much!.  Here's some preliminary results, if you're interested: https://bitcointalk.org/index.php?topic=68655.msg11809512#msg11809512

[achow101]

Is your code for this open source or available for people to download and run themselves? If so, where can I get a copy.

[jonnybravo0311]

Is your code for this open source or available for people to download and run themselves? If so, where can I get a copy.

I didn't publish it... it's just something I wrote up for myself.  To be honest, it's not the cleanest Java code I've ever written.  For example, rather than handle Exceptions, I just throw them.  Everything is in one file - multiple defined private classes and inner classes.  Hardcoded bitcoin daemon connection parameters.  No comments.  No build system (like maven, etc).

Thanks so much!.  Here's some preliminary results, if you're interested: https://bitcointalk.org/index.php?topic=68655.msg11809512#msg11809512

I'll check it out.  Glad to help

[unamis76]

Pretty interesting data... There seem to be more "blank" blocks than I thought. This is definitely not good, in my opinion...

[NeuroticFish]

Pretty interesting data... There seem to be more "blank" blocks than I thought. This is definitely not good, in my opinion...

I agree, especially since the last attack/stress test.
Somebody smarter than me should maybe think if this can be done in the network somehow - if the number of unconfirmed transactions is 10k, the pool / finder includes only 1, or 100 transactions, making the block 1k or 100k, while maximum is 1MB is unfair - the network should force to include more transactions, close to max block size.

While this is not fixed, discussions about bigger block size are .. just not needed.

[jonnybravo0311]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

[p3yot33at3r]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

It's not just the Chinese either. I'm not sure what's worse, the pools that are knowingly throwing out these empty blocks in search of greed - or the miners that mine with them......in search of greed......

[kano]

And our Chinese pools continue to mine empty blocks, even with 13k unconfirmed transactions waiting... 364748 mined by f2pool is empty.

It's not just the Chinese either. I'm not sure what's worse, the pools that are knowingly throwing out these empty blocks in search of greed - or the miners that mine with them......in search of greed......

Well ... it's even more confusing when Luke-jr/Eloipool is saying that people mining the spam (and thus getting rid of it) are doing the wrong thing ... lulz.
https://bitcointalk.org/index.php?topic=441465.msg11826511#msg11826511

Although the spam has caused a lot of problems on the network (tell me about it - I've missed sleep thanks to it ...) it's also meant higher transaction fees:
Last 5 https://kano.is/ block reward txn fees % have ~been 2% 1% 1% 2% and 1% ... normally it averages 0.4%

--

Oh and I ran a 40 block https://kano.is/ vs eligius block change test again last night.

https://kano.is/ was only slower with 4 blocks in that 40 range ... of course the reason was they were Eligius blocks.

https://kano.is/ was faster with the other 36 blocks, and the total average was 1.4% faster for the 40 blocks with an average of 8 seconds faster per block change!

I still have no idea why anyone mines there, I guess they don't compare pools properly or see the '0%' fee comment, but don't realise that there's a high average expected loss due to mining on eligius (vs my pool) that more than makes up for that.

[p3yot33at3r]

It is a little worrying. With the attacks currently taking place & these pools that are not pulling their weight or contributing to the network, it would be nice if the mining community tried to increase awareness of the harm they are doing. It seems to me that these pools are biting the hand that feeds them, leeching even. I've read so many complaints from miners about extended tx confirmation times & fees, yet they continue to mine at the very pools that are making the situation worse - am I right?

If so, what does anyone suggest?

[TheRealSteve]

There's only one thing to suggest: go mine at a pool that aligns more with your ideals.  Problem is that the ideals of many people are trumped by their greed.
All else being equal, would you rather mine at a pool that for a given period pays out BTC1.0 while playing nice, or at a pool that pays out BTC1.01 but they mine coinbase-only blocks more often than strictly necessary?  What if it's BTC1.05?

[kano]

I've had a PM question about what the numbers I've posted mean so ...

When a pool gives out work to the miners, it gives work that can find a block built on the pool's current known best block.

The pool changes it's current best block either when one of it's pool miners finds a new one, or, most often, when someone else on the bitcoin network finds and distributes a 'better' block, around the network.

In simplest terms, what this means is that: most miners will have a small time frame, each time the block changes, from when someone finds the block, distributes it on the network, and then the pool gets the new block, then the pool sends out new work, for the new block, to the pool's miners.

There's 3 parts in that:
1) Some miner somewhere on the network finds a block (and sends it to their pool if they are mining to a pool)
2) The block is distributed on the network and your pool gets it
3) Your pool processes the new block then sends out new work to your pool's miners

If someone on your pool finds the block, then 1) and 2) are simply how long it takes for the miner to send their block winning share to the pool.
Otherwise:
Step 1) is out of the pool's control so you can't certainly know exactly when that happens
Step 2) is dependent upon how well the pool is connected to the the rest of the network and specifically, wherever the block was found.
e.g. if you are mining at PoolA and PoolB finds the block, it depends on how long it takes for that new block message to get from PoolB to PoolA

No matter who finds the block,
Step 3) is dependent on how quickly the pool can process the block and send out new work to the pool's miners.
It will also depend on the size of the block and other factors like the block processing complexity and of course how fast the pool software is and how long it takes to send work to the miners

Ignoring when the pool itself finds a block, the time the pool takes to do step 2) and step 3) represents wasted mining time.
Step 3) is completely 100% wasted time
Step 2) is either wasted mining time, or if a block is found during this time, will represent how likely the pool's chance is of winning an orphan race against the other block

So .........

What is my 8 seconds I posted above?

If you have 2 cgminers each pointing to different pools, then each cgminer will report on the screen each time it gets work that is a block change.
The difference between a block change time on the 2 miners represents wasted mining time.
It is of course Step 2) + Step 3)
It should be small and usually each pool should randomly be faster than the other pool
If one pool is averaging better than another pool, it is clearly doing one or both of:
a) better connected to the rest of the bitcoin network (Step 2)
b) faster at processing blocks and sending out new work (Step 3)

The average time difference represents, on average, the amount of extra wasted time mining on the slower pool.
The faster pool is also wasting some time mining each block change, but the slower pool is wasting, on average, that much more per block change.

Eligius' claim is that their empty blocks make the total of Step 2) + Step 3) faster so miners are wasting less time mining stale work.

But what point is there to that when even with that 'faster' processing, they are averaging slower than a pool that doesn't cheat the network by mining empty blocks ...

[NeuroticFish]

Wow, long text, but interesting!

I see 2 options:

#1. More "advertising" to increase awareness between miners about this problem, maybe some will switch pool.
#2. Implement into the network something to force them to play fair.


I don't know if #2 is actually possible, but that would be the way to go. I find this more important than increasing the block size.

[p3yot33at3r]

Wow, long text, but interesting!

I see 2 options:

#1. More "advertising" to increase awareness between miners about this problem, maybe some will switch pool.
#2. Implement into the network something to force them to play fair.


I don't know if #2 is actually possible, but that would be the way to go. I find this more important than increasing the block size.

Both good suggestions, I was thinking along similar lines. Maybe a sticky or some kind of bold statement from organofcorti in his pools thread, or any respected community members for that matter would help? I'm sure that #2 would be possible somehow, but it would take time to implement I think.....?

EDIT: Some kind of list showing which pools are contributing to the network & which ones aren't - name & shame kinda thing......

[TheRealSteve]

Maybe a sticky or some kind of bold statement from organofcorti in his pools thread, or any respected community members for that matter would help?

Various people quite often state that they would prefer to see people mine at smaller pools (for reasons besides any concerns about coinbase-only blocks).  In the end, well, see my previous post.  I'm happy to mine at Slush's pool, kano.is and BTCDig with the smidgens of hash rate I've got, but then I don't worry about that marginal increase in income or whether I go without a payout for weeks on end..I'm not running anything profitable anyway, even with solar (get more money feeding back into the utility network)

organofcorti posts mostly just verifiable (with a bit of effort) statistics - I think that actively discouraging certain pools on his blog or even his pools thread (which looks like it'll be replaced if the guy volunteering gets his thread off the ground) would just introduce partiality that would both hurt objective interpretations and credibility of his stuff.  As it is, some might think the BTC20 donation 'coincidentally' made after a post that showed the pool involved as being the biggest is of potential concern. ( I'm not, as long as he keeps posting statistics, and the funds are probably welcome for covering expenses and justifying what is mostly a hobby. )

To skip an item...

EDIT: Some kind of list showing which pools are contributing to the network & which ones aren't - name & shame kinda thing......

That certainly is something that organofcorti could do.  This thread started out with some statistics of coinbase-only blocks vs non-, how that relates to previous block time, and in another thread how it relates to the previous block's size, etc.  Presented in a simple table it again just becomes verifiable statistics and people can make up their own mind; and do remember that miners who aren't including any regular transactions are still contributing to the network by building the block chain, and there are differing opinions on whether coinbase-only blocks are actually a bad thing, a good thing, or one of several shades of grey in between.

I'm sure that #2 would be possible somehow, but it would take time to implement I think.....?

It's possible, I think would require another consensus round (considering you'd have to deny certain blocks as being invalid after some time), and a whole lot of discussion.. which is where most of the time would be (I think that was discussed in this thread as well?  It was discussed somewhere anyway).  The required code itself is somewhat trivial once the rules have been fleshed out.  Good luck with that, though... the max block size debate hasn't even been properly settled yet

[drakoin]

The chain had tens of thousands of empty blocks in the beginning.

What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...

How many empty blocks are there this year - and who mined them?

[TheRealSteve]

The chain had tens of thousands of empty blocks in the beginning.
What if you ignore the chain before 2015 (and/or 2014, 2013, 2012, 2011) ...
How many empty blocks are there this year - and who mined them?

Partial answer to that is already in this thread ( https://bitcointalk.org/index.php?topic=1085800.msg11593801#msg11593801 ), or you could divine it from jonny's dataset ( https://bitcointalk.org/index.php?topic=1085800.msg11584786#msg11584786 ).
Updated for blocks in 2015 up to block 365,401:

Edit: updated to show 2-transaction blocks, only because that stood out for one particular pool.

Keep in mind that this data on its own isn't very interesting without also looking at e.g. block times and sizes.