Bitcoin's Empty Blocks Analaysis.

[tranthidung]

Yesterday, I saw that link https://gz.blockchair.com/bitcoin/blocks/ but I gave up when see their message that all files require over 1Tb. If it is what you will have to scrape everything, it is good to download their files to save time.  

I have not yet used or extracted any file with that format. I searched and found TSV means tab separated values. I will try to extract one file to see how it looks.

I still think include blocks that have same amount of generation and reward BTC is good because we will know there are how many percent of fake empty blocks with same generation and reward BTC.

How to include that condition from what you have us?
https://blockchair.com/bitcoin/blocks?q=transaction_count(1)#f=time,guessed_miner,transaction_count

Your current conditions result in less than 90k results!

[mikeywith]

I'll see what I can do. I kinda want to just scrape everything, and make a huge csv. That'll come in handy for other analyses too.

If that's not too much to scape I would encourage you to do that, I am also interested in block-size analysis, kind of want to see if we are really fully utilizing the block-size we have now or we aren't and all these calls for block size increment are unlogical, further analysis like how often do we really get unusual block time such as 1 or 2 hours block is also interesting, different studies require different data, so scrapping everything especially from blockchair.com will be really useful, good luck and please keep me updated.

Yesterday, I saw that link https://gz.blockchair.com/bitcoin/blocks/ but I gave up when see their message that all files require over 1Tb. If it is what you will have to scrape everything, it is good to download their files to save time. 

1TB is TOO MUCH, those files probably contain everything about the blocks and more, the current blockchain size less than 250gb, the data we need shouldn't be larger than 1mb in a text/excel format, I most certainly can't process a 1TB worth of data.

I still think include blocks that have same amount of generation and reward BTC is good because we will know there are how many percent of fake empty blocks with same generation and reward BTC.

The generation is the same for every 210,000 blocks and they have nothing to do with a block being empty or full, the generations are not needed in this study and most likely not in any other analysis, I can simply populate them manually if I had to since we know the first block had 50 btc and then 210,000 later it was 25 btc and then 210,000 later it became 12.5 btc, I don't know how to explain it better, but really we don't need them.

for the reward, it's merely generation + fees, if you include the fee(btc) column you will see that, now similar fees for more than one block proves nothing, they are pretty normal.

Your current conditions result in less than 90k results!

That simply means we had less than 90k empty blocks since Satoshi mined the first block.

It would be cool to see that a huge amount of empty blocks by antpool were 30-60 seconds

vs 1-10 seconds as it would expose a pattern of bad acting that they are accused of.

Phill, I will probably still do that analysis for you and the other person who requested it, but I can tell you beforehand that it will prove NOTHING, bitmain can find blocks in 30 seconds and fake the time in the block header to be 3 seconds unless you could hack into their database and assuming they do keep such records, the study will prove nothing, you are counting on their stupidity, integrity, and kindness, and we both know bitmain doesn't have any of those three aspects.

[pooya87]

I'll see what I can do. I kinda want to just scrape everything, and make a huge csv. That'll come in handy for other analyses too.

It would be cool to see that a huge amount of empty blocks by antpool were 30-60 seconds
vs 1-10 seconds as it would expose a pattern of bad acting that they are accused of.

it is worth noting that timestamps are only as accurate as the clock of the computer that created the block to be mined. specially when the plan is to compare the timestamps with a high accuracy of a couple of seconds you have to take the room for error in mind.

[LoyceV]

Yesterday, I saw that link https://gz.blockchair.com/bitcoin/blocks/ but I gave up when see their message that all files require over 1Tb.

Great find! But it's much, much less than 1 TB. I'm downloading this now (at 10 kB/s) to make a big csv. If it has enough data it's much faster than scraping all blocks.

it is worth noting that timestamps are only as accurate as the clock of the computer that created the block to be mined. specially when the plan is to compare the timestamps with a high accuracy of a couple of seconds you have to take the room for error in mind.

If I'm not mistaken, the time between blocks can even be negative if the miner's clock is a bit off.

[mikeywith]

If I'm not mistaken, the time between blocks can even be negative if the miner's clock is a bit off.

You are right, he only rule that stops miners from mining blocks with timestamp in the past is the MPT rule, which states that timestamp of any given block must be > than the median time of the last 11 blocks, since the median of 11 blocks is block 6, this means

Timestamp of block 12 can be

less than block 11
less than block 10
less than block 9
less than block 8
less than block 7

can NOT be less than block 6 timestamp, provided block 1 to 11 didn't have any negative timestamps, if they do, the calculation method will be different, the code simply takes the past 11 timestamps and get the median of those, which is always the 6 blocks back if they all had positive timestamp differences, I might be mistaken, would love to be corrected.

so timestamps can be negative, not only if the miner's clock is off, it could be done intentionally, but to miners best interest (assuming they are not purposely mining empty blocks and trying to hide it) is to fake the time to be LONGER and not shorter, because that gives the miners a decrease in difficulty while the latter causes an increase in difficulty and thus reducing their rewards in the future.

[pooya87]

so timestamps can be negative,

do you mean the difference between two timestamps (t2-t1<0)?
because if not, and if you are indeed talking about the 4 bytes after the merkleroot hash in a header then i would love to see how this works. is it another weird thing in bitcoin protocol that we let happen?

[mikeywith]

 (t2-t1<0)? 

Given the context i think it is pretty clear what i mean is the quoted part, however you brought up an interesting point,
 the timestamps in the header are the seconds elapsed since 1st jan 1970, can unix timestamp be negative? I don't know.

Let's assume you managed to sequeze in a negative value and kept it at 4bytes, what will actually happen? Is there any piece in the code that specifically checks that?  The only issue is that timestamps - - timestamps will result in a timestamps that way far into the future, which then will be invalidated by this rule.

Code:

MAX_FUTURE_BLOCK_TIME =
2 * 60 * 60

[pooya87]

i'm the worst at reading C++ but it seems like core is storing the blocktime (ntime) as an unsigned 32-bit integer[1] and during comparisons (eg.[2]) the GetBlockTime() method [3] is called which will cast that into a 64-bit signed integer which means it will never be negative (casted UInt32.Max = 0x00000000ffffffff => always positive)

[1] https://github.com/bitcoin/bitcoin/blob/78dae8caccd82cfbfd76557f1fb7d7557c7b5edb/src/primitives/block.h#L27
[2] https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L350
[3] https://github.com/bitcoin/bitcoin/blob/54f812d9d29893c690ae06b84aaeab128186aa36/src/chain.h#L247-L250

[mikeywith]

I am not good at C++ too, but I know it can't be 64 bit both singed and unsigned because 64 bits is 8 Bytes and thus it won't fit into the block header, so it must be 32, by quickly skimming the public class code

Code:

uint32_t nTime{0};

So no negatives here, only 0 and above, all the way to 4294967295 and that so happen to be equivalent to 02/07/2106 @ 6:28am (UTC) according to https://www.unixtimestamp.com/index.php, so in 86 years, some work and a fork will be needed to keep the blockchain going.

[LoyceV]

which means it will never be negative

I just checked (using this work in progress), and using yesterday's data I found 14,145 blocks that had a time earlier than the previous block.
Examples:
628453 - 2020-05-01 19:02:08
628454 - 2020-05-01 19:02:54
628455 - 2020-05-01 19:02:47 (-7 seconds)
628456 - 2020-05-01 19:02:34 (-13 seconds)
628457 - 2020-05-01 19:13:10

I would have expected miners to synchronize their clocks much more.

[DaCryptoRaccoon]

This is a very interesting topic I brought up something about this a long time ago back in 2018 when I spotted strange things happening in the block explorers.

https://bitcointalk.org/index.php?topic=4998410.msg45156234#msg45156234

My post was actually deleted at the time. (Screen shot 2)

I felt it was a valid question to ask as it was not something you normally see happening on the chain it looked to me like someone had worked out how to manipulate the time to ensure they could mine the next block empty before anyone else even started on the block.

So are people now thinking that antpool and bitmain are doing something sketchy to ensure they can mine those empty blocks while the rest of us are trying to play on the un-level playing field?

I was under the impression that most miners used the NTP time pool's for there clock source I know my old miners used NTP and even had the ability to run as NTP server.   

Thanks

Magic

[mikeywith]

someone had worked out how to manipulate the time to ensure they could mine the next block empty before anyone else even started on the block.
 

Nop, can not start mining block N before everybody else knows about block N-1 ( ignoring network delays for certain nodes), block timestamps mean nothing in regards to the chain order. 

So are people now thinking that antpool and bitmain are doing something sketchy to ensure they can mine those empty blocks while the rest of us are trying to play on the un-level playing field

That is not what anybody else is thinking, you see having negative times between blocks only means 1 of 3

1- The clock is off (behind)
2-The clock of the miner who mined the previous block is off (ahead)
3- The miner doing that on purpose to

 a- hide the fact that they had enough time to include transactions but refused doing so. (this is what people are suspecting)

b- The miner wants to artificially increase the difficulty by tricking the protocol as if blocks were mined faster than they actually were (no logical reason for any miner to do so)

[tranthidung]

I just checked (using this work in progress)

It has not yet included block heights. Could you include block heights, please.

[LoyceV]

I just checked (using this work in progress)

It has not yet included block heights. Could you include block heights, please.

I was still working on it, see Bitcoin block data in CSV format.

@mikeywith: is this what you were looking for?

[tranthidung]

see Bitcoin block data in CSV format.

You are actually a data-mining beast, LoyceV. Thanks for another data which is huge this time.  

[BrewMaster]

1- The clock is off (behind)
2-The clock of the miner who mined the previous block is off (ahead)
3- The miner doing that on purpose to

 a- hide the fact that they had enough time to include transactions but refused doing so. (this is what people are suspecting)

b- The miner wants to artificially increase the difficulty by tricking the protocol as if blocks were mined faster than they actually were (no logical reason for any miner to do so)

keeping in mind that a miner first sets the block time in the header and then starts mining it the reason for the differences in matter of seconds between blocks and real time could simply be because the miner took longer to update the time.

for example the miner could set the time then start going through the nonces then after failing to find a good hash they change their "extra nonce" and hash again, and so on. then right before they get around to update the time they could find a good hash and release it. so the result is a couple of seconds in the past.

[mikeywith]

is this what you were looking for?

By the look of it, it seems like that is more than I need, I have yet to download the data I need and see if it fits in perfectly in my excel sheets, you did a great job.

It has not yet included block heights. Could you include block heights, please.

It is there, he refers to it as ID > http://loyce.club/blockdata/id.txt

keeping in mind that a miner first sets the block time in the header and then starts mining it the reason for the differences in matter of seconds between blocks and real time could simply be because the miner took longer to update the time.

You are right indeed, but that difference in time is too small to be considered, updating the time in the block header probably takes a few milliseconds and since it doesn't affect the miner chances of solving the block and also keeping in mind that for miners' best interest the time is either exact or in the future (to make the best use of next difficulty adjustment) it's safe to say at least large mining pools make sure the time is rather in the future, not the past.

but really combining all these factors, small or large, makes the analysis a bit far from accurate, but I will do it either way.

[mikeywith]

As requested by a few members, and after getting the data from LoyceV, here is what I came up with.


I excluded the years before 2016 so the data is only for 2016-2019, the reason behind this is the fact that in the early days, we had a ton of empty blocks, simply because at many times there were no transactions to be added to the block anyway, also most of the blocks come from "unknown" miners.

In this chart, I show the total time for all timestamps of empty blocks - the block which preceded it for the top 5 pools.




But I realized this figure might be misleading because finding more empty blocks will give the miner a lead in this chart, and since Antpool did find a ton of empty blocks most likely do the use of covert Asicboost, the figure above might not show any evidence of delaying transactions on purpose.

The next step was getting the average timestamp for those empty blocks, I got the total time difference and divided it by the total number of empty blocks per pool per year.




The above figure is pretty fair since it takes into account that larger pools tend to find more blocks and therefore find more empty blocks.


Based on the points mentioned in the previous comments, I will not attempt to conclude anything out of this study, everyone is free to interpret
these figures the way they like.

[PrimeNumber7]

Great analysis. Could you factor in the timestamp/the timing received by blockchair into the graph? It would be a lot clearer to see the timings between the empty blocks and the block before it.

I could but there are two problems, the first one is scraping the data of blockchain,took me forever to extract the data for empty blocks which is nothing compared to the total number of blocks, but if someone has a better way of getting those data, put them in a table format like excel and send them to me, I can do it and even more perhaps.

The second problem is that time-stamping isn't exactly accurate,

If you are running a full node, you can use RPC commands to obtain timestamp information about each block, using the RPC command "getblock" using verbosity 1. This will not tell you which pool mined each block, you will need to get this from somewhere else.

The published timestamp has the potential to be off by up to 2 hours, although in most cases, I believe it is closer to accurate. To have an authoritative timestamp, you will need to have a well-connected node that you specifically program to keep track of the time each block is received.  If you only have the published timestamp, that will have to do.

I would be curious if there is a correlation between the percentage of blocks found, and the percentage of zero tx fee blocks/total pool blocks found, and if so, what this correlation is. Does the percentage of zero tx fee blocks/total pool blocks found increase as the percentage of blocks found increases? Are there any outliers in this particular slope?

I would not want to pre-judge data, but I would propose that pools that find more blocks generally can invest in better equipment/network connectivity, to process found blocks, to propagate found blocks, and to identify found blocks. I would also propose that some pools may have different groups miners try to find blocks based on different sets of transactions.

[pooya87]

This will not tell you which pool mined each block, you will need to get this from somewhere else.

if you use Verbosity 2 then each transaction in block will also be fully shown in JSON format and the first one is obviously the coinbase tx which contains the possible name of the pool or miner that found the block. all you have to do is to find result.tx[0].vin.coinbase then use UTF8 to decode the result. for example for block (currently last block) 0000000000000000000fcf675c44734ddbe2af6dddd480f7d7aba324488f138d

Code:

��	4Ÿ^vip/www.okex.com/��mm�d8OR :Y:�ٜ�U��Juf��䶎;x

then search inside that string for known names which could found online with a little search.

it could also split into two calls (reduces the size of the first response):
first with verbosity 1 as before then fetch the first transaction ID then use that in getrawtransaction with verbose true to get the coinbase JSON as a single transaction and do the same UTF8 decode again.

test here if you don't have a node close:
https://chainquery.com/bitcoin-cli/getblock
https://chainquery.com/bitcoin-cli/getrawtransaction