Trying to understand an odd fork propagation pattern (height 928484)

[clex88]

Hello everyone,
I was inspired by a few papers on Bitcoin block propagation to build a small probe network that measures live block propagation on mainnet
(currently Atlanta / Frankfurt / Singapore).

So far I’ve analyzed a bit over 2000 blocks, and I’m seeing some interesting and
fairly consistent propagation patterns emerge (still crunching through most of that data).
However, one fork in particular caught my attention, and I’m trying to understand what I’m actually observing.

This was a fork at height 928484, between GDPool and AntPool.

According to my probes, GDPool’s block was visible across all probe locations roughly a minute and a half before AntPool’s competing block.
I initially assumed I was misunderstanding something, so I spent a few days digging into it.

At first I wondered whether AntPool’s block had some characteristic that caused it to be preferred by the network — for example a higher-weight block, or that it had already reached a larger share of nodes through some distribution path outside of my probes, despite the large timing difference I was observing. But when I looked at the next block, 928485, things got stranger.

When querying my database for 928485, I noticed that AntPool’s block at 928484 and its successor at 928485 appeared to my probes at essentially the same time (within ~10ms across all probes). As I understand it, once a block is mined on top of a competing block, the longer chain is selected regardless of which block arrived first, since there’s no global time authority.

So effectively, GDPool’s block arrived first, but AntPool’s block arrived together with its successor, and the fork resolved immediately.

At that point I thought maybe my probes just caught this in a strange way due to some network fluke or sampling artifact. However, after profiling thousands of blocks, I do see fairly stable regional and miner-specific propagation behavior.

For context, when I look at the global skew between probes (i.e. the time between the first probe seeing a block and the last probe seeing the same block), it’s usually on the order of 70–90 ms, and even at the extreme tail I see at most ~2 seconds globally. I’ve never seen anything remotely close to 90 seconds under normal conditions.

So I’m a bit stuck.

Does anyone here with deeper protocol or mining experience have insight into what might be going on in a case like this?

Is this a known artifact of how headers propagation or chain synchronization works?
Am I misunderstanding how these announcements are observed at the network level?

[gmaxwell]

What matters for ultimate selection is what hashpower sees, not what 'nodes' see.  So monitoring many nodes may not give you a fair assessment of what actually matters.  This is made more likely by the fact that good operational practice is to not accept incoming connections directly on the node controlling mining in order to protect it against inbound DOS attack, so the nodes that matter most for your measure are ones you literally can't connect to.

Also,

Consider a race between A and A'--  if the nodes you are connected to see A first then they suppress transmission of A' by virtue having already accepted A.  However, in realty A' might actually be preferred by significant hashpower while the dominance of A exists mostly just in what you are monitoring.  In any case, even if your view is fair A' will still probably be the first block seen by at least the author of A' and so the probability of it being extended first is non-zero.

Later block A'B is mined, at that point it becomes the preferred tip and all the nodes that previously preferred A will now switch to A'B and relay A' (parent of A'B) to you first.  So what you see is A' then A'B at once, but in reality A' could have existed in public for a long time, it just wasn't making it to you.

So the success of A has suppressed your visibility of A' until the chain with A' overtook.    It's also possible that the author of A' withheld it, but that is a needlessly complicated explanation for an otherwise expected observation.

Unfortunately ever since compact blocks (particularly the use of cut-through announcements) well relayed blocks tend to block the knowledge of competitors pretty effectively.   Prior to it, validation time was in the critical path at every hop and network latency meant that close announced competing tips tended to be more visible.

I've previously suggested a protocol addition where each node would announce just the header of any POW valid block that connects 'near' the current best tip, even if they haven't accepted the block, don't have the block, and/or don't know where to get the block-- just to improve the visibility of races/forks/stale-blocks.  This needs a new P2P message, because the existing blocks/headers messages mean that you have the block and are able to share it.  Fortunately it would be a very trivial change needing only changes in the node software and not the rest of the ecosystem.

I think your kind of monitoring will inherently be very limited and somewhat misleading without that kind of protocol addition--  particularly because I don't think you can distinguish a withholding attack (which would be very interesting!) from the natural suppression of near-ties without something like that.

Going a step further it would be useful if miners published and the network shared header messages for attempted blocks meeting e.g. 10% or even 1% of the difficulty target.  This would give realtime visibility in to what forks the network was attempting to extent.  Such an addition though would require changes to mining software in addition to nodes, so it would be inherently much harder to deploy and people may want to extend the mechanism to help improve block propagation (by using it to pre-announce likely to be mined transactions)-- so unlike the alternative-chain-headers proposal would be likely to suffer scope creep.

Unfortunately the bitcoin community seems to have lost sight of the goal of maintaining a healthy network and instead prefers to navel gaze and debate paid provocateurs about the merits of altering the system to steal coins, censor transactions, or other such nonsense.  So unless you feel like doing that work yourself, I wouldn't suggest holding your breath.