Trying to understand an odd fork propagation pattern (height 928484)

[clex88]

Hello everyone,
I was inspired by a few papers on Bitcoin block propagation to build a small probe network that measures live block propagation on mainnet
(currently Atlanta / Frankfurt / Singapore).

So far I’ve analyzed a bit over 2000 blocks, and I’m seeing some interesting and
fairly consistent propagation patterns emerge (still crunching through most of that data).
However, one fork in particular caught my attention, and I’m trying to understand what I’m actually observing.

This was a fork at height 928484, between GDPool and AntPool.

According to my probes, GDPool’s block was visible across all probe locations roughly a minute and a half before AntPool’s competing block.
I initially assumed I was misunderstanding something, so I spent a few days digging into it.

At first I wondered whether AntPool’s block had some characteristic that caused it to be preferred by the network — for example a higher-weight block, or that it had already reached a larger share of nodes through some distribution path outside of my probes, despite the large timing difference I was observing. But when I looked at the next block, 928485, things got stranger.

When querying my database for 928485, I noticed that AntPool’s block at 928484 and its successor at 928485 appeared to my probes at essentially the same time (within ~10ms across all probes). As I understand it, once a block is mined on top of a competing block, the longer chain is selected regardless of which block arrived first, since there’s no global time authority.

So effectively, GDPool’s block arrived first, but AntPool’s block arrived together with its successor, and the fork resolved immediately.

At that point I thought maybe my probes just caught this in a strange way due to some network fluke or sampling artifact. However, after profiling thousands of blocks, I do see fairly stable regional and miner-specific propagation behavior.

For context, when I look at the global skew between probes (i.e. the time between the first probe seeing a block and the last probe seeing the same block), it’s usually on the order of 70–90 ms, and even at the extreme tail I see at most ~2 seconds globally. I’ve never seen anything remotely close to 90 seconds under normal conditions.

So I’m a bit stuck.

Does anyone here with deeper protocol or mining experience have insight into what might be going on in a case like this?

Is this a known artifact of how headers propagation or chain synchronization works?
Am I misunderstanding how these announcements are observed at the network level?

[gmaxwell]

What matters for ultimate selection is what hashpower sees, not what 'nodes' see.  So monitoring many nodes may not give you a fair assessment of what actually matters.  This is made more likely by the fact that good operational practice is to not accept incoming connections directly on the node controlling mining in order to protect it against inbound DOS attack, so the nodes that matter most for your measure are ones you literally can't connect to.

Also,

Consider a race between A and A'--  if the nodes you are connected to see A first then they suppress transmission of A' by virtue having already accepted A.  However, in realty A' might actually be preferred by significant hashpower while the dominance of A exists mostly just in what you are monitoring.  In any case, even if your view is fair A' will still probably be the first block seen by at least the author of A' and so the probability of it being extended first is non-zero.

Later block A'B is mined, at that point it becomes the preferred tip and all the nodes that previously preferred A will now switch to A'B and relay A' (parent of A'B) to you first.  So what you see is A' then A'B at once, but in reality A' could have existed in public for a long time, it just wasn't making it to you.

So the success of A has suppressed your visibility of A' until the chain with A' overtook.    It's also possible that the author of A' withheld it, but that is a needlessly complicated explanation for an otherwise expected observation.

Unfortunately ever since compact blocks (particularly the use of cut-through announcements) well relayed blocks tend to block the knowledge of competitors pretty effectively.   Prior to it, validation time was in the critical path at every hop and network latency meant that close announced competing tips tended to be more visible.

I've previously suggested a protocol addition where each node would announce just the header of any POW valid block that connects 'near' the current best tip, even if they haven't accepted the block, don't have the block, and/or don't know where to get the block-- just to improve the visibility of races/forks/stale-blocks.  This needs a new P2P message, because the existing blocks/headers messages mean that you have the block and are able to share it.  Fortunately it would be a very trivial change needing only changes in the node software and not the rest of the ecosystem.

I think your kind of monitoring will inherently be very limited and somewhat misleading without that kind of protocol addition--  particularly because I don't think you can distinguish a withholding attack (which would be very interesting!) from the natural suppression of near-ties without something like that.

Going a step further it would be useful if miners published and the network shared header messages for attempted blocks meeting e.g. 10% or even 1% of the difficulty target.  This would give realtime visibility in to what forks the network was attempting to extent.  Such an addition though would require changes to mining software in addition to nodes, so it would be inherently much harder to deploy and people may want to extend the mechanism to help improve block propagation (by using it to pre-announce likely to be mined transactions)-- so unlike the alternative-chain-headers proposal would be likely to suffer scope creep.

Unfortunately the bitcoin community seems to have lost sight of the goal of maintaining a healthy network and instead prefers to navel gaze and debate paid provocateurs about the merits of altering the system to steal coins, censor transactions, or other such nonsense.  So unless you feel like doing that work yourself, I wouldn't suggest holding your breath.

[DaveF]

What matters for ultimate selection is what hashpower sees, not what 'nodes' see.  So monitoring many nodes may not give you a fair assessment of what actually matters.  This is made more likely by the fact that good operational practice is to not accept incoming connections directly on the node controlling mining in order to protect it against inbound DOS attack, so the nodes that matter most for your measure are ones you literally can't connect to.....

I am going to take that comment a step past there as I have before.
For you or me or just about anyone to run a node is not a big deal.
For people running mining pools there is a lot of extra security involved. Or at least there should be....
Including, checking across multiple nodes in different locations.
AND
making sure they are well secured and protected.

Nobody wants to talk about security. But quoting myself from years past on a different issue with some odd mining things.

Taking it a bit past what was said above.
IF they are doing it properly and that is a big IF a large pool is running several nodes all over the world, when a block comes in they all should start validating it. Once a certain percentage of them agree then and only then do they start building the new block for the pool.

Loosing the fees even now is better then building a invalid block that gets rejected.
AND not broadcasting an empty block as soon as possible risks loosing it too.

It just comes down to routing. It takes about 1/2 a second to send 1 packet of data around the world without doing any kind of firewall / security inspection in an ideal setting.
Allowing for the nodes to verify what is in the block, and sending it back out is another couple of seconds and then start building a new block. If all is ideal you should have a new block ready to go in 10 seconds or so.

BUT, if you are waiting for minimum of 3 of 5 to do their thing you might add a few seconds on top of that.

Add in a bloated memory pool and some DPI From firewalls and you add a few seconds again.

All of a sudden you are looking at more time to build a block.

Even now there are 2 schools of thought with what happened with what happened with foundry the other day.
Some people including Kano are saying that they tried to orphan a couple of blocks.
Others are saying that they saw foundrys blocks 1st.

It has always been the way it works.

And part of the problem is that even if you understand BTC perfectly, unless you understand the true issues of internet routing and the true delays that proper DPI can put into network performance then it's never going to seem logical as to what happened.

Dave's pool can put it's nodes out there in public.
A corporation with all kinds of security requirements probably has 3 layers of security devices looking at all data coming in before it even hits the node to be processed. Dave's node has seen, and processed the block before it has even gotten through the security devices of some places. On the flip side, it's a lot harder to take out big corps node(s) by flooding them with bad data since it never even makes it into the network.


-Dave

* DPI = deep packet inspection. https://www.fortinet.com/resources/cyberglossary/dpi-deep-packet-inspection

So Antpool may have not even fully seen the block that they orphaned.

People think because their home 'firewall' can move data at a certain speed anything can.
But proper security and filtering takes real time.


Edit the next AM to add: When I say "takes time" I am not talking minutes but multiples of seconds. That can stack up quickly leading to longer times approaching minutes. I can at home download and open a 5 meg PDF somewhat quickly sa in under 10 seconds. I have some corporate clients that between the click to download and the full open on the desktop is 45+ seconds.
The flip side of that is when someone deliberately tried to send them some very very sophisticated malware. 2 of the 5 things it had to pass though stopped it. Never would have opened after the 1st one, but it still passed through the rest of the scanners anyway to keep checking. But doing things like this and then cross verifying will take time.

-Dave

[clex88]

Wow I kind of wish I didn't post this on the holidays so I could have gotten back to it a bit quicker.
I needed some time to digest all of that but that's why I'm here so let's get into it.

Regarding The Fork

Consider a race between A and A'--  if the nodes you are connected to  see A first then they suppress transmission of A' by virtue having  already accepted A.  However, in realty A' might actually be preferred  by significant hashpower while the dominance of A exists mostly just in  what you are monitoring.  In any case, even if your view is fair A' will  still probably be the first block seen by at least the author of A' and  so the probability of it being extended first is non-zero.

Later  block A'B is mined, at that point it becomes the preferred tip and all  the nodes that previously preferred A will now switch to A'B and relay  A' (parent of A'B) to you first.  So what you see is A' then A'B at  once, but in reality A' could have existed in public for a long time, it  just wasn't making it to you.

So the success of A has suppressed your visibility of A' until the chain with A' overtook. 

This is an interesting insight. It's also actually supported by the data I've collected so far because I've noted that, from the vantage point of my probes, GDPool has a tighter distribution profile than AntPool. Which would imply that if, as you say, public nodes can only request peers on what we might say is the "public boundary" implying that miners inject block broadcasts at layers beneath these and propagate upwards, the probability of a partition happening where A (say from GDPool) and A' (from AntPool) where injected at nearly the same time (say within ~3-20ms apart) then it is very possible that all of my probes would see A first.

However, I should mention, I'm timestamping blocks in

Code:

PeerManagerImpl::ProcessMessage

and classifying them based on the msg_type  chosen. Meaning that I also capture all of the broadcasts from each peer as they come in.

For each block from my 3 probes I typically get 24 entries in my database and for this specific incident all 24 were partial to A before A'B was captured ~90s later.

Which I just mention to say, that if that's true, it means that A' was stuck at a deeper layer because A blocked all of it's exit points to the public nodes layer before it could escape. Alternatively, it may have actually made it to a few public nodes, but it must have been a limited segment of public nodes that I don't currently monitor. If the latter, then I could potentially have captured that if I added more probes to different regions to cast a wider net to get the required resolution to witness an event like this. Otherwise, as you said, it is invisible unless the protocol is modified because it has not surfaced to the public layer (and that is unfortunate).

I would wager though that odds are, both scenarios happen. So, I might be able to capture some fork partitions some of the time if I cast a wide enough net. But unfortunately, I also don't see how this could reliably tell us anything about intentional withholding.

In my defense, I wasn't originally really trying to specifically do this anyway. I just wanted to look for miner propagation invariants and evidence is mounting that I might have found them. Although, ultimately I was hoping the insights would be useful outside of my own curiosity. Of that I'm not so sure yet.

-
Including, checking across multiple nodes in different locations.
AND
making sure they are well secured and protected.
-
So Antpool may have not even fully seen the block that they orphaned.
-
..proper security and filtering takes real time.

This is actually relevant to another assumption I made about a different stage of my analysis which was based on this quote from a paper:

We also found that the probability of blocks created by utilizing nodes
to become orphan blocks is surprisingly smaller than that of the
non-utilizing nodes. Even in the worst case, the value of utilizing
nodes is 15% of the value of non-utilizing nodes.

- Effects of a Simple Relay Network on the Bitcoin Network

I had interpreted their results as that outbound propagation and inbound propagation was symmetric. But, if inbound propagation regularly is required to pass through security steps that outbound doesn't. Then obviously that can't be true. Now I'm realizing it could also have been implied from the partitioning that gmaxwell was talking about.

At this point I have to wonder how it's even possible to implement intentional withholding, especially at the typical global hashrate that exists in present day.

The P2P Protocol Addition

Fortunately it would be a very trivial change needing only changes in the node software and not the rest of the ecosystem.
-
So unless you feel like doing that work yourself, I wouldn't suggest holding your breath.

Well, frankly, this work is my attempt at doing something useful that also plays to my strengths to enter Bitcoin. I'd be willing to attempt a PR if you would be willing to share the details of the change in more depth.

Quick question though, even if a patch was written - if there were miners intentionally withholding - wouldn't they just skip this version to remain undetected?