Mempool / Current number of transactions question

[DaveF]

So I have 4 nodes running in my office.
One is for my electrum server, 2 are just there to play with and the 4th is on a cable modem as an off network test node.

So I noticed it before and did not really think about it but they all show a different number of pending transactions in the mempool.
All are on the same block. But as of now they run from ~400 to ~750 pending transactions. The last block 603381 cleared the mempool.

Earlier today I noticed a similar thing, but figured it was just the lag of me going from one server to the next. But now with 4 consoles up I see it in real time.

Is this normal or do I have a config or other issue someplace?

Thanks,
Dave

[1715313939]

1715313939

[1715313939]

1715313939

[1715313939]

1715313939

[1715313939]

1715313939

[1715313939]

1715313939

[Carlton Banks]

All the nodes should have different peers, so the mempool contents will never be identical, so diagnosis normal.

However, maybe you've changed the minimum relay amount on some of them? Or, do some run 24/7, while others are shutdown for any length of time? Those factors will contribute to how full a given node's mempool is, and/or how full it can potentially be. Plus, the propagation logic is subtle, it certainly doesn't try to get every tx relayed to all peers ASAP.

[DaveF]

All the nodes should have different peers, so the mempool contents will never be identical, so diagnosis normal.

However, maybe you've changed the minimum relay amount on some of them? Or, do some run 24/7, while others are shutdown for any length of time? Those factors will contribute to how full a given node's mempool is, and/or how full it can potentially be. Plus, the propagation logic is subtle, it certainly doesn't try to get every tx relayed to all peers ASAP.

Cool, thanks. I just saw them all drop to 0 and a few minutes later all had wildly different numbers. If one was at 450 and another at 460 and the others also in that range then I would not think twice but 400 to 750 just seemed way off.

However, maybe you've changed the minimum relay amount on some of them? Or, do some run 24/7, while others are shutdown for any length of time? Those factors will contribute to how full a given node's mempool is, and/or how full it can potentially be. Plus, the propagation logic is subtle, it certainly doesn't try to get every tx relayed to all peers ASAP.

Nope, outside of the basic changes on one node to the conf file to run electrum all have no settings in the conf file outside of an addnode= the cable ip or in the case of the cable node it points back to the electrum node.

All are up 24/7
Probably not important in the real world, I had just never seen it before.

How often do you really look at the mempool on your nodes.

Thanks,
Dave

[DaveF]

I have 2 guess :
1. Gossip protocol in Bitcoin isn't perfect where there's no guarantee a transaction will be received by all nodes
2. Some of your devices might have firewall which limit or block connection between node

Going to have to go with #1
The nodes are all behind their own firewalls, but they are all clones of one another. So in that respect they should all be the same.
Looking now they are all within 10 transactions of each other. It was just really weird to see such a difference yesterday with an empty mempool.

Not going to worry about it unless I see it again and can look to see what is different at that moment.

-Dave

[Carlton Banks]

I have 2 guess :
1. Gossip protocol in Bitcoin isn't perfect where there's no guarantee a transaction will be received by all nodes
2. Some of your devices might have firewall which limit or block connection between node

Going to have to go with #1

"perfect" in this case means non-aligned or dis-harmonious, future changes are likely to make mempool contents slightly less aligned than they are now.

Any reason why you need to hear this more than once?

The nodes are all behind their own firewalls, but they are all clones of one another. So in that respect they should all be the same

Are they connected to each other, or to the wider internet? It sounds like the latter, and so you cannot expect them to be receiving the exact same relayed transactions when each node will be connected to different  peers. Again, this is by design. There are both tx privacy and network robustness reasons for this.

Your concern is entirely misplaced. You should be feeling the opposite: if you had completely identical mempools across your separate nodes, it would be a symptom of something being very wrong with the bitcoin network, and that the algorithms to diversify node connections across the internet and to obscure the source of a given tx were not performing well.

[HCP]

OP, it seems that you believe that an identical config should result in an identical local mempool across different/separate nodes.

As Carlton Banks has stated, that is a flawed belief, for the various reasons he has mentioned.

[DaveF]

So I noticed it before and did not really think about it but they all show a different number of pending transactions in the mempool.
All are on the same block. But as of now they run from ~400 to ~750 pending transactions. The last block 603381 cleared the mempool.

HCP / Carlton / whoever else.

Yes I know mempools will never be identical.
That is not what I was asking. I can accept 10% even 30% differences between them. What was odd and the gist of my question was should 1 node have almost 2x more transactions then the node next to it.

The answer seems to be yes from what all of you have said. They can be wildly different.
It just seems like a big variance. Which was why I asked.

I probably could have been clearer in how I asked it, that's on me. Sorry about that, I was doing too many things at once.

-Dave

[Carlton Banks]

so I kind of see what you mean, but it's really not an unexpected difference


Given nodes A and B, running uninterrupted 24/7/365:

• A having 450 and B simultaneously having 750 is not unusual
• A having 4500 and B having simultaneously having 7500 is a little unexpected
• A having 45,000 and B having simultaneously having 75,000 suggests a problem or a config difference between the nodes (max mempool size and min relay rate can be user configured, and could elicit such a disparity)
• A having 45 and B simultaneously having 75 is not unusual if transaction load is low (i.e. you could've seen this on December 25th every year up to now)
• A having 4 and B simultaneously having 7 is very unlikely, and suggests a problem with both nodes (or that you're on a Bitcoin ForkCoin )

So a whole number multiple in difference only really ought to raise eyebrows when the total network load is very high. At relatively low loads, wide variations can easily exist within expectations of proper operation. As you saw yourself, the difference lasted only briefly at low loads anyway. The main material difference is probably that the node with 450 tx mempool dowloaded the whole block, whereas the node with 750 tx mempool could reconstruct the block from it's mempool using compact block hints.