Bitcoin Mempool Mismatch Between Nodes

[mvdheuvel1983]

Also while I certainly appreciate all of the suggestions to try and correct the issue, the real reason for me bringing up this matter isn't to get my two nodes synched up so that they have the same mempool but to rather it's to understand why there is a discrepancy in the first place as this may somehow be some kind of attack that we aren't aware of presently that needs to be looked at (call me paranoid but we are competing with the most powerful and best funded organizations in the history of the planet, right?).  

Thanks all.

I read through the thread and although I am not able to give my opinion to your question, this last paragraph got me. I want to ask that assuming this were some kind of attack as you have mentioned what are some of the measures you would have implemented to counteract it. What proactive steps would you have taken to not only halt the ongoing attack but also to establish safeguards that prevent its recurrence in the future?

[1714228340]

1714228340

[BitcoinCanSaveUsAll]

Hello again everyone. Thanks again for all the replies.


 o_e_l_e_o,
 
Yes I believe that the file is in the correct directory as I've opened it via the method you suggested and edited that file every time. I've also checked for spelling or other issues and haven't seen any.

mvdheuvel1983,

The first thing I'd do is bring the issue here to see if someone smarter than me might have a clue as to properly troubleshoot the exploit and then perhaps have a public dialog to brainstorm the issue.

All,

I'm honestly not interested in running any of the recent versions as I don't support taproot. Reason being is that it appears there's a consensus that this has actually made the ordinals and inscriptions possible which seems to directly correlate to the mismatches I've been seeing between my nodes.

Thanks.

[ABCbits]

All,

I'm honestly not interested in running any of the recent versions as I don't support taproot. Reason being is that it appears there's a consensus that this has actually made the ordinals and inscriptions possible which seems to directly correlate to the mismatches I've been seeing between my nodes.

Thanks.

Unfortunately, it's common misconception among Bitcoiner. Ordinals is fully possible without Taproot since the arbitrary data is stored on witness data. Actual Taproot mess up is no longer have 10000 script size limit[1]. And even without Taproot and SegWit, what Ordinals do could be replicated using OP_RETURN, address/pubkeyhash, multisig address and others with few limitation.

[1] https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#user-content-Resource_limits.

[o_e_l_e_o]

o_e_l_e_o,
 
Yes I believe that the file is in the correct directory as I've opened it via the method you suggested and edited that file every time. I've also checked for spelling or other issues and haven't seen any.

Odd. You could try copying your .conf file from the node which does have the 600 MB limit and pasting it over the .conf file for the node with the limit of 300 MB to see if that works. I don't use Windows, but often files having the wrong permissions can mess things up in Windows. Could this be the case here?

Also check you aren't launching with any command line options. These would override anything in your .conf file.

[BitcoinCanSaveUsAll]

Hi o_e_l_e_o,

Unfortunately I just hit another stumbling point. I tried clicking the "reset options" button which when I did it restarted my node and then tried to relaunch it. I've then started another thread to try to recover from that issue which I also posted in this message board.  Hopefully I can figure out how to get this node back online after the issue I'm having now so I can resume troubleshooting this maxmempool issue.

[BitcoinCanSaveUsAll]

Update:  After resolving the issue in my earlier post by editing the options file located here:  %APPDATA%\Bitcoin I realized that this was not in fact the same file that the GUI is pulling up when I clicked on the Settings>Options>Open Configuration File.  Perhaps this is due to the fact that I'm running my node with a non default path for the blocks directory on an external hard drive?  Either way I've been able to modify the correction options file (not the one the GUI pulls ups but the one here: %APPDATA%\Bitcoin and as soon as I did it correctly updated my maxmempool=600 value.  I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are.  Thanks again to all who gave their feedback here!

[NotATether]

Update:  After resolving the issue in my earlier post by editing the options file located here:  %APPDATA%\Bitcoin I realized that this was not in fact the same file that the GUI is pulling up when I clicked on the Settings>Options>Open Configuration File.  Perhaps this is due to the fact that I'm running my node with a non default path for the blocks directory on an external hard drive?  Either way I've been able to modify the correction options file (not the one the GUI pulls ups but the one here: %APPDATA%\Bitcoin and as soon as I did it correctly updated my maxmempool=600 value.  I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are.  Thanks again to all who gave their feedback here!

There shouldn't be any further discrepancies because when you use such a giant mempool limit, it should capture all of the transactions without dropping any of them - especially the ones with ultra-low fees.

At one point a week or two ago, I looked at mempool statistics and saw that the mempool with all transactions in it was a whole gigabyte large. So as everyone else's mempool reaches breaking point, you might start noticing this issue temporarily.

[o_e_l_e_o]

I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are.

You'll probably need to wait longer than that.

As I explained earlier in this thread, nodes do not automatically rebroadcast transactions, nor do they actively try to fetch transactions they don't know about from other nodes. They simply relay transactions (if valid) the first time they hear about them. So if there is a discrepancy of say 5,000 transactions which one of your nodes knows about and the other doesn't, that discrepancy will only decrease if:

A) These transactions are mined
B) Your node which knows about these transactions drops them, either for exceeding the size limit or exceeding the time limit
C) Someone (most likely the creator of these transactions) rebroadcasts them and there is a path from their node to your node of nodes which don't already know about the transactions (if a node receives a transaction already in its mempool, it will not relay it again)

The best way to check for ongoing issues would be to either wipe your both your mempools or to sync them to each other, so they are both starting from the same point.

[seek3r]

I'll let it run for the next few days to see if there are any further discrepancies and will report back if there are.

You'll probably need to wait longer than that.

As I explained earlier in this thread, nodes do not automatically rebroadcast transactions, nor do they actively try to fetch transactions they don't know about from other nodes. They simply relay transactions (if valid) the first time they hear about them. So if there is a discrepancy of say 5,000 transactions which one of your nodes knows about and the other doesn't, that discrepancy will only decrease if:

A) These transactions are mined
B) Your node which knows about these transactions drops them, either for exceeding the size limit or exceeding the time limit
C) Someone (most likely the creator of these transactions) rebroadcasts them and there is a path from their node to your node of nodes which don't already know about the transactions (if a node receives a transaction already in its mempool, it will not relay it again)

The best way to check for ongoing issues would be to either wipe your both your mempools or to sync them to each other, so they are both starting from the same point.

I think that his problem is already fixed. Correct me if I am wrong but he decided to reset his options via GUI.
He used the wrong data directory for his config file since he stored it on a external hard drive.
Refering to this post in another thread: https://bitcointalk.org/index.php?topic=5470575.msg63005361#msg63005361

He was using the pruning mode and txindex at the same time which caused that conflict. Since he disabled pruning he should be fine?!

[o_e_l_e_o]

He was using the pruning mode and txindex at the same time which caused that conflict. Since he disabled pruning he should be fine?!

Fixing that problem will allow him to launch his node since your cannot use tx-index=1 with a pruned node.

That won't make any difference to his mempools, though. Whether or not the node is pruned or not, or whether the node is using tx-index=0 or =1, should have no bearing at all on the size of its mempool.

[BitcoinCanSaveUsAll]

Hi all,

Yes my issue is resolved in respect to the hang up I experienced after Clicking on Settings> Options > Reset Options .  As I noted in my other thread I added the prune=0 option in the config file.  After doing so however I see that two different options I set in my configuration file there was a message that said:

Options set in the dialog are overridden by the command line or in the configuration file:
-prune=0 -dBcache=5000


The funny thing is that prior to the other thread I never had added the prune option to the option file at all. 

To summarize what I've noted so far with this particular issue however is that that there are apparently 2 different config files that are created with Bitcoin Core v.20.2 (maybe others too?) since when clicking Settings> Options > Open Configuration File the GUI button launches one, but that one isn't the one the system reads on start up.  The startup one is the one that needs to be modified and is stored in the %APPDATA%\Bitcoin directory.

Going forward, I'm curious to know if the 300 MB default limit is going to be adequate for the nodes going forward if there's constant mismatches between nodes when the default mempools being set to 300 MB.  Do anyone here believe this is a serious issue?  Perhaps this can somehow been an attack vector of sorts?  If so hopefully the core team is already looking into this.  Regardless I really do appreciate everyone's feedback here.

[o_e_l_e_o]

Going forward, I'm curious to know if the 300 MB default limit is going to be adequate for the nodes going forward if there's constant mismatches between nodes when the default mempools being set to 300 MB.

300 MB will likely cause a mismatch for some time, yes. The average memory usage of nodes' mempools has been above 300 MB pretty consistently for the last 6 months. Given this, two nodes both with the default limits will be dropping different transactions at different times and therefore will always have a mismatch between their mempools.

Do anyone here believe this is a serious issue?  Perhaps this can somehow been an attack vector of sorts?

There is only a possible attack vector if you start accepting unconfirmed transactions, which has never been safe. If you wait for any incoming transactions to receive at least a couple of confirmations, then whether or not those transactions were in the mempool of one, both, or neither of your nodes prior to being confirmed in a block is irrelevant.

[BitcoinCanSaveUsAll]

Thanks o_e_l_e_o.  I appreciate your feedback.  What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?   Maybe my transaction never gets to a miner's mempool since it's already full how then would it get confirmed?  I think nowadays the default amount of nodes that your node connects to is only 10, right? What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?  Anyway I really hope I'm just being paranoid and ignorant of the technical operations here but this seems quite odd to me.

[DaveF]

Thanks o_e_l_e_o.  I appreciate your feedback.  What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?   Maybe my transaction never gets to a miner's mempool since it's already full how then would it get confirmed?  I think nowadays the default amount of nodes that your node connects to is only 10, right? What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?  Anyway I really hope I'm just being paranoid and ignorant of the technical operations here but this seems quite odd to me.

If your fee is higher then the fees that in the TXs being dropped then you will just push a lower fee TX out of their mempool.

It's not something you should worry about. There are also a lot of well connected nodes and services that will broadcast your TX

https://live.blockcypher.com/btc/pushtx/

You don't even need to be running core, you can build your TX manually using python (and other code too) and push it out through a variety of ways.

-Dave

[seek3r]

Thanks o_e_l_e_o.  I appreciate your feedback.  What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?   Maybe my transaction never gets to a miner's mempool since it's already full how then would it get confirmed?  I think nowadays the default amount of nodes that your node connects to is only 10, right? What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?  Anyway I really hope I'm just being paranoid and ignorant of the technical operations here but this seems quite odd to me.

Yep, the maximum of outgoing connections is 10 for now. We discussed this today in this thread.
8 of them are full-relay and two additional block-relay connections.

To ur concern: When u are sending/broadcasting a new transaction from ur own node this TXID will propgated to the nodes that already connected to ur node. From there it will move forward to other nodes in the network.
If this transaction will not be forwarded to the miners mempool due to cap-limit of the mempool it needs to rebroadcasted again once it cleared or you have to adjust your fee.

I wouldn't necessarily be so paranoid about that because the Bitcoin network is designed to be decentralized. That's the strength of Bitcoin and that's what makes it special. Most bitcoiners resist as best they can when it comes to centralization.

[BitcoinCanSaveUsAll]

All,

I did transmit a transaction from my node (version 20.2) and I can see that there's been 6 blocks added to the blockchain since my transmission was sent.  The odd thing is that there has been also at least 2 retransmissions of my transaction according to my node's log.  I'm starting to think this can be a bigger deal.

Note that this fee was ~5.39 sats per byte and I believe that it was within the 20 minute expected range according to mempool dot space.  Yet it's been almost 2 hours now and nothing but attempted retransmissions from my node. Any thoughts on this would be much appreciated.  Thanks.

[BitcoinCanSaveUsAll]

Wow this is getting even stranger.  The good news is that my transaction did just confirm.  But the weird part about it was that my electrum wallet (it's connected to it's own default nodes not mine) showed that my transaction was confirmed about 10 minutes before my node seen it.  I wonder what would cause such a propagation delay to my node?  It is behind a VPN but it does have inbound connections as well.  Anyway I'm glad to see everything has settled.

Thanks again to everyone who replied to this message thread and helped me understand this process that much better!

[o_e_l_e_o]

What if however I wanted to send a transaction from my own node and the mempool is already clogged up beyond the 300 MB cap for most nodes with other transactions (maybe with inscriptions, ordinals etc.)?

Either you wait for the mempool to empty, or as DaveF says, you increase the fee and your transaction will evict some other lower paying transaction from these nodes' mempools.

I think nowadays the default amount of nodes that your node connects to is only 10, right?

10 outbound connections by default, but up to 115 inbound connections by default. Note that apart from the 2 block-relay connections, outbound and inbound connections are functionally identical - it's only the way the connections are established which are different.

What if this is some sly roundabout way to allow transactions to be made on CEXs/authorized transmitters or something like that?

It isn't. Anyone can increase their transaction's fees and gain access to a mempool which is already at its limit by evicting other transactions. Since mempools are run locally, it is always in miners' best interests to do this, since it allows them to maximize their profits.

I did transmit a transaction from my node (version 20.2) and I can see that there's been 6 blocks added to the blockchain since my transmission was sent.

Looking at the time this happened, you simply got unlucky with when blocks are mined. Keep in mind blocks are mined on average every 10 minutes, but this can vary from a few seconds to over an hour in reality.

Block 812,957 included transactions with a fee of 3 sats/vbyte. At this point, mempool.space was probably recommending a fee at around 5 sats/vbyte, and appropriately so. However, it then took 42 minutes to find the next block, by which point the minimum fee had increased to 24 sats/vbyte. Because of this backlog which was created during this 42 minute time period, it then took the network to block 812,970 to mine transactions at 5 sats/vbyte and include your transaction in a block. This had nothing to do with mempools being full - indeed, if nodes were rejecting your transaction then it would never have been mined at all. This is simply a quirk of how bitcoin works and it happens all the time.

[DaveF]

...I wonder what would cause such a propagation delay to my node?  It is behind a VPN but it does have inbound connections as well....

Could be the VPN. Remember, most VPN endpoints are abused and their IPs do get blocked even if it's not your fault.
I use the same VPN provider you do and have a malicious node, my node does not get blocked my IP does, which just so happens to be the same as yours.
Your node shows a connection but it's really not sending anything.
Have seen it now and then.

Are you running on at least an SSD with 8gb ram and a processor that is less then 10 years old?
Could also be hardware that can not keep up churning away on something.

-Dave

[BitcoinCanSaveUsAll]

Good morning everyone,

@ o_e_l_e_o,

I appreciate your explanation and the breakdown of the quirks of the mempool / mining process.  Also I didn't realize that the inbound and outbound connections were functionally identical but it makes sense when I think about it so thank you for that comment. 

@ DaveF,

Yes my computer has more RAM than that but I'm running the blockchain on an external HD that's connected via USB 3.1.  Also I believe that I am properly sending data to the bitcoin network behind the VPN as certain gateways for our VPN provider do allow for port forwarding which I've poked a hole in my router to open.  In fact over the last 20 hours or so it appears that my node has sent over 5GB of data.  At one point I had 37 inbound connections behind the VPN (now it's down to about 1/3 of that however).

Also just an FYI I did notice that the same issue with the block propagation delay to my node (behind the VPN) happened again as I waited for another transaction I sent shortly after I posted.  This issue however seems to happen fairly regularly as this has happened several times over at least the last year or two I've been spot checking with both with my node in the clear and the one running behind the VPN connection.  Regardless though everything seems to be synched up at this point. Either way I appreciate your feedback. Curious though, how did you know what my VPN provider was?  I'm assuming you seen the IP and looked it up but how did you get the IP?  Thanks.