Bitcoin Core 0.14 mempool saving feature - possibly not the best solution?

[AngryDwarf]

Retaining the Mempool Across Restarts
- -------------------------------------

The mempool will be saved to the data directory prior to shutdown
to a `mempool.dat` file. This file preserves the mempool so that when the node
restarts the mempool can be filled with transactions without waiting for new transactions
to be created. This will also preserve any changes made to a transaction through
commands such as `prioritisetransaction` so that those changes will not be lost.

I thought this was a great idea as someone who runs a node on a multiboot Windows/Linux system. But I am beginning to wonder if this is a very good solution at all.

Perhaps a better solution would be for a newly started node to obtain its transaction pool from its peers?

If a node is down for any length of time its transaction set is so out of date. During its downtime, many transactions may have been missed.

If the node network consists of many high up time nodes, then the chance for a double spend to get through is minimal. However, if there are many part time nodes, there is a chance for a double spend to get through between sender and recipient, which may get rejected by the miners nodes.

I think the reason to do this is to prevent a possible double spending path through 'part-time' nodes which do not run 24x7, when the transaction may get rejected by the mining pools. That is quite important if zero confirmation economic risk is to have any meaning.

The idea of perfectly synchronised transaction pools is probably a fruitless endeavor in an asynchronous peer to peer system. But a system where the transaction pools are close enough to synchronisation is probably enough to minimise the chance of a double spend getting through.

I came to this conclusion when asking for this feature to be applied to alternative node implementations, and it would seem that some people agree:

https://github.com/bitcoinclassic/bitcoinclassic/issues/251
https://github.com/bitcoinxt/bitcoinxt/issues/195
https://github.com/BitcoinUnlimited/BitcoinUnlimited/issues/438

So I have to ask why was this feature implemented in this manner?

[1715093711]

1715093711

[1715093711]

1715093711

[1715093711]

1715093711

[1715093711]

1715093711

[-ck]

Personally I find this feature very useful across node restarts where the downtime is virtually zero. If I upgrade my configuration in such a way that it requires a restart, or upgrade my node to newer code and then restart it, the mempool is otherwise empty which is no good for a mining pool node. This feature is pretty much useful only for virtually immediate restarts.

[AngryDwarf]

Personally I find this feature very useful across node restarts where the downtime is virtually zero. If I upgrade my configuration in such a way that it requires a restart, or upgrade my node to newer code and then restart it, the mempool is otherwise empty which is no good for a mining pool node. This feature is pretty much useful only for virtually immediate restarts.

From what I can work out from the code it looks like this is all it is useful for, and is probably it's primary intention (retaining state like prioritisetransaction as per the release note). The file is only written at clean closedown, so it is of limited use during an unexpected system failure (I think it will just load up the old mempool.dat file from the last clean closedown).
As a pool operator, I assume you have some kind of redundancy in your system anyway? If you had a system failure on one node, you would want a way to populate that before it is brought back online to maximise fees by avoiding mining an empty block?

It's kind of useful for myself for keeping an eye out on the transaction pool size, so I can send transactions during low demand where possible.

So it does make me wonder if there is some value in implementing the solution outlined in my OP.

[gmaxwell]

Perhaps a better solution would be for a newly started node to obtain its transaction pool from its peers?
[...]
So I have to ask why was this feature implemented in this manner?

Without saving the sync would be a _massive_ waste of bandwidth, so it was important that the saving be done first. The two are complementary.

Saving was also substantially driven by the motivation of being able to save prioritization for miners; which is not accomplished via syncing.

Saving is purely a client local feature-- it's something the Core project can decide to do without worrying if other implementations like it (and, in fact, Jeff Garzik showed up out of nowhere, having not sent a single message to the project in months, and attacked the saving vigorously)...  syncing, especially efficiently, requires a BIP and collaboration with other implementations some of which are hostile to the project and reliably oppose every proposal.  So it didn't make sense to hold up the saving for the syncing to be worked out, since saving really is no other implementation's business.

Efficient syncing is also an active area of research, there are techniques to perform the sync in ways which are significantly more efficient than you might guess. (But, of course, if you throw out the data-- it's going to have to be transferred again, saving it is the only way to avoid that!)

The file is only written at clean closedown, so it is of limited use during an unexpected system failure

An earlier version of the proposal did periodic flushes, but we're concerned with creating excessive writes that lower the lifetime of SSD, and having half completed write can be worse than no write since we'll lose the in-progress flush.   The ability to trigger a flush via an RPC will probably come back, so if you want a periodic flush you'll be able to crontab it.

This feature is pretty much useful only for virtually immediate restarts.

Not entirely, it's also useful to pick up some low feerate stuff that otherwise wouldn't be mined for hours or days... so even if you were off for a few hours it can have value.  If you're off for weeks it would be worthless, however.

Saving also improves the accuracy of data for long term fee estimation.

[AngryDwarf]

Thank you for the info. Very useful.