Is the 'fee estimation currently not possible' problem getting worse?

In the years I have used Bitcoin Core wallet, the 'fee estimation is not currently possible' warning only turned up occasionally. Recently, it happens frequently and takes longer to go away and calculate a fee. What could be causing this?

Is your Bitcoin Core fully synchronized? This error pops up if your client doesn't have enough transactions in the mempool to make any estimation (ie. still being synchronized), or if you've got sufficient transactions but Bitcoin Core hasn't seen them being confirmed yet to build a reasonable estimate for their priority system.

This problem should go away if you're fully synchronized, and you've let the client run for a short period of time.

Quote from: nicholas480 on July 16, 2024, 04:30:56 AM

Fee estimation in Bitcoin Core gets problems many times but were fixed and issues you have now can come from your side.

You should not rely on the wallet estimation for fee when you are broadcasting your transaction.

You can use it as one source and double check with some websites for mempool information includes recommended fee rates.

https://jochen-hoenicke.de/queue/#BTC%20(default%20mempool),24h,weight
https://mempool.space/
https://mempool.jhoenicke.de/#BTC,24h,weigh

Or register this bot on Telegram @BitcoinFeesAlert_bot, The bot announcement

Quote from: nicholas480 on July 16, 2024, 04:30:56 AM

Recently, it happens frequently and takes longer to go away and calculate a fee.

I can only say it doesn't happen to me. My Bitcoin Core is always running, although I occasionally put my laptop in sleep mode.
I prefer to set fees manually though: Johoe's Statistics or mempool.space give a better indication of current fees than Bitcoin Core itself.

Quote from: ranochigo on July 16, 2024, 04:35:28 AM

This problem should go away if you're fully synchronized, and you've let the client run for a short period of time.

But if there are many unconfirmed transaction on mempool, I noticed it could take decent amount of time before Bitcoin Core obtain most of those.

Quote from: ABCbits on July 16, 2024, 09:00:40 AM

But if there are many unconfirmed transaction on mempool, I noticed it could take decent amount of time before Bitcoin Core obtain most of those.

It's at most 300 MB by default, that shouldn't take longer than catching up on the latest blocks if you run Bitcoin Core once a day.

Quote from: LoyceV on July 16, 2024, 09:09:56 AM

Quote from: ABCbits on July 16, 2024, 09:00:40 AM

But if there are many unconfirmed transaction on mempool, I noticed it could take decent amount of time before Bitcoin Core obtain most of those.

It's at most 300 MB by default, that shouldn't take longer than catching up on the latest blocks if you run Bitcoin Core once a day.

Based on my past experience which doesn't accept incoming connection, 300MB mempool isn't the problem. The problem is rate of received unconfirmed transaction could be slow (less than 20 TX/s), which is problematic when there are so many unconfirmed transaction. For example, https://mempool.space/ currently state there are about 177K TX with 759MB mempool (about 70K with 300MB mempool). I haven't checked whether it's also happen with newest version of Bitcoin or if you accept incoming connection.

Quote from: ABCbits on July 16, 2024, 09:20:36 AM

It depends on how well-connected, or how loose your peer's mempool policies are. Johoe and Mempool.space are very well-connected and are likely to have transactions that would otherwise be purged from the reference nodes with default settings.

Other than that, Bitcoin Core uses inv messages to notify their peers of the blocks and the transactions that they have. For which, each node will subsequently request for the transactions that they're not aware of using getdata. I'm not exactly sure what is the exact limit, I don't recall exactly seeing a limit for that. Mempool.space and others are displaying the new transactions that are being relayed, as opposed to ones that are offline for too long.

Regardless, by how Bitcoin Core estimates fees, you don't actually have to know the entire mempool to get a good estimate. This is further compounded by the fact that you're only looking at the higher quantiles, which is pretty much guaranteed by the minfees limit set by mempool.

Quote from: nicholas480 on July 16, 2024, 04:30:56 AM

Recently, it happens frequently and takes longer to go away and calculate a fee. What could be causing this?

Since it's specific to your use-case, that can't be answered for certain without providing these info:

Have you updated your Bitcoin Core recently?
Have you set any option in your bitcoin.conf or started using command line args since then?
How often do you use Bitcoin Core or how long do you keep Bitcoin Core running on average?
Regarding the question above, is it the same as your usual usage before the fee estimation begin to initiate slower?

If your node is not fully synced then you will get the following error when you try to estimate the fee (on cli, using estimatesmartfee):

Code:

{
  "errors": [
    "Insufficient data or no feerate found"
  ],
  "blocks": 0
}

A node is considered out-of-synced if it has missed a full day's worth of blocks. In other words, if you shut if off for 24 hours or more then the node will resume the IBD process but at 99% or something, and you won't be able to use a few RPC calls until it's done.

Look if you go here

https://mempool.jhoenicke.de/#BTC,4d,count

it tells you current tx by sats this is about 5 minutes off which is within reason as it is 2-6 minutes slow on average. it reads 3:29 my post is 3:34

it shows 117,000 tx at 3 sats which mean that won't clear
it shows 2,348 tx at 4 sats which means it could clear with a few quick blocks
it shows. 1,474 tx at 5 sats which is likely to clear in 2-3 blocks
it shows 1,154 tx at 6 sats which is likely to clear in 1-2 blocks
it shows 948 tx at 7 sats which is likely to clear next block

all of the above you can see below and mentally place your best guess.

it pretty fucking easy to do it the way I am saying.

If I was doing an important send I would spend 7.15 sats byte and feel pretty safe
If I was doing a critical time based send I would spend 9.15 sats per byte and not worry at all

If I was consolidating my own wallets where time is unimportant I would do 4.46 sats

I have more than one wallet:

big
medium
small

So consolidating them is not time based as I only do 1 at a time. and I almost never consolidate or add to the big wallet.

I hope this helps

Quote from: philipma1957 on July 25, 2024, 07:34:45 PM

Look if you go here

https://mempool.jhoenicke.de/#BTC,4d,count

--snip--

It's great suggestion. But i would recommend https://mempool.space/graphs/mempool instead which show total size (rather than total tx). You could make guess knowing a block can have size up to 1 MvB.

Quote from: ABCbits on July 26, 2024, 09:14:58 AM

It's great suggestion. But i would recommend https://mempool.space/graphs/mempool instead which show total size (rather than total tx). You could make guess knowing a block can have size up to 1 MvB.

Probably a better estimation than transaction counts. However, that also depends on the rate that the mempool is going to be filled up. Mempool right after a block has been mined, or after successive blocks can look very different than after say 10 minutes on average. I find Core's estimation to be more forgiving and accurate for newbies if they don't know what they're doing.

A trick for this would be to look at the fee rates right before the dip. It would give a fair estimate of the mempool state before the last block.