Is bitcoinfees.earn.com systematically overestimating fees?

[Quickseller]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

[1714981854]

1714981854

[1714981854]

1714981854

[1714981854]

1714981854

[TheQuin]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

I wasn't suggesting that there was a scandal and I've stayed away from putting a tinfoil hat and suggesting that they do this deliberately. I started the thread because something seemed wrong with the way it is working and I was trying to find out what. Since then as I've watched more I've become convinced if they just set that -mempoolexpiry to 72 they would improve their fee estimation. Until then I'll just recommend people use other fee estimators.

[Quickseller]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

I wasn't suggesting that there was a scandal and I've stayed away from putting a tinfoil hat and suggesting that they do this deliberately. I started the thread because something seemed wrong with the way it is working and I was trying to find out what. Since then as I've watched more I've become convinced if they just set that -mempoolexpiry to 72 they would improve their fee estimation. Until then I'll just recommend people use other fee estimators.

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

[TheQuin]

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

That's missing where the problem is. From what I've seen all the mining pools are all using a 72hr expiry so earn.com is factoring in transactions that the miners will never include in blocks. That is what is inflating their estimation.

[Quickseller]

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

That's missing where the problem is. From what I've seen all the mining pools are all using a 72hr expiry so earn.com is factoring in transactions that the miners will never include in blocks. That is what is inflating their estimation.

If a valid transaction does not get confirmed after 72 hours of being broadcast, it will be because the tx fee was too low. Including transactions that will never confirm (because the tx fee is too low) means the estimate will be lower, not higher.

That argument also ignores the potential for users to rebroadcast their transactions after xx time, which is often sufficient to get a transaction "unstuck".

[TheQuin]

OK, if you don't think it is the -mempoolexpiry can you explain this another way?

I've just taken the following screenshots immediately after a block has been found.

earn.com


jochen-hoenicke.de


The earn.com mempool has 601 transactions queued at 171+ that the mining pools are ignoring. jochen-hoenicke.de mempool has only 4 transactions above 170.

If they factor in 600 high fee transactions that the mining pools are ignoring then obviously that is going to inflate their fee estimate. The -mempoolexpiry is the best explanation I can come up with, can you think of a better one?
 

[RNC]

I tend to watch this site for fees.

https://bitinfocharts.com/comparison/bitcoin-transactionfees.html

Says the price per transaction is about $2 just now which is a number that I can understand without having
eight decimal places and doing sums.

One year ago the price was about $0.10 per transaction as I remember it and many wallets don't offer the
option of setting your own fees so this could be the reason why the sums don't seem to add up.

[TheQuin]

I tend to watch this site for fees.

https://bitinfocharts.com/comparison/bitcoin-transactionfees.html

Says the price per transaction is about $2 just now which is a number that I can understand without having
eight decimal places and doing sums.

One year ago the price was about $0.10 per transaction as I remember it and many wallets don't offer the
option of setting your own fees so this could be the reason why the sums don't seem to add up.

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

[RNC]

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

Sorry but the title of this thread says "bitcoinfees.earn.com systematically overestimating fees" so
maybe my link to a charts that puts fees today at about $2.00 per transaction is "Off-Topic" but only
to people that are spreading FUD about $0.09 fees and you should contact bitinfocharts.com if you
are saying the numbers are wrong.

[TheQuin]

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

Sorry but the title of this thread says "bitcoinfees.earn.com systematically overestimating fees" so
maybe my link to a charts that puts fees today at about $2.00 per transaction is "Off-Topic" but only
to people that are spreading FUD about $0.09 fees and you should contact bitinfocharts.com if you
are saying the numbers are wrong.

Read my previous reply "That site is giving a historical chart of what people have paid not an estimation of the fee required."

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

[RNC]

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

Anything that's not promoting your fake image of Bitcoin is "off-topic" and if pointing out that fees are much higher than you
are trying to make out upsets you then please go ahead because my estimator program between my ears puts the number at
about $2.00 just now so stop being a drama queen, I have no time for that.

[Thirdspace]

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

Anything that's not promoting your fake image of Bitcoin is "off-topic" and if pointing out that fees are much higher than you
are trying to make out upsets you then please go ahead because my estimator program between my ears puts the number at
about $2.00 just now so stop being a drama queen, I have no time for that.

read the topic! read all the replies in this thread 
all mentioned and discussed about sites that deal with fee estimation, in other word prediction
he, TheQuin, did explicitly say stay on topic talking about "fee estimators" and you insisted on talking about bitinfocharts.com
who is promoting what site?

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.
---snip---

---snip---
... you should contact bitinfocharts.com if you are saying the numbers are wrong.

he never said "the numbers are wrong"  but that site info is irrelevant to the discussion
read the title on that page, it is written in bold
"Bitcoin Avg. Transaction Fee historical chart" and no hints on estimation fee

[TheQuin]

Thankyou @Thirdspace. I would have thought it obvious that the average fee paid has to be by definition higher than the minimum fee required.

Back to bitcoinfees.earn.com and their mysteriously bloated mempool. All these screenshots were taken at the same time just after a block was found with not enough transactions to even fill it.

https://i.snag.gy/lEHQzT.jpg


earn.com showing 3,244 unconfirmed transactions that the mining pools don't see. Resulting in an estimated fee of 30 sats/byte.

https://i.snag.gy/PQGBbz.jpg


jochen-hoenicke.de shows only 67 unconfirmed transactions.

https://i.snag.gy/lvgWM3.jpg


coinb.in/#fees recommends a fee of 1 sat/byte.

[Quickseller]

OK, if you don't think it is the -mempoolexpiry can you explain this another way?

I've just taken the following screenshots immediately after a block has been found.

earn.com


jochen-hoenicke.de


The earn.com mempool has 601 transactions queued at 171+ that the mining pools are ignoring. jochen-hoenicke.de mempool has only 4 transactions above 170.

If they factor in 600 high fee transactions that the mining pools are ignoring then obviously that is going to inflate their fee estimate. The -mempoolexpiry is the best explanation I can come up with, can you think of a better one?
 

Jochen very specifically only includes unconfirmed transactions while earn includes both confirmed and unconfirmed transactions in each of their relevant time periods. Those websites give very different information.

If I broadcast a valid transaction with a transaction fee of 350 sat/byte, my transaction will remain on Jochen until the following block when, as of now, it is almost certain to get confirmed, while it will remain on earn for two weeks.

Also, just because the miners cannot find sufficient transactions to fill up a block in its entirety does not mean there are no remaining paying transactions out there. If a transaction will have a cost to the miner in the risk that including that additional transaction will incrementally increase the risk the block getting orphaned is greater than the transaction fee (both within the transaction, and out of band), then the transaction will not confirm, even if there are no other paying transactions.

[TheQuin]

Jochen very specifically only includes unconfirmed transactions while earn includes both confirmed and unconfirmed transactions in each of their relevant time periods. Those websites give very different information.

The earn.com website also is only showing unconfirmed transactions in the higher (green bars). The discrepancy between this and what Jochen shows and what is getting included in blocks is what I'm pointing out. The legend above could be clearer as to what the two bars are:
https://i.snag.gy/FSAYyn.jpg

"Unconfirmed transactions / Transactions today" makes sense but "# OF TRANSACTIONS IN MEMPOOL IN LAST 336 HOURS" is misleading. If it was really that then the number would HAVE to be much higher than "Transaction Today". They just mean their mempool has a 336hr expiry on unconfirmed transactions.

If I broadcast a valid transaction with a transaction fee of 350 sat/byte, my transaction will remain on Jochen until the following block when, as of now, it is almost certain to get confirmed, while it will remain on earn for two weeks.

No, it will move from the higher bar to the lower bar as it is now confirmed. If you don't believe me then spend some time watching it and see what happens after a block is found.

Also, just because the miners cannot find sufficient transactions to fill up a block in its entirety does not mean there are no remaining paying transactions out there. If a transaction will have a cost to the miner in the risk that including that additional transaction will incrementally increase the risk the block getting orphaned is greater than the transaction fee (both within the transaction, and out of band), then the transaction will not confirm, even if there are no other paying transactions.

That's the heart of the issue here. Every node on the network has its own version of the mempool. In order to make a realistic fee estimation it would be logical to try and have a mempool as similar to the ones the mining pools are using as possible. The fact the earn.com's mempool contains many more transactions than the mining pools mempools is causing them to overestimate fees.

The -mempoolexpiry parameter is the only explanation I can come up with, but I'd like to hear from anyone who has another possible technical explanation for the discrepancy.