Is bitcoinfees.earn.com systematically overestimating fees?

[TheQuin]

I've been intrigued for some time as to why the recommended fee on bitcoinfees.earn.com is usually at least 10 times what you really need to pay to get included in the next block. Over the last few days, I've noticed that they show about 600 transactions at 211+ and about another 650 at 141-150. After each block is found these do not disappear.

This screenshot was taken immediately after a block which included many transactions at 5 sats/byte.



These transactions are not shown on https://jochen-hoenicke.de/queue/#1,2h

So as every node has its own version of the mempool these transactions for some reason must not be getting propagated around the network. Obviously, if the data they are using to estimate fees has 1200 high fee transactions that cannot get confirmed then their estimate is going to be much higher than it needs to be.

How does this happen? Is this a bug?

[1715661807]

1715661807

[1715661807]

1715661807

[1715661807]

1715661807

[OmegaStarScream]

It could be done on purpose because as far as I can remember, their suggested fees were always high compared to what you actually need to pay to get your transaction confirmed so If it was a technical issue, they would've figured it out already. A better alternative (far more accurate): https://coinb.in/#fees

[TheQuin]

I was trying to stay away from the tinfoil hat route of them doing it on purpose, although I do acknowledge that anything is possible. Personally, I like to work out fees from https://jochen-hoenicke.de/queue/#1,2h as it is really easy to see what is included in each block and if there is a trend up or down. I'll even admit that the estimated fee on BTC.COM is pretty reliable for anyone who doesn't want to get involved with working it out.

I've been watching this for a while and I'm just trying to work out any technical explanation for their mempool to have all these transactions seemly stuck and not being propagated around the network.

[buwaytress]

I've been intrigued for some time as to why the recommended fee on bitcoinfees.earn.com is usually at least 10 times what you really need to pay to get included in the next block. Over the last few days, I've noticed that they show about 600 transactions at 211+ and about another 650 at 141-150. After each block is found these do not disappear.

This screenshot was taken immediately after a block which included many transactions at 5 sats/byte.



These transactions are not shown on https://jochen-hoenicke.de/queue/#1,2h

So as every node has its own version of the mempool these transactions for some reason must not be getting propagated around the network. Obviously, if the data they are using to estimate fees has 1200 high fee transactions that cannot get confirmed then their estimate is going to be much higher than it needs to be.

How does this happen? Is this a bug?

Bitcoinfees has been my most-used estimator and has always been very accurate in helping me estimate a good fee to pay (I seldom seek next-block confirmation). However, someone (DH I think) correctly pointed out a few weeks ago that their "fastest and cheapest fee" is far above what is actually efficient. I don't recommend to use it anymore, but their raw numbers still look pretty good to me.

Note that they keep the data in two ranges... the first bar is for 14 days (336 hours) and the second is for 1 day (24 hours) so the data will stay there for a while. Maybe they're doing it on purpose, maybe they just use a different, less flexible dataset. I wouldn't know.

It could be done on purpose because as far as I can remember, their suggested fees were always high compared to what you actually need to pay to get your transaction confirmed so If it was a technical issue, they would've figured it out already. A better alternative (far more accurate): https://coinb.in/#fees

I think when I first started using it, it was pretty okay, but never paid attention to their recommended fees, just used the bars to estimate manually. But yeah, it's been pointed out for a while now so they've had plenty of time to fix it. Negligence or deliberation?

And yeah, coinb.in seems to be the better alternative now!

[jseverson]

I also use it exclusively for my transactions and have never run into problems. I only ever use it to see how much people are paying for their transactions though, and have never once followed their recommended fee. I always assumed that that was just for people who absolutely needed to confirm their transactions by the next block, which is why it's as absurd as it is. The hourFee figure in their API seems to provide a much more reasonable estimate at 10 sats/byte, but it's still quite high for the current volume, and could actually be enough to confirm a transaction by the next block.

For what it's worth, I've never seen their graph look like that, but I don't rule out the possibility that they may be trying to inflate fees.

https://coinb.in/#fees

Will probably use this from now on, thanks!

[TheQuin]

Note that they keep the data in two ranges... the first bar is for 14 days (336 hours) and the second is for 1 day (24 hours) so the data will stay there for a while. Maybe they're doing it on purpose, maybe they just use a different, less flexible dataset. I wouldn't know.

The first bar is what is in the mempool, ie. what is queued up waiting to be confirmed. How long a transaction is held in a node's mempool is configurable so they're just telling you that theirs is set to 14 days. If you look on https://www.blocktrail.com/BTC, for example, they always show a smaller number of transactions queued because they set theirs to kick out anything older than 3 days.

The second bar is the number of transactions that have been confirmed in the last 24hrs. That figure would be the same anywhere regardless of how the mempool is configured, everyone gets the same blockchain.

So what I'm seeing here is queued transactions not being confirmed even though they have higher fees than transactions than those that are being confirmed.

[LoyceV]

I've seen several fee estimating website give very high estimates. I can't help but think some of them might have some financial gains from high fees as a motivator (aka miners).

A better alternative (far more accurate): https://coinb.in/#fees

This has become my new favourite now. Note that you can still go under the recommended if you don't need it confirmed within the next few blocks.

I try to pay as little as possible for two reasons:
1. safe money
2. don't join the fee war by making all fees higher, after which we all still wait.

It's amazing 2 sat/byte is enough now for a fast transaction, in December and January many transactions with even 600 sat/byte got stuck for days.

Bitcoin Core's estimate is 1.177 sat/byte to confirm within 24 hours. For 20 minutes, it estimates 5.4 sat/byte, still more than coinb's recommendation.

[TheQuin]

Update on this. Currently, the mining pools are not finding enough transactions to fill blocks, the mempool is empty as far as they are concerned.



But bitcoinfees.earn.com is still showing many transactions in their mempool including a lot at high fees.



I'm pretty convinced that setting the mempool to keep transactions for 14 days must be inconsistent with the settings used by the mining pools and this is a large part of what is causing them to consistently overestimate fees.

[Samarkand]

...
A better alternative (far more accurate): https://coinb.in/#fees

This has become my new favourite now. Note that you can still go under the recommended if you don't need it confirmed within the next few blocks.

...

The fee estimator by coinb.in is great!

However, I think that the following fee estimator is even better:
https://whatthefee.io/

If you use CELL feerate (sat/vbyte) you will have 1+ confirmations after at
most ROW time with at least COLUMN % certainty.

[Stedsm]

I believe this is probably getting done ON-PURPOSE only, else there's no reason as to why would they keep showing the charts with more than 10 sats/byte (or even more comparatively) to be sent in a transaction to get it confirmed early, while those with even 3-5 sats/byte are getting involved in the next block after they were done. I don't follow Bitcoinfees anymore because the one you gave me - https://jochen-hoenicke.de/queue/#1,2h seems to be providing with the best estimates needed for me to save a lot in my transaction fees.

[TheQuin]

bitcoinfees.earn.com is great website, the only problem is they give recommendation fee for very fast confirmation a.k.a next-block confirmation after broadcast the transaction. Also, i think the data isn't real time since they mentioned that "The predictions are based on blockchain data of the last 3 hours, as well as the current pool of unconfirmed transactions (mempool)." which cause the problem you mentioned.

I think simply see their vertical bar and choose lowest fees with fastest delay 1 block is good enough for everyday usage.

But just have a look at the images I posted above. The data in those bars doesn't match what the miners are doing. They have old transactions that are not getting confirmed that stay in those bars and make the estimate you can make wrong.

[Thirdspace]

"Is bitcoinfees.earn.com systematically overestimating fees?"
It looks just like that!
https://btc.com/stats/unconfirmed-tx 1 Satoshis/vbyte | 0.00001 BTC/KvB
https://bitcoinfees.earn.com/ 90 satoshis/byte
https://coinb.in/#fees 1 Sat/Byte

btc.com uses his own mempool to estimate the recommended fee
coinb.in uses specific tx fee rate used in recent block (previous ~3 blocks)
bitcoinfees.earn.com, I don't know which mempool source they used to estimate the fee
I think last december/january high tx fee rate was the product of misplaced trust in bitcoinfees.earn.com estimation fee

[jtipt]

Seems like someone at bitcoinfees.earn.com saw this thread, because they have now lowered the recommended fees to 10 sat/byte like everyone else.

[squatter]

Seems like someone at bitcoinfees.earn.com saw this thread, because they have now lowered the recommended fees to 10 sat/byte like everyone else.

You'd still be overpaying by following them. It looks like 10 sat/byte is the lowest they estimate -- they round to the top of each fee range and it jumps from 0 to 1-10. That's pretty annoying during low fee periods like this.

None of the estimators are great. I sent an urgent transaction earlier today during an apparent traffic spike. Core estimated the next-block fee ~ 65 sat/byte. Earn.com estimated it at 100 sat/byte. I sent it ~ 20 sat/byte and it got confirmed in the next block. I should have checked the most recent blocks; 1-2 sat/byte transactions were confirming.

Core seems pretty slow to react to drops in mempool size. But earn.com seems slower, by hours.

[TheQuin]

bitcoinfees.earn.com, I don't know which mempool source they used to estimate the fee
I think last december/january high tx fee rate was the product of misplaced trust in bitcoinfees.earn.com estimation fee

I'm not certain but I'm guessing they are running their own node.

Seems like someone at bitcoinfees.earn.com saw this thread, because they have now lowered the recommended fees to 10 sat/byte like everyone else.

You'd still be overpaying by following them. It looks like 10 sat/byte is the lowest they estimate -- they round to the top of each fee range and it jumps from 0 to 1-10. That's pretty annoying during low fee periods like this.

None of the estimators are great. I sent an urgent transaction earlier today during an apparent traffic spike. Core estimated the next-block fee ~ 65 sat/byte. Earn.com estimated it at 100 sat/byte. I sent it ~ 20 sat/byte and it got confirmed in the next block. I should have checked the most recent blocks; 1-2 sat/byte transactions were confirming.

Core seems pretty slow to react to drops in mempool size. But earn.com seems slower, by hours.

Right now the mempool is completely empty and miners can't find enough transactions to fill the blocks but earn.com still shows many transactions queued up at high fees.

https://i.snag.gy/rEwRaU.jpg

[Thirdspace]

Seems like someone at bitcoinfees.earn.com saw this thread, because they have now lowered the recommended fees to 10 sat/byte like everyone else.

yeah bitcoinfees lowered it, but now I'm seeing weird situation
btc.com 100 Satoshis/vbyte | 0.001 BTC/KvB
bitcoinfees.earn.com 10 satoshis/byte
coinb.in 1 Sat/Byte
btc.com's graph shows an increase in mempool size started around early midnight on 2018-02-24 for a whole day
now it's already returned back to normal low mempool size but fee rate still 100s/B while bitcoinfees 10s/B
only coinb.in shows 1s/B because it uses reference fee from transaction in block 510839 (5 blocks back)
like I said... weird

[TheQuin]

Seems like someone at bitcoinfees.earn.com saw this thread, because they have now lowered the recommended fees to 10 sat/byte like everyone else.

yeah bitcoinfees lowered it, but now I'm seeing weird situation
btc.com 100 Satoshis/vbyte | 0.001 BTC/KvB
bitcoinfees.earn.com 10 satoshis/byte
coinb.in 1 Sat/Byte
btc.com's graph shows a spike in mempool size around early midnight 2am on 2018-02-24
now it's already returned back to normal low mempool size but fee rate still 100s/B while bitcoinfees 10s/B
only coinb.in shows 1s/B because it uses reference fee from transaction in block 510839 (5 blocks back)
like I said... weird

Apart from that midnight to 8am period the fee to get into the next block really should be 1 sat/vbyte. The picture from jochen-hoenicke.de the last 2 days is a bit odd, to say the least. 70Mb at 1 sat/vbyte added at a constant rate over 24hr then stops and takes 24hrs to clear.

https://i.snag.gy/h1AmKr.jpg

[vit05]

It could be done on purpose because as far as I can remember, their suggested fees were always high compared to what you actually need to pay to get your transaction confirmed so If it was a technical issue, they would've figured it out already. A better alternative (far more accurate): https://coinb.in/#fees

I'll start using this site and recommend it. Thanks for the sugestion.

I always imagined that bitcoinfees.earn.com was accurate and had never read anything that made me doubt. I hope more people will disclose better alternatives as this type of information is essential to lower fees.

[Quickseller]

Update on this. Currently, the mining pools are not finding enough transactions to fill blocks, the mempool is empty as far as they are concerned.

[...]

But bitcoinfees.earn.com is still showing many transactions in their mempool including a lot at high fees.

[img ]https://i.snag.gy/LWeEO2.jpg[/img]

You are misreading that chart. The chart displays how many transactions have been broadcast in the past two weeks and past 24 hours, not necessarily what is unconfirmed in the mempool of a given node presently. I believe this is done to account for variance in both the number of transactions per second and in block times.

I'm pretty convinced that setting the mempool to keep transactions for 14 days must be inconsistent with the settings used by the mining pools and this is a large part of what is causing them to consistently overestimate fees.

Pools will include the 1MB (weight) of transactions that comply with network rules, and that will result in the highest revenue, either directly through transaction fees, or otherwise. If a transaction has a negative NPV (the transaction fee is lower than the cost of the added risk including the block will be orphaned), then said transaction will not be included in a block, even if there is available space.

It is not possible to know with certainty how many transactions will be broadcast every second, nor is it possible to know when the next block will be found, so various services will create formulas to try to predict appropriate transaction fees to include in order to get your transaction quickly confirmed. bitcoinfees.21.co seems to assume transaction fees are generally steady over time, which is not the case. When transaction fees were skyrocketing in the past, I noticed that bitcoinfees.21.co would often underestimate required transaction fees. This is probably a flaw in the concept of the fee market than anything else.   

[TheQuin]

Update on this. Currently, the mining pools are not finding enough transactions to fill blocks, the mempool is empty as far as they are concerned.

[...]

But bitcoinfees.earn.com is still showing many transactions in their mempool including a lot at high fees.

[img ]https://i.snag.gy/LWeEO2.jpg[/img]

You are misreading that chart. The chart displays how many transactions have been broadcast in the past two weeks and past 24 hours, not necessarily what is unconfirmed in the mempool of a given node presently. I believe this is done to account for variance in both the number of transactions per second and in block times.

That site is somewhat open to interpretation.

https://i.snag.gy/FSAYyn.jpg


The heading "Unconfirmed transactions" would indicate it means what is in their mempool. I've always taken "# OF TRANSACTIONS IN MEMPOOL IN LAST 336 HOURS" to indicate that they have set the "-mempoolexpiry" parameter to 336 rather than 72hrs most nodes use. Watching the site you will see the numbers reducing when a block is found which is consistent with that.

The second heading "Transactions today" and "# OF TRANSACTIONS IN LAST 24 HOURS" doesn't specify but if you watch the numbers it does look like they mean confirmed in the last 24hrs. The numbers certainly increase when a block is found.

If it really meant how many Tx had been broadcast in the last 2 weeks it would always be a much larger number than the number of Tx confirmed in that last 24hrs. That would mean the top bar would be larger than the lower blue bar, which it never is.

. When transaction fees were skyrocketing in the past, I noticed that bitcoinfees.21.co would often underestimate required transaction fees. This is probably a flaw in the concept of the fee market than anything else.   

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

[Quickseller]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

[TheQuin]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

I wasn't suggesting that there was a scandal and I've stayed away from putting a tinfoil hat and suggesting that they do this deliberately. I started the thread because something seemed wrong with the way it is working and I was trying to find out what. Since then as I've watched more I've become convinced if they just set that -mempoolexpiry to 72 they would improve their fee estimation. Until then I'll just recommend people use other fee estimators.

[Quickseller]

Right now they are telling people to pay a $0.67 transaction fee when a $0.20 fee might suffice. Honestly, there is no scandal here. In both cases, transactions are basically free.

This is true, their model is slow to adjust to changes so it lags when fees are heading up as when they are falling. Although my observation is that it is much slower to adjust downwards. That's what made me spend a bit of time trying to work out what is going on with it.

I think this has to do with the fact that fees declined faster than it increased.

I wasn't suggesting that there was a scandal and I've stayed away from putting a tinfoil hat and suggesting that they do this deliberately. I started the thread because something seemed wrong with the way it is working and I was trying to find out what. Since then as I've watched more I've become convinced if they just set that -mempoolexpiry to 72 they would improve their fee estimation. Until then I'll just recommend people use other fee estimators.

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

[TheQuin]

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

That's missing where the problem is. From what I've seen all the mining pools are all using a 72hr expiry so earn.com is factoring in transactions that the miners will never include in blocks. That is what is inflating their estimation.

[Quickseller]

If you assume transaction volume drops off on weekends (and holidays), then changing the formula to only take 72 hours of data into consideration then transaction fee estimates will always be high on the weekend and will be too low on Monday/Tuesday, possibly to the extent using that fee rate will result in a transaction that won’t confirm (until the weekend).

That's missing where the problem is. From what I've seen all the mining pools are all using a 72hr expiry so earn.com is factoring in transactions that the miners will never include in blocks. That is what is inflating their estimation.

If a valid transaction does not get confirmed after 72 hours of being broadcast, it will be because the tx fee was too low. Including transactions that will never confirm (because the tx fee is too low) means the estimate will be lower, not higher.

That argument also ignores the potential for users to rebroadcast their transactions after xx time, which is often sufficient to get a transaction "unstuck".

[TheQuin]

OK, if you don't think it is the -mempoolexpiry can you explain this another way?

I've just taken the following screenshots immediately after a block has been found.

earn.com


jochen-hoenicke.de


The earn.com mempool has 601 transactions queued at 171+ that the mining pools are ignoring. jochen-hoenicke.de mempool has only 4 transactions above 170.

If they factor in 600 high fee transactions that the mining pools are ignoring then obviously that is going to inflate their fee estimate. The -mempoolexpiry is the best explanation I can come up with, can you think of a better one?
 

[RNC]

I tend to watch this site for fees.

https://bitinfocharts.com/comparison/bitcoin-transactionfees.html

Says the price per transaction is about $2 just now which is a number that I can understand without having
eight decimal places and doing sums.

One year ago the price was about $0.10 per transaction as I remember it and many wallets don't offer the
option of setting your own fees so this could be the reason why the sums don't seem to add up.

[TheQuin]

I tend to watch this site for fees.

https://bitinfocharts.com/comparison/bitcoin-transactionfees.html

Says the price per transaction is about $2 just now which is a number that I can understand without having
eight decimal places and doing sums.

One year ago the price was about $0.10 per transaction as I remember it and many wallets don't offer the
option of setting your own fees so this could be the reason why the sums don't seem to add up.

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

[RNC]

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

Sorry but the title of this thread says "bitcoinfees.earn.com systematically overestimating fees" so
maybe my link to a charts that puts fees today at about $2.00 per transaction is "Off-Topic" but only
to people that are spreading FUD about $0.09 fees and you should contact bitinfocharts.com if you
are saying the numbers are wrong.

[TheQuin]

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.

BTW. Can you please stay on-topic. This thread is about the accuracy of earn.com not general whinging about fees.

Sorry but the title of this thread says "bitcoinfees.earn.com systematically overestimating fees" so
maybe my link to a charts that puts fees today at about $2.00 per transaction is "Off-Topic" but only
to people that are spreading FUD about $0.09 fees and you should contact bitinfocharts.com if you
are saying the numbers are wrong.

Read my previous reply "That site is giving a historical chart of what people have paid not an estimation of the fee required."

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

[RNC]

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

Anything that's not promoting your fake image of Bitcoin is "off-topic" and if pointing out that fees are much higher than you
are trying to make out upsets you then please go ahead because my estimator program between my ears puts the number at
about $2.00 just now so stop being a drama queen, I have no time for that.

[Thirdspace]

If you have anything to say about fee estimators I'm all ears. If you continue to try and derail my thread I will report you to the moderators.

Anything that's not promoting your fake image of Bitcoin is "off-topic" and if pointing out that fees are much higher than you
are trying to make out upsets you then please go ahead because my estimator program between my ears puts the number at
about $2.00 just now so stop being a drama queen, I have no time for that.

read the topic! read all the replies in this thread 
all mentioned and discussed about sites that deal with fee estimation, in other word prediction
he, TheQuin, did explicitly say stay on topic talking about "fee estimators" and you insisted on talking about bitinfocharts.com
who is promoting what site?

That site is giving a historical chart of what people have paid not an estimation of the fee required. https://coinb.in/#fees says the current fee required is $0.09.
---snip---

---snip---
... you should contact bitinfocharts.com if you are saying the numbers are wrong.

he never said "the numbers are wrong"  but that site info is irrelevant to the discussion
read the title on that page, it is written in bold
"Bitcoin Avg. Transaction Fee historical chart" and no hints on estimation fee

[TheQuin]

Thankyou @Thirdspace. I would have thought it obvious that the average fee paid has to be by definition higher than the minimum fee required.

Back to bitcoinfees.earn.com and their mysteriously bloated mempool. All these screenshots were taken at the same time just after a block was found with not enough transactions to even fill it.

https://i.snag.gy/lEHQzT.jpg


earn.com showing 3,244 unconfirmed transactions that the mining pools don't see. Resulting in an estimated fee of 30 sats/byte.

https://i.snag.gy/PQGBbz.jpg


jochen-hoenicke.de shows only 67 unconfirmed transactions.

https://i.snag.gy/lvgWM3.jpg


coinb.in/#fees recommends a fee of 1 sat/byte.

[Quickseller]

OK, if you don't think it is the -mempoolexpiry can you explain this another way?

I've just taken the following screenshots immediately after a block has been found.

earn.com


jochen-hoenicke.de


The earn.com mempool has 601 transactions queued at 171+ that the mining pools are ignoring. jochen-hoenicke.de mempool has only 4 transactions above 170.

If they factor in 600 high fee transactions that the mining pools are ignoring then obviously that is going to inflate their fee estimate. The -mempoolexpiry is the best explanation I can come up with, can you think of a better one?
 

Jochen very specifically only includes unconfirmed transactions while earn includes both confirmed and unconfirmed transactions in each of their relevant time periods. Those websites give very different information.

If I broadcast a valid transaction with a transaction fee of 350 sat/byte, my transaction will remain on Jochen until the following block when, as of now, it is almost certain to get confirmed, while it will remain on earn for two weeks.

Also, just because the miners cannot find sufficient transactions to fill up a block in its entirety does not mean there are no remaining paying transactions out there. If a transaction will have a cost to the miner in the risk that including that additional transaction will incrementally increase the risk the block getting orphaned is greater than the transaction fee (both within the transaction, and out of band), then the transaction will not confirm, even if there are no other paying transactions.

[TheQuin]

Jochen very specifically only includes unconfirmed transactions while earn includes both confirmed and unconfirmed transactions in each of their relevant time periods. Those websites give very different information.

The earn.com website also is only showing unconfirmed transactions in the higher (green bars). The discrepancy between this and what Jochen shows and what is getting included in blocks is what I'm pointing out. The legend above could be clearer as to what the two bars are:
https://i.snag.gy/FSAYyn.jpg

"Unconfirmed transactions / Transactions today" makes sense but "# OF TRANSACTIONS IN MEMPOOL IN LAST 336 HOURS" is misleading. If it was really that then the number would HAVE to be much higher than "Transaction Today". They just mean their mempool has a 336hr expiry on unconfirmed transactions.

If I broadcast a valid transaction with a transaction fee of 350 sat/byte, my transaction will remain on Jochen until the following block when, as of now, it is almost certain to get confirmed, while it will remain on earn for two weeks.

No, it will move from the higher bar to the lower bar as it is now confirmed. If you don't believe me then spend some time watching it and see what happens after a block is found.

Also, just because the miners cannot find sufficient transactions to fill up a block in its entirety does not mean there are no remaining paying transactions out there. If a transaction will have a cost to the miner in the risk that including that additional transaction will incrementally increase the risk the block getting orphaned is greater than the transaction fee (both within the transaction, and out of band), then the transaction will not confirm, even if there are no other paying transactions.

That's the heart of the issue here. Every node on the network has its own version of the mempool. In order to make a realistic fee estimation it would be logical to try and have a mempool as similar to the ones the mining pools are using as possible. The fact the earn.com's mempool contains many more transactions than the mining pools mempools is causing them to overestimate fees.

The -mempoolexpiry parameter is the only explanation I can come up with, but I'd like to hear from anyone who has another possible technical explanation for the discrepancy.