Discussion: optimization of mempool fee levels

[s3ton]

I'd like to discuss a possible change to the bitcoin core wallet that does not impact consensus rules.
Right now bitcoin core nodes are still the vast majority and over 60% are recent versions of bitcoin core (>16.3)
Say the mempool is like this:

https://i.ibb.co/M9bq20R/Screenshot-2019-04-03-at-12-59-49.png
What we can see from this visualisation of the mempool is that there are 5.3 MB of transactions that span from 1 sat per byte to 40 satoshi per byte (we will leave extremely expensive transactions (red color) aside for a second).
That means that we could compress all those transactions into just 6 fee levels:
5 MB with fees from 1 to 5 satoshi per byte
0.3 MB with fees of 6 satoshi per byte.
Instead of 40 satoshi per byte!!

Edit:
What I want is to give the option to nodes to set a parameter of maximum allowed overpaying.
For example I would set this parameter to 1 sat/b while you may set it to 5 sat/b and someone else might leave it unlimited like it is now.
Eventually I think that having this option would lead to people using it and force wallets to improve their fee estimation to be sure everyone relays their tx
It is like a cold-war weapon. You don't have to use it, just having it ready does change stuff.
Of course people could just ignore it, but by just implementing it and having it available would be a signal for the network:
- hey be careful that if you overpay you risk to not have an optimal propagation of your transaction.
A block explorer could tag those transactions with a warning saying: "this transaction payed too many fees, this could mean it is not relayed optimally by the network and therefore could not be confirmed quickly"
Please RBF it to a lower fee to make sure it pays a reasonable amount

We can achieve this by having bitcoin full nodes not creating or including in their mempool:
- transactions that pay more than (insert here your selected option, mine is: 1) 1 extra sat/b when compared to the other transactions in the node mempool unless there are already enough tx to fill at least 1block (4000 KWU)

When including transactions in their mempool the nodes would basically start from those paying 1 sat per byte and include those, then move to those paying 2 sat per byte.
After that the nodes would check if there are enough transactions that pay 2 sat/b to fill 1 block (4000KWU). If true then it starts to include the transactions that pay 3 sat/b, if not it stop until there are.
In general I think this would make sure that the mempool grows in a gradual way, without any holes in it like those you can see in the image at 8 sat per bytes where there are very very few transactions.
One possible downside of this is if you have to create an expensive cpfp.

The power of a default setting and the incentive that, if the majority of wallets implement this “best practice”, the prices of the mining fees become rational and optimized should hopefully enough to push people to implement something like this in any wallet.
I know and understand that mining fees need to exist and I think that it is right to develop a fee market. But at the same time we should encourage the users of the wallets to not overpay for their tx to get fast confirmations because, not only do they not gain anything personally by doing that, but they also could create a spiral in the price of the mining fees that can damage Bitcoin.
In some ways it is similar to batching. No one can stop a big exchange if they want to send millions of single  transactions to each of their customers when they buy or sell. Coinbase did that for quite some time.
But eventually it should become a best practice to batch transactions to minimize the footprint on the bitcoin ecosystem.
Exactly the same for this proposal.
No one can stop you to use your own wallet that let you create a tx that pays double the fees of everyone else and get into the next block. But you could’ve been included anyways by paying just 1 more sat/b than the highest fee level. And by doing that you also don’t make it more expensive for everyone else.

[ABCbits]

There are problems with your proposal :
1. Each node's mempool isn't exactly same, so few transaction still could "pass"
2. All users who use wallet which have bad fee estimation or couldn't get mempool (assuming user use SPV wallet) could get their transaction rejected
3. This could be seen as censorship attempt
4. If bitcoin network is attacked by spamming transaction, user can't find way to get their transaction confirmed quickly

[s3ton]

There are problems with your proposal :
1. Each node's mempool isn't exactly same, so few transaction still could "pass"
2. All users who use wallet which have bad fee estimation or couldn't get mempool (assuming user use SPV wallet) could get their transaction rejected
3. This could be seen as censorship attempt
4. If bitcoin network is attacked by spamming transaction, user can't find way to get their transaction confirmed quickly

Hey! 👋
Thanks for your reply, I’m not an expert so please excuse me if I say something stupid.
1. I’m totally ok if some transactions pass even if they overpay. It would be great if this was a default option on the most used wallets so the vast majority of the users would find themselves using it. That alone would optimize things a lot imho.
2.  People using bad spv wallets upgrading to better, safer wallets is something positive imho. Spv wallets should be able to request fee levels from the node they connect to anyways
3. I think some transactions are not relayed by bitcoin nodes if they pay less than 1 sat/b. I don’t think that is censorship.
4. I actually think , but I might be wrong, that with this setting it would be much worse for spammers. Right now all they have to do is wait until there are enough transactions from users and then post their spam to force the users to pay more and more and more in a exponential increase.
With this proposal all they could do is force the users to bump their tx with rbf, but the increase would only be linear: basically 1 sat/b for every spam attack

[squatter]

I'd like to discuss a possible change to the bitcoin core wallet that does not impact consensus rules.

We can achieve this by having bitcoin full nodes not creating or including in their mempool:
- transactions that pay more than 1 extra sat/b when compared to the other transactions in the node mempool unless there are already enough tx to fill at least 1block (4000 KWU)

Users overpaying is a huge problem. People are paying $20 at the toll when a more efficient market would have them paying a quarter for comparable confirmation times.

It's a really hard problem to fix, though. Just like High-priority transaction rules, miners will simply remove this from their clients. Being rational, they will include higher paying fees before lower paying ones since there is no consensus issue. Even if we table the discussion of how refusing to forward transactions that pay higher fees is a form of censorship, miners and others who don't agree -- and there will be many -- will sidestep the issue by removing this code and spinning up globally distributed nodes that do the same.

3. I think some transactions are not relayed by bitcoin nodes if they pay less than 1 sat/b. I don’t think that is censorship.

That's because those transactions aren't going to confirm anyway. It's wasteful to propagate them. Refusing to propagate valid transactions that would otherwise be confirmed is a stickier issue.

[pooya87]

any kind of restrictive changes like this should be avoided as @ETFbitcoin said this can be viewed as censorship. and you will be damaging honest people, a lot of services are forced to pay a moderately higher fee than normal because if they don't and the fee increases in that period where their tx stays in the mempool waiting for confirmation their customers will start social media attacks calling them scammers,... and none of them want this.

besides there are lots of things that should be done first before we start doing something like this.
for starters have you ever considered why are we even incrementing fees 1 satoshi per byte instead of 1 satoshi? for example if i have a tx with 200 byte size i increment my fees like this:
- 1 s/b = 200
- 2 s/b = 400
- 3 s/b = 600
...
and that is not a good design where i should really be increasing it with a smaller value: 200 to 210 to 220,...
and that is only if i have a small one, if i had a big transaction with multi sig (like some exchanges regularly do) and if the size was something like 1500 byte then i would be increasing it 1500 satoshi per step!

[odolvlobo]

...That means that we could compress all those transactions into just 6 fee levels:
5 MB with fees from 1 to 5 satoshi per byte
0.3 MB with fees of 6 satoshi per byte.
Instead of 40 satoshi per byte!!
...

Can you please explain the purpose and benefit of your proposal?

[s3ton]

Users overpaying is a huge problem. People are paying $20 at the toll when a more efficient market would have them paying a quarter for comparable confirmation times.

It's a really hard problem to fix, though. Just like High-priority transaction rules, miners will simply remove this from their clients. Being rational, they will include higher paying fees before lower paying ones since there is no consensus issue. Even if we table the discussion of how refusing to forward transactions that pay higher fees is a form of censorship, miners and others who don't agree -- and there will be many -- will sidestep the issue by removing this code and spinning up globally distributed nodes that do the same.

I totally agree with you that it is hard to solve the problem.
But maybe we could find a way to point the other implementations in the right direction by not allowing users to overpay for transactions.
We all know that miners would hate this kind of implementation because it would cause less income for them in the short term.
But I think in the long therm it would make bitcoin fees much more predictable.
Btw there would be no reason to remove this code because as I see it this would just be a default setting that can be turned off.
Hopefully though, just like segwit, people may realize that if enough people use this, there are benefits like paying less fees and the mempool fee levels increasing in a linear way.

Can you please explain the purpose and benefit of your proposal?

Right now it costs very low amout of money to spam, just a little bit, the mempools so that users naturally start to overpay to get in the next block.
People are forced to pay from 20x to 100x the amount they would have to pay.
spam is an actual economic gain for miners with the current system if done right because even if you burn some money in fees to other mining pools, you also earn much more fees from the blocks you mine, from the honest users that have to pay 50 sat/b instead of 10 sat/b.

More importantly I don't really see any big drawback because this would just be a default setting to help users not overpaying, but not being something mandatory, it should not break anything.
Worst case is that people don't really use this, but I think in the long run this will be something that will be implemented in all wallets.

besides there are lots of things that should be done first before we start doing something like this.
for starters have you ever considered why are we even incrementing fees 1 satoshi per byte instead of 1 satoshi?

The reason we don't pay like this is that transactions that occupy more space on the blockchain should pay more.

[spartacusrex]

Good topic.

Paying a whole satoshi extra _per byte_ seems a little crude..  

Why not go up to 1.1 sat / byte ?

Or 1.01 / byte.  (Then rounding UP to the nearest satoshi )

Is this banned in the client - can it be un-banned ?

The only thing the miners _should_ be doing is ordering the transactions by sat/byte. How much the difference is shouldn't matter - no ? (And I can almost get my head around a minimum 1 sat per byte aswell )

It's _weird_ because it's completely within our powers (the users) to sort this out..


EDIT :
So there are 2 transactions of 200 bytes each. One pays a 200 satoshi fee. The other pays a 201 satoshi fee (rather than 400). Is that not enough to climb the ranks of the mempool ? miner still maximising profit per block.

[s3ton]

EDIT :
So there are 2 transactions of 200 bytes each. One pays a 200 satoshi fee. The other pays a 201 satoshi fee (rather than 400). Is that not enough to climb the ranks of the mempool ? miner still maximising profit per block.

Of course users can just pay 201 instead of 200.
What I propose here is to apply a MAX amount of how much they should be able to pay more if they want to make it into my node's mempool.
Like if there are 2 tx of 200 bytes each and paying 200 sat fee -> you can't pay more than 400 sat for a tx of 200 bytes if you want to make it into my node's mempool.
If instead there are 10k transactions that pay up to 200 sats and other 10k transactions that pay up to 400 sat -> you can start paying up to 600 sat to get into my node's mempool

[spartacusrex]

EDIT :
So there are 2 transactions of 200 bytes each. One pays a 200 satoshi fee. The other pays a 201 satoshi fee (rather than 400). Is that not enough to climb the ranks of the mempool ? miner still maximising profit per block.

Of course users can just pay 201 instead of 200.
What I propose here is to apply a MAX amount of how much they should be able to pay more if they want to make it into my node's mempool.
Like if there are 2 tx of 200 bytes each and paying 200 sat fee -> you can't pay more than 400 sat for a tx of 200 bytes if you want to make it into my node's mempool.
If instead there are 10k transactions that pay up to 200 sats and other 10k transactions that pay up to 400 sat -> you can start paying up to 600 sat to get into my node's mempool

I don't think you can impose a MAX. It just wouldn't be necessary. If the market is functioning correctly people will pay the cheapest they can anyway.

https://bitcoinfees.earn.com/

This page would look the same, as in the ordering of the transactions and timescales, if the scale on the left wasn't in satoshi/byte.. but in tenths of a satoshi/byte.

The problem seems to be the clients simply paying too much.

[s3ton]

The problem seems to be the clients simply paying too much.

That's why you want to incentivize them to double check what they are doing.
If they want to pay too much they should risk to have their transactions being not relayed by all the nodes.
My node simply doesn't understand why another node wants to overpay when they could get into the next block just paying 1 extra sat/b.
Since for my node that overpaying is an attack trying to drive up fees for everyone than it does not include it in its mempool and doesn't relay that tx until it makes sense to pay that much.

[spartacusrex]

The problem seems to be the clients simply paying too much.

That's why you want to incentivize them to double check what they are doing.
If they want to pay too much they should risk to have their transactions being not relayed by all the nodes.
My node simply doesn't understand why another node wants to overpay when they could get into the next block just paying 1 extra sat/b.
Since for my node that overpaying is an attack trying to drive up fees for everyone than it does not include it in its mempool and doesn't relay that tx until it makes sense to pay that much.

Nothing makes more sense than - Exact Same Result but Cheaper. That is where the incentive lies. Not _forcing_ them to be cheaper.

The problem is in the client and fee estimation. That is where the fix lies. No txn blocking required.

[s3ton]

Nothing makes more sense than - Exact Same Result but Cheaper. That is where the incentive lies. Not _forcing_ them to be cheaper.

The problem is in the client and fee estimation. That is where the fix lies. No txn blocking required.

With your idea you don’t really optimize, you only make things cheaper and that may not be good because eventually bitcoin will have to pay for its security with just mining fees.
What I am proposing is not to change the rules to make cheap tx.
What I want is to give the option to nodes to set a parameter of maximum allowed overpaying.
For example I would set this parameter to 1 sat/b while you may set it to 5 sat/b and someone else might leave it unlimited like it is now.
Eventually I think that having this option would lead to people using it and force wallets to improve their fee estimation to be sure everyone relays their tx

[spartacusrex]

ok.

I was unclear as to what you meant by 'optimisation' if not 'make things cheaper'. Kinda still am.

None of what I was proposing before required any changes. It was just fee setting. (although if it is already possible I'm a little confused why it isn't being used.. which since it isn't.. makes me think it's not!

This one does need changes.

This idea has been discussed before - good luck finding it in the depths of BTCtalk - and its very simple.

Everyone pays the same fees in a block, the smallest fee of any of the transactions in the block. A transaction just specifies how much it is prepared to pay to get in. The HOW and mechanism for this to work are not yet specified.

I think this is similar to your idea as it prevents people paying over the current fee price for a transaction, since you all pay the same lowest fee - but doesn't prevent their propagation across the network.


I saw Emin Gur post something on 'twat-ter' about this and claim it as his own.. when I'm positive I've seen it discussed here many years ago..  (Frankly I defy anyone to find anything anywhere that hasn't been chewed over in these forums at some point or other)
 

[s3ton]

Everyone pays the same fees in a block, the smallest fee of any of the transactions in the block. A transaction just specifies how much it is prepared to pay to get in. The HOW and mechanism for this to work are not yet specified.

I think this is similar to your idea as it prevents people paying over the current fee price for a transaction, since you all pay the same lowest fee - but doesn't prevent their propagation across the network.

No, it is actually quite different from my idea.
I think it is necessary to punish wallets that do a bad fee estimation because they damage the user experience and ultimately bitcoin in general.
One of the ways to incentivise them is relaying their transactions only if they don't overpay.
For example if the mempool is basically empty there is zero reason to relay a transaction of 20 sat/b.

Having a parameter that you can define in your bitcoin node that specifies how much clients are allowed to overpay to get their transaction included in the nodes mempool, and so relayed, would help to improve things imho.
it is like cold war weapon.
Of course people could just ignore it, but by just implementing it and having it available would be a signal for the network:
- hey be careful that if you overpay you risk to not have an optimal propagation of your transaction.
A block explorer could tag those transactions with a warning saying: "this transaction payed too many fees, this could mean it is not relayed optimally by the network and therefore could not be confirmed quickly"
Please RBF it to a lower fee to make sure it pays a reasonable amount

[s3ton]

No, it is actually quite different from my idea.
I think it is necessary to punish wallets that do a bad fee estimation because they damage the user experience and ultimately bitcoin in general.
One of the ways to incentivise them is relaying their transactions only if they don't overpay.
For example if the mempool is basically empty there is zero reason to relay a transaction of 20 sat/b.

And you'd punish non-technical user as well who barely know how to use/manage Bitcoin.

Im not even sure that’s the case.
I believe people using light wallets like “blockstream green wallet” or all the other light wallets that get their fee estimation from bitcoin core will be unaffected by this change

[squatter]

I think it is necessary to punish wallets that do a bad fee estimation because they damage the user experience and ultimately bitcoin in general.

Indeed. I've always hoped these wallets would lose users due to higher-than-normal fees -- Copay is one example, and they don't even offer Segwit to boot. But the reality is that they drastically inflate the entire fee market so everybody is forced to similarly overpay to get any reasonable guarantee of confirmation.

I do seriously resent these assholes who immediately jump up 10-20 satoshis/byte just because there's 2,000 transactions in their mempool. This type of deliberate overpayment really screws us all over and it's really rampant. It snowballs when there is any spike in demand for block space. People are cramming $100 bills into a coin slot. It's just a tough philosophical problem because people should be able to compete in the fee market. It just sucks that the tendency of many wallets -- and people who are manually setting fees and rounding up to the nearest 10 sat/byte -- is to drastically overpay to maximize chances of getting in the next block. When everyone is overpaying so much, we all have no choice but to do the same.

You bring up an interesting point, that's for sure. These users are hurting my interests, driving up my costs, and relaying their transactions only reinforces their bad behavior. Miners have financial incentive to reinforce this bad behavior, but what's my incentive as a user and node operator?

[s3ton]

I think it is necessary to punish wallets that do a bad fee estimation because they damage the user experience and ultimately bitcoin in general.

Indeed. I've always hoped these wallets would lose users due to higher-than-normal fees -- Copay is one example, and they don't even offer Segwit to boot. But the reality is that they drastically inflate the entire fee market so everybody is forced to similarly overpay to get any reasonable guarantee of confirmation.

I do seriously resent these assholes who immediately jump up 10-20 satoshis/byte just because there's 2,000 transactions in their mempool. This type of deliberate overpayment really screws us all over and it's really rampant. It snowballs when there is any spike in demand for block space. People are cramming $100 bills into a coin slot. It's just a tough philosophical problem because people should be able to compete in the fee market. It just sucks that the tendency of many wallets -- and people who are manually setting fees and rounding up to the nearest 10 sat/byte -- is to drastically overpay to maximize chances of getting in the next block. When everyone is overpaying so much, we all have no choice but to do the same.

You bring up an interesting point, that's for sure. These users are hurting my interests, driving up my costs, and relaying their transactions only reinforces their bad behavior. Miners have financial incentive to reinforce this bad behavior, but what's my incentive as a user and node operator?

I couldn't have said it better myself.

[gmaxwell]

Other people overpaying is none of your business. Only the fact that they paid more than you has any influence on you at all (they get confirmed ahead of you).

Sometimes people make mistakes, sure, but you shouldn't assume that everyone acting differently from you is making a mistake... or if they are, that its any of your concern.

Nothing I've seen discussed in this thread seems interesting.  If you'd like to make a case that some people are paying higher than you when they'd really rather wait longer than you for confirmation-- that might be interesting. Regardless, trying to fix prices on nodes will just make the network work less well and encourage centeralization when people then realize they need to give their transactions directly to miners.

[squatter]

If you'd like to make a case that some people are paying higher than you when they'd really rather wait longer than you for confirmation-- that might be interesting.

In some cases, users obviously are paying higher than me when they'd rather wait longer.

The Copay wallet (and Bitcoin.com's which is based on it) is very problematic because it's a sleek wallet with easy UI for beginners -- lots of newbies use it. Users have no manual control over fees. If they choose to the lowest fee option -- which means they specifically don't need next-block confirmation and are willing to wait several hours -- they are paying up to 5x-15x what it actually costs to get in the next block. A couple examples:

https://www.reddit.com/r/Bitcoin/comments/7ngbng/how_to_avoid_fee_sending_btc_from_copay_wallet/
https://www.reddit.com/r/Bitcoin/comments/89xx9d/copay_is_the_most_broken_bitcoin_wallet_137/
https://www.reddit.com/r/Bitcoin/comments/6fyxs9/copay_super_economy_confirmed_in_11_minutes_are/

Coming from Bitpay/Bitcoin.com, it feels like a continuing political attack. I think they've been deliberately charging users absurdly high fees, which has had an unfortunate spillover effect on the fee market. This is what newbies see when they choose the lowest fee option (4 hour estimated confirmation time) and are still being overcharged for next block confirmation:



I'm sure there are many users who are happy to pay more than they need to. But I can't imagine all users looking at earn.com's estimator and sending 160 satoshis/byte for the "fastest and cheapest transaction" when there's 1,400 transactions in my mempool down to 100 satoshis/byte want to overpay. They just don't know any better, and unfortunately the Copay wallet and earn.com have been highly recommended for years on social media and this forum.

I agree there's no real way to fix this and the OP's idea won't really work. But I'm willing to acknowledge that it's a shitty problem for the network in spite of my appreciation for the design of the fee market. Users will keep overpaying because of shite fee estimation and ignorance and now that fees are strongly trending up again, they'll start blaming the protocol and complaining more loudly about it like in 2016 and 2017. I sort of wonder if we're gearing up for another political attack.