Discussion: optimization of mempool fee levels

[gmaxwell]

The Copay wallet (and Bitcoin.com's which is based on it) is very problematic because it's a sleek wallet with easy UI for beginners -- lots of newbies use it. Users have no manual control over fees. If they choose to the lowest fee option -- which means they specifically don't need next-block confirmation and are willing to wait several hours -- they are paying up to 5x-15x what it actually costs to get in the next block.

Fair point but to counter: At one point copay leaked users private keys, there is little that can be done about sufficiently severe incompetence much less malice except encourage people to not use that software/service.

Coming from Bitpay/Bitcoin.com, it feels like a continuing political attack. I think they've been deliberately charging users absurdly high fees, which has had an unfortunate spillover effect on the fee market.

Agreed there too, but at least the subject of this thread can't really do anything about maliciously bad wallets.   There are 1001 ways to have the same effect-- they could still overpay and mine them themselves (overpayment for profit hurrah), they could simply make the wallets also spend additional long unconfirmed chain inputs to slow them down etc.

I do agree that things like these adversarial parties masquerading as supporting bitcoin is a problem, but trying to do somethin about overpayment only addresses a symptom at best, it if could work.

I agree there's no real way to fix this and the OP's idea won't really work. But I'm willing to acknowledge that it's a shitty problem for the network in spite of my appreciation for the design of the fee market. Users will keep overpaying because of shite fee estimation and ignorance and now that fees are strongly trending up again, they'll start blaming the protocol and complaining more loudly about it like in 2016 and 2017. I sort of wonder if we're gearing up for another political attack.

+1 Agreed here, sorry for perhaps overstating my case.

[1713980655]

1713980655

[s3ton]

Is there any issue in bitcoin core fee estimation algorithm right now?
Looking at the mempool:
https://i.ibb.co/vsFLTC8/Screenshot-2019-04-12-at-11-50-34.png
While the mempool looks empty, I get +110 sat/b fee estimation from bitcoin core to get into the next block and +90 to get into the next 6 blocks.
Am I the only one seeing these numbers?

[spartacusrex]

https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41

Bitcoin Core Fee Estimation Algorithm.

---------

Hmm.. I've skimmed it twice now..

[kano]

If you don't like the way core estimates your fee, then estimate it yourself ... which you can do.
(... personally I write my own transactions and choose my own fees I want)

As for mining transactions, miners will choose higher fees per byte over lower fees per byte.
So these so-called 'overpaid' transactions will be confirmed first.

Implementing rules in bitcoin to enforce some group's ideals is not going to happen.
The free and uncontrolled market is one of the biggest reasons why people want bitcoin.
Central control (which is what you are effectively suggesting) goes against that ideal and wont be tolerated.
Miners will just ignore it also.

If you are complaining about what happened last week with fees sky-rocketing, then discuss that with Core who implemented Segwit which was "SUPPOSED" to solve such problems ... but failed to as usual.

[Carlton Banks]

The free and uncontrolled market is one of the biggest reasons why people want bitcoin.

indeed

If you are complaining about what happened last week with fees sky-rocketing, then discuss that with Core who implemented Segwit which was "SUPPOSED" to solve such problems ... but failed to as usual.

didn't take you long to abandon "the free and uncontrolled market", lol


Segwit's working as the compromise it's supposed to be. Only 2% of addresses are even using segwit addresses, because many people just aren't aware, and also since the industry as a whole has only really now begun to support it (probably a good idea, taking risks with new tech in the real world could be an expensive mistake).

So until more users and businesses switch, 1.2MB average blocks is as big as they'll get. If you want bigger blocks and cheaper fees, the incentive is there, use it. Or don't. It's a f(r)ee market

[s3ton]

If you don't like the way core estimates your fee, then estimate it yourself ... which you can do.

Of course I do set the fee manually for my transactions.

But if there is a bad algorithm for fee estimation and everyone uses it, for the network, it doesn't really matter what one or 2 users do.

The algorithm used by bitcoin core is likely used by many big players that are not going to set the estimate manually, they have to rely on an algorithm and they trust bitcoin core.

To be more clear:
Right now I'm not advocating for any central planning or stuff like that.
I'm just trying to understand why the vast majority of the network does fee estimation poorly.
It's something everyone can see by looking at the last few days and hours.

[kano]

...

If you are complaining about what happened last week with fees sky-rocketing, then discuss that with Core who implemented Segwit which was "SUPPOSED" to solve such problems ... but failed to as usual.

didn't take you long to abandon "the free and uncontrolled market", lol


Segwit's working as the compromise it's supposed to be. Only 2% of addresses are even using segwit addresses, because many people just aren't aware, and also since the industry as a whole has only really now begun to support it (probably a good idea, taking risks with new tech in the real world could be an expensive mistake).

So until more users and businesses switch, 1.2MB average blocks is as big as they'll get. If you want bigger blocks and cheaper fees, the incentive is there, use it. Or don't. It's a f(r)ee market

I wonder how many years you keep saying this

One could re-word it as:
"People don't use SegWit ... so it's not SegWit's fault"

Then reply with
"People don't do many things, if your solution is worthwhile, then get them to use it ... but if you continue this excuse for years (it has been 'years') then the 'solution' is clearly NOT a solution if it still fails to solve the problem"

Though I could throw into this the fact that BIP100 would have been an immediate solution and would be working today, and had/has a longer expected solution timeframe than SegWit.
P.S. BIP100 had a 70% coinbase vote at it's peak without anyone knowing if it would get implemented - but alas, no one paid core the money to implement it, core implements 'major' changes that they are paid to implement.

[Carlton Banks]

if you continue this excuse for years (it has been 'years')

e.g. the entirety of the year 2018,  during which mempools were half empty?

the 'solution' is clearly NOT a solution if it still fails to solve the problem"

which problem?

blocks are too big? or blocks are too small? By choosing to use P2PKH addresses, blocks stay smaller. If you use segwit addresses, blocks can be made bigger.

Compromise: achieved

[kano]

if you continue this excuse for years (it has been 'years')

e.g. the entirety of the year 2018,  during which mempools were half empty?

Oh, you forgot did you ... memory loss - advanced stage?
segwit was implemented in 2017 ... BIP91 ...

and in Nov 2017 to Jan 2018 it failed to manage the mempool such that txn fees were VERY high
Nov block reward Avg: 118.27%
Dec Avg: 129.11%
Jan Avg: 132.04%

Then total txns fell - some of the blame resides with segwit for that.

the 'solution' is clearly NOT a solution if it still fails to solve the problem"

which problem?

blocks are too big? or blocks are too small? By choosing to use P2PKH addresses, blocks stay smaller. If you use segwit addresses, blocks can be made bigger.

Compromise: achieved

Lulz - we are already talking about yet another recent failure of segwit here ...
I guess you consider it's multiple failures an achievement ... most people consider that simply a failure.

Giving excuses for it's continued failure still makes it a failure.

[Carlton Banks]

segwit was implemented in 2017 ... BIP91 ...

and in Nov 2017 to Jan 2018 it failed to manage the mempool such that txn fees were VERY high

that's unreasonable

From segwit activation (September '17 IIRC) to Jan 2018 was just a few months. Segwit couldn't possibly be used to react to big hike ups in tx backlogs straight after it activated, that wasn't quite enough time for the whole of Bitcoin to move to segwit addresses, and then start spending from them . Good thing too, blocks stayed effectively capped at ~ 1.1MB until Jan '18, compromise: achieved

seems you forgot the reasoning why BIP100 got rejected too, but whatever

Nov block reward Avg: 118.27%
Dec Avg: 129.11%
Jan Avg: 132.04%

Then total txns fell - some of the blame resides with segwit for that.

so....

• mempool full: because segwit
• mempool empty: because segwit

uh, ok

[kano]

segwit was implemented in 2017 ... BIP91 ...

and in Nov 2017 to Jan 2018 it failed to manage the mempool such that txn fees were VERY high

that's unreasonable

From segwit activation (September '17 IIRC) to Jan 2018 was just a few months. Segwit couldn't possibly be used to react to big hike ups in tx backlogs straight after it activated

Well any block size change based on any bitcoin transactions (not just some new transactions that you want to force everyone to use) would have worked.

seems you forgot the reasoning why BIP100 got rejected too, but whatever

Yeah as I said, no one paid for it.

Nov block reward Avg: 118.27%
Dec Avg: 129.11%
Jan Avg: 132.04%

Then total txns fell - some of the blame resides with segwit for that.

so....

• mempool full: because segwit
• mempool empty: because segwit

uh, ok

Total transaction counts fell - seriously you know nothing about the number of transactions in bitcoin over time?
Ignorance is bliss I guess. Helps you make up false arguments.

Aside: the reason that the total transactions fell was due to the price bubble burst, and segwit's design to ignore most transactions.

[Carlton Banks]

Aside: the reason that the total transactions fell was due to the price bubble burst, and segwit's design to ignore most transactions.

Segwit's not doing a good job of "ignoring most transactions", because most transactions are not segwit. That was how this little debate started. you saying segwit failed due to low uptake.


You're a big fan of non-sensical arguments:

• segwit ignores most transactions
• most transactions ignore segwit

both those things are true, depending on which version of "everything is segwit's fault" you're trying to argue

[franky1]

this topics idea is no better than current state of affairs.

firstly if using the scenario of say 20mb(WU)(5mblegacy) mempool size where
5sat is priority(0-4mbWU(0-1mb leg)) next block,
4sat is less priority(4-8mbWU(1-2mb leg)) second block,
3sat is less priority(8-12mbWU(2-3mb leg)) third block,

a spammer pays 5sat, gets confirmed then instantly throws that utxo into a new tx of 5 sat and is instantly in the priority nxt block listing.
meaning those that paid 4sat are not guaranteed to be in the second block as the spammer is in the second block too

this, like current status just leads to games of everyone just paying ever increasing amounts, until people give up and the price then eventually resettles until people realise it settled down to then play another round of ever increasing amounts

what needs to be done is bring back a fee formulae that is a consensus rule (have to abide by) which actually has options.

EG if a tx is only 1confirm deep its 'score' is lower than someone with 144 confirms. so the spammer(1confirm) has to pay more to raise its score to be given priority.
this way it actually makes spammers pay the price, where as occassional spenders are not affected by th price. and by using the score based system its not a everyone is hit by a premium due to a few users.

this way it actually make spammers who multispend daily having to pay more, then become more incentivised to spam less or use offchain solutions. while the occassional non abusive user is not paying out alot but getting their transactions noticed by miningpools collating transactions

to explain better

this is what i see as the logical punishment for bloating/rspending spammers. whilst rewarding moral normal transactors

one which includes a CLTV voluntary option. where users gain priority points if they voluntarily agree to put their funds into a 1-day maturity. but those avoiding the one day before respend or have bloated transactions pay more to get into a block sooner.

EG
if you really need priority you agree that once confirmed you cant respend for a day.
it also means you can be selective of priority. by only putting a 142block wait if your happy to wait a couple blocks because it wont be priority for a couple blocks by not paying quite enough fee. allowing the age/maturity/fee variables to give a better flag of desire.

obviously those moral users that actually need to spend more than once a day could see the niche of LN as a way to transact often and cheaper.
and those that dont spend every day get priority and not need LN or to CLTV mature funds, because they are not spending everyday, anyway.

here is one example of a formulae that does not care about how much people are spending (not a rich gets priority, poor are victimised old formula), but rewards people willing to wait a day, have lean transactions. and penalises those that want to respend often or have bloated transactions



basically
if your transaction is 2x a lean tx. you pay twice as much. if 44x a lean tx you pay 44x
if you dont want to mature your funds for 144 blocks and only want to wait 1 block you pay 144x.
if the tx is both 44x bloated and wants to respend the very next block after getting confirmed then it costs 44x*144X


though my formulae is not finalised or perfect for every utility. i see how changing the priority formulae can cause more benefits for good people and penalise the bad, without making it used just to be snobby about rich vs poor. due to it no longer rewarding the rich with points just for being rich, which the old formulae done

[d5000]

I like pooya87's idea of an "increased granularity" (being able to set fractions of a satoshi per byte). However, Core afaik already is using BTC/kB with a granularity up to a single satoshi (and not sat/B) as the default unit. And afaik you also can always select an absolute fee amount.

But lots of wallets and websites continue to use the "sat/B" unit. The reason is probably that this is the unit "most easily digested", with numbers currently of 1-10 in low-fee periods, 10-200 in mild congested times and 200-1000 in highly congested times. If in the last block the average fee of confirmed transactions was, say, 5 sat/Byte, it's reasonable to pay 6 sat/Byte if you want the tx to be confirmed soon, and not 5001 sat/kB.

So increased "fee granularity" in wallets with sat/B as the default option would certainly be desirable, but the impact would be probably not too high. But as there are no risks associated, I certainly would support this kind of measure.

The OP's proposal, however, would probably not work. It over-estimates the "tx relaying power" of non-mining nodes. It gives incentives, instead, to simply connect directly to nodes that you know that they're mining. This will probably become the default behavior if such a mechanism was implemented, as gmaxwell writes.

@franky1: A "priority-based" tx inclusion by miners like in early bitcoind versions would be certainly interesting, but I struggle to understand how this could really be enforced. Maybe like the "weight" calculation in Segwit transactions? However, I've read some time ago that some miners are sometimes selectively preferring to include non-Segwit transactions over Segwit-type ones. If this is possible (and not punishable by the protocol), then it's probably not reliabily possible to really enforce priority-based tx confirmation. (And I'm sure the reason why the priority mechanism was abolished in newer Core versions has to do with that problem.)

If it's possible to enforce, priority-based calculation would certainly lead to lower fees, because it would incentive people which aren't in a hurry to confirm to pay only a very low fee.

[spartacusrex]

Decimal Places.

People start at the lowest and work up. Normal non-spam users would linger in the lowest levels. Why not.

What would happen if - just hypothetically - Bitcoin had 16 decimal places and not 8 ?

1 satoshini was 1*10^-16 BTC.. transaction fee priced in satoshini / byte.

I think fees would come down, but not by as much as 8 decimal places.

[d5000]

What would happen if - just hypothetically - Bitcoin had 16 decimal places and not 8 ?

1 satoshini was 1*10^-16 BTC.. transaction fee priced in satoshini / byte.

I think fees would come down, but not by as much as 8 decimal places.

Exactly, I think psychology and our limited brain resources would prevent us to use an "optimal" fee strategy (which would be, in this case, to always chose a fee exactly 1 satoshini per byte higher than what we think our competitors would chose). But it could have a small effect.

Apart from that, 8 more decimal places would enlarge transactions - not by that much, but significantly. Simply chosing "sat/kB" instead of "sat/Byte" seems a better strategy.