Bitcoin Forum

I'd like to discuss a possible change to the bitcoin core wallet that does not impact consensus rules.
Right now bitcoin core nodes are still the vast majority and over 60% are recent versions of bitcoin core (>16.3)
Say the mempool is like this:

https://i.ibb.co/M9bq20R/Screenshot-2019-04-03-at-12-59-49.png
What we can see from this visualisation of the mempool is that there are 5.3 MB of transactions that span from 1 sat per byte to 40 satoshi per byte (we will leave extremely expensive transactions (red color) aside for a second).
That means that we could compress all those transactions into just 6 fee levels:
5 MB with fees from 1 to 5 satoshi per byte
0.3 MB with fees of 6 satoshi per byte.
Instead of 40 satoshi per byte!!

Edit:
What I want is to give the option to nodes to set a parameter of maximum allowed overpaying.
For example I would set this parameter to 1 sat/b while you may set it to 5 sat/b and someone else might leave it unlimited like it is now.
Eventually I think that having this option would lead to people using it and force wallets to improve their fee estimation to be sure everyone relays their tx
It is like a cold-war weapon. You don't have to use it, just having it ready does change stuff.
Of course people could just ignore it, but by just implementing it and having it available would be a signal for the network:
- hey be careful that if you overpay you risk to not have an optimal propagation of your transaction.
A block explorer could tag those transactions with a warning saying: "this transaction payed too many fees, this could mean it is not relayed optimally by the network and therefore could not be confirmed quickly"
Please RBF it to a lower fee to make sure it pays a reasonable amount

We can achieve this by having bitcoin full nodes not creating or including in their mempool:
- transactions that pay more than (insert here your selected option, mine is: 1) 1 extra sat/b when compared to the other transactions in the node mempool unless there are already enough tx to fill at least 1block (4000 KWU)

When including transactions in their mempool the nodes would basically start from those paying 1 sat per byte and include those, then move to those paying 2 sat per byte.
After that the nodes would check if there are enough transactions that pay 2 sat/b to fill 1 block (4000KWU). If true then it starts to include the transactions that pay 3 sat/b, if not it stop until there are.
In general I think this would make sure that the mempool grows in a gradual way, without any holes in it like those you can see in the image at 8 sat per bytes where there are very very few transactions.
One possible downside of this is if you have to create an expensive cpfp.

The power of a default setting and the incentive that, if the majority of wallets implement this “best practice”, the prices of the mining fees become rational and optimized should hopefully enough to push people to implement something like this in any wallet.
I know and understand that mining fees need to exist and I think that it is right to develop a fee market. But at the same time we should encourage the users of the wallets to not overpay for their tx to get fast confirmations because, not only do they not gain anything personally by doing that, but they also could create a spiral in the price of the mining fees that can damage Bitcoin.
In some ways it is similar to batching. No one can stop a big exchange if they want to send millions of single  transactions to each of their customers when they buy or sell. Coinbase did that for quite some time.
But eventually it should become a best practice to batch transactions to minimize the footprint on the bitcoin ecosystem.
Exactly the same for this proposal.
No one can stop you to use your own wallet that let you create a tx that pays double the fees of everyone else and get into the next block. But you could’ve been included anyways by paying just 1 more sat/b than the highest fee level. And by doing that you also don’t make it more expensive for everyone else.

There are problems with your proposal :
1. Each node's mempool isn't exactly same, so few transaction still could "pass"
2. All users who use wallet which have bad fee estimation or couldn't get mempool (assuming user use SPV wallet) could get their transaction rejected
3. This could be seen as censorship attempt
4. If bitcoin network is attacked by spamming transaction, user can't find way to get their transaction confirmed quickly

Hey! 👋
Thanks for your reply, I’m not an expert so please excuse me if I say something stupid.
1. I’m totally ok if some transactions pass even if they overpay. It would be great if this was a default option on the most used wallets so the vast majority of the users would find themselves using it. That alone would optimize things a lot imho.
2.  People using bad spv wallets upgrading to better, safer wallets is something positive imho. Spv wallets should be able to request fee levels from the node they connect to anyways
3. I think some transactions are not relayed by bitcoin nodes if they pay less than 1 sat/b. I don’t think that is censorship.
4. I actually think , but I might be wrong, that with this setting it would be much worse for spammers. Right now all they have to do is wait until there are enough transactions from users and then post their spam to force the users to pay more and more and more in a exponential increase.
With this proposal all they could do is force the users to bump their tx with rbf, but the increase would only be linear: basically 1 sat/b for every spam attack

Users overpaying is a huge problem. People are paying $20 at the toll when a more efficient market would have them paying a quarter for comparable confirmation times.

It's a really hard problem to fix, though. Just like High-priority transaction rules, miners will simply remove this from their clients. Being rational, they will include higher paying fees before lower paying ones since there is no consensus issue. Even if we table the discussion of how refusing to forward transactions that pay higher fees is a form of censorship, miners and others who don't agree -- and there will be many -- will sidestep the issue by removing this code and spinning up globally distributed nodes that do the same.


That's because those transactions aren't going to confirm anyway. It's wasteful to propagate them. Refusing to propagate valid transactions that would otherwise be confirmed is a stickier issue.

any kind of restrictive changes like this should be avoided as @ETFbitcoin said this can be viewed as censorship. and you will be damaging honest people, a lot of services are forced to pay a moderately higher fee than normal because if they don't and the fee increases in that period where their tx stays in the mempool waiting for confirmation their customers will start social media attacks calling them scammers,... and none of them want this.

besides there are lots of things that should be done first before we start doing something like this.
for starters have you ever considered why are we even incrementing fees 1 satoshi per byte instead of 1 satoshi? for example if i have a tx with 200 byte size i increment my fees like this:
- 1 s/b = 200
- 2 s/b = 400
- 3 s/b = 600
...
and that is not a good design where i should really be increasing it with a smaller value: 200 to 210 to 220,...
and that is only if i have a small one, if i had a big transaction with multi sig (like some exchanges regularly do) and if the size was something like 1500 byte then i would be increasing it 1500 satoshi per step!

Can you please explain the purpose and benefit of your proposal?

I totally agree with you that it is hard to solve the problem.
But maybe we could find a way to point the other implementations in the right direction by not allowing users to overpay for transactions.
We all know that miners would hate this kind of implementation because it would cause less income for them in the short term.
But I think in the long therm it would make bitcoin fees much more predictable.
Btw there would be no reason to remove this code because as I see it this would just be a default setting that can be turned off.
Hopefully though, just like segwit, people may realize that if enough people use this, there are benefits like paying less fees and the mempool fee levels increasing in a linear way.

Right now it costs very low amout of money to spam, just a little bit, the mempools so that users naturally start to overpay to get in the next block.
People are forced to pay from 20x to 100x the amount they would have to pay.
spam is an actual economic gain for miners with the current system if done right because even if you burn some money in fees to other mining pools, you also earn much more fees from the blocks you mine, from the honest users that have to pay 50 sat/b instead of 10 sat/b.

More importantly I don't really see any big drawback because this would just be a default setting to help users not overpaying, but not being something mandatory, it should not break anything.
Worst case is that people don't really use this, but I think in the long run this will be something that will be implemented in all wallets.

The reason we don't pay like this is that transactions that occupy more space on the blockchain should pay more.

Good topic.

Paying a whole satoshi extra _per byte_ seems a little crude..  :-X

Why not go up to 1.1 sat / byte ?

Or 1.01 / byte.  (Then rounding UP to the nearest satoshi )

Is this banned in the client - can it be un-banned ?

The only thing the miners _should_ be doing is ordering the transactions by sat/byte. How much the difference is shouldn't matter - no ? (And I can almost get my head around a minimum 1 sat per byte aswell )

It's _weird_ because it's completely within our powers (the users) to sort this out..


EDIT :
So there are 2 transactions of 200 bytes each. One pays a 200 satoshi fee. The other pays a 201 satoshi fee (rather than 400). Is that not enough to climb the ranks of the mempool ? miner still maximising profit per block.

Of course users can just pay 201 instead of 200.
What I propose here is to apply a MAX amount of how much they should be able to pay more if they want to make it into my node's mempool.
Like if there are 2 tx of 200 bytes each and paying 200 sat fee -> you can't pay more than 400 sat for a tx of 200 bytes if you want to make it into my node's mempool.
If instead there are 10k transactions that pay up to 200 sats and other 10k transactions that pay up to 400 sat -> you can start paying up to 600 sat to get into my node's mempool

I don't think you can impose a MAX. It just wouldn't be necessary. If the market is functioning correctly people will pay the cheapest they can anyway.

https://bitcoinfees.earn.com/

This page would look the same, as in the ordering of the transactions and timescales, if the scale on the left wasn't in satoshi/byte.. but in tenths of a satoshi/byte.

The problem seems to be the clients simply paying too much.

That's why you want to incentivize them to double check what they are doing.
If they want to pay too much they should risk to have their transactions being not relayed by all the nodes.
My node simply doesn't understand why another node wants to overpay when they could get into the next block just paying 1 extra sat/b.
Since for my node that overpaying is an attack trying to drive up fees for everyone than it does not include it in its mempool and doesn't relay that tx until it makes sense to pay that much.

Nothing makes more sense than - Exact Same Result but Cheaper. That is where the incentive lies. Not _forcing_ them to be cheaper.

The problem is in the client and fee estimation. That is where the fix lies. No txn blocking required.

With your idea you don’t really optimize, you only make things cheaper and that may not be good because eventually bitcoin will have to pay for its security with just mining fees.
What I am proposing is not to change the rules to make cheap tx.
What I want is to give the option to nodes to set a parameter of maximum allowed overpaying.
For example I would set this parameter to 1 sat/b while you may set it to 5 sat/b and someone else might leave it unlimited like it is now.
Eventually I think that having this option would lead to people using it and force wallets to improve their fee estimation to be sure everyone relays their tx

ok.

I was unclear as to what you meant by 'optimisation' if not 'make things cheaper'. Kinda still am.

None of what I was proposing before required any changes. It was just fee setting. (although if it is already possible I'm a little confused why it isn't being used.. which since it isn't.. makes me think it's not! :)

This one does need changes.

This idea has been discussed before - good luck finding it in the depths of BTCtalk - and its very simple.

Everyone pays the same fees in a block, the smallest fee of any of the transactions in the block. A transaction just specifies how much it is prepared to pay to get in. The HOW and mechanism for this to work are not yet specified.

I think this is similar to your idea as it prevents people paying over the current fee price for a transaction, since you all pay the same lowest fee - but doesn't prevent their propagation across the network.


I saw Emin Gur post something on 'twat-ter' about this and claim it as his own.. when I'm positive I've seen it discussed here many years ago..  (Frankly I defy anyone to find anything anywhere that hasn't been chewed over in these forums at some point or other)
 

No, it is actually quite different from my idea.
I think it is necessary to punish wallets that do a bad fee estimation because they damage the user experience and ultimately bitcoin in general.
One of the ways to incentivise them is relaying their transactions only if they don't overpay.
For example if the mempool is basically empty there is zero reason to relay a transaction of 20 sat/b.

Having a parameter that you can define in your bitcoin node that specifies how much clients are allowed to overpay to get their transaction included in the nodes mempool, and so relayed, would help to improve things imho.
it is like cold war weapon.
Of course people could just ignore it, but by just implementing it and having it available would be a signal for the network:
- hey be careful that if you overpay you risk to not have an optimal propagation of your transaction.
A block explorer could tag those transactions with a warning saying: "this transaction payed too many fees, this could mean it is not relayed optimally by the network and therefore could not be confirmed quickly"
Please RBF it to a lower fee to make sure it pays a reasonable amount :)

Im not even sure that’s the case.
I believe people using light wallets like “blockstream green wallet” or all the other light wallets that get their fee estimation from bitcoin core will be unaffected by this change

Indeed. I've always hoped these wallets would lose users due to higher-than-normal fees -- Copay is one example, and they don't even offer Segwit to boot. But the reality is that they drastically inflate the entire fee market so everybody is forced to similarly overpay to get any reasonable guarantee of confirmation.

I do seriously resent these assholes who immediately jump up 10-20 satoshis/byte just because there's 2,000 transactions in their mempool. This type of deliberate overpayment really screws us all over and it's really rampant. It snowballs when there is any spike in demand for block space. People are cramming $100 bills into a coin slot. It's just a tough philosophical problem because people should be able to compete in the fee market. It just sucks that the tendency of many wallets -- and people who are manually setting fees and rounding up to the nearest 10 sat/byte -- is to drastically overpay to maximize chances of getting in the next block. When everyone is overpaying so much, we all have no choice but to do the same.

You bring up an interesting point, that's for sure. These users are hurting my interests, driving up my costs, and relaying their transactions only reinforces their bad behavior. Miners have financial incentive to reinforce this bad behavior, but what's my incentive as a user and node operator?

I couldn't have said it better myself.

Other people overpaying is none of your business. Only the fact that they paid more than you has any influence on you at all (they get confirmed ahead of you).

Sometimes people make mistakes, sure, but you shouldn't assume that everyone acting differently from you is making a mistake... or if they are, that its any of your concern.

Nothing I've seen discussed in this thread seems interesting.  If you'd like to make a case that some people are paying higher than you when they'd really rather wait longer than you for confirmation-- that might be interesting. Regardless, trying to fix prices on nodes will just make the network work less well and encourage centeralization when people then realize they need to give their transactions directly to miners.

In some cases, users obviously are paying higher than me when they'd rather wait longer.

The Copay wallet (and Bitcoin.com's which is based on it) is very problematic because it's a sleek wallet with easy UI for beginners -- lots of newbies use it. Users have no manual control over fees. If they choose to the lowest fee option -- which means they specifically don't need next-block confirmation and are willing to wait several hours -- they are paying up to 5x-15x what it actually costs to get in the next block. A couple examples:

https://www.reddit.com/r/Bitcoin/comments/7ngbng/how_to_avoid_fee_sending_btc_from_copay_wallet/
https://www.reddit.com/r/Bitcoin/comments/89xx9d/copay_is_the_most_broken_bitcoin_wallet_137/
https://www.reddit.com/r/Bitcoin/comments/6fyxs9/copay_super_economy_confirmed_in_11_minutes_are/

Coming from Bitpay/Bitcoin.com, it feels like a continuing political attack. I think they've been deliberately charging users absurdly high fees, which has had an unfortunate spillover effect on the fee market. This is what newbies see when they choose the lowest fee option (4 hour estimated confirmation time) and are still being overcharged for next block confirmation:

https://i.imgur.com/Av0NyRt.png

I'm sure there are many users who are happy to pay more than they need to. But I can't imagine all users looking at earn.com's estimator and sending 160 satoshis/byte for the "fastest and cheapest transaction" when there's 1,400 transactions in my mempool down to 100 satoshis/byte want to overpay. They just don't know any better, and unfortunately the Copay wallet and earn.com have been highly recommended for years on social media and this forum.

I agree there's no real way to fix this and the OP's idea won't really work. But I'm willing to acknowledge that it's a shitty problem for the network in spite of my appreciation for the design of the fee market. Users will keep overpaying because of shite fee estimation and ignorance and now that fees are strongly trending up again, they'll start blaming the protocol and complaining more loudly about it like in 2016 and 2017. I sort of wonder if we're gearing up for another political attack.

Fair point but to counter: At one point copay leaked users private keys (https://bitcoin.stackexchange.com/questions/83559/what-is-the-origin-of-insecure-64-bit-nonces-in-signatures-in-the-bitcoin-chain/), there is little that can be done about sufficiently severe incompetence much less malice except encourage people to not use that software/service.

Agreed there too, but at least the subject of this thread can't really do anything about maliciously bad wallets.   There are 1001 ways to have the same effect-- they could still overpay and mine them themselves (overpayment for profit hurrah), they could simply make the wallets also spend additional long unconfirmed chain inputs to slow them down etc.

I do agree that things like these adversarial parties masquerading as supporting bitcoin is a problem, but trying to do somethin about overpayment only addresses a symptom at best, it if could work.


+1 Agreed here, sorry for perhaps overstating my case.

Is there any issue in bitcoin core fee estimation algorithm right now?
Looking at the mempool:
https://i.ibb.co/vsFLTC8/Screenshot-2019-04-12-at-11-50-34.png
While the mempool looks empty, I get +110 sat/b fee estimation from bitcoin core to get into the next block and +90 to get into the next 6 blocks.
Am I the only one seeing these numbers?

https://gist.github.com/morcos/d3637f015bc4e607e1fd10d8351e9f41

Bitcoin Core Fee Estimation Algorithm.

---------

Hmm.. I've skimmed it twice now..

If you don't like the way core estimates your fee, then estimate it yourself ... which you can do.
(... personally I write my own transactions and choose my own fees I want)

As for mining transactions, miners will choose higher fees per byte over lower fees per byte.
So these so-called 'overpaid' transactions will be confirmed first.

Implementing rules in bitcoin to enforce some group's ideals is not going to happen.
The free and uncontrolled market is one of the biggest reasons why people want bitcoin.
Central control (which is what you are effectively suggesting) goes against that ideal and wont be tolerated.
Miners will just ignore it also.

If you are complaining about what happened last week with fees sky-rocketing, then discuss that with Core who implemented Segwit which was "SUPPOSED" to solve such problems ... but failed to as usual.

indeed



::) didn't take you long to abandon "the free and uncontrolled market", lol


Segwit's working as the compromise it's supposed to be. Only 2% of addresses are even using segwit addresses, because many people just aren't aware, and also since the industry as a whole has only really now begun to support it (probably a good idea, taking risks with new tech in the real world could be an expensive mistake).

So until more users and businesses switch, 1.2MB average blocks is as big as they'll get. If you want bigger blocks and cheaper fees, the incentive is there, use it. Or don't. It's a f(r)ee market

Of course I do set the fee manually for my transactions.

But if there is a bad algorithm for fee estimation and everyone uses it, for the network, it doesn't really matter what one or 2 users do.

The algorithm used by bitcoin core is likely used by many big players that are not going to set the estimate manually, they have to rely on an algorithm and they trust bitcoin core.

To be more clear:
Right now I'm not advocating for any central planning or stuff like that.
I'm just trying to understand why the vast majority of the network does fee estimation poorly.
It's something everyone can see by looking at the last few days and hours.

I wonder how many years you keep saying this :)

One could re-word it as:
"People don't use SegWit ... so it's not SegWit's fault"

Then reply with
"People don't do many things, if your solution is worthwhile, then get them to use it ... but if you continue this excuse for years (it has been 'years') then the 'solution' is clearly NOT a solution if it still fails to solve the problem"

Though I could throw into this the fact that BIP100 would have been an immediate solution and would be working today, and had/has a longer expected solution timeframe than SegWit.
P.S. BIP100 had a 70% coinbase vote at it's peak without anyone knowing if it would get implemented - but alas, no one paid core the money to implement it, core implements 'major' changes that they are paid to implement.

??? e.g. the entirety of the year 2018,  during which mempools were half empty?



which problem? ;)

blocks are too big? or blocks are too small? By choosing to use P2PKH addresses, blocks stay smaller. If you use segwit addresses, blocks can be made bigger.

Compromise: achieved

Oh, you forgot did you ... memory loss - advanced stage?
segwit was implemented in 2017 ... BIP91 ...

and in Nov 2017 to Jan 2018 it failed to manage the mempool such that txn fees were VERY high
Nov block reward Avg: 118.27%
Dec Avg: 129.11%
Jan Avg: 132.04%

Then total txns fell - some of the blame resides with segwit for that.

Lulz - we are already talking about yet another recent failure of segwit here ...
I guess you consider it's multiple failures an achievement ... most people consider that simply a failure.

Giving excuses for it's continued failure still makes it a failure.

that's unreasonable

From segwit activation (September '17 IIRC) to Jan 2018 was just a few months. Segwit couldn't possibly be used to react to big hike ups in tx backlogs straight after it activated, that wasn't quite enough time for the whole of Bitcoin to move to segwit addresses, and then start spending from them :D. Good thing too, blocks stayed effectively capped at ~ 1.1MB until Jan '18, compromise: achieved :)

seems you forgot the reasoning why BIP100 got rejected too, but whatever



so....

• mempool full: because segwit
• mempool empty: because segwit

uh, ok

Well any block size change based on any bitcoin transactions (not just some new transactions that you want to force everyone to use) would have worked.

Yeah as I said, no one paid for it.

Total transaction counts fell - seriously you know nothing about the number of transactions in bitcoin over time?
Ignorance is bliss I guess. Helps you make up false arguments.

Aside: the reason that the total transactions fell was due to the price bubble burst, and segwit's design to ignore most transactions.

Segwit's not doing a good job of "ignoring most transactions", because most transactions are not segwit. That was how this little debate started. you saying segwit failed due to low uptake.


You're a big fan of non-sensical arguments:

• segwit ignores most transactions
• most transactions ignore segwit

both those things are true, depending on which version of "everything is segwit's fault" you're trying to argue

this topics idea is no better than current state of affairs.

firstly if using the scenario of say 20mb(WU)(5mblegacy) mempool size where
5sat is priority(0-4mbWU(0-1mb leg)) next block,
4sat is less priority(4-8mbWU(1-2mb leg)) second block,
3sat is less priority(8-12mbWU(2-3mb leg)) third block,

a spammer pays 5sat, gets confirmed then instantly throws that utxo into a new tx of 5 sat and is instantly in the priority nxt block listing.
meaning those that paid 4sat are not guaranteed to be in the second block as the spammer is in the second block too

this, like current status just leads to games of everyone just paying ever increasing amounts, until people give up and the price then eventually resettles until people realise it settled down to then play another round of ever increasing amounts

what needs to be done is bring back a fee formulae that is a consensus rule (have to abide by) which actually has options.

EG if a tx is only 1confirm deep its 'score' is lower than someone with 144 confirms. so the spammer(1confirm) has to pay more to raise its score to be given priority.
this way it actually makes spammers pay the price, where as occassional spenders are not affected by th price. and by using the score based system its not a everyone is hit by a premium due to a few users.

this way it actually make spammers who multispend daily having to pay more, then become more incentivised to spam less or use offchain solutions. while the occassional non abusive user is not paying out alot but getting their transactions noticed by miningpools collating transactions

to explain better

I like pooya87's idea of an "increased granularity" (being able to set fractions of a satoshi per byte). However, Core afaik already is using BTC/kB with a granularity up to a single satoshi (and not sat/B) as the default unit. And afaik you also can always select an absolute fee amount.

But lots of wallets and websites continue to use the "sat/B" unit. The reason is probably that this is the unit "most easily digested", with numbers currently of 1-10 in low-fee periods, 10-200 in mild congested times and 200-1000 in highly congested times. If in the last block the average fee of confirmed transactions was, say, 5 sat/Byte, it's reasonable to pay 6 sat/Byte if you want the tx to be confirmed soon, and not 5001 sat/kB.

So increased "fee granularity" in wallets with sat/B as the default option would certainly be desirable, but the impact would be probably not too high. But as there are no risks associated, I certainly would support this kind of measure.

The OP's proposal, however, would probably not work. It over-estimates the "tx relaying power" of non-mining nodes. It gives incentives, instead, to simply connect directly to nodes that you know that they're mining. This will probably become the default behavior if such a mechanism was implemented, as gmaxwell writes.

@franky1: A "priority-based" tx inclusion by miners like in early bitcoind versions would be certainly interesting, but I struggle to understand how this could really be enforced. Maybe like the "weight" calculation in Segwit transactions? However, I've read some time ago that some miners are sometimes selectively preferring to include non-Segwit transactions (https://bitcoinist.com/bitmains-policy-questioned-as-antpool-stops-mining-segwit-blocks/) over Segwit-type ones. If this is possible (and not punishable by the protocol), then it's probably not reliabily possible to really enforce priority-based tx confirmation. (And I'm sure the reason why the priority mechanism was abolished in newer Core versions has to do with that problem.)

If it's possible to enforce, priority-based calculation would certainly lead to lower fees, because it would incentive people which aren't in a hurry to confirm to pay only a very low fee.

Decimal Places.

People start at the lowest and work up. Normal non-spam users would linger in the lowest levels. Why not.

What would happen if - just hypothetically - Bitcoin had 16 decimal places and not 8 ?

1 satoshini was 1*10^-16 BTC.. transaction fee priced in satoshini / byte.

I think fees would come down, but not by as much as 8 decimal places.

Exactly, I think psychology and our limited brain resources would prevent us to use an "optimal" fee strategy (which would be, in this case, to always chose a fee exactly 1 satoshini per byte higher than what we think our competitors would chose). But it could have a small effect.

Apart from that, 8 more decimal places would enlarge transactions - not by that much, but significantly. Simply chosing "sat/kB" instead of "sat/Byte" seems a better strategy.