Possible “reasons” for the transaction fee and possible price implications

[jonny1000]

The transaction fee in Bitcoin is an area that has always confused me and I am unable to picture all the complexities behind the fee at the same time.  I also fail to understand how the price of the fee could be determined or if a market based fee could work overall.  Please see below a short list of all the possible “reasons” for transaction fees that I can think of, which I have tried to analyse independently of each other.  Please let me know if there are any I missed or some of these should not be here.  I think we need to work out how these “reasons” can be “combined” to determine the price of the transaction fee and if a market based solution could work overall.  Do some of the “reasons” below make others redundant?

1.   To allocate the limited supply of space on the blockchain
The blockchain is a valuable public resource, which many Bitcoin participants need to store and share.  Due to the limitations of storage space, bandwidth and block propagation time which could increase mining centralisation; the system needs to have incentives to limit increases in the level of data and therefore the amount of transactions on the blockchain.  Therefore an arbitrary block size limit is imposed in Bitcoin's core implementation, which ensures there is a finite supply of space for transactions in each block.  A transaction fee is required to address this problem.  If demand for transactions exceeds the finite supply, the transaction fee should increase to ensure supply and demand meet and the market clears.  Miners then include the transactions with the highest fees.
Market based mechanism appears viable: Yes
Price implications: Potentially high transaction fees, possibly dependent on Moore’s law

2.   To incentivise miners to continue to mine after the block reward becomes too low
Currently miners are incentivised primarily by the block rewards, however the block reward will half approximately every four years.  A transaction fee is therefore necessary to ensure that miners are incentivised to continue to mine when the block reward becomes too small.  Without the transaction fee, at some point in the future, miners would begin to leave the system and core consensus mechanism behind Bitcoin would become insecure.  The aggregate fees need to be high enough to ensure miners remain incentivised.
Market based mechanism appears viable: Unclear, since a tragedy of the commons situation appears to occur
Price implications: Unclear impact, dependant on transaction volume

3.   To compensate for the marginal cost to miners of including the transaction in a block
In a competitive market, price should equal marginal cost.  There is a cost to miners of including transactions in blocks: 1) a small amount of computing power is required to put a hash of the transaction data into the block header, 2) the transaction increases the size of the block and this will increase the block propagation time, which could increase the orphan rate.  A transaction fee is therefore required to compensate miners for this cost.  Since these costs to miners are very low, the transaction fee should also be very low.
Market based mechanism appears viable: Yes
Price implications: Potentially very low transaction fees

4.   The economic reality of how miners will prioritise double spending zero conformation transactions
Currently by default, if a Bitcoin miner sees a double spend attempt with zero conformations, the miner should attempt to put the first transaction it saw into a block.  However the assumption behind this fails to appreciate that miners, as rationale actors, will ultimately act in their own self interest.  Therefore miners will eventually prioritise double spend transactions based on the one with a higher fee.  Transaction fees are therefore necessary to facilitate this.  Transaction fees could be a tool used in low value transactions; for buyers to be able to cancel zero conformation transactions, or a tool for sellers who could insist on a particular transaction fee for a zero conformation transaction.  However if a buyer has no intention of double spending, or intends to wait for conformations, the fee should be zero.
Market based mechanism appears viable: Unclear
Price implications: Potentially moderate or zero transaction fees

[1713462672]

1713462672

[1713462672]

1713462672

[1713462672]

1713462672

[1713462672]

1713462672

[BurtW]

Great post.

I have been wondering about this for a long time.  My basic question is this:  In order to create a market based approach to fees wouldn't we need market information?

In other words don't we need all the pools to publish their current rates so we can determine what fee to use?

I can see a system where the pools all publish their rates and these rates are all collected by the wallet so the wallet can determine the fee needed at the time of the transaction.

[jonny1000]

Thanks BurtW.

I think the market information would be historic transaction fees in old blocks and then a weighted median could be used somehow.  How this will work is unclear, but this could have characteristics of a "perfect market"; in the sense there are many buyers, many sellers and "perfect information".  However if the "sellers" own anything to begin with is unclear.

I think we need a better understanding of a theory behind the price before we can approach this problem.  For example if the scarcity of space in the block is the issue, does the formula consider the remaining space in the historic blocks for further transactions?  Should only blocks which were found in a period of 10 minutes or greater be used in the price formula, since perhaps blocks found in a shorter time period are less representative of the scarcity problem?

I think the arbitrary scarcity of space in blocks may be a necessary tool which can provide some type of market for transaction fees.  This market could then result in fees which would provide incentives for miners to continue to mine in the long run.

Many thanks

[piotr_n]

I can see a system where the pools all publish their rates and these rates are all collected by the wallet so the wallet can determine the fee needed at the time of the transaction.

I made a similar suggestion some time ago
https://bitcointalk.org/index.php?topic=637874.0

The pools publishing the rates is the best way to go IMHO.

At the other side, there are people claiming that they will be able to figure out the current rates just by looking inside the past block.
Personally, I cannot wait to see their system at work, though I don't seriously expect to be enjoying it within the next couple of years.

[theymos]

My basic question is this:  In order to create a market based approach to fees wouldn't we need market information?

In other words don't we need all the pools to publish their current rates so we can determine what fee to use?

I can see a system where the pools all publish their rates and these rates are all collected by the wallet so the wallet can determine the fee needed at the time of the transaction.

All of the pools would lie about their fees. They have little reason to reject transactions with any non-dust fee, but every reason to tell people to pay more. So advertising fees like that would be useless.

Looking at the fee statistics in the last few days' of blocks is probably also not a good idea because pools can send high-fee transactions to themselves to skew statistics. These transactions would be private until accepted into the block, so there'd be very little risk of the fees being claimed by other pools. I predict that no matter how the client attempts to avoid this sort of manipulation, pools will figure out how to manipulate calculated fees anyway.

I think that all we need is for people to be able to cancel 0-conf transactions. People can then experiment and see what fees they can get away with without worrying about transactions getting stuck. The client might even do this automatically, starting transactions at some fee level based on user-specified transaction priority and past data about confirmation times, and increasing the fee if the transaction takes too long.

2.   To incentivise miners to continue to mine after the block reward becomes too low
Currently miners are incentivised primarily by the block rewards, however the block reward will half approximately every four years.  A transaction fee is therefore necessary to ensure that miners are incentivised to continue to mine when the block reward becomes too small.  Without the transaction fee, at some point in the future, miners would begin to leave the system and core consensus mechanism behind Bitcoin would become insecure.  The aggregate fees need to be high enough to ensure miners remain incentivised.
Market based mechanism appears viable: Unclear, since a tragedy of the commons situation appears to occur
Price implications: Unclear impact, dependant on transaction volume

Assurance contracts would probably be a good solution if there is ever insufficient mining power.

[jonny1000]

Dear Theymos

I think your point that “pools can send high-fee transactions to themselves to skew statistics” is interesting and a risk; however I think there could be a degree of natural resilience against this type of attack; depending on what the transaction fee “formula” is trying to achieve.  Let me try to explain below:

In my mind, the transaction fee formula could be trying to achieve a price level of something like; the higher of:
a.   The marginal cost to the miner of including the transaction in a block
b.   The minimum bid required to outbid other transactions trying to get into the limited block space

In respect of b, the limit on block size is the supply and the numbers of transactions are the demand.  The block size limit is the same for every block; however the demand varies depending on how quickly the miners find a block of the required difficulty.

For example:  If a block is found in 2 minutes then there are only 2 minutes worth of new transactions, however the block size limit is designed for the target 10 minute time and remains at the same level.  

If the block is found in less than 10 minutes, the block may not be representative of the level of demand for space in the block and therefore the transaction fee formula should exclude this block from the calculation.

Now let’s consider the implication of this for a miner considering the strategy of sending “high-fee transactions to themselves to skew statistics”.  In order for this to be relevant the miner needs two conditions to hold:
1.   The miner must be the first to find a valid block, and
2.   The miner must find the block in longer than 10 minutes
(If the miner finds the block in less than 10 minutes, the miner is likely to want to release the block immediately otherwise the miner could lose a propagation race).

From each miners perspective the probability of these two conditions holding is unlikely, as from their point of view the longer it takes to find the block the less chance there is of being first.  Therefore this type of mining fee manipulation attack is unlikely to appear viable from the miner’s perspective.  This logic is dependent on my probability theory being correct, so please let me know if I am totally wrong here.

Many thanks

[piotr_n]

Of course it's unlikely to appear.
Why would miners want to lie about what fee they will accept  in a block they are going to mine in hopefully next 10 minutes?
That's what the software is for - you see what they are having to sell and you buy it from them for the price you choose yourself. Obviously you don't want a soft that blindly follows miners announcements.

Anyway,  I don't think you can run a biseness  where you advertise buying something for 10 while in fact you even buy it for 4. And even if you could run such a business ,  why would you even want to?  People are not that stupid - if there is money involved they will figure it out,  sooner or later.

[Gavin Andresen]

Smart (dynamic, floating) fees for the reference implementation wallet was pulled today:
  https://github.com/bitcoin/bitcoin/pull/4250

... and should appear in version 0.10.

The estimation code only considers transactions that are broadcast on the network, enter the memory pool (so are available to any miner to mine), and then are included in a block. So it is immune to miners putting pay-to-self transactions with artificially high fees in their blocks.

Right now if you use the default fee rules your transactions will take 2-6 blocks to confirm:
  http://bitcoincore.org/smartfee/fee_graph.html

The priority estimation code is even more broken; the reference implementation wallet will send a 56-million-priority transaction with no fee, which is nowhere near enough priority to get confirmed quickly:
  http://bitcoincore.org/smartfee/priority_graph.html

(the smart fee code estimates priority, too).

Release notes from doc/release-notes.md in the source tree:

Transaction fee changes
=======================

This release automatically estimates how high a transaction fee (or how
high a priority) transactions require to be confirmed quickly. The default
settings will create transactions that confirm quickly; see the new
'txconfirmtarget' setting to control the tradeoff between fees and
confirmation times.

Prior releases used hard-coded fees (and priorities), and would
sometimes create transactions that took a very long time to confirm.


New Command Line Options
========================

-txconfirmtarget=n : create transactions that have enough fees (or priority)
so they are likely to confirm within n blocks (default: 1). This setting
is over-ridden by the -paytxfee option.

New RPC methods
===============

Fee/Priority estimation
-----------------------

estimatefee nblocks : Returns approximate fee-per-1,000-bytes needed for
a transaction to be confirmed within nblocks. Returns -1 if not enough
transactions have been observed to compute a good estimate.

estimatepriority nblocks : Returns approximate priority needed for
a zero-fee transaction to confirm within nblocks. Returns -1 if not
enough free transactions have been observed to compute a good
estimate.

Statistics used to estimate fees and priorities are saved in the
data directory in the 'fee_estimates.dat' file just before
program shutdown, and are read in at startup.

[theymos]

The estimation code only considers transactions that are broadcast on the network, enter the memory pool (so are available to any miner to mine), and then are included in a block. So it is immune to miners putting pay-to-self transactions with artificially high fees in their blocks.

Ah, that sounds pretty good. I can't think of a way to manipulate that without wasting a lot of money. I guess only bitcoind servers will be online enough to get good data, though.

[Peter Todd]

The estimation code only considers transactions that are broadcast on the network, enter the memory pool (so are available to any miner to mine), and then are included in a block. So it is immune to miners putting pay-to-self transactions with artificially high fees in their blocks.

Ah, that sounds pretty good. I can't think of a way to manipulate that without wasting a lot of money. I guess only bitcoind servers will be online enough to get good data, though.

It's fairly easy to manipulate actually; sybil attacks are not that hard to do. (there's evidence they're being used against people accepting zeroconf) I demonstrated a 1BTC fee tx due to the estimator failing. Right now there is a cap of 100mBTC, but obviously that can drain your wallet pretty quickly.

[Gavin Andresen]

It's fairly easy to manipulate actually; sybil attacks are not that hard to do. (there's evidence they're being used against people accepting zeroconf) I demonstrated a 1BTC fee tx due to the estimator failing. Right now there is a cap of 100mBTC, but obviously that can drain your wallet pretty quickly.

... demonstrated in a completely artificial -regtest environment...

If you can Sybil somebody and control their view of the network, then it seems to me there are more potentially profitable attacks than "make them pay more in fees than they should."

But please feel free to demonstrate an actual, effective Sybil on the fee estimation code. bitcoincore.org is running a wallet-less bitcoind on port 8333 that generates the graphs at bitcoincore.org/smartfee/

(hacking into the web server to make it LOOK like the fee estimation code failed doesn't count, you have to manage to control it's p2p network connections and then manipulate the memory pool).

[Crowex]

 If I’m a miner then I’m selling a service to the consumer (someone who wants to get their transaction in the block chain). e.g. A miner advertises that it has so much hashing power and is likely to get your transaction included in (on average) so many minutes and the cost will be x bitcoins.
 In order for it to be a competitive marketplace the consumer should be able to choose who they buy their product or service from based on the cost of the transaction and the probable time for the transaction to be processed.
 This would be the natural way for market forces to keep transactions fees low.

 How could you implement a model like this without encouraging centralisation of mining?

 One interesting point is that the difficulty of a block is fixed no matter what transactions are included. Maybe it’s not something for bitcoin but are there any alt coins that have tried implementing a formula for a variable difficulty requirement depending on the type/fees of transactions included in the block?

[Peter Todd]

It's fairly easy to manipulate actually; sybil attacks are not that hard to do. (there's evidence they're being used against people accepting zeroconf) I demonstrated a 1BTC fee tx due to the estimator failing. Right now there is a cap of 100mBTC, but obviously that can drain your wallet pretty quickly.

... demonstrated in a completely artificial -regtest environment...

If you can Sybil somebody and control their view of the network, then it seems to me there are more potentially profitable attacks than "make them pay more in fees than they should."

But please feel free to demonstrate an actual, effective Sybil on the fee estimation code. bitcoincore.org is running a wallet-less bitcoind on port 8333 that generates the graphs at bitcoincore.org/smartfee/

Sybil attacks in Bitcoin are merely DoS attacks unless the attacker is willing to pay the rather large expense of making a fake block. You've turned that relatively minor attack into a dangerous and profitable one.

Anyway, I'm busy enough that I'm happy to let attackers demonstrate that attack for me.

[jonny1000]

I think this change could be a good improvement.  However in the long run, it may be better to have a system which focuses only on data in the past blocks.  This could be more robust and secure (as Peter mentions); it could also have some characteristics of a “perfect” market, as there would be perfect information.  Ultimately everybody knows what was in the previous blocks and everybody knows that everybody knows what was in previous blocks, to a high degree of certainty, this is what makes Bitcoin work, but it could also make markets very efficient and robust.  This does not apply to transaction data propagated throughout the network but not in blocks.

Even with this change the long term problems still exists, this fee is determined by demand and supply and currently supply is the arbitrarily defined block size limit.  Perhaps a solution where the block size limit and frees somehow dynamically change depending on each other could work.

[deliciousowl]

This fee exploit is bad news. Better listen to Peter...

[Crowex]

I think this change could be a good improvement.  However in the long run, it may be better to have a system which focuses only on data in the past blocks.  This could be more robust and secure (as Peter mentions); it could also have some characteristics of a “perfect” market, as there would be perfect information.  Ultimately everybody knows what was in the previous blocks and everybody knows that everybody knows what was in previous blocks, to a high degree of certainty, this is what makes Bitcoin work, but it could also make markets very efficient and robust.  This does not apply to transaction data propagated throughout the network but not in blocks.

Even with this change the long term problems still exists, this fee is determined by demand and supply and currently supply is the arbitrarily defined block size limit.  Perhaps a solution where the block size limit and frees somehow dynamically change depending on each other could work.

I'm still trying to get an understanding of all of the things involved here, so I might not have understood everything correctly.

It does seem a bit circular that the software seems to be telling you what fees to pay and then looking at the fees that it has suggested in order to decide what to tell you what to pay? Where is the market information coming from?

I can sort of see how the change could help. The analogy might be where you would be looking at how much to market or buy a property for, but you consider the advertised prices of unsold property in estate agents rather than the sold prices.

I agree it would be better to focus on the data in past blocks.
Can't somebody build a web-service that detects when transactions enter it's memory pool, looks at the transactions fees, and then examines when these transactions enter the block-chain and analyses all of this data to provide a more accurate estimate of required fees which would be based on real market data? Maybe have some sort of api that wallets could use? Or does this exist?

Is it a necessary requirement that each wallet should be able to independently analyse the transaction fee market and provide market information? Why not just go to some other service for this information?

[Gavin Andresen]

Can't somebody build a web-service that detects when transactions enter it's memory pool, looks at the transactions fees, and then examines when these transactions enter the block-chain and analyses all of this data to provide a more accurate estimate of required fees which would be based on real market data? Maybe have some sort of api that wallets could use? Or does this exist?

That is exactly what the 'smartfee' code in the reference implementation does.

RE: where does the market information come from:

Like any market, it comes from the collective action of lots of individual decisions. Different wallet software has different fee policies, and there is already a little bit of "I sent a transaction using wallet XYZ and it took FOREVER to confirm, WTF?!?" (or "why does exchange/wallet/service ABC charge me such high transaction fees").

As wallets mature, I expect them to specialize ("Save on Fees! Use UberWallet!") and/or compete for best cost/speed/reliability/predictability.

The default for the reference implementation will be "follow the herd" -- but the price will be set by the minority of people 'at the margins' who REALLY want their transactions to confirm quickly or REALLY want spend as little as possible on transaction fees. They will set -paytxfee and/or -txconfirmtarget  to override the default behavior.

And "they" are likely to be high-volume-transaction-creators-- like exchanges (probably want their transactions to confirm quickly; fewer customer support calls) or watch-a-video-get-a-few-bits services (probably want to cut costs any way they can, don't care if their customers have to wait a while for a withdrawal to get confirmed...).

RE: sybil/isolation attack:

Again, not a likely attack. You would have to:
1) Find some high-transaction-volume service and identify all of their bitcoin-network-connected nodes
2) Control ALL of those nodes' connections (expensive to do reliably with the 'connect out to 8 random peers' rule) FROM THE BEGINNING OF TIME (well, beginning of when it started running the smartfee code).
3) Let that node see only extremely-high-fee transactions (there aren't very many of those, so you'll need to manage to control the node's connections for a while).
4) Expect the node's operator to send a lot of transactions and not notice that they were paying abnormally high transaction fees.

If you are running a high-transaction-volume service you probably already have several connections into the bitcoin p2p network because you have probably already been the target of a distributed denial of service attack....

Definitely not an issue for Bitcoin-Qt, because you're shown the total amount + fees you'll pay before every transaction.

[Crowex]

ok, thanks. I missed the point in your original post that only transactions that are included in the block are used for the analysis. So in effect we are dealing with real market data.

Why wouldn't this system incentivise large mining pools to only mine transactions with high fees and not mine anything with a low or zero fee in order to manipulate the transaction fee calculated by the reference implementation?