CreateTransaction: suggest/enforce fee for big low-priority transactions

[StevenC]

Hi,  I was affected by this issue too.

I'm totally new to bitcoin;  installed the Debian bitcoin-cli package yesterday, created a separate address from the default one (14mX4LPATYGqHqN16egft9dzwbjGy8j1Zc) and requested 0.05 from the faucet (txid e5017a6831e7034432ca033258254f3fda37ebae9c0de9765918380160d690cd) which has remained pending for almost 18 hours.

When investigating this I noticed lots of errors in ~/.bitcoin/debug.log like this:

Code:

ERROR: ConnectInputs() : 94ba8e6447 mapTransactions prev not found d884bcbf17
ERROR: AcceptToMemoryPool() : ConnectInputs failed 94ba8e6447

Some of these partial txid's seem to 'cascade' on from one another, so I figured there was a dependency chain forming here.  I've managed to render this in the PDF below;  maybe that helps people to visualise what's going on.  If anyone else has transactions not completing, they may try searching for the transaction ID in the PDF document.  I found my transaction in there with at least 5 preceding it.  I count about 275 transactions in total at the moment, but I don't currently have the means to check if any of the displayed transactions eventually got processed.

• http://pyro.eu.org/stuff/bitcoin-txqueue.pdf

Hope this is somehow helpful!

[1714770545]

1714770545

[1714770545]

1714770545

[1714770545]

1714770545

[1714770545]

1714770545

[jgarzik]

Perhaps the client should stop retransmitting, and reclaim, transactions if, oh, 5,000 blocks go by without the transaction getting accepted.

Good idea. There should also be a manual reclaim ability.

I think gavin already has a 'refund transaction' patch in his repo somewhere.

Overall, I think transactions should have a maximum lifetime outside of a block.  If they exist past 24 hours or 150 confirmations, clients should drop the TX from their cache.

[jgarzik]

Why do we think there should be any free transactions?  Of course free is nice, but is it workable?

First, even assuming there is no transaction spam, we must somehow choose an appropriate amount of free space per block.  If we make it large enough that all free transactions get into blocks, adjusting periodically as transaction volume increases, there will never be any fees, which doesn't seem sustainable.  If we limit it, we have usability issues.  How do people know when to include a fee?  An ever-increasing queue of free transactions seems likely.

Second, whatever free space we do allow will be consumed by spam.  It doesn't seem possible in general to distinguish spam from legitimate micropayment use cases.  And do we actually care?  If legitimate micropayments generate huge transaction volume, we probably don't want to allow them for free anyway.

It is a natural consequence of the system that any free TX space will be used up, quickly, once bitcoin is popular.

Therefore, it is expected that adding a transaction fee will be strongly recommended simply to have one's transactions confirm in a reasonable amount of time.

"free tx" is just a nice feature while the bitcoin system is under-utilized.  Once bitcoin network sees more transactions on a daily basis, getting a slot in the free-tx area will be like winning the lottery.

[0x6763]

Why do we think there should be any free transactions?  Of course free is nice, but is it workable?

First, even assuming there is no transaction spam, we must somehow choose an appropriate amount of free space per block.  If we make it large enough that all free transactions get into blocks, adjusting periodically as transaction volume increases, there will never be any fees, which doesn't seem sustainable.  If we limit it, we have usability issues.  How do people know when to include a fee?  An ever-increasing queue of free transactions seems likely.

Second, whatever free space we do allow will be consumed by spam.  It doesn't seem possible in general to distinguish spam from legitimate micropayment use cases.  And do we actually care?  If legitimate micropayments generate huge transaction volume, we probably don't want to allow them for free anyway.

Including transactions in a block has a real, non-zero cost.  If we charge a fair price, a lot of problems go away:

1. No more spam.  Or at least, spam is paid for.  If the price is fair, who cares?

2. No need to increase fees with block size.  If all transactions are required to pay a fair price, we can just include all transactions in a block immediately.

3. Everybody knows what fee to include to have their transaction processed promptly.

Granted, we still need some upper limit on block size to prevent block generators from spamming 1 TB blocks.

We just need to come up with a fair fee.  Filling 1 MB with 5000 200-byte transactions at 0.01 BTC each would cost 50 BTC, or about $50.  Perhaps that's reasonable.  At 0.0001 BTC, it would cost about $0.50.  That seems too low.  I was hoping we could tell people "transactions cost a hundredth of a cent" -- but maybe "transactions cost a penny" is good enough.

You're not suggesting price-fixing the fees, are you?

[twobitcoins]

It is a natural consequence of the system that any free TX space will be used up, quickly, once bitcoin is popular.

Therefore, it is expected that adding a transaction fee will be strongly recommended simply to have one's transactions confirm in a reasonable amount of time.

"free tx" is just a nice feature while the bitcoin system is under-utilized.  Once bitcoin network sees more transactions on a daily basis, getting a slot in the free-tx area will be like winning the lottery.

In the future when bitcoin is heavily used, I don't see any value in a "free transaction lottery".  It adds complexity for no apparent reason.

If bitcoin is underused, sure, free transactions are nice.  I just don't see any good solution to the spam problem, and it already seems that micropayments from pools are bumping into the limits.  I'm not sure it makes sense to invest in a system of free transactions that works in the short term if it will be obsolete in the long term anyway.

[Garrett Burgwardt]

Why do we think there should be any free transactions?  Of course free is nice, but is it workable?

First, even assuming there is no transaction spam, we must somehow choose an appropriate amount of free space per block.  If we make it large enough that all free transactions get into blocks, adjusting periodically as transaction volume increases, there will never be any fees, which doesn't seem sustainable.  If we limit it, we have usability issues.  How do people know when to include a fee?  An ever-increasing queue of free transactions seems likely.

Second, whatever free space we do allow will be consumed by spam.  It doesn't seem possible in general to distinguish spam from legitimate micropayment use cases.  And do we actually care?  If legitimate micropayments generate huge transaction volume, we probably don't want to allow them for free anyway.

Including transactions in a block has a real, non-zero cost.  If we charge a fair price, a lot of problems go away:

1. No more spam.  Or at least, spam is paid for.  If the price is fair, who cares?

2. No need to increase fees with block size.  If all transactions are required to pay a fair price, we can just include all transactions in a block immediately.

3. Everybody knows what fee to include to have their transaction processed promptly.

Granted, we still need some upper limit on block size to prevent block generators from spamming 1 TB blocks.

We just need to come up with a fair fee.  Filling 1 MB with 5000 200-byte transactions at 0.01 BTC each would cost 50 BTC, or about $50.  Perhaps that's reasonable.  At 0.0001 BTC, it would cost about $0.50.  That seems too low.  I was hoping we could tell people "transactions cost a hundredth of a cent" -- but maybe "transactions cost a penny" is good enough.

You're not suggesting price-fixing the fees, are you?

I would like to second this idea, or one like it. A consistent (small) fee for all transactions is ideal, then there's no problems with accepting some transactions as free, etc.

[twobitcoins]

You're not suggesting price-fixing the fees, are you?

The current client has default fees, but they are not enforced, so miners can change them if they like.  My proposal is the same.  Is that price fixing?

The current client has a hard limit on block size to prevent spamming by miners.  My proposal is the same.  If we ever bump into the hard limit, it becomes an artificial restriction on the rate of transactions, driving up prices.  Is that essentially equivalent to price fixing?

I guess I do worry that miners might allow block sizes up to the hard limit while charging no fees or too low fees (too low to compensate the rest of the network for storing that block for all eternity).  I'm not sure what to do about that.  The only solution seems to be for other miners to reject blocks that charge too little fees, which definitely seems like price fixing.

[Steve]

the network has to set the rules regarding what are acceptable blocks and what are not...and with that, what transactions those blocks must include. ...  They would be generating blocks for their own transactions and because of the rules enforced by the network, everyone else's as well.

I believe the network currently does not enforce that any certain transactions be placed into a block.  Miners already have the option to pick and choose which transactions to include according to whatever criteria they wish.  All they have to do is modify their software to do so, and it would still be fully within the rules of the bitcoin network.  The network only cares that the blocks and transactions are valid, not that the block includes transaction X and transaction Y, etc.  I'm not sure it's even possible to enforce that miners include all/certain transactions, since not every node is always aware of every transaction when a block is generated.

What would stop a well funded, bad actor from using a super computer to create blocks with only a few of it's own transactions in it?  Someone that wanted to undermine bitcoin?  I think it's a very bad idea for the network to not set some criteria regarding transaction inclusion into blocks.

Also, regarding another question (from twobitcoins) about whether there should be free transactions at all: first, I'd say that regardless of fee or no fee, dropped transactions are bad...it opens up a possibility of fraud and it will undermine confidence in the system (see the various postings of people wondering and worrying about what happened to their bitcoins if you doubt that).  It is correct that block generation is not free, but people will pay to get transactions into block sooner rather than later and these paying transactions will subsidize the free transactions and free transactions increase the overall value of the bitcoin network.

As for spam transactions, well, there is already a good solution that was proposed years ago to solve the spam dilemma...proof-of-work.  If you want to issue a free transaction, the network could require a small proof of work for such transactions.  The proof of work required for free transactions could even be linked to the current mining difficulty...the higher the difficulty for mining, the higher the difficulty require for free transactions.  There could even be a correlation between the transaction fee and the required difficulty (such that paying a small amount wouldn't eliminate the proof of work, but might let you buy down the difficulty).  And, you could even link that required fee to the average number of transactions in recent blocks such that a block containing only proof-of-work free transactions would amount to some total (say 50 btc)...so, for example, if the average number of transactions in recent blocks is 100, the transaction fee required to completely eliminate the proof of work requirement on any individual transaction would be 0.50 btc.  If the average number of transactions was only 10, then the fee would be 5 btc/transaction.  Clients wouldn't even forward transactions that didn't have the required fee or proof of work.  People could even offer services to compute the proof-of-work on behalf of users on less powerful clients (for a fee of course...or it could be a part of some larger account or hosting service that a user purchases).

[jgarzik]

What would stop a well funded, bad actor from using a super computer to create blocks with only a few of it's own transactions in it?  Someone that wanted to undermine bitcoin?  I think it's a very bad idea for the network to not set some criteria regarding transaction inclusion into blocks.

If you have enough CPU power to dictate specifically which transactions into a block, we have bigger things to worry about (double spending).

There are plenty of validity checks on blocks and tx's in the client.  It's just that fees are not one of those checks, permitting miners to choose their own policy.

[StevenC]

If someone can flood the network with free transactions to/from their own addresses, and there's no way to make sure legitimate transactions are unaffected, there'd soon be no such thing as a free transaction any more.  Would it be so bad to disallow them to conserve resources?  The difficulty I imagine, would be in enforcing a minimum fee that would be remain sensible as BTC's value varies.

...was proposed years ago to solve the spam dilemma...proof-of-work.  If you want to issue a free transaction, the network could require a small proof of work for such transactions.

As for requiring 'proof of work';  requiring a BTC value inherently does that (they could join a mining pool to do the 'proof of work', then use their earnings to pay the fees).

[molecular]

Digesting all comments, maybe this could a workable consensus:

 1.) remove "free transaction space" altogether. Argument (by many): free transactions are not going to make it in the long run anyways, where probably never intended to. This would generate a market for transaction processing by miners, which was intended from the beginning and will be their main income after generation ends. Regarding the pool issue: This moves the problem to the originator (where it belongs): the pool-miner receiving very small payments has to either pay the price by using higher tx-fee to spend it or avoid the problem by increasing his payout treshold. This also mitigates DoS attacks.

 2.) make smaller transaction fee (down to 0.0001) possible. Argument: 0.01 might be too much as minimum fee. There are poor people.

 3.a) Client should also warn about low priority transactions and suggest a fee to increase chance for inclusion.
 3.b) Make it possible to increase transaction fee of an already broadcast transaction. Argument: this would obsolete 3.a) and introduce transaction fees to users in a nice fashion (user sees transaction not being confirmed, user sees the fee is 0.00, some gui hint tells him he can speed things up by increasing fee)

 4.) Let existing backlog trickle in slowly, don't automatically expire or allow reclaiming of transactions in any way. Argument (by Steve): Expiring transaction would open up the possibility of clients deliberately crafting transactions to fail (in order to get something for nothing)

[caveden]

Most miners (including slush's pool) use Bitcoin's getwork, so default fees are important. The ability to modify fee rules would be nice, though.

Can't this code be just separated from the default client?
I don't see a reason to have a miner in the default client anymore... It's useless, since CPU-mining is useless, and besides, things like fee policy for instance are really arbitrary and may change during time. It would be better if the default client was there just to set up the "protocol rules" and nothing more.

Anyway, if we really need to set up a default fee policy, then definitely these free transaction limits should be raised.

By the way, I still think it would be a good idea to make the maximum block size variable, instead of a hardcoded constant as it's right now.

Perhaps the client should stop retransmitting, and reclaim, transactions if, oh, 5,000 blocks go by without the transaction getting accepted.

Good idea. There should also be a manual reclaim ability.

Yes, manual reclaim and manual resending, that should do it.

[theymos]

Can't this code be just separated from the default client?
I don't see a reason to have a miner in the default client anymore... It's useless, since CPU-mining is useless, and besides, things like fee policy for instance are really arbitrary and may change during time. It would be better if the default client was there just to set up the "protocol rules" and nothing more.

The client has to do all the verification and processing for blocks/transactions, anyway, so I don't see the harm in providing the necessary info to separate mining software. It's certainly not something every mining program should have to handle.

Currently clients will not even relay transactions if their fees are "too low". This needs to be changed for the fee market to work.

By the way, I still think it would be a good idea to make the maximum block size variable, instead of a hardcoded constant as it's right now.

The user should never be able to change the max block size without significant effort. You're pretty much guaranteed to end up on a separate chain sooner or later if you change this value.

[caveden]

Currently clients will not even relay transactions if their fees are "too low". This needs to be changed for the fee market to work.

Really? This is bad! It's not up to the client to decide that... at most, relay and then forget about them, but don't ignore them.

By the way, I still think it would be a good idea to make the maximum block size variable, instead of a hardcoded constant as it's right now.

The user should never be able to change the max block size without significant effort. You're pretty much guaranteed to end up on a separate chain sooner or later if you change this value.

No, what I suggest is making such limit adjustable just like the difficulty factor, as a "network rule", not something a user could change. There in the topic I explain more: http://bitcointalk.org/index.php?topic=1865.0

[theymos]

No, what I suggest is making such limit adjustable just like the difficulty factor, as a "network rule", not something a user could change. There in the topic I explain more: http://bitcointalk.org/index.php?topic=1865.0

I'm still against that. Until there is some out-of-band method for lightweight clients to be notified of their own transactions, the max block size must be kept low enough for everyone to keep up with blocks. It's not reasonable to expect every client to download 100MB per hour (or whatever).

Automatic adjustments might be OK if lightweight clients can find their own transactions without downloading full blocks. Then only miners would have to worry about network and disk space usage. But when the max block size is over 10MB, the massive disk space requirements (500GB per year) will cause there to be only a handful of miners, and at that point all the miners will be able to come to an agreement about block size with relative ease due to their small number.

[jgarzik]

Currently clients will not even relay transactions if their fees are "too low". This needs to be changed for the fee market to work.

I disagree.  You cannot change that without vastly increasing the ability to spam the network with useless transactions.

[m0mchil]

My humble opinion - there is no free lunch. Every transaction must have a price. This price should be a constant, bound to current definitions of COIN and CENT.

Anything else is just a dream.

[Steve]

After IRC discussion, I became aware of transaction rate limiting as a means of mitigating the spam problem...I hadn't realized that before, but I think it's a pretty simple solution to the spam problem (much simpler that the proof of work suggestion I made earlier).

I don't agree with eliminating free transactions...in fact, I believe the likely trajectory of things is that transactions will be remain free (I mean just look at all the unprofitable mining that's happening now).  Merchants, exchanges and many others have an interest in facilitating transactions and will always step in to ensure they get processed, even the free ones.  In fact, everyone using bitcoins has an interest in free transactions.  It's a good selling point and a way to attract new users (and without new users, this experiment will fail).  I know this doesn't come as good news to those of you hoping to strike it rich through mining (though maybe there is a future for you in collecting fees from those merchants willing to subsidize the cost of mining).

Other than that, I agree with molecular's other proposals.  I will add that clients need to become smarter about prioritizing the transactions they forward to ensure the p2p traffic remains manageable and clients need to have a strong role in deciding which blocks are accepted and which are rejected (it would be bad if an attacker with a very large budget could buy the resources necessary to dominate block creation and prevent almost everyone's transactions from getting through).

[ribuck]

It needs to be simple for the user, yet effective for controlling spam.

How about a single setting in the standard client, that controls the fee per transaction input. The default can be something very low (0.0001 per transaction input).

That will also incentivize someone to write code to make the client's input coin selection more optimal, to the benefit of the network as a whole.

[theymos]

I disagree.  You cannot change that without vastly increasing the ability to spam the network with useless transactions.

Having a memory pool size limit where transactions are removed based on priority would solve that, I think. Enabling the -limitfreerelay switch by default might also help (with lower minimum fees).