CreateTransaction: suggest/enforce fee for big low-priority transactions

[Gavin Andresen]

This is:  https://github.com/bitcoin/bitcoin/issues#issue/86

currently (2011/03/01) a backlog of transactions that dont seem make it into blocks is building up (according to ArtForz now 670 transactions).

People are starting to complain, see http://bitcointalk.org/index.php?topic=3835.msg57031#msg57031 for an example.

This was discussed on #bitcoin-dev

ArtForz> that tx should have a score of 20567855
ArtForz> so it falls short of the dPriority > COIN * 144 / 250 test
ArtForz> and as tx size is > 4000, fAllowFree is never true for it
theymos> Ah. So it will never confirm?
ArtForz> probably not

ArtForz> looks like the tx-sending code needs some fixing
molecular> like not generate a tx with a score too low, or suggest to add a fee?
ArtForz> yep
ArtForz> check size and priority of transaction, if it's > 4kB and score < 57600000 require min fee
ArtForz> maybe reduce the size limit to 3.5k or so, otherwise it can only get into an otherwise empty block

Maybe the transaction cache should not be unlimited?

[1713529152]

1713529152

[1713529152]

1713529152

[ribuck]

It's not optimal if transactions are not being processed while there's space available in the blocks.

Why not experiment, in the next client, with a setting that allows the user to specify a very low default fee for all transactions (well below 0.01 BTC). Then see what happens.

[Gavin Andresen]

This definitely needs fixing; it is another "people getting lots of very small change from mining pools" issue.

The code that controls this is CTransaction::GetMinFee() in main.h and CreateNewBlock() in main.cpp.

We should think about:

+ Is setting aside a specific amount of space for free transactions the right thing to do?  Maybe blocks should just get filled in reverse priority order (with transactions with fees at the front of the line)

+ What to do with the current transaction backlog.
If old, big, low-priority transactions get flushed, then there needs to be some way for the bitcoin client(s) that sent them to reclaim those coins.  Perhaps the client should stop retransmitting, and reclaim, transactions if, oh, 5,000 blocks go by without the transaction getting accepted.

Or maybe it makes sense to let those old transactions trickle in; perhaps the next version of bitcoin should take the oldest transaction in the memory cache and add it to the generated block.

[m0mchil]

Client should also warn about low priority transactions and suggest a fee to increase chance for inclusion.

[lfm]

Seems like it just should be simpler so the strategies for dealing with it could be seen by everyone and dealt with behaviourly instead of with software patches. Like fifo for free txn with priority for fee txn and a way to add a fee to your txn.

[Binford 6100]

It's not optimal if transactions are not being processed while there's space available in the blocks.

Why not experiment, in the next client, with a setting that allows the user to specify a very low default fee for all transactions (well below 0.01 BTC). Then see what happens.

yay! I support that. if nothing else, collecting .0001 is a nice way of keeping score
now that the full precision display/handling seem to be fixed.

[Mike Hearn]

Right, that's pretty much what dropping the concept of "free space" would do. All space is equal, transactions are ordered by fee and priority score. So you can never get a transaction that's too big/low priority to enter a block.

[Steve]

I don't think it will be a good idea to allow any announced transaction to fail to get processed by the network...the announcement of the transaction on the network should be the act of transacting...inclusion in a block just makes it confirmed and permanent.  I initially thought that some policy of expiring unprocessed transactions would be good, however that would open up the possibility of clients deliberating crafting transactions to fail (in order to get something for nothing).

So, what if both the age and the transaction fee are considered in the priority calculation?  Any transaction over 4 hours (just making something up) old should have priority over any younger, fee bearing transaction, no matter the fee involved.  In other words, transactions under 4 hours old would be processed based on fee first, age second, while transactions older than 4 hours would be processed based purely on age (fees breaking ties I guess...and obviously all the nodes on the network would have to enforce this such that a miner couldn't just ignore those rules).  Under this scheme (after the end of block awards), fee bearing transactions would entirely subsidize the cost of free transactions (otherwise there would be no incentive for block creation).

I don't know what implications this might have on block size limits...I haven't learned enough about the system to understand the importance of transaction or block size limits...if there is a high enough sustained transaction volume, I imagine the block chain would start to fall further and further behind.  Lifting those limits might open the network to DOS attacks but having those limits would seem to create an upper bound on transaction volume and the system would have no choice but to abandon really old transactions that have yet to be processed.

If abandoned transactions were unavoidably a part of the system, I don't think reclaiming those coins is really an issue...at least not at the protocol or network level.  Clients could simply reverse those transactions in their tallies if a certain amount of time has passed without confirmation (such that the network would have flushed it).  The goal of the network would be to never flush any transaction, but if it started to happen (even on a small scale), maybe the block and/or transaction size could be increased...all clients don't necessarily need to retain the transactions inside blocks...they could keep running tallies of bitcoins in various accounts and validate/digest each block as it comes through (maybe with a small tail to handle the case where blocks occur close in time to each other).  You would still want to have a number of super nodes keeping the full history (and making all transactions available to peers as needed), but you would eliminate a lot of storage requirements on a lot of nodes without losing their important block validation role.

[caveden]

Gavin, IMHO, transaction fee policy is a business to miners. You shouldn't care, unless you're developing a miner.

I think that the miner that comes with the default client should just be discontinued and removed from future versions. This way, there's no "default policy" and miners will create their own. If they want to accept free transactions they will, otherwise they won't, and they'll find their ways of prioritizing them.

The only thing that the client should be capable of doing is resending a transaction, with a higher fee.

It would also be nice too if the user could specify the transaction fee per transaction. The GUI could also allow different kind of inputs (absolute value, relative to the transaction size, relative to the amount etc)

[caveden]

By the way, an important question: if I generate a 500KB block with only free transactions in it, it will be accepted by the network, right?
Or this current "default fee policy" is not just for the default client, but for the protocol as a whole?

[Syke]

This definitely needs fixing; it is another "people getting lots of very small change from mining pools" issue.

That sounds like the pool needs to do it differently, not the network. If this is a pool payout to 500 people in a single transaction, then the pool needs to break it down into smaller transactions. I sent a few regular transactions yesterday and they were immediately included in solved blocks.

[Gavin Andresen]

By the way, an important question: if I generate a 500KB block with only free transactions in it, it will be accepted by the network, right?
Or this current "default fee policy" is not just for the default client, but for the protocol as a whole?

You can generate a 1MB block with only free transactions in it and it will get accepted (maximum block size is 1MB, although the standard bitcoin client will never generate blocks larger than 500K).

That sounds like the pool needs to do it differently, not the network. If this is a pool payout to 500 people in a single transaction, then the pool needs to break it down into smaller transactions. I sent a few regular transactions yesterday and they were immediately included in solved blocks.

The problem isn't the pool payout-- the problem is that people participating in the pool end up with wallets full of tiny (e.g. penny-size) transactions.  When they go to spend those tiny transactions, they're bundled up together to make a transaction that is small in value but large in size.

Pools can mitigate the problem by requiring larger minimum payouts (e.g. 1 BTC instead of 0.01 BTC).

[Syke]

The problem isn't the pool payout-- the problem is that people participating in the pool end up with wallets full of tiny (e.g. penny-size) transactions.  When they go to spend those tiny transactions, they're bundled up together to make a transaction that is small in value but large in size.

Pools can mitigate the problem by requiring larger minimum payouts (e.g. 1 BTC instead of 0.01 BTC).

Ok, that makes sense. Changing the pool payouts would mitigate it, but people could still end up with fragmented wallets for any number of other reasons. How about a feature in the client to consolidate one's wallet. It would bundle up many of the small wallets and send them to itself. I really dislike the idea of changing the rules at this point.

[Steve]

Gavin, IMHO, transaction fee policy is a business to miners. You shouldn't care, unless you're developing a miner.

I think that the miner that comes with the default client should just be discontinued and removed from future versions. This way, there's no "default policy" and miners will create their own. If they want to accept free transactions they will, otherwise they won't, and they'll find their ways of prioritizing them.

The only thing that the client should be capable of doing is resending a transaction, with a higher fee.

It would also be nice too if the user could specify the transaction fee per transaction. The GUI could also allow different kind of inputs (absolute value, relative to the transaction size, relative to the amount etc)

I think this is a bad idea, possibly very bad...the network has to set the rules regarding what are acceptable blocks and what are not...and with that, what transactions those blocks must include.  If it's easy to create transactions that the issuer can be confident the network will drop, it opens the whole system up to rampant abuse and fraud.  If this were possible someone could easily craft bitcoin transactions designed to be abandoned and use those for payment...a person could obtain some good or service long before the unsuspecting victim realizes what has happened.  This is not a recipe that will engender confidence in the system.

Furthermore, this would put too much power in the hands of miners...it's the responsibility of the network to validate blocks and transactions and to decide what is acceptable and what is not.  Not the miners.  I cannot stress this point enough.  The miners will still have an equal role in setting the market rate for transactions (which is what I think you're after here).  Setting aside the current 50btc payout for blocks, if the predominate fee that people are offering is low enough, there will be little incentive to mine (based solely on the transaction fees)...so, mining activity will plummet as will the difficulty.  I think what you'd find is that it would get low enough that some merchant would realize they need their transactions verified and would press some old spare PC into service to do the mining (with the difficulty being low, it would be a very cheap thing to do).  They would be generating blocks for their own transactions and because of the rules enforced by the network, everyone else's as well.  He would prioritize his own transactions (but would have to remain within the guidelines of the network)...other merchants, needing the same service would either run a miner themselves, or start attaching a fee to their transactions.  And, soon, anyone wanting to see their transactions clear a little sooner than 4 hours (again, arbitrary number I picked) would start attaching a small transaction fee.  Conversely, if everyone is competing to have their transactions verified quickly and bidding up the fee for that, then more people will enter the mining business to capture a piece of those profits.

[Steve]

Oh...one other thing...I would think it should be fairly trivial in the client to come up with suggested fees for desired verification speeds based on the recent block statistics...I could imagine a selection list in the GUI like:

   Transaction fee (expected confirmation speed):
       0.00 btc (>4 hrs)
       0.01 btc (~1.5 hrs)
       0.05 btc (~30 min)
       1.00 btc (<15 min)

[lachesis]

I agree with caveden: I think fees should be entirely up to the miners.

I like the idea of offering a small (฿0.0001) fee for every transaction, but wouldn't that just lead to more dust and wallet fragmentation?

Alternatively, we could greatly raise the limits for free transactions, or raise their score if they're old so they can get outside the free transaction space. What are the problems with that approach?

[theymos]

Most miners (including slush's pool) use Bitcoin's getwork, so default fees are important. The ability to modify fee rules would be nice, though.

Is setting aside a specific amount of space for free transactions the right thing to do?  Maybe blocks should just get filled in reverse priority order (with transactions with fees at the front of the line)

Setting aside space for free transactions is needed because otherwise someone could waste lots of disk space by sending endless free transactions. The 4kB priority limit should be removed, though -- let the 27kB free space be sorted by priority. The free space should also be increased to 50kB, IMO.

Perhaps the client should stop retransmitting, and reclaim, transactions if, oh, 5,000 blocks go by without the transaction getting accepted.

Good idea. There should also be a manual reclaim ability.

[0x6763]

the network has to set the rules regarding what are acceptable blocks and what are not...and with that, what transactions those blocks must include. ...  They would be generating blocks for their own transactions and because of the rules enforced by the network, everyone else's as well.

I believe the network currently does not enforce that any certain transactions be placed into a block.  Miners already have the option to pick and choose which transactions to include according to whatever criteria they wish.  All they have to do is modify their software to do so, and it would still be fully within the rules of the bitcoin network.  The network only cares that the blocks and transactions are valid, not that the block includes transaction X and transaction Y, etc.  I'm not sure it's even possible to enforce that miners include all/certain transactions, since not every node is always aware of every transaction when a block is generated.

[BitterTea]

Oh...one other thing...I would think it should be fairly trivial in the client to come up with suggested fees for desired verification speeds based on the recent block statistics...I could imagine a selection list in the GUI like:

   Transaction fee (expected confirmation speed):
       0.00 btc (>4 hrs)
       0.01 btc (~1.5 hrs)
       0.05 btc (~30 min)
       1.00 btc (<15 min)

It wouldn't be expected confirmation speed, it would be expected block insertion speed.

Confirmations always happen every ~10 minutes, since they are blocks created after the transaction is inserted into a block.

[twobitcoins]

Why do we think there should be any free transactions?  Of course free is nice, but is it workable?

First, even assuming there is no transaction spam, we must somehow choose an appropriate amount of free space per block.  If we make it large enough that all free transactions get into blocks, adjusting periodically as transaction volume increases, there will never be any fees, which doesn't seem sustainable.  If we limit it, we have usability issues.  How do people know when to include a fee?  An ever-increasing queue of free transactions seems likely.

Second, whatever free space we do allow will be consumed by spam.  It doesn't seem possible in general to distinguish spam from legitimate micropayment use cases.  And do we actually care?  If legitimate micropayments generate huge transaction volume, we probably don't want to allow them for free anyway.

Including transactions in a block has a real, non-zero cost.  If we charge a fair price, a lot of problems go away:

1. No more spam.  Or at least, spam is paid for.  If the price is fair, who cares?

2. No need to increase fees with block size.  If all transactions are required to pay a fair price, we can just include all transactions in a block immediately.

3. Everybody knows what fee to include to have their transaction processed promptly.

Granted, we still need some upper limit on block size to prevent block generators from spamming 1 TB blocks.

We just need to come up with a fair fee.  Filling 1 MB with 5000 200-byte transactions at 0.01 BTC each would cost 50 BTC, or about $50.  Perhaps that's reasonable.  At 0.0001 BTC, it would cost about $0.50.  That seems too low.  I was hoping we could tell people "transactions cost a hundredth of a cent" -- but maybe "transactions cost a penny" is good enough.