TX replacement and nLockTime

[Andrew Vorobyov]

I have other ideas too and way too little time to implement them. I'll at least try to write them up when I get the time in case other people are interested.

That's your duty for community... don't let them die in your head

[1714146084]

1714146084

[1714146084]

1714146084

[blueadept]

Exactly why I want to write them up. I think it could be a killer app for Bitcoin.

[jevon]

DoSing entities would send themselves multiple versions of multiple transactions rapidly, forcing each version to propagate through the network and use disk and CPU resources on each node for verification - thus, each version needs to cost more.  I'd even go so far to say that each new version should have to at least double the fees of the previous one, with the initial/timelocked transaction having a set minimum fee (or a percentage fee, even).  That increases the cost of multiple versions very rapidly while keeping the normal use case of a single replacement cheap.

Increasing the fee on each version is a perfect solution, except not doubling. It should increment by a constant predefined amount that is more than the maximum estimated cost of bandwidth and resources consumed by broadcasts. Then it's a user fee.

The inputs wouldn't have to change as long as there's a change return to reduce.

I'm also considering the idea that if transaction replacement is not enabled, many use cases would be enabled by changing the semantics of nLockTime such that transactions with an nLockTime in the future (either block number OR time) must not be accepted to the memory pool at all.

That would be a good temporary fix that would work in simple cases with only one nLockTime transaction, like an escrow expiration payment.

They shouldn't create multiple versions though, because if replacement is still disabled, then it's just a race at nLockTime to broadcast first and the version doesn't matter.

Another possible temporary fix would be to enable only a single replacement: final replaces non-final. That would allow a single lockTime transaction for expiration default, and is only one line of code without getting into all the issues. It's stupid that a lockTime transaction can sit in the pool and block a final transaction that trumps it.

[etotheipi]

Another possible temporary fix would be to enable only a single replacement: final replaces non-final. That would allow a single lockTime transaction for expiration default, and is only one line of code without getting into all the issues. It's stupid that a lockTime transaction can sit in the pool and block a final transaction that trumps it.

For that, you just set sequence numbers to 0xfffffffe.  The maximum value is 0xffffffff, which means that there's only room for one replacement before the tx becomes actually-final.

EDIT: The point was, that you could make a or node-acceptance rule that only allows replacement on seq=0xfffffffe instead of any value

[jevon]

Either way will work. Just make sure they can't do this, like with 4 inputs:
0xfffffffe-0xfffffffe-0xfffffffe-0xfffffffe
0xfffffffe-0xfffffffe-0xfffffffe-0xffffffff
0xfffffffe-0xfffffffe-0xffffffff-0xffffffff
0xfffffffe-0xffffffff-0xffffffff-0xffffffff
0xffffffff-0xffffffff-0xffffffff-0xffffffff

Final replaces non-final is simple to implement, just add to the replacement conditions:
  if (!IsFinal())
      return false;

In other words, replacement is only allowed if you're replacing with this:
0xffffffff-0xffffffff-0xffffffff-0xffffffff

The only difference between this and blueadept's way is whether the lockTime transaction can get into the pool ahead of time, or they have to wait until lockTime to broadcast it (if they're still around by then).

[etotheipi]

Either way will work. Just make sure they can't do this, like with 4 inputs:
0xfffffffe-0xfffffffe-0xfffffffe-0xfffffffe
0xfffffffe-0xfffffffe-0xfffffffe-0xffffffff
0xfffffffe-0xfffffffe-0xffffffff-0xffffffff
0xfffffffe-0xffffffff-0xffffffff-0xffffffff
0xffffffff-0xffffffff-0xffffffff-0xffffffff

Final replaces non-final is simple to implement, just add to the replacement conditions:
  if (!IsFinal())
      return false;

In other words, replacement is only allowed if you're replacing with this:
0xffffffff-0xffffffff-0xffffffff-0xffffffff

The only difference between this and blueadept's way is whether the lockTime transaction can get into the pool ahead of time, or they have to wait until lockTime to broadcast it (if they're still around by then).

Good point about the multi-input seq numbers.  I forgot they are per-TxIn, not per-Tx (like nLockTime).

Gavin and I debated your last point the other day, because we were determining if it was possible to have a "Seller automatically gets payment after 31 days" in escrow transactions, in case the buyer is too lazy to sign the escrowed Bitcoins to the seller.   What we determined is that if the locktime is in the future and any one sequence number is non-maxxed, then the tx will be accepted to the memory pool but not included in a block.

What this gives is you a very inconvenient implementation of what you have already proposed.  It is replaceable but only if you get a miner to agree to drop the locked tx in their memory pool and accept your final replacement tx.  If they have enough mining power and there is enough time, your tx will eventually make it into a block and invalidate the locked tx in all nodes' memory pools.

The issue, of course, is that it's not really feasible to have regular users contacting miners to help them replace tx like this (in the event that the seller broadcasts the 31-day tx right away, hoping the buyer can't figure out how to replace it).  Also, I didn't like the idea that anyone would be able to bribe a miner to replace their 0-confirmation tx.  But that's a whole 'nother story...

[Gavin Andresen]

The issue, of course, is that it's not really feasible to have regular users contacting miners to help them replace tx like this (in the event that the seller broadcasts the 31-day tx right away, hoping the buyer can't figure out how to replace it).  Also, I didn't like the idea that anyone would be able to bribe a miner to replace their 0-confirmation tx.  But that's a whole 'nother story...

In the long run, I think we have to assume that miners will do what is most profitable, and design escrow/payment protocols around that assumption.

Replacing non-final transactions in the memory pool potentially opens up a whole bucket of worms, but I think the short-term challenge is to figure out if we should change the rules we have now. If we do (I think we should), then I think it would be a mistake to do anything other than "create rules that will maximize miner profits."  Because if we do something else, then sooner or later I believe there will be a "miners special" version of the code that has miner-friendly rules.

If you agree with me that the default rules in the client should maximize miners' profits, then it seems to me there's one simple rule change we should make:

If you've got two otherwise equivalent transactions that spend the same input(s), keep the one with the most fees.

Right now, the rule is "keep the first one you see."

The bag of worms comes into play if you've got two transactions that spend the same input(s) that, for example, look like this:

Transaction 1:  final transaction (can go into a block RIGHT NOW) that has a fee of 0.005 BTC
Transaction 2:  transaction that won't be final for 3 days that has a fee of 0.1 BTC

Should a miner put Transaction 1 into the block they're mining and take the smaller fee now, or not include it, hoping that nobody else mines Transaction 1 in the next 3 days so maybe they can mine Transaction 2 and get the bigger fee?

I'm not an expert in game theory, but I believe the winning strategy in the above situation, assuming everybody knows about both transactions, is to mine Transaction 1 right away (any economists reading who know a lot more about game theory than I do?).

That suggests the rules for transactions that spend the same inputs aught to be:

1. If you have two free transactions, keep the first one you see.
2. If you have a free and a fee-paying transaction, always keep the fee-paying one.
3. If you have two final, fee-paying transactions, keep/mine/relay the one with the higher fee.
4. If one or both of the transactions is non-final, keep/relay the one that will become final first.

[blueadept]

FWIW, I completely agree with Gavin on this. As long as a transaction is valid, the rules should be ones that maximize miner profit so that miners play by the rules and transaction participants can know what to expect. If there's a conflict between rules and miner profit, transaction acceptance is not deterministic because you don't know whether a miner is playing by the official rules or the profit maximizing rules.

My concern is that we need to make sure people know not to trust unconfirmed non-multisignature transactions. If even some miners are taking final transactions in their memory pools and replacing them with double spend transactions having bigger fees, transactions have a significant chance of being reversed before they're confirmed.

This further cements Bitcoin's position as a settlement network, creating the necessity to create clearing mechanisms on top of it as recently mentioned by genjix. But enabling transaction replacement in this way can also make it easier to create such clearing mechanisms.

[etotheipi]

FWIW, I completely agree with Gavin on this. As long as a transaction is valid, the rules should be ones that maximize miner profit so that miners play by the rules and transaction participants can know what to expect. If there's a conflict between rules and miner profit, transaction acceptance is not deterministic because you don't know whether a miner is playing by the official rules or the profit maximizing rules.

My concern is that we need to make sure people know not to trust unconfirmed non-multisignature transactions. If even some miners are taking final transactions in their memory pools and replacing them with double spend transactions having bigger fees, transactions have a significant chance of being reversed before they're confirmed.

This further cements Bitcoin's position as a settlement network, creating the necessity to create clearing mechanisms on top of it as recently mentioned by genjix. But enabling transaction replacement in this way can also make it easier to create such clearing mechanisms.

I am swayed.  Otherwise we're trying to swim upstream.  I just didn't like the idea that anyone can replace their not-in-the-blockchain-yet tx (final or not) just by submitting one with a bigger fee.  It seems to make the zero-conf tx even more useless than before.  On the other hand, maybe it's a good idea to do that and cement zero-conf tx as totally useless outside of the pure-trust tx (at least before, the person has to inject a replacement into the blockchain, not just broadcast it). 

[jevon]

3. If you have two final, fee-paying transactions, keep/mine/relay the one with the higher fee.

It's fine and good to require ascending fees for lockTime transactions, which are explicitly replaceable...

It would be terrible for final transactions! Final transactions are explicitly not supposed to be replaced. Replacement is fraudulent double-spending! Sure, some miners might cheat, but I think most would be honest for smaller POS sized payments. As long as at least some are honest, the double-spender has a substantial chance that the double-spend won't work and he'll end up paying, which breaks most theft cases because of the discount at fencing goods. Only if you throw in the towel and say miners should intentionally side with double-spenders will they have 100% chance of getting away without paying.

The whole point of the Bitcoin network is supposed to be to witness which transactions came first. Don't give up on that!

[blueadept]

It would be terrible for final transactions! Final transactions are explicitly not supposed to be replaced. Replacement is fraudulent double-spending! Sure, some miners might cheat, but I think most would be honest for smaller POS sized payments. As long as at least some are honest, the double-spender has a substantial chance that the double-spend won't work and he'll end up paying, which breaks most theft cases because of the discount at fencing goods. Only if you throw in the towel and say miners should intentionally side with double-spenders will they have 100% chance of getting away without paying.

The whole point of the Bitcoin network is supposed to be to witness which transactions came first. Don't give up on that!

The problem with this isn't apparent now but will be when fees make up most of miner income. When miners compete for every last Satoshi, any rules that aren't enforced as part of block validity will be bent for profit maximization. That means that anything in the memory pool is fair game. It's important to teach people not to rely on zero confirmation transactions before there's a whole lot of infrastructure based on trusting them built while miners can still afford to be benevolent.

[jevon]

Then instead make it in the miner's interest to stay with the first transaction:

When the client software receives a double-spend, display the first transaction as bad and immediately spend its entire amount to fee. It's in the miner's interest to include the first transaction in order to include the all-fee transaction after it.

Double spending is then pointless. You lose your money and still owe the merchant, and it's your fault for spoiling the payment with a double spend.

If the receiver is not online at the time, then he's not a double spending target because he's not even online to act on 0-confirmation transactions.

In the future, this is what any game theory understanding 0-confirmation accepting merchant will do to deter double spend fraud if there are any dishonest miners.

[kjj]

Then instead make it in the miner's interest to stay with the first transaction:

When the client software receives a double-spend, display the first transaction as bad and immediately spend its entire amount to fee. It's in the miner's interest to include the first transaction in order to include the all-fee transaction after it.

Double spending is then pointless. You lose your money and still owe the merchant, and it's your fault for spoiling the payment with a double spend.

If the receiver is not online at the time, then he's not a double spending target because he's not even online to act on 0-confirmation transactions.

In the future, this is what any game theory understanding 0-confirmation accepting merchant will do to deter double spend fraud if there are any dishonest miners.

How do you do the bolded part?  And how do you know which is "first"?

[jevon]

How do you do the bolded part?  And how do you know which is "first"?

The first transaction is the one paying to the recipient (merchant). This is the recipient doing this action.

Recipient receives tx 1, paying to him.
Recipient receives tx 2, which is a double spend of tx 1 paying back to double spender.
Recipient spends tx 1 to all fee.

Including tx 1 to get to the all fee transaction after it is now worth more fee than tx 2.

[kjj]

How do you do the bolded part?  And how do you know which is "first"?

The first transaction is the one paying to the recipient (merchant). This is the recipient doing this action.

Recipient receives tx 1, paying to him.
Recipient receives tx 2, which is a double spend of tx 1 paying back to double spender.
Recipient spends tx 1 to all fee.

The problem with a double spend is that no one really knows which is the "right" spend.  Whichever one ends up in the next block is (by definition) the right one, as far as bitcoin is concerned.  If that is the one sent to the unknown person that you are calling "recipient", wouldn't the "recipient" be better off keeping the money?

Or are you saying that the miners will be willing to ignore tx2 if they see that (tx1+all_fee) goes into their pocket, unlike tx2, which does not?

[jevon]

Or are you saying that the miners will be willing to ignore tx2 if they see that (tx1+all_fee) goes into their pocket, unlike tx2, which does not?

Gavin and blueadept are speculating that could happen in the future. This is all in reply to that concern that there will be dishonest miners that will take the transaction with the largest fee, even if it's wasn't first.

The concern was that a double spender could make his double spend (tx2) with a larger fee to bribe the miners.

In that case, merchants would respond by increasing the fee on tx1 (tx1->all fee) so the double spender won't be rewarded. Double spending would always fail to get any money back, so it would be deterred.

The point is, if there are dishonest miners that can be bribed with fee to take a double spend, then merchants would not sit by while they get ripped off by double spenders.

If that is the one sent to the unknown person that you are calling "recipient", wouldn't the "recipient" be better off keeping the money?

No, in this scenario the double spender is bribing miners with fee so he would always win. The merchant can play that game too and raise his fee too. Then the first spend has the highest fee and is again in the miner's best interest.

[blueadept]

jevon, I see what you're saying and agree with you, though it becomes a balancing act between profit maximization and DoS protection on the part of the miners. To add this kind of dependency resolution for multiple versions of a transaction without any change to the protocol to send sets of transactions in a single message will require miners to be vigilant about resource use consumed by storing multiple versions of a transaction to be able to tell what future transactions depend on them and how they affect fees.

This also doesn't change Gavin's proposed rules. It just makes them friendlier to accepting zero confirmation transactions in certain circumstances.

By the way, this still doesn't work all the time. If a sender sends to a 2 of 2 multisig address for escrow for goods or services that get delivered before a block is found, the sender can double spend without worrying that the cosigner can up the fees to stop him.

ETA: What Gavin and I are saying is that there will be selfish miners. Whether they're considered dishonest or not depends on whether the official rules align with incentives or ignore them. Right now they're mostly benevolent in regards to transactions in the memory pool because the overwhelming majority of their revenue comes from the block subsidy.

[Mike Hearn]

Can we please hold off on yet more rule changes? The BIP 16 effort drained months of effort away from other things, like making the standard client most people use scale. Attempting to change the rules yet again will be another huge, controversial effort at a time when what Bitcoin needs is stability and basic improvements to the user experience.

The proposed change would basically invalidate Satoshis reasoning around how small payments can be accepted for things like vending machines. It would also make it impossible for lightweight clients (ie, most users in future) to accept payments until a block had been mined, basically killing the whole concept of a usable currency. The point of having a mesh in which lots of nodes don't mine and lots do is to make it difficult to distribute double spends into the memory pool like that. Even if some miners change to the proposed for-profit rule, getting your transactions to such miners such that you reliably win should be difficult.

I explicitly disagree the rules should be designed to maximize miner profits against all other considerations. If you truly believe that, just make the software automatically measure the size of the fee increase then automatically fork the block chain so it can be overridden. Turn every miner into a "blackhat miner" that slows down transactions  and rewrites the block chain if there's enough profit to do so. Or, we could assume that miners also mine as part of expressing support for a particular set of ideas and rules - not just for profit. As most mining pools already do today.

Please let's just try and use the tools Satoshi left us with rather than constantly trying to change them. I'm strongly against trying to "improve" parts of the protocol that we've barely even begun to understand yet, like nLockTime and tx replacement.

[Gavin Andresen]

Please let's just try and use the tools Satoshi left us with rather than constantly trying to change them. I'm strongly against trying to "improve" parts of the protocol that we've barely even begun to understand yet, like nLockTime and tx replacement.

Some of the big mining pool operators are already going their own way, experimenting with new types of transactions (e.g. deepbit and eligius), so I suspect the answer to your question "Can 'we' please hold off..." is "No."

Discussions like this one are how I start to understand parts of the protocol like nLockTime and transaction replacement better.


As for BIP 16 draining effort away from making the standard client scale:  I've been saying for a year that my top priorities are network stability and wallet security. Making the standard client scale is, in my opinion, a lower priority; I don't want a super-scalable client with a 4-minute blockchain initial block sync time if it is vulnerable to the user clicking one wrong link and getting infected by a trojan wallet stealer.

[Mike Hearn]

Over time hopefully p2pool will reduce the power of individual pools to impact rule changes, as most miners will just trust the de-facto judgement of the project maintainers in such matters (ie, you). And the biggest pools are all run by friendly, involved operators who aren't driven purely by profit maximization.

The proposed change would make it impossible for users on lightweight clients to do anything with a zero-confirm transaction, even for small amounts. That would make something as trivial as paying for a beer for Bitcoin largely impossible, which seems like a pretty basic litmus test of "does this currency actually work?".

What's more, allowing such double spends would mean that logically it should be extended to already confirmed transactions too. Once merchants all start waiting for 1 block, or 2 blocks, or whatever, large numbers of users would simply start broadcasting double spends after 3 blocks. Once you have enough replacements for transactions 3 blocks back, if all miners use the "profit by default" rule, everyone would start mining on a 3-block back fork automatically in order to claim the higher fees.

The system would become largely useless.