[RFC] When wallets conflict with the block chain

[Pieter Wuille]

Edit: see pull request described here: http://forum.bitcoin.org/index.php?topic=7330.0


It is a situation that is currently hard to reach, but nonetheless possible when moving wallet.dat files around, and can potentially leave you with a permanently corrupted wallet.

Consider this scenario:

• Alice receives a 50BTC coin
• Alice creates backup of her wallet
• Alice goes offline
• Bob imports A's wallet
  
• Bob sends 20BTC to Carol (sending 30BTC change to a reserve key)
• Alice comes online before Bob's transaction ends up in a block, or somehow does not receive the block it is in yet
• Alice sends 10BTC to Ted, using the 50BTC coin as input

The network already knows about Bob's spending of the 50BTC coin, so it will not accept Alice's transaction anymore. Alice should be using the 30BTC change coin, but does not know about this yet. The result is that Alice has a transaction in her wallet which will never be accepted by the network, but Alice still assumes it is valid. Even worse, Alice will try to use the 40BTC change of that transaction in further transactions as well, and those transactions (and everything resulting from its change) won't be accepted by the network either.

You may argue that copying wallet.dat files around is always a dangerous business, but I don't like the idea that being unaware of a transaction happening using one of your own keys can potentially break things.

I'm working on a patch to detect cases where the blockchain conflicts with a wallet, but the question is: what to do when it happens?

Some possibilities:

• Edge case, don't bother
• Silently drop the offending transaction from the wallet (very easy)
• At least give a warning message in the GUI (easy, but what to do in bitcoind?)
• Mark the transaction somehow as inactive, and show this in the GUI and RPC calls (needs some changes, including update to wallet.dat format if you want keep these permanently)
• Try to recreate the transaction, using inputs that do exist (after user confirmation in GUI, or using special RPC call)
• Automatically try to recreate the transactions (probably too dangerous)

[Matt Corallo]

Bitcoin definitely needs to be as bulletproof as possible when it comes to UI (1st rule: assume users are trying to break your program).  I don't think bitcoin should try to recreate the transactions or resend now that the balance has changed.  I'd say GUI should notify the user and bitcoind just silently drops the TX (or maybe marks it as rejected in place of unconfirmed?)

[Mike Hearn]

This is a specific case of a more general problem, which is it's impossible to know if your tx was accepted by the network at all. Nodes just silently drop transactions they don't accept.

A much bigger instance of this problem is setting fees. Imagine if the minimum tx fee is lowered from Ⓑ0.01 to Ⓑ0.001. Rollouts of new software versions are not synchronized. You upgrade your software and send a transaction with the new rules, but by chance you connected only to nodes that had not upgraded yet. Your tx is rejected through no fault of your own and now your wallet is useless. Useless or corrupted wallets are probably the worst failure mode BitCoin has.

The "fix" for this so far has been to try and synchronize changes to the rulesets by using block chain indexes, and hoping everyone has upgraded by the flag day. But it'd be better if a new message was added to the protocol. If you send a tx message and the receiving node doesn't accept it, it could send back a txfail message with an error code and/or English explanation of why it was deemed unacceptable. If all connected nodes replied with a txfail message, the transaction would not be committed to the wallet and show as failed in the UI.

[Matt Corallo]

The "fix" for this so far has been to try and synchronize changes to the rulesets by using block chain indexes, and hoping everyone has upgraded by the flag day. But it'd be better if a new message was added to the protocol. If you send a tx message and the receiving node doesn't accept it, it could send back a txfail message with an error code and/or English explanation of why it was deemed unacceptable. If all connected nodes replied with a txfail message, the transaction would not be committed to the wallet and show as failed in the UI.

I disagree that that is necessary.  IMHO, people should be aware that any tx which has 0 confirmations is perfectly capable of failing.  It is much easier to notify a user that their tx did not make it into the blockchain for whatever reason.

To your specific example of TX fee changes, eventually the tx would go through as the tx is rebroadcasted and eventually it would be sent to a new node which makes its way to a miner.  In any case, the tx fee issue is slightly different than simply not being accepted due to conflicts.  IMHO, the ideas that have been floating around for tx fee increase when a tx fails after x period of time should be considered as it would fix your specific problem and it in combination with simple fails after conficts could fix the rejected tx issue without needing to modify the protocol.

[Mike Hearn]

The problem is not that people think 0/unconfirmed txns can be spent. The client makes that clear. The problem is if you spend half your wallet and it never confirms for some reason, you can't unspend them and try again. There's no such thing as "failure", just permanent limbo. Losing coins is very bad for both the owner and BitCoin globally.

It's not just fees and wallet movements that can cause this, but unusual scripts etc.

[theymos]

I don't think a new message is necessary. Bitcoin should allow you to unspend 0-confirmation transactions, sending a conflicting transaction back to itself. Maybe it can do this automatically after 2 weeks (or whatever), or immediately as soon as a double-spending transaction is confirmed in the block chain.

[error]

I don't think a new message is necessary. Bitcoin should allow you to unspend 0-confirmation transactions, sending a conflicting transaction back to itself. Maybe it can do this automatically after 2 weeks (or whatever), or immediately as soon as a double-spending transaction is confirmed in the block chain.

Two hours is more what I'd like to see.

[theymos]

Two hours is more what I'd like to see.

It's not safe to assume a transaction has been completely rejected after just 2 hours. Maybe it was just lost. You should wait at least a day or two.

[Matt Corallo]

The problem is not that people think 0/unconfirmed txns can be spent. The client makes that clear. The problem is if you spend half your wallet and it never confirms for some reason, you can't unspend them and try again. There's no such thing as "failure", just permanent limbo. Losing coins is very bad for both the owner and BitCoin globally.

It's not just fees and wallet movements that can cause this, but unusual scripts etc.

Yes its not technically "failure", but for the purpose of the user its very similar.  As theymos said, bitcoin could "unspend" a tx by creating a conflicting one  then mark a tx as rejected.  Also, for the purpose of the user, you can call coins in permanent limbo a failed tx. 

[Pieter Wuille]

I don't think a new message is necessary. Bitcoin should allow you to unspend 0-confirmation transactions, sending a conflicting transaction back to itself. Maybe it can do this automatically after 2 weeks (or whatever), or immediately as soon as a double-spending transaction is confirmed in the block chain.

Why another (intentionally) conflicting transaction? Just mark the unconfirmed transaction inactive...

[theymos]

Why another (intentionally) conflicting transaction? Just mark the unconfirmed transaction inactive...

It could end up going through, which might cause you to pay twice.

[Mike Hearn]

I don't see the problem with adding a new message. Old clients will ignore it and act exactly as they do today. Newer clients can understand that if all nodes replied with txfail, there is no need to wait two weeks. The transaction can be unspent immediately and the user informed that they need to take action.

[Matt Corallo]

I don't see the problem with adding a new message. Old clients will ignore it and act exactly as they do today. Newer clients can understand that if all nodes replied with txfail, there is no need to wait two weeks. The transaction can be unspent immediately and the user informed that they need to take action.

I just don't see a clear need for it and I don't see why we should add a message if there is no need for it.  Also, in some cases (such as the connected to out-of-date client who is rejecting txes based on old rules situation) rejecting a tx right away might not be the best way to do things.  It might be better to wait around a couple hours and hope you get connected to an updated client and resubmit the tx.

[Pieter Wuille]

Why another (intentionally) conflicting transaction? Just mark the unconfirmed transaction inactive...

It could end up going through, which might cause you to pay twice.

Maybe, but you'll need to be careful.

• If there was a problem with the inputs (already spent eg.), the correction tx won't be accepted either (and you definitely don't want a correction tx for a correction tx).
• If the problem was with the output (eg. non-standard script), a correction tx that does a simple spend-to-self may get you certainty, by seeing the correction tx end up in a block sooner than the original
• If the original tx never reached the network, a retransmit is the best thing to do, but a correction tx won't harm

So, implementation-wise, i suppose it should be a kind of spend-to-self tx, which is by default not assumed to be active, until it is seen in a block.

[caveden]

A much bigger instance of this problem is setting fees. Imagine if the minimum tx fee is lowered from Ⓑ0.01 to Ⓑ0.001. Rollouts of new software versions are not synchronized. You upgrade your software and send a transaction with the new rules, but by chance you connected only to nodes that had not upgraded yet. Your tx is rejected through no fault of your own and now your wallet is useless.

You mean nodes won't forward a transaction that doesn't suit their own fee policies? This is bad if it's the case... transactions should at least be forwarded, in spite of fees.

[Matt Corallo]

A much bigger instance of this problem is setting fees. Imagine if the minimum tx fee is lowered from Ⓑ0.01 to Ⓑ0.001. Rollouts of new software versions are not synchronized. You upgrade your software and send a transaction with the new rules, but by chance you connected only to nodes that had not upgraded yet. Your tx is rejected through no fault of your own and now your wallet is useless.

You mean nodes won't forward a transaction that doesn't suit their own fee policies? This is bad if it's the case... transactions should at least be forwarded, in spite of fees.

That is the current implementation.  It was done to prevent DDoS attacks.  If I can go sending 10000 txes/sec to nodes and they have to forward all of them. Anyone can easily DDoS the network very quickly.

[caveden]

Bitcoin should allow you to unspend 0-confirmation transactions, sending a conflicting transaction back to itself. Maybe it can do this automatically after 2 weeks (or whatever), or immediately as soon as a double-spending transaction is confirmed in the block chain.

I agree with that (although I also agree with [mike] that a failure message wouldn't harm either).
I just don't think there should be anything automatic... reclaiming should be always manual.

[caveden]

That is the current implementation.  It was done to prevent DDoS attacks.  If I can go sending 10000 txes/sec to nodes and they have to forward all of them. Anyone can easily DDoS the network very quickly.

You mean valid transactions? I agree invalid transactions don't need to be forwarded, I just don't like this mandatory fee policy.

But I see the point... somebody could send millions of 0,01μBTC with no fees, flooding the network....

Eventually something better than a mandatory fee policy could be implemented, don't you think? Like rejecting "too fast" transaction series, for example... [mike] suggestion comes handy there. A node that "suspects" one of its peers is flooding the network could just send him an error message with a proper code or text indicating something like "I think you're a spammer, won't accept more than one transaction of yours per second".

[randomguy7]

We could enforce some proof-of-work to be made before txs are processed by the nodes (kinda like hashmail or whatever its name was).

[theymos]

If there was a problem with the inputs (already spent eg.), the correction tx won't be accepted either (and you definitely don't want a correction tx for a correction tx).

Bitcoin can cancel one input per day until one of the cancellations goes through.