[RFC] When wallets conflict with the block chain

[Pieter Wuille]

I finally have this implemented, see https://github.com/sipa/bitcoin/tree/rejectedtx

Initial testing shows that it can detect such conflicts, and store/restore correctly in wallet.dat.

The GUI will show rejected transactions, the RPC interface will hide them.

It still needs a way for permanently removing rejected transactions (default after X confirmations of the conflicting tx?), and a way to query them from RPC.

[1713540016]

1713540016

[1713540016]

1713540016

[1713540016]

1713540016

[Vladimir]

If anyone needs a test case for 'limbo transactions' simply send some money when your client is not up to date completely with the block chain. You will get a limbo transaction which at the moment basically the same as money lost and it is also unclear what happens with all the "change" involved which can be rather significant.

Seems like a serious bug to me.

A friend of mine has a a few such limbo transactions and no idea how to fix this. I also simply do not know how to help here.

Any suggestions?

[ByteCoin]

I think the transaction cancelling/superseding issue is one of the most technically complex outstanding issue needing to be addressed.

Bitcoin can cancel one input per day until one of the cancellations goes through.

Are you saying that this is something that already happens in the code or is this something you are proposing to be implemented to solve the problem in future?
If it is in the code, could you please indicate where?

Could you please also be a little bit more explicit as to how the input cancelling would solve the problem?
And why only cancel one per day if more are eligible for cancellation?

Thanks,

ByteCoin

[bji]

I think the transaction cancelling/superseding issue is one of the most technically complex outstanding issue needing to be addressed.

Bitcoin can cancel one input per day until one of the cancellations goes through.

Are you saying that this is something that already happens in the code or is this something you are proposing to be implemented to solve the problem in future?
If it is in the code, could you please indicate where?

Could you please also be a little bit more explicit as to how the input cancelling would solve the problem?
And why only cancel one per day if more are eligible for cancellation?

Thanks,

ByteCoin

Transaction handling does seem like a weak point of the bitcoin system.  I think that the design should have included:

1) A proof of work in the transaction to limit the rate at which hosts can create transactions (and with a work function that is scaled to be of a constant difficulty similarly to how the block hash work function is scaled automatically).  Large, powerful hosts still would possibly be able to get around this but it would cost them (in electricity).

2) Transactions have a built-in expiration of 1 hour.  Blocks would be rejected by the client if they included transactions that should have expired before the block was created.  In this way, you can be sure that if you see a block which came after that hour expiration period, and your transaction still is not in the block chain, that it is gone and will never be processed.

[BitterTea]

1) A proof of work in the transaction to limit the rate at which hosts can create transactions (and with a work function that is scaled to be of a constant difficulty similarly to how the block hash work function is scaled automatically).  Large, powerful hosts still would possibly be able to get around this but it would cost them (in electricity).

Perhaps there could be a transaction hash target, similar to that of the block. In order to be relayed, the transaction's hash must be below the target. I'm not sure how you'd get this difficulty to automatically scale with increases in processing power.

[theymos]

Are you saying that this is something that already happens in the code or is this something you are proposing to be implemented to solve the problem in future?

It's something I'm proposing.

Could you please also be a little bit more explicit as to how the input cancelling would solve the problem?

As a solution to the general problem of transactions that will never get into a block, sending a conflicting transaction (a transaction sharing one of the inputs to the "stuck" transaction) is guaranteed to cancel the old transaction if the new transaction gets into a block (and the normal SIGHASH mode is used). You can't just mark the old transaction as "canceled" without creating a conflicting transaction, since the old transaction might go through eventually, which would often be bad.

Sending just one of the stuck inputs at a time increases the chances that your new transaction will go through.

This isn't necessary if you know the transaction is already conflicting with something in the block chain, but often a transaction will not be accepted due to fee problems, and lightweight clients won't know about already-conflicting transactions.

And why only cancel one per day if more are eligible for cancellation?

There isn't a major reason why you couldn't do that, though if you sent all the transactions at once, then your peers would probably reject all but the first. Also, it seems dangerous to have many conflicting transactions floating around: if an attacker gets control of the network, they could do some extra things to transactions based on yours.

[Mike Hearn]

Incidentally, BitCoinJ now implements something like this. Transactions that conflict with the block chain are considered "dead" and an API user can find out about them by handling an event.

There are still a few edge cases that aren't handled quite correctly, like the case of a previously dead transaction becoming live again due to a block chain re-org (it's handled but you are not notified via the event listeners). It's a very complex set of edge cases on top of edge cases.

[TiagoTiago]

What happens if you start the client with the --rescan flag or delete the blockchain and let the client redownload it?

[Pieter Wuille]

What happens if you start the client with the --rescan flag or delete the blockchain and let the client redownload it?

-rescan only adds transactions to your wallet that were missing (and since 0.3.23, also updates them if they were incorrectly marked unconfirmed).

Deleting the block chain is something that should never be needed since 0.3.23.

[netrin]

Sipa just pointed me to this thread after I posted http://forum.bitcoin.org/index.php?topic=26269.0

The transaction in question is now over a month old. I have two versions of the same send in my client (neither send happened). The first send has 7000 confirmations sent to no (blank) address. Another send (with the same timestamp and btc amount) has 0/confirmations to an unknown address in the block chain (presumably an address of mine somewhere).

I believe the one or two transactions are invalid due to double spending from a copied wallet but it's difficult to confirm because I can not be sure which are the sending nor receiving addresses. In any case, I would hope that before 7000 blocks the 0.3.23 client would be convinced that the transaction is never going to happen.

[Gavin Andresen]

Losing double-spent transactions should probably be reported as -1/unconfirmed, -2/unconfirmed, etc -- meaning "there is a N-confirmed transaction that conflicts with this one."

-6/unconfirmed seems like the natural "this transaction ain't never gonna be confirmed, time to report it as -N/orphan."

-1/unconfirmed should be enough to stop re-transmitting it.

And -120/orphan seems like a safe time to "unspend" any inputs (let them be spent by new transactions) that weren't spent by the other, conflicting transaction.

... he says nonchalantly, knowing full well that actually making bitcoin behave that way means a lot of very careful coding and lots and lots of careful testing...

The shortcut, maybe-good-enough-for-now solution:  export all the private keys from all the messed up wallets.  Start with a clean wallet, then re-import all the private keys and let the clean-slate bitcoin figure it all out.

[TiagoTiago]

Would the transaction be removed from the transactions history or would it be marked as "REJECTED" or somthing like that?

[Pieter Wuille]

Losing double-spent transactions should probably be reported as -1/unconfirmed, -2/unconfirmed, etc -- meaning "there is a N-confirmed transaction that conflicts with this one."

I like this idea. It would need to be extended to "there is a N-confirmed transaction that conflicts with this one, or one of its dependencies", though.

-6/unconfirmed seems like the natural "this transaction ain't never gonna be confirmed, time to report it as -N/orphan."

-1/unconfirmed should be enough to stop re-transmitting it.

And -120/orphan seems like a safe time to "unspend" any inputs (let them be spent by new transactions) that weren't spent by the other, conflicting transaction.

Not sure about this. As soon as you are at -1, the chance for the transaction to ever get into the block chain is already very small. I think you should stop counting its inputs at that point already.

When -6 is reached, you could call it "orphan" or "rejected". The latter may be a bit clearer to users.

When -120 is reached, you can indeed assume that it won't confirm anymore ever. Maybe a "Remove transaction from wallet" option can be enabled at that point?

... he says nonchalantly, knowing full well that actually making bitcoin behave that way means a lot of very careful coding and lots and lots of careful testing...

Well, i've already implemented a rejection system (see http://forum.bitcoin.org/index.php?topic=7330.0 and https://github.com/bitcoin/bitcoin/pull/195). It needs rewriting for the current codebase though, but the idea above shouldn't be too hard to incorporate.

The shortcut, maybe-good-enough-for-now solution:  export all the private keys from all the messed up wallets.  Start with a clean wallet, then re-import all the private keys and let the clean-slate bitcoin figure it all out.

That works, indeed.

EDIT: I definitely think you should not count the spending caused by rejected transactions. What if you have two transactions in your wallet that spend the same input (one confirmed, one rejected with outputs to yourself). This is a possible situation. If you still count the rejected one, your balance will be wrong.

[gmaxwell]

Transaction handling does seem like a weak point of the bitcoin system.  I think that the design should have included:

1) A proof of work in the transaction to limit the rate at which hosts can create transactions (and with a work function that is scaled to be of a constant difficulty similarly to how the block hash work function is scaled automatically).  Large, powerful hosts still would possibly be able to get around this but it would cost them (in electricity).

It wouldn't have to be in the blockchain. You could hashcash complete txn to get them forwarded/mined.   But why? We have a _reusable_ proof of work system built into bitcoin:  transaction fees.

2) Transactions have a built-in expiration of 1 hour.  Blocks would be rejected by the client if they included transactions that should have expired before the block was created.  In this way, you can be sure that if you see a block which came after that hour expiration period, and your transaction still is not in the block chain, that it is gone and will never be processed.

Creates a huge to split the chain in order to kill transactions forever even after they are confirmed, even absent obviously fraudulent respend activities and just by accident:   Consider the  the possible block a txn could have gone in: it goes in,  but the chain splits and it falls onto an orphan block, now it can't ever go in, even if it wasn't respent.   It basically allows random miners to denyable create reverse&respend like theft events without even having to control the private keys for respending.

If there were something like this the horizon would need to be much longer than an hour I think.

[netrin]

The shortcut, maybe-good-enough-for-now solution:  export all the private keys from all the messed up wallets.  Start with a clean wallet, then re-import all the private keys and let the clean-slate bitcoin figure it all out.

I would have assumed that's what --rescan would do. Throw away all block references and scan every address for it's balance record in the block chain.

[Eri]

their should be a fix for this in the client itself, relying on a response from a node is not good enough. this is my take on a few different ideas I've seen in this thread and vary glad to see someone working on it!

when a transaction is initially made it should be listed as "0/Pending", each new block the client receives that doesn't have the transaction in it should increase the number. when it hits some number, for example 10 the client can allow access to the funds again but give some sort of warning that shows the transaction may still go through(though it may be worth it to simply transfer all funds to a new private key to invalidate the other transaction?).  you can combine this with the idea of the node sending a msg back if the transaction is rejected, which would allow it to immediately set the transaction to 'failed' without having to wait which would allow respending of the funds immediately.

as someone else mentioned, rescan should delete all data except private keys and public keys, rebuilding the wallet from scratch allowing us to see it as the network sees it. a plain ole export for private keys would be awesome too :3
 
i also liked the idea of having a timeout as someone mentioned, if not *in* a block that is in the process of being solved within an hour of the transactions creation all nodes would simply ignore it. making it safe to say that if its been 2 or 3 hours and new blocks have been created and received, the transaction will /never/ go through as you would have already received it or all nodes would have safely ignored it.

i think a combination of all of these for such a serious issue, even if it is rare, would not be overkill.

[TiagoTiago]

Wouldn't that create the risk that if someone send a payment before downloading the whole blockchain they'll have to send it again once it's fully downloaded?

[Eri]

the client can allow access to the funds again but give some sort of warning that shows the transaction may still go through(though it may be worth it to simply transfer all funds to a new private key to invalidate the other transaction?).

Wouldn't that create the risk that if someone send a payment before downloading the whole blockchain they'll have to send it again once it's fully downloaded?

yes and no, that would only be an issue if you wanted to invalidate the transaction automatically(and not just have the funds back). that should likely also be left up to the user and in a case like this they would know if if they are still downloading the block chain or not. as far as being automatic goes i think you can tell when you have the latest block, something to do with requesting the next block in the blockchain. which may not give you the most recent one in cases with a slow connection.. but maybe requesting it from multiple sources? do clients keep track of when the latest block was created? if not they could include a timestamp and send that with the latest block or something. it doesn't seem like a big issue.

as far as a UI goes id have 2 categories for funds, "balance"  and "pending transfers". 'pending transfers' shows the total funds in transactions we have sent out that have not been seen in a block in the block chain yet. this would make people aware of funds that haven't been accepted into the block chain yet. at the same time we can add a new tab next to 'sent' and 'received' called 'pending transfers' and include all outgoing transactions that have yet to be seen in the block chain and include a header called 'status' with a description like 'no confirmations' and have a count for how many blocks it has yet to be seen in and possibly have a check box/button for each that would allow for 'canceling' the transaction manually.