How to get rid of doubled transactions

[djeZo]

I am running a bitcoin payable service and for some customers some doubled transactions stacked up. Now we already know that these will never be confirmed, but how can I get rid of them, because they do affect current balance of customers.

I need solution that will not take whole day (like downloading whole blockchain). Is restart of bitcoin-qt wallet enough?

[raveneye]

The problem looks like it is related to an "orphaned transaction" caused by a mutated transaction.

Transaction TX1 was issued from address A1 to A2+OutputAddress.
Transaction TX2 was issued from address A2(TX1-000) to A3+OutputAddress before TX1 was confirmed.

TX1 got mutated to TX1a, which invalidated and orphaned TX2, because TX2's input transaction no longer existed.
TX1 and TX1a are showing up in blockchain.info as a duplicate spend of each other.  TX2 does not appear at all on blockchain.info.

[djeZo]

Restart of bitcoin-qt did not help. Some files need to be deleted, but I don't know which ones...

[maaku]

Do not delete files.

Look up "pywallet" in the forum search. There are other people who have had this problem and dealt with it.

[djeZo]

How long does the whole process take? Like redownloading whole blockchain? Half a day?  

Actually, this process f*cked up everything. All accounts are messed up now, there is missing balance... Hope I can salvage this :/

[djeZo]

I was able to restore previous wallet state. Luckily.

Looks like we will have to wait for solution from either bitcoin devs or 3rd party app that could clean up these double transactions due to recent attacks.

[djeZo]

So far no solution for small merchants using bitcoin-qt?

Is there a way to manually remove each doubled transaction?

[meanig]

So far no solution for small merchants using bitcoin-qt?

Is there a way to manually remove each doubled transaction?

This is a patch that stops the qt wallet from spending change outputs with less than 2 confirms. It might help.

http://www.reddit.com/r/Bitcoin/comments/1xm49o/due_to_active_malleable_transaction_relayers_it/

[DeathAndTaxes]

So far no solution for small merchants using bitcoin-qt?

Is there a way to manually remove each doubled transaction?

Why do you need to remove them?
Why are you giving credit to users for 0 confirm txs?

They will drop off automatically when they are deleted from the memory pool to make room for newer transactions.

If you absolutely need to delete them you can use a tool like pywallet to manually remove the tx from your wallet.  Still before you do that you should question why do you need to remove them.  Think of 0-confirm tx as unpaid orders.  Do you need to delete unpaid orders? 

[djeZo]

The only issue is that amounts in accounts are reduced as long as these unconfirmed transactions are on. I only want to release this "locked" amount back to customers, so they can spend it.

I already use min 6 conf. (do not consider spending coins if they are not min 6 conf).

Basically what I need is a tool that would allow me to manually remove these long unconfirmed transactions - they do not get removed on their own - at least they didn't so far (except if bitcoin-qt restart prolongs removal of them).

And the issue are not incoming transactions, but outgoing.

An example is; customer A loads on 1 BTC. He then pays for rig B amount 0.5 BTC. Because of the attacks, there are 2 transactions created, each 0.5 BTC (total 1 BTC). One gets confirmed, another one is in state of never being confirmed. But side effect is that his account would show final balance 0 - he is unable to spend the remaining 0.5 BTC for anything else. That is the issue I would like to fix. I hope you understand me now.

[DeathAndTaxes]

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

[raveneye]

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

The problem is a fault in the Bitcoin client itself.  Despite everyone's instance that the problem is outside of that, there is still a fault in the bitcoin client.

If 2 transactions are made from the same wallet on the same block and the client chains them (i.e. uses the outputs of the first transaction for the second transaction) and the first transaction is mutated by the network, then the second transaction becomes invalided and rejected by the network.  The client does not clean up the second transaction and it's coins become un-spendable without a lot of technical knowledge of how to fix the corrupted wallet.

Small time users that only do a few transactions a day are not seeing the problem.  Large users like mtgox and coinedup that do many transactions from the same wallet in the same block on a regular basis are getting hammered by this fault.

[djeZo]

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

I am not, this is what bitcoin-qt does by default (I use RPC API).

[meanig]

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

The problem is a fault in the Bitcoin client itself.  Despite everyone's instance that the problem is outside of that, there is still a fault in the bitcoin client.

If 2 transactions are made from the same wallet on the same block and the client chains them (i.e. uses the outputs of the first transaction for the second transaction) and the first transaction is mutated by the network, then the second transaction becomes invalided and rejected by the network.  The client does not clean up the second transaction and it's coins become un-spendable without a lot of technical knowledge of how to fix the corrupted wallet.

Small time users that only do a few transactions a day are not seeing the problem.  Large users like mtgox and coinedup that do many transactions from the same wallet in the same block on a regular basis are getting hammered by this fault.

But a simple patch of not spending unconfirmed change outputs would work around this problem?

[DeathAndTaxes]

Why are you deducting the account based on 0-confirm transactions?
Why are you not either
a) deducting the account based on the actual request
OR
b) deducting the account based on the 1-confirm transactions

There is a tool called pywallet which allows you to manually modify your wallet.  You can permanently corrupt your wallet if you don't understand what you are doing.   However before you start using pywallet maybe you should take a step back and say "am I pulling a MtGox here" by using 0-confirm tx to handle your accounting.

I am not, this is what bitcoin-qt does by default (I use RPC API).

Sorry about that.  Yeah the built in qt accounts system is well ... garbage for reasons beyond this.   Sorry I assumed when you said "account" you were talking about your own internal accounting.  That would be a problem.

So if you want to use QT built in "accounts" your only option would be to either wait or use a tool like pywallet. The amount of time before they fall from the memory pool is not deterministic, it depends on your nodes resources and the tx volume so it could be hours or weeks.  Obviously some patch for handling this better will be needed but in the interim those are your only options.  As a heads up if you delete the "copy" prior to one of them being confirmed, it probably will come right back, this is because other nodes will still be relaying it.  If you delete the unconfirmed copy AFTER the other one is confirmed the duplicate should not be relayed anymore (as all nodes will see the unconfirmed one as a double spend).

The other option (and in the long run I think you will find this works better) is to scrap the QT "accounts" system and just use the client/wallet as a transaction engine.  In your own database you would track user balances, payments, withdraws, etc, and just query (using RPC API) the client for low level data.  This provides more scalability and flexibility.

For example you can assign a user a deposit address by calling getnewaddress and recording the result in your database linked to the particular user.  When a payment is received at that address you can look it up and know which user to assign it to.  To make detecting transactions and blocks, bitcoind supports the ability to assign a callback function (could be an external program, web call, etc).  It will pass the tx or block id to your system when an inbound tx (or confirmation) or new block is detected.

[DeathAndTaxes]

But a simple patch of not spending unconfirmed change outputs would work around this problem?

That alone won't help, the issue is that the doubled transaction affects the QT clients "account" recording.  The sender isn't spending unconfirmed change, the tx is being modified (by those trying to prove a point) between the sender and receiver.

One relatively simple option would be to patch the QT client to DELETE unconfirmed mutated transactions (but not traditional double spends) once the confirmed version has one or more confirmations.  If the developer didn't want to delete it, a flag could be added "duplicate" instead which by default would hide the double from transaction display and the "accounts" balance.

An even better patch would be "smart" enough to realize that in a doubled transaction (same inputs, same outputs, different hash) only one copy of the transaction can ever end up confirmed.  Since they both have the same value it would never even consider the second copy.

So
Copy1 receive - unconfirmed balance increased (or decreased for spend)
Copy2 receieved - no change (maybe a special symbol indicating a double and by default doubles aren't even shown

If copy 1 ends up confirmed there is no change other than the confirmed balance increases and unconfirmed balance decreases.
If copy 2 ends up confirmed, copy1 gets the "double" flag, copy2 "double" flag is removed.  The user sees no change in total balance, and the visible tx changes.  The only change is the hash so to the user they simply see the tx hash change.

[djeZo]

Yes, this is what I need. Didn't know BTC's internal accouting was garbage... So far it worked without any issues.

[DeathAndTaxes]

Yes, this is what I need. Didn't know BTC's internal accouting was garbage... So far it worked without any issues.

Well that is just my opinion.  I know Gavin likes it.  For a personal wallet it might be fine but I think you will find that building a custom datalayer which connects to bitcoind to be a better long term solution.  If nothing else it provides better flexibility and scalability.

I guess it is kinda like quickbooks (or other accounting software) vs online banking.  It is far easier to use accounting software and then link to online banking to pull the incoming and outgoing txs then it is to somehow try to use online banking for accounting.

[raveneye]

But a simple patch of not spending unconfirmed change outputs would work around this problem?

The problem is that the client does this by default.  For example, look at these 2 transactions on block 285303:

http://blockexplorer.com/tx/359e7c0c45f8633e3b85e0c5de79c8c0e7600fc1b4e17ba67e7bf4541379aa2a#o0
Inputs:
txid: 290fd5edf00e...:1 amount: 0.0035285 from address:   1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
Outputs:
0 23885ee7630b... amount: 0.00290698 1LuckyY9fRzcJre7aou7ZhWVXktxjjBb9S
1 3393e29876b7... amount: 0.00052152 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ

Notice that the client has split the amount into 2 amounts, one to the payee (1LuckyY...) for the desired amount and the other which credits the remaining balance back to the same account (1FSRTDG...).  This is because each transaction must use the entire balance of the inputs in the outputs.
Then this transactions is logged on the same block that uses the output from the first transaction.

http://blockexplorer.com/tx/3393e29876b7090e439762a0c3bf359818821bfb2c44317e25c144e9b2cdcab9#i74103801
Inputs:
359e7c0c45f8...:1 amount: 0.00052152 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
00c8195a8fea...:1 amount: 0.00079582 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
2b473eb3853d...:1 amount: 0.0003604 1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ
Outputs:
0 444f9a598a6a... 0.00145349   1LuckyR1fFHEsXYyx5QK4UFzv3PEAepPMK
1 07c00c8a3ada... 0.00012425   1FSRTDG5nsMn2CHL472iZZjjYyWw74G6hQ

If the first transaction gets mutated and the hash is changed from 359e7c0c45f8... into something else then the 2nd transaction will become orphaned, corrupting the wallet and preventing the coins from being spent until it is cleaned up.

[CIYAM]

I guess it is kinda like quickbooks (or other accounting software) vs online banking.  It is far easier to use accounting software and then link to online banking to pull the incoming and outgoing txs then it is to somehow try to use online banking for accounting.

Actually as a user of traditional accounting software I can assure you that "it is nothing like that" and why I (and many others) don't even *touch* the Bitcoin "accounts" functionality.

Although the Bitcoin devs are amazing in what they can do - they don't seem to understand "accounting" much at all.