Bitcoin Forum

I'm getting some understandably upset customers at the moment.

I have five send transactions dating back as far as 21 hours that have 0 confirmations on my client, and don't even show up at http://bitcoincharts.com/bitcoin/txlist/ as unconfirmed transactions.

I have 84 other send transactions that confirmed as normal during the same time period.

I am using the official bitcoin client, version 0.6.2-beta. I have scanned the forums and haven't seen that anyone else is having the same problem over the last 24 hours. If anyone else has seen a related thread, please direct me to it.

The transactions are as follows (times are AEST):

Status: 0/unconfirmed
Date: 5/06/2012 13:32
To: 19rWnQxBtXUUxqs9CytG4xnUoQXTUqRKao
Debit: -39.67 BTC
Transaction fee: -0.0005 BTC
Net amount: -39.6705 BTC
Transaction ID: ccf585bcfed1283d11b5012c5f3f3429c5695a8d2280287c44e605223b6318b5

Status: 0/unconfirmed
Date: 5/06/2012 17:13
To: 1LMFpG2ho7bWfndFpgf2L7Le6XghjRiXpQ
Debit: -39.74 BTC
Transaction fee: -0.0005 BTC
Net amount: -39.7405 BTC
Transaction ID: ee188b9e048d4889d7adcafc2216346878197af798e96ae9ade30c85bee47e43

Status: 0/unconfirmed
Date: 5/06/2012 17:21
To: 1FYrSH6Fcqme2AyewkMpzTbszuxKLjzVXt
Debit: -164.99 BTC
Transaction fee: -0.0005 BTC
Net amount: -164.9905 BTC
Transaction ID: b1f331143fbf75f0e8ce402df64019bed7fe4f0bb02bcea31f907dcd8bf3019e

Status: 0/unconfirmed
Date: 5/06/2012 17:21
To: 1EKQxT88b3FqppUzUi6spa1q4bHj7tPFyq
Debit: -49.59 BTC
Transaction fee: -0.0005 BTC
Net amount: -49.5905 BTC
Transaction ID: 11fd5b6aed10e545136a91bcb2e3dd50eb34345896c30896c5abca3c9a304e1b

Status: 0/unconfirmed
Date: 5/06/2012 17:23
To: 1LXuMiKB1kHpLt5wXSxi2uT93LYybuebsS
Debit: -66.12 BTC
Transaction fee: -0.0005 BTC
Net amount: -66.1205 BTC
Transaction ID: 4b87953accada36255e87b2d0562473fe66838d6820dd78aa37265c0b2c5fc1c

Any help would be greatly appreciated by both myself and my customers.

Highest Regards,
Jeremy

bump

4 btc bounty for successfully helping me resolve this.

Is it possible that your internet connection/firewall is/was somehow stopping the tx's from being broadcast?

I don't know. 84 other transactions sent just fine yesterday during the same time period and 35 have sent as normal today.

If for some reason they were blocked from being broadcast, why would they not have broadcast when I came back online? If that is the case, is there any way to cancel these transactions so I have access to those coins again to resend?

I did have (just the one) tx that didn't send due to not having internet connectivity and am pretty sure I had to actually restart the Bitcoin client to get it to re-send (although maybe I was just impatient).

Hmmmm... WHen you say restart the bitcoin client do you mean starting the download of the blockchain from scratch or just closing the client and opening it again?

I have already closed and reopened the client several times and even uninstalled and reinstalled it (but with the blockchain folder still intact so it started where it left off).

I had this happen once on the android client and it cancelled the transaction when I actually deleted the blockchain files and redownloaded from scratch. I was hoping to avoid this as last time it took over 12 hours to download the blockchain from scratch, but this is the solution I am going to try if no one can come up with a quicker solution within the next few hours.

Thanks Ian!

Have you tried running it with a --rescan ?

For the tx I had trouble with all I did was exit the program and then restarted it (I think it still took a while for the tx to be re-sent from memory).


Indeed you should certainly do this before resorting to downloading the whole block chain again.

No. Can anyone explain to me how to do this? I'm on Windows 7.

Or open up a command prompt and cd to the directory with bitcoin-qt.exe and type:

bitcoin-qt --rescan

And away it goes!

Does this essentially update the balance that the app sees according to the block chain transaction history?

Thanks for that guys. I'll be trying this as soon as I get home in half an hour and doling out bounties accordingly if it works!

Hi all,

Unfortunately these solutions didn't work. I'll be redownloading the blockchain scratch tonight, it seems.

Sorry to hear that - and from a bit of searching it seems that if the issue is actually within your wallet then even reloading the entire blockchain may not fix the problem.

I found the following that might be of help in this case:

https://bitcointalk.org/index.php?action=printpage%3btopic=11331.0

Well, the easiest solution at this point would be to revert to a recent backup of your wallet (you have one, right?) and double-spend these coins by sending the entire contents of the wallet to another of your addresses.

Otherwise, you can set-up pywallet (https://bitcointalk.org/index.php?topic=34028.0) and delete the individual transactions from your wallet. It's easy to do this, luckily, but setup is a pain (lots of sitting and clicking until everything is installed).

this suggests that your client is OK.

if you reload the blockchain from scratch it will interrupt your service, and you'll get more angry customers.

just stop your bitcoind and restart it, and check if these transactions are still displayed by your client;
they might have been rejected by the network (eg double spend)

If you do end up deciding you need to reload the blockchain you might also be interested in the following (at least for testing before you end up wasting too much time):

http://eu1.bitcoincharts.com/blockchain/

(these are snapshots of the blockchain taken daily)

If I do this will it mess up the 120 or so transactions I've done since the corrupted transactions?

I've restarted it several times, to no avail.

I use a separate wallet for spend transactions and there will be no buy transactions due for 19 hours now which should be enough time to reload the blockchain, plus I have a backup in case I do need to get in earlier. :)

Note:

I searched for the 5 transactions you reported in the debug.log of my Electrum server  (it is always up).
My server has not seen these transactions; I guess they were not propagated by the network.
It could be a double spend attack, or a problem with the bitcoin client used by your customers.

why don't you ask your angry customers to provide details on the transactions they did?
are they really 5 separate customers?

If the backup contains all the private keys, a rescan should be enough for it to see transactions that were confirmed. It will see them in the chain.
What's important is that the backup contain all private keys, but not all transactions.
Keep a copy of your current wallet before doing this, of course.

I thought these transactions were initiated by Jeremy's wallet...
If they are incoming transactions then the problem is not with him, but with those trying to send the money.

These are transactions FROM me not TO me. And I am aware from bitcoincharts.com that they are not even on the network as unconfirmed. However, even with a rescan they are no longer showing in my balance.

Good point. I'll try this. Thanks!

sorry for the confusion :-)
it looks like they were rejected by the network.
check your own debug.log to see the cause (probably a blockchain reorg).

Maybe you are the victim of a doublespend ?

What should I search for?

I think the default keypool is only 100.  If you're doing this many transactions you should set the keypool size to 1500 or some other high number so your backups are at least valid for a few days.

Anyway, if over 100 of those transactions are receiving coins to new addresses (this includes any sends with change), you might lose some bitcoins.  Otherwise you should be okay with your backup.  If you've used too many addresses, I'd set up pywallet and remove the transactions from your live wallet after making a backup (without bitcoin running).

Finally, to figure out what went wrong, as ThomasV suggests, check your debug.log.  I imagine yours may be fairly dense if you are doing so many transactions.  Searching for the first 10 digits of the transaction id should get you in the right area.

search for ERROR.
on Linux, you can use grep :  
you should see lines that look like this:

(with different numbers)

you can also use the first chars of the transaction hashes in the grep command:
 

Sorry for the ignorance. This log means nothing to me. Can anyone find anything in here that would tell me what to do?

https://www.dropbox.com/s/o9cq3u68emdt4hb/debug.log

I found the following in your debug.log:


Not sure exactly what that tells us but those are the starting characters of your 5 tx's.

Also found this:


The above actually appears a number of times (so I guess it kept trying to resend these tx's unsuccessfully).

Currently trying Maged's idea of restoring a backup bitcoin folder. Less than 2 days behind, should be up to date soon and know whether I'm all sorted. :)

Damn it! Ended up with the same 5 unconfirmed transactions! Why would that be if they never ended up on the network?

I think the problem is that the tx's are known to your wallet so will just keep coming back.


Perhaps you will need to do this (and note that there is a Windows version of the tool).

The backup you used was done after the 5 bogus transactions. For that to work, you'd need to find a backup which was done before the bogus transactions, but which happens to contains all your private keys. Seeing the number of transactions you make, that might be difficult. 
You'll probably have to resort to manually deleting the transactions...


To the devs: isn't this a bug in the Satoshi client? Shouldn't the client try to rebroadcast unconfirmed transactions once in a while? If it does rebroadcast, just not fast enough for Jeremy's needs, then isn't this an enhancement issue that should be logged anyway? (allowing the user to configure the rebroadcasting periods, and/or adding a command line switch (perhaps even GUI) to manually force the rebroadcast...)

Or is there something peculiar with those transactions that make them be refused by everyone every time they are sent? I'm ruling out voluntary double-spends. (if it's an involuntary one, it's possibly an even more serious issue...)

I think that using Gavin's bitcointools to mark the 5 rogue transactions as 'unspent' should fix your problem.

Get it from here:
  https://github.com/gavinandresen/bitcointools

Then something like:
  python fixwallet.py --tweakspent ccf585bcfed1283d11b5012c5f3f3429c5695a8d2280287c44e605223b6318b5

I'll experiment here to see if it works, and report back.

It is rebroadcasting the transactions over and over.  From the log Jeremy put on dropbox:


I'm wondering if the transactions are invalid for some reason, and so his peers aren't accepting the transactions.  I don't know why that would be though.

I used my current wallet with a backup of all the rest of the files in the bitcoin folder from 1 day prior to the unconfirmed transactions. My understanding was that the wallet contained nothing but the private keys. Is this incorrect?

Could someone isolate his node and refuse transactions ?

Unfortunately it contains a lot more than just the private keys which I guess is exactly why the pywallet tool was created.

No, I believe the wallet.dat also contains transactions stored. If you use the same it won't work.

Theoretically possible, but very unlikely. Hard to do, and, why would the attacker cherry-pick these 5 transactions?


Jeremy, if the backup thing doesn't work, would you mind adding some trusted peers in your list manually?
https://en.bitcoin.it/wiki/Fallback_Nodes

the debug file suggests that these stored transactions are resent without being checked.
they might have become double-spents because of a reorg.
I guess it would make sense for ResendWalletTransactions() to check old transactions inputs again. Does it redo this check?

Yes, that's incorrect.

See http://bitcoin.stackexchange.com/questions/3173/what-information-does-a-wallet-contain

I got the Windows version of pywallet installed and running (and yes you do need to click Next/I Agree many times).

Once you have "pywallet.bat" running then you can use http://localhost:8989/ as the interface (you need to have a web server running locally for this - if using Windows 7 then assuming IIS was installed then this is how to start it http://technet.microsoft.com/en-us/library/cc732317(v=ws.10).aspx).

Assuming you got the Pywallet Web Interface to appear then if you scroll down to Delete a key from your wallet you should be able to enter the txid of those 5 tx's to remove them one at a time (of course make sure you have a backup of the wallet before doing this).

I just asked on the dev channel, and I got confirmation of what I feared:

so, this might explain your problem, even if it does not really solve it.
I guess you could try to remove the transactions from your wallet using pywallet. make sure you back it up before.

It seems you have transactions in your wallet that conflict wih others in the blockchain (i.e., attempted double spends). Have you sent transactions from an instance that used a copy of the wallet.dat as well as from an instance running the original? That could explain the behaviour.

The only solution for now is removing those transactions from your wallet. I suppose tools like pywallet can do this, or manual tinkering using the bdb tools to dump/remove/load the wallet file.

There are plans for adding logic to the wallet for detecting transactions that conflict with the blockchain, and at least temporarily disable them, and eventually delete them if the transactions that they conflict with are buried too deep in the chain.

The pywallet method definitively works, i had this same issue a while back with one tx and used it without other alternative. You will have to remove the transaction from wallet.dat, which is a database that stores this kind of info, manually. Make sure you have a backup copy of your wallet before doing any of this.

Like others have said in this thread, get yourself pywallet (https://github.com/jackjack-jj/pywallet) by downloading the zip and unpack it somewhere accessible on your main windows drive (for ex. c:\pywallet).
 
1. Open up a terminal (http://java.about.com/od/gettingstarted/ss/firstprogram_4.htm) window by running "cmd" and make sure you have python interpreter installed by running "python" in the console window.

2. If you get an "unknown command" error, download and install python here (http://www.python.org/download/) then optionally add it  to your path (http://stackoverflow.com/questions/3701646/how-to-add-to-the-pythonpath-in-windows-7).

3. You will have to install some python libraries like bsddb and twisted to have it work correctly. Download setuptools 0.6 from here (http://pypi.python.org/packages/2.7/s/setuptools/).

4. After installing it run this in your terminal:


where X: is the drive where your python installation is located.

5. Now comes the fun part, run the following command by replacing X: with the correct drive letter in both places:


If it doesn't give you an error don't close the terminal, just minimize it and open up your usual web browser at this address http://localhost:8989, which is the pywallet web interface page that it serves locally, on port 8989 by default.

6. The next step is to find out the id of the troubling transaction. Just fire up your bitcoin wallet and when it finishes go to "Transactions" tab and double click the unconfirmed tx, you should see it's id down on the list. Copy it and shut down your bitcoin wallet, completely, or else pywallet would not be able to work with the wallet safely.

7. Now go to your browser on the pywallet's page and paste your tx id in "Delete a key from your wallet" section and hit "Delete" without changing anything else. You should see a success message right away.

8. Start your bitcoin wallet normally and check if the transaction has been deleted, so now you can resend it again but make sure you include some fees so it gets confirmed right away. Hope this helps.

EDIT: Added twisted and bsddb libs install.

Actually the current installation will create a pywallet.bat that you just need to run (it calls pywallet.py with the args provided for you).

Most importantly make sure you have a web server running locally in order to use http://localhost:8989.

EDIT - (see below)

I have tried to install this stuff and just hit a brick wall somewhere. Unfortunately I'm a customer service guru and other than that just a general user of computers and bitcoin, not a coder or technical person in any way.

I've given up for the evening and gone ahead and paid these five transactions again, hoping that I can figure out this pywallet stuff and recover the lost bitcoins tomorrow. ;(

Thanks for all your help so far everybody! The bounty will go to the person who writes up step by step instructions that make sense to a total non-techie like myself. :)

I've been unable to get bitcointools to do what I wanted.

https://bitcointalk.org/index.php?topic=29765.0 seems to be a similar issue to Jeremy's, but the solution that was found there is on an old git branch that never made it back to the master branch.

Although not exactly the easiest thing for a typical end-user to accomplish I actually found the instructions fairly straight forward.

If you could perhaps explain exactly how far you got with pywallet and what did/didn't happen at that point perhaps it shouldn't be too hard to get over the final hurdle.

I know this is going to make me sound incredibly ignorant, but we'll start with the fact that I don't know what you mean when you say I need to have a local web server running. I just have normal DSL internet access. Is it still possible to do this?

Sure - when running a local web server you don't need internet access at all.

So - firstly you will need to make sure you have IIS installed (check this http://www.howtogeek.com/howto/windows-vista/how-to-install-iis-on-windows-vista/).

EDIT - (see below)

Please don't recommend installing IIS for this, the pywallet page is generated by python with the help of twisted library so there is no need to have the microsoft IIS involved at all.

@Jeremy West, just follow the steps i posted earlier (https://bitcointalk.org/index.php?topic=85689.msg944529#msg944529) and you should be back in business in no time  :)

Oh - sorry my bad - I guess that the twisted library actually is the web server (doh! of course that's why 8989).

Forget the IIS web server installation then and after running pywallet.bat try http://localhost:8989 (you may have to allow an exception in your firewall for the web server to run on port 8989).

I just followed your instructions and it worked for me.  I had made a mess of my wallet and failed to tidy it up using bitcointools.  Following your instructions I was able to remove the double-spend transaction and get the wallet back to normal.

I already had Python installed, and am running Linux, so I've not tested it on Windows.

I think i know where you got stuck, i forgot to put how to install the needed python libraries. I'm making the corrections right now, sorry.

EDIT: Please give it another try (https://bitcointalk.org/index.php?topic=85689.msg944529#msg944529) and give us feedback if you get stuck on something.

@dooglus great, seems like on linux this kind of "hacking" is done allot easier and I love it  :)

There were some very good points made in this thread and just so that they aren't overlooked I'm putting in a recap / summary:


Yes, step 1 of any offers given to help someone in a scenario like this should include:

Before you try reverting to an old wallet, or performing pywallet surgery on your wallet, make sure that you make a backup (or two) of your current wallet.

In this instance, Jeremy made over 100 transactions.  Had he reverted to an old wallet, and wasn't using a large keypool already, there is the possibility that he would have given out Bitcoin address and received payments to them where these addresses didn't exist yet when the most recent backup was made.  

Hopefully everyone actively using Bitcoin at these levels is well aware of the keypool and has adjusted the keypools size to be large enough so that the backup and recovery strategy will make it so there is not ever result in lost bitcoins because of not having backups that include new keys.

Here's notme's mention of that:


The Bitcoin.org client isn't clogged up to accommodate every use case out there, so merchants should know that the -keypool=<n> configuration setting and a good backup strategy require proper attention to prevent the situation where there are lost bitcoins.  If your backup strategy is to take a nightly snapshot (shut down the client & copy the wallet.dat) and retain that backup copy for each night over the past 7 days, then you probably want a key pool of multiple days worth of transactions (or, more specifically large enough for multiple days worth of addresses that are consumed from the keypool).  As notme suggests, 15X daily usage is not rediculous (they're really cheap so why not!).


The client will rebroadcast within in a half hour.  The normal use case is for this to not occur on startup as that lessens the transaction anonymity.  The ability to force a rebroadcast was discussed but the conclusion was that the range of zero to 30 minutes was sufficient so the ticket was closed:
 - https://github.com/bitcoin/bitcoin/pull/421

So after the cosmetic problem where there are 0/unconfirmed transactions is resolved for Jeremy it would be useful to try to understand what might have caused this to occur in the first palce.  There is an attack in which this exact situation could be the result (the vector76 attack or "one confirmation attack") but that probably isn't what happened here.  Further information on this attack here:
 - https://github.com/bitcoin/bitcoin/issues/1428

Ending up in this situation where you have these 0/unconfirmed spend transactions that will never confirm isn't something that is easy to do unless either you are doing something wrong (e.g., either purposely trying to double spend, or accidentally double spent by using the same wallet on multiple nodes and unwittingly created a double spend in the process) or you happened to not only be the target of a double spend but you happened to also use funds from that double spend for a spend transaction yourself.

Any guess as to what might have happened here?

Thanks again for all the help you've all given so far. One of you has sent me a private message offering to walk me through this over skype (he/she hasn't given me authority to say who, so although I'm sure they wouldn't mind, I'll keep it private since they sent me a PM) and I'm going to take you up on that as soon as we can connect.

Just thought I'd update where I'm at with all of this right now:

--I sent the customers their bitcoins again yesterday. This is my problem not theirs, and it sounds like these won't rebroadcast and I'll be able to retrieve them anyway.
--I had another transaction hang like this late yesterday, so I've just set up a new wallet and transferred my coins there in 100 btc lots (so if any more get caught it won't affect the whole amount).

Phew, this has been a week of putting out fires. And it's supposed to be my study week for finals. I haven't cracked a book once! :(

Oh well, the life of a bitcoin entrepreneur. Crazy roller coaster, but worth every minute. :)

Haha, sounds just like me, except I study for finals the class before ;)

I've been studying all semester, but Australia gives an extra week for studying between the last week of classes and the final exams.

Hi,
I sent some bitcoins to Jeremy, the amount was deducted from my wallet, but the second day I noticed the transaction is still sitting unconfirmed. Today I checked it again and it finally passed (after two days).
Can this be related to the issue Jeremy had?

Unlikely.  If the payment doesn't make it into a block, then SpendBitcoins probably wouldn't have been able to use it in a payment, at least not with the stock Bitcoin.org client.

I think he's asking if he had the same issue as me, but it doesn't sound like it because mine never broadcast while his eventually did.

I have the exact same problem, a very werid thing, tx's going through and others not all within a short time frame, it began with opening my wallet and finding more coins than I had before, dono if that triggers anything in someones brain but i reloaded the whole blockchain like you said your going to do, i didnt help, i havn't tried this rescan though, not sure how they are different but ill check it out, hope this helps a little at least

That would be me.

I never heard from you on Skype.  I don't know if you tried to contact me.

Did you ever get the issue resolved?  I had forgotten about this thread until trying to answer a similar issue for a SatoshiDICE player here: https://bitcointalk.org/index.php?topic=77870.msg958137#msg958137

I think he has the same problem as you, except that his bad transactions are incoming, not outgoing.  Either way, deleting them using PyWallet should fix the problem.

Edit: I'll subscribe to this thread this time so I won't miss your reply again.

Hi Dooglus. I'll definitely be contacting you on skype. I haven't tried yet because I haven't had a moment to. :(

Sorry to bump an old thread, but just figured I'd mention that I ran into the same issue and give the details in case someone who works in-depth with bitcoin has more information to work with.

Anyhow, sent a small transaction that had the status of "0/unconfirmed" followed by something about being broadcasted out through 12 nodes or something like that.  Later went to just "0/unconfirmed" though.


It deducted the coins but never showed up in the blockchain.  Someone checked the bitcoincharts for me as well for a low priority transaction that was queued but came up with nothing.

Used pywallet to delete all the keys and rescanned (returned ~2.9BTC) as well as trying to delete the specific transactions that were unconfirmed and rescanning (put me at a balance of 0BTC).  *Note: There was 1 other unconfirmed transaction besides the one quoted above.

I did have an issue about a month ago where I got a message about a double-spend when my bitcoin wallet crashed as I went to send payment for something.  The double-spend message later disappeared from the blockchain as well.  However, that transaction was for 4.0BTC so I'm not sure why fixing my wallet gave me ~2.9BTC

I've had 90 successful transactions and 2 bad ones within the past 30 day period.

Grrr... I haven't had a chance to do this yet. I was hoping doing this would give me all my bitcoins back. :(

Yea, I hear you, my 4BTC is peanuts compared to yours.  :-\

But you did mention having backups so hopefully you'll get it resolved.  I never backed mine up although I'm starting to realize that moving forward I should definitely back up my wallet weekly or be super paranoid and back it up a couple times a week even though I do an extremely low volume of transactions.

Or, run 'bitcoin-qt -keypool=9999', wait for it to generate 9999 pool addresses, then back up the wallet, and you don't need to make another backup until you've used those 9999 addresses.  "Using" an address happens each time you make a transaction which generates change back to yourself, and each time you click 'new address' in the 'receive coins' tab.  Some RPC calls will also use a pool address, but if you don't use RPC you don't need to worry about that.  9999 can be any sufficiently large number, of course.  By default it's only 100, meaning you need to back up at least every 100 spends you make.