HELP! Transactions not confirming ;( - 4 btc bounty

[caveden]

I used my current wallet with a backup of all the rest of the files in the bitcoin folder from 1 day prior to the unconfirmed transactions. My understanding was that the wallet contained nothing but the private keys. Is this incorrect?

No, I believe the wallet.dat also contains transactions stored. If you use the same it won't work.

[caveden]

I'm wondering if the transactions are invalid for some reason, and so his peers aren't accepting the transactions.  I don't know why that would be though.

Could someone isolate his node and refuse transactions ?

Theoretically possible, but very unlikely. Hard to do, and, why would the attacker cherry-pick these 5 transactions?


Jeremy, if the backup thing doesn't work, would you mind adding some trusted peers in your list manually?
https://en.bitcoin.it/wiki/Fallback_Nodes

[ThomasV]

I used my current wallet with a backup of all the rest of the files in the bitcoin folder from 1 day prior to the unconfirmed transactions. My understanding was that the wallet contained nothing but the private keys. Is this incorrect?

No, I believe the wallet.dat also contains transactions stored. If you use the same it won't work.

the debug file suggests that these stored transactions are resent without being checked.
they might have become double-spents because of a reorg.
I guess it would make sense for ResendWalletTransactions() to check old transactions inputs again. Does it redo this check?

[dooglus]

I used my current wallet with a backup of all the rest of the files in the bitcoin folder from 1 day prior to the unconfirmed transactions. My understanding was that the wallet contained nothing but the private keys. Is this incorrect?

Yes, that's incorrect.

See http://bitcoin.stackexchange.com/questions/3173/what-information-does-a-wallet-contain

[CIYAM]

I got the Windows version of pywallet installed and running (and yes you do need to click Next/I Agree many times).

Once you have "pywallet.bat" running then you can use http://localhost:8989/ as the interface (you need to have a web server running locally for this - if using Windows 7 then assuming IIS was installed then this is how to start it http://technet.microsoft.com/en-us/library/cc732317(v=ws.10).aspx).

Assuming you got the Pywallet Web Interface to appear then if you scroll down to Delete a key from your wallet you should be able to enter the txid of those 5 tx's to remove them one at a time (of course make sure you have a backup of the wallet before doing this).

[ThomasV]

I just asked on the dev channel, and I got confirmation of what I feared:

Code:

<ThomasV> does ResendWalletTransactions() check tx inputs?
<sipa> ThomasV: how do you mean?
<sipa> ThomasV: it just resends all unconfirmed transactions
<ThomasV> sipa: double spends, etc
<ThomasV> sipa: https://bitcointalk.org/index.php?topic=85689.msg944505#msg944505
<sipa> ThomasV: the current wallet code basically always assumes that its transactions are valid and will eventually confirm
<sipa> that's a major flaw
<ThomasV> sipa: what if a tx has become a double spent because of a reorg?
<ThomasV> could this explain the error reported there?
<sipa> ThomasV: i suppose
<ThomasV> sipa: what should this user do?
<ThomasV> (you might get a bounty if you answer in the thread)

so, this might explain your problem, even if it does not really solve it.
I guess you could try to remove the transactions from your wallet using pywallet. make sure you back it up before.

[Pieter Wuille]

It seems you have transactions in your wallet that conflict wih others in the blockchain (i.e., attempted double spends). Have you sent transactions from an instance that used a copy of the wallet.dat as well as from an instance running the original? That could explain the behaviour.

The only solution for now is removing those transactions from your wallet. I suppose tools like pywallet can do this, or manual tinkering using the bdb tools to dump/remove/load the wallet file.

There are plans for adding logic to the wallet for detecting transactions that conflict with the blockchain, and at least temporarily disable them, and eventually delete them if the transactions that they conflict with are buried too deep in the chain.

[paraipan]

The pywallet method definitively works, i had this same issue a while back with one tx and used it without other alternative. You will have to remove the transaction from wallet.dat, which is a database that stores this kind of info, manually. Make sure you have a backup copy of your wallet before doing any of this.

Like others have said in this thread, get yourself pywallet by downloading the zip and unpack it somewhere accessible on your main windows drive (for ex. c:\pywallet).
 
1. Open up a terminal window by running "cmd" and make sure you have python interpreter installed by running "python" in the console window.

2. If you get an "unknown command" error, download and install python here then optionally add it to your path.

3. You will have to install some python libraries like bsddb and twisted to have it work correctly. Download setuptools 0.6 from here.

4. After installing it run this in your terminal:

Code:

X:\Python27\Scripts\easy_install bsddb3
X:\Python27\Scripts\easy_install twisted

where X: is the drive where your python installation is located.

5. Now comes the fun part, run the following command by replacing X: with the correct drive letter in both places:

Code:

X:\Python27\python X:\pywallet\pywallet.py --web

If it doesn't give you an error don't close the terminal, just minimize it and open up your usual web browser at this address http://localhost:8989, which is the pywallet web interface page that it serves locally, on port 8989 by default.

6. The next step is to find out the id of the troubling transaction. Just fire up your bitcoin wallet and when it finishes go to "Transactions" tab and double click the unconfirmed tx, you should see it's id down on the list. Copy it and shut down your bitcoin wallet, completely, or else pywallet would not be able to work with the wallet safely.

7. Now go to your browser on the pywallet's page and paste your tx id in "Delete a key from your wallet" section and hit "Delete" without changing anything else. You should see a success message right away.

8. Start your bitcoin wallet normally and check if the transaction has been deleted, so now you can resend it again but make sure you include some fees so it gets confirmed right away. Hope this helps.

EDIT: Added twisted and bsddb libs install.

[CIYAM]

and then run the following command:

Code:

python pywallet.py --web

Actually the current installation will create a pywallet.bat that you just need to run (it calls pywallet.py with the args provided for you).

Most importantly make sure you have a web server running locally in order to use http://localhost:8989.

EDIT - (see below)

[Jeremy West spendbitcoins.com]

and then run the following command:

Code:

python pywallet.py --web

Actually the current installation will create a pywallet.bat that you just need to run (it calls pywallet.py with the args provided for you).

Most importantly make sure you have a web server running locally in order to use http://localhost:8989.

I have tried to install this stuff and just hit a brick wall somewhere. Unfortunately I'm a customer service guru and other than that just a general user of computers and bitcoin, not a coder or technical person in any way.

I've given up for the evening and gone ahead and paid these five transactions again, hoping that I can figure out this pywallet stuff and recover the lost bitcoins tomorrow. ;(

Thanks for all your help so far everybody! The bounty will go to the person who writes up step by step instructions that make sense to a total non-techie like myself.

[dooglus]

I think that using Gavin's bitcointools to mark the 5 rogue transactions as 'unspent' should fix your problem.

Get it from here:
  https://github.com/gavinandresen/bitcointools

Then something like:
  python fixwallet.py --tweakspent ccf585bcfed1283d11b5012c5f3f3429c5695a8d2280287c44e605223b6318b5

I'll experiment here to see if it works, and report back.

I've been unable to get bitcointools to do what I wanted.

https://bitcointalk.org/index.php?topic=29765.0 seems to be a similar issue to Jeremy's, but the solution that was found there is on an old git branch that never made it back to the master branch.

[CIYAM]

I have tried to install this stuff and just hit a brick wall somewhere.

Although not exactly the easiest thing for a typical end-user to accomplish I actually found the instructions fairly straight forward.

If you could perhaps explain exactly how far you got with pywallet and what did/didn't happen at that point perhaps it shouldn't be too hard to get over the final hurdle.

[Jeremy West spendbitcoins.com]

I have tried to install this stuff and just hit a brick wall somewhere.

Although not exactly the easiest thing for a typical end-user to accomplish I actually found the instructions fairly straight forward.

If you could perhaps explain exactly how far you got with pywallet and what did/didn't happen at that point perhaps it shouldn't be too hard to get over the final hurdle.

I know this is going to make me sound incredibly ignorant, but we'll start with the fact that I don't know what you mean when you say I need to have a local web server running. I just have normal DSL internet access. Is it still possible to do this?

[CIYAM]

I know this is going to make me sound incredibly ignorant, but we'll start with the fact that I don't know what you mean when you say I need to have a local web server running. I just have normal DSL internet access. Is it still possible to do this?

Sure - when running a local web server you don't need internet access at all.

So - firstly you will need to make sure you have IIS installed (check this http://www.howtogeek.com/howto/windows-vista/how-to-install-iis-on-windows-vista/).

EDIT - (see below)

[paraipan]

I know this is going to make me sound incredibly ignorant, but we'll start with the fact that I don't know what you mean when you say I need to have a local web server running. I just have normal DSL internet access. Is it still possible to do this?

Sure - when running a local web server you don't need internet access at all.

So - firstly you will need to make sure you have IIS installed (check this http://www.howtogeek.com/howto/windows-vista/how-to-install-iis-on-windows-vista/).

If not already installed then most likely the service will be automatically started after you have installed it (to test just type http://localhost - if you get a web page that is not an error then you have a web server running).

Please don't recommend installing IIS for this, the pywallet page is generated by python with the help of twisted library so there is no need to have the microsoft IIS involved at all.

@Jeremy West, just follow the steps i posted earlier and you should be back in business in no time  

[CIYAM]

Please don't recommend installing IIS for this, the pywallet page is generated by python with the help of twisted library so there is no need to have the microsoft IIS involved at all.

@Jeremy West, just follow the steps i posted earlier and you should be back in business in no time  

Oh - sorry my bad - I guess that the twisted library actually is the web server (doh! of course that's why 8989).

Forget the IIS web server installation then and after running pywallet.bat try http://localhost:8989 (you may have to allow an exception in your firewall for the web server to run on port 8989).

[dooglus]

The pywallet method definitively works, i had this same issue a while back with one tx and used it without other alternative. You will have to remove the transaction from wallet.dat, which is a database that stores this kind of info, manually. Make sure you have a backup copy of your wallet before doing any of this.

I just followed your instructions and it worked for me.  I had made a mess of my wallet and failed to tidy it up using bitcointools.  Following your instructions I was able to remove the double-spend transaction and get the wallet back to normal.

I already had Python installed, and am running Linux, so I've not tested it on Windows.

[paraipan]

I think i know where you got stuck, i forgot to put how to install the needed python libraries. I'm making the corrections right now, sorry.

EDIT: Please give it another try and give us feedback if you get stuck on something.

@dooglus great, seems like on linux this kind of "hacking" is done allot easier and I love it 

[Stephen Gornick]

There were some very good points made in this thread and just so that they aren't overlooked I'm putting in a recap / summary:

Keep a copy of your current wallet before doing this, of course.

Yes, step 1 of any offers given to help someone in a scenario like this should include:

Before you try reverting to an old wallet, or performing pywallet surgery on your wallet, make sure that you make a backup (or two) of your current wallet.

In this instance, Jeremy made over 100 transactions.  Had he reverted to an old wallet, and wasn't using a large keypool already, there is the possibility that he would have given out Bitcoin address and received payments to them where these addresses didn't exist yet when the most recent backup was made.  

Hopefully everyone actively using Bitcoin at these levels is well aware of the keypool and has adjusted the keypools size to be large enough so that the backup and recovery strategy will make it so there is not ever result in lost bitcoins because of not having backups that include new keys.

Here's notme's mention of that:

I think the default keypool is only 100.  If you're doing this many transactions you should set the keypool size to 1500 or some other high number so your backups are at least valid for a few days.

The Bitcoin.org client isn't clogged up to accommodate every use case out there, so merchants should know that the -keypool=<n> configuration setting and a good backup strategy require proper attention to prevent the situation where there are lost bitcoins.  If your backup strategy is to take a nightly snapshot (shut down the client & copy the wallet.dat) and retain that backup copy for each night over the past 7 days, then you probably want a key pool of multiple days worth of transactions (or, more specifically large enough for multiple days worth of addresses that are consumed from the keypool).  As notme suggests, 15X daily usage is not rediculous (they're really cheap so why not!).

To the devs: isn't this a bug in the Satoshi client? Shouldn't the client try to rebroadcast unconfirmed transactions once in a while? If it does rebroadcast, just not fast enough for Jeremy's needs, then isn't this an enhancement issue that should be logged anyway?

The client will rebroadcast within in a half hour.  The normal use case is for this to not occur on startup as that lessens the transaction anonymity.  The ability to force a rebroadcast was discussed but the conclusion was that the range of zero to 30 minutes was sufficient so the ticket was closed:
 - https://github.com/bitcoin/bitcoin/pull/421

So after the cosmetic problem where there are 0/unconfirmed transactions is resolved for Jeremy it would be useful to try to understand what might have caused this to occur in the first palce.  There is an attack in which this exact situation could be the result (the vector76 attack or "one confirmation attack") but that probably isn't what happened here.  Further information on this attack here:
 - https://github.com/bitcoin/bitcoin/issues/1428

Ending up in this situation where you have these 0/unconfirmed spend transactions that will never confirm isn't something that is easy to do unless either you are doing something wrong (e.g., either purposely trying to double spend, or accidentally double spent by using the same wallet on multiple nodes and unwittingly created a double spend in the process) or you happened to not only be the target of a double spend but you happened to also use funds from that double spend for a spend transaction yourself.

Any guess as to what might have happened here?

[Jeremy West spendbitcoins.com]

Thanks again for all the help you've all given so far. One of you has sent me a private message offering to walk me through this over skype (he/she hasn't given me authority to say who, so although I'm sure they wouldn't mind, I'll keep it private since they sent me a PM) and I'm going to take you up on that as soon as we can connect.

Just thought I'd update where I'm at with all of this right now:

--I sent the customers their bitcoins again yesterday. This is my problem not theirs, and it sounds like these won't rebroadcast and I'll be able to retrieve them anyway.
--I had another transaction hang like this late yesterday, so I've just set up a new wallet and transferred my coins there in 100 btc lots (so if any more get caught it won't affect the whole amount).

Phew, this has been a week of putting out fires. And it's supposed to be my study week for finals. I haven't cracked a book once!

Oh well, the life of a bitcoin entrepreneur. Crazy roller coaster, but worth every minute.