[GUIDE] Simple wallet recovery using bitcointools

[Matt Corallo]

THIS THREAD IS VERY OUT-OF-DATE AND SAYS SOME THINGS THAT AREN'T TRUE ANYMORE. IT APPEARS THAT FOLLOWING IT HAS CAUSED SOME USERS TO HAVE INCREASED DIFFICULTY RECOVERING THEIR WALLETS, SO PLEASE DONT DO THIS UNTIL (IF) THIS THREAD IS UPDATED

So, you open Bitcoin to check on your Bitcoins and to your dismay it won't open.  Now what?
You probably see errors like

EXCEPTION: NSt8ios_base7failureE
CDataStream::read() : end of data
bitcoin in AppInit()

Don't worry, your keys are almost always still there, they are just hiding, I'm gonna help you find them

Prerequisites: A willingness to open your Terminal application, Python

Methodology: Usually when a wallet gets corrupted, its not the private keys that get corrupted but some setting, transaction or public key.  In order to get Bitcoin to open and allow you to access your Bitcoins, we will use bitcointools' fixwallet.py to delete settings, accounts, public keys, and transactions and see if we can get you access to your private keys to spend your coins again.

Notes:

• This guide will refer to your .bitcoin directory several times, by default this is on Linux, /home/folder/.bitcoin, on Windows %AppData%/Bitcoin (probably C:\Users\UserName\AppData\Roaming\Bitcoin on Win Vista/7 or C:\Documents and Settings\UserName\Application Data\Bitcoin on XP) and on Mac OSX, /Users/UserName/Library.  If you use a different bitcoin folder, use it instead and add a --datadir=/path/to/your/bitcoin/folder each time you call one of the bitcointools programs.

Guide:

• 1. BACKUP YOUR ENTIRE .bitcoin FOLDER Period, even corrupted wallets will still have useful data in them and the database folder may have enough BDB logs for a real BDB expert to fix your wallet if all of this fails.  Also, several points in the guide require you to restore the original corrupted wallet.dat, so you have to have it.
  
• 2. Often times the BDB logs or other DBs get corrupted and Bitcoin refuses to start.  Delete all the files in your .bitcoin directory except for your wallet.dat and try Bitcoin again.  This should solve 90% of the corrupted DB errors you see with Bitcoin.  If this doesn't work, the corruption lies in your wallet, but don't worry, it should still be fixable.
  
• 3. Grab yourself a copy of Gavin's excellent bitcointools either by using git clone https://github.com/gavinandresen/bitcointools.git or by using the download tar.gz from https://github.com/gavinandresen/bitcointools.  Untar it to a directory of your choice and cd /path/to/that/directory in your Terminal application.
  
• 4. First, lets do a basic sanity check on the wallet.  Call ./dbdump.py --wallet once you have cd'd to the bitcointools directory.  You should get some out put that looks like

  PubKey 0402...de94 1XXXXXXXXXXXXXXXXXXXXXXXXXX: PriKey 3082...de94

  some that looks like

  ADDRESS 11XXXXXXXXXXXXXXXXXXXXXXXXXX :

  Change Pool key 1: 1XXXXXXXXXXXXXXXXXXXXXXXXXX (Time: Fri May 27 16:27:28 2011)

  and finally some settings like

  fUseUPnP: True
  

  If some of those are missing, dont worry, all that really matter is the first type, which contains your bitcoin private keys. As long as you dont see any errors along the lines of ERROR parsing wallet.dat, type key, ekey, or wkey, and you have several lines of the first type, you should have no problem getting access to your bitcoins. If you dont have any lines that look like the first then you are going to have to work a bit harder to get your keys back, which is outside of the scope of this post.
  
• 5. Now that you know your wallet still has your private keys in a readable state, its time to recover.  Run ./fixwallet.py and replace your old wallet.dat with walletNEW.dat that bitcointools just created in your bitcoin folder.  If you see several ERROR messages that are of a type other than key, you know it is working.  Try to open Bitcoin, you may find that it opens perfectly now, or you may find several issues in this wallet wrt your settings, accounts, etc which you may need to re-enter.  If you want to clear your accounts, transactions, settings, etc or Bitcoin still isnt working for you, keep reading.
  
• 6. This involves some trial-and-error.  The goal here is to run ./fixwallet.py with options to drop the corrupted parts of your wallet.  The options are --noaccounts, --nosettings, --notxes and --nopubkeys.  First off, replace the wallet.dat you created in step 5 with the original, corrupted, one.  Then run ./fixwallet.py --noaccounts --nosettings --notxes --nopubkeys; replace your wallet.dat in your .bitcoin folder with walletNEW.dat and see if Bitcoin opens (note that your transactions and balance will be gone when you use --notxes, this is intentional, read on for instructions on how to get them back).  If Bitcoin doesnt open, the recovery of your wallet falls outside the scope of this guide; you might try asking an expert who you trust to see if they can get further.  If it does work, however, repeat this step with less and less flags until Bitcoin refuses to open.
  If you used --notxes, open Bitcoin with -rescan to repopulate them from the block database. DO NOT use the --notxes option if you had transactions in your wallet which are not yet in a block.
  If you used --noaccounts, you will need to recreate your accounts.
  If you used --nopubkeys your list of public keys in your address book may be cleared, note that your old addresses will still work, however you may wish to change your publicized addresses to those which you can add to your address book so that they appear properly in your client.
  If you used --nosettings, please go back to the Options dialog and re-enter all your settings.  
  

Comments, suggestions, sticky?

[RaTTuS]

OK I'm not panicking too much yet but .....

my machine [windows vista 64] rebooted - as requested but when bitcoins loaded it errored with this exception ...
I've fixed my wallet with
fixwallet.py
and copied the wallet.new.dat over the old one [I've still got copies]
I've run
bitcoin -rescan
and i'm left with
all my transactions ok
but
0 connections , 0 blocks , 7 transactions
and balance 0.00

so for some reason I'm not connection to the network ?

only one comment - add a pay address in your sig 

[Matt Corallo]

The second step will delete your addr.dat too, so making new connections after that could take quite some time. There is a fix in the pipeline hopefully to come out very soon, but until then, I recommend forwarding your ports and trying -dnsseed or -addnode=(the ip of a known-up fallback node)

[RaTTuS]

yeah I've seen some other reported problems about connections....
I've added a couple of the backup ones and things are progressings [3 connections now and 200k blocks], when it all settings down and I have BTC available I'll smudge some over to you .... 

[RaTTuS]

 
recovered now,
and I've sent a small donation [though not from this address]

[mewantsbitcoins]

Thanks Matt

[nazgulnarsil]

when I type dbdump.py or anything else I get an error that says "no module named bsddb.db"  is there a bunch of setup I have to do that I'm neglecting?  I downloaded python and put all the bitcoin tools in my bitcoin folder.

[Matt Corallo]

when I type dbdump.py or anything else I get an error that says "no module named bsddb.db"  is there a bunch of setup I have to do that I'm neglecting?  I downloaded python and put all the bitcoin tools in my bitcoin folder.

You are gonna need the bsddb python plugin which you can probably get from http://www.jcea.es/programacion/pybsddb.htm

[nazgulnarsil]

switching to py 2.7 fixed that.  Now I don't know how to install the pycrypt library.

reading this: http://docs.python.org/install/index.html

but a little lost as to where the relevant info is.

[arnoldrimmer]

Do I need to intall Pyton?? How is that a simple guide?

I deletet everything exept the wallet.dat now I see all my Keys but my account i on 0.0

[Matt Corallo]

Do I need to intall Pyton?? How is that a simple guide?

I deletet everything exept the wallet.dat now I see all my Keys but my account i on 0.0

Well it was more targeted towards Linux/Unix users where python is pretty much part of a base install on every distro.
That said, if you install python you should be able to follow pretty cleanly on Windows as well.
If you deleted everything except wallet.dat and your client is showing 0 balance, wait for it to redownload the blockchain, as the 0 balance is to be expected. 

[joulesbeef]

when i do this ./dbdump.py --wallet

i get this

Code:

D:\Users\joulesbeef\Downloads\gavinandresen-bitcointools-294c080\gavinandresen-b
itcointools-294c080>dbdump.py --wallet
Traceback (most recent call last):
  File "D:\Users\joulesbeef\Downloads\gavinandresen-bitcointools-294c080\gavinan
dresen-bitcointools-294c080\dbdump.py", line 89, in <module>
    main()
  File "D:\Users\joulesbeef\Downloads\gavinandresen-bitcointools-294c080\gavinan
dresen-bitcointools-294c080\dbdump.py", line 58, in main
    dump_wallet(db_env, options.dump_wallet, dump_tx, options.wallet_tx_filter)
  File "D:\Users\joulesbeef\Downloads\gavinandresen-bitcointools-294c080\gavinan
dresen-bitcointools-294c080\wallet.py", line 247, in dump_wallet
    parse_wallet(db, item_callback)
  File "D:\Users\joulesbeef\Downloads\gavinandresen-bitcointools-294c080\gavinan
dresen-bitcointools-294c080\wallet.py", line 34, in parse_wallet
    for (key, value) in db.items():
bsddb.db.DBRunRecoveryError: (-30974, 'DB_RUNRECOVERY: Fatal error, run database
 recovery -- PANIC: Invalid argument')

[joulesbeef]

ok. i've managed to get dbdump to produce something that looks hopeful

PubKey 04ef...1f37 : PriKey 3082...1f37
PubKey 04f0...db99 : PriKey 3082...db99
PubKey 04f0...9954 : PriKey 3082...9954
PubKey 04f0...998b : PriKey 3082...998b
PubKey 04f0...ea42 : PriKey 3082...ea42
PubKey 04f1...6a44 : PriKey 3082...6a44
PubKey 04f1...9187 : PriKey 3082...9187
PubKey 04f1...2f2a : PriKey 3082...2f2a
PubKey 04f5...e867 : PriKey 3082...e867
PubKey 04f5...0ce2 : PriKey 3082...0ce2
PubKey 04f6...8b0b : PriKey 3082...8b0b
PubKey 04f8...d789 : PriKey 3082...d789
PubKey 04f8...e93f : PriKey 3082...e93f
PubKey 04f9...e1d5 : PriKey 3082...e1d5
PubKey 04fa...4a0f : PriKey 3082...4a0f
PubKey 04fa...125f : PriKey 3082...125f
PubKey 04fa...1d60 : PriKey 3082...1d60
PubKey 04fa...06cc : PriKey 3082...06cc
PubKey 04fd...b76f : PriKey 3082...b76f
ADDRESS 1PqphMPWGU73WY6mAEPwqJoAWAsbizve1L :
Change Pool key 2:  (Time: Fri Jul 22 16:18:22 2011)
Change Pool key 3:  (Time: Fri Jul 22 16:18:23 2011)

but when i do a fixwallet and replae the wallet and run bitcoin it starts up but my wallet is empty and no addys or nothing.

any help?

[Matt Corallo]

Thats the entire output? Thats only 19 private keys, there should be >100 at all times.  Its possible you might have lost some of your keys.  Yes, if you fixwallet the wallet and replace with that you will only have the address 1PqphMPWGU73WY6mAEPwqJoAWAsbizve1L, which is not the default key, so will just be another one in your address book.  If there are transactions to any of the private keys there, they will be found if you do a -rescan or as you download the block chain.

[joulesbeef]

Thats the entire output?

nope the list was way long.. goes past the buffer of xp's command prompt

and I only coppied a portion of what it outputted

If there are transactions to any of the private keys there, they will be found if you do a -rescan or as you download the block chain.

launched with bitcoin -rescan

so i guess i just wait and pray.. it isnt much but i'd still like to get it back
.

thanks for the help

[joulesbeef]

It worked.. there were only 0.9 coins in there .. i thought 3 but it was a good learning experience.

[RevolutionMaster]

This worked well for me, and I recovered 3 bitcoins with the -notxes flag on the git pull. Thanks OP!

[Sy]

Still working, fixed my chncoin wallet

[FalconFour]

Yeah, this doesn't seem to work. All I get is this:

Code:

falcon@ubuntu:~/bitcointools$ ./fixwallet.py 
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.

I don't have Bitcoin installed in my Linux VM, so I made a ".bitcoin" folder and placed the corrupt wallet.dat (112kb and full of binary data similar to my working wallet.dat) inside in order to run these tools.

I can't seem to get any of this to work. Nothing wants to touch this database file. Since it's all proprietary binary data, it's now stuck inside this corrupt database file. This begs the question of exactly how hard could it have possibly been to simply make the wallet - an extremely important and valuable file - as an open XML file or something actually extensible and manageable instead of this gibberish binary Berkeley DB format?

So still, nothing. The file doesn't even appear corrupt, and yet, it won't even load...

The file is from a very old and forgotten Bitcoin wallet (an earlier Bitcoin-QT on Windows) that turns out to have had some coins left in it. Not sure what version.

edit: Turns out, the file was opened+saved in Notepad, completely corrupting the binary data - replacing 0x00 with 0x20 all throughout the file. That's completely unacceptable if the wallet file can be irreparably corrupted that easily by someone. I'm working with a hex editor now to replace 0x20 with 0x00 and rescuing the stragglers (where it should've actually been 0x20 - an ASCII space (" ") character). So far so good - this time, the debug log file reports it found 130-something records. So we'll see...

edit: there was a backup, so yay. all is well. ^_^

[edgar]

if anyone is able to add an updated step-by-step walkthrough of this process it may be appreciated.

i have downloaded python 2.7, pywallet, twisted 11.0, zope interface 4.1.3 and bitcointools-master.

all but pywallet are in the site-packages folder (pywallet is in the Scripts folder)

assuming this is all correct i am unable to 'cd into the bitcointools directory'