Wallet.Dat Recovery... Recover Your Own Lost Bitcoins!

[casascius]

I have been doing some tinkering around, thinking about other people's wallet disasters, and believe I have come to the following conclusion...

If you have lost your wallet.dat for whatever reason (deleted it, formatted your drive, file corruption, etc.) it's possible that it may still be lurking on your computer.  If so, recovery is no longer purely theoretical.  With a little knowledge of what to search for, you can use a hex editor to potentially find usable remnants of your wallet.dat file and get back your bitcoins, even if the original file isn't fully recoverable.

So here goes...

If you can use a hex-editor to do a sector-by-sector search/edit on your entire hard drive, then search your entire hard drive for occurrences of the following byte sequence:

01 03 6B 65 79 41 04...........

the middle four of these bytes represent the string "keyA" in ASCII.

Each time this byte sequence occurs, a Bitcoin private key is probably stored nearby, about 180 bytes later.  The 32-byte private key is the only thing you need to recover your bitcoins!... as long as you find the right one(s).

Approximately 180 bytes after this sequence, you may find the byte sequence 04 20 (hex).  These two bytes seem to precede every private key (the 0x20 suggests a length of 32 bytes).  If you find this sequence, the thirty-two bytes that come after 04 20 are the private key representing a Bitcoin address and might be the private key that recovers some of your lost bitcoins!  Your wallet will have numerous private keys (at least one hundred, due to the pre-allocation of keys)... get as many as you can find.  Carefully search the sectors adjacent to any sector containing the "keyA" sequence above.  Then yell for help!  (But don't share the private keys in public, unless you want to give away your wallet.)

An example of a hex editor that can scan an entire disk volume for specific byte sequences for Windows is WinHex.  In WinHex, use Tools, Open Disk (F9), and choose the disk you want to scan.  Scanning a full disk can take hours.  WinHex must "run as administrator" to be able to scan a physical disk.  Someone please recommend a good way to do this in Linux, preferably with a known Live CD, if possible.  Also, any time you are scanning a disk for potentially lost data, you should NEVER boot the disk you're searching - always boot from another disk and install the target disk as secondary.

[1714586944]

1714586944

[1714586944]

1714586944

[niooron]

Would this work on SSDs? I heard they don't let you physically scan the flash memory.

[casascius]

Would this work on SSDs? I heard they don't let you physically scan the flash memory.

Yes, you can do it on SSD's, it's just that your chances of success will be somewhat lower.  When you scan a disk in this manner, you are simply doing a sector-by-sector read of the entire disk, which will always succeed (in the sense that the disk can be read end-to-end without running into an error or a protest from the disk)... but it just may not turn up any lost bitcoins.

[dserrano5]

Say bye-bye to this method as soon as wallets are encrypted on disk.

[-]

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.

[chungy]

Wouldn't help if the file alone was encrypted (say, with GPG or similar).  Still, even if that's implemented in the client, I hope such a feature would be optional, it might contain a warning such as: Encrypting your wallet may help thwart theft of your wallet file, though it also diminishes chances of recovery on accidental file deletion.

[dserrano5]

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.

I was talking about encrypted wallet as implemented by the bitcoin client, which is a strongly demanded feature these days. Once deleted, it's lost forever. Keep backups!

[jackjack]

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.

I was talking about encrypted wallet as implemented by the bitcoin client, which is a strongly demanded feature these days. Once deleted, it's lost forever. Keep backups!

If bitcoin use encrypted wallets, copying the wallet.dat several times becomes possible without danger

[amincd]

Casascius, may I suggest considering to offer this as a service?

You already have the skill and knowledge to do a search for the wallet data, and are relatively trusted, so others might prefer to send their hard-drives to you in the event of an accidental deletion of a wallet than do it themselves.

[smoothie]

Wouldn't a windows system restore bring back a deleted wallet file?

[Maged]

Casascius, may I suggest considering to offer this as a service?

You already have the skill and knowledge to do a search for the wallet data, and are relatively trusted, so others might prefer to send their hard-drives to you in the event of an accidental deletion of a wallet than do it themselves.

This is now pretty much automated:
http://forum.bitcoin.org/index.php?topic=25091.0

[amincd]

Wouldn't a windows system restore bring back a deleted wallet file?

I think in some cases that would work. Windows doesn't create restore points all the time, so I think there's a high chance that the data that a person deletes will not have been in existence when the latest restore point was created.

By the way, looking at your profile pic makes me want to drink a smoothie.

This is now pretty much automated:
http://forum.bitcoin.org/index.php?topic=25091.0

Thanks for that. This should be mentioned in the bitcoin wiki if it already isn't.

[etotheipi]

Is it expected that the private key would come way after the public key?  I'm doing a brute force search of my wallet file, using my crypto library to check whether each sequence of 64 consecutive bytes is on the secp256k1 elliptic curve (meaning it's a public key), then searching for a 32-byte sequence that gives you this public key when you interpret it as the private key.  What I am finding is that (1) this is really slow, (2) not every public key has an associated private key in the wallet file and (3) the private keys I'm finding are located about 200-300KB past where the public key was found.

This works under the assumption that both keys are encoded in big-endian and the public keys are encoded as [0x04 [X] [Y]] and private keys are encoded as [0x0420 [secretInt]].  But it doesn't work well enough to be confident that I'm extracting everything--why am I finding so many public keys that don't have matching private keys?   Could keys be stored in different endiannesses?  are some of them not preceded by '0x0420'?

I'd like to see the keys stored in a flat file that makes key recovery a million times easier.  The keypairs can be encoded as constant-length binary strings, just like the headers are serialized.  The private keys can be encrypted individually in this flat file, without encrypting the entire file.  Instead, you just convert the 256-bit private-keys to a 256-bit encrypted-private-key before writing it to file.  Then your wallet file still works for tracking transactions (since the public keys/addresses are still in plaintext), but the private key will be useless without transforming it back to the unencrypted bitstring.

[casascius]

But it doesn't work well enough to be confident that I'm extracting everything--why am I finding so many public keys that don't have matching private keys?   Could keys be stored in different endiannesses?  are some of them not preceded by '0x0420'?

Are you searching just wallet.dat or your entire hard drive?  The block chain database is going to be loaded with numerous public keys - belonging to other people, of course.

Each record containing a key in wallet.dat also contains the ASCII text "key" nearby as well.  (Because wallet.dat can hold multiple kinds of records, and this is how the software knows that this record is a key.)

[etotheipi]

I'm only searching the wallet.dat file.  I wanted to get my keys into a flat file for fun, maybe come up with a way to convert between the wallet.dat and wallet.flat.txt, which would make manual key management and recovery easier.

I figured out that the other keys, I believe, are addresses to which I've sent coins before... the ones that show up under the "Sent" addresses tab in the client. 

However, it does seem that there are multiple instances of your public keys in the file.  One of my public keys (for which I have the private key) shows up 10+ times.  Others show up two or three times.  I guess the wallet file holds transaction information, in addition to the keys themselves.

[etotheipi]

So, I have created this script which pulls out every public and private key in your wallet and stores them into flat files.  The "keylistpub.txt" will contain every public key in the wallet file whether it's yours or not, and "keylistpair.txt" contains a list of just the keys for which there is an associated private key in the wallet, and includes both.  This script should avoid duplicate keys in the output files.

http://dl.dropbox.com/u/1139081/extractKeys.tar.gz

Keep in mind, this utility is going to create a new file on your computer with your private keys.   Handle with care! 

Now, I want to create something that converts the flat file back into a wallet.  This will allow someone to extract keys from multiple wallets, combine them into a single file, and the create a merged wallet.  The output file format looks like this:

Code:

1MRAs5doMqqbLQVuAUqGcxHBzrexMiTBG:
	PubKey: 
		cfd41f6ab9a217380bd2dc370592635797759c7de172f5cc6b228c1d4f83dde2
		44f5a373bf80e66db4c0d34a892def09d1f605aef0d94f6b2c3e0322dfdd331e
	PrivKey: 
		7bb1e283fe1007757c75966706553e16cdb5f148c22712811a78e6bcf30c9a1b

1QXg28gA7mLBB9LSMgf4sjoB9batJBXEtB:
	PubKey: 
		47634c35731a35b5b70d4959418dae2e1c6676a1007626092eef8bceb80e1b16
		0d048e2a917c80a3f5f085a06ce4c88f78d66c82abf2f2a1683c171f8bbdb7ab
	PrivKey: 
		6cd77c1cc66929e6db9bf4b502f4ce4868cb76037b66d630fd931a0ea2fb8bce
...

(don't get too excited, I have mangled the private keys)

-- The first string is the address, which should be fairly obvious. 
-- The PubKey is two 32-byte numbers (x,y), which correspond to a point on the secp256k1 elliptic curve (the ECDSA curve used by the bitcoin network)
-- The PrivKey is literally a random 32-byte number, which gives the public key when you multiply the generator point by this number.  Yes, a private key is just a random number.  As such, there is no way to identify whether a string of digits is a private key, without having a public key to compare to.  Or rather, every 256-bit number is a private key, so a "private key" is only meaningful in the context of a public key point (x,y).   (all hex numbers are encoded in BigEndian)


Anyone want to help converting pub/priv keypairs into a wallet file?  I believe it can be done with the bsddb package in Python, but I haven't gotten it to work, myself, yet.

[jackjack]

So, I have created this script which pulls out every public and private key in your wallet and stores them into flat files.  The "keylistpub.txt" will contain every public key in the wallet file whether it's yours or not, and "keylistpair.txt" contains a list of just the keys for which there is an associated private key in the wallet, and includes both.  This script should avoid duplicate keys in the output files.

http://dl.dropbox.com/u/1139081/extractKeys.tar.gz

Keep in mind, this utility is going to create a new file on your computer with your private keys.  Handle with care!  

Now, I want to create something that converts the flat file back into a wallet.  This will allow someone to extract keys from multiple wallets, combine them into a single file, and the create a merged wallet.  The output file format looks like this:

Code:

1MRAs5doMqqbLQVuAUqGcxHBzrexMiTBG:
	PubKey: 
		cfd41f6ab9a217380bd2dc370592635797759c7de172f5cc6b228c1d4f83dde2
		44f5a373bf80e66db4c0d34a892def09d1f605aef0d94f6b2c3e0322dfdd331e
	PrivKey: 
		7bb1e283fe1007757c75966706553e16cdb5f148c22712811a78e6bcf30c9a1b

1QXg28gA7mLBB9LSMgf4sjoB9batJBXEtB:
	PubKey: 
		47634c35731a35b5b70d4959418dae2e1c6676a1007626092eef8bceb80e1b16
		0d048e2a917c80a3f5f085a06ce4c88f78d66c82abf2f2a1683c171f8bbdb7ab
	PrivKey: 
		6cd77c1cc66929e6db9bf4b502f4ce4868cb76037b66d630fd931a0ea2fb8bce
...

(don't get too excited, I have mangled the private keys)

-- The first string is the address, which should be fairly obvious.  
-- The PubKey is two 32-byte numbers (x,y), which correspond to a point on the secp256k1 elliptic curve (the ECDSA curve used by the bitcoin network)
-- The PrivKey is literally a random 32-byte number, which gives the public key when you multiply the generator point by this number.  Yes, a private key is just a random number.  As such, there is no way to identify whether a string of digits is a private key, without having a public key to compare to.  Or rather, every 256-bit number is a private key, so a "private key" is only meaningful in the context of a public key point (x,y).   (all hex numbers are encoded in BigEndian)


Anyone want to help converting pub/priv keypairs into a wallet file?  I believe it can be done with the bsddb package in Python, but I haven't gotten it to work, myself, yet.

Joric's pywallet is the tool you need
However, I modified it a bit and it's more practical

Basically I added the possibility to import a key:
 - with its label
 - in a wallet not named 'wallet.dat'
 - as a reserve key (the hidden ones not shown in bitcoin adress book)

Here are both:
https://github.com/jackjack-jj/pywallet
https://github.com/joric/pywallet



Edit: I now read your entire message
I made a script too to backup the keys of a wallet into a new one automatically using my pywallet, without writing keys to the hdd
You may take a look: http://forum.bitcoin.org/index.php?topic=31418.0

[etotheipi]

This tool doesn't have to backup to the HDD, it's just that there's no other place to put it, at the moment.  I wanted this for my own use, to have a human-readable list of keys -- for backup or for easy input into other scripts/programs I want to write to do things with the keys without having to understand the wallet.dat format.   

As I expected, I'm not the first person to extract keys from a wallet, but I did want to recreate the wallet without the transaction history.  Sounds like pywallet will get me there.

[marcus_of_augustus]

04 20

 

[jackjack]

Pywallet can't import hex keys yet, please wait a few hours