Bitcoin Forum
April 16, 2014, 08:35:11 PM *
News: ♦♦ A bug in OpenSSL, used by Bitcoin-Qt/Bitcoin Core, could allow your bitcoins to be stolen. Immediately updating Bitcoin Core to 0.9.1 is required in some cases, especially if you're using 0.9.0. Download. More info.
The same bug also affected the forum. Changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2  All
  Print  
Author Topic: Wallet.Dat Recovery... Recover Your Own Lost Bitcoins!  (Read 8964 times)
casascius
Mike Caldwell
VIP
Hero Member
*
Offline Offline

Activity: 1204


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW

Ignore
June 26, 2011, 07:41:04 AM
 #1

I have been doing some tinkering around, thinking about other people's wallet disasters, and believe I have come to the following conclusion...

If you have lost your wallet.dat for whatever reason (deleted it, formatted your drive, file corruption, etc.) it's possible that it may still be lurking on your computer.  If so, recovery is no longer purely theoretical.  With a little knowledge of what to search for, you can use a hex editor to potentially find usable remnants of your wallet.dat file and get back your bitcoins, even if the original file isn't fully recoverable.

So here goes...

If you can use a hex-editor to do a sector-by-sector search/edit on your entire hard drive, then search your entire hard drive for occurrences of the following byte sequence:

01 03 6B 65 79 41 04...........

the middle four of these bytes represent the string "keyA" in ASCII.

Each time this byte sequence occurs, a Bitcoin private key is probably stored nearby, about 180 bytes later.  The 32-byte private key is the only thing you need to recover your bitcoins!... as long as you find the right one(s).

Approximately 180 bytes after this sequence, you may find the byte sequence 04 20 (hex).  These two bytes seem to precede every private key (the 0x20 suggests a length of 32 bytes).  If you find this sequence, the thirty-two bytes that come after 04 20 are the private key representing a Bitcoin address and might be the private key that recovers some of your lost bitcoins!  Your wallet will have numerous private keys (at least one hundred, due to the pre-allocation of keys)... get as many as you can find.  Carefully search the sectors adjacent to any sector containing the "keyA" sequence above.  Then yell for help!  (But don't share the private keys in public, unless you want to give away your wallet.)

An example of a hex editor that can scan an entire disk volume for specific byte sequences for Windows is WinHex.  In WinHex, use Tools, Open Disk (F9), and choose the disk you want to scan.  Scanning a full disk can take hours.  WinHex must "run as administrator" to be able to scan a physical disk.  Someone please recommend a good way to do this in Linux, preferably with a known Live CD, if possible.  Also, any time you are scanning a disk for potentially lost data, you should NEVER boot the disk you're searching - always boot from another disk and install the target disk as secondary.

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
1397680511
Hero Member
*
Offline Offline

Posts: 1397680511

View Profile Personal Message (Offline)

Ignore
1397680511
Reply with quote  #2

1397680511
Report to moderator
BitcoinBULLBEARGET THE UNBEATABLE
BITCOIN TRADING FORECAST

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1397680511
Hero Member
*
Offline Offline

Posts: 1397680511

View Profile Personal Message (Offline)

Ignore
1397680511
Reply with quote  #2

1397680511
Report to moderator
1397680511
Hero Member
*
Offline Offline

Posts: 1397680511

View Profile Personal Message (Offline)

Ignore
1397680511
Reply with quote  #2

1397680511
Report to moderator
1397680511
Hero Member
*
Offline Offline

Posts: 1397680511

View Profile Personal Message (Offline)

Ignore
1397680511
Reply with quote  #2

1397680511
Report to moderator
1397680511
Hero Member
*
Offline Offline

Posts: 1397680511

View Profile Personal Message (Offline)

Ignore
1397680511
Reply with quote  #2

1397680511
Report to moderator
niooron
Full Member
***
Offline Offline

Activity: 190


View Profile

Ignore
June 26, 2011, 08:08:12 AM
 #2

Would this work on SSDs? I heard they don't let you physically scan the flash memory.

14dxwuQwkQiLbZjJFfciZ26xSGdRU5mKEp
casascius
Mike Caldwell
VIP
Hero Member
*
Offline Offline

Activity: 1204


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW

Ignore
June 26, 2011, 08:12:57 AM
 #3

Would this work on SSDs? I heard they don't let you physically scan the flash memory.

Yes, you can do it on SSD's, it's just that your chances of success will be somewhat lower.  When you scan a disk in this manner, you are simply doing a sector-by-sector read of the entire disk, which will always succeed (in the sense that the disk can be read end-to-end without running into an error or a protest from the disk)... but it just may not turn up any lost bitcoins.


Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
dserrano5
Staff
Hero Member
*****
Offline Offline

Activity: 686



View Profile

Ignore
June 26, 2011, 11:45:42 AM
 #4

Say bye-bye to this method as soon as wallets are encrypted on disk.

PGP :: OTC :: Localbitcoins :: bitcoind -addnode=bk5ejfe56xakvtkk.onion
I prefer to be sent PGP-encrypted PMs :: Prefiero recibir MPs cifrados con PGP.
-
Jr. Member
*
Offline Offline

Activity: 50


View Profile

Ignore
June 26, 2011, 12:49:26 PM
 #5

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.
chungy
Jr. Member
*
Offline Offline

Activity: 38


View Profile

Ignore
June 26, 2011, 01:01:29 PM
 #6

Wouldn't help if the file alone was encrypted (say, with GPG or similar).  Still, even if that's implemented in the client, I hope such a feature would be optional, it might contain a warning such as: Encrypting your wallet may help thwart theft of your wallet file, though it also diminishes chances of recovery on accidental file deletion.
dserrano5
Staff
Hero Member
*****
Offline Offline

Activity: 686



View Profile

Ignore
June 26, 2011, 02:43:56 PM
 #7

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.

I was talking about encrypted wallet as implemented by the bitcoin client, which is a strongly demanded feature these days. Once deleted, it's lost forever. Keep backups!

PGP :: OTC :: Localbitcoins :: bitcoind -addnode=bk5ejfe56xakvtkk.onion
I prefer to be sent PGP-encrypted PMs :: Prefiero recibir MPs cifrados con PGP.
jackjack
Hero Member
*****
Offline Offline

Activity: 504


May Bitcoin be touched by his Noodly Appendage


View Profile

Ignore
June 26, 2011, 02:45:59 PM
 #8

Say bye-bye to this method as soon as wallets are encrypted on disk.

It still works if you can decrypt the archive (or the entire disk) before doing the search. So mount the encrypted volume, then do the search.

I was talking about encrypted wallet as implemented by the bitcoin client, which is a strongly demanded feature these days. Once deleted, it's lost forever. Keep backups!
If bitcoin use encrypted wallets, copying the wallet.dat several times becomes possible without danger

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
amincd
Hero Member
*****
Offline Offline

Activity: 669


View Profile

Ignore
July 13, 2011, 07:09:28 AM
 #9

Casascius, may I suggest considering to offer this as a service?

You already have the skill and knowledge to do a search for the wallet data, and are relatively trusted, so others might prefer to send their hard-drives to you in the event of an accidental deletion of a wallet than do it themselves.
smoothie
Hero Member
*****
Offline Offline

Activity: 882


Lealana Physical Bitcoins and Litecoins


View Profile

Ignore
July 13, 2011, 07:41:49 AM
 #10

Wouldn't a windows system restore bring back a deleted wallet file?

Maged
Global Moderator
Hero Member
*
Offline Offline

Activity: 1064


View Profile

Ignore
July 13, 2011, 08:04:57 AM
 #11

Casascius, may I suggest considering to offer this as a service?

You already have the skill and knowledge to do a search for the wallet data, and are relatively trusted, so others might prefer to send their hard-drives to you in the event of an accidental deletion of a wallet than do it themselves.
This is now pretty much automated:
http://forum.bitcoin.org/index.php?topic=25091.0

amincd
Hero Member
*****
Offline Offline

Activity: 669


View Profile

Ignore
July 13, 2011, 01:23:47 PM
 #12

Quote from: smoothie
Wouldn't a windows system restore bring back a deleted wallet file?

I think in some cases that would work. Windows doesn't create restore points all the time, so I think there's a high chance that the data that a person deletes will not have been in existence when the latest restore point was created.

By the way, looking at your profile pic makes me want to drink a smoothie.

Quote from: Mage
This is now pretty much automated:
http://forum.bitcoin.org/index.php?topic=25091.0

Thanks for that. This should be mentioned in the bitcoin wiki if it already isn't.
etotheipi
Hero Member
*****
Offline Offline

Activity: 1036


Core Armory Developer


View Profile WWW

Ignore
July 23, 2011, 10:24:36 PM
 #13

Is it expected that the private key would come way after the public key?  I'm doing a brute force search of my wallet file, using my crypto library to check whether each sequence of 64 consecutive bytes is on the secp256k1 elliptic curve (meaning it's a public key), then searching for a 32-byte sequence that gives you this public key when you interpret it as the private key.  What I am finding is that (1) this is really slow, (2) not every public key has an associated private key in the wallet file and (3) the private keys I'm finding are located about 200-300KB past where the public key was found.

This works under the assumption that both keys are encoded in big-endian and the public keys are encoded as [0x04 [X] [Y]] and private keys are encoded as [0x0420 [secretInt]].  But it doesn't work well enough to be confident that I'm extracting everything--why am I finding so many public keys that don't have matching private keys?   Could keys be stored in different endiannesses?  are some of them not preceded by '0x0420'?

I'd like to see the keys stored in a flat file that makes key recovery a million times easier.  The keypairs can be encoded as constant-length binary strings, just like the headers are serialized.  The private keys can be encrypted individually in this flat file, without encrypting the entire file.  Instead, you just convert the 256-bit private-keys to a 256-bit encrypted-private-key before writing it to file.  Then your wallet file still works for tracking transactions (since the public keys/addresses are still in plaintext), but the private key will be useless without transforming it back to the unencrypted bitstring.


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
casascius
Mike Caldwell
VIP
Hero Member
*
Offline Offline

Activity: 1204


The Casascius 1oz 10BTC Silver Round (w/ Gold B)


View Profile WWW

Ignore
July 24, 2011, 04:23:47 PM
 #14

But it doesn't work well enough to be confident that I'm extracting everything--why am I finding so many public keys that don't have matching private keys?   Could keys be stored in different endiannesses?  are some of them not preceded by '0x0420'?

Are you searching just wallet.dat or your entire hard drive?  The block chain database is going to be loaded with numerous public keys - belonging to other people, of course.

Each record containing a key in wallet.dat also contains the ASCII text "key" nearby as well.  (Because wallet.dat can hold multiple kinds of records, and this is how the software knows that this record is a key.)

Companies claiming they got hacked and lost your coins sounds like fraud so perfect it could be called fashionable.  I never believe them.  If I ever experience the misfortune of a real intrusion, I declare I have been honest about the way I have managed the keys in Casascius Coins.  I maintain no ability to recover or reproduce the keys, not even under limitless duress or total intrusion.  Remember that trusting strangers with your coins without any recourse is, as a matter of principle, not a best practice.  Don't keep coins online. Use paper wallets instead.
etotheipi
Hero Member
*****
Offline Offline

Activity: 1036


Core Armory Developer


View Profile WWW

Ignore
July 24, 2011, 06:54:20 PM
 #15

I'm only searching the wallet.dat file.  I wanted to get my keys into a flat file for fun, maybe come up with a way to convert between the wallet.dat and wallet.flat.txt, which would make manual key management and recovery easier.

I figured out that the other keys, I believe, are addresses to which I've sent coins before... the ones that show up under the "Sent" addresses tab in the client. 

However, it does seem that there are multiple instances of your public keys in the file.  One of my public keys (for which I have the private key) shows up 10+ times.  Others show up two or three times.  I guess the wallet file holds transaction information, in addition to the keys themselves.


Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
etotheipi
Hero Member
*****
Offline Offline

Activity: 1036


Core Armory Developer


View Profile WWW

Ignore
July 25, 2011, 10:34:27 PM
 #16

So, I have created this script which pulls out every public and private key in your wallet and stores them into flat files.  The "keylistpub.txt" will contain every public key in the wallet file whether it's yours or not, and "keylistpair.txt" contains a list of just the keys for which there is an associated private key in the wallet, and includes both.  This script should avoid duplicate keys in the output files.

http://dl.dropbox.com/u/1139081/extractKeys.tar.gz

Keep in mind, this utility is going to create a new file on your computer with your private keys.   Handle with care! 

Now, I want to create something that converts the flat file back into a wallet.  This will allow someone to extract keys from multiple wallets, combine them into a single file, and the create a merged wallet.  The output file format looks like this:

Code:
1MRAs5doMqqbLQVuAUqGcxHBzrexMiTBG:
PubKey:
cfd41f6ab9a217380bd2dc370592635797759c7de172f5cc6b228c1d4f83dde2
44f5a373bf80e66db4c0d34a892def09d1f605aef0d94f6b2c3e0322dfdd331e
PrivKey:
7bb1e283fe1007757c75966706553e16cdb5f148c22712811a78e6bcf30c9a1b

1QXg28gA7mLBB9LSMgf4sjoB9batJBXEtB:
PubKey:
47634c35731a35b5b70d4959418dae2e1c6676a1007626092eef8bceb80e1b16
0d048e2a917c80a3f5f085a06ce4c88f78d66c82abf2f2a1683c171f8bbdb7ab
PrivKey:
6cd77c1cc66929e6db9bf4b502f4ce4868cb76037b66d630fd931a0ea2fb8bce
...
(don't get too excited, I have mangled the private keys)

-- The first string is the address, which should be fairly obvious. 
-- The PubKey is two 32-byte numbers (x,y), which correspond to a point on the secp256k1 elliptic curve (the ECDSA curve used by the bitcoin network)
-- The PrivKey is literally a random 32-byte number, which gives the public key when you multiply the generator point by this number.  Yes, a private key is just a random number.  As such, there is no way to identify whether a string of digits is a private key, without having a public key to compare to.  Or rather, every 256-bit number is a private key, so a "private key" is only meaningful in the context of a public key point (x,y).   (all hex numbers are encoded in BigEndian)


Anyone want to help converting pub/priv keypairs into a wallet file?  I believe it can be done with the bsddb package in Python, but I haven't gotten it to work, myself, yet.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
jackjack
Hero Member
*****
Offline Offline

Activity: 504


May Bitcoin be touched by his Noodly Appendage


View Profile

Ignore
July 25, 2011, 10:41:33 PM
 #17

So, I have created this script which pulls out every public and private key in your wallet and stores them into flat files.  The "keylistpub.txt" will contain every public key in the wallet file whether it's yours or not, and "keylistpair.txt" contains a list of just the keys for which there is an associated private key in the wallet, and includes both.  This script should avoid duplicate keys in the output files.

http://dl.dropbox.com/u/1139081/extractKeys.tar.gz

Keep in mind, this utility is going to create a new file on your computer with your private keys.  Handle with care!  

Now, I want to create something that converts the flat file back into a wallet.  This will allow someone to extract keys from multiple wallets, combine them into a single file, and the create a merged wallet.  The output file format looks like this:

Code:
1MRAs5doMqqbLQVuAUqGcxHBzrexMiTBG:
PubKey:
cfd41f6ab9a217380bd2dc370592635797759c7de172f5cc6b228c1d4f83dde2
44f5a373bf80e66db4c0d34a892def09d1f605aef0d94f6b2c3e0322dfdd331e
PrivKey:
7bb1e283fe1007757c75966706553e16cdb5f148c22712811a78e6bcf30c9a1b

1QXg28gA7mLBB9LSMgf4sjoB9batJBXEtB:
PubKey:
47634c35731a35b5b70d4959418dae2e1c6676a1007626092eef8bceb80e1b16
0d048e2a917c80a3f5f085a06ce4c88f78d66c82abf2f2a1683c171f8bbdb7ab
PrivKey:
6cd77c1cc66929e6db9bf4b502f4ce4868cb76037b66d630fd931a0ea2fb8bce
...
(don't get too excited, I have mangled the private keys)

-- The first string is the address, which should be fairly obvious.  
-- The PubKey is two 32-byte numbers (x,y), which correspond to a point on the secp256k1 elliptic curve (the ECDSA curve used by the bitcoin network)
-- The PrivKey is literally a random 32-byte number, which gives the public key when you multiply the generator point by this number.  Yes, a private key is just a random number.  As such, there is no way to identify whether a string of digits is a private key, without having a public key to compare to.  Or rather, every 256-bit number is a private key, so a "private key" is only meaningful in the context of a public key point (x,y).   (all hex numbers are encoded in BigEndian)


Anyone want to help converting pub/priv keypairs into a wallet file?  I believe it can be done with the bsddb package in Python, but I haven't gotten it to work, myself, yet.
Joric's pywallet is the tool you need
However, I modified it a bit and it's more practical

Basically I added the possibility to import a key:
 - with its label
 - in a wallet not named 'wallet.dat'
 - as a reserve key (the hidden ones not shown in bitcoin adress book)

Here are both:
https://github.com/jackjack-jj/pywallet
https://github.com/joric/pywallet



Edit: I now read your entire message
I made a script too to backup the keys of a wallet into a new one automatically using my pywallet, without writing keys to the hdd
You may take a look: http://forum.bitcoin.org/index.php?topic=31418.0

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
etotheipi
Hero Member
*****
Offline Offline

Activity: 1036


Core Armory Developer


View Profile WWW

Ignore
July 25, 2011, 10:51:09 PM
 #18

This tool doesn't have to backup to the HDD, it's just that there's no other place to put it, at the moment.  I wanted this for my own use, to have a human-readable list of keys -- for backup or for easy input into other scripts/programs I want to write to do things with the keys without having to understand the wallet.dat format.   

As I expected, I'm not the first person to extract keys from a wallet, but I did want to recreate the wallet without the transaction history.  Sounds like pywallet will get me there.

Founder and CEO of Armory Technologies, Inc.
Armory Bitcoin Wallet: Bringing cold storage to the average user!
Only use Armory software signed by the Armory Offline Signing Key (0x98832223)

Please donate to the Armory project by clicking here!    (or donate directly via 1QBDLYTDFHHZAABYSKGKPWKLSXZWCCJQBX -- yes, it's a real address!)
marcus_of_augustus
Hero Member
*****
Offline Offline

Activity: 1134



View Profile

Ignore
July 26, 2011, 12:12:43 AM
 #19

04 20

 Wink

Monetary Freedom - a basic human right
"Disarming money as a tool for tyranny."
"Disintermediating the State."
jackjack
Hero Member
*****
Offline Offline

Activity: 504


May Bitcoin be touched by his Noodly Appendage


View Profile

Ignore
July 26, 2011, 12:30:02 AM
 #20

Pywallet can't import hex keys yet, please wait a few hours

Own address: 19QkqAza7BHFTuoz9N8UQkryP4E9jHo4N3 - Pywallet support: 1AQDfx22pKGgXnUZFL1e4UKos3QqvRzNh5 - Bitcointalk++ script support: 1Pxeccscj1ygseTdSV1qUqQCanp2B2NMM2
Pywallet: instructions. Encrypted wallet support, export/import keys/addresses, backup wallets, export/import CSV data from/into wallet, merge wallets, delete/import addresses and transactions, recover altcoins sent to bitcoin addresses, sign/verify messages and files with Bitcoin addresses, recover deleted wallets, etc.
Pages: [1] 2  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!