Can the encrypted wallet be recovered using the unencrypted?

[sethsethseth]

Here is what happened:

Made new wallet using satoshi client on windows 7
Made a backup
Sent a large amount of bitcoins to it
Realized the wallet wasn't encrypted, and encrypted it
Apparently did not back up the encrypted wallet
Made a few tx using the encrypted wallet, causing the balance to get sent as change to an address on the encrypted version
Got laptop stolen
Spent the last couple days trying to figure out why the backup wallet had a zero balance and couldn't see the change that had come back to it in the next tx

Hoping it is possible to replicate the process and recover the encrypted version.  There are enough bitcoins on this wallet to put a huge amount of work into recovering them.  I would guess that the encryption process is repeatable, but if there are any random factors in the new generation of 100 addresses, I am screwed.

[1714881798]

1714881798

[1714881798]

1714881798

[1714881798]

1714881798

[1714881798]

1714881798

[1714881798]

1714881798

[Revalin]

Yes, your coins are probably still there.  The client pre-generates a bunch of addresses for change, so it's likely all there.

Shut it down and rerun it in a cmd window: "bitcoin-qt.exe -rescan".

[CIYAM]

Any change addresses that had been used prior to the encryption will still be present (otherwise the encrypted wallet would not be able to find those funds) it is only those addresses in the key pool that were unused (as far as the client could tell at that time) will have been discarded so as per Revalin's advice do a rescan and all should be good.

[sethsethseth]

edited OP to state that I made tx after I encrypted the wallet

[CIYAM]

edited OP to state that I made tx after I encrypted the wallet

In that case unfortunately you really have a problem as the unused addresses in the original backed up key pool were discarded when the wallet was encrypted (and re-encrypting the backup with the same password won't help as am pretty sure the addresses will be random).

[Revalin]

Ouch, I wasn't aware of that - why would it throw away the pool?

[CIYAM]

Ouch, I wasn't aware of that - why would it throw away the pool?

Ironically (for the OP) this is done just in case someone managed to get a copy of the unencrypted wallet.

[sethsethseth]

Something needs to be changed if I really can't get these coins back.  It is reasonable for me to think that if I have an unencrypted file, back it up, and then encrypt one of them, that I can derive the encrypted version from the unencrypted one if I know the password.  There should be a warning in there somewhere.  I am sick over losing this many coins.

[CIYAM]

This type of recovery is possible with deterministic wallets but AFAIA the Satoshi client doesn't support this.

Sorry to give you such bad news (and would be happily corrected if I've got anything wrong).

One thing that probably should be added to the Satoshi client is a message to tell you to backup your wallet as soon as you have encrypted it.

[sethsethseth]

It is remotely possible there is a deleted version of the encrypted backup on a thumb drive, if someone can recommend a program to unerase....

[CIYAM]

That's great - be sure not to copy any new files to it before using file recovery software.

You could try this software: http://www.nchsoftware.com/data-recovery/index.html?gclid=CM3ktL3Ww7ICFS5U4godHhkA4w

(I haven't used it myself though so can't personally recommend it)

[scintill]

Some topics that might help (recovering using hex editor from raw disk image):
https://bitcointalk.org/index.php?topic=14698.0
https://bitcointalk.org/index.php?topic=7116.0
https://bitcointalk.org/index.php?topic=8274.0
https://bitcointalk.org/index.php?topic=25091.0

I say might because none of them seem to indicate if this works for encrypted wallets.  Good luck!

[flatfly]

It is remotely possible there is a deleted version of the encrypted backup on a thumb drive, if someone can recommend a program to unerase....

I can vouch for tokiwa datarecovery, it's free and fantastic. Saved my ass many times.

http://tokiwa.qee.jp/EN/dr.html

[Puppet]

It is remotely possible there is a deleted version of the encrypted backup on a thumb drive, if someone can recommend a program to unerase....

If you didnt overwrite that space, there is a very good chance of being able to recover it. Use testdisk:
http://www.cgsecurity.org/wiki/TestDisk

I would also advice you to make a copy of the drive first. Not a regular copy of course, but using dd under linux so you copy "every bit" of the drive.
Boot linux and run

dd if=/dev/yourusbdrive of=backup.dat

That way if you mess up the undelete, nothing is lost and you can try again.

[Remember remember the 5th of November]

Well this is what happens when people mindlessly encrypt their wallets. Obviously better safe than sorry, but again, this is what happens when people mindlessly encrypt their wallets.

[JoelKatz]

Ironically (for the OP) this is done just in case someone managed to get a copy of the unencrypted wallet.

Typically, this is what a user would want. The reason you opt to encrypt the wallet is to prevent someone from being able to get your coins without knowing the key to decrypt the wallet. It is generally understood that your coins are only assuredly protected if you backup your wallet after your last transaction.

However, I 100% firmly believe that non-deterministic wallets should be abandoned entirely as a failed experiment. Perhaps they should be left as an option with clear warnings about their problems.

People keep losing their money and it's getting ridiculous. One of the biggest problems with Bitcoin is it's damn near impossible to use it safely.

[caveden]

However, I 100% firmly believe that non-deterministic wallets should be abandoned entirely as a failed experiment. Perhaps they should be left as an option with clear warnings about their problems.

+1

While the Satoshi client doesn't use deterministic wallets, an alternative for its users would be to have the wallet.dat be a symbolic link towards a safe backup account, like Wuala for example.
This way your backup is updated on real-time.

People keep losing their money and it's getting ridiculous. One of the biggest problems with Bitcoin is it's damn near impossible to use it safely.

You use strong words, but yeah, you have to understand in details how things work to be safe. We cannot expect that of most people. I still don't recommend my parents for example, to store wealth in bitcoins. And they're not totally computer-illiterate, they know how to send e-mails, to chat, to use facebook and watch youtube. My father even has a blog. But I don't think they would be able to protect their keys. It's a pity.

[JoelKatz]

You use strong words, but yeah, you have to understand in details how things work to be safe. We cannot expect that of most people. I still don't recommend my parents for example, to store wealth in bitcoins. And they're not totally computer-illiterate, they know how to send e-mails, to chat, to use facebook and watch youtube. My father even has a blog. But I don't think they would be able to protect their keys. It's a pity.

I'm a computer security person, and I don't trust myself to keep a significant amount of money in Bitcoins. It's just too difficult to balance making sure you can always access the coins no matter what might go wrong with making sure others can never access them no matter what might go wrong.

[bitvientiane]

I think at this point in time everybody's best bet is an electrum deterministic wallet, like Joel hinted at.

[sethsethseth]

The reason you opt to encrypt the wallet is to prevent someone from being able to get your coins without knowing the key to decrypt the wallet.

However, I 100% firmly believe that non-deterministic wallets should be abandoned entirely as a failed experiment. Perhaps they should be left as an option with clear warnings about their problems.

It is generally understood that your coins are only assuredly protected if you backup your wallet after your last transaction.

This is absolutely not generally understood, and the main dev team should not assume this.  Multiple people with thousands of posts have told me that the wallet can always be recovered as long as you don't use more than 100 addresses on it after the backup.  The 100 address generation is the part that is generally understood.  The client should accommodate this view.  If I have the password, I should not be able to destroy the coins before 100 addresses if it is common knowledge that you cannot.  The new 100 keys should be able to be generated from the unencrypted ones if you have the password.

People keep losing their money and it's getting ridiculous. One of the biggest problems with Bitcoin is it's damn near impossible to use it safely.

You use strong words, but yeah, you have to understand in details how things work to be safe. We cannot expect that of most people. I still don't recommend my parents for example, to store wealth in bitcoins. And they're not totally computer-illiterate, they know how to send e-mails, to chat, to use facebook and watch youtube. My father even has a blog. But I don't think they would be able to protect their keys. It's a pity.

I'm a computer security person, and I don't trust myself to keep a significant amount of money in Bitcoins. It's just too difficult to balance making sure you can always access the coins no matter what might go wrong with making sure others can never access them no matter what might go wrong.

A large part of the reason I think I did not back up as well as I should have is that I didn't want someone with access to the wallet file I was storing on the internet to be able to see the balance and transactions.  I always reencrypted the wallet file, making it more of a hassle to do a backup.  This may be a stupid line of thinking, but it may be helpful for devs to know this in making the client more user-friendly and secure.

I have spent several hours a day for the last two years reading the forums, and still made this error.  I knew it was risky to put them on my computer, but I thought with my knowledge of bitcoin it was less risky than keeping them online.  Obviously I was wrong.

That said, I had to make at least 4 catastrophic errors that are not at all logical to allow this to happen.  If I only made any 3 of these errors I would still have the coins:

Used a wallet with a huge balance as a main wallet to conduct tiny transactions on bitcoin-otc
Only backed up the wallet on dropbox.  Need to email, dropbox, and flash drive the backup
Left laptop in insecure location
Didn't do a wallet backup when I made an account on OTC a few days after encrypting the wallet.  There was a warning to do so.

I lost over 1300 btc.  Where does that put me on the loser's list of losses due to carelessness?

[helloworld]

I lost over 1300 btc.  Where does that put me on the loser's list of losses due to carelessness?

My guess is "nowhere near the top of the list" unfortunately.

[JoelKatz]

The developers and in fact the entire community have failed you and lots of other people. This and the investment scams are the two biggest problems facing Bitcoin right now.

[scintill]

I lost over 1300 btc.

Ouch, sorry.  When you said, "There are enough bitcoins on this wallet to put a huge amount of work into recovering them" I misread and thought there weren't very many coins.

So, have you learned anything about the possible backup on the thumb drive?  For this much money, I think it worth it to hire someone trustworthy with experience in this.  I think Casascius has recovered (unencrypted?) wallets and seems willing to do it for a smallish payment.

Good luck.

[flatfly]

Approaching this from a different angle (assuming the thief hasn't found and spent the coins already - hopefully your password was strong enough, but check on blockchain.info whether the coins have moved):

try everything you can to recover the stolen laptop. This may be possible with some luck and if the thief isn't too smart.

You mentioned you are a dropbox user. Try the following: log on to the dropbox website and check your access logs - perhaps you'll discover the thief's ip address. Some people have been known to track and recover their stolen laptop thanks to dropbox logs. 

Or perhaps you have some other software running on that laptop allowing you to remotely access it? Teamviewer, remote desktop, ftp, telnet perhaps? 

[Revalin]

This is absolutely not generally understood, and the main dev team should not assume this.

+1.  It caught me by surprise.

I lost over 1300 btc.  Where does that put me on the loser's list of losses due to carelessness?

Ouch.  That's enough to warrant some serious exploration of your USB stick.  Definitely make a dd block dump of it before you do anything.  Even if the undelete utilities can't find anything, we may be able to scrounge something.  There's a good utility here for recovering unencrypted keys by looking for their raw signature:  https://bitcointalk.org/index.php?topic=25091.0 .  If they were encrypted you can probably place a bounty for someone to extend the program to recognize encrypted keys.

[Jutarul]

Ouch.  That's enough to warrant some serious exploration of your USB stick.  Definitely make a dd block dump of it before you do anything.  Even if the undelete utilities can't find anything, we may be able to scrounge something.  There's a good utility here for recovering unencrypted keys by looking for their raw signature:  https://bitcointalk.org/index.php?topic=25091.0 .  If they were encrypted you can probably place a bounty for someone to extend the program to recognize encrypted keys.

+1

There are several threads where raw data inspection has been discussed to find out whether wallet data is on the deleted drive. E.g. a recommended a procedure is a deep scan, here:
https://bitcointalk.org/index.php?topic=91702.msg1016998#msg1016998

If you are not familiar with these techniques, find someone trustworthy who is.

[Dabs]

What's the update on the OP? You can try piriform's recuva. That works on deleted files that have not been wiped.

The current bitcoin client (satoshi) is not making any specific warnings when encrypting the wallet about making a backup.

I did an experiment and made a backup of my wallet file, encrypted it, and made a backup of the encrypted wallet file. I then used pywallet to extract the keys and made a comparison.

It seems that my unencrypted wallet has 108 keys. The encrypted wallet has 208 keys. Getting the keys into a spreadsheet and doing a basic lookup, and / or dupe check, it seems the encrypted wallet now has 100 new additional keys, in addition to the existing keys that were in the unencrypted wallet.

This tells me, than the unencrypted wallet has all the old keys, the encrypted wallet has new keys, but it's possible that the change bitcoins were sent to one of the new keys instead of the old keys.

When you lost the new encrypted wallet, you essentially lost the change bitcoins that were sent to one of the new keys.

In my experiment, all the new keys are of the compressed type (private keys beginning with L or K instead of 5.)

[sethsethseth]

Still a tiny chance of recovering the laptop.  The encrypted wallet is not on the flash drives.  In my mind at the time, I thought that encrypting the wallet was a repeatable process just like encrypting any other file, so I did not think to back it up again.

[Jutarul]

Still a tiny chance of recovering the laptop.  The encrypted wallet is not on the flash drives.  In my mind at the time, I thought that encrypting the wallet was a repeatable process just like encrypting any other file, so I did not think to back it up again.

Yes the developers really screwed that one up. That's why bitcoin still deserves beta status.

[casascius]

To answer the question as to whether the deleted encrypted wallet material is still on the drive, I think it should be deterministically possible to confirm it or rule it out with a simple disk scan with a hex editor.

My understanding is that an encrypted wallet is simply an unencrypted database with encrypted records in it.  I think the wallet file itself, the database overhead being unencrypted, could easily be found on a hard drive simply by scanning all disk sectors for unique string markers that are plenty easy to find with a tool like WinHex.

I don't have a computer with an encrypted wallet.dat handy, but I will bet I could find tons of strings in the file that would be highly likely to appear in every wallet.dat, and I am sure others can provide a suitable search string as well.

[Jutarul]

My understanding is that an encrypted wallet is simply an unencrypted database with encrypted records in it.  I think the wallet file itself, the database overhead being unencrypted, could easily be found on a hard drive simply by scanning all disk sectors for unique string markers that are plenty easy to find with a tool like WinHex.

Exactly. https://bitcointalk.org/index.php?topic=91702.msg1016998#msg1016998

I suppose the OP explored this option already. If not - I recommend reading the advice more carefully...

[casascius]

My understanding is that an encrypted wallet is simply an unencrypted database with encrypted records in it.  I think the wallet file itself, the database overhead being unencrypted, could easily be found on a hard drive simply by scanning all disk sectors for unique string markers that are plenty easy to find with a tool like WinHex.

Exactly. https://bitcointalk.org/index.php?topic=91702.msg1016998#msg1016998

I suppose the OP explored this option already. If not - I recommend reading the advice more carefully...

Ironically, I don't think the wallet file stores Bitcoin addresses in human-readable text form anywhere in the wallet.dat, even in an unencrypted wallet.  I believe it only stores the actual underlying public key and hash, which are just random-looking binary data.  So even if this wallet existed on the drive, if I'm right about this, the script wouldn't be able to find it.

You would, rather, be searching for strings that appear, such as "blockindex", "bestblock", "pool"... the most valuable record is "key", and surely there has got to be a few relatively static bytes that could be searched.

[scintill]

I believe it only stores the actual underlying public key and hash, which are just random-looking binary data.

Maybe if he knows an address, he could search for that in hex?  (Going on the assumption there might be something to recover.)

I just pulled two addresses out of my wallet, one I received coins at and one I sent coins to; base58-decoded them into hex at brainwallet.org, and searched wallet.dat for them in a hex editor, found both in the clear in my encrypted wallet.  (This is pretty disconcerting, depending on what you were hoping an encrypted wallet would do for you.)

So, if you do this on a whole disk and find stuff, there's a good chance they're wallet fragments.  From there I guess you start looking for nearby markers like Casascius talked about.

[Jutarul]

My understanding is that an encrypted wallet is simply an unencrypted database with encrypted records in it.  I think the wallet file itself, the database overhead being unencrypted, could easily be found on a hard drive simply by scanning all disk sectors for unique string markers that are plenty easy to find with a tool like WinHex.

Exactly. https://bitcointalk.org/index.php?topic=91702.msg1016998#msg1016998

I suppose the OP explored this option already. If not - I recommend reading the advice more carefully...

Ironically, I don't think the wallet file stores Bitcoin addresses in human-readable text form anywhere in the wallet.dat, even in an unencrypted wallet.  I believe it only stores the actual underlying public key and hash, which are just random-looking binary data.  So even if this wallet existed on the drive, if I'm right about this, the script wouldn't be able to find it.

You would, rather, be searching for strings that appear, such as "blockindex", "bestblock", "pool"... the most valuable record is "key", and surely there has got to be a few relatively static bytes that could be searched.

BTC addresses are stored in binary form. The referenced script works, since the tool grep searches for the supplied key in binary form. Try it out. Replace /dev/sdaX with your wallet file.

[flatfly]

Errr.. Guys, read the OP (and subsequent posts) again. Your advice is good but unfortunately not relevant at this stage. Before any of that can happen, the laptop must be recovered from the thief.

[Jutarul]

Errr.. Guys, read the OP (and subsequent posts) again. Your advice is good but unfortunately not relevant at this stage. Before any of that can happen, the laptop must be recovered from the thief.

https://bitcointalk.org/index.php?topic=110781.msg1206155#msg1206155

[flatfly]

Sorry, I should've emphasized today's update:

Still a tiny chance of recovering the laptop.  The encrypted wallet is not on the flash drives.  

[Jutarul]

Sorry, I should've emphasized today's update:

Still a tiny chance of recovering the laptop.  The encrypted wallet is not on the flash drives.  

Yes. That assumes the OP has done a deep scan. casascius implicitly asked the OP to confirm that.

[Dabs]

There are many laptop tracking software available. A commercial one is Lo Jack for computers. An open source one is Prey.

BUT, they have to be installed and activated prior to your laptop being stolen. (Works with desktops and smart phones too.)

[sethsethseth]

I do not believe I backed the encrypted wallet up on the flash drive.  If on some off chance I did, it is not showing up on the recovery tools.  Perhaps I will mess with this hex thing once I determine if I cannot recover the laptop

[flatfly]

I do not believe I backed the encrypted wallet up on the flash drive.  If on some off chance I did, it is not showing up on the recovery tools.  Perhaps I will mess with this hex thing once I determine if I cannot recover the laptop

Have you tried to check your Dropbox access logs as I mentioned?
If you can't find them, let me know and I'll help you out.

[sethsethseth]

yeah i checked that.  the computer had a logon password, and they didnt steal the charger, and it was constantly crashing and I had to boot it like 8 times to get it to load windows properly, so the thief would probably just wipe it.  Probably still recoverable if I get it back though.

[kjj]

Still a tiny chance of recovering the laptop.  The encrypted wallet is not on the flash drives.  In my mind at the time, I thought that encrypting the wallet was a repeatable process just like encrypting any other file, so I did not think to back it up again.

Yes the developers really screwed that one up. That's why bitcoin still deserves beta status.

https://github.com/bitcoin/bitcoin/issues/1884

Don't be too hard on the devs.  Wallet encryption has been around for like a year, and we are just now noticing this problem, which suggests that it is not obvious except in hindsight.

[FactoredPrimes]

Why does the standard client always send the change to a new address?? It would make more sense to send the change back to one of the sending addresses.

[kjj]

Why does the standard client always send the change to a new address?? It would make more sense to send the change back to one of the sending addresses.

Sending to a new address leaks the least privacy.  Also, there is no concept of a sending address in the bitcoin system.

[sunnankar]

Something needs to be changed if I really can't get these coins back.  It is reasonable for me to think that if I have an unencrypted file, back it up, and then encrypt one of them, that I can derive the encrypted version from the unencrypted one if I know the password.  There should be a warning in there somewhere.  I am sick over losing this many coins.

As far as creating a safe and secure backup of private keys, what about (1) using an offline computer to create a wallet address via bitaddress.org's html/javascript then (2) placing the keys into a TrueCrypt volume and (3) transferring the TrueCrypt volume, or offline generated transaction, via USB to an online computer and backing up to multiple places like Dropbox, AWS, Google Drive, etc.

Here is the code to create offline wallets and generate offline transactions.

Is there a flaw in this procedure that I am not seeing?

[FreeMoney]

Why does the standard client always send the change to a new address?? It would make more sense to send the change back to one of the sending addresses.

Sending to a new address leaks the least privacy.  Also, there is no concept of a sending address in the bitcoin system.

Sending back to one of the input addresses could be an optional setting.