Lost Savings Wallet Addresses?!

[Sad Puppy]

I think I just managed to lose a large number of BTC.  Here's what happened:

1. I had a wallet with all my BTC.  I quit Bitcoin (version 0.3.21) and renamed the entire Bitcoin directory Bitcoin-checking.
2. I re-opened Bitcoin, which created a new Bitcoin directory and downloaded all the blocks again.
3. I copied the address shown, quit Bitcoin, renamed this directory Bitcoin-savings, encrypted it as Bitcoin-savings-encrypted, and saved it in multiple remote locations.
4. I renamed Bitcoin-checking to Bitcoin, then restarted the Bitcoin application.
5. I sent 0.02 BTC to the address from step 3.
6. I quit Bitcoin and renamed the Bitcoin directory to Bitcoin-checking.
7. I unencrypted a copy of Bitcoin-savings-encrypted, renamed the directory to Bitcoin, and restarted the Bitcoin application.
8. My 0.02 BTC showed up in this savings wallet.
9. I copied another address, quit Bitcoin again, renamed the directory as Bitcoin-savings, swapped in Bitcoin-checking and sent lots of BTC to this new savings address.
10. I never updated the Bitcoin-savings-encrypted file after step 3, because I thought the wallet automatically contained 100 pre-generated addresses to start.
11. I securely deleted my unencrypted Bitcoin-savings directory with multiple passes.
12. Later I unencrypted a copy of the Bitcoin-savings-encrypted directory, renamed it Bitcoin, opened the Bitcoin app, and only my original 0.02 BTC are shown even after all the new blocks are downloaded.

So it looks like I lost all the BTC I transferred to my savings wallet!  I downloaded bitcointools from here:
https://github.com/gavinandresen/bitcointools
and viewed the contents of my savings wallet.dat from Bitcoin-savings-encrypted with this command:
python dbdump.py --wallet

The output only shows a single PubKey and PrivKey pair (where I sent 0.02 BTC).  It also shows two lines that say "Unknown key type: bestblock".

I was under the impression that wallets automatically have 100 pre-generated keys as soon as the wallet is created as mentioned here:
https://en.bitcoin.it/wiki/Securing_your_wallet

So why did my savings wallet that I encrypted in step 3 only have a single address?  I clearly completely screwed up by not updating Bitcoin-savings-encrypted after the large transfers, but I thought I only needed to do that after 100 keys had been used for 100 transactions.  What went wrong?  When does the Bitcoin application actually create those 100 queued keys?  Does it only created the pool of pre-generated keys after the first address is actually used?

- Very Sad Puppy

[1714140447]

1714140447

[1714140447]

1714140447

[fornit]

assuming that you use windows, the wallet.dat is usually not in the bitcoin directory but in C:\Users\Username\AppData\Roaming\Bitcoin

as i understand it, you made several bitcoin directories but did nothing to backup the actual wallet.dat?

[Sad Puppy]

I'm using Mac OS 10.6.7.  I definitely backed up the directory containing the actual wallet.dat file.  My initial 0.02 BTC transfer showed up in the wallet.dat file that I restored from my Bitcoin-savings-encrypted. 

On a Mac, the Bitcoin directory is at:
/Users/username/Library/Application\ Support/Bitcoin

[vuce]

I can confirm, newly created wallet shows as having only one key pair with dbdump.py.

I also had a problem where I lost a few bitcoins that were sent to a new address, when restoring the wallet from backup (the difference was the last transaction only).

[Sad Puppy]

I can confirm, newly crated wallet shows as having only one key pair with dbdump.py.

I also had a problem where I lost a few bitcoins that were sent to a new address, when restoring the wallet from backup (the difference was the last transaction only).

Thanks for confirming.  This makes me very depressed.  Everything I had read made me believe that 100 pre-generated addresses were created for new wallets.

If I re-open the Bitcoin application with this wallet.dat, wait for it to show my 0.02 BTC transaction, then quit Bitcoin, I can use dbdump.py to see that it now has 100 addresses.  However, none of them are the addresses where I transferred all my savings.  I guess the pool of 100 addresses isn't created until a transaction is completed.  We should update the Wiki to clarify this.  I would edit the Wiki myself, but I'm busy contemplating jumping out a window in despair.

[fornit]

I would edit the Wiki myself, but I'm busy contemplating jumping out a window in despair.

was it that much?

[Sad Puppy]

I would edit the Wiki myself, but I'm busy contemplating jumping out a window in despair.

was it that much?

Yes, unfortunately.  I hate to admit it, but I lost 7208 BTC.  At least for me, that's a lot.  My savings just got destroyed.  Ugggggghhhh.  I seriously can't believe it.  I'm totally screwed.

[theymos]

There is a bug where the creation of the first-ever address (with label "Your Address") does not trigger filling of the keypool. The keypool is only filled once the next address is created, and then it takes tens of minutes to fill completely.

[Sad Puppy]

There are many instances where the Bitcoin client doesn't properly associate the transaction history with the wallet.  For example, if while running with wallet A, a block comes in with coins for wallet B, and you then substitute wallet B and rerun Bitcoin, the coins won't appear in the client even though they should.  What fixes this is if you remove all the data files other than wallet.dat, and download the block chain again.

Unless you have actually destroyed (deleted) a wallet.dat and irreversibly lost private keys you need, you are probably not screwed.

When reviving a wallet, start Bitcoin with a data directory that contains only that wallet.dat and let it rebuild everything.

I suppose if you are familiar with the wallet dumper, you are probably already familiar with how to compare the addresses in your wallet(s) to the block chain.  Though at no point do you say you have deleted a wallet, so I'm not sure how any key could be lost.

I tried starting Bitcoin with only wallet.dat and let it rebuild from scratch, but it only has the 0.02 BTC.

The problem is that I initially created a new empty savings wallet, encrypted it, and backed it up.  That encrypted savings wallet apparently only had 1 single address.  I only have a copy of that initial encrypted savings wallet with one address (although I thought it was supposed to have 100 addresses).

After my large transfers to the savings wallet, I did delete my later copy of the wallet.dat.  I thought my initial encrypted copy of the savings wallet had 100 pre-generated addresses, but it didn't.  Newly created wallets DO NOT actually have a key pool of 100 address, they just have 1 single address.  

[vuce]

There is a bug where the creation of the first-ever address (with label "Your Address") does not trigger filling of the keypool. The keypool is only filled once the next address is created, and then it takes tens of minutes to fill completely.

tried it, it works. Interesting that this hasn't yet been fixed in the latest beta.

I suppose addresses that are created internally also don't trigger this?

[SgtSpike]

I would edit the Wiki myself, but I'm busy contemplating jumping out a window in despair.

was it that much?

Yes, unfortunately.  I hate to admit it, but I lost 7208 BTC.  At least for me, that's a lot.  My savings just got destroyed.  Ugggggghhhh.  I seriously can't believe it.  I'm totally screwed.

Wow, that sucks. 

[Sad Puppy]

There is a bug where the creation of the first-ever address (with label "Your Address") does not trigger filling of the keypool. The keypool is only filled once the next address is created, and then it takes tens of minutes to fill completely.

I guess I can confirm this bug, and I demonstrated it in a spectacular fashion.  

The sad thing is, I think the 100-address pool is meant to help protect against loss like happened to this guy:
http://forum.bitcoin.org/index.php?topic=782.0

But if this 100-address pseudo protection didn't exist, I would have been obsessively making completely new backups after every transaction.  The 100-address pool sort of tricked me into a false sense of security.

I know it's my fault for relying on the pool without verifying that it actually existed.  I don't think I've ever been this depressed, I'm actually crying.  I can't believe this happened.

[casascius]

Wow, that sucks. 

I second that.  And hope that the BTC lost was part of a BTC windfall, not something you traded some other valuable asset for.  

I would of course advise not jumping out the window regardless of how painful it must be.

[SgtSpike]

You might try slaving your hard drive to a second computer and using some file recovery software to try to recover the wallet.dat.  Of course, I would advise immediately ceasing use of the computer (just powering it off with the power button would be better than shutting down, lest the wallet file be overwritten by some files that are written during shutdown).  A decent piece of software might cost $50.

If that doesn't work, there are professional labs that will read the actual data on the disk, if it hasn't already been overwritten.  A decent lab would cost $1000 or so.  Don't cheap out on this if you try it.  I did, and I regret it, a lot.

And beyond that, if you use a mechanical hard drive, forensics labs can actually tell what bits were written to a given sector BEFORE the current bits, because the previous bits leave some sort of magnetic bias behind.  That's why military erasing standards dictate that a drive must be overwritten 7 times with random bits before the data that was on it is considered securely erased.  I have no idea what it might cost to get a forensics expert to look at your drive though.

[unk]

And beyond that, if you use a mechanical hard drive, forensics labs can actually tell what bits were written to a given sector BEFORE the current bits, because the previous bits leave some sort of magnetic bias behind.  That's why military erasing standards dictate that a drive must be overwritten 7 times with random bits before the data that was on it is considered securely erased.  I have no idea what it might cost to get a forensics expert to look at your drive though.

unfortunately, this is essentially an urban myth. for more information, see the epilogue to gutmann's updated paper on secure erasure from magnetic media. i don't believe there's even one reported case of data recovery after a single pseudorandom pass of overwriting. it would take something like manual scanning using an electron microscope if it were even possible at all. as gutmann wrote:

with the ever-increasing data density on disk platters and a corresponding reduction in feature size and use of exotic techniques to record data on the medium, it's unlikely that anything can be recovered from any recent drive except perhaps a single level via basic error-cancelling techniques... with modern high-density drives, even if you've got 10KB of sensitive data on a drive and can't erase it with 100% certainty, the chances of an adversary being able to find the erased traces of that 10KB in 200GB of other erased traces are close to zero.

if the data was overwritten by even one pseudorandom pass, it would take well more than the bitcoins' value to recover them. it won't be worth the time and expense.

it is worth walking through the drive with software to see whether you indeed overwrote wallet.dat. some wiping programs don't do what you expect, either because they're poorly written or because the filesystem is more complicated than the case for which they were intended. if that doesn't work, though, you'd do better to assume there's no magical solution in hardware.

[Sad Puppy]

You might try slaving your hard drive to a second computer and using some file recovery software to try to recover the wallet.dat.  Of course, I would advise immediately ceasing use of the computer (just powering it off with the power button would be better than shutting down, lest the wallet file be overwritten by some files that are written during shutdown).  A decent piece of software might cost $50.

If that doesn't work, there are professional labs that will read the actual data on the disk, if it hasn't already been overwritten.  A decent lab would cost $1000 or so.  Don't cheap out on this if you try it.  I did, and I regret it, a lot.

And beyond that, if you use a mechanical hard drive, forensics labs can actually tell what bits were written to a given sector BEFORE the current bits, because the previous bits leave some sort of magnetic bias behind.  That's why military erasing standards dictate that a drive must be overwritten 7 times with random bits before the data that was on it is considered securely erased.  I have no idea what it might cost to get a forensics expert to look at your drive though.

Unfortunately, I securely deleted the plain text wallet before I realized there was an issue, so it was probably overwritten many times with random bits.  I was basically following the steps I found in a thread describing how to create a secure savings wallet.  

[casascius]

Doesn't Mac use a journaled file system?

Is journaling for the directory info only, or does it include files?  In NTFS, small files AFAIK are just dumped into the directory inline, rather than given their own sectors, which would make them journaled.

Bottom line, it is remotely possible that the journaling functionality may have left that data on your drive in a way you could never "find" it except through a brute force search.  Who knows for sure if that's the case.  But - you  know one key - so a brute force search of the drive to rule it out isn't out of the question.

Although recovery is very voodoo, there is a specific method that could be followed:

0 - if considering it, power off now...don't shut down properly, just grab the electrical cord and pull it out of the wall.

1 - you already know ONE of the keys in this lost wallet.dat, right?  Find the bytes comprising the representation of the one known key from the wallet.dat you kept, and then take that hard drive out, use another machine and do a sector-by-sector search for those bytes from another machine.  Of course you'll probably bump into the file you kept, but maybe you'll bump into fragments of the deleted file.  And if you do, you may never know how it survived.

You don't need recovery of the file, just the few (32?) bytes representing the private key of the address you lost, which might be in the same sector adjacent to the key you have.  the wallet can be rebuilt if you know just that number.

If you exported the lost wallet to a text file at any point, then by all means, search for the plaintext of the known key too, where the plaintext of the lost key could be expected adjacent.

This is pie in the sky of course, but if some sector on your drive contains that key, this is also doable without any science fiction.

[bitlotto]

I had something similar happen to me. I tried -rescan but it still didn't work. The only way I could get it to work was to delete everything, re-download the blockchain, replace the wallet, then run -rescan.   
Good luck getting it back. I'd have a panic attack losing that much!

[unemployed]

Dude make a copy of your whole hard drive with dd command. Maybe there is nonzero chance, that your file was copied somewhere? Maybe the filesystem stored it in two places??
What's the point of doing secure deletion on your own computer anyway?

[BitCointransfers]

I feel bad for you dude and hope you can recover it all back......Did you mine these coins or buy them?