Help - lost bitcoins (now found!) !!

[mintymark]

So I had a machine crash. I was not that worried I had several backups of wallet.dat.

I wiped the .bitcoin directory and coppied wallet.dat and waited for blockchain download.

I was surprised that a ballance came up almost at once and that it was about 1200 BTC short, but I waited for the download.

Now the download has completed, it seems that many transactions are missing, payments to addresses created a long time ago seem present. Those to recently created addresses not so.

I have done a bitcoin --rescan, and that made no difference.

Now the various backups of wallet.dat that I have are significantly different in size.

Perhaps I just have to do the reload with a different, older or bigger wallet.dat?

Advice?

[1715666988]

1715666988

[1715666988]

1715666988

[Deafboy]

Sorry to hear that. Yes, recently created keys are probably gone
To prevent loosing your BTCs in future, add keypool=somelargenumber to bitcoin.conf. Default value is 100, so you need to back up your wallet after every 100 used addresses.

To recover most of your bitcoins, use most recent backup you have.

[Blazr]

Are all of your addresses showing up in the "Receive coins" tab?

[mintymark]

No, the recent addresses are missing, but I do believe I may not have used the most recent backup.

At the moment swapped the wallet.dat, and am doing a rescan.

In fact it really doesnt make any difference how long this takes, but from a biting the fingernails approach, it would seriously reduce my stress levels if I knew these were safe.

Now how could I forget that a recent backup of wallet.dat was needed, I thought any wallet.dat backup would do.

Really kicking myself here.

[mintymark]

There is hope, the recent wallet.dat shows the correct ballance, with 4 blocks to download, so I am hopeful that this will work out well.

I feel a bit stupid for bothering everyone with this.....

[Jaw3bmasters]

There is hope, the recent wallet.dat shows the correct ballance, with 4 blocks to download, so I am hopeful that this will work out well.

I feel a bit stupid for bothering everyone with this.....

This thread reminds me of a quote....
"One often meets his destiny on the road he takes to avoid it."

I'm guessing you probably have two firewalls on a Windows system with several cold-storage all over the place....

[Stephen Gornick]

I feel a bit stupid for bothering everyone with this.....

Don't -- it is a good reminder to others who might just have made a recent backup after empathizing with you.

The wallet.dat contains, by default. a key pool of the next 100 addresses your client will use.  An address is consumed each time you click "New Address" and then each time a change transaction (back to yourself) is made it pulls one address from the key pool.   The keypool is topped up after each time an address is drawn from it.  (with a few exceptions).

So you as long as your backup is newer than the past 100 transactions it should have all the keys in it.

[mintymark]

Well, it seems I'm not out of the woods yet. Its been an hour, and still there are 4 blocks to download. It still says Last received block was generated 17 minutes ago.

So maybe I need to delete the blockchain data and download all from scratch.

I tried making a small payment to myself and got:

A fatal error occurred. Bitcoin can no longer continue safely and will quit.

EXCEPTION: St13runtime_error       
CDB() : can't open database file wallet.dat, error -30974       
bitcoin in Runaway exception       

So now I have a choice. Redownload the entire blockchain using this wallet, or use a slightly older backup that was like this one, approx double the size of earlier ones.  That one was a better quality backup in that it was taken from a non-running bitcoin if I remember.

Which to try first?

[DannyHamilton]

. . . Redownload the entire blockchain using this wallet, or use a slightly older backup . . . Which to try first?

Which ever you try, make sure you keep a copy of this wallet.dat.

If you find that the slightly older wallet.dat doesn't have the private keys for all of your addresses, you could alwyas try using a tool like pywallet to extract the missing private keys from this wallet and import them into the slightly older one.

[Stephen Gornick]

So now I have a choice. Redownload the entire blockchain using this wallet, or use a slightly older backup that was like this one

If you know what the balance should be (i.e., to know if the backup was recent enough) simply rename the existing wallet.dat (to something like wallet.err) and restore from the other backup.   [Edit: Use -rescan if needed] If the balance is correct, use that one.

If that did not have all the keys, then you want to try to salvage the wallet (available since v0.7.1):

* -salvagewallet command-line option, which moves any existing wallet.dat
  to wallet.{timestamp}.dat and then attempts to salvage public/private
  keys and master encryption keys (if the wallet is encrypted) into
  a new wallet.dat. This should only be used if your wallet becomes
  corrupted, and is not intended to replace regular wallet backups.

[John (John K.)]

Keep multiple backups of the wallet.dat file FIRST.
Is the blockchain finished? Or is it still downloading?

[farlack]

For $16,000 it might be a good idea to send the hard drive to a data recovery company.

[casascius]

His wallet file sounds completely recoverable without needing hard drive recovery.

[Evolvex]

Arghhh what a nightmare, hope your get your bitcoins back dude!

Happy new year everyone btw 

[🏰 TradeFortress 🏰]

Your bitcoins are safe. You should not have deleted the blockchain.

[Jutarul]

For his own sake I hope he didn't use the same hard drive for the next system.

If he used the same drive, chances are slim - but not 0%. If the harddrive is big enough there's a good chance that the old data is still physically stored on the old drive and thus recoverable with specialized software tools. Switch off and use a different drive to test your wallet backups.

[Gyrsur]

so please tell me what was the intention of the core developers to create a keypool with serveral addresses? do one address fix this issue?

keypool=1

thanks!

[drakahn]

so please tell me what was the intention of the core developer to create a keypool with serveral addresses? do one address fix this issue?

keypool=1

thanks!

keypool = 1 means you would need a new backup with every transaction

[Gyrsur]

so please tell me what was the intention of the core developer to create a keypool with serveral addresses? do one address fix this issue?

keypool=1

thanks!

keypool = 1 means you would need a new backup with every transaction

this is not my understanding of how it works. if you use only one public/private key and you make a transaction you send bitcoins to the receiver and the rest of your bitcoins will go to this one public/private key which you created thru "keypool=1". isn't it?

[Scrat Acorns]

this is not my understanding of how it works. if you use only one public/private key and you make a transaction you send bitcoins to the receiver and the rest of your bitcoins will go to this one public/private key which you created thru "keypool=1". isn't it?

A new address will be generated to hold the change.

[DannyHamilton]

this is not my understanding of how it works. if you use only one public/private key and you make a transaction you send bitcoins to the receiver and the rest of your bitcoins will go to this one public/private key which you created thru "keypool=1". isn't it?

Bitcoin-Qt never sends change outputs back to the same address that the inputs came from.  It always uses a new address for the change.  The keypool is a pool of unused addresses to be used in the future whenever the client needs a new address for some purpose.  This means that when you create a backup, you have backed up the next {keypool} number of addresses that will be used.  That way you can use that many addresses as new receiving addresses or change addresses and if you recover the backup, the coins will still be accessible, since the addresses were backed up before you used them. When the client uses an address from the keypool, it generates a new address to add to the keypool to keep the keypool at the same size.

If your keypool=1, and then you perform a backup, you will have backed up only the one next address to be used.

If you then send some coins in a transaction that requires change, you'll use that one address from the keypool for the change, and a new address (that is not part of the previous backup) will be generated and added to the keypool.

If you don't backup before your next transaction, then the next transaction with change will use this new address (which is not in the backup) from the keypool for your change.

If you then recover your backup, you will lose all the bitcoins that were sent to the most recent change address since that address isn't in the backup.

Example:
My wallet has a total of 100 BTC and it was all received in a single transaction meaning that it is a single "output". I have keypool=1, and so far I have never sent or received any other bitcoins.

My receiving address is 1abcdef...
My wallet has a hidden unused keypool address of 1zyxwvu...

I perform a backup of my wallet and the backup now includes the 2 addresses listed above.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1abcdef... has 100 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... has 100 BTC
1zyxwvu... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... has 100 BTC
1zyxwvu... has never been used and therefore has 0 BTC

I send 1 BTC to a friend who has a receiving address of 1hijklm...

My Bitcoin-Qt client creates a transaction that has 1 input and 2 outputs:

Code:

INPUT: The 100 BTC output from 1abcdef... (my initial receiving address)
OUTPUT: 1 BTC to 1hijklm... (my friends receiving address)
OUTPUT: 99 BTC to 1zyxwvu... (my original keypool address used as a change address)

My Bitcoin-Qt generates a new address 1nopqrs... to replace the address that was taken from the keypool.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1zyxwvu... has been used as a change address, has 99 BTC
1hijklm... my friend's address has 1 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... has been used as a change address, has 99 BTC
1nopqrs... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... has been used as a change address, has 99 BTC

So I can still access my 99 BTC if I recover the backup, since that address was in the backed-up keypool.

I send a 3 BTC transaction to pay a merchant for some purchase at their receiving address of 1tsrqpo...

My Bitcoin-Qt client creates a transaction that has 1 input and 2 outputs:

Code:

INPUT: The 99 BTC output from 1zyxwvu... (my previous change address)
OUTPUT: 3 BTC to 1tsrqpo... (the merchants receiving address)
OUTPUT: 96 BTC to 1nopqrs... (my new keypool address used as a change address)

My Bitcoin-Qt client generates a new address 1tuvwxy... to replace the address that was taken from the keypool.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1nopqrs... has been used as a change address, has 96 BTC
1hijklm... my friend's address has 1 BTC
1tsrqpo... merchant's address has 3 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... all outputs spent, has 0 BTC
1nopqrs... has been used as a change address, has 96 BTC
1tuvwxy... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... all outputs spent, has 0 BTC

I can no longer access my 96 BTC if I recover the backup, since that address was NOT in the backed-up keypool.

[mintymark]

Well I am happy to say there was a happy solution to this. The machine it was running on became progressively less reliable and eventually refused to boot. It was a thermal or power supply problem. At last a good use for one of those old bitcoin mining machines that are still on my credit card, dam it! A new machine with the same harddisk is now running. And I have downloaded the blockchain to resync bitcoin-qt from scratch.

This is clearly what happened. After the first sudden power down event, the existing blockchain data and wallet.dat seemed to be corrupt.
I then used an old backup of wallet.dat that unknown to me was taken before the new chunk of receiving addresses were generated - none of the new addresses including some big transactions where in the resynced client! Hence my panic. At this stage I found a more recent backup up used that and all was well.

But lets look a little more deeply into this ... (Because believe me I am psychologically scarred, wouldnt you be?? )
Lets suppose I didnt have that backup. I think recovery would still be really easy. All I would need to do would be to generate some new receiving addresses, and suddenly a new chunk of addresses would be generated. (Or force this using command line arguments.) Because this is a deterministic process, suddenly, poof, my bitcoins would come back. I mean one wallet.dat is based on one initial set of private keys, so from that point of view, any old wallet.dat backup will do wont it? Please be aware I am speculating here, I am saying you only need one back up of wallet.dat, but I am saying it quietly for fear of misleading people in case I am wrong, and also because its clearly more convenient to have more recent backups given a choice. 

Is this right?

[Stephen Gornick]

Because this is a deterministic process
[...]
I mean one wallet.dat is based on one initial set of private keys

Nope.  The Bitcoin.org client might someday use a deterministic wallet, but today it does not.

Additionally, those clients that do you a deterministic wallet and also allow the user to import private keys also need to be backed up after each import due to those Bitcoin addresses not being generated from the deterministic key.

And to reiterate:

The wallet.dat contains, by default. a key pool of the next 100 addresses your client will use.  An address is consumed each time you click "New Address" and then each time a change transaction (back to yourself) is made it pulls one address from the key pool.   The keypool is topped up after each time an address is drawn from it.  (with a few exceptions).

So you as long as your backup is newer than the past 100 transactions it should have all the keys in it.