Help - lost bitcoins (now found!) !!

[DannyHamilton]

this is not my understanding of how it works. if you use only one public/private key and you make a transaction you send bitcoins to the receiver and the rest of your bitcoins will go to this one public/private key which you created thru "keypool=1". isn't it?

Bitcoin-Qt never sends change outputs back to the same address that the inputs came from.  It always uses a new address for the change.  The keypool is a pool of unused addresses to be used in the future whenever the client needs a new address for some purpose.  This means that when you create a backup, you have backed up the next {keypool} number of addresses that will be used.  That way you can use that many addresses as new receiving addresses or change addresses and if you recover the backup, the coins will still be accessible, since the addresses were backed up before you used them. When the client uses an address from the keypool, it generates a new address to add to the keypool to keep the keypool at the same size.

If your keypool=1, and then you perform a backup, you will have backed up only the one next address to be used.

If you then send some coins in a transaction that requires change, you'll use that one address from the keypool for the change, and a new address (that is not part of the previous backup) will be generated and added to the keypool.

If you don't backup before your next transaction, then the next transaction with change will use this new address (which is not in the backup) from the keypool for your change.

If you then recover your backup, you will lose all the bitcoins that were sent to the most recent change address since that address isn't in the backup.

Example:
My wallet has a total of 100 BTC and it was all received in a single transaction meaning that it is a single "output". I have keypool=1, and so far I have never sent or received any other bitcoins.

My receiving address is 1abcdef...
My wallet has a hidden unused keypool address of 1zyxwvu...

I perform a backup of my wallet and the backup now includes the 2 addresses listed above.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1abcdef... has 100 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... has 100 BTC
1zyxwvu... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... has 100 BTC
1zyxwvu... has never been used and therefore has 0 BTC

I send 1 BTC to a friend who has a receiving address of 1hijklm...

My Bitcoin-Qt client creates a transaction that has 1 input and 2 outputs:

Code:

INPUT: The 100 BTC output from 1abcdef... (my initial receiving address)
OUTPUT: 1 BTC to 1hijklm... (my friends receiving address)
OUTPUT: 99 BTC to 1zyxwvu... (my original keypool address used as a change address)

My Bitcoin-Qt generates a new address 1nopqrs... to replace the address that was taken from the keypool.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1zyxwvu... has been used as a change address, has 99 BTC
1hijklm... my friend's address has 1 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... has been used as a change address, has 99 BTC
1nopqrs... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... has been used as a change address, has 99 BTC

So I can still access my 99 BTC if I recover the backup, since that address was in the backed-up keypool.

I send a 3 BTC transaction to pay a merchant for some purchase at their receiving address of 1tsrqpo...

My Bitcoin-Qt client creates a transaction that has 1 input and 2 outputs:

Code:

INPUT: The 99 BTC output from 1zyxwvu... (my previous change address)
OUTPUT: 3 BTC to 1tsrqpo... (the merchants receiving address)
OUTPUT: 96 BTC to 1nopqrs... (my new keypool address used as a change address)

My Bitcoin-Qt client generates a new address 1tuvwxy... to replace the address that was taken from the keypool.

Code:

THE PUBLIC BLOCKCHAIN HAS THE FOLLOWING UNSPENT OUTPUTS ASSOCIATED WITH MY TRANSACTIONS: 
1nopqrs... has been used as a change address, has 96 BTC
1hijklm... my friend's address has 1 BTC
1tsrqpo... merchant's address has 3 BTC

MY ACTIVE Bitcoin-Qt CLIENT CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... all outputs spent, has 0 BTC
1nopqrs... has been used as a change address, has 96 BTC
1tuvwxy... has never been used and therefore has 0 BTC

THE BACKUP CONTAINS THE KEYS TO THE FOLLOWING LIST OF ADDRESSES:
1abcdef... all outputs spent, has 0 BTC
1zyxwvu... all outputs spent, has 0 BTC

I can no longer access my 96 BTC if I recover the backup, since that address was NOT in the backed-up keypool.

[mintymark]

Well I am happy to say there was a happy solution to this. The machine it was running on became progressively less reliable and eventually refused to boot. It was a thermal or power supply problem. At last a good use for one of those old bitcoin mining machines that are still on my credit card, dam it! A new machine with the same harddisk is now running. And I have downloaded the blockchain to resync bitcoin-qt from scratch.

This is clearly what happened. After the first sudden power down event, the existing blockchain data and wallet.dat seemed to be corrupt.
I then used an old backup of wallet.dat that unknown to me was taken before the new chunk of receiving addresses were generated - none of the new addresses including some big transactions where in the resynced client! Hence my panic. At this stage I found a more recent backup up used that and all was well.

But lets look a little more deeply into this ... (Because believe me I am psychologically scarred, wouldnt you be?? )
Lets suppose I didnt have that backup. I think recovery would still be really easy. All I would need to do would be to generate some new receiving addresses, and suddenly a new chunk of addresses would be generated. (Or force this using command line arguments.) Because this is a deterministic process, suddenly, poof, my bitcoins would come back. I mean one wallet.dat is based on one initial set of private keys, so from that point of view, any old wallet.dat backup will do wont it? Please be aware I am speculating here, I am saying you only need one back up of wallet.dat, but I am saying it quietly for fear of misleading people in case I am wrong, and also because its clearly more convenient to have more recent backups given a choice. 

Is this right?

[Stephen Gornick]

Because this is a deterministic process
[...]
I mean one wallet.dat is based on one initial set of private keys

Nope.  The Bitcoin.org client might someday use a deterministic wallet, but today it does not.

Additionally, those clients that do you a deterministic wallet and also allow the user to import private keys also need to be backed up after each import due to those Bitcoin addresses not being generated from the deterministic key.

And to reiterate:

The wallet.dat contains, by default. a key pool of the next 100 addresses your client will use.  An address is consumed each time you click "New Address" and then each time a change transaction (back to yourself) is made it pulls one address from the key pool.   The keypool is topped up after each time an address is drawn from it.  (with a few exceptions).

So you as long as your backup is newer than the past 100 transactions it should have all the keys in it.