something i didn't realize when backing up my wallet

[unk1911]

i was under the impression that if i back up my wallet at time T0  and continue to use the MAIN wallet to send and receive bitcoins,  that if i open the backed-up wallet at time T1, assuming i'm sync'ed with the blockchain,  that my backed up wallet would reflect all of the transactions.  however, this doesn't appear to be the case.  there's a finite number of addresses that the bitcoin client keeps around (in a pool of 100 addresses) which get used up every time you send a payment.  if you run out of your addresses,  new addresses will be added to the pool but your BACKED UP wallet.dat file will not be privy to these new addresses and thus if you are unfortunate enough to ever lose the original file, you will forever lose some fraction of your coins.. this is because whenever you send bitcoins,  it generates 2 transactions:  tx1 goes to the recipient, tx2 gets sent to one of the 100 addresses that are stored in your account pool... so if these were depleted and new ones added, the backup wallet doesn't know about it and thus the loss of bitcoin can occur... was i like the only one who didn't know about this?   now i have to be a lot more careful and backup the wallet more frequently i guess...

[1715415556]

1715415556

[1715415556]

1715415556

[1715415556]

1715415556

[420]

Clients with deterministic wallets (Armory, for example) do not have this problem.

could you later import your wallet into one of those and be good? (get those 'lost' coins back)

[DeathAndTaxes]

Clients with deterministic wallets (Armory, for example) do not have this problem.

could you later import your wallet into one of those and be good? (get those 'lost' coins back)

No once lost the coins are lost.  The coins are "lost" because the wallet generates an address not in the backup and if no updated backup is made there is no record of the private key (if the original is lost/damaged).   There is nothing to import, they private key is missing.

You can however expand the keypool with the -keypool=xxxx command line argument.  Make the keypool 1,000 keys to give yourself some more "slack" between backups.

[unk1911]

Clients with deterministic wallets (Armory, for example) do not have this problem.

could you later import your wallet into one of those and be good? (get those 'lost' coins back)

No once lost the coins are lost.  The coins are "lost" because the wallet generates an address not in the backup and if no updated backup is made there is no record of the private key (if the original is lost/damaged).   There is nothing to import, they private key is missing.

You can however expand the keypool with the -keypool=xxxx command line argument.  Make the keypool 1,000 keys to give yourself some more "slack" between backups.

thanks for the advice, DeadAndTaxes.  will do that to give myself a bit more runway.

[jbreher]

Most computer files work this way. If you haven't backed up the latest version, you lose all changes since the last backup. I don't know why this would be a surprise.

The qt wallet gives you 100 changes worth of bonus slop, and deterministic wallets give you unlimited slop. But that is not the norm for backups in general.

[Singlebyte]

Unfortunately new users believe they are backing up their one private key.  Not realizing behind the scenes new one are being created.  New wallets should somehow make the user aware of this so they know to make new backups.

[auzaar]

When wallet creates a new key automatically, it should ask for a backup with warning and get confirmation before proceeding, this is a very silly bug in software claiming to be ver 0.8

[DeathAndTaxes]

When wallet creates a new key automatically, it should ask for a backup with warning and get confirmation before proceeding, this is a very silly bug in software claiming to be ver 0.8

So every transaction you mean.  You want the wallet to warn you to backup after every single send?

[BitThink]

I didn't get it. Do you mean there will be a new address generated after each payment, or after 100 payments? Currently, i can only see the addresses manually created by myself, and I will only backup when I created a new address. I am using multibit.

[auzaar]

When wallet creates a new key automatically, it should ask for a backup with warning and get confirmation before proceeding, this is a very silly bug in software claiming to be ver 0.8

So every transaction you mean.  You want the wallet to warn you to backup after every single send?

Why would wallet create a new key after each transaction, can't/doesn't it have a pool of keys?

[DeathAndTaxes]

I didn't get it. Do you mean there will be a new address generated after each payment, or after 100 payments? Currently, i can only see the addresses manually created by myself, and I will only backup when I created a new address. I am using multibit.

In the QT client the keypool is continually refreshed.  While the keypool contains 100 keys every send requires a new change address so the oldest key from the keypool is used AND a new key created and added to the keypool.  The keypool is always at 100 keys (baring some unusual situations where wallet can't be unlocked).

So yes the wallet would need to warn you to backup every single time you sent a transaction OR clicked New Address.

[auzaar]

In the QT client the keypool is continually refreshed.  While the keypool contains 100 keys every send requires a new change address so the oldest key from the keypool is used AND a new key created and added to the keypool.  The keypool is always at 100 keys (baring some unusual situations where wallet can't be unlocked).

So yes the wallet would need to warn you to backup every single time you sent a transaction OR clicked New Address.

So what is the benefit of keeping the pool always 100? instead why not let it deplete and when it reaches threshold refill the pool and ask for backup?

[DeathAndTaxes]

When wallet creates a new key automatically, it should ask for a backup with warning and get confirmation before proceeding, this is a very silly bug in software claiming to be ver 0.8

So every transaction you mean.  You want the wallet to warn you to backup after every single send?

Why would wallet create a new key after each transaction, can't/doesn't it have a pool of keys?

It does but it doesn't allow the keypool to run out.  It adds a new key to the keypool everytime one is used.

Imagine a wallet has a keypool which contains 100 address.  We will number them 1 to 100.

So:
Active Addresses: None
Keypool: 1 ... 100

If you took a backup right now you will notice you have all the addresses backed up to #100.


You need a new address.  The wallet pulls the oldest key from the keypool "1".  It immediately add a NEW key to the keypool lets call that 101.

Active Addresses: 1
Keypool: 2 ... 101

Next address used
Active Addresses: 1, 2
Keypool: 3 ... 102


48 more addresses used.
Active Addresses: 1, 2 .... 50
Keypool: 51 ... 150

50 more addresses used
Active Addresses: 1, 2 .... 100
Keypool: 101 ... 200

If you used another address it wouldn't be included in your backup.

[DeathAndTaxes]

In the QT client the keypool is continually refreshed.  While the keypool contains 100 keys every send requires a new change address so the oldest key from the keypool is used AND a new key created and added to the keypool.  The keypool is always at 100 keys (baring some unusual situations where wallet can't be unlocked).

So yes the wallet would need to warn you to backup every single time you sent a transaction OR clicked New Address.

So what is the benefit of keeping the pool always 100? instead why not let it deplete and when it reaches threshold refill the pool and ask for backup?

It would make backups before the pool exhausts useless.  As it works now any backup made at anytime covers all active addresses and the NEXT 100 future addresses.   It also means that prior backs (if not overwritten) likely are also effective and can be used as a backup for the backup if you do more than 1 backup per 100 key uses.

My keypool is 1,000 keys.  My highest weekly usage ever was ~130 keys.  I backup automatically once a week with sequentially named backups.  The likelihood of losing any keys is essentially zero.

[auzaar]

It would make backups before the pool exhausts useless.

isn't that a good thing

[DeathAndTaxes]

It would make backups before the pool exhausts useless.

isn't that a good thing

No unless you like losing keys, unpredictability, or bitcoind halting your automated system because you didn't do a backup you didn't know you needed to do until it halted.

As it stands now every backup is useful.  It resets the "future key clock" to the limit of the pool (default 100).  So if you do a daily or weekly backup your backup will ALWAYS have all keys and the next 100 future keys.

Backups should always be periodic.  Setup your system to backup the wallet.dat once per week and set your keypool large enough to cover two weeks usage and at all times you should have two valid backups (in case of a file corruption issue).   Nothing to think about, no risk of creating keys which aren't backed up, no intrusive system where bitcoind has to prevent you from using your own money because it just foolishly exhausted the keypool and needs you to stop and manually make a backup right now.

[auzaar]

No unless you like losing keys, unpredictability, or bitcoind halting your automated system because you didn't do a backup you didn't know you needed to do until it halted.

I don't understand, if bitcoin-qt instead of generating new keys every-time did it periodically then wallet state will not change so often, it will also enable it to ask for backup when wallet state changes, that way user knows when to do backup, currently only safe way is to backup after every transaction.

e.g if it generates 1000 keys to start with and uses those only and once it reaches 100, generates 900 more and asks user to back it up, what is the harm in that?

[auzaar]

It would make backups before the pool exhausts useless.

isn't that a good thing

Backups should always be periodic.

No, if data is critical (lots of money) it should not be periodic it should be backuped at each change, if I do weekly backup I risk loosing some coins.

[DeathAndTaxes]

No unless you like losing keys, unpredictability, or bitcoind halting your automated system because you didn't do a backup you didn't know you needed to do until it halted.

I don't understand, if bitcoin-qt instead of generating new keys every-time did it periodically then wallet state will not change so often, it will also enable it to ask for backup when wallet state changes, that way user knows when to do backup, currently only safe way is to backup after every transaction.

e.g if it generates 1000 keys to start with and uses those only and once it reaches 100, generates 900 more and asks user to back it up, what is the harm in that?

bitcoind is often used in automated processes.  I much prefer my company wallet being automatically backed up every day and knowing I have multiple backups then an intrusive system which waits to refill the keypool and forces me to do a manual backup on a unpredictable schedule.

What if the user makes a backup right before the keypool exhausts and being a user (think your grandmother) assumes he is "covered" he just made a backup 5 minutes ago it must be good.  Then they keypool refreshes making his backup 5 minutes ago worthless?

What happens if you are right in the middle of sending out a batch of payments.  Will the system halt and lock you out of your own money until it forces you to make a backup?   Will it actively prevent you from using the wallet.   If it doesn't and the wallet fails you just lost keys and funds.

What happens if the backup is corrupted?  Your prior backups are completely useless because backups between refreshes contain less and less new keys?

Of course the client is open source.  Feel free to fork it and implement a different backup system.    Honestly given all the potential use and applications I can't see a more universal solution than make regular backups and the backup will contain all used keys and X future keys.  Still this is somewhat academic, long term all wallets will be deterministic anyways.

[Mageant]

I think a good feature of the Bitcoin client would be a warning after making 50 transactions or so without making a backup (executing the backup command).