Key pool feature for safer wallet backup

[satoshi]

SVN rev 163 (ver 0.3.13.3) has the key pool feature.  Pre-generated new keys are aged in a queue before use, so that backups of wallet.dat hold keys you'll use in the future.

For now I made the default pool size 100.  It can be configured with -keypool=.  Be aware, it takes a little time to increase the pool size, so don't go crazy with it.  Disk space is about 1K per key.

I have not addressed the recovery side of this yet.  If you actually did restore an old wallet.dat, I think you may have to delete blk*.dat to rediscover your own transactions during the redownload.

I've only tested this moderately.  You might not want to use this for a website server until it's had some more testing.

[nelisky]

Much appreciated, this feature will be

Now, is there/will there be a way to know how many free keys are in the pool at some moment in time, and trigger generation of a new pool of addresses in a determined time point?

The idea is to be able to schedule backups when new adresses are going to be generated, and to never use an address not already in a backup. These keys will be used for both change and new addresses, correct?

Thank you!

[Gavin Andresen]

RE: how many free keys are in the pool:

By default, there are at least 100 free keys in the pool, always.

When a key is taken out, if the number of free keys drops below 100 (or the -keypool= number) another is generated.

Keys are put back if they're unused-- for example, a key is needed for every miner hashing thread, so if you're on a 4-core machine and turn on coin generation and then turn it back off you'll wind up with 104 keys in the free pool.

But for most people most of the time there will be exactly 100 free keys.

[nelisky]

RE: how many free keys are in the pool:

By default, there are at least 100 free keys in the pool, always.

When a key is taken out, if the number of free keys drops below 100 (or the -keypool= number) another is generated.

Keys are put back if they're unused-- for example, a key is needed for every miner hashing thread, so if you're on a 4-core machine and turn on coin generation and then turn it back off you'll wind up with 104 keys in the free pool.

But for most people most of the time there will be exactly 100 free keys.

So, 100 transactions from now and I'm in the same exact backup problem. It just moves the safety zone into the future "backup now and you'll be safe for X time" X being really small in a very busy service.

Is it possible to have hysteresis in the pool? once it reaches 20 it generates the other 80, or something or the kind? That and getfreekeys on the rpc side and I can make sure I don't need to do backups for every transaction, I don't need to do backups based on time and I'm 100% safe. Or I can do time based pooled backup and be 99% safe.

Even better, hysteresis as described and after every time the server generates new keys for the pool, it does a wallet backup with time signature. My backup job can simply detect new files as save them, a simple rsync would do the job in a guaranteed safe way.

[LZ]

As I understood generated coins will not be in the key pool. Am I right?

[theymos]

As I understood generated coins will not be in the key pool. Am I right?

They will be. Everything is.

Every received IP transaction creates a new address (taken from the pool), so be careful if you are accepting those. Someone could empty your pool in a few minutes.

[LZ]

Okay. Seems I will see Generated string after 120 confirmations.

I am not accepting IP transactions. Thanks.

[Gavin Andresen]

Backup every 30 sendtoaddress or generatenewaddress and you'll be fine-- you should always have at least 3 backup copies of all your keys.

If you're running a very busy service so backing up every 30 is too often, then run with -keypool=1000 and backup at least every 300 sends/generates.

I worry about bitcoin accumulating too many features and not doing any of them very well.  I suppose it wouldn't hurt to add an option so it automatically creates timestamped wallet backups... but should it erase old backups?  (if it doesn't, I KNOW people will forget to erase them and will be upset when their disk fills up with wallet backups and they're left to figure out how to clean up the resulting mess).  Should it encrypt them?  What should it do if an automatic time-stamped wallet backup fails?  When encrypted wallets are implemented, what should happen to old backups if the wallet encryption key is changed?

[nelisky]

Backup every 30 sendtoaddress or generatenewaddress and you'll be fine-- you should always have at least 3 backup copies of all your keys.

You obviously understand that from the software perspective, the two approaches are different, right? Having a fixed trigger to backup is very easy and can be completely stateless, but backup every X whatevers means keeping a running count, harder to get redundant work with redundant data, etc.

If you're running a very busy service so backing up every 30 is too often, then run with -keypool=1000 and backup at least every 300 sends/generates.

I worry about bitcoin accumulating too many features and not doing any of them very well.  I suppose it wouldn't hurt to add an option so it automatically creates timestamped wallet backups... but should it erase old backups?  (if it doesn't, I KNOW people will forget to erase them and will be upset when their disk fills up with wallet backups and they're left to figure out how to clean up the resulting mess).  Should it encrypt them?  What should it do if an automatic time-stamped wallet backup fails?  When encrypted wallets are implemented, what should happen to old backups if the wallet encryption key is changed?

That's why I like simple, the kiss approach, within limits. The perfect approach for me would be manual triggering of key pool refilling, because that way the backup agent makes sure that there are enough keys from the last backup and, failing that, generates a new (variable size based on the average needed in the past) batch. This is very simple on the bitcoind side and very powerful for me.

I just need a way to know if the addresses on the wallet are covered by the last backup, and if there are still leftover addresses in the wallet, without depending on running counts and persistent data not stored in bitcoin directly.

[Gavin Andresen]

I just need a way to know if the addresses on the wallet are covered by the last backup, and if there are still leftover addresses in the wallet, without depending on running counts and persistent data not stored in bitcoin directly.

Keys in the keypool store when they're generated (and the oldest are always used first).   Asking "what's the oldest key in the key pool" seems reasonable, and I think it would give you what you want-- you could compare that timestamp to the timestamp of your last backup to see if you're covered.

Although if you're going to periodically check to see what the oldest timestamp is (or you're going to periodically check to see if bitcoin has written a new timestamped backup file or periodically check WHATEVER) then it seems simpler to me to just periodically always call backupwallet.  Disk space and bandwidth is cheap these days...

[nelisky]

I just need a way to know if the addresses on the wallet are covered by the last backup, and if there are still leftover addresses in the wallet, without depending on running counts and persistent data not stored in bitcoin directly.

Keys in the keypool store when they're generated (and the oldest are always used first).   Asking "what's the oldest key in the key pool" seems reasonable, and I think it would give you what you want-- you could compare that timestamp to the timestamp of your last backup to see if you're covered.

Although if you're going to periodically check to see what the oldest timestamp is (or you're going to periodically check to see if bitcoin has written a new timestamped backup file or periodically check WHATEVER) then it seems simpler to me to just periodically always call backupwallet.  Disk space and bandwidth is cheap these days...

Yes, I will keep calling backupwallet regularly regardless, but how often do I do it? That's the answer this whole thing will be able to provide, as it's not a static thing. I can then check every minute but backup every day, unless I hit a peak usage and have to put an intermediate backup. And if these happen a few times in a row, I'll adjust the auto backup time frame.

[bitcoinex]

How can I ensure that the pool is generated? debug.log

--pool is not listed in the "usage screen" bitcoind --help (0.3.14 beta)

[mintymark]

It really does seem to me that the OP has a point here, and that control over keyy generation is very important and that the client does not do this in a satisfactory manner.

Once suggestion is to just generate a very large number, perhaps 20000 or even 200000 as a one off operation and leave it at that. Assume you will never generate any more. 20000 keys is about 15Mb. But it is a bit clunky. I tried this with keypool=10000 and when I generated a new address, there was a 10 minute delay and the wallet.dat grew, I then put keypool back to 1, and everything seemed fine, but I have not done a good test.

What happens if you put keypool=0 ?  Intuition suggests that this would generate a new address every time you ask for one, with none in the pool. Setting to 0 did not seem to give any strange behaviour.

What happens with keypool=-1 ? I tried this and could not get any strange behavior.

What I would like is that there is a setting where generating a new key just fails. So I could generate 100 or 1000 or 10000 keys and then set keypool=-1 and NO MORE WOULD BE GENERATED. Any attempt to generate new key would would take them from the pool or fail if the pool was empty.

Is there such a setting or combination of settings? I dont know, but I dont think so.

Can we have it?

[bitplane]

I'd personally prefer a cyclic pool of keys (along with an anonymity warning) rather than having to back up my wallet every time I make a transaction.