Increasing the number of keys in key pool

[jl2012]

Again, someone lost >1300BTC due to misconception about wallet backup: https://bitcointalk.org/index.php?topic=110781.0

It might be difficult to implement deterministic wallet in the next release. However, it is very easy to increase the key pool from 100 keys to 10000 keys. Adding 2 or 0 zeros in the source code will solve 99% of the problem

[1713998980]

1713998980

[1713998980]

1713998980

[1713998980]

1713998980

[1713998980]

1713998980

[1713998980]

1713998980

[1713998980]

1713998980

[CIYAM]

Actually it wouldn't matter how many keys were in the key pool in the case you have linked to as his backup was unencrypted then he encrypted his wallet and then sent BTC.

He lost the BTC because it was sent to a "change" address and that address (for security reasons) is not taken from the unencrypted pool but instead from a new pool that is created when you encrypt the wallet.

The lesson is to immediately backup after encrypting (and I believe the software does warn you that you need to).

[runeks]

The lesson is to immediately backup after encrypting (and I believe the software does warn you that you need to).

I don't think it does. At least there's an open issue regarding this on the bug tracker: https://github.com/bitcoin/bitcoin/issues/1884

[CIYAM]

I don't think it does. At least there's an open issue regarding this on the bug tracker: https://github.com/bitcoin/bitcoin/issues/1884

Ouch - if that is true then certainly it needs to be changed (and the notice should probably be presented to the end user in a bold font with a bright color).

YOU NEED TO BACKUP YOUR ENCRYPTED WALLET BEFORE MAKING ANY TRANSACTION OR YOU COULD LOSE ALL YOUR BITCOINS!

[runeks]

I just hacked together a quick patch in this pull request: https://github.com/bitcoin/bitcoin/pull/1890
Here's what the dialog looks like now:

If the devs wish they can make the text red.

[CIYAM]

Good stuff!

I think that should get the message across (red text is probably not necessary).

[Stephen Gornick]

it is very easy to increase the key pool from 100 keys to 10000 keys. Adding 2 or 0 zeros in the source code will solve 99% of the problem

As has been described, a larger key pool wouldn't have solved the problem described where enabling wallet key encryption flushes the entire pool, but if you want to the key pool can size be controlled with a config setting:

 -keypool=<n>       Set key pool size to <n> (default: 100)
 
 - http://en.bitcoin.it/wiki/Running_Bitcoin

[Stephen Gornick]

I just hacked together a quick patch in this pull request: https://github.com/bitcoin/bitcoin/pull/1890

Should the instructions really be to "replace" your old backups?

Instead you might want to describe how no prior backups will have the new encrypted keys and thus it is recommended to make new backups now.  Or something to that effect.

[FreeMoney]

That warning isn't quite right because it implies that the old backups won't be dangerous in the wrong hands, but they will until you have used all existing inputs.

[BC12345]

1. +1 to this whole backup/encrypt wallet issue. I think this is really important, especially to people who are new with bitcoins (but apparently not only for them)

2. I do not think that flushing the key pool when the wallet is encrypted is a good idea. I understand why it's done, but when a program gives you the option to encrypt something I expect it to do right that: encrypt. I don't expect it to somehow modify/change my data.

[kjj]

1. +1 to this whole backup/encrypt wallet issue. I think this is really important, especially to people who are new with bitcoins (but apparently not only for them)

2. I do not think that flushing the key pool when the wallet is encrypted is a good idea. I understand why it's done, but when a program gives you the option to encrypt something I expect it to do right that: encrypt. I don't expect it to somehow modify/change my data.

The keys existed on your disk in an unencrypted state.  They are not safe, they should not be used.  Marking them and generating new ones is the right thing to do.  Key encryption has been around for something like a year, and we are just now noticing that a few people have lost keys in strange situations.

[Dabs]

Actually, I prefer that the default keypool be made smaller. But that's just me, I don't have 100 transactions, and it is convenient for me to just take care of a dozen keys.

In my experiment, which anyone can duplicate, the newly encrypted wallet contains all the old keys AND 100 more new keys. The old keys are not useless, but the new keys will not yet have been backed up.

It's possible that the client just picked a key at random to send the change and it just so happens that it was not one of the old keys.

[DeathAndTaxes]

Actually, I prefer that the default keypool be made smaller. But that's just me, I don't have 100 transactions, and it is convenient for me to just take care of a dozen keys.

In my experiment, which anyone can duplicate, the newly encrypted wallet contains all the old keys AND 100 more new keys. The old keys are not useless, but the new keys will not yet have been backed up.

It's possible that the client just picked a key at random to send the change and it just so happens that it was not one of the old keys.

Not exactly it is important to understand there is a difference between keys in the "active/used" wallet and keys in the keypool.

The keys (i.e. used keys) in the wallet transfer over.  Obviously they would have to or you would lose all coins associated with those keys.   The entire keypool is erased and new keys generated.  The Satoshi client always uses the next key in the keypool for change.   It is never up to chance.   Also a larger keypool doesn't increase your administrative burden.  If you are doing manual extraction/key work on a wallet the keypool represents "future" keys.  100 or 10,000 it doesn't really matter.  If you only have a dozen active keys you simply need those keys.  

TL/DR
Used/active keys =/= keypool.
The satoshi client always uses the "next" key from the keypool for change and new addresses.

[Dabs]

If the satoshi client discarded unused keys, then I would not have all the old unused keys inside the new encrypted wallet. My experiment had 108 old keys (I think I have 8 transactions). The new encrypted wallet has 208 keys, 108 from the old wallet including used and unused keys, and 100 more new keys.

I used bitcoin version 0.7 (the one just released) for Windows, and pywallet to export the keys from both the old unencrypted wallet, and the new encrypted wallet.

The difference between the two is I just encrypted the wallet, then I made a backup of the encrypted wallet.

Or maybe it's just my wallet since I created it when it was version 0.5 or so at the time (March 2012). Maybe the behavior for new wallets is different than for old wallets created with an old client? But I remember upgrading my wallet to a newer format (maybe it was version 0.6)

[CIYAM]

From what I've gathered it doesn't throw away those keys but after the encryption doesn't use them (at least for change addresses) either (I guess in the off chance somehow you managed to get funds sent to one of those addresses).

[DeathAndTaxes]

If the satoshi client discarded unused keys, then I would not have all the old unused keys inside the new encrypted wallet. My experiment had 108 old keys (I think I have 8 transactions). The new encrypted wallet has 208 keys, 108 from the old wallet including used and unused keys, and 100 more new keys.

I used bitcoin version 0.7 (the one just released) for Windows, and pywallet to export the keys from both the old unencrypted wallet, and the new encrypted wallet.

The difference between the two is I just encrypted the wallet, then I made a backup of the encrypted wallet.

Or maybe it's just my wallet since I created it when it was version 0.5 or so at the time (March 2012). Maybe the behavior for new wallets is different than for old wallets created with an old client? But I remember upgrading my wallet to a newer format (maybe it was version 0.6)

Weird.  Maybe CIYAM is right.   To avoid losing funds in the event a key was used in a non-standard way (say from a keydump) it probably doesn't delete them just marks them so they will never be used in future tx by the client.  

I didn't notice another part of your question.  You can make the keypool smaller if you wish.  100 is the default (IMHO should be larger given the trivial amount of space) but you can set it to any value even 0.  With keypool=0 you have no keypool the only keys in the wallet will be existing keys and new keys (when needed) will be "created on the fly".  NOOB WARNING (not you dabs but anyone who happens to read this): with a keypool of 0 you would need an updated backup after every tx involving a new key ("new address button", or spend w/ change) to avoid the risk of irrecoverable loss of funds.

Still all of this is somewhat academic.  Deterministic wallets are the future.  The ability to backup and/or print an encrypted deterministic seed and thus avoid a whole category of potential data loss scenarios provides huge value in making bitcoin "easier to use" while providing no real risk/downside. 

[Dabs]

The risk with deterministic wallets is obvious. If someone cracks the deterministic seed used, he has all your money, now and into the future, for that particular wallet. With a randomly generated wallet, someone would have to crack or bruteforce every private key. Of course, if someone cracks the passphrase used for encrypted wallets, he has everything in your current wallet up to the key pool maximum (or future 100 transactions) but not your 101st transaction. (for default keypool=100).

I say, give the user the choice. I prefer randomly generated wallets.

I think an option for the client would be to set a maximum number of keys used, so that, for example, if you have already 1000 keys, and you have made 1000 transactions, the client would just re-use an old key. This way, you can keep a backup of your wallet once (after generating 1000 keys).

Take the example of BitcoinSpinner. It only uses 1 key. That's an extreme, but you can have the satoshi client fix your maximum keys to an arbitrary number of your choice (and probably force it to pre-generate that number already.)

To prevent the client from generating too many keys and taking over your computer, maybe the satoshi client will also have a hard coded limit on what the maximum wallet size would be, like 65000 keys is pretty large for one wallet, for one person.

[kjj]

The risk with deterministic wallets is obvious. If someone cracks the deterministic seed used, he has all your money, now and into the future, for that particular wallet. With a randomly generated wallet, someone would have to crack or bruteforce every private key. Of course, if someone cracks the passphrase used for encrypted wallets, he has everything in your current wallet up to the key pool maximum (or future 100 transactions) but not your 101st transaction. (for default keypool=100).

I say, give the user the choice. I prefer randomly generated wallets.

I prefer random keys too, but realistically speaking, cracking the master seed is equivalent to either 1) breaking all EC math, including bitcoin, or 2) stealing your wallet and cracking your password.  (I'm pretty sure armory encrypts the master seed at least as well as the reference client encrypts the private keys.)

In either case, you have big problems.  EC wallets have plenty of advantages, and only insignificant theoretical disadvantages.  They should probably be the default.

[runeks]

I just hacked together a quick patch in this pull request: https://github.com/bitcoin/bitcoin/pull/1890

Should the instructions really be to "replace" your old backups?

Instead you might want to describe how no prior backups will have the new encrypted keys and thus it is recommended to make new backups now.  Or something to that effect.

That warning isn't quite right because it implies that the old backups won't be dangerous in the wrong hands, but they will until you have used all existing inputs.

I considered making the message longer and more descriptive. But I'm afraid people will just ignore it if a giant wall of text appears.

Please feel free to compose a better message, and post it on the Github page for the issue. I think you are right, but I'm not quite sure how to get it across without the message becoming too verbose.

[Dabs]

Make it simple:

Warning: After encryption, make a backup of your new wallet and discard all previous backups. See *insert*webpage* for detailed explanation.

You have 17.29382 bitcoins from your previous unencrypted wallet, would you like to send them to a new address in your new encrypted wallet? (click yes / no / maybe / i'm not sure.)