Bitcoin Forum

i was under the impression that if i back up my wallet at time T0  and continue to use the MAIN wallet to send and receive bitcoins,  that if i open the backed-up wallet at time T1, assuming i'm sync'ed with the blockchain,  that my backed up wallet would reflect all of the transactions.  however, this doesn't appear to be the case.  there's a finite number of addresses that the bitcoin client keeps around (in a pool of 100 addresses) which get used up every time you send a payment.  if you run out of your addresses,  new addresses will be added to the pool but your BACKED UP wallet.dat file will not be privy to these new addresses and thus if you are unfortunate enough to ever lose the original file, you will forever lose some fraction of your coins.. this is because whenever you send bitcoins,  it generates 2 transactions:  tx1 goes to the recipient, tx2 gets sent to one of the 100 addresses that are stored in your account pool... so if these were depleted and new ones added, the backup wallet doesn't know about it and thus the loss of bitcoin can occur... was i like the only one who didn't know about this?   now i have to be a lot more careful and backup the wallet more frequently i guess...

could you later import your wallet into one of those and be good? (get those 'lost' coins back)

No once lost the coins are lost.  The coins are "lost" because the wallet generates an address not in the backup and if no updated backup is made there is no record of the private key (if the original is lost/damaged).   There is nothing to import, they private key is missing.

You can however expand the keypool with the -keypool=xxxx command line argument.  Make the keypool 1,000 keys to give yourself some more "slack" between backups.

thanks for the advice, DeadAndTaxes.  will do that to give myself a bit more runway.

Most computer files work this way. If you haven't backed up the latest version, you lose all changes since the last backup. I don't know why this would be a surprise.

The qt wallet gives you 100 changes worth of bonus slop, and deterministic wallets give you unlimited slop. But that is not the norm for backups in general.

Unfortunately new users believe they are backing up their one private key.  Not realizing behind the scenes new one are being created.  New wallets should somehow make the user aware of this so they know to make new backups.

When wallet creates a new key automatically, it should ask for a backup with warning and get confirmation before proceeding, this is a very silly bug in software claiming to be ver 0.8

So every transaction you mean.  You want the wallet to warn you to backup after every single send?

I didn't get it. Do you mean there will be a new address generated after each payment, or after 100 payments? Currently, i can only see the addresses manually created by myself, and I will only backup when I created a new address. I am using multibit.

Why would wallet create a new key after each transaction, can't/doesn't it have a pool of keys?

In the QT client the keypool is continually refreshed.  While the keypool contains 100 keys every send requires a new change address so the oldest key from the keypool is used AND a new key created and added to the keypool.  The keypool is always at 100 keys (baring some unusual situations where wallet can't be unlocked).

So yes the wallet would need to warn you to backup every single time you sent a transaction OR clicked New Address.

So what is the benefit of keeping the pool always 100? instead why not let it deplete and when it reaches threshold refill the pool and ask for backup?

It does but it doesn't allow the keypool to run out.  It adds a new key to the keypool everytime one is used.

Imagine a wallet has a keypool which contains 100 address.  We will number them 1 to 100.

So:
Active Addresses: None
Keypool: 1 ... 100

If you took a backup right now you will notice you have all the addresses backed up to #100.


You need a new address.  The wallet pulls the oldest key from the keypool "1".  It immediately add a NEW key to the keypool lets call that 101.

Active Addresses: 1
Keypool: 2 ... 101

Next address used
Active Addresses: 1, 2
Keypool: 3 ... 102


48 more addresses used.
Active Addresses: 1, 2 .... 50
Keypool: 51 ... 150

50 more addresses used
Active Addresses: 1, 2 .... 100
Keypool: 101 ... 200

If you used another address it wouldn't be included in your backup.

It would make backups before the pool exhausts useless.  As it works now any backup made at anytime covers all active addresses and the NEXT 100 future addresses.   It also means that prior backs (if not overwritten) likely are also effective and can be used as a backup for the backup if you do more than 1 backup per 100 key uses.

My keypool is 1,000 keys.  My highest weekly usage ever was ~130 keys.  I backup automatically once a week with sequentially named backups.  The likelihood of losing any keys is essentially zero.

isn't that a good thing

No unless you like losing keys, unpredictability, or bitcoind halting your automated system because you didn't do a backup you didn't know you needed to do until it halted.

As it stands now every backup is useful.  It resets the "future key clock" to the limit of the pool (default 100).  So if you do a daily or weekly backup your backup will ALWAYS have all keys and the next 100 future keys.

Backups should always be periodic.  Setup your system to backup the wallet.dat once per week and set your keypool large enough to cover two weeks usage and at all times you should have two valid backups (in case of a file corruption issue).   Nothing to think about, no risk of creating keys which aren't backed up, no intrusive system where bitcoind has to prevent you from using your own money because it just foolishly exhausted the keypool and needs you to stop and manually make a backup right now.

I don't understand, if bitcoin-qt instead of generating new keys every-time did it periodically then wallet state will not change so often, it will also enable it to ask for backup when wallet state changes, that way user knows when to do backup, currently only safe way is to backup after every transaction.

e.g if it generates 1000 keys to start with and uses those only and once it reaches 100, generates 900 more and asks user to back it up, what is the harm in that?

No, if data is critical (lots of money) it should not be periodic it should be backuped at each change, if I do weekly backup I risk loosing some coins.

bitcoind is often used in automated processes.  I much prefer my company wallet being automatically backed up every day and knowing I have multiple backups then an intrusive system which waits to refill the keypool and forces me to do a manual backup on a unpredictable schedule.

What if the user makes a backup right before the keypool exhausts and being a user (think your grandmother) assumes he is "covered" he just made a backup 5 minutes ago it must be good.  Then they keypool refreshes making his backup 5 minutes ago worthless?

What happens if you are right in the middle of sending out a batch of payments.  Will the system halt and lock you out of your own money until it forces you to make a backup?   Will it actively prevent you from using the wallet.   If it doesn't and the wallet fails you just lost keys and funds.

What happens if the backup is corrupted?  Your prior backups are completely useless because backups between refreshes contain less and less new keys?

Of course the client is open source.  Feel free to fork it and implement a different backup system.    Honestly given all the potential use and applications I can't see a more universal solution than make regular backups and the backup will contain all used keys and X future keys.  Still this is somewhat academic, long term all wallets will be deterministic anyways.

I think a good feature of the Bitcoin client would be a warning after making 50 transactions or so without making a backup (executing the backup command).

Strangely I have never lost any coins with periodic backups.  Like I said fork the QT client and implement a different backup system.  I was advising the OP on how to increase the keypool and on how it works.

If bitcoind could generate event for backup then that can be backed up exactly when needed, for now it will generate event every transaction which I think is a bad design, wallet should change state and generate event periodically and user should have option to be alerted, ignored, or run a command.

Long term is long term away :)

It seems multibit sometimes send change back to the origin address. What's the problem of this? Does this mean in this case no new change address is created?

You are correct.

MultiBit does not create new change addresses. The only private keys in the wallet are for the addresses you see in the 'Request bitcoin' tab.

This is not very good for privacy, but does make the backups life longer.
If you create new receiving addresses (i.e. new private keys) then you need to make new backups.

(The next version of MultiBit - MultiBit HD - will have a deterministic wallet)

Address reuse issue.


And another pointer to Amory's deterministic wallet. One paper backup is enough to restore all future private keys.

As far as I know, it is not feasible to deduce the private key from the public key. That's the basic requirement of any asymmetric encryption algorithm. If this is cracked, then many networking protocol is not safe any more. For example, ssh and https are both based on SSL (yes they are RSA or DSA based, not exactly ECDSA but quite similar).

But I agree exposing only 'address' is a good idea anyway if it is really a cold storage.

I agree that it is a little on the paranoid side.

However, think back on the recent Android RNG debacle. Had those wallets not reused addresses, the private key could not have been computed.

I agree.

Backup1a
Backup1b
Backup1c
etc.

appears to be much better than

Backup
overwrite Backup
overwrite Backup
overwrite Backup
etc.

As long as you don't do more than 100 transactions between backups, you should be fine. :)

Backup integrity checks?
Backup at different locations?
Backup encryption?
Boot to the head type memory loss?

Paper backup with deterministic wallet ... yay.

as bitcoin becomes more popular,  you can't expect people without a computer science degree to understand the intricacies of the algorithm.  i think the most sensical thing to do is to warn someone who is creating a backup about what will happen with default settings if 100 keys are exhausted...

as someone WITH a computer science degree,  and as someone who has been around bitcoin since 2011,  i didn't really start backing up my wallet until recently.  i used to hold the coins in my mtgox account until mtgox disappointed me with a certain transaction went sour. the manner in which they handled it (i got my money back eventually but it was quite the agonizing 2 weeks), i have decided to part ways with them and decided to use the physical client (bitcoin-qt) as the primary store of the coins.  i started backing up my wallet and one day decided to run through a COB scenario in which i imagined what i would do in the event my computer had crashed... and then i realized the whole keypool=100 issue...   

at the time of creating the backup, i understood the concept of private keys and the idea that the backup of the wallet.dat file essentially stores the private key, along with my addresses i used, which is all that is really needed to store the value of the coins-- however,  at that time, i wasn't privy to the implementation details inside bitcoin-qt involving the keypool size=100,  and that once you run out of those keys, the new keys that are generated will not be in the old backup file... so that seems like an implementation detail that the user shouldn't really be concerned with or exposed to,  in the same way that when i use dollar bills,  i do not know all the intricacies involved in the various inks and paper types used to create the bills... i just transact with the bills without giving it a second thought...  isn't this an obvious flaw/issue with the bitcoin-qt?

In an interview, the lead developer of Armory says that the reference client bitcoin-qt will adopt the deterministic wallets, similar to those Armory uses.

Theoretically the concept is simple. If the seed for the pseudo random number generator is saved, all the generated addresses will always be the same. For example, given a certain seed, the 10th private key will always be the same.

Interview:
http://letstalkbitcoin.com/e55-happy-birthday-bitcoin/#.Unj_z-VX98E (around 16:00)

To me it is obvious that in order for Bitcoin to be used as a day-to-day currency it must appeal to the lowest common denominator, otherwise people will just continue using cash and credit-cards, ceteris paribus. In order for Bitcoin to succeed as a day-to-day currency it must be simplistic enough to the point at which all one needs to know is, "send coins to (address)," "receive coins at (address)."

I am pretty sure long before Bitcoin is mainstream there will be no "random key" wallets outside of niche applications used by professionals.   Deterministic wallets are far easier to understand, and safer for the average user.   Remember the QT client is the reference client and as such it is rather conservative at adopting enhancements.  In time I am sure it will either be deterministic as well.  With a deterministic wallet one simply needs to backup and/or print the deterministic seed once and the backup is valid forever.

Given enough time you probably will even see hardware deterministic wallets which are completely "user proof" in that the deterministic wallet comes with a preprinted "THIS IS YOUR WALLET RECOVERY DOCUMENT.  PLACE IT IS A SAFE PLACE.  TREAT IT LIKE CASH AND ENSURE IT IS NOT STOLEN. IF YOUR WALLET FAILS YOU FUNDS CAN NOT BE RECOVERED WITHOUT THIS DOCUMENT."   User buys the wallet, puts the recovery doc in a safe, and if their wallet fails buy another one, scans the QR code on the recovery doc and is back up and running with no loss of funds.


Bitcoin isn't mainstream yet.
QT client is the reference node and increasingly not the best choice for casual users.
There has to be a complete, solid, well tested reference node to ensure the network operates as expected.  It probably will always be "rougher" than clients designed for casual users.  The need for full blockchain is probably going to drive casual users to more "friendly" (SPV) wallets anyways.