Improved wallet backups coming in v0.5.13

[jim618]

There was an issue raised this week where unfortunately the user lost 53 BTC.
It is here:
https://github.com/jim618/multibit/issues/173

It was caused by a combination of:

+ the laptop completely run out of juice multiple times (this eventually foiled the rolling wallet backups)
+ having the wallet unencrypted (i.e. no automatic private key exports available)
+ no manually created wallet backups
+ no manually created private key exports


I have been thinking of how to improve the MultiBit wallet/ private key backups and thought I would write up what I plan to do.

Current file structure

Say your wallet is called 'savings'. The timestamps below are just examples.

There is currently stored:

+ savings.wallet                            (The main wallet, private keys + transactions etc)
+ savings.info                               (Labelling info)
+ savings-20130605111604.wallet    (Rolling wallet backup)
+ savings-20130605111203.key       (Encrypted wallets only. Encrypted private key export, created when new keys added or password changes)


New file structure

I am thinking of changing it to:

+ savings.wallet
+ savings.info
+ savings-data                                (A directory. The name of the wallet suffixed with '-data')
+ savings-data/rolling-backup            (A directory to store rolling backups)
+ savings-data/key-backup                (A directory to store the private key exports for encrypted wallets)
+ savings-data/wallet-backup             (A directory to store wallet backups for encrypted wallets)
+ savings-data/wallet-unenc-backup    (A directory to store wallet backups for unencrypted wallets)

The rolling backups and the private key exports would be the same.

in addition both encrypted and unencrypted wallets would be backed up into their respective directories in the format:
"name of wallet"-"timestamp".wallet e.g. savings-20130505120617.wallet

New wallet backups would be made when the wallet is first created, new keys are added to the wallets or when the password changes.
They are stored in separate directories because when you encrypt an unencrypted wallet you want to make sure there are no unencrypted keys left on disk. When an unencrypted wallet is encrypted, all the wallets in the saving-data/wallet-unenc-backup directory will be AES 256 encrypted using the wallet password you are using. The unencrypted backups will be secure deleted.

There will be a maximum number of wallets in the backup directories, say, 60. Once the limit is reached new wallet backups will 'thin out' the existing backups. The backup that is most quickly replaced by a later backup will be deleted. Delete will be a secure delete. (I might always keep the first few backups and last few as aglets).

When a password is changed with this scheme you still have encrypted wallets and private key exports using the old password(s). I'll put a warning message up to mention this but am reluctant to delete them automatically - deleting stuff like this is a good way of losing your private keys. Very secure minded users will have to delete these manually.


I think this scheme offers more protection to the user (especially when they have unencrypted wallets as they will have some automatic backups now) without complicating matters too much (complicated things have a habit of breaking just when you need them).

The new 'per-wallet' directory in which the backups are stored would also tidy things up. It would also be a good location to store other wallet specific things in the future (eg a per-wallet blockstore which would speed up resyncs).


edit: Changed wallet backup replacement algorithm to 'thinning out' from 'median deleting'.
edit: Changed example wallet name, backup directory names, number of backups to match what is actually implemented.

[xeroc]

sounds very usefull ... had overwritten my wallet.dat myself a few days ago ..
backups ARE useful

[Mike Hearn]

I'm not sure about the median age thing. Why would an old backup be more useful than  median aged backup?

Andreas puts up a GUI notice when you first get some balance encouraging users to make an explicit backup. It'd be a good idea to do that in MultiBit too. Although this particular failure might have been avoidable with a larger number of rolling backups, the guys laptop could have broken or been stolen and then it wouldn't have mattered. That's a huge amount of money to have in a wallet that has no backups at all. Also, it was unencrypted. I wonder if he did not know about these features or just didn't think about it.

Idea: once the balance goes above zero for the first time, show a permanent on-screen notice encouraging the creation of a backup. It disappears once you do the backup. Once the balance goes above $100, show a permanent on-screen notice advertising the encryption feature. Once the user has encrypted their wallet, replace it with the backup advert again.

[jim618]

Good suggestions Mike. They are primarily UI changes I think rather than file structure changes.


i think the very early backups are valuable because if "something goes wrong at a point in time" then anything after that may have problems but anything earlier than that won't. I think a fair number of users just use the default wallet with a single address in ie they never create a new receiving address. To keep the First Copy of that wallet makes sense I think.

Also the very latest backups are valuable because they are newer and have the complete set of private keys.

If the earliest and latest backups are more valuable, then I think it makes sense to 'start deleting from the middle'.



When people read this they may be thinking: what about deterministic wallets ?
Well - yes - they add more protection for sure. It'll be great when they go in.

I think of quality in terms of 'the nines'. I.e. is it working 99% of the time, 99.9% ? 99.99% ? etc.
Currently I think there are somewhere around 50,000 multibit users and the quality level is between 3 and 4 nines. For something dealing with money I think you want at least 5 nines. You want improvements on all fronts: backup, GUI (as per Mike's suggestions), deterministic wallets etc.

[flatfly]

Not sure if this has been tried already, but it might be worth using the OSX time machine feature to restore previous versions of the  wallet files...

[jim618]

I have mentioned that to OSX users in the past but of course it only covers a proportion of total users.

[jim618]

I have been working over the last week on improving the MultiBit backups.

Say you have a wallet stored as "savings.wallet".
There is now a per wallet directory stored next to the wallet file called:
   savings-data

It has subdirectories:
   savings-data/key-backup  
   savings-data/rolling-backup  
   savings-data/wallet-backup  
   savings-data/wallet-unenc-backup

The private key backups and rolling backups are the same as before.

The wallet-backup is for backups of encrypted wallets and a new timestamped wallet file is put in there when you first create the wallet, create new private keys/ import keys etc. The corresponding info file is also put in there.

The wallet-unenc-backup is for backups of unencrypted wallets, backed up with the same criterion.

Also implemented is a thinning algorithm so that there is a maximum number of backups (60 per backup directory). The algorithm thins out the backups so that they are spread as much as possible over time. It always keeps the first 2 and last 8 backups regardless as I think these are marginally more valuable.

When you add a password to your wallet all the UNencrypted wallet backups are encrypted with the wallet password you have chosen. This is a "whole file" encryption using AES 256 - it is the same encryption algorithm as for the private keys but for the whole file. When I put in a separate "Backup Wallet" UI command I will offer this so that you can make your wallet backup completely opaque. (I haven't done this yet and it won't get into the next release).

You can open these "whole file" encrypted files using "File | Open Wallet" - obviously you need to know the password to decrypt the file and use it.

The 'whole file' encryption is a similar strategy to the blockchain.info double encrypted backups.

When you first use the new code, it tidies up your existing key and backup files (that are currently stored in the same directory as your wallet). This also makes them available for wallet recovery.

Wallet recovery is also improved - if the main wallet cannot be loaded it tries to load both the rolling backup and last wallet backup (newer first) before reporting a failure.

The help has also been improved to give people more 'Wallet Backup Advice'. It also describes what backup files are available in case the automatic wallet recovery fails and they have to look thru the backups manually. I'll push the new help onto the website later this week.

I'll spend the rest of the week testing things and plan to have the next version, 0.5.13, out at the beginning of next week.

TL;DR The next version of MultiBit will have improved wallet backups.