Wallet.dat backups may lose transactions prior to backup (and this is not a bug)

[kermit]

The correct answer is "everything ever".

This is very counterintuitive and I don't really understand why, but you MUST backup after every send, or after creating new addresses, or you risk losing the ENTIRE wallet.  Also, use the bitcoind backupwallet command, or exit bitcoin first.   For that to work with the GUI, the GUI had to be run with -server.

Since that always leaves a window of unbacked up wallet, a better solution is to have a savings wallet that doesnt do much sending, and send everything there.

(Way at the bottom of the FAQ this is mentioned http://www.bitcoin.org/faq#How_do_I_backup_my_wallet)

[nelisky]

This has been discussed in many other threads, but it might be a good idea to do again, to avoid misunderstandings.

The reason, as far as I understand, for the above stated is change. When you send coins, one or more transactions received and emptied, but that may not sum up to the exact transfer amount, so a new inbound transfer is created with the change. This will be a new address for you, and thus not backed up before. Imaging:

addr1: 50BTC
addr2: 2.5BTC
addr3: 50000BTC

All this is backed up, now you send 60 to me, and end up with:
addr1+addr2+addr3 -> TX (50052.5), broken in
  myaddr: 60 (transfered to me)
  addr4: 49992.5 (your change)

So a new address is automagically created for you, and all your coins put there. This was not in the previous backup, so if you loose this wallet... you are hosed!

[kermit]

This has been discussed in many other threads, but it might be a good idea to do again, to avoid misunderstandings.

Yeah, I'm trying to increase awareness.. the poll is to show the powers that be how low the current awareness level is so it's made more clear to the user in some way.. mainly so this doesn't get really bad PR at some later point down the line when any influential start losing their wallet.   (Like how XFS got bad PR for losing data merely because its default age time was higher than ext3)

And thanks for that explanation.

[wscott]

And the proposed solution is to pre-generate a lot of future addresses and store them in your wallet too.  Now every transaction will use one of these pre-generated but unused addresses for the change and also generate a new address to replace it.

This means that if we save 100 future addresses in your wallet file, you just need to have a backup of your wallet from less than 100 transactions ago to recover your money.  Much better.

-Wayne

[grondilu]

Is there a clean, complete HOWTO about this subject (How to backup)  ?

AFAIC I just made backup.sh script in my .bitcoin folder :

Code:

#!/bin/bash

gpg -r $USERNAME -e wallet.dat
sftp $USERNAME@SOME_STORAGE_SERVICE.com <<EOF
put wallet.dat.gpg
EOF

I wonder if I should or not keep a backup history or something (backup.dat.1.gpg, backup.dat.2.gpg, etc.).

What do you think ?

[nelisky]

Is there a clean, complete HOWTO about this subject (How to backup)  ?

AFAIK I just made backup.sh script in my .bitcoin folder :

Code:

#!/bin/bash

gpg -r $USERNAME -e wallet.dat
sftp $USERNAME@SOME_STORAGE_SERVICE.com <<EOF
put wallet.dat.gpg
EOF

I wonder if I should or not keep a backup history or something (backup.dat.1.gpg, backup.dat.2.gpg, etc.).

What do you think ?

I think you are asking for trouble

The correct way would be to add an rpc call to backupwallet (which is supported from 0.3.12) passing a file path, and then do the gpg/sftp dance on that file. It might require you to, if running a GUI, start the program with the -server switch, though.

[grondilu]

The correct way would be to add an rpc call to backupwallet (which is supported from 0.3.12) passing a file path, and then do the gpg/sftp dance on that file. It might require you to, if running a GUI, start the program with the -server switch, though.

Ok, thks.  I'm looking forward to see this pretty much automated, though.

[Beelzebuddy]

Thanks for the tip re: backupwallet.  That's not listed in the online API documentation.

Two questions, if you'd be so kind:

1) Is there any benefit to multiple backups?  Or do newer backups strictly invalidate older ones?  That is, should I call my file backup.dat or backup.datetime().dat? 

2) This is a lotta damn foolishness we have to deal with here, and it's all caused by sending the change to a newly generated address.  What's wrong with sending it to an existing, possibly designated one?

How this might work
*If there's an address labeled "Change," that address receives all change payments.  Uber-strict backups not necessary.  Maybe even something more intuitive like "Your Address," the designated one that's already there.
*If not, create a new address as usual, for anonymity or whatever.

[nelisky]

Thanks for the tip re: backupwallet.  That's not listed in the online API documentation.

Two questions, if you'd be so kind:

1) Is there any benefit to multiple backups?  Or do newer backups strictly invalidate older ones?  That is, should I call my file backup.dat or backup.datetime().dat? 

There's always some benefit to multiple time point backups. You might have bad ram in the computer and one of the backups is corrupted, same goes for hard drive or even just a slightly loose cable. But for the most part, assuming all goes well, the latest backup does invalidate older ones.
Now, don't feel too safe because you have one backup, even 100 of them, on the same physical location. Same computer is asking for trouble, same house is calculated risk. Better to make a daily or so backup to an external storage service. It's always about balance; how much do you keep in the wallet, how often you do transactions, what would it cost you to loose such wallet.

2) This is a lotta damn foolishness we have to deal with here, and it's all caused by sending the change to a newly generated address.  What's wrong with sending it to an existing, possibly designated one?

How this might work
*If there's an address labeled "Change," that address receives all change payments.  Uber-strict backups not necessary.  Maybe even something more intuitive like "Your Address," the designated one that's already there.
*If not, create a new address as usual, for anonymity or whatever.

I don't see any technical reason to not do so that way. Other may think differently, but from the top of my head there's nothing preventing that. It would, however, create a very simple path to follow and find you, and while this may not present an issue to you personally, if your address is compromised there's a good chance the receiver is also traceable now. I'm paranoid, you probably aren't, so my opinion is highly biased

[Beelzebuddy]

I'm not saying you shouldn't have the ability to be as completely anonymous as possible, I just don't think it should trump usability as the default setting. 

Right now, new users (like me) find a shitload of scary warnings, and debate, and fud, over something that can be quickly and easily prevented entirely.  Unlike me, most people aren't going to dig deep enough to find out the root trouble and deal with it.  They're going to logically conclude that the software is buggy, or at least untrustworthy, and move on.  And yes, since the documentation for the "lose all yer bitcoins" behavior is found only in threads like this, that's a bug and a damn big one.

Freenet had a similar misstep a few years back, when it would only run in darknet mode.  Hey, it's great  that it can run like that, but it meant that random people who just wanted to dick around and help out the network had to idle in a designated IRC channel for a few hours collecting connection information (completely defeating the purpose of a darknet, btw) before they could join.  So mostly no one did, Freenet stagnated because of it, and i2p left them in the dust.

What bitcoin needs right now is adoption.  What it needs to see greater adoption is greater usability, which right now means stability.  People aren't going to invest their time and money in a program which can burp and randomly delete their shit because they momentarily took their tinfoil hat off.  That's not responsible behavior for a piece of financial software.  Either don't make new addresses or auto-backup after each transaction.  Hell if I know why it isn't doing that to begin with.

[theymos]

And yes, since the documentation for the "lose all yer bitcoins" behavior is found only in threads like this, that's a bug and a damn big one.

A warning about this was in the very first revision of the wiki's "backup" page (from July).

An elegant and anonymous solution will be implemented at some point (a list of queued addresses will be included in wallet.dat for use with change). Until then, deal with the fact that this beta software requires more frequent backups.