auto backing up of wallet.dat

[nelisky]

So, as many others, I need to backup wallet.dat. And since this one is on a server, I need it to happen unattended. And because this server is being used by the lottery, I must not shutdown bitcoind.

So far I just copy the file over, and do it often so I can outlive one corrupted file. But this is not ideal, as I should backup after every transaction (sent, right? no new addresses are created on receiving a transfer ) or after everytime I create a new address.

For that I thought about instead of copying the file I could use db_dump to dump it's contents, using the -r flag just in case. Would that work?

The ideal solution would be an rpc call that would either:
- flush and lock updates until a new rpc call (any call, didn't need to be an unlock command) would come over
- flush and cp wallet.dat to a side file
- flush and dump through jsonrpc. If each key would come separate in an array, we could then call this with 'lastseen=X' to just get new keys

Could this work? Which would work best?

[mizerydearia]

http://bitcointalk.org/index.php?topic=873.msg10728#msg10728 is somewhat related

[satoshi]

I started posting in the other topic but I'll repeat here, this thread seems more specific to the topic.

The main backup improvement will be a pre-generated pool of keys and a rescan at load to scrape missed transactions from the block history.  Then a backup will last forward for a long time.

I was starting to post the same idea you said nelisky.

How about a json-rpc command that locks the wallet, flushes it, copies wallet.dat to a location you specified, then unlocks it?  That would be a smaller project than the pooled keys, so maybe it could be done first.

What's the simplest portable way to copy a file?  Is there something in Boost?

What should it be named?  maybe:
backupwallet <destination>

[nelisky]

I started posting in the other topic but I'll repeat here, this thread seems more specific to the topic.

The main backup improvement will be a pre-generated pool of keys and a rescan at load to scrape missed transactions from the block history.  Then a backup will last forward for a long time.

I was starting to post the same idea you said nelisky.

Yes, I saw your other post and I really like the poll of addresses, but we still need a way to easily back up when those get all used, don't we? I know the address space is huge, but there might be applications that deliver thousands of addresses per day.

How about a json-rpc command that locks the wallet, flushes it, copies wallet.dat to a location you specified, then unlocks it?  That would be a smaller project than the pooled keys, so maybe it could be done first.

What's the simplest portable way to copy a file?  Is there something in Boost?

What should it be named?  maybe:
backupwallet <destination>

Great on all accounts, name and implementation approach.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps. In C++ you can just use standard file streams, can't you? Something like http://www.dreamincode.net/code/snippet2306.htm (out of a quick google, didn't try but looks correct).

To put the cherry on top, why not add a trigger that makes the copy whenever there's a change to the wallet? Well, this would depend on file locking to work properly and that sucks on windows iirc. Been a while since I had to code there

[jgarzik]

What's the simplest portable way to copy a file?  Is there something in Boost?

mmap(2) + memcpy(3) ?  Boost::Iostreams already has a mapped_file Source.

What should it be named?  maybe:
backupwallet <destination>

Sounds quite useful!

[satoshi]

If you read it into memory and write it out, it could fail in tight memory situations.

I'm looking for something like copyfile(const char* from, const char* to) or copyfile(path from, path to), preferably something in Boost if it has it.  If you find it for me, it's more likely I'll get to implementing it.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps.

We require Boost for JSON and a dozen things replacing dependencies on wxWidgets.  Boost is good, portable stuff, we should not shy away from it.

[nelisky]

If you read it into memory and write it out, it could fail in tight memory situations.

I'm looking for something like copyfile(const char* from, const char* to) or copyfile(path from, path to), preferably something in Boost if it has it.  If you find it for me, it's more likely I'll get to implementing it.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps.

We require Boost for JSON and a dozen things replacing dependencies on wxWidgets.  Boost is good, portable stuff, we should not shy away from it.

Ok, so what's wrong with the simple standard fstream usage from that snippet I mentioned? I think simple is best

But if you are already using features from boost::filesystem you can use copy_file from that. I just think that, if not already required for something else, it's a tad overkill.

[jgarzik]

If you read it into memory and write it out, it could fail in tight memory situations.

I think you misunderstand what mmap does?  mmap / CreateFileMapping does not read a file into heap memory:  http://en.wikipedia.org/wiki/Mmap

[satoshi]

I doubt there's an mmap(2) on Windows.  I'd rather call an existing file copy function than make and test my own.

But if you are already using features from boost::filesystem you can use copy_file from that. I just think that, if not already required for something else, it's a tad overkill.

Thanks.  I thought it would be in there somewhere.

We already use boost::filesystem in a dozen places.  It's not a new added dependency.  It gives us a lot of portable stuff that we would otherwise have to have a #ifdef for each OS and test everywhere.

[jgarzik]

I doubt there's an mmap(2) on Windows.

The Windows version of mmap was mentioned in the message to which you replied:  CreateFileMapping

In my earlier message, I mentioned how to use this from boost:  Boost::Iostreams already has a mapped_file Source.

[satoshi]

Sorry, I've been so busy lately I've been skimming messages and I still can't keep up.

We want to avoid Windows API calls whenever possible.  They usually take about 6-8 parameters and a lot of testing to get right, it takes a page of code to do something simple.

I usually shy away from iostreams.  Seems like I too often hit limitations.  They kind of botched the C++ streams standard in the 90's, which is too bad, streams can be very powerful and useful when done right.  Using it in rpc.cpp may still turn out to be a mistake.

Bottom line is I'd rather call an existing file copy function than make and test my own.

[satoshi]

rpc backupwallet <destination> is in SVN rev 147.

[BioMike]

Nice. How about being able to set <destination> as a config option and have the function called after every send(/receive?) of bitcoins?

[nelisky]

Nice. How about being able to set <destination> as a config option and have the function called after every send(/receive?) of bitcoins?

I'm doing backups every time I create a new address and when I send btcs. Receiving should be safe? Also, I'm collecting other stuff and doing offsite scp based backups of the wallet, meaning I have the trigger embedded in my program logic, but auto backups would be great, as one can look for changes in the backup file and use that as trigger to do the offsite.

But what I *really* think would rock is some kind of event mechanism