Bitcoin Forum
March 19, 2024, 07:23:39 AM *
News: Latest Bitcoin Core release: 26.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: auto backing up of wallet.dat  (Read 9356 times)
nelisky (OP)
Legendary
*
Offline Offline

Activity: 1540
Merit: 1001


View Profile
August 25, 2010, 03:52:13 AM
 #1

So, as many others, I need to backup wallet.dat. And since this one is on a server, I need it to happen unattended. And because this server is being used by the lottery, I must not shutdown bitcoind.

So far I just copy the file over, and do it often so I can outlive one corrupted file. But this is not ideal, as I should backup after every transaction (sent, right? no new addresses are created on receiving a transfer ) or after everytime I create a new address.

For that I thought about instead of copying the file I could use db_dump to dump it's contents, using the -r flag just in case. Would that work?

The ideal solution would be an rpc call that would either:
- flush and lock updates until a new rpc call (any call, didn't need to be an unlock command) would come over
- flush and cp wallet.dat to a side file
- flush and dump through jsonrpc. If each key would come separate in an array, we could then call this with 'lastseen=X' to just get new keys

Could this work? Which would work best?
The block chain is the main innovation of Bitcoin. It is the first distributed timestamping system.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1710833019
Hero Member
*
Offline Offline

Posts: 1710833019

View Profile Personal Message (Offline)

Ignore
1710833019
Reply with quote  #2

1710833019
Report to moderator
mizerydearia
Hero Member
*****
Offline Offline

Activity: 574
Merit: 507



View Profile
August 25, 2010, 11:11:16 AM
 #2

http://bitcointalk.org/index.php?topic=873.msg10728#msg10728 is somewhat related
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364
Merit: 6610


View Profile
August 26, 2010, 12:57:40 AM
 #3

I started posting in the other topic but I'll repeat here, this thread seems more specific to the topic.

The main backup improvement will be a pre-generated pool of keys and a rescan at load to scrape missed transactions from the block history.  Then a backup will last forward for a long time.

I was starting to post the same idea you said nelisky.

How about a json-rpc command that locks the wallet, flushes it, copies wallet.dat to a location you specified, then unlocks it?  That would be a smaller project than the pooled keys, so maybe it could be done first.

What's the simplest portable way to copy a file?  Is there something in Boost?

What should it be named?  maybe:
backupwallet <destination>

nelisky (OP)
Legendary
*
Offline Offline

Activity: 1540
Merit: 1001


View Profile
August 26, 2010, 01:21:57 AM
 #4

I started posting in the other topic but I'll repeat here, this thread seems more specific to the topic.

The main backup improvement will be a pre-generated pool of keys and a rescan at load to scrape missed transactions from the block history.  Then a backup will last forward for a long time.

I was starting to post the same idea you said nelisky.

Yes, I saw your other post and I really like the poll of addresses, but we still need a way to easily back up when those get all used, don't we? I know the address space is huge, but there might be applications that deliver thousands of addresses per day.


How about a json-rpc command that locks the wallet, flushes it, copies wallet.dat to a location you specified, then unlocks it?  That would be a smaller project than the pooled keys, so maybe it could be done first.

What's the simplest portable way to copy a file?  Is there something in Boost?

What should it be named?  maybe:
backupwallet <destination>



Great on all accounts, name and implementation approach.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps. In C++ you can just use standard file streams, can't you? Something like http://www.dreamincode.net/code/snippet2306.htm (out of a quick google, didn't try but looks correct).

To put the cherry on top, why not add a trigger that makes the copy whenever there's a change to the wallet? Well, this would depend on file locking to work properly and that sucks on windows iirc. Been a while since I had to code there Smiley
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1091


View Profile
August 26, 2010, 01:42:26 AM
 #5

What's the simplest portable way to copy a file?  Is there something in Boost?

mmap(2) + memcpy(3) ?  Boost::Iostreams already has a mapped_file Source.

What should it be named?  maybe:
backupwallet <destination>

Sounds quite useful!




Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364
Merit: 6610


View Profile
August 27, 2010, 01:13:42 AM
 #6

If you read it into memory and write it out, it could fail in tight memory situations.

I'm looking for something like copyfile(const char* from, const char* to) or copyfile(path from, path to), preferably something in Boost if it has it.  If you find it for me, it's more likely I'll get to implementing it.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps.
We require Boost for JSON and a dozen things replacing dependencies on wxWidgets.  Boost is good, portable stuff, we should not shy away from it.
nelisky (OP)
Legendary
*
Offline Offline

Activity: 1540
Merit: 1001


View Profile
August 27, 2010, 01:21:09 AM
 #7

If you read it into memory and write it out, it could fail in tight memory situations.

I'm looking for something like copyfile(const char* from, const char* to) or copyfile(path from, path to), preferably something in Boost if it has it.  If you find it for me, it's more likely I'll get to implementing it.

As for the file copy, why add to the boost dependency? I for one would love to get a core lib with very little deps.
We require Boost for JSON and a dozen things replacing dependencies on wxWidgets.  Boost is good, portable stuff, we should not shy away from it.

Ok, so what's wrong with the simple standard fstream usage from that snippet I mentioned? I think simple is best Smiley

But if you are already using features from boost::filesystem you can use copy_file from that. I just think that, if not already required for something else, it's a tad overkill.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1091


View Profile
August 27, 2010, 02:14:19 AM
 #8

If you read it into memory and write it out, it could fail in tight memory situations.

I think you misunderstand what mmap does?  mmap / CreateFileMapping does not read a file into heap memory:  http://en.wikipedia.org/wiki/Mmap


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364
Merit: 6610


View Profile
August 27, 2010, 02:54:07 AM
 #9

I doubt there's an mmap(2) on Windows.  I'd rather call an existing file copy function than make and test my own.

But if you are already using features from boost::filesystem you can use copy_file from that. I just think that, if not already required for something else, it's a tad overkill.
Thanks.  I thought it would be in there somewhere.

We already use boost::filesystem in a dozen places.  It's not a new added dependency.  It gives us a lot of portable stuff that we would otherwise have to have a #ifdef for each OS and test everywhere.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1596
Merit: 1091


View Profile
August 27, 2010, 03:42:25 AM
 #10

I doubt there's an mmap(2) on Windows.

The Windows version of mmap was mentioned in the message to which you replied:  CreateFileMapping

In my earlier message, I mentioned how to use this from boost:  Boost::Iostreams already has a mapped_file Source.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364
Merit: 6610


View Profile
August 27, 2010, 03:47:57 PM
Last edit: August 27, 2010, 04:16:05 PM by satoshi
 #11

Sorry, I've been so busy lately I've been skimming messages and I still can't keep up.

We want to avoid Windows API calls whenever possible.  They usually take about 6-8 parameters and a lot of testing to get right, it takes a page of code to do something simple.

I usually shy away from iostreams.  Seems like I too often hit limitations.  They kind of botched the C++ streams standard in the 90's, which is too bad, streams can be very powerful and useful when done right.  Using it in rpc.cpp may still turn out to be a mistake.

Bottom line is I'd rather call an existing file copy function than make and test my own.
satoshi
Founder
Sr. Member
*
qt
Offline Offline

Activity: 364
Merit: 6610


View Profile
September 06, 2010, 09:45:10 PM
 #12

rpc backupwallet <destination> is in SVN rev 147.
BioMike
Legendary
*
Offline Offline

Activity: 1658
Merit: 1001


View Profile
September 07, 2010, 05:10:36 AM
 #13

Nice. How about being able to set <destination> as a config option and have the function called after every send(/receive?) of bitcoins?
nelisky (OP)
Legendary
*
Offline Offline

Activity: 1540
Merit: 1001


View Profile
September 07, 2010, 09:46:51 AM
 #14

Nice. How about being able to set <destination> as a config option and have the function called after every send(/receive?) of bitcoins?

I'm doing backups every time I create a new address and when I send btcs. Receiving should be safe? Also, I'm collecting other stuff and doing offsite scp based backups of the wallet, meaning I have the trigger embedded in my program logic, but auto backups would be great, as one can look for changes in the backup file and use that as trigger to do the offsite.

But what I *really* think would rock is some kind of event mechanism Smiley
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!