Bitcoin Forum

Hi, I would like to know if we need to keep taking backups of our wallet.dat as we keep having transactions or we only need to backup once for the entire lifetime?

Let's say I backup my wallet.dat in January, have some transactions in February, then my PC crashed in March.
When I restore my wallet.dat, are my February transactions gone?

if you use a deterministic wallet, you can back it up once and for all.
if you use the official client, then you need to back it up every 100 transactions

I don't think so. As the BTC Wallet downloads new blocks (blk001.dat & blkindex.dat) it will see your bitcoin address (as stored in your wallet.dat) and update the transactions?

I think the way it works is, All transactions EVER made, are in the block container, then bitcoin.exe just filters the results by wallet.dat

So no matter what, you'll always have your transaction history as it's on the interwebs!

ThomasV, How do I use a deterministic wallet?
Also, what will happen if I fail to backup every 100 transactions? (when my PC crash)

Already discussed over, and over, and over again, e.g. here (https://bitcointalk.org/index.php?topic=60604.msg706263#msg706263)
Do your research first, post advice later.

The traditional wallet contains, among other goodies, private keys for your bitcoin addresses. A pool of 100 yet-unused keys is stored inside it by default.
Every transaction you send, some of the incoming transactions, and every receive address you generate inside the client use one of those pre-generated keys which causes the key-pool to be repopulated with new keys.
A sufficiently old wallet backup will be out of keys which might render some of your bitcoins unaccessible.


Deterministic wallets are only supported by alternative bitcoin clients (https://bitcointalk.org/index.php?board=37.0).

here are the deterministic wallets I know of:
 - Electrum (lightweight client written by me) https://bitcointalk.org/index.php?topic=50936
 - Armory (full client)
 - Bitcoinspinner (lightweight client for mobile phones)

if you fail to backup your wallet with the official client, you might lose some coins when your PC crashes.
but I guess the client warns you when its pool of addresses gets deprecated.

If it only were that simple... The pool is always being repopulated, no problem there. Nothing for the client to warn about.
The problem arises when one has to rely on a very old wallet backup missing some of the recently-generated keys.

exactly. but I was under the impression that the client warns you when the pool gets repopulated.
if it doesn't, how do you know when you need to back up?

It doesn't. The pool is ALWAYS being repopulated.  The client also has no idea when your last backup was.

As example might help:
Your keypool by default has 100 keys.  These are unused keys, the "next 100".
You make a backup today.  The backup contains all existing transactions plus the next 100 into the future.
If you lose your wallet.dat in the future as long as no more than 100 new addresses have been used you lose nothing.
If >100 new addresses have been used any coins at any addresses beyond the 100 are lost forever.

EVERYTIME you click [New Address] button or send funds (new address used for change) one address from keypool is used.  The client then puts 1 new address into the keypool so the keypool size doesn't change however the addresses in it do.  Thus your backup eventually will be out of date.  

You can set a larger keypool and honestly given how small keys are I don't know why client doesn't use a more "noob safe" number like 500 keys in the keypool by default.

This is not how I thought it worked.
Given that 100 addresses are pre-generated, I thought it would wait until they are all used until it generates another batch of 100.
This would be way safer, because you could warn the user that it is time to do another backup.

No problem.  Keypool confuses everyone at least once.  :)
A non-refilling pool would require backups exactly at every 100 transaction increments and forgetting would make your client operating without backup immediately.  The pool continual refill method is better because you can make backups at anytime and regain 100 (or 1000) addresses of future protection.  

One way to combine both methods is to have a backup button in the client.  Click backup, client makes a copy of wallet.dat and records the last valid key in keypool.  Thus it could warn the user when last backup is out of date. 

I run with a keypool of 1000 keys.  Each key takes up about 1KB of space so my backup is ~1MB larger but ensures than any backup in last 1000 addresses is valid.  Given the trivial cost to store 1MB it seems silly to run w/ a smaller keypool.

Way safer? No way ;D
Your approach does not resolve the issue of having to fall back to an ancient backup of the wallet file.

It's the end user's responsibility to properly manage wallet backups.
I can't say this is the prudent approach as only a minority of users can be expected to follow a secure backup routine.

EDIT::*nods head* Hello DAT. I requested that auto backups be done by default once the wallet has been encrypted but the devs seem to have turned the deaf ear.

are you saying that the official client is written for a minority of users?
why doesn't it have a deterministic wallet?

The official client likely will never be casual user friendly.  It is the reference design.  It tests out new functionality but it likely will never be the best client for casual users.  Personally I think this is a good thing.  We don't want one defacto client. 

Er... thanks guys, this is suddenly too much information for me to digest.
So basically if I have no transactions or not expecting any, my backup will be good for years?
(I am using the official client)

Unfortunately, such seems to be the truth. The official client is convoluted about the most important aspects. It buries the wallet.dat file where the average non-techie user can't find it, it does ZERO backups, and offers NO encryption by default. I raised all those issues at least twice but no one seems to want to listen.
Yeah, why didn't the devs (or any other user) write the code and merge it into the client? I suspect insufficient motivation.

well, the official Bitcoin client is the defacto client proposed at bitcoin.org
it does not look like their policy is to redirect noobs to more user-friendly clients.

Pretty much.
I suggest you encrypt the wallet (Settings -> Encrypt Wallet) and do a backup once a month. Just don't store the backups on the same machine.

No but in my opinion Bitcoin isn't casual user ready yet.  There is a reason the major version number is 0.

An analogy.  Last time you went looking for a credit card did you go to VISA.COM? :)

The reality is the official client is a reference implementation.  A basis for designing other client and something to reference.  It likely will never be casual user friendly.  I don't think that is a deal killer.  No client is going to be optimal for every user and the Satoshi client has the obligation of being the reference model.

yes.
do not forget to wait until the blockchain is fully synchronized everytime you use it; if you use it rarely the synchronization will take some time.

True but not exactly a surprise that they don't want the responsibility of having to worry about the alternative clients they endorsed.
Not gonna happen.
DAT is absolutely right comparing the official client to a reference-design device.

this is understandable.
I guess the official client should use a deterministic wallet by default.

I guess you should make your alternative client as user-friendly⁽¹⁾ as possible and go for the kill.
Easy, safe, secure: Electrum. Why not?
Be the nonref triple-fan design slapping the loud and inefficient reference one around.

My understanding is, Bitcoin desperately needs better clients if it's going to grow.
Look at the issues being constantly raised here: lost or stolen wallet files, slow block chain download, having to run -rescan... the issues are all client-side.

Notes:
(1) I'm not talking only about the UI here.

Don't use time like years.  The backup is good for 100 transactions using new addresses.  That might be years, possibly centuries or it might be days.  Obviously if you never use a new address the backup will be good forever.

The backup is good for 100 transactions using new addresses.

This made me think of something, is the wallet.dat cross client compatible?

To be honest, I just got into Bitcoin because I hear the value could go up and the only transaction I have is the one from Bitcoin Faucet. >.<

Short answer: No.

Longer answer: it is possible for other clients to use same format & database as the Satoshi client but you shouldn't rely on that.  Yes a universal format or maybe wallet conversion tool would be useful.

Funny you should mention that DAT.
I've had an open-source wallet decrypter/converter/key extractor somewhere on my todo list for quite awhile. Seems I just can't get there :)

I was going to start a new thread, but I figure I'll just a bump a relevant thread to reduce clutter (please let me know if that's not good mojo on these boards).

I'm currently running Bitcoin-QT on both Windows and Mac. The Windows software was just a test, the Mac is mine and likely where I'll keep any actual store of bitcoins. I use the OSX backup software "Time Machine" (for those who don't know, it does a differential backup every hour). I assumed this was sufficient and in the case that wallet.dat was corrupted, I could restore from the most recent uncorrupted version. However, some of the keypool stuff I'm reading in threads like these, as well as the info on deterministic wallets, is really throwing me for a loop.

Please bear with me, I geek out over this stuff and I really like to understand how it works so I don't make any mistakes (and because BTC is such an incredible idea)

How I thought it works:
 - wallet.dat contains the private keys associated with each address (each address is really just a hash of the private key?)
 - the balance in your wallet is based on using the private keys for each addresss and verifying against the Block Chain.
 - if the wallet is corrupted after the creation of a new address but before it's freshly backed up, you could lose any coins sent to that new address
After reading about the keypool I assumed that this addressed the above problem scenario, but further reading made me realize that this is not the case

How I now think the keypool works:
 - wallet.dat contains 100 private/public key pairs
 - key pairs are associated with your addresses as well as some (but not all?) transactions
 - when a new address is created or a transaction occurs to/from an address a key is pulled "from the bottom" of the key pool, with remaining keys left to be used at a future date — and a new key repopulates "at the top"
 - my wallet balance is based on private keys for both addresses and transactions, and then verified against the block chain
 - restoring from a backup will maintain my current balance, so long as fewer than 100 actions have taken place since the backup (address creation and transactions)
 - the keypool can be made larger (in some way that I haven't figured out how to do on my mac yet)

How I now think a deterministic wallet works:
 - wallet.dat stores a seed or some other piece of info that determines all future key-pairs (based on the seed? that part might be beyond my current ability to understand)
 - in the case of a corrupt wallet.dat, restoration from any backup, even the original, will restore your full address list, balance, and transaction history to the client, re-generating the "determined" keys and verifying against the block chain.

I think that's it. Am I completely far off? In the ballpark? Assuming I've got the basics down, further questions:
 - How tested are deterministic wallets? Has the re-generation ever been shown to be incorrect?
 - I assume that my current system backup is fine (I have 2 addresses and 3 transactions to date, not exactly a power user)?

Thanks in advance for any help.

Clarification a new address isn't used when you RECEIVE funds to an existing address.  It is only used when you SEND funds from an existing address (the new address is used for change).

Ok I think this is the part that was initially confusing to me. Is this what's being described on blockexplorer.com when viewing an address history?

So the key pool contains 100 addresses, and a new one is used for this process (https://en.bitcoin.it/wiki/Change) when you send coins.

Either way, my understanding of how the software determines the balance is correct?

Yes and Yes.

What might fit it all together is:

1) Bitcoin only has the concept of inputs & outputs. 
2) All transactions (except coinbase) have an input and all transactions have outputs. 
3) The input of a transaction (every transaction) is the output of some prior transactions. 
4) You can't spend part of an output.

So say you have an output of a prior transaction with 5 BTC.  You want to spend 1 BTC.  It is impossible.  So the client does this.

Input A - 5 BTC
Output B - 1 BTC
Output C - 4 BTC

B is the address of the person you want to pay.  C is a NEW ADDRESS from your keypool.

Coinbase transactions are the only exception.  They are what reward miners and they mint 50 BTC out of thin air.

Transaction fees are simply a difference between input and output.  Using the example above if you wanted to pay a 0.01 BTC tx fee it would look like this:

Input A - 5 BTC
Output B - 1 BTC
Output C - 3.99 BTC

Notice the Input is 0.01 more than the combined outputs.  When the block is constructed the miner adds up all inputs and all outputs and adds the difference to the block subsidy and that becomes the coinbase.  So if this was the only transaction for the block the coinbase would look like this

Input: (NOTHING)
Output:  Address Z 50.01 BTC (miner's reward address)

An example of a real transaction:
http://blockchain.info/tx-index/3395028/f1bec123cb4a1ecef1f6aa1086de44be781f813de359e7da4d2b1a4aa29a7181
Notice the input is 4.68577 (you can click the link as see the output that produced this input and go all the way back to where the coins were born).

There are two outputs
2.70527 & 1.98
One is the change.  We don't know for sure which one but most likely it is the 2.70527

There is a difference between the input 4.68577 and the combined outputs 4.68527.  The difference 0.005 is collected by the miner as a tx fee.  The fee doesn't need to be specified.  Miners collect any difference between inputs and outputs as a fee.

An example of a real coinbase transaction:
http://blockchain.info/tx-index/3395037/09af493daf707bf7e01f930134315cc74c9a6439b332cc648dde6d477109322b
Notice it is NOTHING as input and 50.02351 as the output.
50 BTC is the block subsidy (going to 25 in Dec 2012)
0.02351 is the total tx fees for the block (if you added up all inputs and all outputs in the block there would be a difference of 0.02351)

This definitely ties it together for me, thank you. I wasn't sure why coins were "sent back" to me if I transferred them, but that makes sense now. I really think I have a handle on the basics.

It seems like every simple question I ask ends up being a few days long adventure in learning and reading, but it's worth it. Thanks again for your help and for being so patient. On to the next adventure!