Need more clarification on backing up wallet

[jake262144]

well, the official Bitcoin client is the defacto client proposed at bitcoin.org
it does not look like their policy is to redirect noobs to more user-friendly clients.

True but not exactly a surprise that they don't want the responsibility of having to worry about the alternative clients they endorsed.
Not gonna happen.
DAT is absolutely right comparing the official client to a reference-design device.

[ThomasV]

well, the official Bitcoin client is the defacto client proposed at bitcoin.org
it does not look like their policy is to redirect noobs to more user-friendly clients.

True but not exactly a surprise that they don't want the responsibility of having to worry about the alternative clients they endorsed.
Not gonna happen.

this is understandable.
I guess the official client should use a deterministic wallet by default.

[jake262144]

I guess you should make your alternative client as user-friendly⁽¹⁾ as possible and go for the kill.
Easy, safe, secure: Electrum. Why not?
Be the nonref triple-fan design slapping the loud and inefficient reference one around.

My understanding is, Bitcoin desperately needs better clients if it's going to grow.
Look at the issues being constantly raised here: lost or stolen wallet files, slow block chain download, having to run -rescan... the issues are all client-side.

Notes:
(1) I'm not talking only about the UI here.

[DeathAndTaxes]

Er... thanks guys, this is suddenly too much information for me to digest.
So basically if I have no transactions or not expecting any, my backup will be good for years?
(I am using the official client)

Don't use time like years.  The backup is good for 100 transactions using new addresses.  That might be years, possibly centuries or it might be days.  Obviously if you never use a new address the backup will be good forever.

The backup is good for 100 transactions using new addresses.

[yyohko]

Er... thanks guys, this is suddenly too much information for me to digest.
So basically if I have no transactions or not expecting any, my backup will be good for years?
(I am using the official client)

Don't use time like years.  The backup is good for 100 transactions using new addresses.  That might be years, possibly centuries or it might be days.  Obviously if you never use a new address the backup will be good forever.

The backup is good for 100 transactions using new addresses.

This made me think of something, is the wallet.dat cross client compatible?

To be honest, I just got into Bitcoin because I hear the value could go up and the only transaction I have is the one from Bitcoin Faucet. >.<

[DeathAndTaxes]

This made me think of something, is the wallet.dat cross client compatible?

Short answer: No.

Longer answer: it is possible for other clients to use same format & database as the Satoshi client but you shouldn't rely on that.  Yes a universal format or maybe wallet conversion tool would be useful.

[jake262144]

Funny you should mention that DAT.
I've had an open-source wallet decrypter/converter/key extractor somewhere on my todo list for quite awhile. Seems I just can't get there

[Count Schlick]

I was going to start a new thread, but I figure I'll just a bump a relevant thread to reduce clutter (please let me know if that's not good mojo on these boards).

I'm currently running Bitcoin-QT on both Windows and Mac. The Windows software was just a test, the Mac is mine and likely where I'll keep any actual store of bitcoins. I use the OSX backup software "Time Machine" (for those who don't know, it does a differential backup every hour). I assumed this was sufficient and in the case that wallet.dat was corrupted, I could restore from the most recent uncorrupted version. However, some of the keypool stuff I'm reading in threads like these, as well as the info on deterministic wallets, is really throwing me for a loop.

Please bear with me, I geek out over this stuff and I really like to understand how it works so I don't make any mistakes (and because BTC is such an incredible idea)

How I thought it works:
 - wallet.dat contains the private keys associated with each address (each address is really just a hash of the private key?)
 - the balance in your wallet is based on using the private keys for each addresss and verifying against the Block Chain.
 - if the wallet is corrupted after the creation of a new address but before it's freshly backed up, you could lose any coins sent to that new address
After reading about the keypool I assumed that this addressed the above problem scenario, but further reading made me realize that this is not the case

How I now think the keypool works:
 - wallet.dat contains 100 private/public key pairs
 - key pairs are associated with your addresses as well as some (but not all?) transactions
 - when a new address is created or a transaction occurs to/from an address a key is pulled "from the bottom" of the key pool, with remaining keys left to be used at a future date — and a new key repopulates "at the top"
 - my wallet balance is based on private keys for both addresses and transactions, and then verified against the block chain
 - restoring from a backup will maintain my current balance, so long as fewer than 100 actions have taken place since the backup (address creation and transactions)
 - the keypool can be made larger (in some way that I haven't figured out how to do on my mac yet)

How I now think a deterministic wallet works:
 - wallet.dat stores a seed or some other piece of info that determines all future key-pairs (based on the seed? that part might be beyond my current ability to understand)
 - in the case of a corrupt wallet.dat, restoration from any backup, even the original, will restore your full address list, balance, and transaction history to the client, re-generating the "determined" keys and verifying against the block chain.

I think that's it. Am I completely far off? In the ballpark? Assuming I've got the basics down, further questions:
 - How tested are deterministic wallets? Has the re-generation ever been shown to be incorrect?
 - I assume that my current system backup is fine (I have 2 addresses and 3 transactions to date, not exactly a power user)?

Thanks in advance for any help.

[DeathAndTaxes]

How I now think the keypool works:
 - wallet.dat contains 100 private/public key pairs
 - key pairs are associated with your addresses as well as some (but not all?) transactions
 - when a new address is created or a transaction occurs to/ from an address a key is pulled "from the bottom" of the key pool, with remaining keys left to be used at a future date — and a new key repopulates "at the top"
 - my wallet balance is based on private keys for both addresses and transactions, and then verified against the block chain
 - restoring from a backup will maintain my current balance, so long as fewer than 100 actions have taken place since the backup (address creation and transactions)
 - the keypool can be made larger (in some way that I haven't figured out how to do on my mac yet)

Clarification a new address isn't used when you RECEIVE funds to an existing address.  It is only used when you SEND funds from an existing address (the new address is used for change).

[Count Schlick]

Clarification a new address isn't used when you RECEIVE funds to an existing address.  It is only used when you SEND funds from an existing address (the new address is used for change).

Ok I think this is the part that was initially confusing to me. Is this what's being described on blockexplorer.com when viewing an address history?

Every time a transaction is sent, some bitcoins are usually sent back to yourself at a new address (not included in the Bitcoin UI), which makes the balance of a single address misleading.

So the key pool contains 100 addresses, and a new one is used for this process when you send coins.

Either way, my understanding of how the software determines the balance is correct?

[DeathAndTaxes]

Yes and Yes.

What might fit it all together is:

1) Bitcoin only has the concept of inputs & outputs. 
2) All transactions (except coinbase) have an input and all transactions have outputs. 
3) The input of a transaction (every transaction) is the output of some prior transactions. 
4) You can't spend part of an output.

So say you have an output of a prior transaction with 5 BTC.  You want to spend 1 BTC.  It is impossible.  So the client does this.

Input A - 5 BTC
Output B - 1 BTC
Output C - 4 BTC

B is the address of the person you want to pay.  C is a NEW ADDRESS from your keypool.

Coinbase transactions are the only exception.  They are what reward miners and they mint 50 BTC out of thin air.

Transaction fees are simply a difference between input and output.  Using the example above if you wanted to pay a 0.01 BTC tx fee it would look like this:

Input A - 5 BTC
Output B - 1 BTC
Output C - 3.99 BTC

Notice the Input is 0.01 more than the combined outputs.  When the block is constructed the miner adds up all inputs and all outputs and adds the difference to the block subsidy and that becomes the coinbase.  So if this was the only transaction for the block the coinbase would look like this

Input: (NOTHING)
Output:  Address Z 50.01 BTC (miner's reward address)

An example of a real transaction:
http://blockchain.info/tx-index/3395028/f1bec123cb4a1ecef1f6aa1086de44be781f813de359e7da4d2b1a4aa29a7181
Notice the input is 4.68577 (you can click the link as see the output that produced this input and go all the way back to where the coins were born).

There are two outputs
2.70527 & 1.98
One is the change.  We don't know for sure which one but most likely it is the 2.70527

There is a difference between the input 4.68577 and the combined outputs 4.68527.  The difference 0.005 is collected by the miner as a tx fee.  The fee doesn't need to be specified.  Miners collect any difference between inputs and outputs as a fee.

An example of a real coinbase transaction:
http://blockchain.info/tx-index/3395037/09af493daf707bf7e01f930134315cc74c9a6439b332cc648dde6d477109322b
Notice it is NOTHING as input and 50.02351 as the output.
50 BTC is the block subsidy (going to 25 in Dec 2012)
0.02351 is the total tx fees for the block (if you added up all inputs and all outputs in the block there would be a difference of 0.02351)

[Count Schlick]

Yes and Yes.

What might fit it all together is:

1) Bitcoin only has the concept of inputs & outputs. 
2) All transactions (except coinbase) have an input and all transactions have outputs. 
3) The input of a transaction (every transaction) is the output of some prior transactions. 
4) You can't spend part of an output.

So say you have an output of a prior transaction with 5 BTC.  You want to spend 1 BTC.  It is impossible.  So the client does this.

Input A - 5 BTC
Output B - 1 BTC
Output C - 4 BTC

B is the address of the person you want to pay.  C is a NEW ADDRESS from your keypool.

[... snip]

This definitely ties it together for me, thank you. I wasn't sure why coins were "sent back" to me if I transferred them, but that makes sense now. I really think I have a handle on the basics.

It seems like every simple question I ask ends up being a few days long adventure in learning and reading, but it's worth it. Thanks again for your help and for being so patient. On to the next adventure!