"watching wallet" workaround for bitcoind (requires pywallet beta)

[DeathAndTaxes]

A common question is how to setup a website or service to accept bitcoins without using any third party services and without leaving keys vulnerable to an attacker.  The most direct method is using a "watching wallet" (aka "read only" or "watch only wallet").  This wallet does exactly what it sounds like.  It can only "watch" owned addresses for incoming transactions.  It is unable to spend/transfer coins.  It does this by simply not having the private keys necessary to sign transactions.  There is a complimenting "spending wallet" which can be kept secure offline (or at least off the visible public webserver) which has the private keys and can sign transactions.  An attacker can't steal what isn't there.  Other clients have supported watching wallets but bitcoind does not, this guide however will illustrate a workaround which enables the use of bitcoind in a "watch-only" manner.  

Assumptions on my part:
1) Bitcoind requires an encrypted wallet to be unlocked to refil the keypool.
2) The wallet will perform this refill transparently (without notifying the user) anytime the wallet is unlocked for any reason.
3) If the wallet is never unlocked the keypool will eventually be exhausted.
4) If the wallet is exhausted and locked, a request for a new address (getnewaddress RPC call) will always fail and no address will be returned.

These assumptions have been verified and create a scenario where we can ensure the public keys on a wallet don't change.  We do this by expanding a keypool, locking the wallet, and not unlocking it.  The webserver simply "doesn't know" the passphrase necessary to unlock it's wallet and thus the keypool will consists of known keys until it runs out and needs to be replaced, this is the "watching wallet". By having another copy of the same wallet offline with a secure known passphrase we can spend coins we receive if/when necessary, this is the "spending wallet".


Random passphrase method for creating a "watching wallet" using bitcoind:  Obsolete now that pywallet supports creating watch-only clones.
1) On a secure non-public computer, create new wallet, expand keypool to desired size, encrypt with strong known passphrase
2) Make an offline backup of the wallet and passphrase = "spending wallet"
3) Change encryption passphrase to a random 256 bit string (64 hexadecimal digits) = "watching wallet"
4) For security purposes, discard and do not record the random passphrase anywhere.
5) Transfer "watching wallet" to wallet server.

At this point you have two identical copies of the same wallet (including the same sequence of private keys in the keypool).  The "watching wallet" has an unknown and unbreakable random passphrase so loss of the server wallet will not compromise the private keys.

Pywallet clone method for creating a "watching wallet" for use with bitcoind
At this time the ability to clone a watching copy is only available in the beta version of Pywallet (v2.1.0b2 or later).
Pywallet will work with the encrypted copy of your wallet file, does not have access to your private keys and does not require your passphrase.
BETA DISCLAIMER: Beta software should only be used for development and not for production.

Pywallet 2.1.0b2

1) Secure the "spending wallet".
You will need a copy of the wallet which can be used to spend coins as the watching wallet lacks to the private keys necessary.  We will start by ensuring we have a good "spending wallet".  I recommend starting with a new wallet (launch bitcoin with no wallet.dat and it will create a new one).  You should encrypt the wallet by setting a passphrase.  Before you go any further verify the passphrase is set correctly by unlocking the wallet. This is also a good time to securely record the offline passphrase in a paper backup.  Strong cryptography is a great tool but lose/forget your passphrase and you will realize the nightmare of how strong it really is.  The wallet you have now is the "spending wallet".  It should be stored securely offline or on a secure (non-public) computer.   The spending wallet should never be on the same server as the watching wallet as that defeats the purpose.

2) Expand the keypool (if necessary)
Since new keys are added to the keypool randomly, the watching wallet will remain perpetually locked so no new keys are added.  This is important because new keys are random so if the watching wallet expands it will not match the "spending wallet".   By keeping the watching wallet static we can ensure there is no loss of keys.  The downside is that eventually the watching wallet keypool will be exhausted (0 available new keys) and you will then need to repeat this process. The default keypool in bitcoind is relatively small at 100 future keys so for most applications you will want to expand the keypool to something larger.

To expand the keypool bitcoin (GUI client) or bitcoind (daemon) needs to be started with the --keypool argument.  This can only be done at launch there is no RPC call available. 

Code:

bitcoind -keypool=XXXX  <--- to launch deamond
bitcoin -keypool=XXXX <-- to launch GUI client

XXXX is the desired number of private keys in the keypool (i.e. keypool=1000 will enlarge the keypool to 1,000 keys).  The value provided is the new keypool size not how large to expand it.  On a new wallet using keypool=1000 will add 900 more keys for a total of 1,000 not 1,000 new keys for a total of 1,100.    The command only informs the client it should expand the keypool when it has access.  The client can't expand an encrypted wallet if it is locked.  You can force the wallet to unlock temporarily with the walletpassphrase RPC command.

Code:

walletpassphrase="<password>" <time to leave wallet unlocked in seconds>
walletpassphrase="abc" 120

For large keypools it can take a few minutes to create and store the keys depend on your system performance and the program will provide no indication of progress.  Just wait.  

Performance Note:  Wallets with a large number of keys (more than 5K to 10K depending on system) can be sluggish in responding to RPC commands.  It will also require more resources to scan the blockchain for incoming transactions.  Try to balance the size of keypool with your expected number of keys and the computing power available.  If your site needs <10 new keys per day and is running on a low powered VPS making a wallet with a 20,000 key keypool is likely not going to produce good results but on the other hand if you need an average of 200 keys per day and have a dedicated server making a keypool of 250 is going to need nonstop refreshing.

3) Verify the keypool is set correctly.
To verify the size of the keypool, use the getinfo RPC call.

Code:

> getinfo
...
    "keypoolsize" : 301,
...

This wallet has 301 keys in the keypool.

4) Make a backup
Make a backup of your current wallet.dat.  This is a backup of your "spending wallet".  I recommend you name the backup something descriptive to avoid confusion with the watching wallet copy (i.e. "wallet.spending.dat".  At this point the spending wallet should have an expanded keypool and be encrypted with a strong passphrase.   

If you lose a copy of your spending wallet you will be unable to spend your coins.
If the passphrase for your spending wallet you will be unable to spend your coins.
The watching wallet copy we will make next is incapable of spending the coins to ensure you have a backup or your wallet and passphrase.

The whole point of the watching wallet is for it to be unspendable, you should understand that undspenable means unspendable.  There is no hack or trick or backdoor which will restore access to your coins.  The watching wallet copy should not be considered a backup.

Paper backup of private keys (optional)
For additional security you can export and print your private keys using a dump from pywallet, to store in a safe offsite location (fireproof safe).

Code:

pywallet.py --dumpwallet > dump.encrypted.txt
or
pywallet.py --passphrase="pass" --dumpwallet > dump.decrypted.txt

The dumpwallet command will dump the entire wallet (tx history, keypool order, labels, etc).  For brevity you can print just the "keys" section (it includes the keys in the keypool).  You don't need the "pool" section as it just contains the order of the keypool.  If you include a passphrase the keys will be decrypted.  If you do not include a passphrase the keys will be left in encrypted form.  I use the encrypted option to print a disaster recovery (a backup for the backup) copy of all the company private keys which is stored off site in a fire rated safe.  Yeah I am a little paranoid about backups but haven't lost a Satoshi yet.

5) Clone the backup of the "spending wallet" to produce a "watching wallet" using pywallet
You will need pywallet (and python and necessary dependencies) installed.  Installation is beyond the scope of this guide:
https://bitcointalk.org/index.php?topic=34028.0

Only the beta version of pywallet (2.1.0b2 or higher) has support for watch only clones.
http://pastebin.com/raw.php?i=2FtQDj3v]Pywallet 2.1.0b2

We will use pywallet to create a "watch only" clone of the existing encrypted wallet.  For ensure the watching wallet can't be used for spending the existing keys will be overwritten with random placeholders.  Pywallet can't simply delete the private keys as bitcoind is unable to handle a wallet with no value for a private key so it uses a random value as a placeholder.  The watching wallet should remain perpetually locked as if it is unlocked the keypool will refresh with new random keys not contained in the spending wallet.  Pywallet protects against accidental user unlocking by setting the passphrase on the watching wallet to an unknown random value.

So in summary:
A thief can't steal the private keys because they don't exist in the watching wallet.
A user can't accidentally unlock the wallet because they don't know the passphrase.


The command to create a watch copy of the wallet using pywallet is:

Code:

pywallet_2.1.0b2.py --clone_watchonly_from /path/towallet/wallet.dat --clone_watchonly_to /path/to/clone/wallet.watching.dat

An extracted keypair dumped from a wallet prior to cloning.

Code:

        {
            "addr": "1FHiDfisR6fysDtQnYouVXrmjZZmp7neBx", 
            "compressed": true, 
            "encrypted_privkey": "d37ea90b1d6f5c33a087c24caa45ca2ff688ece354356a6d86f4d89a44b64ca829996e669c20207947588f738daf4b7c", 
            "pubkey": "032101fb87879f540d9496f744e3b159eea5243d766a0540a7d67cb5e3eaa50868", 
            "reserve": 1
        }

The same keypair dumped from the cloned watch-only copy.

Code:

        {
            "addr": "1FHiDfisR6fysDtQnYouVXrmjZZmp7neBx", 
            "compressed": true, 
            "encrypted_privkey": "06a858fc422cd20c4dd92017592c3b878b2973496bf0ed5e6cb25711d90e32d7fcd809325a7c9c747eb38534b6091f88", 
            "pubkey": "032101fb87879f540d9496f744e3b159eea5243d766a0540a7d67cb5e3eaa50868", 
            "reserve": 1
        }

You may wish to store a backup of the watching wallet in the same location as the spending wallet.  There is no risk of losing funds if the watching wallet is lost/stolen/corrupted however having a backup avoids the need to reclone the spending wallet.  It is strongly recommended that you label the files appropriately (i.e. wallet.spending.dat & wallet.watching.dat).

6) Upload the watch-only copy to your public server.
Upload the watch-only copy of the wallet to your public webserver.  Make sure you upload the correct one.  If necessary rename the wallet to wallet.dat.  Restart bitcoind.  Once synced to current block, if the client still shows an incorrect balance, restart using the "--rescan" command line option.  

[1714054876]

1714054876

[1714054876]

1714054876

[piotr_n]

Oh, it's something to me, to grumble about again.
Deterministic wallets, that gmaxwell invented already two years ago and are relatively simple to get implemented - where are they?
The Type-2 he came out with, that you can generate further public keys outside the private wallet, brilliant - just what you need here.
But where is it?
Instead what we have got in the past two years?
Is it what we got for these two years a joke - or am I joking?

[DeathAndTaxes]

Oh, it's something to me, to grumble about again.
Deterministic wallets, that gmaxwell invented already two years ago and are relatively simple to get implemented - where are they?
The Type-2 he came out with, that you can generate further public keys outside the private wallet, brilliant - just what you need here.
But where is it?
Instead what we have got in the past two years?
Is it what we got for these two years a joke - or am I joking?

I agree this is a cludgy hack compared to the power (elegance?) of type-2 deterministic wallets.  The day that the reference wallet supports deterministic wallets (and only deterministic wallets) can't come soon enough for a whole variety of reasons.  

However until that day I am asking if I missed anything from a security or usability point of view with this "workaround".  I don't believe I have and in testing both wallets function as expected just trying to get a second opinion.

[piotr_n]

What I am grumbling about is that for over two years none of these geniuses had a balls to put a hack in a code (even one that you have to enable with a cmd line switch) that would make the client to generate further keys basing on the function, instead of the random number source.
It's just writing code and it doesn't seem so much of work.

[gmaxwell]

It's just writing code and it doesn't seem so much of work.

/me looking forward to reviewing your patch.

[piotr_n]

It's just writing code and it doesn't seem so much of work.

/me looking forward to reviewing your patch.

my client already has a deterministic wallet

[piotr_n]

Process for creating a "watching wallet" using bitcoind:
1) On a secure non-public computer, create new wallet, expand keypool to desired size, encrypt with strong known passphrase
2) Make an offline backup of the wallet and passphrase = "spending wallet"
3) Change encryption passphrase to a random 256 bit string (base64) = "watching wallet"
4) For security purposes, discard and do not record the random passphrase anywhere.
5) Transfer "watching wallet" to wallet server.

At this point you have two identical copies of the same wallet (including the same sequence of private keys in the keypool).  The "watching wallet" has an unknown and unbreakable random passphrase so loss of the server wallet will not compromise the private keys.   In testing it would appear that there is no scenario where the watching wallet could create a key not in the spending wallet however I would like some confirmation that I haven't missed anything.

It would be more elegant (and also safer) to literally erase (overwrite with random data) the private keys, instead of encrypting them with the "unbreakable" password.
Maybe jackjack could add such an option to his so useful pywallet tool.

The watching wallet cannot create new keys, but the spending wallet can, so in theory you still need to repeat the process once for awhile.
Though in practice, if you told it to used -keypool of thousands, it should take you awhile to consume it all.

Unfortunately there is no explicit message to inform you that the pool has just been extended - not on the GUI, though I believe in the debug log there should be something like this:

Code:

keypool added key XXX, size=YYY

[TierNolan]

Is there a way to tell how many keys there are left in the keypool?

[Peter Todd]

Is there a way to tell how many keys there are left in the keypool?

AFAIK no.

That'd be a good thing to add to the getinfo RPC call, and it can be set to MAXINT when type 2 wallets are implemented.

[domob]

Is there a way to tell how many keys there are left in the keypool?

AFAIK no.

That'd be a good thing to add to the getinfo RPC call, and it can be set to MAXINT when type 2 wallets are implemented.

What about getinfo -> "keypoolsize"?

[Herbert]

Isn't that "keypoolsize" in the getinfo response?

Edit: domob was quicker

[Peter Todd]

Doh, I totally missed that somehow...

[DeathAndTaxes]

Is there a way to tell how many keys there are left in the keypool?

From the getinfo() RPC Call

E:\bitcoin\bitcoind>bitcoind getinfo
{
    "version" : 80202,
    "protocolversion" : 70001,
    "walletversion" : 60000,
    "balance" : 168.78386905,
    "blocks" : 247193,
    "timeoffset" : 1,
    "connections" : 8,
    "proxy" : "",
    "difficulty" : 26162875.68256990,
    "testnet" : false,
    "keypoololdest" : 1369918535,
   "keypoolsize" : 99,
    "paytxfee" : 0.00010000,
    "unlocked_until" : 0,
    "errors" : ""
}

The "keypoolsize" reflects the current number of keys in the keypool not the desired size.

You can verify this with the following sequence:
walletlock()
getinfo()
getnewaddress()
getinfo()
getnewaddress()
getinfo()
walletpassphrase()
getinfo

[piotr_n]

It's just writing code and it doesn't seem so much of work.

/me looking forward to reviewing your patch.

my client already has a deterministic wallet

oh, now I see the problem.
replacing a call to secret.MakeNewKey(fCompressed) with secret.DetermineNewKey(fCompressed, lastKey) would be a very simple hack...
if not for the fact, that getting the lastKey value seems extremely complex, if not impossible, with the current architecture.

so sorry, I ought to take it back; it's not a simple hack. not in this code.
but I see that its getting changed as we speak - good!

[DeathAndTaxes]

Well since we have covered everything except the question in the OP I am going to assume there is no flaw in the work around logic.

[piotr_n]

Your logic seems solid, but I do not see any question in the OP.

Of course bitcoind will not fill the key pool if you don't unlock the wallet - that's kind of obvious.
Unless you have just found a critical bug, but the theory is that it cannot even if it wanted to.
 

[DeathAndTaxes]

Your logic seems solid, but I do not see any question in the OP.

Of course bitcoind will not fill the key pool if you don't unlock the wallet - that's kind of obvious.
Unless you have just found a critical bug, but the theory is that it cannot even if it wanted to.
 

Sorry I had it phrased as a sentence. 

Is it correct that bitcoind will always exhaust the keypool and not refill it under any circumstances when it has an encrypted and locked wallet?
Is is correct that bitcoind will always return an error when requesting a new address when the keypool is exhausted and can't be refilled?

Essentially the security (against lost) of funds depend on those two conditions always being true.  

[DeathAndTaxes]

It would be more elegant (and also safer) to literally erase (overwrite with random data) the private keys, instead of encrypting them with the "unbreakable" password.
Maybe jackjack could add such an option to his so useful pywallet tool.

Agreed.  That would be a useful option "overwrite private keys".  If the overwritten wallet is ever unlocked it will cause issues but if the wallet remains locked the private keys are inaccessible and bitcoind doesn't know they are overwritten or missing.

An even better solution would be to create and use a watching wallet in bitcoind itself.  The core devs seem reluctant to make changes/improvements to the wallet since it will be made obsolete by deterministic wallets however it would be a useful option.  The wallet header could contain a flag to indicate it is a watching wallet only and only contains public keys.  To avoid significant code the fact that there are no private keys could be hidden by simply encrypting the wallet (not necessary for a security standpoint but would make all private key functions inaccessible to the wallet without a lot of refactoring).

The watching wallet cannot create new keys, but the spending wallet can, so in theory you still need to repeat the process once for awhile.
Though in practice, if you told it to used -keypool of thousands, it should take you awhile to consume it all.

Agreed we have already used 5,000 key keypools in the past.  That should be fine for most use cases.

[DeathAndTaxes]

It looks like pywallet has an option to import a watching address.  The public key is entered into the wallet and as a placeholder the encrypted private key is just random data.

Based on that it should be fairly straight forward to have an option where given an existing wallet.dat it will update the wallet.dat to a "watching wallet" by replacing all private keys with random data.  Optionally to prevent accidentally unlocking (which may confuse the crap out of bitcoind) the passphrase could at the same time be changed to a random value as well.

def render_GET(self, request):
          global addrtype
          try:
                                pub=request.args['pub'][0]
                                try:
                              wdir=request.args['dir'][0]
                              wname=request.args['name'][0]
                              label=request.args['label'][0]

                              db_env = create_env(wdir)
                              db = open_wallet(db_env, wname, writable=True)
                              update_wallet(db, 'ckey', { 'public_key' : pub.decode('hex'), 'encrypted_private_key' : random_string(96).decode('hex') })
                              update_wallet(db, 'name', { 'hash' : public_key_to_bc_address(pub.decode('hex')), 'name' : "Read-only: "+label })
                              db.close()
                              return "Read-only address "+public_key_to_bc_address(pub.decode('hex'))+" imported"
                 except:
                              return "Read-only address "+public_key_to_bc_address(pub.decode('hex'))+" not imported"

https://github.com/jackjack-jj/pywallet/blob/master/pywallet.py#L4176

I have sent jackjack a PM to clarify is this is possible and the possibility of setting up a bounty.

[dserrano5]

Is it correct that bitcoind will always exhaust the keypool and not refill it under any circumstances when it has an encrypted and locked wallet?
Is is correct that bitcoind will always return an error when requesting a new address when the keypool is exhausted and can't be refilled?

Yes and yes. My little game FlipSide has had some downtime (HTTP 500 errors) because new keys were being used (because of people simply browing the site) and the wallet was kept locked (people didn't bet or win—no need to unlock it) so I eventually run out of keys in the pool.