Wallet import/export: bitkeys format

[Pieter Wuille]

Hello all,

I've been working on a patch for bitcoin which allows exporting and importing of wallets in the so-called bitkeys format (initially suggested here: http://bitcointalk.org/index.php?topic=3638.msg52066#msg52066).
Now there are still some issues with this, so i thought maybe it's a good idea to discuss some details in a separate thread.

First of all, there are basically four classes of keys:

• (1) Used keys, which have unredeemed transaction outputs linked to them (and thus contain funds)
• (2) Used keys, which only have redeemed transaction outputs linked to them (worthless, but the addresses may be used still by others)
• (3) Unused keys (whose corresponding addresses may have been published)
• (4) Reserve keys (whose corresponding addresses have typically never been divulged, but may have been divulged at the time the export is imported again)

Furthermore, for the first three classes, each key may either be in the address book or not, and have a label/account name or not (which may or may not be considered relevant information for a wallet export).

Especially for the reserve keys, I'm not sure what the best way to proceed is when importing. Clearly, you don't want to add them to the reserve keys when importing (either they've been used in between already, in which case they should definitely not become a reserve key but still imported, or they haven't ever been used, probably never will, and importing them would just clutter the wallet).

The current proposed bitkeys format does not contain information about how the key was used, whether it is a reserve key, or what label it has. Maybe it is useful to add that information to the format?

Currently, my dev branch (https://github.com/sipa/bitcoin/tree/walletdump) produces exports with lines like this (warning: valid, importable, testnet keys):

v=1
91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY,10806 # moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L,10870 # mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV,10870 # midiVjwgBCkMSup4X2FifFinwLhpHYe2wn (unused)
93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT,10870 # (reserve)

The first line is a class-1 key, with an unmatured generation output (so not yet really 50.01 BTC). The second line is of class 2, the third of class 3, and the fourth of class 4. Notice that the block number for keys without balance is always the current block number (since there is definitely - unless in case of a chain split - no available money transferred to these before), and that the address for reserve keys is not revealed.

So, I suggest to a third field, which distinguishes between those classes, either a boolean active/inactive, or some more elaborate form. Maybe an optional fourth field that contains the label for adding it to the address book is useful as well?

The format of the private keys is described on http://bitcointalk.org/index.php?topic=3906.msg55643#msg55643, and also contains a version number and a checksum, like addresses.

Ideas/comments/suggestions/remarks/bitcoins?

[Mike Hearn]

As you're writing the code for this, it's best if you take the lead. Adding additional fields for labels sounds OK to me, however please be aware that allowing arbitrary user-provided text in the format complicates it significantly (character encodings, escaping of the commas/newlines/hashes etc).

[Gavin Andresen]

I'd suggest using a standard, existing format instead of inventing a new one.

Either get rid of the version number and do standard CSV:

Code:

91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY,10806,moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD,"50.01 BTC"
92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L,10870,mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx,"0.00 BTC"
 ... etc (always 4 columns)

Or maybe .ini-format, with each private key a different section:

Code:

version=1
[91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY]
block=10806
publickey=moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD
amount=50.01
[92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L]
block=10870
  ... etc

(bitcoin already contains .ini-file-parsing code, from boost)


I think there are lots of advantages to using an already-standard file format.

[BitterTea]

I'd suggest using a standard, existing format instead of inventing a new one.

I'm assuming this functionality is going to be exposed via JSON-RPC, at least at first. Why not have it return a JSON representation of the private keys?

[jgarzik]

Keys have a standard PEM format...

[Hal]

I would focus on use cases.

1. Back up all the keys in a wallet, and later restore the wallet state if the wallet is deleted or corrupted.

2. Merge two wallets by exporting all the keys in one, and importing to another.

3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.

Any others?

I wonder whether it makes sense to use bitkeys format for case 1 and 2. Wallets do have other things, which is part of your problem. I'd suggest that case 1 & 2 should use wallet formats.

This would mean you don't have to worry about reserve keys. I'd treat the other key classes the same.

Case 3 & 4 are very similar. They suggest you should have an export-and-delete function. Gavin suggested: 1. export key to disk file; 2. read disk file, make sure key is there; 3. delete key from wallet.

Case 4 probably should delete the exported key. You can save a copy and re-import it if the recipient never uses it.

For the case 4 import you really want to do: 1. import keys; 2. do a spend from these keys to one of yours; 3. delete imported keys. That assures that ownership transfers. You don't need this for case 3 import as it's your key.

Finally, as far as the address and balance: I don't like putting them into CSV fields, because they are redundant and unnecessary. Your comment fields seem okay, but note that the balance may be out of date. How about a command that would take a bitkeys file, and display the address and current balance? Then you could leave the comments out.

[BitterTea]

Keys have a standard PEM format...

According to wikipedia and this site, PEM is specifically used for X.509 certifications (public keys).

edit... Yeah, I don't think PEM is a good fit, though maybe something following that general convention...

-----BEGIN BITCOIN PRIVATE KEY-----
...
-----END BITCOIN PRIVATE KEY-----

I would focus on use cases.

I agree.

3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

In addition to the uses for security, this use case would also work for transferring some coins between devices you own without having to do a transaction. For instance, if you wanted to transfer 100 BTC to your mobile device to spend, you could do so using this method.

[Gavin Andresen]

Focusing on the use cases is the right approach.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.
Any others?

One variation:

5. Export keys to transfer funds to somebody else.  Recipient is expected to import and then immediately send-to-self.  Sender keeps the keys 'set aside' in the wallet until they are spent by recipient, or, if not spent by recipient (lost in transit maybe, or recipient chokes on a lollipop before having a chance to redeem), may be added back into the sender's balance.

And one usability comment:

Is "selected keys" the right way to go?  Just selecting an amount of bitcoins to export/send seems like a better way to go (with a send-to-self transaction generated, if necessary, to create the right amount), with maybe an "advanced" option later that lets you select specific inputs to export/send.

[BitterTea]

And one usability comment:

Is "selected keys" the right way to go?  Just selecting an amount of bitcoins to export/send seems like a better way to go (with a send-to-self transaction generated, if necessary, to create the right amount), with maybe an "advanced" option later that lets you select specific inputs to export/send.

Yes, this is a very good idea, and what I had in mind as a way to manage separate pools of encrypted keys. You could either let Bitcoin choose keys as it does now to create a transaction, and leave it at "around the amount you entered", or do as you say, with a send to self transaction to get the exact amount.

[Pieter Wuille]

I would focus on use cases.

1. Back up all the keys in a wallet, and later restore the wallet state if the wallet is deleted or corrupted.

2. Merge two wallets by exporting all the keys in one, and importing to another.

3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.

Any others?

I wonder whether it makes sense to use bitkeys format for case 1 and 2. Wallets do have other things, which is part of your problem. I'd suggest that case 1 & 2 should use wallet formats.

I don't like the wallet as more than an internal database for one specific implementation of a bitcoin client. It's not transparent or human readable, hard to keep updated when accessing it using external tools, and maybe very different from what other implementations like to use. It is very fast and hard to corrupt, so it is perfect for an internal database, but for all other applications, i prefer a separate, more interchangeable format.

Simply by adding reserve keys (which are in fact harder to filter out when exporting, since they are available in mapKeys like change keys and used keys), you have everything to have a bitkeys file function as a backup or by limiting its contents, as an interchange format.

Finally, as far as the address and balance: I don't like putting them into CSV fields, because they are redundant and unnecessary. Your comment fields seem okay, but note that the balance may be out of date. How about a command that would take a bitkeys file, and display the address and current balance? Then you could leave the comments out.

I agree, I suggested exporting them as comments because they help making a bitkeys files also a useful way to inspect your wallet (which is maybe also a use case?), and possibly select which keys from it you want to give to someone else (essentially doing use case 3 or 4 by hand, by copy-pasting a part of a bitkeys file).

[Pieter Wuille]

I'd suggest using a standard, existing format instead of inventing a new one.

I'm assuming this functionality is going to be exposed via JSON-RPC, at least at first. Why not have it return a JSON representation of the private keys?

I've though about it. It's a very simple way to avoid any parsing/escaping difficulty, while making it easy to extend it further afterwards. On the other hand, CSV is so simple that you don't even need any library for parsing.

Still, a proposal:

[
  { // moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
    "privkey" : "91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY",
    "firstblock" : 10806
  },
  { // mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
    "privkey" : "92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L",
    "firstblock" : 10870,
    "label" : "My Address"
  },
  { // midiVjwgBCkMSup4X2FifFinwLhpHYe2wn
    "privkey" : "91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV",
    "firstblock" : 10870
  },
  { // reserve
    "privkey" : "93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT",
    "firstblock" : 10870,
    "reserve" : true
  }
]


Does anyone know how JSON libraries cope with comments?

[jgarzik]

Still, a proposal:

[
  { // moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
    "privkey" : "91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY",
    "firstblock" : 10806
  },
  { // mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
    "privkey" : "92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L",
    "firstblock" : 10870,
    "label" : "My Address"
  },
  { // midiVjwgBCkMSup4X2FifFinwLhpHYe2wn
    "privkey" : "91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV",
    "firstblock" : 10870
  },
  { // reserve
    "privkey" : "93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT",
    "firstblock" : 10870,
    "reserve" : true
  }
]

Very nice.  If you have this JSON, you shouldn't need anything else IMO.

Does anyone know how JSON libraries cope with comments?

According to my research, the sad fact is comments are outside JSON scope  You may add faux-entries like

Code:

    "_comment1" : "privkey is the private key associated with your request",
    "_comment2" : "firstblock is the first block where it appeared",

Ugly hack, I know.

[Pieter Wuille]

Ok, I've since decided to use a CSV-like format anyway. It is more compact (for use in QR-encoded form, eg.), and easier to process using for example command-line tools like grep and cut.

Here is a preliminary branch that supports importing and exporting transactions: https://github.com/sipa/bitcoin/tree/walletdump

Working:

• dumping a wallet to file
• importing a file
• watching for others doing transactions with your keys
• rescanning the relevant portion of the block chain after importing
• correct updating of transactions and balance in the GUI

Todo:

• error-handling when a double spend has been tried anyway
• importing of reserve keys
• test

The file contains lines of the form:

Code:

<private key>,<block number>[,<flags[,label]] [# <comment>]

Current flags are:

• R: reserve key
• L: label is present

For example:

92263nMUE7SkR62tMn1Ucu2AxY5nANmQhXRhxHZP6MVxpfLC2EH,10801,L,"Your Address"
934xQoFPHzYkwTvJQwQSPw8QspwVB8kPd1S3wRZsZGbgLKVQPb7,10814
92xRg4At7vN25B1esc768NvW7Exrr2PcgZ78MFNmd58MzXjXGYM,10955,R

[Mike Hearn]

That's great! Keep going! :-)

[bitcoinex]

Keys have a standard PEM format...

+1

http://bitcointalk.org/index.php?topic=2507.msg34009#msg34009

so the format should be :
-----BEGIN EC PARAMETERS-----
BgUrgQQACg==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHQCAQEEII84GK/wX8stbmJWYL/WUM1nPAK1miIBDBeyNuo2vyf4oAcGBSuBBAAK
oUQDQgAE/yRZIKrOj4GBfLFtMYuocJ5QF1Tr9rWMh2ixCyfodDWRWTIU21v3ehDR
NJiAXHKwkhDqQ//i46NHoNUhjvx/lw==
-----END EC PRIVATE KEY-----

(I got this output from "openssl ecparam -out key.pem -name secp256k1 -genkey")

[Mike Hearn]

PEM structures have a lot of data we don't need, and lack data that is useful like block numbers, labels etc.

[Pieter Wuille]

PEM looks useful as well, especially when people want to use bitcoin's keys to sign other things. Shouldn't be too hard to implement that as well.

However, what I'm trying to do is create a self-contained, compact, human readable, stable form of storage for wallets. PEM files are standard but not-so-compact ways of storing purely the cryptographic part.

[bitcoinex]

PEM structures have a lot of data we don't need,

?

and lack data that is useful like block numbers, labels etc.

-----BEGIN EC PRIVATE KEY-----
Label: blahblahblah
Block-number: 12345

MHQCAQEEII84GK/wX8stbmJWYL/WUM1nPAK1miIBDBeyNuo2vyf4oAcGBSuBBAAK
oUQDQgAE/yRZIKrOj4GBfLFtMYuocJ5QF1Tr9rWMh2ixCyfodDWRWTIU21v3ehDR
NJiAXHKwkhDqQ//i46NHoNUhjvx/lw==
-----END EC PRIVATE KEY-----

[Pieter Wuille]

PEM structures have a lot of data we don't need,

?

We only need the 256-bit private key, not curve parameters, generator point and public key, which are stored in the OpenSSL structures (and some of those in .pem files too, judging by their size) as well.

[Hal]

Simply by adding reserve keys (which are in fact harder to filter out when exporting, since they are available in mapKeys like change keys and used keys), you have everything to have a bitkeys file function as a backup or by limiting its contents, as an interchange format.

There's other stuff in wallets, including preferences, address book entries for payees, and account information. bitkeys format won't be suitable for wallet backup if it loses all this.