Bitcoin Forum
April 20, 2014, 02:00:17 AM *
News: Due to the OpenSSL heartbleed bug, changing your forum password is recommended.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2  All
  Print  
Author Topic: Wallet import/export: bitkeys format  (Read 9747 times)
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 14, 2011, 11:00:11 AM
 #1

Hello all,

I've been working on a patch for bitcoin which allows exporting and importing of wallets in the so-called bitkeys format (initially suggested here: http://bitcointalk.org/index.php?topic=3638.msg52066#msg52066).
Now there are still some issues with this, so i thought maybe it's a good idea to discuss some details in a separate thread.

First of all, there are basically four classes of keys:
  • (1) Used keys, which have unredeemed transaction outputs linked to them (and thus contain funds)
  • (2) Used keys, which only have redeemed transaction outputs linked to them (worthless, but the addresses may be used still by others)
  • (3) Unused keys (whose corresponding addresses may have been published)
  • (4) Reserve keys (whose corresponding addresses have typically never been divulged, but may have been divulged at the time the export is imported again)
Furthermore, for the first three classes, each key may either be in the address book or not, and have a label/account name or not (which may or may not be considered relevant information for a wallet export).

Especially for the reserve keys, I'm not sure what the best way to proceed is when importing. Clearly, you don't want to add them to the reserve keys when importing (either they've been used in between already, in which case they should definitely not become a reserve key but still imported, or they haven't ever been used, probably never will, and importing them would just clutter the wallet).

The current proposed bitkeys format does not contain information about how the key was used, whether it is a reserve key, or what label it has. Maybe it is useful to add that information to the format?

Currently, my dev branch (https://github.com/sipa/bitcoin/tree/walletdump) produces exports with lines like this (warning: valid, importable, testnet keys):

v=1
91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY,10806 # moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L,10870 # mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV,10870 # midiVjwgBCkMSup4X2FifFinwLhpHYe2wn (unused)
93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT,10870 # (reserve)

The first line is a class-1 key, with an unmatured generation output (so not yet really 50.01 BTC). The second line is of class 2, the third of class 3, and the fourth of class 4. Notice that the block number for keys without balance is always the current block number (since there is definitely - unless in case of a chain split - no available money transferred to these before), and that the address for reserve keys is not revealed.

So, I suggest to a third field, which distinguishes between those classes, either a boolean active/inactive, or some more elaborate form. Maybe an optional fourth field that contains the label for adding it to the address book is useful as well?

The format of the private keys is described on http://bitcointalk.org/index.php?topic=3906.msg55643#msg55643, and also contains a version number and a checksum, like addresses.

Ideas/comments/suggestions/remarks/bitcoins?

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
1397959217
Hero Member
*
Offline Offline

Posts: 1397959217

View Profile Personal Message (Offline)

Ignore
1397959217
Reply with quote  #2

1397959217
Report to moderator
1397959217
Hero Member
*
Offline Offline

Posts: 1397959217

View Profile Personal Message (Offline)

Ignore
1397959217
Reply with quote  #2

1397959217
Report to moderator
Buy a Blade, Get a 5-Chip Free!
Start Mining with GAWMiners.com
24/7 Live Phone & Tech Support
Free Hosting & Electricity for 1 Year!

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1397959217
Hero Member
*
Offline Offline

Posts: 1397959217

View Profile Personal Message (Offline)

Ignore
1397959217
Reply with quote  #2

1397959217
Report to moderator
1397959217
Hero Member
*
Offline Offline

Posts: 1397959217

View Profile Personal Message (Offline)

Ignore
1397959217
Reply with quote  #2

1397959217
Report to moderator
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
March 14, 2011, 03:26:25 PM
 #2

As you're writing the code for this, it's best if you take the lead. Adding additional fields for labels sounds OK to me, however please be aware that allowing arbitrary user-provided text in the format complicates it significantly (character encodings, escaping of the commas/newlines/hashes etc).

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
Gavin Andresen
Hero Member
*****
qt
Offline Offline

Activity: 1330


Chief Scientist


View Profile WWW

Ignore
March 14, 2011, 05:21:35 PM
 #3

I'd suggest using a standard, existing format instead of inventing a new one.

Either get rid of the version number and do standard CSV:
Code:
91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY,10806,moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD,"50.01 BTC"
92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L,10870,mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx,"0.00 BTC"
 ... etc (always 4 columns)

Or maybe .ini-format, with each private key a different section:
Code:
version=1
[91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY]
block=10806
publickey=moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD
amount=50.01
[92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L]
block=10870
  ... etc
(bitcoin already contains .ini-file-parsing code, from boost)


I think there are lots of advantages to using an already-standard file format.

Will I see you in Amsterdam?
  http://bitcoin2014.com/
BitterTea
Sr. Member
****
Offline Offline

Activity: 294



View Profile

Ignore
March 14, 2011, 05:25:16 PM
 #4

I'd suggest using a standard, existing format instead of inventing a new one.

I'm assuming this functionality is going to be exposed via JSON-RPC, at least at first. Why not have it return a JSON representation of the private keys?
jgarzik
Staff
Hero Member
*****
qt
Offline Offline

Activity: 1260


View Profile

Ignore
March 14, 2011, 07:43:32 PM
 #5

Keys have a standard PEM format...

Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Hal
VIP
Sr. Member
*
expert
Offline Offline

Activity: 314



View Profile

Ignore
March 14, 2011, 07:53:48 PM
 #6

I would focus on use cases.

1. Back up all the keys in a wallet, and later restore the wallet state if the wallet is deleted or corrupted.

2. Merge two wallets by exporting all the keys in one, and importing to another.

3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.

Any others?

I wonder whether it makes sense to use bitkeys format for case 1 and 2. Wallets do have other things, which is part of your problem. I'd suggest that case 1 & 2 should use wallet formats.

This would mean you don't have to worry about reserve keys. I'd treat the other key classes the same.

Case 3 & 4 are very similar. They suggest you should have an export-and-delete function. Gavin suggested: 1. export key to disk file; 2. read disk file, make sure key is there; 3. delete key from wallet.

Case 4 probably should delete the exported key. You can save a copy and re-import it if the recipient never uses it.

For the case 4 import you really want to do: 1. import keys; 2. do a spend from these keys to one of yours; 3. delete imported keys. That assures that ownership transfers. You don't need this for case 3 import as it's your key.

Finally, as far as the address and balance: I don't like putting them into CSV fields, because they are redundant and unnecessary. Your comment fields seem okay, but note that the balance may be out of date. How about a command that would take a bitkeys file, and display the address and current balance? Then you could leave the comments out.

Hal Finney
BitterTea
Sr. Member
****
Offline Offline

Activity: 294



View Profile

Ignore
March 14, 2011, 08:05:13 PM
 #7

Keys have a standard PEM format...
According to wikipedia and this site, PEM is specifically used for X.509 certifications (public keys).

edit... Yeah, I don't think PEM is a good fit, though maybe something following that general convention...

-----BEGIN BITCOIN PRIVATE KEY-----
...
-----END BITCOIN PRIVATE KEY-----

I would focus on use cases.

I agree.

Quote
3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

In addition to the uses for security, this use case would also work for transferring some coins between devices you own without having to do a transaction. For instance, if you wanted to transfer 100 BTC to your mobile device to spend, you could do so using this method.
Gavin Andresen
Hero Member
*****
qt
Offline Offline

Activity: 1330


Chief Scientist


View Profile WWW

Ignore
March 14, 2011, 08:43:08 PM
 #8

Focusing on the use cases is the right approach.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.
Any others?

One variation:

5. Export keys to transfer funds to somebody else.  Recipient is expected to import and then immediately send-to-self.  Sender keeps the keys 'set aside' in the wallet until they are spent by recipient, or, if not spent by recipient (lost in transit maybe, or recipient chokes on a lollipop before having a chance to redeem), may be added back into the sender's balance.

And one usability comment:

Is "selected keys" the right way to go?  Just selecting an amount of bitcoins to export/send seems like a better way to go (with a send-to-self transaction generated, if necessary, to create the right amount), with maybe an "advanced" option later that lets you select specific inputs to export/send.

Will I see you in Amsterdam?
  http://bitcoin2014.com/
BitterTea
Sr. Member
****
Offline Offline

Activity: 294



View Profile

Ignore
March 14, 2011, 08:46:20 PM
 #9

And one usability comment:

Is "selected keys" the right way to go?  Just selecting an amount of bitcoins to export/send seems like a better way to go (with a send-to-self transaction generated, if necessary, to create the right amount), with maybe an "advanced" option later that lets you select specific inputs to export/send.

Yes, this is a very good idea, and what I had in mind as a way to manage separate pools of encrypted keys. You could either let Bitcoin choose keys as it does now to create a transaction, and leave it at "around the amount you entered", or do as you say, with a send to self transaction to get the exact amount.
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 14, 2011, 09:15:45 PM
 #10

I would focus on use cases.

1. Back up all the keys in a wallet, and later restore the wallet state if the wallet is deleted or corrupted.

2. Merge two wallets by exporting all the keys in one, and importing to another.

3. Export and remove selected keys from wallet, for moving to safe long-term storage in case the wallet is hacked; later, move the keys back to a wallet for spending.

4. Export selected keys for transferring their funds to someone else; the recipient imports the keys and gains access to the funds.

Any others?

I wonder whether it makes sense to use bitkeys format for case 1 and 2. Wallets do have other things, which is part of your problem. I'd suggest that case 1 & 2 should use wallet formats.

I don't like the wallet as more than an internal database for one specific implementation of a bitcoin client. It's not transparent or human readable, hard to keep updated when accessing it using external tools, and maybe very different from what other implementations like to use. It is very fast and hard to corrupt, so it is perfect for an internal database, but for all other applications, i prefer a separate, more interchangeable format.

Simply by adding reserve keys (which are in fact harder to filter out when exporting, since they are available in mapKeys like change keys and used keys), you have everything to have a bitkeys file function as a backup or by limiting its contents, as an interchange format.

Finally, as far as the address and balance: I don't like putting them into CSV fields, because they are redundant and unnecessary. Your comment fields seem okay, but note that the balance may be out of date. How about a command that would take a bitkeys file, and display the address and current balance? Then you could leave the comments out.

I agree, I suggested exporting them as comments because they help making a bitkeys files also a useful way to inspect your wallet (which is maybe also a use case?), and possibly select which keys from it you want to give to someone else (essentially doing use case 3 or 4 by hand, by copy-pasting a part of a bitkeys file).

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 14, 2011, 11:11:50 PM
 #11

I'd suggest using a standard, existing format instead of inventing a new one.

I'm assuming this functionality is going to be exposed via JSON-RPC, at least at first. Why not have it return a JSON representation of the private keys?

I've though about it. It's a very simple way to avoid any parsing/escaping difficulty, while making it easy to extend it further afterwards. On the other hand, CSV is so simple that you don't even need any library for parsing.

Still, a proposal:

[
  { // moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
    "privkey" : "91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY",
    "firstblock" : 10806
  },
  { // mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
    "privkey" : "92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L",
    "firstblock" : 10870,
    "label" : "My Address"
  },
  { // midiVjwgBCkMSup4X2FifFinwLhpHYe2wn
    "privkey" : "91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV",
    "firstblock" : 10870
  },
  { // reserve
    "privkey" : "93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT",
    "firstblock" : 10870,
    "reserve" : true
  }
]


Does anyone know how JSON libraries cope with comments?


aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
jgarzik
Staff
Hero Member
*****
qt
Offline Offline

Activity: 1260


View Profile

Ignore
March 14, 2011, 11:36:49 PM
 #12

Still, a proposal:

[
  { // moura6ybHvwzMtrWdyqiEN6n2mMakf8HxD (50.01 BTC)
    "privkey" : "91iwnurxhWmDF9gqwc4CbuLCB4HxuMXf5B2vEhTPDftY3rkMgDY",
    "firstblock" : 10806
  },
  { // mmFPR1oaDExBfJqpRohWBvbE2aCg86Kwcx (0.00 BTC)
    "privkey" : "92JsNVMJgG4RiC2ARxpduJp8DXKgdDMj9WREE5jo66Hg8UMdr3L",
    "firstblock" : 10870,
    "label" : "My Address"
  },
  { // midiVjwgBCkMSup4X2FifFinwLhpHYe2wn
    "privkey" : "91s3swvX2a3F7FDoypEV7rTVhWUKiw8RCBmDuHMN8xMksLn3YtV",
    "firstblock" : 10870
  },
  { // reserve
    "privkey" : "93NM1B7y35d9VxqGBvc8DZcAvEB9kE5UZV7WqNGMTNWhPBdhsJT",
    "firstblock" : 10870,
    "reserve" : true
  }
]


Very nice.  If you have this JSON, you shouldn't need anything else IMO.


Quote
Does anyone know how JSON libraries cope with comments?

According to my research, the sad fact is comments are outside JSON scope Sad  You may add faux-entries like

Code:
   "_comment1" : "privkey is the private key associated with your request",
    "_comment2" : "firstblock is the first block where it appeared",

Ugly hack, I know.


Jeff Garzik, bitcoin core dev team and BitPay engineer; opinions are my own, not my employer.
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 20, 2011, 02:37:40 AM
 #13

Ok, I've since decided to use a CSV-like format anyway. It is more compact (for use in QR-encoded form, eg.), and easier to process using for example command-line tools like grep and cut.

Here is a preliminary branch that supports importing and exporting transactions: https://github.com/sipa/bitcoin/tree/walletdump

Working:
  • dumping a wallet to file
  • importing a file
  • watching for others doing transactions with your keys
  • rescanning the relevant portion of the block chain after importing
  • correct updating of transactions and balance in the GUI

Todo:
  • error-handling when a double spend has been tried anyway
  • importing of reserve keys
  • test

The file contains lines of the form:
Code:
<private key>,<block number>[,<flags[,label]] [# <comment>]
Current flags are:
  • R: reserve key
  • L: label is present
For example:

92263nMUE7SkR62tMn1Ucu2AxY5nANmQhXRhxHZP6MVxpfLC2EH,10801,L,"Your Address"
934xQoFPHzYkwTvJQwQSPw8QspwVB8kPd1S3wRZsZGbgLKVQPb7,10814
92xRg4At7vN25B1esc768NvW7Exrr2PcgZ78MFNmd58MzXjXGYM,10955,R



aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
March 21, 2011, 09:29:52 AM
 #14

That's great! Keep going! :-)

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
bitcoinex
Sr. Member
****
Offline Offline

Activity: 350


probiwon.com


View Profile WWW

Ignore
March 21, 2011, 09:40:08 AM
 #15

Keys have a standard PEM format...

+1

http://bitcointalk.org/index.php?topic=2507.msg34009#msg34009

Quote
so the format should be :
-----BEGIN EC PARAMETERS-----
BgUrgQQACg==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHQCAQEEII84GK/wX8stbmJWYL/WUM1nPAK1miIBDBeyNuo2vyf4oAcGBSuBBAAK
oUQDQgAE/yRZIKrOj4GBfLFtMYuocJ5QF1Tr9rWMh2ixCyfodDWRWTIU21v3ehDR
NJiAXHKwkhDqQ//i46NHoNUhjvx/lw==
-----END EC PRIVATE KEY-----

(I got this output from "openssl ecparam -out key.pem -name secp256k1 -genkey")

New bitcoin lottery: probiwon.com
- Может, ты ещё и в Невидимую Руку Рынка веруешь? - Зачем же веровать в то, что можно наблюдать непосредственно?
Mike Hearn
Hero Member
*****
expert
Offline Offline

Activity: 1232


View Profile

Ignore
March 21, 2011, 09:47:32 AM
 #16

PEM structures have a lot of data we don't need, and lack data that is useful like block numbers, labels etc.

12LMm82ZgAzf7yNDpPydEYxEr4Ap7XtSSK
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 21, 2011, 10:12:40 AM
 #17

PEM looks useful as well, especially when people want to use bitcoin's keys to sign other things. Shouldn't be too hard to implement that as well.

However, what I'm trying to do is create a self-contained, compact, human readable, stable form of storage for wallets. PEM files are standard but not-so-compact ways of storing purely the cryptographic part.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
bitcoinex
Sr. Member
****
Offline Offline

Activity: 350


probiwon.com


View Profile WWW

Ignore
March 21, 2011, 10:31:45 AM
 #18

PEM structures have a lot of data we don't need,

?

Quote
and lack data that is useful like block numbers, labels etc.

-----BEGIN EC PRIVATE KEY-----
Label: blahblahblah
Block-number: 12345

MHQCAQEEII84GK/wX8stbmJWYL/WUM1nPAK1miIBDBeyNuo2vyf4oAcGBSuBBAAK
oUQDQgAE/yRZIKrOj4GBfLFtMYuocJ5QF1Tr9rWMh2ixCyfodDWRWTIU21v3ehDR
NJiAXHKwkhDqQ//i46NHoNUhjvx/lw==
-----END EC PRIVATE KEY-----

New bitcoin lottery: probiwon.com
- Может, ты ещё и в Невидимую Руку Рынка веруешь? - Зачем же веровать в то, что можно наблюдать непосредственно?
Pieter Wuille
Hero Member
*****
qt
Offline Offline

Activity: 938


View Profile WWW

Ignore
March 21, 2011, 11:45:20 AM
 #19

PEM structures have a lot of data we don't need,
?

We only need the 256-bit private key, not curve parameters, generator point and public key, which are stored in the OpenSSL structures (and some of those in .pem files too, judging by their size) as well.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
Hal
VIP
Sr. Member
*
expert
Offline Offline

Activity: 314



View Profile

Ignore
March 21, 2011, 06:30:23 PM
 #20

Simply by adding reserve keys (which are in fact harder to filter out when exporting, since they are available in mapKeys like change keys and used keys), you have everything to have a bitkeys file function as a backup or by limiting its contents, as an interchange format.
There's other stuff in wallets, including preferences, address book entries for payees, and account information. bitkeys format won't be suitable for wallet backup if it loses all this.

Hal Finney
Pages: [1] 2  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!