Bitcoin Forum

Hi everybody,

after short discussion with someone42, we decided to start separate thread discussing wire protocol used for future hardware wallets. We would like to find some universal and flexible solution which can become a standard, to make easier life for programmers of bitcoin desktop clients.

My current proposal is to use generic class USB HID device, which can be used without installation of drivers across every major operating systems (especially MS Windows). There are some limitations, like 64 bytes of message payload and communication can be initiated only by the computer (polling).

I want to use HID protocol just like a transport for higher level protocol, instead of using some custom binary format. No, this time I'm not going to propose JSON-RPC :-), but I think Protocol Buffers is a good choice. There exists some lightweight implementations for microcontrollers, it is super-easy to use PB from every major programming language and it is well defined high level protocol (in comparsion to some custom-built protocol on top of HID messages).

There's my current draft of proto file: https://github.com/slush0/bitkeylib-python/blob/master/protobuf/bitkey.proto

I completely dropped the idea of storing custom private keys on the device, I plan to use the seed for BIP32 or Electrum algorithms on the device instead. It also makes the protocol a bit easier, because there's no need for messages handling with custom keys. I'll try to describe every message in my next post.

How to encode protobuf message into the stream?

Encoded protobuf message contains only message payload itself. There's no header or terminator; for this reason we need to define stream encoder/decoder for protobuf messages.

Every protobuf message will be encoded into the stream in following way:

a) First two characters are magic identifier "##" (0x23, 0x23).
b) 2B of message ID (encoded as big-endian unsigned short).
c) 4B of message length (encoded as big-endian unsigned int).
d) The of the payload is binary-encoded PB message.

There must be standardized mapping between message ID and protobuf message definition. For now I'm using following mapping: https://github.com/slush0/bitkeylib-python/blob/master/bitkeylib/mapping.py but I'm open to discussion on this topic.

Demonstration encoder/decoder in python is implemented here: https://github.com/slush0/bitkeylib-python/blob/master/bitkeylib/transport.py

How to encode protobuf stream into HID message:

1. Compose PB message stream as described above
2. Split string to 63-bytes long payload chunks
3. For every chunk create HID message in format: 1B = chunk size (0x01-0x3f), 0-63B PB payload

Why to encode chunk size into the message? Some USB HID controllers use higher values of first character (>0x3f) for custom commands like modifying re-programming vendor ID etc. Storing chunk size at 0th message character will make the protocol compatible with wide range of existing controllers.

Note that I'll start playing with some HID controllers next week (I'm waiting for samples) so maybe this will be the subject of change.

Standard message flow:

All communication is always initiated by the computer. Device is passive and responds with pre-defined sets of responses for every request.

1. Computer must start communication with "Initialize" message. This tells the device to restart it's current state and respond with "Features" message.
2. Computer can request device's UUID. This is unique binary string indentifying device's MCU (serial number), NOT device's USB controller. Although USB controllers can report vendor ID, product ID and so on, it is very likely that DYI hackers won't be able to modify USB controllers on their own (it requires some additional effort), so bitcoin client should use UUID to distinct between two tokens.
3. Although some responses are sent by device instantly (like GetUUID), most of them are blocking, because they requires manual confirmation (pressing the button) by the user. Bitcoin client should be aware of this and implement communication with the device in separate thread, to not block client's UI or other functionality.

Additional protection:

All important responses must be confirmed by pressing the button on the device by the user. Some users still want to use additional protection:

1. One Time Password - when enabled, device prints few characters on it's internal display and send "OtpRequest" response to the computer. User must retype the OTP from display to computer. Computer then sends "OtpAck" message. When correct, device sends the response of original request to the computer. OTP prevents user to pressing the button accidentally. By typing four characters to the keyboard, user confirms that he really want to perform this action.

2. PIN (password) protection - Although stealing of bitcoins from the hacked machine is impossible, attacker still have a physical access to the token. For this reason, device can be protected by the password. In this case, device responds with "PinRequest" to the computer and user must type correct password to the device's keyboard. Computer then sends "PinAck" message, containing the password. When corect, device send the response of original request to the computer. Although PIN isn't perfect protection (especially because you're typing the PIN to the computer's keyboard), with PIN-protected device, attacker must gain the physical access to the token AND have an access to hacked computer where user previously typed the password.

Device can combine OTP+PIN protection. The message flow will be following:

 The most paranoid setup will require pressing of physical button to confirm the action, then rewriting the OTP and writing down the PIN. As far as I can say, it is still pretty comfortable. Rewriting four or five alphanumeric characters is super easy and the password itself don't need to be super-long, as it is just additional security feature.

I'm don't mind which USB class device is used. I think the primary criteria is how easy it is to work with the most common combinations of programming languages and operating systems (which unfourtunately is a lot of combinations). The only other appropriate USB classes I can think of are HID and CDC (i.e. a serial port).

There are problems (you appear to be aware of them as well) with USB serial ports in Windows, notably the need to supply a .inf, and the bugginess of usbser.sys. I was going to work around these problems by instead emulating a FTDI USB-to-serial chip. Every major operating system has reliable drivers for the FTDI chips included, and the protocol seems quite simple (from looking at the Linux driver sources).

The advantages of implementing a serial port are:

• Easy to work with; often OSes allow you to treat them as a non-seekable file.
• It's possible to use common terminal programs to do debugging or firmware updating.

There's one major disadvantage: device discovery is hard. There's no easy, portable way to "know" what a serial port is connected to without sending something and waiting for the correct response.

I think USB HID is a good choice. Advantages:

• Allows you to choose a VID/PID (the FTDI emulation solution above requires you to use an FTDI VID/PID).
• Easy device discovery; just query the product descriptor string.

The only disadvantage I've come across in my research: apparently, in Linux, you need to detach a kernel driver in order to properly communicate with a HID class device. This can be done on most systems, but I worry that it will break on some Linux configurations. Please correct me if the situation is different now.

Currently I'm coding it for serial port as well, because I still don't have USB controllers from the vendor.


FTDI don't work in Windows out of the box and the installation is a bit weird. I spent some time to get my BFL Single working on Win7 :-/.

We want to use USB-to-serial for the Raspberry Pi shield (as RPi don't have usable USB port for HID device), too. But from the computer side I think there should be just one supported transport - USB HID.

I was going to come here and whinge about your choice of protocol buffers, but after a bit more research, I'm liking them more. Their encoding is quite lean; I was worried that it would be like XML. Also, there are tiny parser libraries (eg. nanopb) written in C out there.

So now I think it's a good idea to use protocol buffers. My only request is that, for all the non-signing messages, there is a maximum message size specified. A reasonable value would be something like 256 bytes. This allows buffers to be statically allocated (i.e. without the use of malloc). The limit cannot apply to signing messages, because transactions can be very large.

Want crazy talk? How about Bluetooth?

Applications...
-Transfer of files, contact details, calendar appointments, and reminders between devices with OBEX. (http://en.wikipedia.org/wiki/OBEX)
-Replacement of previous wired RS-232 serial communications in test equipment, GPS receivers, medical equipment, bar code scanners, and traffic control devices.

I was mistaken about FTDI drivers on Windows.

Looks like it is USB HID then.

No way because of battery...

Edit: However HID over Bluetooth is still possible, so even if we will standardize HID device, some future wallet devices can use Bluetooth and still have compatible protocol.

Detaching a kernel driver is just one function call from libusb or a similar library. Also some combinations of VID/PID (e.g the ones Silabs CP2110 use) are IIRC not interpreted in Linux kernel. So I don't think this is a big issue ...

Same here. I was a little bit sceptical when I saw the generated code from http://code.google.com/p/protobuf-c/ (pure C generator for Protocol Buffers, original implementation supports just C++), but I was quite happy when I saw the code that came out of nanopb - http://code.google.com/p/nanopb/

A few points and questions:

Timeout
You state:

Standard message flow:
3. Although some responses are sent by device instantly (like GetUUID), most of them are blocking, because they requires manual confirmation (pressing the button) by the user. Bitcoin client should be aware of this and implement communication with the device in separate thread, to not block client's UI or other functionality.

Is it worth adding in a timeout which is, from the computer's perspective, equivalent to a 'Cancel' operation ? i.e Device asks user to 'Confirm or Cancel'. User does nothing and after, say, 60 seconds computer says "Ok, that is a cancel" and moves to a more general "nothing happening state". It stops you getting stuck waiting forever for a button press that never comes.

PIN/ OTP exchanges
In 'Standard message flow, first paragraph' you mention that the computer initiates the conversation but with the PIN and OTP the device sends the PinRequest / OTPRequest. Do these occur as responses to the GetEntropy computer request only ?

i.e. valid exchanges are:

C: GetEntropy()
D: Entropy(entropy)

or

C: GetEntropy()
D: OtpRequest()
C: OtpAck(otp)
D: Entropy(entropy)

or

C: GetEntropy()
D: OtpRequest()
C: OtpAck(otp)
D: PinRequest()
C: PinAck(pin)
D: Entropy(entropy)

also, is it valid to have the PinRequest/PinAck before the OtpRequest/OtpAck ?

(just trying to pin down what are valid exchanges and which are invalid).

Checksum and Chunking/ Dechunking
Does the bytestream for USB have guaranteed integrity ?  IE is it necessary to have a checksum for each of the 64 byte message or is that taken care of in the transport protocol ? If required we could simply have 1 checksum byte = XOR(payload bytes) i.e.

1B = payload length
up to 62B = payload
1B = checksum defined as XOR(payload)

Also, you state that the PB message is chunked into 64 byte packets. When they are 'dechunked' how do you know the boundaries of the packets to stitch them back together to recreate the PB messages? Do you need "this protobuf message is chunked over X packets' at the beginning of the PB message.

I am just thinking of how to make the 'chunking' and 'dechunking' as simple and unambigous as possible. Ideally you want it a dumb transport layer that understands nothing of what is in the messages.

I think this should be invented layers first if it's intended to become a credible standard.

I would suggest starting with a message protocol that would be workable over a simple RS232-like serial line.  Start with the assumption that basic error detection and recovery will be needed, so the protocol isn't restricted to media that has it built in.  Then worry about adding abstractions to it like chunking.  USB-CDC and bluetooth can both abstract a serial line "out of the box".  If you invent a protocol that's centered around USB-HID, then nobody who wants to extend that protocol to work over TCP/IP or radio waves or named pipes or tin can telephones or light sensors vs flashing boxes on a screen or any of the other useful ways to move data is going to want to deal with implementing "USB-HID" contingencies where they aren't needed.

USB does have some error checking (one CRC16 per packet), so there's probably no need for us to implement it anywhere.

As for how to determine the packet boundaries, protocol buffers appear to lack a terminator. So length information must be out of band:
1. One option is to include the length of the packet somewhere. For example, the length (eg. as a packed, little-endian 32 bit integer) can be prepended to the protocol buffer.
2. Another option is to rely on the chunk lengths. When you see a chunk with less than 63 bytes of payload, you know it's the end. Note that if the packet length is a multiple of 63, to avoid confusion, an extra 0-payload packet needs to be sent as well.

Lastly, a bit of detail: how is the message type sent? A single byte prepended to the protocol buffer will do; but does anyone have a better solution?

Strictly speaking, device itself is just a state machine. I don't think defining some timeout is required. Computer can reset the device at any time by sending "Initialize" message, as I defined above.

From computer perspective, some "Ok, nothing happened" is completely implementation dependent and we don't need to specify this. Confirmation dialog can be opened for unlimited time and nothing wrong happen...


PinRequest and OtpRequest can basically appear as a response for any sensitive call. GetEntropy was just an one example.

I personally implement Pin/Otp handling in bitkeylib (python) independently to the call itself. When device respond with PinRequest or OtpRequest, library simply ask user for input.


Yes, all these combinations can happen. Although asking for Otp before Pin make sense, it makes guessing/bruteforcing of the pin almost impossible, because very attempt requires unique OTP...


USB HID is very simple protocol and yes, it guarantee the order and integrity. So no need for adding it manually.


You're right, I completely forgot to describe "magic character" and "message size" while encoding PB message into the stream. I'll complete the documentation above now.


That's exactly how I defined it. There are three layers:

1. Transport - it just transport bytes from one side to another. I want to have only one transport based on USB HID in the specs, but technically it is possible to use serial port or system sockets as well. As I said, I'm already testing the protocol using named pipes. Part of transport specification is (for example) that one byte defining message length (report id in term of HID).

2. Stream encoding/decoding. It contains magic character + PB message length, to make decoding of the stream as easy as possible. I'll complete the docs above.

3. Payload encoding/decoding. I proposed Protocol Buffers, which seems to be pretty good choice for our needs.

Whole messaging stack doesn't need to understand transferred messages themselves, so everything is very flexible.

Firstly, don't forget that we're not inventing new internet, but just specialized protocol for talking with quite a dumb device. Secondly, protocol itself (PB) isn't specific for some particular transport. And integrity checking should be a part of the transport itself, not the part of application protocol built on top of it.

Shortly said, it is possible to use the same protocol as described above over TCP without any modification.

I just added the chapter "How to encode protobuf message into the stream?"

Good stuff.  Keep up the good work!

I think there are USB simulators to be found, which would make testing this easier.

Your dumb device is actually a brilliant device and is probably the future of Bitcoin as we know it.  In the interest of having as few competing protocols as possible, I'm just submitting to you my request that you do this one right, because the next person to make a device intended to be interoperable with yours will probably make some improvement to it that none of us can conceive of today, or run it over some transport that seems crazy here and now but makes total sense somewhere else (e.g. analog radio links in Africa, or DTMF over antiquated voice lines, or printed barcodes on pieces of paper), and you want his job to be as unencumbered as possible.

Simple integrity checking at the application level is commonplace and best practice.  Bitcoin addresses have integrity checking in them, despite them most commonly being passed through TCP and other channels that also have error checking.  Application-level error checking isn't supposed to reinvent transport-level error checking at the application level, it's there just to throw out corrupted requests as a failsafe and substituting a low-risk action (e.g. doing nothing) instead of taking uncommanded actions with costly consequences when you're dealing with real money.  A simple 32-bit CRC and a practice of ignoring messages failing the CRC check would be a minimum that any hardware should have no problem handling without being an unreasonable burden on application-level code.

In a round-about way... please just don't make this specific to hardware supporting USB.

As far as Jim and Alan will be open to implement support for Pigeon transport, there's no limitation in protocol itself.

I would like to propose the convention that if the wallet sees a field in a protocol buffer that it does not recognise, if that field is 0, false or "" (depending on the type of the field), the wallet can safely ignore the field. On the other hand, if an unrecognised field contains something other than 0, false or "", a wallet must not ignore the field and must return a "feature not supported" failure message. This convention makes it easier to add additional fields in the future.

As it stands, nanopb currently doesn't allow this (it skips all fields it doesn't recognise). But I'm sure it's possible to modify the runtime decoder to follow this convention.

I see the approach slush has taken is to include a "Features" message which enumerates a wallet's features. I think a features list is somewhat open to interpretation. For example, what happens if a wallet only supports a subset of feature X? Does it report that it supports feature X or not? Another advantage of the ignore-on-zero convention is that client software doesn't have to put as many ifs all over the place - "feature not supported" is handled like any other wallet error.

I'm affraid that there's no safe solution how to handle forward compatibility by the wallet. I think that if there will be some incompatible messages in the future, clients should ask for wallet version (part of Features message) and then pick appropriate set of messages. This may turn into the mess when wallet developers won't be sane enough, but I'm also affraid of wallet which will try to recover from some messages which it doesn't fully understand...


Forking protocol buffer standard? No way, at least not for me :-).


I'm affraid of "partial support" of some features. Once you support OTP/PIN messages or not. I'm writing test suite which should cover 100 % of device use cases, so I believe this test suite will help developers to be sure all "features" are well implemented.


It should not report support for some feature which is not fully implemented. As I said, we're handling with valuable information and we should not try to recover from unexpected state by heuristic or some guessing.

I'm not aware of any standard saying that protobuf parsers must skip unrecognised fields. In fact, modifying a parser to fail on unrecognised fields would make things more secure: it would protect against client software accidentally using an unsupported feature and then misleadingly reporting success to a user.

I agree that supporting OTP and PIN is a yes or no thing. But sometimes it is not so clear, and we can't anticipate what advanced features might be added to wallets in the future.

The ignore-on-0 convention would not encourage "unexpected state", as long as 0/false/"" is always defined to mean that a feature is disabled. For example, say an "is_hidden" field is added to the LoadDevice message. Wallets which do not support hidden volumes can safely ignore is_hidden = false. If such a wallet ever gets a message with is_hidden = true, it fails with a "feature unsupported" failure message.

Such a convention is safely used in microcontrollers, where reserved bits in peripheral hardware registers are often supposed to be set to 0. Then, if features are added to those registers, the designers will take 0 to mean disabled and 1 to mean enabled.

Edit: Here's a compromise: if a wallet sees an unrecognised field, it always fails with "feature unsupported", no matter what the value of the field. All new fields must be optional. If client software doesn't want to use a new feature, it must not include the corresponding fields.

In my experience the response to the Ping message should contain a non-optional sequence number. This is particularly important in case of the device like yours that may require slight physical rotation to be able to read the LCD and/or may be jostled while pressing the button.

Theoretically the OS should issue USB disconnect/connect events in case of the brief interruption sensed on the USB port, Windows even plays a warning chime. But in practice this doesn't work 100% reliably and the USB device may get reset while being handled by the user to complete the required interaction.

There could be also an opposite situation where the USB device is plugged to the powered USB hub. The user interaction could cause hub disconnection, but the device's internal state would not be affected, despite the application receiving and handling the disconnect/connect events. If the application can detect that the device hadn't lost the internal state it may save the user some aggravation caused by a repeated initialization and retry.

Unrelated to the above, there is one more design decision that you may need to make.

It seems that as it stands right now the protocol may be training the user to blindly press the button without actually looking at the LCD screen.

In the medical device field there is an ongoing discussion: what is the correct level of interaction required to dismiss a dialog about potentially dangerous and irreversible action.

1) pressing always the same button to approve after hearing a beep or always another button to cancel and hear another acknowledgement tone;

2) pressing exactly one of 4-10 buttons after reading the message, and where the message tells which button to press and the number of the button needs to change with repeated presentation of the same message. Pressing any other button than indicated in the message is interpreted as cancellation.

In case of the Piglet the button on the device would be still the same, single one, but the user would be required to look at the screen and the screen would tell the user which key to press on the computer's keyboard.

The medical field apparently still cannot agree which way to go on this.

For the upgrade path of the protobuf format there are a few things we do in the bitcoinj wallet:
1) there is a wallet major and minor version. The major version gets updated on non backwards compatible changes. That way something like MultiBit can look at the version number and say: this is a wallet version 6 - I only know about wallets up to 5 so sorry I cannot load it.

2) also you can put in a new mandatory field to effectively jam a spanner in the machine. A protobuf parser cannot 'legally' load a mandatory field that it does not know about. You can use this to really really stop messages from the future being loaded.

3) in the bitcoinj wallet protobuf there is an extension message with a name/value pair. You can use it to add in additional information - it is designed so that, say, I want to put in some org.multibit fields I can add them in. Other clients will just read them and faithfully write them back (to avoid data loss) but not do anything with them.

Watching.

It should speak RS232. Almost all point of sale systems speak it and the usb bridges are ubiquitous.

I think what we try to establish in this thread is an application protocol. Physical transport is not that important (RS232, PS/2, USB HID, Infrared, ...)

Hard-encoding maximum message size into the protocol sounds problematic. It already requires hacks for message signing, so you still need some workaround in the code. Why to limit other messages then?

PB message can be decoded in smaller chunks, i.e. you don't need to allocate receiving buffer to fit whole message into it.

I just added "session_id" variable into Initialize/Features messages, so the computer can easily restart the device's internal state and check that all buffers have been cleared, because computer received the same session id back.

Handling buffers and delayed responses may be a bit tricky as I already see in my serial transport prototype. However session_id/ping messages are quite effective way how to detect possible issues on the wire.

I really don't want to interfere with your design, but my feeling is that a rarely-changing session_id will not be sufficient.

Fundamentally, you have a problem of synchronizing two state machines over a channel with erasures.

I'm not sure that the above problem has any simpler solution that the "sequence number" patterned after TCP over IP (or X-MODEM or Kermit protocol.)

The session_id would be then equivalent to seqence numbers starting with a value different than zero and different for each session.

Again, those are my feelings. I don't have a concrete proof, neither for nor against.

Edit: On the other hand I still remember the lesson you taught me in the Stratum thread: the protocol has to be conceptually simple enough to be reasonably implemented by the majority of the Bitcoin application programmers. If the price is an occasional, rare failure then it is worth to pay it.

Just a suggestion (I found default TI code quite bugged, especially under linux).
Since you want to use Stellaris Launchpad. The best way would be, ofc, to use a different chip for USB com which also be in charge to reset the arm chip by toggling its reset pin. E.g. if you use a USB<->SER converter, like one made by FTDI, you can use its RTS and DTR pins to perform reset.
If you want to have the ARM chip running wallet to do the USB device part too (which is bad for security imho), you can do something I tested already with TI code. In my case, the ARM chip runs a CDC, not HID, but it can easily converted to some HID command.
In usb_dev_serial, there is ControlHandler() which calls SetControlLineState() on USBD_CDC_EVENT_SET_CONTROL_LINE_STATE event (this happens when /dev/ttyACM is opened and closed).
In SetControlLineState() I added:
   
   
In main() I added:
   

After having a closer look at nanopb's API, I withdraw any request for limited sizes of messages or fields. It is indeed possible to decode smaller chunks of messages. There is even a mechanism to decode smaller chunks of individual fields.

I have been starting to think about what UI changes will be needed for MultiBit for BIP32 wallets.

One of the limitations of MultiBit/ bitcoinj is that currently you do not have quick access to the complete set of unspent outputs for an arbitary key (or BIP32 sequence of keys).

MultiBit has to 'replay the blocks' i.e. get the blocks starting with a far-enough-back-in-time block and set what transactions are relevant. It is very useful to have a date "slightly before the first transaction or the birthdate of the wallet". You can then set the blockchain head to then and get the blocks from a bitcoind. Bloom filters will speed this up but it is conceptually the same.

Anyhow, is it possible to know the birthdate of a wallet ? By this I mean the date when the wallet seed was created.
Is there a real-time network-synced clock on the device that could be remember this and expose it? Or the host computer could give the device the current time perhaps.

Otherwise I will have to ask the user for the creation date of the wallet (which of course they will not know accurately) or get it from another channel other than bitcoind (possible but then I am leaking pretty important information to some server like blockchain.info or an electrum server and I am no longer strictly P2P).

Interesting ideas, Jim. Technically this is not a problem, however thanks to LoadDevice() method, device can contain seed used before the device was created. For example when user lost the device and will try to reload seed to new one. As this is not 100% bulletproof, we should look at another solution...


Looks like this is issue not directly related to Multibit, but to chainless clients generally. Maybe we should encode something into the mnemonic format itself? Full timestamp isn't possible because it requires too much bits, but maybe some lighter solution?

Edit: After quick brainstorming with Stick I realized that this is stupid idea, for many various reasons. So I think that asking user for some date is the best and the most transparent solution so far.

Yes I think I will have to ask the user or get it from a non P2P server.

I presume the device does not even have a battery and gets it's power from the USB so it wouldn't have a real time clock.

Correct.

Generally speaking, yes, you should store the key birthday (or seed birthday) somewhere.  When [re]scanning, this makes a huge difference in startup and network-sync times.

While it might make a huge difference now (because seeded private keys are relatively new concept), it won't make a big difference in a future. I agree it might help to avoid unnecessary rescanning, but relying on that feature is flawed ...

Can you elaborate?

All the major client developers agree this is a good idea.  The benefits into the future increase, as the block chain gets larger.  It makes an even bigger difference in the future.

It has significant advantage as your wallet has been created recently. You created wallet around block 200k, so you need to synchronize only around 10k of blocks (20x speedup). However in few years of usage, on block 1M, you'll need to synchronize 800k of blocks. Then the advantage of timestamp is almost neglible, because the speedup will be only 0.2x (probably a lot less, as blocks are bigger and bigger).

All this is predicted on an invalid assumption:  the common case will be older, existing bitcoin users carrying around one big wallet.

First, wallet creation date is much different than individual key creation dates.

Keys are constantly created, therefore a portion of the wallet state is always created recently.  As blocks get bigger and bigger, key birthday reduces the number of keys one must scan in each block, especially if key retirement is also supported.

Long term you will have a rolling window of keys, no matter when the wallet is created.  Key birthdays will always be relevant and useful, for this reason.

Furthermore, there will always be new users creating new wallets, and users with multiple wallets.

...except the device doesn't with the keys, but with the seed (BIP32/Electrum). Device don't store private keys or any metadata for particular addresses.

However such metadata can be stored on the computer, using master public key as a wallet identifier.

Edit: Device can teoretically store timestamp of the seed generation, as some quick-and-dirty workaround. However this still must be provided by the computer, which will probably just ask the user.

Birthday is of continuing relevance to:  new users, new wallets.

I've been battling this concept in Armory with its deterministic wallets, for a while.  I implemented "blockCreated" and "timeCreated" variables for each address, but I have been hesitant to actually use them for anything.  Because, I determined that getting it wrong was not worth the performance benefit of when it is right:  if you trust the timestamps are created and stored correctly, but it turns out that something was wrong (incorrect clock when it was stored), the user could be missing a big chunk of their wallet and it could be a mess trying to figure out what happened (during recovery of a deterministic wallet, that is).   Plus, even if it is right, unless the user knows exactly how much they had they don't know that it's right unless you do the full scan anyway.  At least for right now, doing the full scan isn't that much more cumbersome than a partial scan, so it's easier to just do it.

I don't know how the HW device will be dealing with addresses (or what is possible memory-wise), but I have found it extremely useful to store the keys even though they are derived from the root.  Not only does it allow me to store birthdays and comments/labels, but (most importantly) I can search for an address and know if it's in my wallet without re-doing all the computation.  If it's possible to at least maintain addr160 list in non-volatile memory, it will make everything quite a bit easier.

I am hesitant to "endorse" requiring key-indexes to be included in the TxDPs (as I call them in BIP 10, perhaps another phrase+acronym would be better).  The reason is to do with multi-sig, which makes sense to accommodate here -- since there's like 90% overlap between what's needed for offline wallets and multi-sig.   There will be situations where the TxDP will be created by another party who doesn't have your watching-only wallet, they are only supplied a public key to include in the transaction.  They don't know how to specify what your address index is, and your offline device will have no way to detect what public keys belong to you.  I guess you could have an optional field so that your online computer could "amend" the TxDP with your address index, but that sounds complicated.

I know it sounds premature to think about multi-sig for your HW device, but I don't think it makes sense to over-complicate that use case if it can be avoided.  If we must accommodate devices that can't even store their own address list, then we have to deal with it, but it's worth trying to avoid.  And in the end, even with 5000 addresses, that's only about 100 kB to store a raw addr160 list...

The clock might be wrong... but it's not gonna days or weeks wrong.

You have to include variability anyway, because blocks can vary.  So given birthday X, start scanning at X minus 2 days, or X minus 1 week.

That's wrong assumption. If the battery on your motherboard is empty the date would be 1.1.1970. I can come up with some other scenarios where the date/time is seriously incorrect.

You're stretching.  Anything is "possible" but using a 0.01% case to justify giving new users a bad experience is not reasonable.

I'm not sure.  The "cost of misclassification" is not symmetric.  It takes 5 minutes to do it right 99.9% of the time, but it takes 7 min to do it right 100.0% of the time.  And given the heartache associated when it's done wrong, why not just take the 7 minutes?  At least, that's been my attitude, even if you don't agree with the trade-off. 

Hardcore Americanism detected!  ;D

The month-day-year versus day-month-year ambiguity with dates is an endless source of the errors that stick is thinking about. People in USA don't see is as often because most of the BIOSes and other low-level tools use the American layout. But it trips over many Europeans.

Wait till Americans start importing Czech-made wallets that use day-month-year order...

I personally hate the American system.  It doesn't make any sense.  Though, I'm not sure how relevant it is here:  UTC unix time is the "path of least resistance" for storing timestamps in binary files, and thus wouldn't have that problem.

Part of the reason I avoided this is because there's a lot that can go wrong with offline computers.  The clocks could be totally wrong with no access to the internet, and thus they don't know the block height at birth, either.  I could insert special cases where I know one or the other based on what information is available at the time it was recorded in the log file.  But then I have to treat all conditions of {have/donthave} {block/time}.  And as I said... I think getting it wrong and silently missing parts of the user's wallet could be a disaster -- those could literally be lost forever, because no program that ever reads the file would check before the recorded time.

The end result is that users would just want to scan everything anyway, just to be sure. 

The european system is just as stupid, just incompatibly so.  ISO8601 puts magnitude in the proper order on all scales, and there hasn't been a valid excuse for using anything else in a long, long time (when you need actual dates, not seconds-from-epoch).

If you don't know the proper birthdate of a key or wallet, set it to zero so you get a full rescan.  Key birthdates is something I wish for about twice a week.  I do a lot of private key importing because I work with offline key generation.  The raw transaction API lets me bypass a lot of that, but not all.

If you don't think that key birthdays are important, just give it a few more years.  If bitcoin is going to take off, transaction volume (and thus scan complexity) is going to grow much faster than Moore's law, and the rescan that you think is just fine now will be downright painful soon.

P.S.  NTP has been around since 1985, if someone doesn't know if their clock is right or not, they deserve what they get.

I'm fully with you on the use of ISO format for dates, but I disagree on the NTP for autoconfiguration.

NTP abuse is so rampant that it now has its own notable Wikipedia article:

http://en.wikipedia.org/wiki/NTP_server_misuse_and_abuse

And thats just a tip of iceberg. In many locales servers like "time.windows.com" are unreachable because of the rampant abuse by the various software licensing/subscription enforcement schemes. McAfee private-label products were implicated in DDoS-ing Swiss government NTP servers, if I recall correctly. And apparently the situation is slowly getting worse, not better.

Yeah, I know.  On the other hand, DHCP can provide local NTP servers.

Timekeeping is one of the things that really pisses me off.  There is absolutely no excuse for any network operator to fail to provide and advertise local NTP servers.  GPS receivers with PPS interfaces have been dirt cheap and quite common for 10 or 15 years now, and ntpd supports them all.  Doing it right is well within the technical and financial means of any network provider.  Hell, within the means of most college students.

And device makers suck.  It isn't hard at all to implement NTP properly, but no one seems to until they end up on slashdot.  And times are changing; even the dinosaurs that have kept PBX systems in the dark ages have started to catch on.

Implementing NTP directly in the device is without a chance, it is completely out of the focus.

However there can be some API for storing/retrieving timestamp to/from the device. But it will be just a dumb storage of the value and it will be up to the computer's software if it will use it or not. I cannot imagine anything useful what can be done with the timestamp by the device.

You're right that timestamps will be probably more and more in the future as people typically use recent wallets. Except that the device is designed more like a cold storage, so that timestamp optimization won't improve user experience for most of the users. Still, if storing timestamp of the seed generation will be useful, we can include it.

Yeah, we don't need to get too sidetracked on that topic.  I was just stating that even if you could store timestamps with each address, I'm not sure you're getting too much value out of it.

However, let me get back to the previous topic I brought up -- what's the chance you could store a raw addr160 list on the device?  Here's what I said before:


If the hardware device can only store the root, then it's going require either:  (1) a computer holding the watching-only wallet for the offline device to create the TxDP to be signed, or (2) When you give someone your public key for a multi-sig tx you gotta supply an address index with it somehow so they can include it, or (3) The TxDP must pass through the computer holding the watching-only wallet so that it can be amended with the correct address index. 

Why we should store addresses on the device, when the address can be generated on the request from the seed?


Generate such list of addresses is a matter of second for the desktop machine. Using the device's memory for storing the list will limit the usability. For example we're using platform with 256kB memory for both application and data. Thanks to memory fragmentation, only limited part of that 256kB can be used for data storage.


As far as I understand the BIP10, it is out of the scope for such device. Device requires very minimal interface and it needs prepared data by the computer. For example it requires streaming of inputs in separate messages to allow signing of huge transactions. However desktop software using the device for signatures can be BIP10 compatible, of course.


Public keys can be derived from master public keys stored in offline wallet, so yes, desktop software can compute indexes just from public keys.


Yes, this is expected. Software can download master public key from the device, store it offline and then handle it as an offline wallet, generate addresses or so.

I think we diverged here.  I was kind of under the impression that we were coming up with a BIP 10+ that was more versatile (like, having a compact binary representation) so it could also be implemented in your device.   I'd really like to see this "protocol" standardized across devices and applicaitons -- which was the original goal of BIP 10, but I was the only one using it, so it never really got the versatility of a real "standard"... I stopped developing it when it worked for Armory, knowing it would be expanded later.

So, I was hoping that your device would simply implement the same BIP 10+, and everyone is happy.  I'm looking for the possibility that someone who creates a TxDP on their desktop, and requiring 3 other signatures, some of which may come from regular desktops, offline computers and/or your devices, that there won't need to be any special conversion for your device to be able to understand it, recognize its own keys, and sign it (after user confirmation).

With that in mind, I think someone42 mentioned the device won't hold its own address list, and thus this protocol requires telling the device which keys belong to it and what address index those keys are in the wallet.   I really wanted to avoid that, if possible, since it complicate other use cases.  It sounded like you said the device could regenerate all of its addresses/public keys from the root ... but given the other limitations of the device, I would think that calculating 200 public keys from a root private key every time would take a while.  And once you've used it a lot, 200 may not be enough.  If this is an unavoidable implementation, then we may be required to split the use cases into different "standards", but I hope you recognize the value of aiming for a single standard.

tl;dr  The protocol can easily be unified with all other devices/applications (i.e. BIP 10+) if you can have the device recognize its own addresses/pubkeys.  I was hoping for that unification, but it may not be achievable/efficient to do so.

The protocol which I proposed is built on top of Protocol buffers. Having a separate parser for another format just to comply with BIP10 doesn't look smart enough.


The conversion from BIP10 to something built on top of PB can be quite straightforward, so even BIP10-compatible software might convert the format for the device easily. To be honest, I didn't study internals of the BIP10 protocol yet, I just watched the protocol format.


You still need some desktop software to work with the device. As far as Armory or Electrum will support this device and Armory or Electrum implements BIP10, there's really no need why device itself must understand BIP10. Or any simple desktop (command line) tool can convert BIP10 format and pass the transaction into the device over USB.


It will be computed by the computer, not by the device. I don't see problem here.


This protocol is low-level layer for communication between the device and desktop software. This protocol must be optimized for low-resource devices and must be really minimal. There's no need why this should use BIP10 at all.

So the real divergence here was that I assumed the device would be somewhat "independent."  It sounds like, instead, your device will be wholly dependent on a host computer to do most of the work for it, and the device will literally only sign.  There will always be a computer-in-the-loop with your watching-only wallet to reformat the thing-to-be-signed into this low-level format for the device.   I had anticipated the device take, from any computer anywhere, a BIP10-like data stream, recognize it's own public keys/addresses, and get user confirmation to sign it.

Okay, we're on the same page now.  My point is valid, that a BIP10+ could potentially serve this purpose if we added extra complexity to accommodate devices that don't even know their own address pool, but it's not necessary.

So, if you have some enormous transaction to sign, let's say with 100 inputs and the transaction data was 100 kB, can you at least hold 100 OutPoint-Value pairs in RAM?  i.e. you feed in first supporting transaction and output index (because your transaction will be spending that output from that tx) and the device will hash it and confirm that the hash and then store the hash and output value in RAM.  Then repeat for all supporting transactions to collect a list of "verified" outpoint-value pairs.  Then when you feed in the transaction to be signed, it can go through it's local memory to confirm each outpoint that belongs to you to be signed.

Then, it sounds like, if that transaction is actually 100 kB, then you can't even hold the whole thing at once.  If you have to sign 30 inputs, you would pass it in 30 times, each time with a different txin-script-in-txout-script-place-and-all-others-blanked.  The thing can accumulate the hash as it goes, and then pass out the signatures one at a time when it's done reading the final transaction.

Is that too crazy?  Or should the device be limited to handling transactions that will fit nicely into its cache?  It could be a requirement of the software that interfaces it (Armory, Electrum, etc), that it will prevent the user from sending tx bigger than X kB...?  However, even if this tx is within the size limit, there's no guarantee that any of your supporting transactions is within the size limit.

Yes, exactly. You cannot expect full bitcoin implementation from the device with 256kB flash, 32kB RAM and a passive USB HID interface ;-). The purpose of the device is:
* Do not leak seed/private keys to the computer
* Confirmation of tx signing by the user, by displaying the transaction on the display and waiting for button press.

...and nothing more.


Exactly!


I designed something similar for input transactions, I call it "input streaming". In this way device don't need to store all inputs in the memory so the device can sign transaction with unlimited counts of inputs.

Although very high count of inputs may happen naturally (like when people are consolidating tiny pool payouts), having so many output transactions isn't so common and usually such complex transaction can be split into more smaller txes. However I over-looked the case of multisig transactions, where having very high number of outputs may be required as well. I'll try to propose some solution for this case too. Thank you for pointing to the problem...


Not at all. I'd like to have a protocol which allow signing of data streams, without any limit to input and output transactions. As I described above, I already have a solution for input streaming, now I just need to think about output streaming. But this can be definitely solved...

By the way:  amusingly-relevant article on slashdot today about clocks getting off by 12 years...

http://news.slashdot.org/story/12/11/20/216234/ntp-glitch-reverts-clocks-back-to-2000

Even without multi-sig, some people include a large number of outputs.   Many times, because it's inconvenient to actually execute an offline transaction, they may pool together many different outgoing transactions into a single transaction.  I know Armory users do this (I do it myself, sometimes), but usually because my offline computer needs to be booted in order to execute the tx and I'm just lazy :)  But for this reason, you probably need the device to be able to scroll so that it can display an arbitrary number of outputs.   I even have it on my own TODO list, to put Armory's confirmation list onto a scroll area to accommodate the rare times users specify two dozen outputs... (but I haven't had anyone complain yet, so I guess it's not common to do that many).

Something else that you need to accommodate:  since the device can't recognize its own addresses, it can't recognize its own change address.  Therefore, your protocol needs some way to tell the device which output is change -- perhaps specify the address index so that it can check that the marked change address truly is in your wallet.  Without it, the device doesn't know which is which, and even if the user understands how transactions and change work, a compromised desktop computer could swap the change address and the user wouldn't realize it's malicious. 

You can do this if you re-order the fields* in the protocol buffer messages to match the order of the Bitcoin protocol (inputs before outputs, previous output reference before input script etc.). Then, you send the entire transaction through once per required signature. Each time an output is encountered, you prompt the user about it. This is merely an extension of your "streaming" idea, but applied to the whole transaction.

You can probably see the problem with this: the user will be prompted repeatedly, for each required signature. This makes for an unnecessarily bad user experience. This is how I intend to deal with the problem in my implementation:
1. While processing the transaction, a hash for signing needs to be calculated. In addition to this "signing hash", calculate an "input invariant hash" which is just like the signing hash, except it ignores all input scripts.
2. The input invariant hash will be the same regardless of which input is being signed (that's why it's called the "input invariant hash").
3. If the user fully approves a transaction, save that transaction's input invariant hash.
4. Include a field in the SignTx message which specifies whether the required signature is for a repeat of the previous transaction.
5. If the host specifies that the transaction is repeated, do not prompt the user.
6. In order to stop the host from lying about the status of the repeated transaction, compare the input invariant hash of the current repeated transaction with the saved input invariant hash, returning a Failure message if there is a mismatch.

The host signs a series of inputs like this:

*This brings me to something which will probably be a non-issue, but we should be aware of: (according to https://developers.google.com/protocol-buffers/docs/encoding (https://developers.google.com/protocol-buffers/docs/encoding)) field ordering in a protocol buffer message is not guaranteed. Every serialiser will write out fields in field number order, but certain operations may not preserve this ordering.

If it is own address, desktop software should indicate this by filling "address_n" field in the message structure. Device can recalculate the address from the seed and this vector, so malicious machine cannot steal coins in this way.

Yes, similar idea here.


Yes, I had this idea yesterday as well, I've been just lazy do write it down O:-).

Etotheipi, someone42, I'd like to know your opinion how to handle change addresses. There are basically two approaches:

a) When transaction is coming to self-address (validity can be verified by the device itself), hide output to change address from the user.
b) Show output to change address as normal output, it can be just displayed in slightly different way on the display.

ad a)
This is how bitcoin software is doing it now, although I don't like it much. User should see full information, otherwise there are some possible attacks. For example, by hiding change address from the UI, malicious (modified) software can send coins literally to /dev/null, by using some very high address vector for change address. Attacker don't take the money, but they're practically lost by the original owner.

ad b)
I see this approach as much safer. User see all outputs and it's only up to the desktop software to explain what's going up here. Change addresses are for example fully visible in Electrum and I don't think that people have any problem with it. Device still must support some kind of scrolling for multiple transaction outputs, so having one more output is not a problem.

It really depends on the target audience.  You and I are intimately aware of how Bitcoin works, under-the-hood, but non-techies think of their own wallet as a single number -- their balance.  They don't understand what "unspent outpoints" are, and why there is change, etc.  To them, they send 20 BTC to one address, they expect to see 20 BTC go to one address and their balance reduced by that much.  Forcing them to understand more than that is... a judgment call.

I've gotten lots of questions about "change addresses", and why it looks like users are sending coins to multiple people.  I tried to make it clear in the Armory tx display, but they don't always catch on.  For that reason, the "Standard" usermode in Armory tries to avoid making any mention of "change," for fear of confusing the user.

On the other hand, your device is not for absolute newbies (although many would argue that Armory isn't, either :)).  I'm sure most of your users would understand what change is.  But I'm not sure displaying the change would make a difference -- even if they see it and understand it, they aren't going to know for sure that the displayed address is part of their own wallet.  The device verifying its own address is the roadblock to a malicious attack, not the user confirming it.

In fact, what will happen if the first 100 tx, the user sees the change output and confirms it.  On tx 101, the second addr pops up but isn't actually change (because it was maliciously replaced).  Is the user going to notice this is different?  Or just hit the "confirm" button twice like he always does because he's always confirming two outputs? 

I'm trying to avoid some "extended usermodes", which just makes the flow more complicated and less universal.

Personally I'm not a fan of change addresses, but I understand that there's some demand for this feature. Change addresses are false promise of anonymity, because addresses in one wallet are still usually tainted enough that it's obvious they have common owner.

Now I have another idea - hide just transaction outputs which send coins to some input address which is mine. For wallets not using change addresses (like Elecrum has such option), you won't see "change" output. In this case users still can imagine wallet as you described - as an amount stored on some virtual account.

There's not enough space on the internal display to explain what change address is. If desktop software wants to use change addresses, then the change address (which is mine, but not in transaction input) can be visually marked (like inverted colors on the line or something) and it is up to the desktop software to explain its meaning to users.

I'm not so sure that displaying the change address deals with the "malicious send to /dev/null" attack described above. For example, say the device displays "change of 0.05581992 BTC was sent to 1HoYQQeA3BxWisDK3VtGFNbvnNPcoZTE5n". How do you even know whether that's a "genuine" change address?

Perhaps a better solution is to display the change address index instead. There is an immediately apparent difference between "change was sent to address 18" and "change was sent to address 1156397331". But will users understand this?

Another option is to require change addresses to be explicitly and sequentially generated by the wallet. However, this does require an internal counter that is incremented every time a change address is generated. The malicious change address attack is prevented because the wallet is presumably unable to generate millions of change addresses in any reasonable time.

Yes, that's why I don't like the concept of change addresses. You must trust the bitcoin software that it generates some reasonable address for you, instead of just checking if one output address is going to the address you want and second output address is going back to one of original addresses (which can be even checked by the device).


No :-(. I've been thinking about this too, but I'm affraid that "address index" is out of game for end users.


Internal counter don't solve anything, because you can use the same seed on another device. This creates completely different set of problems.

I really don't think you're getting any value out of this.  The device can internally confirm that the change address belongs to your own wallet, and asking for user confirmation doesn't add any security -- but it does create confusion.  Even if users know what they're doing, they may accidentally confirm through a malicious change address because they are always asked to confirm two addresses when they send a tx with one target. 

I'd prefer, instead, that the second address only comes up when it is not part of their wallet.  If the user created the tx to go to 2 people, they'll expect to confirm twice, etc.  If they are asked for 3 confirmations, they'll immediately recognize something isn't right.  

Is there really any benefit of using newly created change addresses? Except that this is default in Satoshi's client. This is the main question, because the risk is exactly in using freshly generated address which validity cannot be confirmed by the device or by the user...

If there's no clear benefit in using new change addresses, I'm inclining to display them to confirm as any other addresses.

There's the non-zero benefit of:  ECDSA is secure right now... but if it wasn't, using new addresses still keeps your coins secure because your private key can't be broken if your public key isn't even known.  i.e. -- if you send back to an address that has its public key already in the blockchain, then in the far future if ECDSA is broken and/or quantum computers start becoming a reality, then you are vulnerable.

And of course -- the privacy aspect is improved. As you said, privacy is not guaranteed when using new addresses, but it's also not as bad as just reusing old ones.

If this become a problem, then it is only the matter of software update to start using newly generated addresses. For now this is non-issue.


Well, I accept that. Still we need to balance between security/usability/anonymity and I see using artifically created new addresses as a quite big risk. There's still enough time to think about it a bit more...

I don't understand the risk.  The desktop computer is perfectly capable of creating a new address in the same deterministic wallet, and telling the device what the address index is so that it can recognize it.  The user doesn't even have to know it's there, as long as the device is doing its job to verify ownership of the address. 

(1) If the device doesn't recognize the second address, the user must confirm the extra output, which will look suspicious.
(2) If the device does recognize it, then the user only confirms the original output they were expecting to see

And what if computer ask for using address with index [2^32, 2^32, 2^32, 2^32] ? Don't forget that BIP32 is hierarchical. Good luck recovering coins send somewhere to this address space...

Oh, I missed that part.  I forgot that BIP 32 doesn't require computing addresses in order (Armory current wallets do require computing in the chain, but that has turned out to be more of a curse than a blessing, but would be useful here).  But I see your point, now.   And the suggestion about storing an index on the device and or displaying it for the user, makes more sense, now.

I'll think about that one...

Well here's an idea:  can the device at least store how many times it's ever created signatures?  Or something else that is at >=1 for each transaction it ever deals with?

Here's what I'm getting at:  BIP 32 allows for random access to any address, but it also specifies using a separate subchain for change addresses (the "internal" subchain).  This may have been overlooked by you early on for simplicity reasons, but it is part of the spec.   The idea is that change addresses will only ever come from this the internal chain, and that chain will never receive coins from any other source except the "external" chain (the main chain that is used for distributing payment addresses). 

The relevance is that the "internal" chain is very dense.  There will not be any gaps in the internal chain.  In fact, the highest index ever used in this chain should be about equal to the number of transactions ever signed by the device.   Therefore, if you implement both roots on the device, the device can safely declare that "a change address with an index higher than the number of transactions I've ever signed is invalid." 

Yes, the device can store some simple counter, but I'm affraid that the overal design is becoming a bit fragile.

Technically you can have many devices or desktop wallets working on the same seed, but the device don't have independent access to the full blockchain, so it depends on information provided by the host computer.

Now we must decide what should device do if computer propose change address above current counter state...

After quick brainstorming with Stick we've composed following interaction diagram of transaction signing: signtx_workflow.pdf (https://github.com/slush0/bitkey-python/raw/master/docs/signtx_workflow.pdf)

The purpose of this is to stream everything, to minimize RAM requirements for the device.

Basic description is:
a) Device receives SignTx message, which contains mainly count of inputs and outputs.
b) Device asks for inputs and outputs in separate messages.
c) Every output is confirmed by the user.
d) In first iteration, device computes hash of outputs and blank inputs (without scripts).
e) After first iteration, signature of first input is generated and returned back to computer.
f) In every next iteration, device compare hashed outputs with the hash built during first iteration (which has been confirmed by the user).
g) Every iteration produces exactly one input signature.

I focused to change addresses in BIP32 proposal and everything I see is that (simplified=) "it is possible to use internal chain for change addresses", however it's not a requirement, so device also should not enforce it. When user wants to use multiple chains for multiple offices to keep their balances separate, he'll probably want to use multiple internal chains for change addresses as well...

My point was not that you should use it, for fun, but because using it might make the "what if offline tx uses change address 1e7?" problem a bit easier.  The online computer could hand out millions of addresses on the external chain, but the number of addresses used on the internal chain will be strictly limited.

However, if the private keys are distributed across multiple devices, it's not quite so simple...

Seems like we should start really simple targeted for the smallest USB device, like the cypress CY7C64316. 

Define a protocol that could sign a bunch of outputs; and not even keep track of the act balance.

Base it off of a "transport" that consists of 2 APIs:

send(buffer, length)
recv(buffer,length)

because any underlying transport can implement these 2.


Then additional protocol functionality can be layered on top (to support more advanced HW wallets)... and additional transports can be layered below.

In a related note, is there any web service that will just post an arbitrary bitcoin TXN to the network based on some RESTful probably binary-coded API?

Here are some of my findings regarding USB HID-based communication. Perhaps they will be useful to anyone else working on interfacing with hardware wallets.

• The maximum throughput I've seen in either direction when using HIDAPI is 31500 bytes/s. This is fast enough to transfer most Bitcoin transactions instantly (from a user's perspective), but it is less than 3% of full-speed USB maximum throughput.
• There needs to be a way to identify hardware wallets. HIDAPI's hid_enumerate() function gives you a list of HID devices, but it includes things like keyboards and mice. USB device properties which HIDAPI can query include: vendor ID, product ID, manufacturer string, product string and serial number. Identification via. vendor/product ID is probably not a good idea because they're constrained by other factors.
• The latest version of HIDAPI available on https://github.com/signal11/hidapi/downloads (v0.7.0) is quite outdated. It has issues on Windows relating to report sizes (see https://github.com/signal11/hidapi/pull/28).
• For the CP2110-based protocol that slush has proposed, a long, repetitive HID report descriptor is required.
• On the device side, things are complicated by the fact that the HID specification allows reports to be sent/received from both the control endpoint and an interrupt endpoint. Both these methods have their own subtleties.

Some Windows XP-specific things:

• Windows will refuse to enumerate your HID device unless it provides a valid report descriptor. Windows will refuse to process reports unless that report is correctly described in the report descriptor. This is why that "long, repetitive HID report descriptor" is not just a good idea - it seems to be required for the interface to work at all.
• Upon success, HIDAPI's hid_read() always returns the size of the largest report, even if a smaller report was received. So its return value should be ignored and the report ID (first byte) should be used to determine the report size.
• Windows will eat up received reports even if hid_read() isn't being called.

Some Linux-specific things:

• HIDAPI provides two backends: one based on the hidraw kernel driver, and one based on libusb-1.0. Certain kernel versions have some hidraw bugs.
• No matter what backend is used, access to a USB HID device requires root permissions. Alternatively, udev rules can be used to whitelist certain devices. Care must be taken to write the whitelist filter so that it includes all hardware wallets.
• By default, when a USB HID device is plugged in, the hidraw driver will use "get report" control requests, presumably to initialise its view of the state of the device. This can end up eating up the first byte sent from the hardware wallet.