I'll try and answer as I've imagined it playing out - these are all just proposals. I have enough things on my plate with BitCoinJ and more to keep me busy for a while.
My plan was to do some changes to the official codebase to support serving
lightweight clients but not to actually make it be lightweight itself. The main reason is I don't have enough confidence to do serious surgery on the code without any unit tests to catch errors. Others who have made changes more often (like Gavin) might be able to do a better job. Satoshi sent us a partially complete patch before he left, I didn't look at how complete it is but I think it only implements the initial first-launch download optimization, not everything.
OK, with that, here goes. Firstly some terminology. I'm going to call these things SPV clients for "simplified payment verification". Headers-only is kind of a mouthful and "lightweight client" is too vague, as there are several other designs that could be described as lightweight like RPC frontend and Stefans WebCoin API approach.
What are the engineering assumptions we will make, WRT headers-only clients?
Minimum hardware goal: smartphone (of two years ago) or better class of devices, all the way up to desktops. SPV for servers/merchant nodes might be interesting in a few years but for now, the additional security is probably worth running a full node. Non-goal: smartcards, J2ME candybar phones etc.
What that means concretely, memory usage <16mb total, startup time <500msec.
Protocol: a slightly extended (backwards compatible) of the existing P2P protocol. Connecting to newer nodes means a more optimal experience but is otherwise not necessary.
Is the merkle tree or any other data besides block header required?
Currently SPV clients need to download full blocks from the moment their wallet is created (first key). The reason is that's the only way to discover new transactions. getheaders can be used as an optimization for new users. It's not even required to download the full chain. You can start from the last checkpoint and not go all the way back to the genesis block.
If just the block header, how will a client verify a received TX? Surely at least the block's merkle tree is needed?
Sort of. Merkle tree
doesn't help you because you don't know which nodes in the tree are yours. Merkle branch on the other hand ...
To avoid the need to download full blocks, a simple pattern matching command can be added to the protocol. Strawman proposal, a new txmatch command is added. The contents are:
- a request id
- a range of block numbers to search between
- a regularly formatted transaction, except that the scriptSigs and scriptPubKeys can contain pattern matching opcodes as used in the solver today. Other fields are ignored. The SIGHASH flags might be useful for controlling how the inputs/outputs are matched - need to think about it more.
The full node then loads each block in the range and checks each transaction for a match. This could be quite intensive so some pushback message might be necessary. If the node is overloaded, the client can try another.
The response is a "txmatches" message containing the same request id plus a list of txmatch structures:
- block hash it appeared in
- merkle branch linking the tx to the header
- transaction data itself
The SPV client can now check the server is not lying to it by verifying the merkle branch against the block headers it has downloaded (possibly in parallel with this operation).
How will a lightweight client find new TX's sent to it, when it only knows its own keypairs? Either it must request full blocks w/ all TX's, or someone queries nodes for TXs/keys in a privacy-squelching manner?
Yes, indeed. A simple implementation forces you to choose between requesting everything (ick) or revealing your public keys to a trusted node. There are some other ways to do it:
- Extend the pattern matching opcode language to allow for prefix matches. Choose a short prefix. Now you get some transactions that aren't relevant but also the ones that are. A better design might allow you to provide arbitrary Bloom filters over addresses/pubkeys.
- Split up your request across many nodes so any individual node only sees a few.
- Do something a bit like onion routing. Contact 10 nodes, request a public key from them. Now connect to 10 other nodes and send a txmatch message but including an IP address of the one of the 10 nodes and a request ID (so they can be multiplexed together). The pattern match tx is encrypted under that public key (you can assume there's a stable ip to pubkey mapping). Now it's hard for nodes to know which computers own which keys. They just get requests from random IPs.
I haven't thought about the last one much. I think for now just using a bunch of trusted nodes would work OK, kind of like checking your mail. They can tie a few keys together but not all.
What are the ramifications of some nodes on the P2P network having only partial blocks? Will we need to introduce some sort of seek-nodes-with-full-block-contents logic for lightweight clients to find supernodes?
SPV clients do not accept connections nor do they register with peer discovery mechanisms. They are not really participants in the P2P network at all. The P2P traffic is really just between the full backbone type nodes.
What are the ramifications of partial blocks on the JSON-RPC API, if any?
For now just disabling JSON-RPC if you're not running a full node is probably fine. If you want to mine or be a merchant requiring you to handle the block chain isn't too much to ask. In future it might be but by such a point, there'll be a lot more manpower available
How will old clients behave, when faced with partial blocks? Surely we want them to keep working?
They won't request them so the transition is backwards compatible.
My initial thoughts turn towards pruning spent tx's, because, if you can do that (and the network survives), you can handle partial blocks.
I think the two are probably unrelated though both worth doing. Pruning is really about optimizing full node storage. Disk space is so cheap today it's not a big deal IMHO. In future if nodes start regularly running out of IOPs then people may be forced to host the block chain in RAM. At that point pruning would certainly become more interesting.