If you're not interested in using any of the features of the type-2 derivation, consider the extended public key the same way you'd consider a private key. It is not observable by the network, so if you don't reveal it, it is yours alone. This essentially turns the scheme in a type-1 derivation (you derive extended privkey from extended privkey).

What do you gain by not using all available data? I consider using one 512-bit hash much more elegant than doing several 256-bit hashes.

Or a QR code that contains the file.

Well, all bitcoin releases to date have been beta.

This is about the code that hasn't even been released, but will eventually become 0.8.

The first 193000 blocks should go much faster than the rest, that is normal. Before the last checkpoint, signatures are not verified.

One block per 5-10 seconds sounds slow. What kind of CPU/RAM are you on?

Because you yourself proposed putting "xpub" or "xprv" at the beginning. There is never a question of which one you are working on.

Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.

This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?

Can you put your debug.log online somewhere, or send it to me?

No (unless you're just talking about virtual memory). What OS are you on?

Your earlier post makes me think there is some memory leak or resource leak of some kind (locks in LevelDB maybe).

Yes, since 0.3.21.


Yes, it will. It does this rescan automatically at startup, so there is no feedback in the GUI - when it is loaded, the wallet should always be up-to-date with the block chain.

After the last checkpoint (block 193000 currently), signature verification is enabled, and in all likelyhood this is the bottleneck after that point.

Well, I have little problem with changing the serialization of the extended public/private keys if it's worth it. Whatever preliminary implementations exist, I doubt they have more than the internal key derivation mechanism.

I've generally given up on the human recognizability aspect of Base58 - if we wanted that, base64 or base32 would be far better choices, as they allow prefixing/suffixing with bit/byte arrays without affecting every single encoded character.


I don't consider this a priority anymore. A human will need to know what type of data it encodes anyway before using it.


Ok, I admit I like that. I'll ask around whether anyone objects. Instead of 0x00 + privkey, I'll use privkey + 0x01, so that it matches the serialized private key format exactly (which has a 0x01 at the because the corresponding public key is compressed).

None at all, it just isn't implemented.

What is a problem, is separating the wallet and the database environment. If your client (or your computer) crashes, you must have the database environment (the files in $DATADIR/database) to recover the data. If those are not available, the wallet may get corrupted. BDB isn't really intended to be used with free-floating database files - it expects them to be part of such an environment.

Now, we are slowly moving away from BDB. 0.8 (and this next-test build) do not use it anymore for the block chain indexes (it uses LevelDB instead), and we'll likely stop using BDB for wallet files as well. At that point, it will be perfectly safe to select any file/location for your wallet.

The question is really weighing the risks and benefits.

Some methods for block relaying will have less data transfer in case of invalid data, and other methods will have fewer round-trips.

An extreme of the first case is sending every node of the merkle tree separately, verifying it, and requesting its children. Eventually, when the entire merkle tree is transferred and verified, request the transactions one by one, and verify their hash matches what is claimed by the merkle leaf nodes. This requires 2*N-1 round-trips, and will never download more than a single invalid transaction.

An extreme of the second case is what we do now: send the entire block at once. There is at most 1 round trip, and we'll never download more than a single invalid full block.

What you are suggesting is a compromise, which is very close to the first example. You risk downloading one invalid level of the merkle tree, but at the cost of log(N) round trips.

Time for some math: with a very slow EDGE (2G GSM mobile data) connection, you have up to 600ms of latency, and a bandwidth of 20-60 kbit/s. That means that during one additional round trip, you could have transferred 1.5-4.5 kbyte. Unless the actual amount of data traffic is important, you're most likely trying to optimize transfer time. The calculation above means that an additional round trip is not worth it if it saves you less than a few kilobytes (and in more modern networks, much more). As we don't expect to receive many invalid blocks/trees/transactions, the actual amount saved by systems with more round trips is almost nothing, but they can protect against a worst case.

The technique proposed by BIP 37 involves sending just the necessary part of the merkle tree + transactions not known to be known to the receiver. This saves some trivial bandwidth (the transactions already known to the receiver), but doesn't introduce additional latency. I believe this will in practice be significantly faster than what you're proposing.

The immediate plans are simply to have the reference client support peers that request filtered blocks. BitcoinJ and derived SPV clients would be candidates for requesting filtering. We're not making the reference client SPV, or making it request filtered blocks.

Something related is switching the synchronisation mechanism in the reference client to be "headers first". This is different from "headers only" - it simply means we're first going to request only headers, so that the client knows what the best chains are its peers knows about, and then in a second (mostly parallel) step, fetch the actual blocks along the best known chain. This makes it much easier to make the initial synchronization parallellized. I believe most core developers agree this is the way to go, but it requires a lot of delicate implementation and testing.

However, once that is done, the step to making the program headers-only, where the step to download the actual blocks is skipped altogether, becomes quite small. Some believe a good way forward is to make the program start in such a headers-only mode (effectively SPV mode), and when resources (cpu/bandwidth/storage) allow it, suggest upgrading itself to becoming a full node by downloading and verifying all blocks in history. This will not be for very soon, though.

Ok, I'm afraid I misunderstood your proposal from the start. Sorry.

You're not proposing a way to send just the unspent outputs of old transactions - you're proposing a way to send transactions marked as 'fully spent'.

Yes, yes I believe this would work. It would allow a fully verifying node to bootstrap with SPV-level trust in the network and reduced download/cpu usage. I need to think harder about edge cases, but it sounds reasonable. Interesting. There's no risk in doing this all the way even - you still only need SPV trust.

Also, I believe some recent developments (for filtered blocks, which is a different use case, but still applicable) may improve your proposal. To transfer filtered blocks, we designed a very compact format for transferring any subset of the transactions while not losing the ability to verify the proof-of-work. See https://github.com/sipa/bitcoin/commit/partialmerkle.

Ok, maybe we first need to clear this up: how are you transferring the actual pruned transactions? The specification you give only contains a merkle root.

So all you are proposing here is a hack to have a UTXO merkle root per-transaction transmitted using the existing tx/block messages?

That means you'll still need some mechanism for transferring the actual unspent outputs... why not use that other mechanism immediately?

Also, there is no way to prove that the txid of the transaction matches what is claimed for a particular set of unspent outputs. A server can simply invent UTXO's and claim that they belong to some existing (and valid) transaction. So a server that serves pruned blocks can replace an UTXO of a very old but unspent transaction by one that credits an address of himself (with the same amount). There is no way the receiver can ever detect this, except by rebuilding the full UTXO set from scratch himself, which requires the full blocks.

Known problem in 0.7.1; will be fixed in 0.7.2 and 0.8.0.

I don't understand. At no point is a snapshot or a hash of the state of the UTXO set committed to the blockchain. There have been plans for adding a sort of "merkle root of the UTXO set" to the coinbase, which would change this discussion, but for now, no such data exists.

So, I don't see how you can verify anything a server claims about the UTXO set, except by building it yourself. The sender may arbitrarily prune a transaction which is spent, or leave a spent transaction in (making you think it can still be spent). Done correctly, no merkle root will break, and the PoW of all blocks will be a perfect match. You can indeed verify that the sum of the coinbases matches (ignoring fees that were forgotten to be claimed...), but this is trivial to bypass.

The point is not that transferring the UTXO set is not useful - it certainly is - but anything that requires trust in the sender has no place in the P2P protocol in my opinion. Maybe an RPC to export/import the UTXO set from/to a file... but at the P2P level, nodes shouldn't need to trust each other at all.