|
bits of proof beta release 0.9 now features:
- support for SPV clients using bloom filter
- reduced disk footprint increased speed on server side:
. block chain with indices is 10GB now.
. loads from scratch under 4 hours
. retrieval of transaction history for a set of addresses in a few seconds roundtrip over the bus.
- a redesigned API
. with bloom filtered subscription to validated transactions on the bus
. encrypted wallet local file with several accounts (BIP32) and colors
|
|
|
|
I give up arguing for being explicit, and will also follow those conventions just as if they were specified.
Please don't - we really need to work together to get the conventions and specifications at least "singing from the same hymn book". (maybe "guidelines" to go along with the specifications?) No worries. I am willing to comply with what is written into the proposal. I tried to include more, if not applicable for the spec, then lets add a recommendations section enumerating, that: 1. use consecutive indexes at generation 2. record highest generated index per level 3. do not use more than 3 levels 4. record time point an extend key (level) was created Any other? I actually implemented the spec as close as it can get here: (not yet merged to main of bits of proof): https://github.com/bitsofproof/supernode/blob/lean/api/src/main/java/com/bitsofproof/supernode/api/ExtendedKey.java |
|
|
|
I don't want to specify how wallet software should work.
OK, you do not want to specify but expect that they follow conventions. I give up arguing for being explicit, and will also follow those conventions just as if they were specified. |
|
|
|
Keys generated within a single sub account might also use any random index.
That's just complicating things for yourself, but yes, you can. grau, I think the part you are missing is that there is a "convention" tied in with BIP 32. I think the proposal misses to make these conventions explicit, and I tried to point out what problems this imposes. In fact I would want to go further, not calling them conventions but be explicit part of the "standard". The suggestions I made would put further limits on how the proposal is used for the sake of better scan performance and simpler backup procedures. |
|
|
|
|
The time point a key was created would (if serialized) also greatly aid the reconstruction of the wallet since it would reduce the scan to blocks after that time point.
|
|
|
|
Where does the 2^32 figure come from?
The proposal does not require consecutive generation of the keys but defines a 32 bit integer as index. Sure, but I was talking about the default wallet layout, with consecutive indices (1,2,3,...) for each subaccount. And I believe that other layouts that users might choose to use would be just as easy to scan. Keys generated within a single sub account might also use any random index. |
|
|
|
Where does the 2^32 figure come from?
The proposal does not require consecutive generation of the keys but defines a 32 bit integer as index. The reason I don't like putting a maximum index in the serialized form is because that doesn't tell you what to do when you run out of indices.
An answer could be: re-use keys. This reduces privacy, but it does not have to be without prior warning: e.g. the client could warn if 90% of key interval is used that the interval should be increased and wallet backup created to avoid reduced privacy as pool exhausted. I see that the look ahead scan you describe would also practically work (provided increasing not random indices are used), but like the maximum better because it supports the use case of an audit as the serialized key would provide the set of potentially used keys without heuristics involving the block chain content. |
|
|
|
I still believe that for the default wallet layout (subaccounts at level 1 and linear path for each subaccount) and for other non-pathological layouts, you could let your computer run for a couple of hours and retrieve all your privkeys, because the depths of the branches (including branches that have keys that were never active on the blockchain) would be quite small.
I guess a couple of hours will not cut it. Even if you have an UTXO indexed by keys (that is not even the case for the Satoshi client), it is still prohibitive to scan 2^32 possible generated keys. I believe that you either store what much smaller subset was potentially used, or you can not practically reconstruct the set of keys used, which defeats a purpose of the proposal, that is not to have the need of regular backups. |
|
|
|
I say this is all implementation specific, and doesn't belong in the serialization.
I thought the advantage of this proposal would be to have a wallet that is able to provide 1. a high number of keys to support privacy 2. does not require regular backup, at least to avoid key loss. If serialization format (that is the ultimate backup) fails to deliver enough information to recreate all keys that might have output in a practical time span, then it fails the second goal. If you would add a maximum sequence to the serialization of the extended key then backups would only be needed if a new extended child key is created (sub account), or if the maximum is exhausted and need to be increased to retain the same privacy level. The client could also warn of loss of privacy and re-use already used keys if hitting the maximum sequence of already known keys, until the user increases the maximum and creates a backup. |
|
|
|
I suggest to set and serialize maximum i for each extended key. Such serialized string would define the wallet keys without the need of a regular backup until maximum number of keys not exceeded.
Huh? Why would regular backups be needed? The major advantage of deterministic wallets is that backups aren't needed (see here). Assume you lost your wallet storing the unspent transactions, but you have the root key for the BIP32 generator from a cold store. Theoretically nothing is lost since you can scan the block chain for all outputs that can be spent with descendants of the root key, but practically you are not able to do this if you do not know what sequences were used. Therefore you either have to backup the sequences used or simpler set a limit to them in beforehand that is a practicable magnitude for a scan. As Pieter mentioned in the link that I gave you, it should be very practical to scan in mostly all cases of wallet layouts, the branching factor and depth are quite tiny. The assumption there is that if a key with a sequence number is not on the block chain, then higher sequences were not used, right? I think that would not hold. A shop usually offers unique addresses to receive payment for individual orders. Some orders never get paid, therefore their key will not be present in the block chain, however keys with higher sequence number might be. A reconstruction of the wallet would fail at such a gap. |
|
|
|
I suggest to set and serialize maximum i for each extended key. Such serialized string would define the wallet keys without the need of a regular backup until maximum number of keys not exceeded.
Huh? Why would regular backups be needed? The major advantage of deterministic wallets is that backups aren't needed (see here). Assume you lost your wallet storing the unspent transactions, but you have the root key for the BIP32 generator from a cold store. Theoretically nothing is lost since you can scan the block chain for all outputs that can be spent with descendants of the root key, but practically you are not able to do this if you do not know what sequences were used. Therefore you either have to backup the sequences used or simpler set a limit to them in beforehand that is a practicable magnitude for a scan. |
|
|
|
I suggest to set and serialize maximum i for each extended key. Such serialized string would define the wallet keys without the need of a regular backup until maximum number of keys not exceeded.
It would also facilitate audit since the set of possible descendants of a key would be of a practicable size. Also the bloom filter could be precomputed for SPVs including all possible addresses derivable.
I'm not convinced. Part of the point is to be able to generate more keys on the fly as needed, without all of them being known in advance, or even be known how many there are. There are use cases where knowing all possible keys (or addresses) in advance is an advantage. I mentioned two: audit and bloom filter computation. The limit could be optional with default 2^31 as is by the definition. |
|
|
|
Any comments from iddo, or other people?
I suggest to set and serialize maximum i for each extended key. Such serialized string would define the wallet keys without the need of a regular backup until maximum number of keys not exceeded. It would also facilitate audit since the set of possible descendants of a key would be of a practicable size. Also the bloom filter could be precomputed for SPVs including all possible addresses derivable. |
|
|
|
I suggested that perhaps he just left it running, saving the best result, while he did the final preparation for the release.
Didn't he need to mine both block 0 and block 1? And unless he knew the Times headline before it was printed, that means he mined 2 blocks within 6 days, right? Isn't that the point of the headline, to prove that he hadn't premined more than that? This is an appealing explanation. He wanted the headline in the genesis. He needed a few days to patch and run the client for a strong POW on it, then deleted the superfluous code and started over with block 1,2... |
|
|
|
|
2011 was enough to wake me up. Even if though MtGox managed to press the snooze button the second time in the row, the entire world will wake up.
|
|
|
|
Thanks for your answers. What ECDSA implementation do you use and how many signature checks/sec it can perform?
Bouncy castle. Just measured on my laptop with 2.5GHz i5 single threaded validation of ECDSA: 830 sigs/second. Beware that 1. bits of proof has multithreaded transaction validation, therefore this scales about linear with number of processors available and their speed. 2. In practice signature validation is distributed over time since the server usually receives and validates transactions while unconfirmed, therefore block validation only triggers validation of those transactions it did not receive before. We currently have a few hundred transactions per block, 95+% of them already validated in the minutes between blocks, therefore block validation is a fraction of a second even with this implementation of ECDSA. Storing the block is much more effort than validating it in regular production. The only use case where validation speed plays a role at the moment is initial block chain load and verification. There it can be up to about 2 seconds per block in worst case. This is addressed by both Satoshi and bits of proof by not validating signatures until a checkpoint height (well behind current last block). In summary, although this pure Java implementation of the signature validation is about a magnitude slower than the C heavily optimized for secp256k1, I do not consider it a bottleneck for now. If it ever becomes one, it should not be a big deal to use JNI to call such an optimized native one. bits of proof already uses JNI to drive native LevelDB. |
|
|
|
Do you have host running bitsofproof 24/7?
What are you using for ECDSA? How many sigchecks/sec does it perform?
What's maximum number of peers you tested it with?
What anti-DoS rules do you have?
Thanks.
Yes, bits of proof runs since months in parallel on bitsofproof.com, with a few restarts for upgrades or data migration. I run it with peers 20 regularly, sometimes 50. There are a few ban rules similar to bitcoind and limits on message size (identical to max block) also limits on caches of orphan blocks and unconfirmed transactions. There is a limit on unsolicited connections. |
|
|
|
|
Show Posts
