Message to devs from merchant

[Gavin Andresen]

You guys say that hard drives are cheap but you still have to scan the block chain at first. It takes a very long time to do but is the only trust less solution, isn't it?

Yes, today. But not at some point in the future. Please read about "UTXO commitments" in https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

[doof]

Look at chain size growth in time - it's parabolic!

Its exponential, e^x not x^2

[hhanh00]

You guys say that hard drives are cheap but you still have to scan the block chain at first. It takes a very long time to do but is the only trust less solution, isn't it?

Yes, today. But not at some point in the future. Please read about "UTXO commitments" in https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

It's good to know there is a plan out there to have ultra pruned tree verification. The hashes you mentioned are going to incorporated in Merkle trees? I thought that the Bloom filter mechanism was going to be extended to skip over used tx.

In any case, would it be compatible with deterministic wallets? Without the used txs, they will stop scanning for addresses too early.

[deepceleron]

You guys say that hard drives are cheap but you still have to scan the block chain at first. It takes a very long time to do but is the only trust less solution, isn't it?

Yes, today. But not at some point in the future. Please read about "UTXO commitments" in https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

It's good to know there is a plan out there to have ultra pruned tree verification. The hashes you mentioned are going to incorporated in Merkle trees? I thought that the Bloom filter mechanism was going to be extended to skip over used tx.

In any case, would it be compatible with deterministic wallets? Without the used txs, they will stop scanning for addresses too early.

A deterministic wallet is simply a way of generating addresses. Bitcoin classically generates new private keys using a true random number generator for each address. When a new address is needed to request a payment, it is generated at random and is not related to any other key. A deterministic wallet instead uses a pseudo-random number formula to create addresses, with only the initial seed state being a generated random number. This prevents the wallet backup from becoming obsolete, because any past or future address you created can be recreated from the seed backup.

Pruning is a technique for reducing blockchain disk space consumption, and is unrelated to your wallet addresses. We assume that the blockchain history has already been thoroughly validated by many miners after it is buried under many blocks and that every transaction in it was properly signed by the owner. Therefore, as long as the hash chain in the headers is correct, we don't need to verify the chain-of-ownership of every single bitcoin amount back to its original creation for our own security. This allows an individual client to remove the spent transactions from the blockchain, potentially freeing hard drive space, and even may enable a 'pruned network' p2p client to exchange a lightweight blockchain between peers.

The "plan out there" for pruning was put forth in the original Bitcoin paper: http://we.lovebitco.in/how-bitcoin-works/bitcoin-paper/#ch7 - it is just that nobody has written a complete client implementation that does this yet.

[hhanh00]

You guys say that hard drives are cheap but you still have to scan the block chain at first. It takes a very long time to do but is the only trust less solution, isn't it?

Yes, today. But not at some point in the future. Please read about "UTXO commitments" in https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

It's good to know there is a plan out there to have ultra pruned tree verification. The hashes you mentioned are going to incorporated in Merkle trees? I thought that the Bloom filter mechanism was going to be extended to skip over used tx.

In any case, would it be compatible with deterministic wallets? Without the used txs, they will stop scanning for addresses too early.

A deterministic wallet is simply a way of generating addresses. Bitcoin classically generates new private keys using a true random number generator for each address. When a new address is needed to request a payment, it is generated at random and is not related to any other key. A deterministic wallet instead uses a pseudo-random number formula to create addresses, with only the initial seed state being a generated random number. This prevents the wallet backup from becoming obsolete, because any past or future address you created can be recreated from the seed backup.

Pruning is a technique for reducing blockchain disk space consumption, and is unrelated to your wallet addresses. We assume that the blockchain history has already been thoroughly validated by many miners after it is buried under many blocks and that every transaction in it was properly signed by the owner. Therefore, as long as the hash chain in the headers is correct, we don't need to verify the chain-of-ownership of every single bitcoin amount back to its original creation for our own security. This allows an individual client to remove the spent transactions from the blockchain, potentially freeing hard drive space, and even may enable a 'pruned network' p2p client to exchange a lightweight blockchain between peers.

The "plan out there" for pruning was put forth in the original Bitcoin paper: http://we.lovebitco.in/how-bitcoin-works/bitcoin-paper/#ch7 - it is just that nobody has written a complete client implementation that does this yet.

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

[fbueller]

Spv could work, but youd want to be connected to a few nodes at a time to be confident. Just randomly hop around servers in the electrum server list. You can subscribe to callbacks for events from stratum servers if I recall correctly..

[gmaxwell]

Spv could work, but youd want to be connected to a few nodes at a time to be confident. Just randomly hop around servers in the electrum server list. You can subscribe to callbacks for events from stratum servers if I recall correctly..

Be careful with assumptions like these.  All (reachable) peers are malicious peers if an attacker controls your network connection. (E.g. compromised your ISP or router, or they are your ISP)

[grau]

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

Nodes might prune transactions but maintain a bloom filter of addresses they saw.

[ThomasV]

Electrum uses a 3rd party server, which if you connect to a cancer server, then you can be fooled to think you have bitcoins you really may not.

This is wrong. Electrum verifies all transactions in its history using SPV.

[hhanh00]

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

Nodes might prune transactions but maintain a bloom filter of addresses they saw.

A bloom filter cannot do that. An invertible bloom filter might but it is not decided yet if it is going to used (cf Gavin's message).

[hhanh00]

Electrum uses a 3rd party server, which if you connect to a cancer server, then you can be fooled to think you have bitcoins you really may not.

This is wrong. Electrum verifies all transactions in its history using SPV.

Yes but it could be fooled by omitting some transactions. I switched to Electrum from Armory. I still think it's the best solution at the moment for someone who doesn't want to spend hours downloading the blockchain but it's not suitable for a merchant.

Back to topic,
In my opinion, there should be a client that lets you:
- use a bootstrap file to catchup to a recent checkpoint
- catchup the remaining part of the blockchain by downloading simultaneously from multiple peers (bitcoin core uses 1 peer)
- performs validation in parallel and progressively as the chain gets reconstructed
- prunes the spent transactions locally. It could also prune the blocks if it doesn't care about relaying the blockchain.

If we are allowed to change the protocol, let there be a way to transfer a verifiable list of spent transactions and real transactions. I think this is what Gavin talks about in his post but I need to see more detail to be sure.

[grau]

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

Nodes might prune transactions but maintain a bloom filter of addresses they saw.

A bloom filter cannot do that. An invertible bloom filter might but it is not decided yet if it is going to used (cf Gavin's message).

false positives only extend the scan. All funds would be found.

BTW, an invertible bloom filter is not a bloom filter.

[hhanh00]

Thanks. If a node prunes spent transactions, a hierarchical wallet will not 'see' the empty transactions anymore and it will stop scanning after the gap limit is passed.

Nodes might prune transactions but maintain a bloom filter of addresses they saw.

A bloom filter cannot do that. An invertible bloom filter might but it is not decided yet if it is going to used (cf Gavin's message).

false positives only extend the scan. All funds would be found.

BTW, an invertible bloom filter is not a bloom filter.

You seem to be pretty sure of what you say but I really don't see how it would work. The client sends the filter parameters and the server creates a filter per session. The server scans its blocks. If the spent tx are removed from the server, how would it know how to match the filter?
I'm ok being wrong but could you describe the actual process?

[grau]

The server might prune but accumulate a bloom filter of addresses it ever saw.

I assume the scan knows the master public key. Addresses can be generated sequentially and checked against the bloom filter instead of the block chain until the bloom filter is negative for longer than the gap. Thereafter transactions can be fetched from pruned set by address.

[hhanh00]

You have it reversed. The client is the one sending the filter (with the filterload message) and the server is scanning the transactions. Of course, what you described could work but it is not the current protocol.

[grau]

You have it reversed. The client is the one sending the filter (with the filterload message) and the server is scanning the transactions. Of course, what you described could work but it is not the current protocol.

You assumed constraints I did not. The scan does not need to use the P2P protocol if it is about the wallet of the node
or if the client have extended ways to communicate with nodes it trusts to some level.