Block Batch Filters for Light Clients BIP draft

[bitaps]

Here is a link for a draft of a BIP for  compact probabilistic block filters alternative of BIP 158

https://github.com/bitaps-com/bips/blob/master/bip-block-batch-filters.mediawiki

https://github.com/bitaps-com/bips/blob/master/bip-mempool-transactions-filters.mediawiki

Summary:

- Block filters batching reduce total filter size. Total filters size  is smaller then BIP 158 at all total size savings  22% (3.36 GB vs 4.3 GB)
- Separation of filters by address type allows lite client not to download redundant information without compromising privacy.
- filter is deterministic  and could be committed as commitment in an coinbase transaction in future.
- unlocked ability to filter unconfirmed transaction for SPV nodes used BIP 157 instead of BIP 37 due privacy leak in BIP 37
- more bandwidth consumption reduced in contrast with transaction filters and downloading full blocks for affected addresses
- fully compatible for future consensus layer soft-fork to make block filters commitment one of the block validation rule


Any feedback or discussions are welcome

[Carlton Banks]

This might be a naive question, but are there any particular difference between your draft and Neutrino?

BIP158 essentially is Neutrino (Neutrino is one implementation of it)


Sounds interesting, although I'd be most curious to know whether these filters can be used to check for zero-confirm announcements (which BIP37 can do, but BIP158 cannot)

[bitaps]

Zero-confirm announcements is interesting feature. Need to think about how this could be implemented.

[bitaps]

Here is a link for a draft of a BIP for  mempool transactions filters

https://github.com/bitaps-com/bips/blob/master/bip-mempool-transactions-filters.mediawiki

Summary:

- unlocked ability to filter unconfirmed transaction for SPV nodes used BIP 157 instead of BIP 37 due privacy leak in BIP 37
- more bandwidth consumption reduced in contrast with block filters and downloading full blocks for affected addresses
- proposal for future consensus layer soft-fork to make block filters commitment one of the block validation rule

[Carlton Banks]

Here is a link for a draft of a BIP for  mempool transactions filters

https://github.com/bitaps-com/bips/blob/master/bip-mempool-transactions-filters.mediawiki

forgive a possibly ignorant question, but is it not the case that BIP152 messages are only processed with respect to the current top block? Doesn't this mean that if a headers-only client cannot get incoming transactions for block X if it misses an opportunity to get a compact block message for block X? Could BIP-37 handle this case?

[bitaps]

Here is a link for a draft of a BIP for  mempool transactions filters

https://github.com/bitaps-com/bips/blob/master/bip-mempool-transactions-filters.mediawiki

forgive a possibly ignorant question, but is it not the case that BIP152 messages are only processed with respect to the current top block? Doesn't this mean that if a headers-only client cannot get incoming transactions for block X if it misses an opportunity to get a compact block message for block X? Could BIP-37 handle this case?

The main reason why light client should not be used BIP 37 this is privacy leak. This is a reason  BIP-157/158 is exist, so mixing BIP-37 with BIP-157 is not correct idea. I don't know about limitation for BIP152 to process only the current top block. I will check this in bitcoin core source code. But in any way we can request full block (getdata message) to retrieve transactions with affected addresses like it purposed in BIP-157/158. This is not so effective like in combination with BIP-152 but it will work correctly and privacy safe.

Also the best practice of privacy safe  for light client is retrieve all data like full transaction or full blocks over tor connection with new identity and new IP for each item.
We need this to prevent attempts to collect and accumulate addresses owned by light client wallet

 

[Carlton Banks]

The main reason why light client should not be used BIP 37 this is privacy leak.

right, the NODE_BLOOM peer knows exactly which tx announce the headers-client is interested in.

This is a reason  BIP-157/158 is exist, so mixing BIP-37 with BIP-157 is not correct idea.

I concur, it would be totally pointless to mix BIP37 with BIP157/8

What I mean is: can BIP37 actually do what I'm saying? Can BIP37 on a headers-client get 0-confirm announcements for blocks during which the headers-client was offline? If not, no-one should care too much about BIP152 only being available for current top block, as BIP37 cannot do this either. To answer the question, we would need full understanding of BIP37's capabilities when catching up from an offline state.

[bitaps]

BIP 37 does not receive  0-confirm announcements for blocks during which the headers-client was offline. To get synchronized status in the lightweight BIP 37 client, it downloads all headers, and then block by block, requesting blocks with MSG_FILTERED_BLOCK (a special type of inv). since the complete response node sent a Merkleblock message with a list of affected transactions. The full node responds with a Merkleblock message with a list of affected transactions
(partial merkle tree). Then light node requests each affected transaction with getdata message to full node. In this schema we have no 0-confirm announcements but  can download filtered tx hash list in missed blocks.

[bitaps]

Drafts updated

[gmaxwell]

Receiving all unconfirmed txn without witnesses is 1.6kbyte/s or less.

Sitting logged into many websites more bandwidth than that.

The proposed protocol seems like it uses a pretty significant fraction of the bandwidth as sending the transaction. It also requires hashing the transactions in the mempool to do the 'short id matching'.

The BIP text is also underspecified in that that BIP152 hashing scheme is salted with the block hash, and there is no block hash for unconfirmed transactions. Without an attacker unpredictable salt BIP152's short hashes are particularly vulnerable to collision.

Unconfirmed transactions are generally a major vulnerability in lite wallets:  A number of years ago there were a bunch of nodes on the network that generated fake 21 million btc payments to every recently used address and handed them to lite wallets that connected to them, causing them to display enormous fake unconfirmed transations-- and for some software like android wallet, it made them almost completely unable to make payments until you restored the wallet from a backup. Not to mention the hiding issue.

Is there really a particular reason to filter unconfirmed transactions at the expense of privacy--since what txn you fetch is still visible?  With BIP 157 when a block matches the user fetches the entire block, at least somewhat preserving the anonymity set.

Aside, the graph should show getblocktxn not blocktxn.  I believe the 'getcfproof' message could also work by index instead of txid like getblocktxn does, for a massive bandwidth reduction.

I doubt there will be much appetite for any commitment structure that requires recomputing the commitment any time the txn list in the block is changed. Particularly any that involves doing it for 5 different kinds of filters and which will go out of date as scriptpubkey types change.  The BIP157 design being completely agnostic to the content of the scriptpubkey was an intentional design to conserve storage and optimize for the possibility of eventually turning it into a viable commitment.

[bitaps]

The proposed protocol seems like it uses a pretty significant fraction of the bandwidth as sending the transaction. It also requires hashing the transactions in the mempool to do the 'short id matching'.

The idea is receive transactions flow with stripped data (inputs and outputs converted to distinct set of 4 byte items for specified filter type). And yes transaction flow means a lot of  sending the transaction. I got your point about "short id" will think how to implement it more properly.

Is there really a particular reason to filter unconfirmed transactions at the expense of privacy--since what txn you fetch is still visible?  With BIP 157 when a block matches the user fetches the entire block, at least somewhat preserving the anonymity set.

I suppose that most common case light client receive about 2-3 payments daily, so in case we got positive test we be able use TOR with new identity and different nodes to request complete transaction. I think it should protect privacy Or am I missing something?

Aside, the graph should show getblocktxn not blocktxn.

. Yes.

believe the 'getcfproof' message could also work by index instead of txid like getblocktxn does, for a massive bandwidth reduction.

getcfproof is not use txid it use block hash. getcfproof is only actual in case we have commitments. In case no block filter commitments we should omit this step.

Have you read this one  https://github.com/bitaps-com/bips/blob/master/bip-block-batch-filters.mediawiki? any feedback about this draft?

[bitaps]

I doubt there will be much appetite for any commitment structure that requires recomputing the commitment any time the txn list in the block is changed. Particularly any that involves doing it for 5 different kinds of filters and which will go out of date as scriptpubkey types change.  The BIP157 design being completely agnostic to the content of the scriptpubkey was an intentional design to conserve storage and optimize for the possibility of eventually turning it into a viable commitment.

The idea of separating filters by address type gives a significant reduction in the amount of required data requested by the light node. Refusing to divide by type will actually give a better degree of filter compression and the overall size will be even smaller than what we can achieve now. Block batch filters total size  is smaller then BIP 158 at all total size savings  22% (3.36 GB vs 4.3 GB). We could win another 3-4 percent. But assuming that mostly light nodes use only one type of address, separation provides significant gains in bandwidth reduction. Is the calculation of several additional hashes more important than reducing traffic several times?

In case we try use BIP 158 for commitment this is also  requires recomputing the commitment any time the txn list in the block is changed. Is additional double sha256 is so big problem? 
Also I think we can redesign commitment structure to exclude from commitment filter types which have no any affected script types in recent block it will solve problem with  out of date script types.