BIP 37 automatic filter update

[F.Madsen]

I'm having a problem with bloom filters when using the getdata message.

As I understand BIP 37, setting the bloom filter on a connection using the loadfilter message with flag BLOOM_UPDATE_P2PUBKEY_ONLY set, the responding client automatically inserts all outpoints (TXID and output index) in the bloom filter of transactions matching the filter. However, when I attempt to do exactly this, the node only ever sends transactions containing outputs TO the address I am filtering for, never transaction that contain outputs FROM the address I am filtering for.

My approach is as follows:

• Add the following public key to the bloom filter: 1JbFvyaMyHHmvvjc54h4YARq3uoX2ZZ4Qj (randomly picked)
• Send the filterload message containing the filter (as seen in the wireshark snippet below).

Code:

Bitcoin protocol
Packet magic: 0xf9beb4d9
Command name: filterload
Payload Length: 261
Payload checksum: 0x2bb02c45
Filterload message
    Filter
        Count: 251
        Data: 000000040004010000000100000000000000020000400000...
    nHashFunc: 13
    nTweak: 0x1a689ff4
    nFlags: BLOOM_UPDATE_P2PUBKEY_ONLY (0x02)

• Send the getdata message with the hash of all blocks containing transactions to and from the wallet (which I have identified on a block explorer for the purpose of testing this). The inventories are typed MSG_FILTERED_BLOCK.

The node responds with the following messages in order:

Code:

MerkleBlock: 000000000000000001201ab2679db7c87a432967c060596bad0ac093dac94111
Tx:......... 187b00588e8d7844c405da9a15d78dbd7ebb9b2887fb5aace6651719fede923e
Tx:......... dbbca0d839435ac77db65378a9607d17b0aa965f3027d84dcf0ee5115dc8d32f
Tx:......... 17c12f45956082a913c0c862d573cf067fa5ee96ba5b954914371da6bef51f7f
Tx:......... 36cce25d34a5da0f7d69253743eae7bb7c576e5543d0eb16db0f72967edb4bc6
Tx:......... 6ea48c6286b1691cd1ecec271a3f52a3fe099467bb6c926b41ba8942e21d4aa6
MerkleBlock: 0000000000000000049ee67931f6d4970a7d06b14b8aadd406223ab65efe5025
MerkleBlock: 0000000000000000019f3fc2c8da43e8a81373ae189ace4caec6c59969f46ff6
MerkleBlock: 000000000000000000f67dad206798db2e6c5dffec84d213397784be894e66fa

The 5 transactions in the list are correct, as the contain outputs targeting said wallet. However, block 0000000000000000019f3... contains transactions with inputs from said wallet, but the merkleblock message of this block is not followed by these transactions. Shouldn't the node add the outputs of these 5 transactions to the bloom filter, and thereby match the later transactions containing inputs pointing at them?

I have confirmed that the node is in possession of the relevant transactions. If i load the filter manually, adding the outputs myself, this works as intended.

Am I misunderstanding how BIP 37 is used?

(I also posted this at http://bitcoin.stackexchange.com/questions/43989/bip-37-automatic-filter-update-not-inserting-outputs)

[belcher]

Transactions with outputs to a certain address usually contain the hash of the pubkey rather than the pubkey itself.

It may be that your bloom filter only contains the pubKeyHash and not the pubkey itself, which would lead to behavour of only matching spend-to addresses. If so, to fix you need to add the pubkey itself to the bloom filter.

As I'm sure you're already aware, a bitcoin address like 1JbFvyaMyHHmvvjc54h4YARq3uoX2ZZ4Qj is an encoded pubKeyHash.

If you're working with bip37 bloom filters, you should be aware of their bad privacy properties. Any wire eavesdropper can usually derive all the addresses in your wallet. http://jonasnick.github.io/blog/2015/02/12/privacy-in-bitcoinj/

[F.Madsen]

Thanks for the answer!
Do you mean that our bloom filter should contain the unhashed public key? In this case, is it not possible to fetch UTXOs of wallets for which you do not know the full public key?

[belcher]

When you find a transaction that has an output to one of your addresses, add that txid to the bloom filter, then it will match to an input of a spending transaction. That way you can get spends without knowing the pubkey.