Idea: Transaction propagation optimization

[nmnkgl]

Hi everyone,

Recently I’ve been thinking about ways to improve transaction propagation protocol.

It turns out that straightforward flooding scales poorly in terms of bandwidth required to run a Bitcoin Node. In other words, increase in a number of full nodes in the network or in transaction rate will dramatically increase bandwidth requirements for the infrastructure (more precisely, for nodes which accept incoming connections). I can provide a more detailed summary of the problem on request.

I’ve been running a Bitcoin Node and measurements showed that there is a significant redundancy in messages I’m sending. More precisely, only 5% of inv messages I’ve sent got a get_data back.

So I had a couple of ideas to make transaction propagation protocol more efficient while preserving and potentially expanding current privacy guarantees, the robustness of the network and end-to-end throughput of the network.

One of my ideas is applying filters to links to send only a subset of transaction inv messages. At the time of a handshake, a joining node can set a filter, in accordance to which the second node will advertise transactions. A joining node allocates the space of transactions among its neighbors so that it still has a full blockchain.

A full version of the protocol will include partial redundancy, malicious behavior detection, rotation of filters, ....

What do you think of the idea?
Were there any related prior research/proposals in Bitcoin/blockchains space?

[1715225152]

1715225152

[achow101]

Your proposal is not all that new, it's very similar to BIP 37. BIP 37 uses bloom filters to allow a node to only receive transactions that the node wants. It is specifically used by lite wallets in order to avoid downloading the entire blockchain. However BIP 37 has its downfalls; specifically it is computationally expensive for the full node serving the data, and it does not preserve privacy well (an issue specific to wallets).

However, there is one problem with your idea: what do you choose to filter? And with a filter, how do things not in the filter get transmitted through the network to the miners? How do you guarantee that someone will relay your transaction? The flood protocol practically guarantees this. But having everyone filter out transactions to relay would result in a lot of transactions being missed and thus not relayed.

[nmnkgl]

Your proposal is not all that new, it's very similar to BIP 37. BIP 37 uses bloom filters to allow a node to only receive transactions that the node wants. It is specifically used by lite wallets in order to avoid downloading the entire blockchain. However BIP 37 has its downfalls; specifically it is computationally expensive for the full node serving the data, and it does not preserve privacy well (an issue specific to wallets).

However, there is one problem with your idea: what do you choose to filter? And with a filter, how do things not in the filter get transmitted through the network to the miners? How do you guarantee that someone will relay your transaction? The flood protocol practically guarantees this. But having everyone filter out transactions to relay would result in a lot of transactions being missed and thus not relayed.

Thanks for the feedback!

Yeah, this was originally inspired by BIP 37. The real question here is what we need to add so that BIP 37 is applicable for full nodes on both ends (in terms of being robust to Eclipse and other attacks).

As for the second part, I think I need to provide more explanation.

For example, full node N joins the network (and connects to existing A1-A8 nodes), and it obviously wants to get all the transactions. However, there is no real reason to get 8 inv messages (worst case). So, this node sends a request to A1-A8 so that they filter transactions being sent to the node N. Thus,
A1 sends to N all the transaction invs within a range 0x0-0x2
A2 sends to N all the transaction invs within a range 0x2-0x4
...
A8 sends to N all the transaction invs within a range 0xD-0xF


so that node N sooner or later gets all the transactions.


This is very simplified version, and is vulnerable to some obvious attacks. I mentioned some ideas to make it more robust in the original post.

If you are concerned about the very first hop — it is a very good point, even though most of the nodes rely somebody's transactions most of the time. I need to think more about first hop strategy.