A bitcoin UDP P2P protocol extension

[jgarzik]

Here is a rough draft implementation of a UDP P2P protocol extension
for bitcoin:

     https://github.com/jgarzik/bitcoin/tree/udp
     http://yyz.us/bitcoin/udp-v0.patch

Protocol specification (such that it is):

- UDP, bound to same port as TCP P2P (normally 8333)
- Active, simultaneous TCP P2P connection required (useful against DoS and other attacks)
- Same message format as TCP P2P, same pchMessageStart conventions, etc. (my CNetMessage pull req would be helpful here)
- Multiple P2P messages per UDP packet permitted
- Max UDP packet size 100*1024 bytes
- Advertises NODE_UDP in nServices
- New "getudpcook" TCP P2P command returns a "udpcook" message, containing a yummy cookie
- UDP P2P commands "inv", "tx" and "addr" are handled as if received via TCP
- UDP P2P command "udpsub" sets a mask, that subscribes to one or more data broadcasts.
- When USM_INV_BCAST mask bit is set, receive "inv" messages via UDP rather than TCP.

Project and design goals (or, why do this?):

- It is theorized that UDP may be useful for some messages we broadcast throughout the network
- A productive discussion STARTS with code, otherwise discussion continues forever.
- Investigate improving "inv" relay speed
- Investigate unconditional "tx" broadcasting via UDP, as "inv" alternative, for small tx's.
- Investigate improving block relay speed (or perhaps block header relay speed)
- Open up new design avenues, for P2P patterns more suited to UDP than TCP in general.

It must be emphasize that this is supposed to be a starting point, not a set-in-stone requirement that This Is How The Protocol Must Look.  I don't think we will know how the UDP protocol should look until after testing and experimentation, or even if it is a good idea at all.

[Come-from-Beyond]

- Max UDP packet size 100*1024 bytes

Is it possible to get rid of this magic number? To avoid problems like we have now with 1MB limit.

[r.willis]

Not looked at the code yet, some thoughts:

1) In IPv4 max UDP data size is 65507 bytes.
2) What about big messages (block)?
3) I think relay speed can be increased only by reducing network diameter.  Introducing super-nodes with thousands of connected peers can greatly help here.
4) The only (IMO) UDP advantage: hole punching. But requirement of active TCP connection defies it. And hole punching will need protocol extension.
5) unsolicited tx broadcast will increase traffic of full nodes (especially well-connected) and less people will run them.

[Zeilap]

Not looked at the code yet, some thoughts:

1) In IPv4 max UDP data size is 65507 bytes.
2) What about big messages (block)?
3) I think relay speed can be increased only by reducing network diameter.  Introducing super-nodes with thousands of connected peers can greatly help here.
4) The only (IMO) UDP advantage: hole punching. But requirement of active TCP connection defies it. And hole punching will need protocol extension.
5) unsolicited tx broadcast will increase traffic of full nodes (especially well-connected) and less people will run them.

Maybe I'm reading this as overly negative, but I think you need to look at this in terms of how it can provide a foundation for experimentation and further work, rather than how it won't immediately solve every problem and will create others.

The point is that the protocol as it stands has shortcomings, and these may become problems as we scale up (in number of transactions, nodes or block size). As such, the protocol will have to evolve. Having work in place for sending and receiving some messages over UDP allows for more possibilities when considering solutions to larger network issues, and will give us a better understanding of the benefits of different approaches.

[r.willis]

I just don't see how addition of UDP can help to solve any problems - current of future. But it will add some problems - that's for sure.

[2112]

I just don't see how addition of UDP can help to solve any problems - current of future.

Supporting UDP is the first step required to support IP broadcast, multicast and anycast. This would realize tremendous savings of bandwidth. The current p2p network emulates broadcast in an extremely inefficient way with TCP/IP flooding, just because majority of the current ISPs don't have any way to properly route IP multicast traffic.

 

[r.willis]

Tremendous? Hardly.
Let's see, how much can we save on inv messages.
We connected to, for example, 20 peers, current transaction rate is 60000 tx/day.
inv message is <100 bytes including TCP/IP overhead.
So, overall it's maximum 114 MB/day. Do not think it's too much for a full node.
Bandwidth is cheap today (~2$ per TB), so why bother?
Bloom filter makes inv spam non-issue (and can't be applied to broacast/multicast).
Invstigation into network topology and means to reduce

[2112]

Tremendous? Hardly.
Let's see, how much can we save on inv messages.
We connected to, for example, 20 peers, current transaction rate is 60000 tx/day.
inv message is <100 bytes including TCP/IP overhead.
So, overall it's maximum 114 MB/day. Do not think it's too much for a full node.
Bandwidth is cheap today (~2$ per TB), so why bother?
Bloom filter makes inv spam non-issue (and can't be applied to broacast/multicast).
Invstigation into network topology and means to reduce

I think you've made a mistake in your calculations.

At the source of a transaction or a block none of the peers heard about it. So all the peers will ask the full transaction or the full block. Only at the later stages of the network flood the redundancy of p2p starts approaching the efficiency of multicast. The first stage is most crucial both in the cumulative sum of bandwith and the cumulative sum of bandwith*latency product, especially for the well-connected peers, for which number of connections is significantly greater than the default 8.

This is especially important for blocks, because multicasting of new blocks will reduce the probability of creating orphans.

In addition to facilitating IP broad-/multi-/any-cast the UDP is the first step to facilitating the transmission of the Bitcoin protocol over non-IP media, e.g. raw radio protocols.

Last but not least, learning about network multicasting and bandwidth sharing will probably help dispel other misconceptions prevalent amongst Bitcoin developers, epecially related to sharing of the blockchain storage. C.f. gmaxwell's defense of storing blockchain in a raw stdio/iostream file:

In this case, because the blockchain is inherently read only an append only stream is a fantastic, highly robust, extremely durable, and perfectly efficient data store for it.

[r.willis]

I think you've made a mistake in your calculations.

Can you point exact point I made mistake?

At the source of a transaction or a block none of the peers heard about it. So all the peers will ask the full transaction or the full block.

Every full note will receive every block and every transaction exactly once - be it p2p, or multicast. Difference only comes from inv messages.

This is especially important for blocks, because multicasting of new blocks will reduce the probability of creating orphans.

It will be true if multicasting was reliable/censorship-resistant. Any miner should run well-connected
node to help network.

EDIT: typo fix.

[misterbigg]

The reference client doesn't need new UDP features, it needs to be tightened up and put out as an official 1.0 release that is neither an alpha nor a beta. In other words, something that is ready for the masses. Furthermore, adding UDP only increases the amount of behavior which is not described in any official specification document. I find it puzzling that while on the one hand, developers want to progress in a "slow conservative fashion" because Bitcoin is still in beta and the software is not ready yet but then on the other hand we're talking about new features?

I guess this is a consequence of Bitcoin development not being driven by commercial interests. There's no deadline, no boss to be accountable to, and no proper customers. Businesses like MtGox, BitInstant , or SatoshiDICE probably care little to not at all about whether the software is "ready" or not, or even if new features are added, so they don't really have much to say as long as they keep getting customers and making money.

Capitalism works best when there is competition for profits, which drives the product cycle in the direction that satisfies customers the most. The Bitcoin software development model is not a capitalist model. It more closely resembles a government-run industry with all of its inefficiencies. For example, like the United States Post Office. "Prices" and business rules are not set by the market but rather central planners who think they know best. But no amount of good intentions can substitute for information contained in price signals. The core developers work on what they think is best. There are no performance reviews, and no competitors for their job. The Bitcoin software is free, so a loss of customers is not felt as a drop of revenue. The Bitcoin foundation doesn't answer to the users either. In short there is no free market pressure for the Bitcoin software to evolve in a direction that most satisfies consumers.

I believe that if we don't get some sort of commercial rigor into the Bitcoin software development process, it will forever share the characteristics of many open-source projects that are governed by idealistic neck-beards who work a "real" job at the same time as they contribute their spare time to the open source project as a "hobby." Specifically, that we will get software whose development is driven not by the needs of users, business, or the peformance pressures of the free market, but rather what shiny new feature or personal opinions dictate should happen next. The lack of any formal specification is one of the symptoms of this malaise. You can tell when a project is heading in this direction because there is a chorus of excuses like "if you don't like it, then work on it yourself", "if you want specs then write them", or "I'm working on this for free so I'll do what I want." Talk that would normally get you fired from a typical business.

Bitcoin software development needs feature planning, milestones, deadlines, release schedules, and quality control. Not a never ending litany of features which will keep it in a perpetual "beta" state.

- Investigate improving "inv" relay speed
- Investigate unconditional "tx" broadcasting via UDP, as "inv" alternative, for small tx's.
- Investigate improving block relay speed (or perhaps block header relay speed)
- Open up new design avenues, for P2P patterns more suited to UDP than TCP in general.

Premature optimizations. Let's get 1.0 out first before we start improving things?

[r.willis]

Bitcoin software development needs feature planning, milestones, deadlines, release schedules, and quality control.

And to make all this realistic: funding.

[Schleicher]

Many cheap routers have problems with forwarding UDP packets.
I'm not sure if this would be useful for an p2p application.

[2112]

Can you point exact point I made mistake?

Sure. Network router load is calculated using two quantities: bps and pps; bits per second and packets per second. In addition you need to account for the round-trip latency of the peer-to-peer links.

Every full note will receive every block and every transaction exactly once - be it p2p, or multicast. Difference only comes from inv messages.

This is only asymptotically true. In fact it is completely false for a "well-connected miner". Lets assume the default maximum of 150 connections. After discovering a new block it will push 150 "Hurray!" messages. It will get 150 "Gimme!" messages from his peers, and will send 150 copies of the new block. And I'm discounting PSH+ACK packets that may be required (Hi Mr.Nagle!, if you are reading this; I guess not: Last Active: 2012-09-05, 14:06:16).

So this is minimum of 450 IP packets and 1 additional round-trip with TCP/IP flooding vs. 1 packet and 0 delay with UDP/IP multicast. I'm not going to even bother counting the bps load required, just the pps load savings are tremendous.
 

It will be true if multicasting was reliable/censorship-resistant. Any miner should run well-connected
node to help network.

I suggest that we leave the issues of cooperation vs. competition of Bitcoin users to an another occasion.

I'm not going to launch into an impromptu networking lecture. Anyone really interested with "why UDP?" and "why multicast?" and "how to make them reliable?" can search the ample documentation and deployment experience of financial extranets, e.g. NASDAQ.

[r.willis]

Sure. Network router load is calculated using two quantities: bps and pps; bits per second and packets per second. In addition you need to account for the round-trip latency of the peer-to-peer links.

This would realize tremendous savings of bandwidth.

I was talking about bandwidth, and so was you. Do not see any mistakes on my side, and some
claims without proofs on yours.

This is only asymptotically true. In fact it is completely false for a "well-connected miner".

It is absolutely true. Note: I was talking about receiving, not sending. For sending you will save
some bandwidth - network-wide, your saving will be nearly total volume of blocks and transactions.
So, total bandwidth will be halved. Is it worth it? I doubt it. And miners can easily handle such
levels of bandwidth.

can search the ample documentation and deployment experience of financial extranets, e.g. NASDAQ.

It would be more interesting to see successful usage of multicast for p2p application.

[grau]

I guess this is a consequence of Bitcoin development not being driven by commercial interests.

Exactly.

[kjj]

Good to see that multicast is still widely misunderstood.  While it is true that a message needs to be received the same number of times in multicast (that number being the number of nodes), what changes is how many times it needs to be sent.  It also reduces latency, since the distance across the network becomes n routing hops + 0 verifications, rather than the current n routing hops + (n-1) verifications.

[alexkravets]

The reference client doesn't need new UDP features, it needs to be tightened up and put out as an official 1.0 release that is neither an alpha nor a beta. In other words, something that is ready for the masses. Furthermore, adding UDP only increases the amount of behavior which is not described in any official specification document. I find it puzzling that while on the one hand, developers want to progress in a "slow conservative fashion" because Bitcoin is still in beta and the software is not ready yet but then on the other hand we're talking about new features?

For example,

1. Enabling wallet encryption by default with a pass phrase being collected from the user on the first launch.
2. Adding an "automatic backup of encrypted wallet by email upon new addresses or contacts being added or removed" (similar to what Blockchain.info does).
3. How about a simple "Import Private Key" in addition to currently obscure hidden RPC command to do the same ?

Etc.

This would actually be useful.

- Investigate improving "inv" relay speed
- Investigate unconditional "tx" broadcasting via UDP, as "inv" alternative, for small tx's.
- Investigate improving block relay speed (or perhab]ps block header relay speed)
- Open up new design avenues, for P2P patterns more suited to UDP than TCP in general.

Premature optimizations. Let's get 1.0 out first before we start improving things?

Agreed. There is definitely much bigger "fish to fry" at this point with much larger positive impact on the ecosystem and adoption rate.