Small protocol changes for flexibility

[Cdecker]

Ok, some of you already know that we are trying to create an alternative client in order to play around with the network a bit, and try to diversify the way people build on top of the bitcoin network. Currently we are trying to create a python client that interacts with the network, but does not itself participate in the race for the next block.

Now we noticed that the whole protocol is layed out for only one client, so for example the client version is sent in the initial handshake as an integer. Out idea would be not to use that version number as the client version but as the protocol version the client is using (right now it's 310), this way clients could agree on which version of the protocol to communicate with. Another improvement would be an additional message that would simply list all the capabilities the client can understand. This way the underlying protocol would be decided by the version integer and then clients could allow or disallow extension. There are already many ideas for extensions on this forum, a broadcast mechanism for example that would alert old clients that there is an update, ...

All in all it would allow us to make a more effective use of the network and gradually add new feature, without having to go through a whole release cycle.

What do you all think?

[davout]

+1 on replacing the client version in the handshake by a protocol version.
actually both could be sent, protocol version as number and client signature as string

[caveden]

I didn't know about this, that the protocol was "hard-wired" to the default client.
Your suggestion is really important. Client software and protocol should be separate.
It would help to have a protocol specification for this. The protocol version would only change when changes to the specification are made, not at every release of the client.

[davout]

Also some thought should be given on how the protocol should evolve.
What governance rules should be put in place ?

[jgarzik]

Protocol versioning is very "coarse."

Smarter protocols export lists of capabilities, as well as a "major" version number.  The major version number is left unchanged absent a major protocol rewrite.  The capabilities list is how clients notify each other of what features / bugs / etc. they support.

bitcoin has a bit of this already.  grep for 'nServices'

[RHorning]

There is a "sub-version" field that I believe it is a string rather than a numeric value that is transmitted with the version message.  This might be useful to be used in a fashion like the HTTP headers that identify the browser/operating system/version identification for a particular client for general identification purposes on the network and could be useful for an individual client which is playing around with experimental features to put that into the header.  A dozen extra bytes would not overwhelm the protocol in this case, particularly as it is transmitted only once per session and not per message.

In this case, the "version" field would indicate the protocol version (however that is defined) for the network as a whole that you are using that you guarantee to follow that protocol and any extensions are simply on top of that.  Any implementations ought to be 100% compatible with that protocol so far as anything transmitted from your node to any other node should follow that protocol unless it specifically identifies a fellow peer with a specific sub-version.

Also some thought should be given on how the protocol should evolve.
What governance rules should be put in place ?

I agree that the governance of the protocol documentation ought to be established in some fashion so far as "blessing it" as "official" when the time comes.  I created this specification based upon the source code but I know full well that it is missing some gaps, particularly right now on block and message acceptance rules and chain forking policies, as well as references to the hashing algorithms used.  My hope in creating that document was to eventually get it to some point that it would be "blessed" as the formal spec for clients like Cdecker is doing right now.

I really don't know what the official client is putting into that field right now, but it might solve the problem at least temporarily in regards to a specific re-implementation of the protocol.

[davout]

In my opinion, the governance of the protocol and its versioning are two sides of the same problem.
Once it is decided how the protocol is governed, versioning is pretty trivial.

I don't think a single individual should be in charge of that, I'd rather some not-for-profit organization be created and some procedures get agreed upon. I think it would be a great step towards making the protocol much more stable, predictable and transparent.

[Cdecker]

I'm quite happy that my proposal finally got some attention :-)

Protocol versioning is very "coarse."

Smarter protocols export lists of capabilities, as well as a "major" version number.  The major version number is left unchanged absent a major protocol rewrite.  The capabilities list is how clients notify each other of what features / bugs / etc. they support.

bitcoin has a bit of this already.  grep for 'nServices'

+1 for the capabilities.
The Protocol spec should have a minimal set of instructions and an overall message format, while all the rest should be built on top using capabilities or as they are called in BitTorrent protocol extensions. And I'd rather use an array of descriptive names instead of the bitfield bittorrent is using, but that's a detail.

[ribuck]

I don't think a single individual should be in charge

Agreed. But neither should a single organization be "in charge".

Anyone can make changes, "just" by getting 50% of the network to accept them. Right now, I don't think anyone except Satoshi, nor any organization, could achieve 50%.

[davout]

I don't think a single individual should be in charge

Agreed. But neither should a single organization be "in charge".

Anyone can make changes, "just" by getting 50% of the network to accept them. Right now, I don't think anyone except Satoshi, nor any organization, could achieve 50%.

That's a misconception.
Getting an incompatible change in the protocol to be used by 51% of the clients will just lead to two distinct block chains, not the 49% remaining "accepting" the change.

[ribuck]

Getting an incompatible change in the protocol to be used by 51% of the clients will just lead to two distinct block chains, not the 49% remaining "accepting" the change.

Well sure, but in practice if it's a worthwhile change the minority will see "the writing on the wall" and will switch to the majority.

I suppose there are still clients working on the block chain that had the overflow bug, but there aren't enough of them to matter to the majority.

[Cdecker]

Well the trick is to make the base protocol incredibly solid and easy. Just make it hard to get wrong. Then give people the ability to build their protocol extensions on top of that, all the while requiring that the initial minimal set of functionality must be maintained compatible, and then you'll have a good protocol that challenges people to improve it, but not to break it.

I think the capabilities extension is the way to go 

[davout]

I think it should focus on handling its own evolution elegantly.

[Cdecker]

I think it should focus on handling its own evolution elegantly.

Well making the network extensible is a way to allow evolution while keeping things backward compatible.