Bitcoin Protocol Specification

[slush]

Hi,

I'm surprised that there is no complete documentation of current protocol specification at this time. I found something on wiki http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1 and some fragments on pybitcoin pages https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol. But there are many blank pieces (unknown bytes in format etc).

I'm missing also some standard way for protocol proposals - everything is done here on the forum in some obscure process (at least obscure for me). I think it is because bitcoin community is still quite small, but we should define standard processes for that when we want to grow.

I would like to implement own library (in python) to support bitcoin protocol, but I realized that there is no easy way until I'm familiar with cpp and official client sources. There are also many 'hacks' like limited block size, which are related to protocol itself than on client implementation.

Also one dumb question - it is really needed to have binary protocol for our intentions? I think something more standardized should be more friendly to programmers in another languages (say java) and on another platform (I don't need to solve 32/64 bit problems on datatypes etc) when I defined protocol as (for example) gzipped xml (like other opensource data formats).

Currently there is big barrier to bring new clients with cool new features, but because I don't know lowlevel internals of bitcoin client, I don't know how to improve this situation for now :-(.

[1713909095]

1713909095

[1713909095]

1713909095

[1713909095]

1713909095

[1713909095]

1713909095

[1713909095]

1713909095

[1713909095]

1713909095

[RHorning]

Hi,

I'm surprised that there is no complete documentation of current protocol specification at this time. I found something on wiki http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1 and some fragments on pybitcoin pages https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol. But there are many blank pieces (unknown bytes in format etc).

I, for one, would love to work with somebody or a group who is interested in helping put together a formal specification for Bitcoins.  I think it is something whose time has come and has been discussed on a number of other thread.  It isn't strictly necessary, but a formal specification rather than depending on the reference implementation (something also needed BTW!) is incredibly useful.

I've plowed through some of the source code in an attempt to try and find out how it all works, but it has proven to be a very tough nut to crack in terms of trying to "grok" the source code and figure out just what is happening.  I really could use a "guide" to at least help out in terms of navigating the source code and knowing what to look for.  The question is also how to organize the documentation effort for the protocol.

I think the wiki itself is sufficient in terms of the collaborative writing of the protocol document, but there may need to be some place for discussions beyond that in terms of organizing the content.  Any ideas on that would be appreciated.  Yes, there are a number of holes in the documentation and things that could be improved towards that end.

[Cdecker]

I'm all for a specification of some kind of the protocol. We gave it a first shot when trying to get a working Bitcoin Client to work with pybitcoin, but it turned out the protocol is both complex and hard to understand at times. Also a high level overview of how the different scenarios play out would be incredibly helpful.

We do need alternative clients for the Network, as proven by the huge interest in a native Android client, and for that to ever happen we need a specification of the underlying protocol.

[da2ce7]

I am not a professional programmer, or technical document writer.
How much work, (man hours), would it take to do a Formal Bitcoin Protocol Specification?
I would like to help organize to get this done.

[RHorning]

I am not a professional programmer, or technical document writer.
How much work, (man hours), would it take to do a Formal Bitcoin Protocol Specification?
I would like to help organize to get this done.

I've seen specifications take years to get completed, so much of it depends upon how much detail that you put into the specification document. Some of them are pretty good, some are abysmal in terms of how close they are to reality and even in terms of ease of re-implementation based off of the specification.

The one nice thing about having a clean spec to work off of, besides the fact that you can create a new implementation without having to copy software, is that you can also argue about specific parts of the specification as a common framework in a fashion like you argue about legislation.  In fact, the process is very similar in terms of how laws are passed in a legislative environment.  The difference is that engineers, not lawyers have to be able to use whatever it is that you come up with.

One of the best real specifications that I've ever read was the PNG image file specification:

http://www.libpng.org/pub/png/spec/1.2/

This is of course not similar to what we have here with Bitcoins, which is instead a networking protocol.  That would be more similar to the Internet RFCs which can be found here:

http://www.ietf.org/rfc.html

If you want to read a real specification and at the same time see a bit of nerd humor, I'd suggest that you read this particular spec here:

http://tools.ietf.org/html/rfc1149

I would expect this to take several man-months of effort at the least in terms of getting it prepared for one of these international bodies.  Preferably, I'd love to see this submitted as a formal network RFC once we get everything nailed down real good.  That would give Bitcoins some solid legitimacy and is a pretty high quality target to hit.

There are other standards bodies we need to work with as well, including the ECMA (European Computer Manufacturer's Association) and the W3C (Web standards committee).

[theymos]

Creating an IETF-ready specification would be a waste of time. The simple Bitcoin network used now is not what will be used in the future. Like Usenet or IP+BGP, different protocols will be used for generator-to-generator and generator-to-client connections. None of this is implemented yet, and no one knows how it will behave.

Bitcoin is incomplete. Getting the message format is easy compared to figuring out what something like this is supposed to mean:

Code:

// Subscription methods for the broadcast and subscription system.
// Channel numbers are message numbers, i.e. MSG_TABLE and MSG_PRODUCT.
//
// The subscription system uses a meet-in-the-middle strategy.
// With 100,000 nodes, if senders broadcast to 1000 random nodes and receivers
// subscribe to 1000 random nodes, 99.995% (1 - 0.99^1000) of messages will get through.

Guess what doesn't exist in the code? "Product" and "Table" appear nowhere in this sense. Subscriptions are only mentioned a few more times. This "broadcast and subscription system", whatever it is, is some feature that currently only exists in Satoshi's head. There are several other examples of this kind of thing in the code.

Describing how the system currently works in detail would be very useful, but there is no chance that a "specification" will still be complete in a year from now.

[ribuck]

I think an incremental approach is the only practical one here.

Specify first those parts that are (a) relatively easy to specify, and (b) for which it is particularly useful to have a specification. For everything else, just rely on the reference implementation for now.

For this incremental approach, the wiki is ideal. Some time in the future a more formal process might be considered, but for now it would just slow things down.

[nanotube]

fwiw, artforz, on irc, claims to have some working python bitcoin node code... he may be a good one to hit up on help about understanding the protocol.

[RHorning]

Creating an IETF-ready specification would be a waste of time. The simple Bitcoin network used now is not what will be used in the future. Like Usenet or IP+BGP, different protocols will be used for generator-to-generator and generator-to-client connections. None of this is implemented yet, and no one knows how it will behave.

Bitcoin is incomplete. Getting the message format is easy compared to figuring out what something like this is supposed to mean:

I suppose it is my time to waste getting such an effort done.  The issue here is that the protocol is the network.  I know that some software developers hate to be hamstrung with a formal spec that they must work from, under the presumption that perhaps too much planning in advance will straight jacket their ability to code.

My own personal experience is considerably different, where the time spent in planning and documenting ahead of time is time very much worth the effort and makes the job of any sort of coding considerably easier.  It also tends to make for much, much cleaner software that is easier to maintain, easier to extend, and as an indirect result less dependent upon a single person to make all of the decisions.

This is essentially the difference between a mere computer programmer and a real software engineer too.  A programmer is somebody who pulls out the compiler/editor to figure out how to get a project working, where a software engineer starts with a word processor.  That coding needs to happen eventually is true, but setting out a roadmap for how the project is to be developed is usually a good start.

We could debate and discuss various models used for software development here too, but by its very nature Bitcoins is something that deserves to have some significant effort at bringing eyeballs into its development, and the more eyeballs that we have the better that the network will behave.  At the moment it is just a minor amount of money involved, but it soon may represent some significant economic activity.  To me, this deserves some solid software engineering principles which includes documenting the effort going on here.

I'm not expecting anybody to necessarily participate in this effort, but it seems strange that there is resistance to even start such an effort and potentially thwarting such an effort.  If Bitcoins is incomplete, let's make it more complete.  If there are gaps, those gaps need to be filled.

[Timo Y]

My personal experience is that strict specification creates software that is extremely good at doing what the specification wants, but for all its elegance, it's not always what the user wants or needs.  We haven't fully figured out the details of how Bitcoin is supposed to behave yet (see discussion on transaction fee policy). We are finding out as we go along.  Nothing against specification, but isn't it a bit early right now?

[Gavin Andresen]

As a veteran of the premature standardization trenches (I wrote most of the ISO/IEC 14772-1 "VRML97" specification before I changed my last name from "Bell" to "Andresen")... I agree with foreverdamaged.  It is too early to try to create a formal specification.

However, it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.

[RHorning]

As a veteran of the premature standardization trenches (I wrote most of the ISO/IEC 14772-1 "VRML97" specification before I changed my last name from "Bell" to "Andresen")... I agree with foreverdamaged.  It is too early to try to create a formal specification.

However, it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.

I've also seen software written without any sort of specification or for that matter any sort of planning at all turn out horribly too.  There obviously needs to be some sort of balance to the whole thing, and I'm not against people trying experiments out before formalizing the behavior.

Indeed, it might be useful to simply state that any sort of desired inclusion into a specification ought to be implemented in code first to see if the thing works at all before it is added to a formal specification for Bitcoins.  I wouldn't mind supporting such a notion too.  A semi-formal process or even informal process for writing the spec documents would especially be useful at the moment.

I remember Steve Wozniak complaining about a full day meeting he was involved with in terms of arguing about the placement of semi-colons in the implementation of Pascal on the Apple II computers.  He thought that effort was a wasted day in his life and certainly I'd like to avoid that kind of minutiae debates.  A wiki goes a long way to fix that kind of argument too.

[slush]

it's not always what the user wants or needs

Well, there is no way how to implement unofficial clients for many users/programmers (like me), because they are not enough skilled in C++ and reverse engineering. But I'm capable to write alternate client with at least basic specification how whole thing works. Unfortunatelly because I'm not capable to write own client, I'm also not capable to help anybody with specs. At this time, I'm dependent to somebody else who starts specification process.

I'm absolutely not talking about any formal standard, wiki should help a lot in this stage.

[The Madhatter]

I'm still advocating for a few small changes to the protocol now before it becomes too much of a PITA to change later. (I mentioned this on the forum about 10 months ago.)

1. The handshake should be reversed. An open Bitcoin port shouldn't identify what it is. The connecting client should initiate the handshake. This improves privacy a lot. Think nmap. Think spies. Think any tool that can fingerprint (I use telnet) a service by simply connecting to an open port.

2. The connections should be SSL. We should try to emulate FF connecting to Apache or DPI will eventually become our worst enemy. We should take what the Tor developers learned the hard way into account early on.

3. The Bitcoin client should choose a random unused port to listen on when it is first installed. For a ISP or even a nation to block port 8333 is quite easy and is becoming easier all the time.

4. UPnP is a must. The Bitcoin client should automatically open up whatever port it decided on with UPnP. This will relive a lot of NAT problems and will extend the P2P network a lot better.

[MoonShadow]

#1 & #3 seem contradictory.  How can a connecting client initiate a handshake on an unknown port without spamming the target?  Wouldn't it still show up on a portscan that was specificly looking for a running bitcoin client?

[The Madhatter]

The port isn't unknown. The IP/port are published to the network once the client has seeded successfully. Every other node writes that to their addr.dat.

As far as I can tell the addr.dat contains IP/port already.

[MoonShadow]

The port isn't unknown. The IP/port are published to the network once the client has seeded successfully. Every other node writes that to their addr.dat.

As far as I can tell the addr.dat contains IP/port already.

I see.  Well, I'm not a programmer, yet I cannot see the value in obscuring, encrypting or otherwise trying to hide the port.  The port can be blocked for those who wish to hide their client, and still work; while the data in transit is only openly coded transactions and blocks.  The only risks to the port being open is a sign to potential crackers that there is a running client (and therefore a wallet.dat) on the machine.  Just close the port if that is a concern.

[wumpus]

I'm still advocating for a few small changes to the protocol now before it becomes too much of a PITA to change later. (I mentioned this on the forum about 10 months ago.)

1. The handshake should be reversed. An open Bitcoin port shouldn't identify what it is. The connecting client should initiate the handshake. This improves privacy a lot. Think nmap. Think spies. Think any tool that can fingerprint (I use telnet) a service by simply connecting to an open port.
On the other hand, this would be a lot of trouble for existing clients. A more breaking protocol change is hard to think of.

2. The connections should be SSL. We should try to emulate FF connecting to Apache or DPI will eventually become our worst enemy. We should take what the Tor developers learned the hard way into account early on.

3. The Bitcoin client should choose a random unused port to listen on when it is first installed. For a ISP or even a nation to block port 8333 is quite easy and is becoming easier all the time.

4. UPnP is a must. The Bitcoin client should automatically open up whatever port it decided on with UPnP. This will relive a lot of NAT problems and will extend the P2P network a lot better.

Agreed.
1. This would counter simple port scan/identification attacks by script kiddies. Bitcoin (or any protocol) should not announce what it is. Let the connecter speak first. Just break the connection if it is not what it expected. It will not be impossible to identify the service, just a lot harder.

2. I'm all for this. SSL support is always a good addition. It would at least provide a level of security. Potential issue (specific to SSL) is key/certificate management.

3. Why not. The range in which to randomize should be configurable though, so that firewalls that only leave through a certain range can be used (same as with bittorrent)

4. Yep, that would help with a lot of home routers.

[RHorning]

it's not always what the user wants or needs

Well, there is no way how to implement unofficial clients for many users/programmers (like me), because they are not enough skilled in C++ and reverse engineering. But I'm capable to write alternate client with at least basic specification how whole thing works. Unfortunatelly because I'm not capable to write own client, I'm also not capable to help anybody with specs. At this time, I'm dependent to somebody else who starts specification process.

I'm absolutely not talking about any formal standard, wiki should help a lot in this stage.

I've thrown some additional information onto the wiki already, at least enough to start.  I've found at least some of the relevant sections in the source code for Bitcoins and will try to get some more information put on there, as well as some threads to look through as well.  More theory certainly should be put together, and perhaps an evaluation of some of the decisions already made... which can certainly be useful.

[da2ce7]

[... , I] it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.

This is the most important thing to happen, IMHO, doing so would dramatically lower the barriers of entry of creating 2nd generation bitcoin clients independent of the reference implementation.

So if it would take many man_months of work to develop a formal specification, then how long would it take to develop a 'good enough' informal specification?