Bitcoin Forum

Hi,

I'm surprised that there is no complete documentation of current protocol specification at this time. I found something on wiki http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1 (http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1) and some fragments on pybitcoin pages https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol (https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol). But there are many blank pieces (unknown bytes in format etc).

I'm missing also some standard way for protocol proposals - everything is done here on the forum in some obscure process (at least obscure for me). I think it is because bitcoin community is still quite small, but we should define standard processes for that when we want to grow.

I would like to implement own library (in python) to support bitcoin protocol, but I realized that there is no easy way until I'm familiar with cpp and official client sources. There are also many 'hacks' like limited block size, which are related to protocol itself than on client implementation.

Also one dumb question - it is really needed to have binary protocol for our intentions? I think something more standardized should be more friendly to programmers in another languages (say java) and on another platform (I don't need to solve 32/64 bit problems on datatypes etc) when I defined protocol as (for example) gzipped xml (like other opensource data formats).

Currently there is big barrier to bring new clients with cool new features, but because I don't know lowlevel internals of bitcoin client, I don't know how to improve this situation for now :-(.

I, for one, would love to work with somebody or a group who is interested in helping put together a formal specification for Bitcoins.  I think it is something whose time has come and has been discussed on a number of other thread.  It isn't strictly necessary, but a formal specification rather than depending on the reference implementation (something also needed BTW!) is incredibly useful.

I've plowed through some of the source code in an attempt to try and find out how it all works, but it has proven to be a very tough nut to crack in terms of trying to "grok" the source code and figure out just what is happening.  I really could use a "guide" to at least help out in terms of navigating the source code and knowing what to look for.  The question is also how to organize the documentation effort for the protocol.

I think the wiki itself is sufficient in terms of the collaborative writing of the protocol document, but there may need to be some place for discussions beyond that in terms of organizing the content.  Any ideas on that would be appreciated.  Yes, there are a number of holes in the documentation and things that could be improved towards that end.

I'm all for a specification of some kind of the protocol. We gave it a first shot when trying to get a working Bitcoin Client to work with pybitcoin, but it turned out the protocol is both complex and hard to understand at times. Also a high level overview of how the different scenarios play out would be incredibly helpful.

We do need alternative clients for the Network, as proven by the huge interest in a native Android client, and for that to ever happen we need a specification of the underlying protocol.

I am not a professional programmer, or technical document writer.
How much work, (man hours), would it take to do a Formal Bitcoin Protocol Specification?
I would like to help organize to get this done.

I've seen specifications take years to get completed, so much of it depends upon how much detail that you put into the specification document. Some of them are pretty good, some are abysmal in terms of how close they are to reality and even in terms of ease of re-implementation based off of the specification.

The one nice thing about having a clean spec to work off of, besides the fact that you can create a new implementation without having to copy software, is that you can also argue about specific parts of the specification as a common framework in a fashion like you argue about legislation.  In fact, the process is very similar in terms of how laws are passed in a legislative environment.  The difference is that engineers, not lawyers have to be able to use whatever it is that you come up with.

One of the best real specifications that I've ever read was the PNG image file specification:

http://www.libpng.org/pub/png/spec/1.2/ (http://www.libpng.org/pub/png/spec/1.2/)

This is of course not similar to what we have here with Bitcoins, which is instead a networking protocol.  That would be more similar to the Internet RFCs which can be found here:

http://www.ietf.org/rfc.html (http://www.ietf.org/rfc.html)

If you want to read a real specification and at the same time see a bit of nerd humor, I'd suggest that you read this particular spec here:

http://tools.ietf.org/html/rfc1149 (http://tools.ietf.org/html/rfc1149)

I would expect this to take several man-months of effort at the least in terms of getting it prepared for one of these international bodies.  Preferably, I'd love to see this submitted as a formal network RFC once we get everything nailed down real good.  That would give Bitcoins some solid legitimacy and is a pretty high quality target to hit.

There are other standards bodies we need to work with as well, including the ECMA (European Computer Manufacturer's Association) and the W3C (Web standards committee).

Creating an IETF-ready specification would be a waste of time. The simple Bitcoin network used now is not what will be used in the future. Like Usenet or IP+BGP, different protocols will be used for generator-to-generator and generator-to-client connections. None of this is implemented yet, and no one knows how it will behave.

Bitcoin is incomplete. Getting the message format is easy compared to figuring out what something like this is supposed to mean:


Guess what doesn't exist in the code? "Product" and "Table" appear nowhere in this sense. Subscriptions are only mentioned a few more times. This "broadcast and subscription system", whatever it is, is some feature that currently only exists in Satoshi's head. There are several other examples of this kind of thing in the code.

Describing how the system currently works in detail would be very useful, but there is no chance that a "specification" will still be complete in a year from now.

I think an incremental approach is the only practical one here.

Specify first those parts that are (a) relatively easy to specify, and (b) for which it is particularly useful to have a specification. For everything else, just rely on the reference implementation for now.

For this incremental approach, the wiki is ideal. Some time in the future a more formal process might be considered, but for now it would just slow things down.

fwiw, artforz, on irc, claims to have some working python bitcoin node code... he may be a good one to hit up on help about understanding the protocol.

I suppose it is my time to waste getting such an effort done.  The issue here is that the protocol is the network.  I know that some software developers hate to be hamstrung with a formal spec that they must work from, under the presumption that perhaps too much planning in advance will straight jacket their ability to code.

My own personal experience is considerably different, where the time spent in planning and documenting ahead of time is time very much worth the effort and makes the job of any sort of coding considerably easier.  It also tends to make for much, much cleaner software that is easier to maintain, easier to extend, and as an indirect result less dependent upon a single person to make all of the decisions.

This is essentially the difference between a mere computer programmer and a real software engineer too.  A programmer is somebody who pulls out the compiler/editor to figure out how to get a project working, where a software engineer starts with a word processor.  That coding needs to happen eventually is true, but setting out a roadmap for how the project is to be developed is usually a good start.

We could debate and discuss various models used for software development here too, but by its very nature Bitcoins is something that deserves to have some significant effort at bringing eyeballs into its development, and the more eyeballs that we have the better that the network will behave.  At the moment it is just a minor amount of money involved, but it soon may represent some significant economic activity.  To me, this deserves some solid software engineering principles which includes documenting the effort going on here.

I'm not expecting anybody to necessarily participate in this effort, but it seems strange that there is resistance to even start such an effort and potentially thwarting such an effort.  If Bitcoins is incomplete, let's make it more complete.  If there are gaps, those gaps need to be filled.

My personal experience is that strict specification creates software that is extremely good at doing what the specification wants, but for all its elegance, it's not always what the user wants or needs.  We haven't fully figured out the details of how Bitcoin is supposed to behave yet (see discussion on transaction fee policy). We are finding out as we go along.  Nothing against specification, but isn't it a bit early right now?

As a veteran of the premature standardization trenches (I wrote most of the ISO/IEC 14772-1 "VRML97" specification before I changed my last name from "Bell" to "Andresen")... I agree with foreverdamaged.  It is too early to try to create a formal specification.

However, it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.

I've also seen software written without any sort of specification or for that matter any sort of planning at all turn out horribly too.  There obviously needs to be some sort of balance to the whole thing, and I'm not against people trying experiments out before formalizing the behavior.

Indeed, it might be useful to simply state that any sort of desired inclusion into a specification ought to be implemented in code first to see if the thing works at all before it is added to a formal specification for Bitcoins.  I wouldn't mind supporting such a notion too.  A semi-formal process or even informal process for writing the spec documents would especially be useful at the moment.

I remember Steve Wozniak complaining about a full day meeting he was involved with in terms of arguing about the placement of semi-colons in the implementation of Pascal on the Apple II computers.  He thought that effort was a wasted day in his life and certainly I'd like to avoid that kind of minutiae debates.  A wiki goes a long way to fix that kind of argument too.

Well, there is no way how to implement unofficial clients for many users/programmers (like me), because they are not enough skilled in C++ and reverse engineering. But I'm capable to write alternate client with at least basic specification how whole thing works. Unfortunatelly because I'm not capable to write own client, I'm also not capable to help anybody with specs. At this time, I'm dependent to somebody else who starts specification process.

I'm absolutely not talking about any formal standard, wiki should help a lot in this stage.

I'm still advocating for a few small changes to the protocol now before it becomes too much of a PITA to change later. (I mentioned this on the forum about 10 months ago.)

1. The handshake should be reversed. An open Bitcoin port shouldn't identify what it is. The connecting client should initiate the handshake. This improves privacy a lot. Think nmap. Think spies. Think any tool that can fingerprint (I use telnet) a service by simply connecting to an open port.

2. The connections should be SSL. We should try to emulate FF connecting to Apache or DPI will eventually become our worst enemy. We should take what the Tor developers learned the hard way into account early on.

3. The Bitcoin client should choose a random unused port to listen on when it is first installed. For a ISP or even a nation to block port 8333 is quite easy and is becoming easier all the time.

4. UPnP is a must. The Bitcoin client should automatically open up whatever port it decided on with UPnP. This will relive a lot of NAT problems and will extend the P2P network a lot better.

:)

#1 & #3 seem contradictory.  How can a connecting client initiate a handshake on an unknown port without spamming the target?  Wouldn't it still show up on a portscan that was specificly looking for a running bitcoin client?

The port isn't unknown. The IP/port are published to the network once the client has seeded successfully. Every other node writes that to their addr.dat.

As far as I can tell the addr.dat contains IP/port already. :)

I see.  Well, I'm not a programmer, yet I cannot see the value in obscuring, encrypting or otherwise trying to hide the port.  The port can be blocked for those who wish to hide their client, and still work; while the data in transit is only openly coded transactions and blocks.  The only risks to the port being open is a sign to potential crackers that there is a running client (and therefore a wallet.dat) on the machine.  Just close the port if that is a concern.

Agreed.
1. This would counter simple port scan/identification attacks by script kiddies. Bitcoin (or any protocol) should not announce what it is. Let the connecter speak first. Just break the connection if it is not what it expected. It will not be impossible to identify the service, just a lot harder.

2. I'm all for this. SSL support is always a good addition. It would at least provide a level of security. Potential issue (specific to SSL) is key/certificate management.

3. Why not. The range in which to randomize should be configurable though, so that firewalls that only leave through a certain range can be used (same as with bittorrent)

4. Yep, that would help with a lot of home routers.

I've thrown some additional information onto the wiki already, at least enough to start.  I've found at least some of the relevant sections in the source code for Bitcoins and will try to get some more information put on there, as well as some threads to look through as well.  More theory certainly should be put together, and perhaps an evaluation of some of the decisions already made... which can certainly be useful.

This is the most important thing to happen, IMHO, doing so would dramatically lower the barriers of entry of creating 2nd generation bitcoin clients independent of the reference implementation.

So if it would take many man_months of work to develop a formal specification, then how long would it take to develop a 'good enough' informal specification?

I think this is the wrong way to look at it, particularly given the mostly volunteer nature involve with the operation of Bitcoins at the moment.  There have been several attempts to start the documentation process, and the important thing to do now is to build upon those efforts and get what information anybody knows down into some usable form.  Documentation of Bitcoins all around is sort of weak, and even if you aren't a programmer it would still be useful to at least try to explain the concepts of Bitcoins in some way that perhaps even non geeks can understand them.

There is also a whole bunch of useful information which is now getting buried in these forum threats, so indexing these discussions would also be helpful in some way, although for the specific details of the operation of Bitcoins ultimately falls upon the source code of the reference implementation written by Satoshi.

Like trying to eat an elephant, it takes time and patience where you can only take one bite at a time.  If you can read the source code and understand even a portion of it, get that knowledge recorded or simplified if you can.  At that point we can debate the merit or lack there of for specific decisions in the current design.  My experience is also that once something is established and not challenged, that it tends to become something permanent in nature even on an "open source" project.  Right now, most people don't even know what to start challenging because the details are buried in code.  I'm hoping that a "good enough" documentation effort can at least bring some of those issues to the front.

For those familiar with the network level protocols, what is the difference between getblocks and getdata?  Both seem to be a list of hashes representing blocks which need to be sent to the requesting node.

One difference I can see is with the "getblocks" command/packet type will request a range of blocks, while getdata requests individual blocks.  Is this the only difference or is there something more significant that I'm missing here?  I'm trying to figure out when this particular packet type might be used instead or why there seems to be a duplication of block request methods seemingly doing the same thing.

Have you seen this?
http://www.bitcoin.org/wiki/doku.php?id=network

Getdata requests a specific block or transaction by hash. You generally only send a getdata after you receive an inv listing a block/tx that you don't already have. Getblocks requests an inv containing the hashes of all blocks in a range (max 500 at a time). It's used for initial block download and re-syncing after some downtime.

Getblocks (client) -> inv (server) -> getdata (client) -> block (server)
Send one getblocks, get an inv with 500 entries, send 500 getdata messages, receive 500 block messages. This sounds inefficient, but the download is actually very fast (it's the verification that eats up most of the "download" time).

As a matter of fact, I missed that page.  Thank you so much for putting the effort into writing that explanation.  It really does make a difference.

As a side note, we really need to put together some menus or something that links deep into the wiki, or at least put references to it on other pages.

I've been trying to collect content related to the protocol for some time, so every little bit helps.  Again, thanks!

Let's try to keep this thread alive and unbury it with new findings while we go along. One fact that I stumbled over (for several hours today, hurting myself as I went) is that all numbers in the protocol are not encoded in network byteorder, but rely on little endian. I guess that would be pretty important if we are to create a documentation.

I think there are two ways to look at the protocol, a high level one, where everything is expressed in nice words and comparisons, and another dearly needed one that details the actual information and format on the wire.

One nice detail to add is for example that each messahe starts with a 4-byte magic .

Also in the original design a lot of attention went into how the size of a message is encoded:

But message types are simply encoded with a padded 16 byte string. So I'm starting to wonder about the design choices. Why make the size field optimized when the other part of the message is large always? No offense intended, but this kind of things just make it hard to implement.

Oh and when using Java you might pay close attention on how to read unsigned data types (again, something I had to bang my head against before realizing my error  ::))

I hope you've looked at the "draft spec" (http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1) that I've been writing where I've put some of this information in, but your input is very much appreciated.  I forgot to mention the byte order as it is a huge detail, but something I've come to expect from projects like this.  About the only thing that is recorded in "network byte order" that I'm aware of at the moment is timestamp structure, and that is in part because the structure is defined in a library not written by Satoshi.  Nothing personal against Satoshi here either, as all that is going on is that he isn't re-ordering the bytes as the vast majority of the clients are using Intel architecture on their computers.  It simply makes the software a whole lot easier to write so far as transmitting the data.

This is also a pet peeve of mine as it opens up the whole little endian vs. big endian debate.  This is also where Intel going against the grain on this issue has sort of messed things up and a tale of how architecture decisions made decades ago continue to come back and impact everybody in sometimes significant ways.  For the most part, other than as a potential bug when you are trying to read/write data on a shared data format used by multiple computer systems (aka on a CD-ROM or via the internet) it rarely is even a problem.

At the moment I'm trying to wrap my head around the transaction and block formats in the network data sharing protocol.  A whole bunch is buried in there and isn't very well documented in terms of what it is doing.  If you could help in that regard, let me know too!

Going over the transaction specs, I noticed a "lock time" attribute on each transaction.  With this, there is apparently some sort of protocol envisioned for being able to push transactions to various nodes but also require them to be included at some future block instead of being processed immediately.  In other words, it is a request to miners to not include the transaction "no earlier than" some particular block number.  In addition, there is the ability for details about the transaction to be modified subsequent to its inclusion into a block.

My question is in a couple parts:  Is this in the roadmap for getting implemented in the future or is this simply an idea that hasn't really been completely thought through?  What kind of security issues are there in terms of a 3rd party "changing" the transaction information and simply updating to a new transaction version?  Or is this a "no later than" type of notification where the transaction expires after a certain block number has been created?

It is an interesting feature to Bitcoins if it could be pulled off.  Apparently most miners are not paying attention to this attribute as well, and it may be something to reconsider.

A transaction can't be included in a block if its lock time is in the future. Even now blocks breaking this rule will be rejected.

The feature is designed to work with in-memory transaction replacement, which is currently disabled (it was enabled in older versions):
IsNewerThan() checks that the input's sequence number is lower than the other version. Lower sequence=newer.

This disabled feature is not network-enforced in any way, so it could be enabled at any time.

You can't replace a transaction unless you can sign it. So it should be safe. It might be unsafe if you're using inputs that can be redeemed by more than one person: the other person could make your transaction invalid (but not steal your other inputs).

It was probably disabled because it makes accepting transactions with 0 confirmations really unsafe. It could be safely re-enabled if transactions were only replaceable if they actually specify a non-zero lock time, and this was marked in the UI.

Sometimes I cant resist but question satoshis choices: a UINT64 size field? It's incredibly hard to implement in Java ( well not really BigInteger helps) and do we really need messages larger than 4GB (4 bytes)? UINT 64 would allow for messages of 18.45 Exabytes. That's more than all the world movies put together.

I think I'll simply drop messages requiring UINT64 sizes.

Are you asking about the message header "size" field, indicating how large the message packet itself is?  I thought that was just a simple 4-byte int value followed by a 4-byte checksum.  That format information comes from main.cpp and also implemented in net.h:


On the whole, most messages are quite small, with the exception of the transaction messages themselves which can grow to sizes on the order thousands of bytes (10k is the limit for a single script per input or output).  In theory some of the other messages could get fairly large, but still on that order of magnitude peaking at about 50k in extreme situations.  I can see where a shortint is perhaps too small and that a complex transaction with dozens of inputs and outputs might need more than 64k bytes, but you are correct that there is no need to get past the gigabyte range for message sizes.

*Edit* I also found this little snippet of code relevant to this discussion:


(from serialize.h)

This is the current maximum size for any single message on the network at the moment, as something larger than this is simply going to be rejected.

I wonder when they changed that one. It was incredibly hard to get the size out of the message, and we had to switch between size length according to the first byte.

What exactly is this used for then: http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1#variable_sized_data ?

The only place I currently see that being used is in scripts.  Thanks to Theymos the ideas behind scripting are less opaque but it still is pretty arcane for those who really want to get into the gritty details of Bitcoin.

In theory it could be put into the protocol eventually as a way to save bandwidth, but so far I haven't seen it used in that way.  If that was a goal, it would seem that there would be some other concepts in place that would facilitate data compression more effectively and perhaps even be more extensible too.

The best suggestion I've seen for saving bandwidth is that completed blocks should not contain full transaction bodies but only the Merkle tree. Nodes that somehow missed the original tx broadcast for some nodes in the tree would then just getdata peers for them in the usual manner.

This isn't an issue today but it would make running nodes cheaper if BitCoin goes to extreme scales, like thousands of transactions a second.

According to Gavin's https://github.com/gavinandresen/bitcointools/blob/master/NOTES.txt (https://github.com/gavinandresen/bitcointools/blob/master/NOTES.txt), serialization of any vector object gets preceded by a count of the number of elements in the vector, in the variable-length 1/3/5/9 byte format. I added this count field to the new wiki, e.g. to addr messages. Also, block messages contain a vector of their transactions, so that part is also preceded by a variable-length count.

7 years in production and no official specification. it's a shame!