I've thought about this and I'm surprised I've not seen (or can find) very much discussion eluding to this. At the moment, for anyone that wants to understand the bitcoin protocol, they would be able to use the bitcoin wiki somewhat, as well as forums and other websites but ultimately have to look at the source code of bitcoin implementations, or rely on the knowledge of other people.

It would be very useful and wise, in my opinion, if there was a formalised document describing the protocol to every detail, but in a way that is easy for anyone to follow. It would be a document that would be used as a reference for developers and would reflect all of the agreed (In majority use/Majority mining power) protocol features. The protocol standards document would then be amended as the protocol is modified. A separated set of documents could describe other features which are not core to the protocol such as wallet formats or whatever.

I had a hunch this would be something the Bitcoin Foundation was set up for, but it seems not. Do other people think this would be very useful to work upon? Otherwise the information will continue to be disorganised and a nightmare to piece together.

The paper is a design overview, not a specification. It presents the argument that something like bitcoin can work at all, but doesn't tell you the details of building something compatible with it.

Well by formalised I meant, but together professionally into a specification document (as opposed to now). I din't mean much more than that. It doesn't necessarily have to go by any usual conventions, if that means the document can be both easy to follow and fully detailed.

Isn't most of the protocol right here?

https://en.bitcoin.it/wiki/Protocol_specification

Flying 3D sharks from the past!

That sounds like a statement that might apply at a particular moment in time.  It isn't an argument against specifying the protocol separately from a particular implementation in future, though it is obviously something to take account of while writing the specification.

There was probably a time in 1990 when the reality of how the web worked meant that the CERN httpd implementation was the protocol definition for HTTP (I don't know for sure, but this a pattern seen with many protocols and other interface elements: original PGP implementation -> OpenPGP protocol; this was more or less the way ssh went as well regarding initial implementation to decent public protocol definition too).  That doesn't mean it was a bad idea to create standardisation processes once the technology took off and there was interest in multiple compatible implementations, and in managing changes / extensions.


That's good stuff too -- but is certainly not an argument against trying to get the protocol specified in some sort of, for example, IETF-RFC-like document.  Specifications and test suites go together really well, but are not alternatives for one another.  Test suites are sometimes good for clarifying intent where a spec's ambiguous, and as you say above they're also great aiding implementors with completeness and correctness.

Bitcoin needs a protocol spec for the technology to mature.  One doesn't want to do it while the design's in flux, but Bitcoin's past that stage now.  Any incompatible design changes would be brought about as a new crypto currency rather than as changes to existing Bitcoin.

It is frankly pretty worrying to see Gavin and Mike be so dismissive of MatthewLM's suggestion.  Hopefully some others involved have more wisdom and experience in protocol engineering at Internet scale.

I agree with Gavin's point of view.  Unit Tests are the final and most complete form of behavior specification and the implementation is the final and most complete form of design.  It's best when both are expressed in languages free of a lot of syntactic noise.  C++ is far from ideal in that regard, but you live with compromises born out of practicality.  In languages that are less encumbered by syntactic noise, this perspective is much more readily apparent.  The tests and the implementation are so easily comprehensible that other documentation isn't worth the effort to maintain (and can even be a detriment). 

Check out OMeta and some of the papers at vpri.org if you're really into this sort of thing...with OMeta, they managed to create a system that could almost directly execute TCP/IP from the RFCs.  It was a complete TCP/IP implementation in under 200 lines of code (including the parser specification for the RFC ascii art).  See this summary about it: http://www.moserware.com/2008/04/towards-moores-law-software-part-3-of-3.html  ...to me, this is a proof point that code really should be regarded as self documenting (with little more than annotations to accompany it)...if it's too challenging for people to easily comprehend, it points to a shortcoming of the language, not of the concept that the code is the documentation.

It bothers me that this topic keeps coming up. The fact that Bitcoin is different to other technologies isn't intuitive but by the time you're writing an actual implementation, it should be obvious. Maybe it's worth reading the thread with grau about his reimplementation also?

The fact is that re-implementing Bitcoin exposes not only you, but all participants, to a class of "chain splitting" bugs that don't really exist in other network technologies, or at least are nowhere near as severe. The browser wars of the 90s were bad, but at least developers could check which browser the user ran and adapt to it on the fly. The Bitcoin equivalent is dramatically worse.

When you reimplement Bitcoin, it's not enough to build things as you think they should work. You have to implement them exactly as Satoshi did, including all his bugs. And because some parts of the protocol are directly exposed to underlying libraries like OpenSSL, you have to match their behaviour exactly as well, including all their bugs. Failure to do so can lead to people losing money.

At some point, if you realize you have to match the behaviour of another codebase exactly, down to the tiniest detail, you realize that the only precise enough specification for that is the source code. Which means if you can't read C++ fluently you can't reimplement Bitcoin, yes, but who cares? If you can't keep up, don't step up.

Having detailed protocol documentation is something I'd agree with in any other project except this one. In this one, it will simply mislead people into thinking they can reimplement Bitcoin. Unless they are willing to make absolutely massive effort and take serious risks, they can't.

Note that SPV nodes are much less risky. But Matthew isn't implementing an SPV client.

The unit tests are just there to ensure things are working as expected. They aren't designed to provide a reference to how things are supposed to work.


The English language, combined with diagrams, tables etc. are designed for humans to understand and are thus the ideal format, as opposed to C++. And the Satoshi client is not very friendly to human eyes, so indeed another implementation could make it much easier to understand, but why waste time on doing that if you could write a human-friendly specification?


For some reason I couldn't find very much when I searched for it, except vague references.


All the more reason to make the protocol as clear and easy to understand as possible.


Things have gotten better when things have become more standardised.


Once again, more reason to have clear documentation. Though you do not need to do everything the same way as the Satoshi client. You only need to conform to the protocol requirements.


Sorry? People can re-implement bitcoin. You know that, obviously.


I'm implementing code which can be used for full validation or headers only validation. My plans for a client will includes a mixture between the two, offering the best of both worlds. The block-chain will thus be checked by headers-only validation against full validation done by a server.

No, there are cases where other people can suffer from chain splitting bugs too. Let's say you're a high volume merchant or payment processor that runs an alternative implementation. I can make a transaction which it believes is invalid but the rest of the network believes is valid, for whatever reason. Once that gets included into a block, your business will grind to a halt because it'll split off onto a chain that no longer gets extended, or only gets extended very slowly, meaning you can't process payments anymore until the problem is noticed and you find and fix the conformance bug.

If this causes you to lose X coins per hour of business, then I can try to anonymously extort you for a bit less than X by claiming I know of such a bug in your software. It's very hard to prove it's not there. You'd have to have a lot of confidence in the robustness of the testing of your implementation.

What about if you accept invalid transactions? If you're providing a good or service in return for unconfirmed transactions then this obviously can undermine your risk model because you'll receive a transaction that you believe is valid, not a double spend, you don't see any double spend alerts or conflicting transactions - but it'll never confirm and I can still spend the money. I don't have to wait and mine a block any more like I would if doing a Finney attack, so it's much cheaper.


Such a document would end up being nearly as long as the source code, and not much easier to read. I'm all for adding more detailed comments to the source, though.


Unfortunately, what you've got is the latter and it's not really easy to fix. We keep discovering new odd edge cases where what the software does, isn't what you'd actually expect given a description of how it's meant to work.


I think the Satoshi client is quite straightforward to read, for the most part. A few parts are somewhat inscrutable  because they're written very tightly, but unfortunately there is no substitute for just puzzling it out - if you write a description of what you think the code does it may not match reality. We have seen this demonstrated several times, like with the merkle tree calculation. What people thought it did, wasn't quite what it actually did. If you simply duplicated Satoshis algorithm, you would duplicate his bug too so no chain-split attack would have been present. If you re-implemented it based on an English description, you'd have introduced an exploit.


I've implemented the SPV mode (not re-implemented, as for a long time there was no other implementation of this). Matt Corallo went ahead and has extended my work to do full validation. He's done a TON of testing, very in depth testing, despite that I would be very concerned if I heard that a big mining pool or BitPay or whoever was using it. At least in it's current state. It's not clear to me how much work would be required until I felt comfortable with high value operations using bitcoinj in full mode, and the documentation when 0.7 is released will make that clear.