Formalised Bitcoin Protocol Standard

[caveden]

So, how many more major developers need to show up here to explain that a spec would be 1) not very useful, and/or 2) actually dangerous, before you believe it?

Well, you must admit that they have a little bias: they can all read C++ fluently. Such a documentation would be useful precisely for those who can't. But Armory guy makes a good point:

Write all the "documentation" you want, and put as many comments into the code as you want.  But do not use the word "specification" because nothing except the code can qualify for it.

Maybe simply not calling it "specification" would be enough to avoid most problems raised here.

[1714200362]

1714200362

[CIYAM]

Well, you must admit that they have a little bias: they can all read C++ fluently. Such a documentation would be useful precisely for those who can't.

I read C++ fluently (and have been doing hardcore C++ development since the mid 90's) and although I do agree that "what we have is a spec written in C++" I do not think we really want this to be the case in the future (even C++ itself has a "spec" - ISO/IEC 14882:2003 currently).

Transitioning from "C with Classes" to C++ took many years and it took many more for vendors to get "up to spec" (Microsoft being one of the main lagers until Herb Sutter joined them).

The same "slow" process will need to take place with Bitcoin as I don't think having "source code" as the "specification" (written in any programming language) is the correct (or desirable) way that things should be done (however I do appreciate that until the "source code" has any quirks that the don't make sense to put in the "specification" removed over time then the "specification" will also have to include them).

Algorithms are not described by "source code" and neither are any of the major internet protocols or programming languages (that aren't privately owned) - I can't see why Bitcoin *needs* to be an exception in the long term (although I do agree that in until a spec that matches the way the "reference client" behaves perfectly is written it cannot take precedence).

I guess the main thing I disagree with is that "it is useless/pointless to even try" (and BTW C++ had a "draft spec" for many years before finally becoming approved and adopted as being "standard").

This is not something that can or should be "rushed into" but something that like Bitcoin itself would need to evolve over time.

[flipperfish]

Maybe simply not calling it "specification" would be enough to avoid most problems raised here.

Yep, maybe the wording is the real problem here. But I still think, that it is better to have a documentation of the protocol with all its bugs and quirks, that have to be recreated for a successfull alternative implementation than forcing alternative client devs to read carefully through the satoshi code. The probability of missing some important quirk/bug in the latter case is IMO much higher. Maybe we have a case of "Curse of Knowledge" here, where some people are so familiar with the satoshi code, that they can't imagine other people finding it hard to understand?

Of course such a documentation has to be maintained, which easily could become a fulltime job, because it's rather worthless if it's outdated. On the other side, if someone would go for this and core devs would support this endeavour by pointing out errors and differences to the satoshi client, it could be of great value. And finally even the satoshi client would profit, because all this bugs and quirks could be evened out (or made part of the final specification).

[MatthewLM]

It's understandable to say any document would need to have plenty of warnings about non-conformity with the network. And yes, you can't rush something like this.

And since there are people afraid about this, why does this wiki page still exist as it is? https://en.bitcoin.it/wiki/Protocol_specification

That's obviously no where near a complete specification of the protocol. That page should really just be called "Bitcoin messages" and then some of the irrelevant stuff should be moved elsewhere.

[Gavin Andresen]

And since there are people afraid about this, why does this wiki page still exist as it is?

Because people like you noticed that it isn't quite right, but didn't bother to change it?

[MatthewLM]

I stay away from wikis. They are awkward to use and whenever I change anything, it gets turned back almost instantly, so what is the point?

[Gavin Andresen]

I completely agree that more documentation is better, as long as you don't fall into the trap of "if the documentation says it it MUST be true."

Suggestions on how to make the wiki better welcome (there is a discussion elsewhere about banning certain people from the wiki because they have a history of starting edit wars, which I would support, and discussion on the Foundation forums about making the wiki infrastructure independend of Mt. Gox, which I also support). Or volunteers to move technical documentation from the wiki to someplace better also welcome.

[MatthewLM]

I completely agree that more documentation is better, as long as you don't fall into the trap of "if the documentation says it it MUST be true."

Suggestions on how to make the wiki better welcome (there is a discussion elsewhere about banning certain people from the wiki because they have a history of starting edit wars, which I would support, and discussion on the Foundation forums about making the wiki infrastructure independend of Mt. Gox, which I also support). Or volunteers to move technical documentation from the wiki to someplace better also welcome.

That is all fair enough. I do agree that such documentation should have clear warnings and describe the risks, and not pretend that the documentation overrules the active software. But I do think there being documentation is better than none. I also think more documentation of the source code is a good idea, perhaps using a doxygen syntax and separating prototypes/definitions from implementations because at the moment it's a bit all over the place.

[Boussac]

I completely agree that more documentation is better, as long as you don't fall into the trap of "if the documentation says it it MUST be true."

Suggestions on how to make the wiki better welcome (there is a discussion elsewhere about banning certain people from the wiki because they have a history of starting edit wars, which I would support, and discussion on the Foundation forums about making the wiki infrastructure independend of Mt. Gox, which I also support). Or volunteers to move technical documentation from the wiki to someplace better also welcome.

+1
From a business perspective, I suppose the bitcoin story sounds a lot better if one can say something like:

There is at least one high level documentation of the protocol (wiki run by the Bitcoin Foundation or else) AND any bitcoin client implementation can be tested for compliance with a single reference implementation called the Satoshi client.
The Satoshi client is open source, regularly updated, peer reviewed and provides the basis for a self-certifying ecosystem.

Through the reference implementation, bitcoin can free online merchants and payment processors from the burden of costly certification requirements imposed by proprietary, legacy payment schemes.

[Johnathan]

In my experience something the reference client does not capture well are the "gotchas" that have been solved over time that are relevant to all Bitcoin peer implementations.
When reading the reference code you might not realize that a piece of code evolved to its current state to solve a serious issue and that the naive implementation wouldn't be sufficient.

This.

There are (I would assume) many things learned along the way in any protocol development, but what lives on is the solution, not the reasons that made it that way.  For example, in reading the network protocol handling of the 'addr' message, there are what seem arbitrary choices for different time periods, selection of address to respond with, etc.  I'm certain there is a reason for each and every one of those, and if you've been with the project from the beginning, you'd know them.  However, newcomers have to reverse engineer the thinking of the developers to get at this.

This isn't a complaint, by the way, just an observation that as a project grows beyond a certain size and age, some things need to get written down that were previously "tribal knowledge amongst the devs".  An IETF-style protocol specification would be useful.  (And I'd be willing to help in some capacity.)

[etotheipi]

By the way, to answer the original question... about a year ago, user "bittrick" submitted something resembling documentation about the technical details of Bitcoin:

https://bitcointalk.org/index.php?topic=41718.0

He spent a lot of time in the code and attempted to capture some of the subtler behaviors of the client.  It's not a "specification" but it is an excellent starting point for understanding what's going on.

Back to argument about "specification"... I absolutely support documentation, and especially written nuggets of information that are difficult to deduce (such as that OpenSSL behavior that Mike Hearn mentioned).  I think it's critical to have these kinds of things to help people understand what they're looking at/for.  Especially if they plan to dig into the C++ code afterwards.  And certain parts of the system very well could be specified (perhaps much of the networking protocol).  But not all of it, especially the script-validation engine.

On the topic of OpenSSL... is it possible to reimplement the OpenSSL behavior, or at least extract it from where it is currently, and have it hardcoded into the Satoshi client?  Or rather, does the license allow you to redistribute the OpenSSL project/source with the project so that it can be static compiled?  I guess you only need a .a/.lib, but I'm not sure what the OpenSSL license is like.  That would certainly resolve any issues related to dynamic linking to system libraries that might be different versions on different computers.  It would make the project immune to system library upgrades, which is probably better for security anyway (an attacker could simply swap out the system OpenSSL with a different version having a known bug that he could exploit, and even a diligent user wouldn't notice because the signatures and checksums on the Bitcoin-Qt source distribution are correct).

[jgarzik]

On the topic of OpenSSL... is it possible to reimplement the OpenSSL behavior, or at least extract it from where it is currently, and have it hardcoded into the Satoshi client?  Or rather, does the license allow you to redistribute the OpenSSL project/source with the project so that it can be static compiled?  I guess you only need a .a/.lib, but I'm not sure what the OpenSSL license is like.  That would certainly resolve any issues related to dynamic linking to system libraries that might be different versions on different computers.  It would make the project immune to system library upgrades, which is probably better for security anyway (an attacker could simply swap out the system OpenSSL with a different version having a known bug that he could exploit, and even a diligent user wouldn't notice because the signatures and checksums on the Bitcoin-Qt source distribution are correct).

I have thought about doing this on more than one occasion:  Creating a libcrypto_satoshi snapshot with the bignum, DER, ECDSA etc. bits we currently use.  This would work around an IMO incorrect exclusion of ECDSA from Fedora's openssl package, as well as "freeze" certain bits that are exported globally via the bitcoin blockchain.

But as always, there are costs to maintaining a fork.  It inevitably gets less eyes, use and maintenance than the main fork.

[gene]

• You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation.

Sorry but it sounds like "We can't tell what the program does really."

Yup. Without a spec and conforming code, you're not doing software engineering. You're just a code monkey wielding a roll of duct tape.

I think the specification would be desirable here, otherwise it's like fixing a running car really.
If later some case is found that the reference client disagrees with the specification:
* Bug in the specification? - Fix the specification.
* Bug in the reference client? - Fix the reference client, or if not possible to fix now and hardfork is needed, then write a note in the specification.
Maybe there are more important matters to look at now but this looks inevitable one day ...

Agreed. No good reason not to do this.

[paybitcoin]

On the topic of OpenSSL... is it possible to reimplement the OpenSSL behavior, or at least extract it from where it is currently, and have it hardcoded into the Satoshi client?  Or rather, does the license allow you to redistribute the OpenSSL project/source with the project so that it can be static compiled?  I guess you only need a .a/.lib, but I'm not sure what the OpenSSL license is like.  That would certainly resolve any issues related to dynamic linking to system libraries that might be different versions on different computers.  It would make the project immune to system library upgrades, which is probably better for security anyway (an attacker could simply swap out the system OpenSSL with a different version having a known bug that he could exploit, and even a diligent user wouldn't notice because the signatures and checksums on the Bitcoin-Qt source distribution are correct).

I have thought about doing this on more than one occasion:  Creating a libcrypto_satoshi snapshot with the bignum, DER, ECDSA etc. bits we currently use.  This would work around an IMO incorrect exclusion of ECDSA from Fedora's openssl package, as well as "freeze" certain bits that are exported globally via the bitcoin blockchain.

This absolutely needs to be done with a high priority. OpenSSL changing at all at this point would require recertifying the new version of OpenSSL anyways against older satoshi code + implementations, correct? Or is there some version pegged already as 'Satoshi's version of OpenSSL'? Or have we just been lucky?

But as always, there are costs to maintaining a fork.  It inevitably gets less eyes, use and maintenance than the main fork.

I'm sure some people trusting their bank accounts to the strength of OpenSSL's ECDSA implementation would have great interest in any code changes upstream - you would probably hear 'how does this affect Bitcoin?' pretty quickly.

IMO, the cost of maintaining a fork is much lower than the potential cost of having an OpenSSL library delivered with a certain distro introduce a split because of a bugfix, code cleanup, or optimization.

[MatthewLM]

At least on the Mac version, the OpenSSL library is packaged with bitcoin. If it wasn't, that would be alarming.

[kjj]

I would absolutely recommend against forking openssl, for reasons that jgarzik gave, and more.  If a day ever comes when we must, then we must.  Until then, just because we can does not mean that we should.

[arturh]

What we need is a reference implementation. A program which is obviously correct. The current implementation is a bit of a mess (DoS checking code inside of "class CTransaction"? Give me a break). Don't get me wrong, even the better written C++ implementation would arise suspicion (see the Underhanded C competition to see what I mean). What we need is a higher level of abstraction, at least for the code dealing with validating blocks and transactions. No need to put the IRC code there.

I think that a reference implementation written in Haskell would be a big plus. This could be used in the official client (even if the GUI/IRC/etc code is written in C++) and would make it easier to trust the program. I started toying around in translating some of the C++ code (https://github.com/arturh/hbitcoin) but there's a lot of work left, this could definitely use some help.

I think this would:
  - make the inner workings of what really matters crystal clear (as long as they were willing to learn some haskell)
  - attract math-heads to giva a look at the code

[xxjs]

When I started in computing in the seventies, it was mosltly trying and failing, after that we got a development standard of 2 shelf meters, after that a binder, after that a thin brochure, and now we are back to trying and failing, which works best, after all.

[2112]

I think that a reference implementation written in Haskell would be a big plus.

Ah, the obligatory visit from a silver-bullet salesman. This year's it is arturh selling Haskell. Last year it was Vandroiy selling aspect-oriented programming.

https://bitcointalk.org/index.php?topic=30200.msg614861#msg614861

It is fun to re-read the posts from late 2011 entitled "Request for Standarization". The more things change the more they stay the same, or in its original French "plus ça change, plus c'est la même chose".

Check out how etotheipi changed from 2011 when he was "out-of-core" to 2013 when he is almost "in-core".

Check out how my lame attempts at satire or Commedia dell'arte didn't change.

If we can't get an official Bitcoin standard can we at least find an official Bitcoin core development jester?

[eldentyrell]

Mike, I agree with all of your technical points, and I agree that the Satoshi client code is and always will be "the spec".  However I'm not sure I'm comfortable with the sociological conclusions you're drawing, especially the whole implementation-diversity-is-unavoidably-bad angle.

I'll set that aside for a moment and just comment on this:

Note that SPV nodes are much less risky.

SPV is less risky precisely because it trusts the difficultywise-longest chain regardless of transaction validity.  Transaction validity is the only complicated/subtle part; judging difficultywise-length requires only SHA-256, which is about two dozen lines of easy-to-test code.

This advantage can be replicated in full-chain-validating clients by having them watch for any invalid chain which is difficultywise-longest by more than the confirmation threshold.  If such a situation arises, activate safe mode: halt all activities except possibly mining.  I assume that the kerfuffle over "losing money" isn't about mining revenue.  Safe mode can be automatically deactivated if a valid chain once again becomes difficultywise-longest.

I suppose clients that do this are still "second class" in the sense that if you find a chain-splitting bug you can get them to pause until a human intervenes.  But it's definitely a distinct security class compared to either SPV or bitcoin-qt.