Formalised Bitcoin Protocol Standard

[davout]

No, there are cases where other people can suffer from chain splitting bugs too. Let's say you're a high volume merchant or payment processor that runs an alternative implementation. I can make a transaction which it believes is invalid but the rest of the network believes is valid, for whatever reason. Once that gets included into a block, your business will grind to a halt because it'll split off onto a chain that no longer gets extended, or only gets extended very slowly, meaning you can't process payments anymore until the problem is noticed and you find and fix the conformance bug.

If this causes you to lose X coins per hour of business, then I can try to anonymously extort you for a bit less than X by claiming I know of such a bug in your software. It's very hard to prove it's not there. You'd have to have a lot of confidence in the robustness of the testing of your implementation.

What about if you accept invalid transactions? If you're providing a good or service in return for unconfirmed transactions then this obviously can undermine your risk model because you'll receive a transaction that you believe is valid, not a double spend, you don't see any double spend alerts or conflicting transactions - but it'll never confirm and I can still spend the money. I don't have to wait and mine a block any more like I would if doing a Finney attack, so it's much cheaper.

You're arguing that financial software should be developed and tested carefully to which I agree. I find nothing in what you say that validates the "a specification document is a bad idea" point.

Such a document would end up being nearly as long as the source code,

So what ?

and not much easier to read.

You can't possibly be serious.

I'm all for adding more detailed comments to the source, though.

That's a good idea obviously but it's not what's being argued here.

Unfortunately, what you've got is the latter and it's not really easy to fix. We keep discovering new odd edge cases where what the software does, isn't what you'd actually expect given a description of how it's meant to work.

I never said it would be easy, I never said anyone has to do anything about it, we're all scratching our own itches after all.
I'm just saying we'd be better off with a complete spec, because it would decouple the protocol from its implementation.
Just because something is hard doesn't make it less necessary.

I think the Satoshi client is quite straightforward to read, for the most part. A few parts are somewhat inscrutable  because they're written very tightly, but unfortunately there is no substitute for just puzzling it out - if you write a description of what you think the code does it may not match reality. We have seen this demonstrated several times, like with the merkle tree calculation. What people thought it did, wasn't quite what it actually did. If you simply duplicated Satoshis algorithm, you would duplicate his bug too so no chain-split attack would have been present. If you re-implemented it based on an English description, you'd have introduced an exploit.

Here you're simply saying that the job is a hard one, but just because some folks have failed in the past doesn't mean it's not a necessary and useful thing to do.

[1714799139]

1714799139

[1714799139]

1714799139

[1714799139]

1714799139

[1714799139]

1714799139

[1714799139]

1714799139

[CIYAM]

and not much easier to read.

You can't possibly be serious.

To make a point I guess everyone familiar with the Bitcoin source can explain what this does and why something like it is needed for a "smart pointer" class?

Code:

class bool_test
{
   public:
   bool_test( ) { }

   private:
   void operator delete( void* );
};

operator bool_test*( ) const
{
   if( !p_instance )
      return 0;
   static bool_test test;
   return &test;
}

(hint: you might want to take a look at http://www.artima.com/cppsource/safebool2.html as even the above doesn't solve all the potential problems for a simple "bool" operator in a smart pointer class in C++)

[Mike Hearn]

So you really think rather than using RFC 1939 one should read through someone's source code (and worse yet "comments" that the compiler ignores) to work out how to do POP3?

I am seriously now beginning to wonder whether anyone here has worked on large scale software projects at all (say > 100 devs and say > 100 million USD).

Ad hominem, great. I spent the last three years designing and building major upgrades to the Google accounts and login system - security critical software. Three years before that on Maps. So yes, I've worked on large scale software projects, in C++.

I don't think what I'm saying is getting through to you. Bitcoin is not like other software projects. It just isn't. If you badly reimplement POP3 then, at worst, some mail clients don't talk to your server or vice-versa. The user swaps to a different client and goes about their day. Big deal. The POP3 spec doesn't need to have huge health warnings at the top of it.

If you badly reimplement Bitcoin then people can and will lose money. Possibly a lot of money. In the worst case, people who don't even use that software can lose money.

By "badly reimplement", I mean making such trivial mistakes as swapping out OpenSSL for a crypto library that follows the DER specs correctly and rejects signatures with negative numbers. OpenSSL doesn't do this, it silently makes them positive to fix the signature. Arguably, that's a bug in OpenSSL, but there it is. It's a part of Bitcoin now. If you don't match that mistake - chain splitting bug. Fortunately, that's one we discovered early through happy accident (somebody actually generated such transactions without realizing and got them into the chain). Now imagine if that hadn't happened until a reimplementation got popular.

How many other such inconsistencies might there be? Can you be sure you got them all? What kind of an audit do you need to find code that isn't wrong or incorrect, it's just different to what it should be?

I don't know of any other systems where such tiny deviations from the spec (in ways that are arguably "correct") can cause such damage. POP3 certainly doesn't count. Perhaps crypto libraries is getting close, in that apparently minor issues (like side channel timing attacks) can cause a total failure .... and so, most competent developers will tell you to not implement your own cryptography for this reason.

Yes, documenting Satoshis code exactly is both hard and theoretically possible. There is already documentation on the basic layout of the protocol messages, which is sufficient for many apps. For the rest, well, feel free to try but don't put your readers under any illusions. They will not be able to safely implement Bitcoin based on your documents - so why bother writing them?

[CIYAM]

I don't think what I'm saying is getting through to you. Bitcoin is not like other software projects. It just isn't. If you badly reimplement POP3 then, at worst, some mail clients don't talk to your server or vice-versa. The user swaps to a different client and goes about their day. Big deal. The POP3 spec doesn't need to have huge health warnings at the top of it.

I do understand exactly what you are saying and unfortunately Bitcoin was released *before* any real spec was written (so the code is what is being used as the spec which IMHO is unfortunate) - whilst it is not possible to "go back and start again" I do think it would be a very important thing to "specify" the Bitcoin protocol in much the same way as other protocols and computer languages are (preferably as an ISO spec) but of course this would not be in any way a trivial thing to do (but something I would certainly donate and/or contribute to if the Foundation were to get behind it).

And as I also pointed understanding C++ *deeply* is not a skill that many people at all have (so how can they *ever* fully understand Bitcoin if they don't *fully* understand every single function in say "boost" works little own OpenSSL and the rest?).

[flipperfish]

I think a documentation/specification like IETF-RFC is a good idea. I really see no reason to NOT do it. People will reimplement bitcoin sooner or later, to some degree it's happening already. The main advantage of Bitcoin is its decentralization. Take that away and I'll stay with fiat instead. So actually I'm happy, that there are efforts to create alternative implementations of the client (especially since I really don't like some parts of the satoshi client). Why not improve these alternative implementations with an official specification and at least try to prevent loss of money and a blockchain-split?

[davout]

I do understand exactly what you are saying and unfortunately Bitcoin was released *before* any real spec was written (so the code is what is being used as the spec which IMHO is unfortunate) - whilst it is not possible to "go back and start again" I do think it would be a very important thing to "specify" the Bitcoin protocol in much the same way as other protocols and computer languages (preferably as an ISO spec) are but of course this would not be in any way a trivial thing to do.

And as I also pointed understanding C++ *deeply* is not a skill that many people at all have (so how can they *ever* fully understand Bitcoin if they don't *fully* understand every single function in say "boost"?).

This.

It's unreasonable to assume that code is a sufficient spec. For someone with your background maybe. But for the rest of us probably not.

A spec would allow for progressively fixing all inconsistencies such as the OpenSSL incorrect behaviour you mention by either documenting them as part of the protocol (and including this behavior in alternate implementations) OR by adding a consistency check at the Bitcoin level.

Mike, simple question, what happens when OpenSSL fixes the buggy behavior?

Bitcoin hitting 1.0 without a complete protocol spec would be a serious mistake IMHO.

Oh and BTW, bugs happen, fuckups happen, it still beats the alternative of keeping all our eggs in one basket.

[flipperfish]

It's unreasonable to assume that code is a sufficient spec. For someone with your background maybe. But for the rest of us probably not.

A spec would allow for progressively fixing all inconsistencies such as the OpenSSL incorrect behaviour you mention by either documenting them as part of the protocol (and including this behavior in alternate implementations) OR by adding a consistency check at the Bitcoin level.

Mike, simple question, what happens when OpenSSL fixes the buggy behavior?

Bitcoin hitting 1.0 without a complete protocol spec would be a serious mistake IMHO.

Oh and BTW, bugs happen, fuckups happen, it still beats the alternative of keeping all our eggs in one basket.

+1
Couldn't have said it better.

What came to my mind just now: The buggy behavior of OpenSSL could not be known by just having read the code of the satoshi client, correct? In this case, this is rather an argument for a formal specification besides the code than against.

[2112]

Mike, simple question, what happens when OpenSSL fixes the buggy behavior?

More work, praise for work, and respect from work to the core development team. In the absence of the usual, salaried work arrangement respect is about the only motivating factor everyone can count on.

There is a problem with the alternative compensation schemes: big payoff after successfull delivery: it works for well managed IPO or well executed long con.

Bitcoin hitting 1.0 without a complete protocol spec would be a serious mistake IMHO.

I think that the fixed specification is not a worthwhile goal. The goal should be to establish a protocol management organization that has inclusiveness written into the statute.

But in order to achieve the above Bitcoin millieu would need to have a credible economic force to counterbalance the American-Japanese coalition that currently steers Bitcoin.

It would be very easy to completely derail the specification writing by overdesigning it or overformalizing it. I'm somewhat familiar with the infamous history of the VRML (Virtual Reality Modeling Language) that Gavin Andresen co-authored. It was primarily developed by the consortium of hardware manufacturers with very little input from the primary users of modeling: 3d sculptors, graphic artists, industrial designers, etc. VRML became an idealized superset tool that wasn't really supported by the actual tools used in the industry from Wavefront, Alias, Softimage, etc. It appealed only to programmers and probably to hardware vendors.

Decent brief on the historical context can be read here:

https://design.osu.edu/carlson/history/lessons.html

in sections 7 and 8.

So the VRML misdesign equivalent would be to design Bitcoin according to the needs of cryptographers and programmers to the detriment of accountants and other users of financial software.

Do you recognize the similarities to the current situation?

Anyway, to have a credible standarization effort you'll need cash flow to support it. There is very little point in comparing it with efforts like HTTP or VRML or other "(say > 100 devs and say > 100 million USD)" undertakings. I don't think that there won't be much benefit to it unless there is at least 4 people drawing 4 salaries doing full-time Bitcoin development. I don't think Bitcoin is at this stage yet.

[Steve]

Someone should just write a spec and stop debating it.  If you write it in such a way that it's directly executable (as the OMeta guys did with the TCP/IP spec), you get bonus points.  But I agree with Mike's arguments.  I've worked on large projects where there wasn't a lick of external design documentation...the mentality was that the code was the spec.  And I have to say it was the most well written system I've ever had the pleasure of working on.  It was also the best tested system I've ever worked on...all testing was 100% automated...there was no traditional QA cycle before a release.  Now, the system was written in Smalltalk, which is far easier to read and comprehend than C++, but the code was the spec (also, the fact that people had to read and understand the code to comprehend the system was a major contributing factor to the quality of the code).

P.S.  This reminds me...the original specification for Smalltalk was published in a Book (called the Blue Book)...in that book, the specification for the Smalltalk virtual machine was written in Smalltalk (in the book)...years later, a transcription of that implementation from the book was used as the basis for Squeak's virtual machine which is written in Smalltalk and translated to C for compilation to any target architecture...but you can directly execute Squeak's virtual machine inside Squeak without translating to C...it's really slow, but it works.  I say all of this to drive the point that the ideal scenario is that the spec is directly executable (or, stated another way, that the code is the spec).  I understand that C++ is hard to read and that separate documentation is sometimes justifiable, but it's not ideal.

[caveden]

Someone should just write a spec and stop debating it.  

This.
If you really want such specification, DIY or offer a bounty to someone else.

[2112]

P.S.  This reminds me...the original specification for Smalltalk was published in a Book (called the Blue Book)...in that book, the specification for the Smalltalk virtual machine was written in Smalltalk (in the book)...years later, a transcription of that implementation from the book was used as the basis for Squeak's virtual machine which is written in Smalltalk and translated to C for compilation to any target architecture...

Smalltalk blue book is a beautifull example of how to write a great specification and simultaneously how to lose money on a business venture. The book(s) are just beautifull; but the business was really nasty: Xerox jeaously guarded the trade secrets and copyrights and demanded tremendous royalities and fees. Even after the Smalltalk died as a viable business competitor Xerox lawyers with great effort guarded the copyrights for what essentially was an abandonware. Thus came Squeak: a clean-room legaly-unimpeachable re-implementation to avoid paying Xerox.

With the business advice like that Bitcoin doesn't need enemies.

[Steve]

P.S.  This reminds me...the original specification for Smalltalk was published in a Book (called the Blue Book)...in that book, the specification for the Smalltalk virtual machine was written in Smalltalk (in the book)...years later, a transcription of that implementation from the book was used as the basis for Squeak's virtual machine which is written in Smalltalk and translated to C for compilation to any target architecture...

Smalltalk blue book is a beautifull example of how to write a great specification and simultaneously how to lose money on a business venture. The book(s) are just beautifull; but the business was really nasty: Xerox jeaously guarded the trade secrets and copyrights and demanded tremendous royalities and fees. Even after the Smalltalk died as a viable business competitor Xerox lawyers with great effort guarded the copyrights for what essentially was an abandonware. Thus came Squeak: a clean-room legaly-unimpeachable re-implementation to avoid paying Xerox.

With the business advice like that Bitcoin doesn't need enemies.

Yep, it's incredible how badly they botched so many things relating to PARC...but that's a whole other subject.

[Mike Hearn]

Mike, simple question, what happens when OpenSSL fixes the buggy behavior?

Hopefully by that time the data being sent to OpenSSL is being carefully checked and pre-filtered. Although the OpenSSL guys don't perceive it to be a bug, so it may never be fixed.

The problem is, we don't know where other such issues are lurking. You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation. And unlike other systems, if the majority implementation disagrees with the spec, unless the bug is so severe it undermines the entire system the implementation won't be changing to agree with the spec. The "spec", if you want to call it that, would change to meet the majority. That's just the inevitable outcome when you have people losing money for every hour they're on an alternative chain. There's no incentive to wait for the theoretically correct version of the majority implementation to roll out. You want to get yourself back into the economic majority ASAP and if that means automatically correcting the sign of the signature components (or whatever), then so be it.

That's why Bitcoin is not like other systems. If you want to write a protocol spec go ahead, but don't be under any illusions. If the behaviour of the Satoshi client doesn't match your written spec and the behaviour isn't totally broken, it'll be your spec changing not the software. Which means it's not really a standard.

[2112]

That's just the inevitable outcome when you have people losing money for every hour they're on an alternative chain.

What's with that "losing money" FUD? As far as fixing OpenSSL I understand that the existing blockchain would fail the self-test. So the outcome would be that nobody using Satoshi's client would upgrade until core developers come up with an explicit way to reproduce the old OpenSSL bugs. Is that what you are afraid of?

Anyway, weekend is a good time to post my favourite picture of an amber.

[MatthewLM]

Such a document would end up being nearly as long as the source code, and not much easier to read. I'm all for adding more detailed comments to the source, though.

Well I for one think this page: https://en.bitcoin.it/wiki/Protocol_specification is easier to read than trying to find how the data is structured in the source code. And the script page is much better than looking through the script interpreter source code. When implementing parts to cbitcoin I did find some wiki pages helpful, but I did look at the satoshi client () to much sure in cases since I can't put faith in the wiki articles.

I think the Satoshi client is quite straightforward to read, for the most part. A few parts are somewhat inscrutable  because they're written very tightly, but unfortunately there is no substitute for just puzzling it out - if you write a description of what you think the code does it may not match reality. We have seen this demonstrated several times, like with the merkle tree calculation. What people thought it did, wasn't quite what it actually did. If you simply duplicated Satoshis algorithm, you would duplicate his bug too so no chain-split attack would have been present. If you re-implemented it based on an English description, you'd have introduced an exploit.

An English description isn't going to be "You need to calculate the merkle root for this". Obviously it's going to describe the algorithm used.

They will not be able to safely implement Bitcoin based on your documents - so why bother writing them?

If that were true, how would you expect anyone to safely implement bitcoin looking at the satoshi code, a job which is much harder?

So the VRML misdesign equivalent would be to design Bitcoin according to the needs of cryptographers and programmers to the detriment of accountants and other users of financial software.

What I'm talking about is a technical specification for developers, not a manual for end-users.

[jgarzik]

Someone should just write a spec and stop debating it.

+1

If you want a spec then write something.  That's what's happening by default anyway on the bitcoin wiki.

But Mike Hearn makes two key points that are nonetheless valid:

• You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation.
• The consequences of getting details wrong is far more severe than that which most programmers are used to; there is simply no comparison to, e.g., POP3.

If it were an IETF draft, I would add a section at the top that states "the reference implementation is canonical, this spec is subordinate. any differences are spec bugs."

[commonancestor]

• You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation.

Sorry but it sounds like "We can't tell what the program does really."
I think the specification would be desirable here, otherwise it's like fixing a running car really.
If later some case is found that the reference client disagrees with the specification:
* Bug in the specification? - Fix the specification.
* Bug in the reference client? - Fix the reference client, or if not possible to fix now and hardfork is needed, then write a note in the specification.
Maybe there are more important matters to look at now but this looks inevitable one day ...

[kjj]

So, how many more major developers need to show up here to explain that a spec would be 1) not very useful, and/or 2) actually dangerous, before you believe it?

[etotheipi]

I'll chime in:  Mike Hearn is absolutely right.  

There's nothing wrong with writing "documentation" to help describe at a higher level what is going on.  As a developer of a non-validation client, such documentation has been quite useful.  I fully support "extra stuff the describes at a high level what's going on".  But you must understand that once you put the word "specification" on any such document, you are promising the reader that the document is sufficient for creating a compliant implementation of what is being described.  But with Bitcoin, anything short of the source code itself is not sufficient for implementing it.  And worse, the consequences of not doing so will result in people losing money -- because if there's an inefficiency in the system an attacker will exploit it, guaranteed (especially one where your target uses code that validates differently than much of the rest of the network).  This is why Satoshi did not support alternative implementations.


Write all the "documentation" you want, and put as many comments into the code as you want.  But do not use the word "specification" because nothing except the code can qualify for it.

[adamstgBit]

I'll chime in:  Mike Hearn is absolutely right.   

There's nothing wrong with writing "documentation" to help describe at a higher level what is going on.  As a developer of a non-validation client, such documentation has been quite useful.  I fully support "extra stuff the describes at a high level what's going on".  But you must understand that once you put the word "specification" on any such document, you are promising the reader that the document is sufficient for creating a compliant implementation of what is being described.  But, anything short of the source code itself is not sufficient for implementing it.  And worse, the consequences of not doing so will result in people losing money -- because if there's an inefficiency in the system an attacker will exploit it, guaranteed (especially one where your target uses code that validates differently than much of the rest of the network).  This is why Satoshi did not support alternative implementations.


Write all the "documentation" you want, and put as many comments into the code as you want.  But do not use the word "specification" because nothing except the code can qualify for it.

+1
low level detailed documentation is useless and could be dangerous. but a high level diagram can be very useful for a quick refresher as to how a particular module dose what it dose, and more importantly, when your talking to someone about making changes a good diagram will felicitate communicating ideas.