Bitcoin Forum

I've thought about this and I'm surprised I've not seen (or can find) very much discussion eluding to this. At the moment, for anyone that wants to understand the bitcoin protocol, they would be able to use the bitcoin wiki somewhat, as well as forums and other websites but ultimately have to look at the source code of bitcoin implementations, or rely on the knowledge of other people.

It would be very useful and wise, in my opinion, if there was a formalised document describing the protocol to every detail, but in a way that is easy for anyone to follow. It would be a document that would be used as a reference for developers and would reflect all of the agreed (In majority use/Majority mining power) protocol features. The protocol standards document would then be amended as the protocol is modified. A separated set of documents could describe other features which are not core to the protocol such as wallet formats or whatever.

I had a hunch this would be something the Bitcoin Foundation was set up for, but it seems not. Do other people think this would be very useful to work upon? Otherwise the information will continue to be disorganised and a nightmare to piece together.

What about Satoshi's paper on Bitcoin? Isn't that the official specification?

The paper is a design overview, not a specification. It presents the argument that something like bitcoin can work at all, but doesn't tell you the details of building something compatible with it.

Perhaps if you post an example of a specification that is both "formal" and "easy for anyone" we could make a better comments. Common way of thinking leans toward saying that those are polar opposites.

Anyway, the major points against are:

0) extremely expensive
1) a lot of work with comparatively little benefit
2) hard to prove internal consistency
3) hard to verify consistency with non-formal, but actual implementations

When asked for pitfalls of "formal modeling" I nowadays point towards the ARM Architecture Manual and the way how multi-million company with clearly clever and well motivated staff ended up with BE32 and BE8 (a.k.a. just plain BE): two largely incompatible ways to implement big-endianess.

Well by formalised I meant, but together professionally into a specification document (as opposed to now). I din't mean much more than that. It doesn't necessarily have to go by any usual conventions, if that means the document can be both easy to follow and fully detailed.

The reality of how Bitcoin works means that Satoshis code is the protocol definition.

Isn't most of the protocol right here?

https://en.bitcoin.it/wiki/Protocol_specification

In my experience, developers are really good at either ignoring documentation or interpreting it in a way different than the way the author intended.

And spec authors are really good at getting details wrong, no matter how careful they are. And they're really bad at keeping track of changes.

That's why I spent a lot of time over the past year developing test cases and tools that you can run your code against instead of writing specs.

I may just be cynical because I spent so much time in 1997 working on the ISO/IEC-14772-1 Official, Formal Standard.

Flying 3D sharks from the past! :)

You have the format of the messages on there but nothing about the network operation, validation, scripts etc. THere are other wiki articles that have more information but the information is incomplete and scattered around.


Yes maybe, but surely it's better than developers trying to decipher source code and learn bits here and there?


Well since bitcoin is an open protocol, there can be any number of people contributing to a bitcoin protocol specification, and anyone could spot mistakes and suggest improvements. It doesn't have to be bureaucratic or closed in nature.

That sounds like a statement that might apply at a particular moment in time.  It isn't an argument against specifying the protocol separately from a particular implementation in future, though it is obviously something to take account of while writing the specification.

There was probably a time in 1990 when the reality of how the web worked meant that the CERN httpd implementation was the protocol definition for HTTP (I don't know for sure, but this a pattern seen with many protocols and other interface elements: original PGP implementation -> OpenPGP protocol; this was more or less the way ssh went as well regarding initial implementation to decent public protocol definition too).  That doesn't mean it was a bad idea to create standardisation processes once the technology took off and there was interest in multiple compatible implementations, and in managing changes / extensions.


That's good stuff too -- but is certainly not an argument against trying to get the protocol specified in some sort of, for example, IETF-RFC-like document.  Specifications and test suites go together really well, but are not alternatives for one another.  Test suites are sometimes good for clarifying intent where a spec's ambiguous, and as you say above they're also great aiding implementors with completeness and correctness.

Bitcoin needs a protocol spec for the technology to mature.  One doesn't want to do it while the design's in flux, but Bitcoin's past that stage now.  Any incompatible design changes would be brought about as a new crypto currency rather than as changes to existing Bitcoin.

It is frankly pretty worrying to see Gavin and Mike be so dismissive of MatthewLM's suggestion.  Hopefully some others involved have more wisdom and experience in protocol engineering at Internet scale.

I'm also concerned with needing to refer to the reference client source code but the reference client is called reference for a reason  :)

My concerns are that as the reference client struggles to stay relevant for end users the core developers focus on performance rather than for use as a reference.
Performance and readability do not tend to go hand in hand.

Is the creation of better protocol documentation a better solution?
I personally think so, as the core aspects of the protocol are effectively set in stone we should be able to document them in an accessible/understandable manner.

As Gavin identified the creation and maintenance of specs however is time consuming.
The reference client developers are free to spend there time however they feel is best. There are always issues to be fixed and new features to be implemented. We wouldn't want to stop the reference client from moving forward.

I'd like to offer my help in updating/maintaining the documentation. I've made a few minor edits to the Bitcoin wiki for some of the under specified or unclear areas that I've found.

Where do you feel the content of the Wiki currently falls short?

In my experience I've found that the status of some of the BIPs are out of date and I've tracked a few of them down and updated their status.
Once a BIP is accepted I think we should aim to roll its implications into the base documentation.
This information is recoverable by comparing the reference implementation to the BIPs.

In what ways do you feel that the Reference client falls short as use as a reference?

In my experience something the reference client does not capture well are the "gotchas" that have been solved over time that are relevant to all Bitcoin peer implementations.
When reading the reference code you might not realize that a piece of code evolved to its current state to solve a serious issue and that the naive implementation wouldn't be sufficient.
Again this information isn't lost we can recover it from the history and issue tracker and present it in a more accessible way.

I agree with Gavin's point of view.  Unit Tests are the final and most complete form of behavior specification and the implementation is the final and most complete form of design.  It's best when both are expressed in languages free of a lot of syntactic noise.  C++ is far from ideal in that regard, but you live with compromises born out of practicality.  In languages that are less encumbered by syntactic noise, this perspective is much more readily apparent.  The tests and the implementation are so easily comprehensible that other documentation isn't worth the effort to maintain (and can even be a detriment). 

Check out OMeta and some of the papers at vpri.org if you're really into this sort of thing...with OMeta, they managed to create a system that could almost directly execute TCP/IP from the RFCs.  It was a complete TCP/IP implementation in under 200 lines of code (including the parser specification for the RFC ascii art).  See this summary about it: http://www.moserware.com/2008/04/towards-moores-law-software-part-3-of-3.html  ...to me, this is a proof point that code really should be regarded as self documenting (with little more than annotations to accompany it)...if it's too challenging for people to easily comprehend, it points to a shortcoming of the language, not of the concept that the code is the documentation.

I disagree with the folks that find tons of reasons not to document. I'm not really surprised though, this has come up a few times already and the answer was already pretty much along these lines.

As much as I understand that the core contributors don't really feel like doing it for various reasons (they already write tests and contribute code after all), I'm really surprised that no one really seems to encourage MatthewLM to go forward with it.

Yes, tests are good, but add a complete spec and it gets even better. Yes, it's a fact that the main implementation is currently both the specification and the implementation, nobody can argue that. However, arguing that it's a good thing, that it shouldn't change, that a full protocol documentation is unnecessary isn't quite the same thing IMHO.

It bothers me that this topic keeps coming up. The fact that Bitcoin is different to other technologies isn't intuitive but by the time you're writing an actual implementation, it should be obvious. Maybe it's worth reading the thread with grau about his reimplementation also?

The fact is that re-implementing Bitcoin exposes not only you, but all participants, to a class of "chain splitting" bugs that don't really exist in other network technologies, or at least are nowhere near as severe. The browser wars of the 90s were bad, but at least developers could check which browser the user ran and adapt to it on the fly. The Bitcoin equivalent is dramatically worse.

When you reimplement Bitcoin, it's not enough to build things as you think they should work. You have to implement them exactly as Satoshi did, including all his bugs. And because some parts of the protocol are directly exposed to underlying libraries like OpenSSL, you have to match their behaviour exactly as well, including all their bugs. Failure to do so can lead to people losing money.

At some point, if you realize you have to match the behaviour of another codebase exactly, down to the tiniest detail, you realize that the only precise enough specification for that is the source code. Which means if you can't read C++ fluently you can't reimplement Bitcoin, yes, but who cares? If you can't keep up, don't step up.

Having detailed protocol documentation is something I'd agree with in any other project except this one. In this one, it will simply mislead people into thinking they can reimplement Bitcoin. Unless they are willing to make absolutely massive effort and take serious risks, they can't.

Note that SPV nodes are much less risky. But Matthew isn't implementing an SPV client.

For one that is true only if miners are to run alternative implementations. Secondly I find your statements a little FUDdy because even in the case of a chain split, most transactions would make it to both chains until it's resolved.

Maybe documenting the protocol could lead to fixing said bugs at an agreed-upon block height leading to a clearer and more consistent protocol. I really don't see the harm in documenting what happens under the hood, bugs included.

Maybe that's the sign that the specification-software is getting too convoluted, which will ultimately lead to unmaintainable poor quality software.

What I want to see are competing implementations of a clearly defined protocol, not a centralized black-box maintained by a few who know exactly which bugs should be treated as features.

Putting all your eggs in a single basket is never a good idea (especially when they're golden eggs), what happens the day a critical exploit is discovered in the reference implementation ? Does everything collapse ?

Oh, and there's a reason why Bitcoin is still not 1.0 ;)

The unit tests are just there to ensure things are working as expected. They aren't designed to provide a reference to how things are supposed to work.


The English language, combined with diagrams, tables etc. are designed for humans to understand and are thus the ideal format, as opposed to C++. And the Satoshi client is not very friendly to human eyes, so indeed another implementation could make it much easier to understand, but why waste time on doing that if you could write a human-friendly specification?


For some reason I couldn't find very much when I searched for it, except vague references.


All the more reason to make the protocol as clear and easy to understand as possible.


Things have gotten better when things have become more standardised.


Once again, more reason to have clear documentation. Though you do not need to do everything the same way as the Satoshi client. You only need to conform to the protocol requirements.


Sorry? People can re-implement bitcoin. You know that, obviously.


I'm implementing code which can be used for full validation or headers only validation. My plans for a client will includes a mixture between the two, offering the best of both worlds. The block-chain will thus be checked by headers-only validation against full validation done by a server.

I really think that MatthewLM has a very valid point here - if the C++ standard was just "read GCC" then I think the language would never have even been used to write Bitcoin (or anything else of value just like languages such as D).

Although such documents are very hard to write there is a point in having such standards (or do you guys prefer Microsoft or Google to *make* standards in code rather than using ISO ones?).

Anyone thinking that "Bitcoin is an exception" is kidding themselves (or are wanting to become the next Microsoft or Google themselves most likely).

No, there are cases where other people can suffer from chain splitting bugs too. Let's say you're a high volume merchant or payment processor that runs an alternative implementation. I can make a transaction which it believes is invalid but the rest of the network believes is valid, for whatever reason. Once that gets included into a block, your business will grind to a halt because it'll split off onto a chain that no longer gets extended, or only gets extended very slowly, meaning you can't process payments anymore until the problem is noticed and you find and fix the conformance bug.

If this causes you to lose X coins per hour of business, then I can try to anonymously extort you for a bit less than X by claiming I know of such a bug in your software. It's very hard to prove it's not there. You'd have to have a lot of confidence in the robustness of the testing of your implementation.

What about if you accept invalid transactions? If you're providing a good or service in return for unconfirmed transactions then this obviously can undermine your risk model because you'll receive a transaction that you believe is valid, not a double spend, you don't see any double spend alerts or conflicting transactions - but it'll never confirm and I can still spend the money. I don't have to wait and mine a block any more like I would if doing a Finney attack, so it's much cheaper.


Such a document would end up being nearly as long as the source code, and not much easier to read. I'm all for adding more detailed comments to the source, though.


Unfortunately, what you've got is the latter and it's not really easy to fix. We keep discovering new odd edge cases where what the software does, isn't what you'd actually expect given a description of how it's meant to work.


I think the Satoshi client is quite straightforward to read, for the most part. A few parts are somewhat inscrutable  because they're written very tightly, but unfortunately there is no substitute for just puzzling it out - if you write a description of what you think the code does it may not match reality. We have seen this demonstrated several times, like with the merkle tree calculation. What people thought it did, wasn't quite what it actually did. If you simply duplicated Satoshis algorithm, you would duplicate his bug too so no chain-split attack would have been present. If you re-implemented it based on an English description, you'd have introduced an exploit.


I've implemented the SPV mode (not re-implemented, as for a long time there was no other implementation of this). Matt Corallo went ahead and has extended my work to do full validation. He's done a TON of testing, very in depth testing, despite that I would be very concerned if I heard that a big mining pool or BitPay or whoever was using it. At least in it's current state. It's not clear to me how much work would be required until I felt comfortable with high value operations using bitcoinj in full mode, and the documentation when 0.7 is released will make that clear.

So you really think rather than using RFC 1939 one should read through someone's source code (and worse yet "comments" that the compiler ignores) to work out how to do POP3?

I am seriously now beginning to wonder whether anyone here has worked on large scale software projects at all (say > 100 devs and say > 100 million USD).

You're arguing that financial software should be developed and tested carefully to which I agree. I find nothing in what you say that validates the "a specification document is a bad idea" point.


So what ?

You can't possibly be serious.

That's a good idea obviously but it's not what's being argued here.

I never said it would be easy, I never said anyone has to do anything about it, we're all scratching our own itches after all.
I'm just saying we'd be better off with a complete spec, because it would decouple the protocol from its implementation.
Just because something is hard doesn't make it less necessary.

Here you're simply saying that the job is a hard one, but just because some folks have failed in the past doesn't mean it's not a necessary and useful thing to do.

To make a point I guess everyone familiar with the Bitcoin source can explain what this does and why something like it is needed for a "smart pointer" class?


(hint: you might want to take a look at http://www.artima.com/cppsource/safebool2.html as even the above doesn't solve all the potential problems for a simple "bool" operator in a smart pointer class in C++)

:)

Ad hominem, great. I spent the last three years designing and building major upgrades to the Google accounts and login system - security critical software. Three years before that on Maps. So yes, I've worked on large scale software projects, in C++.

I don't think what I'm saying is getting through to you. Bitcoin is not like other software projects. It just isn't. If you badly reimplement POP3 then, at worst, some mail clients don't talk to your server or vice-versa. The user swaps to a different client and goes about their day. Big deal. The POP3 spec doesn't need to have huge health warnings at the top of it.

If you badly reimplement Bitcoin then people can and will lose money. Possibly a lot of money. In the worst case, people who don't even use that software can lose money.

By "badly reimplement", I mean making such trivial mistakes as swapping out OpenSSL for a crypto library that follows the DER specs correctly and rejects signatures with negative numbers. OpenSSL doesn't do this, it silently makes them positive to fix the signature. Arguably, that's a bug in OpenSSL, but there it is. It's a part of Bitcoin now. If you don't match that mistake - chain splitting bug. Fortunately, that's one we discovered early through happy accident (somebody actually generated such transactions without realizing and got them into the chain). Now imagine if that hadn't happened until a reimplementation got popular.

How many other such inconsistencies might there be? Can you be sure you got them all? What kind of an audit do you need to find code that isn't wrong or incorrect, it's just different to what it should be?

I don't know of any other systems where such tiny deviations from the spec (in ways that are arguably "correct") can cause such damage. POP3 certainly doesn't count. Perhaps crypto libraries is getting close, in that apparently minor issues (like side channel timing attacks) can cause a total failure .... and so, most competent developers will tell you to not implement your own cryptography for this reason.

Yes, documenting Satoshis code exactly is both hard and theoretically possible. There is already documentation on the basic layout of the protocol messages, which is sufficient for many apps. For the rest, well, feel free to try but don't put your readers under any illusions. They will not be able to safely implement Bitcoin based on your documents - so why bother writing them?

I do understand exactly what you are saying and unfortunately Bitcoin was released *before* any real spec was written (so the code is what is being used as the spec which IMHO is unfortunate) - whilst it is not possible to "go back and start again" I do think it would be a very important thing to "specify" the Bitcoin protocol in much the same way as other protocols and computer languages are (preferably as an ISO spec) but of course this would not be in any way a trivial thing to do (but something I would certainly donate and/or contribute to if the Foundation were to get behind it).

And as I also pointed understanding C++ *deeply* is not a skill that many people at all have (so how can they *ever* fully understand Bitcoin if they don't *fully* understand every single function in say "boost" works little own OpenSSL and the rest?).

I think a documentation/specification like IETF-RFC is a good idea. I really see no reason to NOT do it. People will reimplement bitcoin sooner or later, to some degree it's happening already. The main advantage of Bitcoin is its decentralization. Take that away and I'll stay with fiat instead. So actually I'm happy, that there are efforts to create alternative implementations of the client (especially since I really don't like some parts of the satoshi client). Why not improve these alternative implementations with an official specification and at least try to prevent loss of money and a blockchain-split?

This.

It's unreasonable to assume that code is a sufficient spec. For someone with your background maybe. But for the rest of us probably not.

A spec would allow for progressively fixing all inconsistencies such as the OpenSSL incorrect behaviour you mention by either documenting them as part of the protocol (and including this behavior in alternate implementations) OR by adding a consistency check at the Bitcoin level.

Mike, simple question, what happens when OpenSSL fixes the buggy behavior?

Bitcoin hitting 1.0 without a complete protocol spec would be a serious mistake IMHO.

Oh and BTW, bugs happen, fuckups happen, it still beats the alternative of keeping all our eggs in one basket.

+1
Couldn't have said it better.

What came to my mind just now: The buggy behavior of OpenSSL could not be known by just having read the code of the satoshi client, correct? In this case, this is rather an argument for a formal specification besides the code than against.

More work, praise for work, and respect from work to the core development team. In the absence of the usual, salaried work arrangement respect is about the only motivating factor everyone can count on.

There is a problem with the alternative compensation schemes: big payoff after successfull delivery: it works for well managed IPO or well executed long con.

I think that the fixed specification is not a worthwhile goal. The goal should be to establish a protocol management organization that has inclusiveness written into the statute.

But in order to achieve the above Bitcoin millieu would need to have a credible economic force to counterbalance the American-Japanese coalition that currently steers Bitcoin.

It would be very easy to completely derail the specification writing by overdesigning it or overformalizing it. I'm somewhat familiar with the infamous history of the VRML (Virtual Reality Modeling Language) that Gavin Andresen co-authored. It was primarily developed by the consortium of hardware manufacturers with very little input from the primary users of modeling: 3d sculptors, graphic artists, industrial designers, etc. VRML became an idealized superset tool that wasn't really supported by the actual tools used in the industry from Wavefront, Alias, Softimage, etc. It appealed only to programmers and probably to hardware vendors.

Decent brief on the historical context can be read here:

https://design.osu.edu/carlson/history/lessons.html

in sections 7 and 8.

So the VRML misdesign equivalent would be to design Bitcoin according to the needs of cryptographers and programmers to the detriment of accountants and other users of financial software.

Do you recognize the similarities to the current situation?

Anyway, to have a credible standarization effort you'll need cash flow to support it. There is very little point in comparing it with efforts like HTTP or VRML or other "(say > 100 devs and say > 100 million USD)" undertakings. I don't think that there won't be much benefit to it unless there is at least 4 people drawing 4 salaries doing full-time Bitcoin development. I don't think Bitcoin is at this stage yet.

Someone should just write a spec and stop debating it.  If you write it in such a way that it's directly executable (as the OMeta guys did with the TCP/IP spec), you get bonus points.  But I agree with Mike's arguments.  I've worked on large projects where there wasn't a lick of external design documentation...the mentality was that the code was the spec.  And I have to say it was the most well written system I've ever had the pleasure of working on.  It was also the best tested system I've ever worked on...all testing was 100% automated...there was no traditional QA cycle before a release.  Now, the system was written in Smalltalk, which is far easier to read and comprehend than C++, but the code was the spec (also, the fact that people had to read and understand the code to comprehend the system was a major contributing factor to the quality of the code).

P.S.  This reminds me...the original specification for Smalltalk was published in a Book (called the Blue Book)...in that book, the specification for the Smalltalk virtual machine was written in Smalltalk (in the book)...years later, a transcription of that implementation from the book was used as the basis for Squeak's virtual machine which is written in Smalltalk and translated to C for compilation to any target architecture...but you can directly execute Squeak's virtual machine inside Squeak without translating to C...it's really slow, but it works.  I say all of this to drive the point that the ideal scenario is that the spec is directly executable (or, stated another way, that the code is the spec).  I understand that C++ is hard to read and that separate documentation is sometimes justifiable, but it's not ideal.

This.
If you really want such specification, DIY or offer a bounty to someone else.

Smalltalk blue book is a beautifull example of how to write a great specification and simultaneously how to lose money on a business venture. The book(s) are just beautifull; but the business was really nasty: Xerox jeaously guarded the trade secrets and copyrights and demanded tremendous royalities and fees. Even after the Smalltalk died as a viable business competitor Xerox lawyers with great effort guarded the copyrights for what essentially was an abandonware. Thus came Squeak: a clean-room legaly-unimpeachable re-implementation to avoid paying Xerox.

With the business advice like that Bitcoin doesn't need enemies.

Yep, it's incredible how badly they botched so many things relating to PARC...but that's a whole other subject.

Hopefully by that time the data being sent to OpenSSL is being carefully checked and pre-filtered. Although the OpenSSL guys don't perceive it to be a bug, so it may never be fixed.

The problem is, we don't know where other such issues are lurking. You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation. And unlike other systems, if the majority implementation disagrees with the spec, unless the bug is so severe it undermines the entire system the implementation won't be changing to agree with the spec. The "spec", if you want to call it that, would change to meet the majority. That's just the inevitable outcome when you have people losing money for every hour they're on an alternative chain. There's no incentive to wait for the theoretically correct version of the majority implementation to roll out. You want to get yourself back into the economic majority ASAP and if that means automatically correcting the sign of the signature components (or whatever), then so be it.

That's why Bitcoin is not like other systems. If you want to write a protocol spec go ahead, but don't be under any illusions. If the behaviour of the Satoshi client doesn't match your written spec and the behaviour isn't totally broken, it'll be your spec changing not the software. Which means it's not really a standard.

What's with that "losing money" FUD? As far as fixing OpenSSL I understand that the existing blockchain would fail the self-test. So the outcome would be that nobody using Satoshi's client would upgrade until core developers come up with an explicit way to reproduce the old OpenSSL bugs. Is that what you are afraid of?

Anyway, weekend is a good time to post my favourite picture of an amber.

Well I for one think this page: https://en.bitcoin.it/wiki/Protocol_specification is easier to read than trying to find how the data is structured in the source code. And the script page is much better than looking through the script interpreter source code. When implementing parts to cbitcoin I did find some wiki pages helpful, but I did look at the satoshi client () to much sure in cases since I can't put faith in the wiki articles.


An English description isn't going to be "You need to calculate the merkle root for this". Obviously it's going to describe the algorithm used.


If that were true, how would you expect anyone to safely implement bitcoin looking at the satoshi code, a job which is much harder?


What I'm talking about is a technical specification for developers, not a manual for end-users.

+1

If you want a spec then write something.  That's what's happening by default anyway on the bitcoin wiki.

But Mike Hearn makes two key points that are nonetheless valid:

• You can write a spec, but you can't guarantee it actually matches the behaviour of the majority implementation.
• The consequences of getting details wrong is far more severe than that which most programmers are used to; there is simply no comparison to, e.g., POP3.

If it were an IETF draft, I would add a section at the top that states "the reference implementation is canonical, this spec is subordinate. any differences are spec bugs."

Sorry but it sounds like "We can't tell what the program does really."
I think the specification would be desirable here, otherwise it's like fixing a running car really.
If later some case is found that the reference client disagrees with the specification:
* Bug in the specification? - Fix the specification.
* Bug in the reference client? - Fix the reference client, or if not possible to fix now and hardfork is needed, then write a note in the specification.
Maybe there are more important matters to look at now but this looks inevitable one day ...

So, how many more major developers need to show up here to explain that a spec would be 1) not very useful, and/or 2) actually dangerous, before you believe it?

I'll chime in:  Mike Hearn is absolutely right.  

There's nothing wrong with writing "documentation" to help describe at a higher level what is going on.  As a developer of a non-validation client, such documentation has been quite useful.  I fully support "extra stuff the describes at a high level what's going on".  But you must understand that once you put the word "specification" on any such document, you are promising the reader that the document is sufficient for creating a compliant implementation of what is being described.  But with Bitcoin, anything short of the source code itself is not sufficient for implementing it.  And worse, the consequences of not doing so will result in people losing money -- because if there's an inefficiency in the system an attacker will exploit it, guaranteed (especially one where your target uses code that validates differently than much of the rest of the network).  This is why Satoshi did not support alternative implementations.


Write all the "documentation" you want, and put as many comments into the code as you want.  But do not use the word "specification" because nothing except the code can qualify for it.

+1
low level detailed documentation is useless and could be dangerous. but a high level diagram can be very useful for a quick refresher as to how a particular module dose what it dose, and more importantly, when your talking to someone about making changes a good diagram will felicitate communicating ideas.

Well, you must admit that they have a little bias: they can all read C++ fluently. Such a documentation would be useful precisely for those who can't. But Armory guy makes a good point:


Maybe simply not calling it "specification" would be enough to avoid most problems raised here.

I read C++ fluently (and have been doing hardcore C++ development since the mid 90's) and although I do agree that "what we have is a spec written in C++" I do not think we really want this to be the case in the future (even C++ itself has a "spec" - ISO/IEC 14882:2003 currently).

Transitioning from "C with Classes" to C++ took many years and it took many more for vendors to get "up to spec" (Microsoft being one of the main lagers until Herb Sutter joined them).

The same "slow" process will need to take place with Bitcoin as I don't think having "source code" as the "specification" (written in any programming language) is the correct (or desirable) way that things should be done (however I do appreciate that until the "source code" has any quirks that the don't make sense to put in the "specification" removed over time then the "specification" will also have to include them).

Algorithms are not described by "source code" and neither are any of the major internet protocols or programming languages (that aren't privately owned) - I can't see why Bitcoin *needs* to be an exception in the long term (although I do agree that in until a spec that matches the way the "reference client" behaves perfectly is written it cannot take precedence).

I guess the main thing I disagree with is that "it is useless/pointless to even try" (and BTW C++ had a "draft spec" for many years before finally becoming approved and adopted as being "standard").

This is not something that can or should be "rushed into" but something that like Bitcoin itself would need to evolve over time.

Yep, maybe the wording is the real problem here. But I still think, that it is better to have a documentation of the protocol with all its bugs and quirks, that have to be recreated for a successfull alternative implementation than forcing alternative client devs to read carefully through the satoshi code. The probability of missing some important quirk/bug in the latter case is IMO much higher. Maybe we have a case of "Curse of Knowledge" here, where some people are so familiar with the satoshi code, that they can't imagine other people finding it hard to understand?

Of course such a documentation has to be maintained, which easily could become a fulltime job, because it's rather worthless if it's outdated. On the other side, if someone would go for this and core devs would support this endeavour by pointing out errors and differences to the satoshi client, it could be of great value. And finally even the satoshi client would profit, because all this bugs and quirks could be evened out (or made part of the final specification).

It's understandable to say any document would need to have plenty of warnings about non-conformity with the network. And yes, you can't rush something like this.

And since there are people afraid about this, why does this wiki page still exist as it is? https://en.bitcoin.it/wiki/Protocol_specification

That's obviously no where near a complete specification of the protocol. That page should really just be called "Bitcoin messages" and then some of the irrelevant stuff should be moved elsewhere.

Because people like you noticed that it isn't quite right, but didn't bother to change it?

I stay away from wikis. They are awkward to use and whenever I change anything, it gets turned back almost instantly, so what is the point?

I completely agree that more documentation is better, as long as you don't fall into the trap of "if the documentation says it it MUST be true."

Suggestions on how to make the wiki better welcome (there is a discussion elsewhere about banning certain people from the wiki because they have a history of starting edit wars, which I would support, and discussion on the Foundation forums about making the wiki infrastructure independend of Mt. Gox, which I also support). Or volunteers to move technical documentation from the wiki to someplace better also welcome.

That is all fair enough. I do agree that such documentation should have clear warnings and describe the risks, and not pretend that the documentation overrules the active software. But I do think there being documentation is better than none. I also think more documentation of the source code is a good idea, perhaps using a doxygen syntax and separating prototypes/definitions from implementations because at the moment it's a bit all over the place.

+1
From a business perspective, I suppose the bitcoin story sounds a lot better if one can say something like:

There is at least one high level documentation of the protocol (wiki run by the Bitcoin Foundation or else) AND any bitcoin client implementation can be tested for compliance with a single reference implementation called the Satoshi client.
The Satoshi client is open source, regularly updated, peer reviewed and provides the basis for a self-certifying ecosystem.

Through the reference implementation, bitcoin can free online merchants and payment processors from the burden of costly certification requirements imposed by proprietary, legacy payment schemes.

This.

There are (I would assume) many things learned along the way in any protocol development, but what lives on is the solution, not the reasons that made it that way.  For example, in reading the network protocol handling of the 'addr' message, there are what seem arbitrary choices for different time periods, selection of address to respond with, etc.  I'm certain there is a reason for each and every one of those, and if you've been with the project from the beginning, you'd know them.  However, newcomers have to reverse engineer the thinking of the developers to get at this.

This isn't a complaint, by the way, just an observation that as a project grows beyond a certain size and age, some things need to get written down that were previously "tribal knowledge amongst the devs".  An IETF-style protocol specification would be useful.  (And I'd be willing to help in some capacity.)

By the way, to answer the original question... about a year ago, user "bittrick" submitted something resembling documentation about the technical details of Bitcoin:

https://bitcointalk.org/index.php?topic=41718.0

He spent a lot of time in the code and attempted to capture some of the subtler behaviors of the client.  It's not a "specification" but it is an excellent starting point for understanding what's going on.

Back to argument about "specification"... I absolutely support documentation, and especially written nuggets of information that are difficult to deduce (such as that OpenSSL behavior that Mike Hearn mentioned).  I think it's critical to have these kinds of things to help people understand what they're looking at/for.  Especially if they plan to dig into the C++ code afterwards.  And certain parts of the system very well could be specified (perhaps much of the networking protocol).  But not all of it, especially the script-validation engine.

On the topic of OpenSSL... is it possible to reimplement the OpenSSL behavior, or at least extract it from where it is currently, and have it hardcoded into the Satoshi client?  Or rather, does the license allow you to redistribute the OpenSSL project/source with the project so that it can be static compiled?  I guess you only need a .a/.lib, but I'm not sure what the OpenSSL license is like.  That would certainly resolve any issues related to dynamic linking to system libraries that might be different versions on different computers.  It would make the project immune to system library upgrades, which is probably better for security anyway (an attacker could simply swap out the system OpenSSL with a different version having a known bug that he could exploit, and even a diligent user wouldn't notice because the signatures and checksums on the Bitcoin-Qt source distribution are correct).

I have thought about doing this on more than one occasion:  Creating a libcrypto_satoshi snapshot with the bignum, DER, ECDSA etc. bits we currently use.  This would work around an IMO incorrect exclusion of ECDSA from Fedora's openssl package, as well as "freeze" certain bits that are exported globally via the bitcoin blockchain.

But as always, there are costs to maintaining a fork.  It inevitably gets less eyes, use and maintenance than the main fork.

Yup. Without a spec and conforming code, you're not doing software engineering. You're just a code monkey wielding a roll of duct tape.


Agreed. No good reason not to do this.

This absolutely needs to be done with a high priority. OpenSSL changing at all at this point would require recertifying the new version of OpenSSL anyways against older satoshi code + implementations, correct? Or is there some version pegged already as 'Satoshi's version of OpenSSL'? Or have we just been lucky?

I'm sure some people trusting their bank accounts to the strength of OpenSSL's ECDSA implementation would have great interest in any code changes upstream - you would probably hear 'how does this affect Bitcoin?' pretty quickly.

IMO, the cost of maintaining a fork is much lower than the potential cost of having an OpenSSL library delivered with a certain distro introduce a split because of a bugfix, code cleanup, or optimization.

At least on the Mac version, the OpenSSL library is packaged with bitcoin. If it wasn't, that would be alarming.

I would absolutely recommend against forking openssl, for reasons that jgarzik gave, and more.  If a day ever comes when we must, then we must.  Until then, just because we can does not mean that we should.

What we need is a reference implementation. A program which is obviously correct. The current implementation is a bit of a mess (DoS checking code inside of "class CTransaction"? Give me a break). Don't get me wrong, even the better written C++ implementation would arise suspicion (see the Underhanded C competition to see what I mean). What we need is a higher level of abstraction, at least for the code dealing with validating blocks and transactions. No need to put the IRC code there.

I think that a reference implementation written in Haskell would be a big plus. This could be used in the official client (even if the GUI/IRC/etc code is written in C++) and would make it easier to trust the program. I started toying around in translating some of the C++ code (https://github.com/arturh/hbitcoin) but there's a lot of work left, this could definitely use some help.

I think this would:
  - make the inner workings of what really matters crystal clear (as long as they were willing to learn some haskell)
  - attract math-heads to giva a look at the code

When I started in computing in the seventies, it was mosltly trying and failing, after that we got a development standard of 2 shelf meters, after that a binder, after that a thin brochure, and now we are back to trying and failing, which works best, after all.

Ah, the obligatory visit from a silver-bullet salesman. This year's it is arturh selling Haskell. Last year it was Vandroiy selling aspect-oriented programming.

https://bitcointalk.org/index.php?topic=30200.msg614861#msg614861

It is fun to re-read the posts from late 2011 entitled "Request for Standarization". The more things change the more they stay the same, or in its original French "plus ça change, plus c'est la même chose".

Check out how etotheipi changed from 2011 when he was "out-of-core" to 2013 when he is almost "in-core".

Check out how my lame attempts at satire or Commedia dell'arte (http://en.wikipedia.org/wiki/Comedia_del_arte) didn't change.

If we can't get an official Bitcoin standard can we at least find an official Bitcoin core development jester?

Mike, I agree with all of your technical points, and I agree that the Satoshi client code is and always will be "the spec".  However I'm not sure I'm comfortable with the sociological conclusions you're drawing, especially the whole implementation-diversity-is-unavoidably-bad angle.

I'll set that aside for a moment and just comment on this:


SPV is less risky precisely because it trusts the difficultywise-longest chain regardless of transaction validity.  Transaction validity is the only complicated/subtle part; judging difficultywise-length requires only SHA-256, which is about two dozen lines of easy-to-test code.

This advantage can be replicated in full-chain-validating clients by having them watch for any invalid chain (https://en.bitcoin.it/wiki/Invalid_block) which is difficultywise-longest by more than the confirmation threshold.  If such a situation arises, activate safe mode (https://en.bitcoin.it/wiki/Alerts): halt all activities except possibly mining.  I assume that the kerfuffle over "losing money" isn't about mining revenue.  Safe mode can be automatically deactivated if a valid chain once again becomes difficultywise-longest.

I suppose clients that do this are still "second class" in the sense that if you find a chain-splitting bug you can get them to pause until a human intervenes.  But it's definitely a distinct security class compared to either SPV or bitcoin-qt.

Miners are the ones most exposed to losing money! At some point DeepBit accidentally got split onto a side chain because of a database corruption and he lost many thousands of dollars :(


There are hybrid proposals floating around for SPV clients that, on receipt of a "proof of problems" from a node would do enough work to fully verify the chain. The proof would contain enough data to help the SPV clients figure out the rule violation without needing a full database. It could work, somebody would need to try it.

If that's what you meant, then I think it takes the discussion a big step forward: done properly, alternative implementations place at risk only the mining revenue of people who choose to use those implementations for mining.  This is a lot easier to agree with -- the risks of implementation diversity are limited to lost mining revenue and are a mining concern, not a concern for general users.  But I don't think that's the impression people got from this thread so far.



Sounds complicated… simply noticing that more than half the computing power is devoted to a ruleset that doesn't match your own ought to be alarming enough!  I'd worry that attempting automated recovery from such a situation could open new attack vectors.

No. I'm not sure how to make this any clearer. All participants are put at risk by chain splitting bugs. As it happens the direct financial risk to miners who use that implementation is the largest, but obviously, if you split a significant part  of mining power off onto a side chain the cost of a double spend drops significantly, which increases the risk for all other participants as well.


It's not simple to notice that the ruleset has changed if you aren't actually validating those rules! But if you do notice, then picking the correct chain should be a fairly simple consequence of the proving process.

I think it is definitely worth for full nodes to have some sort of safe mode triggered by detection of longer chains. It's simple enough and the problem becomes a disturbance in operation, rather than a problem of forking. And indeed mining software is most critically important when it comes to correct validation.

Perhaps you could give an example scenario where a non-mining user loses money as a result of using a fully-validating client with long invalid branch detection (https://bitcointalk.org/index.php?topic=134318.msg1441171#msg1441171)?  That would definitely make it clearer (at least for me).



So, to restate, it seems like you're claiming that alternative fully-validating implementations create only two risks: (a) miners that choose to use these implementations may lose revenue and (b) the network hashrate might drop, possibly by 50% at the very most.  I actually have reason to believe that the latter will self-correct immediately (and in any case that the hashrate is already plenty high to prevent double-spends, by more than a factor of two) but before proceeding with that I just want to get this straight: are there other "people will lose money" outcomes you are concerned about?  Or is this only about mining revenue and network hashrate?

Sorry if I am being picky and obtuse here.  But I keep seeing these extremely broad comments about "people will lose money"… while there may be some element of truth to that I think the reality is very, very much narrower than that connotes.  So please forgive me for probing the details.

Some kinds of bug can cause a node to be permanently forked onto a side chain, which is essentially a DoS against that business. If you have a way to DoS a business you can try extorting money from them, things like that. There are a variety of ways.

The wisest man on the board.

No, they're only in safe mode until a human intervenes.  You can't deny service, only interrupt it once.

This won't work for extortion -- "send me money or else I'll <what goes here?>"  Usually "stop flooding you with traffic" or "not flood you again" goes in the blank, which is what makes DDoS extortion work.  But to trigger this condition you need to have found a zero-day behavior-splitting bug.  To use it the splitting transaction must be mined into the chain, which advertises the exploit to the whole world including the alternative implementation's dev team -- it's a self-reporting bug and the exploit is one-time use only.  In other words, antifragile (http://en.wikipedia.org/wiki/Antifragile:_Things_That_Gain_from_Disorder).

The Satoshi client already has this "problem", by the way -- it reacts to unusual conditions by going into safe mode.  If this really were concern, you should be more worried about people finding bugs in the Satoshi client that put it into safe mode and somehow extorting MtGox.

AFAICT the situation is still no risk except potential loss of revenue by those who mine with it.

Yeah, but as I said before, there are services where being offline for even a few hours can lead to huge losses. DeepBit used to be in this category, I assume most large pools are similar.

Zero-Day exploits also tend to have somewhat short lives, yet they still have value.

Hrm, I thought I was pretty clear about the fact that loss of mining revenue was a risk (the only one, in fact).  I've yet to see a non-mining example.

Wow, five days after I wrote this:


...it actually happened:


All the 0.7 Satoshi clients just went into (https://github.com/bitcoin/bitcoin/blob/1a9ee5da327d8079a297ad292a1c16745b75df91/src/bitcoinrpc.cpp#L1064) safe mode when they saw (https://github.com/bitcoin/bitcoin/blob/1a9ee5da327d8079a297ad292a1c16745b75df91/src/main.cpp#L2944) the 0.8 clients build a longer chain.  And the world didn't end.  Some miners (probably including me) wasted, in aggregate, 625 BTC worth of hashpower (https://blockchain.info/en/orphaned-blocks).  They'll get over it.

So, I'll say it again regarding alternative fully-validating implementations: no risk except potential loss of revenue by those who mine with it.

Friendly reminder: no matter what client you use there are rare situations which may lead to you being unable to process transactions for a few hours at a time.  If your business can't handle this situation maybe bitcoin isn't right for you.

I fully understand all the reasonable arguments made by all the devs made earlier in this thread, but come on.

We just experienced a chain fork due to a "failure to configure" a 3rd party library (BDB) ...

My point is: however convoluted the must-be-kept-forever bugs and rules are, however difficult the process of extracting the spec from C++ would be, do you REALLY wanna keep running the whole block chain on top of a monoculture of identical clones built from the same convoluted code base ?  

This would be like the entire web running on the same identical version of Appache's httpd and every time a Russian or Chinese hacker discovers a bug, all the websites in the world would become vulnerable to be hacked, defaced or shutdown.

Guys ... it's time to grow up and extract the spec from C++, then target that spec for 1.0 release, it's that or keep trusting ever increasing amounts of wealth to an army of "perfect" clones, seriously ...


Cheers ...

+1

I went through the pain of extracting the spec from several code bases to pour it into my implementation, that started as a way to study the system for my own intellectual curiosity. After months of work in my free time I decided that it might have value for others and open sourced it.

I received some help and much more FUD and warnings for attempting to do the undoable by definition. I comprehend the arguments presented to be cautious to replicate every even untested "feature" of the Satoshi hairball and to look out for the unknown. I took them seriously, worked hard and am not done yet. Yet, I remain confident that capturing the behavior of the implementation is possible to the extent, that it can be used with similar confidence one is forced to put into any software that manages serious money.

It might be that my implementation is key to nothing more than that I understand the depth of the protocol. I wish that the collected knowledge of developers were more accessible to those after us. Unfortunately we all work hard toward goals that are apparently not aligned to this, at least because writing specs and tests is neither sexy nor rewarded.

People using an implementation or even upgrading to a new version of Satoshi know that there is a risk and they will not use an implementation or version if they decide the risk is above their appetite for the return it promises. Having standards, specs and tests is not a guarantee, but hey, they are what human came up with for similarly hard and potentially costly problems.

Can you imagine a bitcoin economy expanding by magnitudes served by a single implementation maintained by a chosen couple in a single version that fits all purposes?

Let's rather find a framework and funding to build the documentation and test suite that enables us to handle this protocol more confidently.

This mostly shows how a specific implementation of a spec can make it really hard to even figure out what the actual spec is that it is attempting to implement.

If we write an actual spec it does not matter whether verifying some prespectoric span of blocks from the dark ages needs to use some ancient ritual mumbo-jumbo to verify that it is indeed a span of blocks that totally violates the spec due to violating the spec having been superstitiously worshipped as heroic back in the dark ages. The spec will make it clear that in the dark ages, the blockchain was totally broken, which led to a shaman caste that made dark and ugly sacrifices to propitiate the invisible hand that brings the cargo upon which they prospered.

The number of locks in the Berkeley DataBase seems a modern example of the shamanic cargo-cult approach too. When you notice that some to whom the cargo is the all-important thing, the raison d'etre for the cult whether or not for the experiment it purports to administer, seem to consider that to be part of the spec instead of a clear bug, it becomes clear that the implementation cannot be the spec: the implementation clearly specifies in capital letters that blocks can be up to one megabyte in size, but clearly neglected to ascertain exactly how many BDB locks are actually in real life required in order that blocks of that size be incapable of needing more locks than are configured.

There we have a clear case of the spec, the in capital letters constant specifying the max block size the functions and classes and such are intended to enable, not actually being met by the code intended to meet it.

-MarkM-

I would prefer to assert that the reference implementation might not itself actually meet the spec, it is merely the closest thing we have so far to a correct implementation of the spec.

We already do not nitty-gritty verify blocks prior to the most-recent checkpoint, so all we need to be able to derive from pre-checkpoint spans of the blockchain is the unspent outputs, isn't it? Is there anything else in that midden of ancient archaeological relics that we need?

About the negative signatures thing, has OpenSSL already turned them positive by the time they get enshrined in the blockchain, or are they in the blockchain as negative?

-MarkM-

This still seems like superstition.

You are basically stating that if the specification specifies that on zero day all your money is forfeit, then sorry, all your money is forfeit, look at the specification (the code) and see right there, the exact code the zero day hacker used? Its in the spec, so the hacker was correctly using the system and correctly took all your money exactly as the spec intended him or her to.

Bullshit.

Before even getting to the protocol specification part of the docs, the requirements specification should specify that zero day exploits are not intended to be part of the spec, regardless of which implemention or how many implementations, or how many users use such implementations or how much money rides on them.

So I guess top of the protocol spec should say "first, read the requirements specification carefully..."

-MarkM-

I'd prefer "The Satoshi client is open source, regularly updated, peer reviewed and provides the basis for a self-certifying ecosystem, but does not itself actually meet the specification at this time, due to certain bugs errors or omissions that have been discovered, and furthermore may contain more such bugs errors or omissions yet to be discovered. Nonetheless it is currently the most accurate rendition of the spec in implementation form that we currently have, and work proceeds apace on bringing it fully up to spec and correcting all bugs errors or omissions in as timely a manner as the need to allow users plenty of time to upgrade to the newer, more-correct versions - and our development and certification budget - allows."

Or something along such lines.

-MarkM-

I would prefer it to be called a protocol description rather than specification until the majority of the network meets it. 

Sorry if I seem to be emulating MPOE-PR, but... No.

If it is just a description of what the current mess actually does, then the current mess already meets it, and in fact it would be more a case of it meeting the current mess than the current mess meeting it.

We can even start at the code level as I did with the BDB misconfiguration example above, in which I separated out the specified max block size as being a specification then checked the BDB configuration to see whether it actually properly met that specification.

On the other hand... semantics.

A specification is a description. It probably should first describe requirements before then going on to describe a protocol which it intends should meet the requirements but which could turn out itself to not meet the requirements if it itself turns out to be buggy.

It is styled a "specification" to indicate that it is a description of what it intends ought to be, not necessarily a description of something that has actually yet been accomplished / implemented in actual running code.

-MarkM-

(copied from misterbigg's thread, by request)

As Pieter wrote on bitcoin-development list,


Or restated:  The fundamental problem being solved by bitcoin at a technical level, on a daily basis, is the distributed consensus problem (link (http://en.wikipedia.org/wiki/Consensus_%28computer_science%29)).

We fully support the writing of specifications and documentation, which you can see here
    https://en.bitcoin.it/wiki/Protocol_specification (https://en.bitcoin.it/wiki/Protocol_specification)

And changes to the existing protocol are formally documented here,
    https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals (https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals)

Ultimately the operational definition of consensus comes from what the network accepts/expects, not a theoretical paper.  Specification practices are healthy as a manual, human-based method of achieving consensus on network protocol rules.  Alternate client implementations (c.f. heterogeneous environment (http://en.wikipedia.org/wiki/Heterogeneous_computing)) are another good practice.

But the collective software rules are always the final specification, by definition.  That is what bitcoin does, achieve consensus.

A few other observations:

Gnutella had a business and project environment with co-motivated individuals working on a few key codebases.  The reference codebase in bitcoin, in contrast, has one paid developer (Gavin@BF) and a few part time unpaid volunteers.

All the big bitcoin businesses seem to either (a) contribute to BF, (b) use bitcoind without contributing back any testing/dev/specification resources, or (c) do their own thing entirely, not contributing back any testing/dev/specification resources.

Bitcoin is a thing, an invention, not a funded project with a built-in set of professionals paid to ensure full spec/dev/test engineering effort.  If you want something, DO IT.  You cannot expect the engineering resources to do X to magically appear, just because you complained on an Internet forum.

In an unfunded open source project, arguing all day about the lack of full-engineering-team rigor is entirely wasted energy.  Blame the dev team if that is your favorite target, that will not magically create extra time in the day or extra manpower to accomplish these extra tasks being demanded by non-contributors.

The time spent whining about what an unfunded effort fails to do could be better spent, say, creating a test network of full nodes running all known bitcoind versions, 0.3 through present.  And test, test, test changes through that.