Bitcoin protocol standarization

[gmaxwell]

The question is how best to objectively advance it over the next 10 years. Should a standard drive a reference implementation, or a reference implementation drive the standard?

As several people have pointed out, only one of these is _technically possible_ because Bitcoin is a globally consistent consensus system implemented by machines that execute implementations and not prose specifications, regardless of whatever would be considered "best" in a world of frictionless spherical cows unbounded by constraints like the possible.  (I am assuming you're using a narrow definition of "standard": An implementation is merely a infinitely precise specification constructed for the benefit of machine enactment, after all. That you present it as a choice must mean that you're defining a "standard" as something inherently too imprecise to act as a concrete embodiment itself.)

It's easy for me to suffer some frustration when people continue to just repeat "but standards are good!" without demonstrating an understanding of the specific technical and political considerations in the problem space, and also while retaining a very narrow definition of what can constitute a "standard", "open", or "transparent".

Bitcoin is broken, insecure, and worthless if the machines implementing it are not able to immediately achieve globally consistency. This is a security requirement of the application domain: If the behaviour of the systems and the specification disagree, you _must_ change to match the systems or be left behind and have your funds stolen.  It's not that I don't think standards are important-- I've been involved with the IETF for many years, and attend much of the meetings and contribute to a number of working groups myself; but Bitcoin has specific technical and social requirements that make standards documents weak (even, at times, counter productive) tools when talking about the behaviour of the established production system.  Unlike TCP, SMTP, HTTP, etc. Bitcoin doesn't simply need to be somewhat inter-operable: when it comes to consensus behaviour all of the systems that constitute the network must behave in deeply identical ways. If an attacker can expose a widespread inconsistency in behaviour, however small or burred, the system faults globally and funds can be stolen even from people whos own systems are not having problems.  If the system were not demanding of consistency Bitcoin would be insecure in other ways instead.  This is just a technically harder challenge than faced by other protocols.

There is another layer to this--  Standards process ultimately rest on the politics of mankind. Most standards efforts are remarkably malleable to politics, easily bent by monied interests to their own ends... as elegantly demonstrated by some of the backdoored cryptography thats made its way into various standards, or rent seeking patent traps that required in many other international standards. And even as we speak, various groups who have little understanding of Bitcoin technically or involvement in the ecosystem have been holding workshops to "standardize it". It may well be that the people behind these efforts are completely innocent of ill-intent (beyond, perhaps, a little vanity and self-promotion), but levers that create a point of control will be exploited in the future if they can be exploited; doubly so when the people involved do not understand the risks.

Bitcoin was created to build a system whos behaviour was more deterministic than is otherwise possible in a world where political expediency can sometimes override "constitutional" guarantees. While it's useful for people to discuss Bitcoin, and nudge it here and there, without the ability to largely elevate itself above political expediency, Bitcoin serves little purpose. I think we should all support efforts to communicate better about Bitcoin, including building better descriptive text to build an on-ramp to a more complete understanding and aid people who only need an informal view. I've contributed time to help improve the developer's guide, help publish BIPs, etc. But there is authority and autonomy risk from crossing over into the realm of proscriptive text that ultimately seeks to control the system rather than the system controlling the text. This is why, for example, the BIP process as a whole is intended to be rather descriptive and there are BIPs for a number of things that quite a few experienced engineers in the space consider to be seriously ill-advised.

Another way I could present your question is "Should a human readable but inherently ambiguous definition requiring politicized interpretation and adjustment drive the definition of Bitcoin? or should Bitcoin be defined by math... even though math is not always easy to read?"

I think the Bitcoin community is smart enough to realize this, and not be deceived by people peddling a political and inherently unrepresentative standard process "driving" Bitcoin to change out from under them and potentially against their interest as "transparency" and "openness",  when it's really is anything but... making something more readable at the cost of being well defined may not be an improvement to transparency at all. A lot of contemporary standards processes could learn a lot about openness, equality, and transparency from Bitcoin. Even inside the IETF it's not too uncommon to hear the sentiment that the IETF has somewhat lost its way, now too influenced by politics and personalities than by running code. In the context of conventional standards this may just be a source of unfortunate inefficiency and avoidable failure, but in the context of Bitcoin the risk is more fundamental and so we must step carefully.

When someone is interested in gravity we can describe it informally, but when they want or need to really understand it-- in some way that has consequences-- we don't pretend that we can shy away from precise formal descriptions-- to do so would be to undermine engineering, and it would result in people dying. We certainly don't seek to "specify" it with prose and expect the universe to obey. Nor should we with Bitcoin. Bitcoin is the physics of a virtual universe, if you will. And like actual physics there is no replacement for a precise, formal, description--- especially if you're in the business of building the machinery of that universe yourself.

I'd like to help build a platform that can provide vast capabilities

What I was talking about there isn't about plug-ability or extensibility; it's orthogonal to it and wouldn't itself offer any additional flexibility. The motivation there is being able to achieve absolute consistency in the parts of the system which MUST be absolutely consistent while still allowing for multiple implementations by reducing the surface area where things could go wrong.

The kind of flexibly you're thinking of might also benefit from some of the same tools... but a consensus system like Bitcoin has some pretty incredible overheads that preclude a good bit of application space by their very nature.

[spin]

Wow, great post.  Learning much from it.  Perhaps worth perserving somewhere on the wiki?

[lontivero]

I've been thinking about all this, about a standard, the code and the available documentation and I have to admit that there were several misunderstanding from my side.

1) Source code is the more detailed specification you can find. This is always true.

2) The reference implementation is not messy code, in fact it is pretty clear now. Last time I took a look at it was in 2011/2012.

3) Available documentation is very good. I didn't know about the developer guide before this post and it is rather detailed (the wire protocol and other low level data structures are well described in other places).

I think that my misinformation stole time to @gmaxwell and others however maybe it is a good thing because other developers could have the same questions about standards.

I'd love to have a library (a .dll) to link in my project with all (or at least the hard part) what is needed to implement a node and I am sure it is the way to move forward. If a VM is a better option I would like to understand why.

   

[CIYAM]

I am also keen to understand the VM suggestion and why @gmaxwelll thinks that would be better than just creating a C/C++ lib with wrappers for other languages to use.

Are we talking about a new VM that is just for Bitcoin?

[theymos_away]

With the sort of VM that gmaxwell is talking about, all of Bitcoin's consensus rules would be specified in a simple and exact programming language (something like Bitcoin's Script, maybe), and then every full node would include this consensus specification verbatim in its code and directly interpret it when checking things against the consensus rules. The specification would not describe or dictate the rules -- it would literally *be* the rules. This ensures that all full nodes have exactly the same behavior, but since each implementation designs its own VM/interpreter for processing the specification, different designs for the lower-level storage operations (etc.) would be possible.

This seems to me like a very good/interesting idea.

[CIYAM]

I very much doubt that Bitcoin's script could be of any use as it is not Turing complete and although actual Bitcoin transactions are handled by it things like re-orgs cannot be handled by it at all (nor could it handle any of the p2p stuff).

[theymos_away]

I very much doubt that Bitcoin's script could be of any use as it is not Turing complete and although actual Bitcoin transactions are handled by it things like re-orgs cannot be handled by it at all (nor could it handle any of the p2p stuff).

That's why I said "something like". Script would indeed be completely unsuitable. But the specification language would have a similar purpose, and maybe a similarly-simple stack-based language would be appropriate. (I don't know much about how something like this should actually be designed, though.)

[CIYAM]

... maybe a similarly-simple stack-based language would be appropriate....

Personally I would hope not - I was involved in a hugely funded project to build a stack based VM system before Java existed in the early 90's and as a programmer (who worked on that project) I can say that I am no fan of such stack based languages (they are far less elegant than using a more traditional CPU style approach).

[2112]

Are we talking about a new VM that is just for Bitcoin?

It wouldn't be "just for Bitcoin", but would be "where Bitcoin-related programs are particuarly short". I would presume that it would have (at least some) 256-bit registers and the elliptic curve operations would be single instructions.

For the similar projects please skim through the zk-SNARK paper to get a high-level overview of the Tiny-RAM machine (and related GNU C compiler back-end) implemented there.

https://eprint.iacr.org/2013/507.pdf

The goals are also somewhat similar to the goals of SystemC, a C++ subset/superset.

http://en.wikipedia.org/wiki/SystemC

The main objective is that the architecture must be so precisely defined as to allow automatic synthesis of equivalent digital circuits and machine-generated proofs about some programs expressed in that language/bytecode.

as it is not Turing complete

It has function call operator (used in P2SH) that allows implementation of iteration through recursion, so this old chestnut should better die.

[erik777]

The question is how best to objectively advance it over the next 10 years. Should a standard drive a reference implementation, or a reference implementation drive the standard?

As several people have pointed out, only one of these is _technically possible_ because Bitcoin is a globally consistent consensus system implemented by machines that execute implementations and not prose specifications

implementations of what?  machines that communicate in a predetermined agreed language implement a specification, regardless of how that spec is defined, unless it is just one program.  saying that a program implements a spec defines by solely by its own code isn't really an implementation of anything.   

It's easy for me to suffer some frustration when people continue to just repeat "but standards are good!" without demonstrating an understanding of the specific technical and political considerations in the problem space, and also while retaining a very narrow definition of what can constitute a "standard", "open", or "transparent".

People who are pro-standards, or anti-standards, are both extremists.  There are good standards, bad standards, and mediocre standards.  We learn and improve.  How did the good standards come about and avoid becoming bad standards?  Bitcoin is built on many standards which, if you like Bitcoin, you hopefully agree they are good standards to have, such as TCP.   

Bitcoin is broken, insecure, and worthless if the machines implementing it are not able to immediately achieve globally consistency.

True of any communications protocol.  It's a good thing HTTP works, or we wouldn't be here right now discussing this. 

This is a security requirement of the application domain: If the behaviour of the systems and the specification disagree, you _must_ change to match the systems or be left behind and have your funds stolen.  It's not that I don't think standards are important-- I've been involved with the IETF for many years, and attend much of the meetings and contribute to a number of working groups myself; but Bitcoin has specific technical and social requirements that make standards documents weak (even, at times, counter productive) tools when talking about the behaviour of the established production system.  Unlike TCP, SMTP, HTTP, etc. Bitcoin doesn't simply need to be somewhat inter-operable: when it comes to consensus behaviour all of the systems that constitute the network must behave in deeply identical ways. If an attacker can expose a widespread inconsistency in behaviour, however small or burred, the system faults globally and funds can be stolen even from people whos own systems are not having problems.  If the system were not demanding of consistency Bitcoin would be insecure in other ways instead.  This is just a technically harder challenge than faced by other protocols.

It is true there is greater security risk with bitcoin.  But, that doesn't mean that new code preceding specification is inherently more secure than having a spec to define and discuss a change beforehand.  In fact, if you really felt comfortable with the way new code is being written to create new implementations of bitcoin, you wouldn't be proposing a VM to address this concern.  To be sure, I think the VM idea is a very good one.  I'm just using it as an example because you clearly don't believe things are perfect the way they are today and the potential risk of new implementations.  If it aint broken, don't fix it, right? 

There is another layer to this--  Standards process ultimately rest on the politics of mankind....

Politics sure is a fuzzy word in any scientific conversation.  Let's reverse this.  Are you saying that open source projects don't have politics?  Are all developers always in perfect agreement.  That's why I like the early days of the IETF.  That was about as minimal on the politics as you could get.  It was engineers who worked together very well to create new things together.  It wasn't until about the mid 90s that the culture began to change.  To some extent, I think we both agree that culture is always a risk.  But, I believe that risk is just as real without a standards dialog.  As for some of your concerns, that's about WHO is influencing the standards, or what type of person, not the fact that the standardization process is visible and perhaps a bit formal.  That said, I haven't proposed any type of process.  I agree that if one were to develop a process, it might need to be a bit unique for Bitcoin to optimize the primary objectives of Bitcoin: Security, Scalability and De-centralization, with a high bar for any direct impact ability (e.g., voting) to ensure technical credibility and common ground on core principles.  Yet, the non-direct impacting open discussions could prove incredibly value-able towards Bitcoin's objectives.  And the visibility on the process cannot possibly hurt.       

Bitcoin was created to build a system whos behaviour was more deterministic than is otherwise possible in a world where political expediency can sometimes override "constitutional" guarantees. While it's useful for people to discuss Bitcoin, and nudge it here and there, without the ability to largely elevate itself above political expediency, Bitcoin serves little purpose. I think we should all support efforts to communicate better about Bitcoin, including building better descriptive text to build an on-ramp to a more complete understanding and aid people who only need an informal view. I've contributed time to help improve the developer's guide, help publish BIPs, etc. But there is authority and autonomy risk from crossing over into the realm of proscriptive text that ultimately seeks to control the system rather than the system controlling the text. This is why, for example, the BIP process as a whole is intended to be rather descriptive and there are BIPs for a number of things that quite a few experienced engineers in the space consider to be seriously ill-advised.

+1 and I understand your concerns.  Keep it scientific with the core team dedicated demonstrating technical excellence and a common vision.  Note, the latter hasn't always been clear to-date, such as with anonymity and fungibility. 

Another way I could present your question is "Should a human readable but inherently ambiguous definition requiring politicized interpretation and adjustment drive the definition of Bitcoin? or should Bitcoin be defined by math... even though math is not always easy to read?"

Like SSL and PKI? 

You again make it sound ilke a choice between math and politics.  Math and politics will always be there.  The only question is HOW you manage the politics.  Do you hide it and pretend it's not there, because there is so much math?  Or do you limit it in your core charter with checks and balances and hopefully an agreement on guiding principles?     

... in the context of Bitcoin the risk is more fundamental and so we must step carefully.

True, except the "..." part.  hahaha 

When someone is interested in gravity we can describe it informally, but when they want or need to really understand it-- in some way that has consequences-- we don't pretend that we can shy away from precise formal descriptions-- to do so would be to undermine engineering, and it would result in people dying.

ROFL!  Not sure how you think Bitcoin is as complex as physics computations, or how you think people will die if the current political status quo has increased visibility.

We certainly don't seek to "specify" it with prose and expect the universe to obey. Nor should we with Bitcoin. Bitcoin is the physics of a virtual universe, if you will. And like actual physics there is no replacement for a precise, formal, description--- especially if you're in the business of building the machinery of that universe yourself.

Except Bitcoin isn't a virtual universe. 

I'd like to help build a platform that can provide vast capabilities

What I was talking about there isn't about plug-ability or extensibility; it's orthogonal to it and wouldn't itself offer any additional flexibility. The motivation there is being able to achieve absolute consistency in the parts of the system which MUST be absolutely consistent while still allowing for multiple implementations by reducing the surface area where things could go wrong.

The kind of flexibly you're thinking of might also benefit from some of the same tools... but a consensus system like Bitcoin has some pretty incredible overheads that preclude a good bit of application space by their very nature.

I know.  I was digressing a bit into a vision that in itself has nothing to do with Bitcoin.  I do see a new M2M virtual currency being required to grease the wheels of trustless resource and task distribution, developed from the ground up to meet the needs of M2M.  Examples of how requirements would differ from Bitcoin: transactions would be incredibly micro.  Yet, because these are virtual machines in a virtual universe, they could settle debt in much larger time spans, such as once a day, decreasing the load on the primary order book.  It could be a combination of trusted and trustless, as well. 

I appreciate your passion for the success of Bitcoin.  You do bring a lot of insight and I do empathize with your concerns even when we don't totally agree on whether or not the current political status quo is enough to ensure the success of Bitcoin over the next 10 years.