Bitcoin protocol standarization

[lontivero]

Hi,

Everytime I tell some people about bitcoin they ask me "who controls it?". I tell them nobody does it because it is a protocol just like http or smtp. However sometimes I think that that is not true. Of course nobody controls the currency but the bitcoin core team is who takes the decisions about the "reference" implementation, what means it is the team that update the protocol even when there are other full-node implementations out there.

In fact, if we have a "reference implementation" is because the thruth is in the source code instead of being in a specification document and that's also a problem. Moreover, you need to be a top hacker to understand the reference implementation source code because it is a bit messy so, if I am right, the thruth is in a messy piece of code.

These facts don't look very well because it means centralization and unilateral control. And there are other signals here and there, in this moment I can read "News: Bitcoin Core 0.9.3 has been released. Download." in the this forum header. I mean, it is more and more like the "official" node. Oh, the alert message described in the protocol specs looks like a centralised broadcast system (yes, I understand it is okay and is there for a good reason)

I think something should be done in order to improve this situation. I know it sounds awful but we need an agreement system like a IETF for the bitcoin protocol.

Is something similar in the roadmap?

[1715177133]

1715177133

[1715177133]

1715177133

[cbeast]

You must be new here. These questions gets asked about once per week. I'm probably not the best one to answer these, but I'll give it  shot.

Hi,

Everytime I tell some people about bitcoin they ask me "who controls it?". I tell them nobody does it because it is a protocol just like http or smtp. However sometimes I think that that is not true. Of course nobody controls the currency but the bitcoin core team is who takes the decisions about the "reference" implementation, what means it is the team that update the protocol even when there are other full-node implementations out there.

The core team leads the development, but doesn't dictate adoption. If fact, they encourage people to not use new updates unless they are a bug fix, until they are confident it is useful to them. Miners themselves make the choice and some are still using older versions of the protocol. If an implementation is not popular, it will be dropped.

In fact, if we have a "reference implementation" is because the thruth is in the source code instead of being in a specification document and that's also a problem. Moreover, you need to be a top hacker to understand the reference implementation source code because it is a bit messy so, if I am right, the thruth is in a messy piece of code.

Basically you're saying that "math is hard" and I won't disagree. Computers are complex, but I use them. Cars too. There are many versions, I can choose the one I want based on my needs. Bitcoin is holding up against thousands of competitors because it's that good.

These facts don't look very well because it means centralization and unilateral control. And there are other signals here and there, in this moment I can read "News: Bitcoin Core 0.9.3 has been released. Download." in the this forum header. I mean, it is more and more like the "official" node. Oh, the alert message described in the protocol specs looks like a centralised broadcast system (yes, I understand it is okay and is there for a good reason)

I think something should be done in order to improve this situation. I know it sounds awful but we need an agreement system like a IETF for the bitcoin protocol.

Is something similar in the roadmap?

We don't really need the Bitcoin alert system at all, it's just a convenience. We need better watchdogs with more powerful analytic tools. They can develop their own alert systems.

[lontivero]

Thank you, @cbeast. Yes, I am new in the development sub-forum and I didn't know this is a common question ;(

Is some people doing something about it?

[doge94]

The issue in creating a standard protocol is documenting all the bugs in the current reference implementation. Until this happens most other implementations will fork the chain.

[gmaxwell]

The issue in creating a standard protocol is documenting all the bugs in the current reference implementation. Until this happens most other implementations will fork the chain.

No, they'll continue to fork the chain regardless; many (most?) of the previously observed implementation behaviour discrepancies have been in clearly well documented behaviour; or even in behaviour triggered on the conformance testing harness. Turns out that getting radically different software to behave in a manner which is absolutely identical under all conditions is quite hard.

[BlindMayorBitcorn]

You must be new here. These questions gets asked about once per week. I'm probably not the best one to answer these

I disagree.

You, sir, are true gentlemen.

[coldcryptos]

I'll take one perfect world, please.

[cbeast]

I'll take one perfect world, please.

Hold that thought.

[altcoinex]

I think cbeast covers this topic very well, but I wanted to add a Consideration:

In fact, if we have a "reference implementation" is because the thruth is in the source code instead of being in a specification document and that's also a problem. Moreover, you need to be a top hacker to understand the reference implementation source code because it is a bit messy so, if I am right, the thruth is in a messy piece of code.

It is my opinion that this statement is inaccurate. While it's development path has left it as -- code that by many programming standards might be 'messy' -- It is very far from being unreadable. I think the reputation for it being such a disaster has been greatly exaggerated for specific effect in a few interviews, leaving those unfamiliar with it to assume it far worse off than it actually is. Any developer with basic knowledge of software engineering, c++, and cryptography concepts are all that is required to read and understand the vast majority of the reference code. The few advanced programming concepts implemented in the reference client or design plans typically have clear explanation available, and while may take a moment to fully grasp are well within reason for an average developer. I think it is important to not let the rumor of bitcoin's code being 'a mess' demotivate people who would otherwise be interested in it's development..

[CIYAM]

The issue that @gmaxwell has mentioned is that any alternative implementation that fails to do something that the current implementation does (which may have never been even witnessed before) would result in a fork.

So the unfortunate situation is that Bitcoin is not a protocol like HTTP but instead is the behaviour of a specific C++ program (who's every currently unknown *quirk* would have to be replicated in any other implementation).

The only way you could create a Bitcoin equivalent in another language would be to have something that behaves identically at the binary level (and that would most likely only perform slower than Bitcoin does).

[gmaxwell]

One of my longer term hopes around the refactoring of Bitcoin core into a separate library for consensus is that it allows us to compile the consensus parts into a simple bytecode with a narrow interface which can be executed by an easily implemented and testable virtual machine, so every implementation can just use the same bytecode and be confident that they'll be consistent.  Considering how hard it has been to get people to understand the unusual requirements of consensus, it may turn out to be hard to get people to accept the performance hit (and to avoid doing crazy JIT stuff which breaks the obvious-correctness of the approach).

[lontivero]

The issue that @gmaxwell has mentioned is that any alternative implementation that fails to do something that the current implementation does (which may have never been even witnessed before) would result in a fork.

So the unfortunate situation is that Bitcoin is not a protocol like HTTP but instead is the behaviour of a specific C++ program (who's every currently unknown *quirk* would have to be replicated in any other implementation).

The only way you could create a Bitcoin equivalent in another language would be to have something that behaves identically at the binary level (and that would most likely only perform slower than Bitcoin does).

That's exactly what I mean.

I've heard people saying that one of the developer's perversion is that they want to rewrite all in another programming language, everybody wants to have a library available for being used in their favorite language. It is easy to understand why devs want to use a confortable language for their projects so, even when that could be too hard in the case of bitcoin full node, new implementations are going to appear and that could be a problem.

It is clear that bitcon is not a commnon thing and there are lost and lots of details to have in mind and tests (like security). However, if I am rigth and new implementations are developed then the bitcoin core team will need to do something.

One of my longer term hopes around the refactoring of Bitcoin core into a separate library for consensus is that it allows us to compile the consensus parts into a simple bytecode with a narrow interface which can be executed by an easily implemented and testable virtual machine, so every implementation can just use the same bytecode and be confident that they'll be consistent.  Considering how hard it has been to get people to understand the unusual requirements of consensus, it may turn out to be hard to get people to accept the performance hit (and to avoid doing crazy JIT stuff which breaks the obvious-correctness of the approach).

 

I think that separate library (or specialized libraries) is an very good idea. I don't understand why a VM could be required.   

[CIYAM]

I think that separate library (or specialized libraries) is an very good idea. I don't understand why a VM could be required.   

I agree with this - Linux has had a long history of libs written in C that end up being available for other languages - why the need to go against that and create a Java thing (as I assume is being suggested)?

[TierNolan]

One of my longer term hopes around the refactoring of Bitcoin core into a separate library for consensus is that it allows us to compile the consensus parts into a simple bytecode with a narrow interface which can be executed by an easily implemented and testable virtual machine, so every implementation can just use the same bytecode and be confident that they'll be consistent.  Considering how hard it has been to get people to understand the unusual requirements of consensus, it may turn out to be hard to get people to accept the performance hit (and to avoid doing crazy JIT stuff which breaks the obvious-correctness of the approach).

This VM would need to be able to have enough memory for the UTXO set.  All the crypt functions would presumably be opcodes.

Accepting blocks into the memory pool presumably doesn't need to be handled by the VM, since that won't cause a fork.  Each miner would check their own blocks before mining.

With UTXO set commitments, block verification could be more self-contained.  The VM would run on a block + Merkle path based metadata + the header chain.

[erik777]

I agree that it would be a good idea to define a language agnostic spec for the communications protocol, and I haven't heard a single valid argument against it.  If you are against defining an open standard, at least be honest and just say no one is motivated to do it or share whatever concern you have would come out of such an effort.  Honestly is ok.  But, obfuscating lack of motivation with "the code is OK" is just a distraction that undermines the value others perceive.  Regardless of whether or not someone currently seasoned in C++ finds it OK the way it is, an open specification would provide value to those of us not so motivated to pick apart a C++ program just to understand the communications going over the wire.  This could also enhance peer review by increasing the respective peer review audience, and possibly foster more innovation around the network.    

EDIT: The next thing I read after posting this happened to be an excellent real example of the value of better peer review of the protocol:

https://bitcointalk.org/index.php?topic=359582.0

Can someone tell me what a VM means in this conversation?  I certainly doesn't sound like a virtual machine that runs an OS, nor does it sound like a JVM, unless I'm just not connecting the dots.    


  

[gmaxwell]

I agree that it would be a good idea to define a language agnostic spec for the communications protocol, and I haven't heard a single valid argument against it.

The communications are already speced out in reasonable enough detail that multiple people have implemented talking to it from just the documentation. (e.g. from he developer guide and Bitcoin Wiki/

If you are against defining an open standard, at least be honest and just say no one is motivated to do it or share whatever concern you have would come out of such an effort.  Honestly is ok.

You're accusing people of being dishonest?  Really?  Try engaging in a little professionalism. Next time someone gives you an explanation you don't understand, you could try not accusing them as dishonesty as a first step. It certainly doesn't motivate people to spend their time educating you on a subject about which you are clearly ill-informed when you take that approach.

the value others perceive

Sometimes people perceive value incorrectly. It's not that the code is "fine" its that for the normative operation of a consensus system it's all that actually matters. Additional informative documentation is useful too, and people have written a fair amount of it.

EDIT: The next thing I read after posting this happened to be an excellent real example of the value of better peer review of the protocol:

Funny, I don't see how thats an example of better peer review of the protocol at all: It requires only an understanding of the system at the level in the original Bitcoin whitepaper (which has been available since 2009). OTOH, I've used that work before as an example of how academia needs more peer review from industry-- because that design breaks decentralization which is the whole point of Bitcoin, and Amiller proposed a very similar design (and arguably better, since there were some additional attacks it resisted) back in 2011 which the work should have cited.

Can someone tell me what a VM means in this conversation?  I certainly doesn't sound like a virtual machine that runs an OS, nor does it sound like a JVM, unless I'm just not connecting the dots.

A bytecode simulator, like the JVM. One intentionally constructed to be very simple in order to avoid the extreme reimplementation risk in consensus code.

[gmaxwell]

This VM would need to be able to have enough memory for the UTXO set.  All the crypt functions would presumably be opcodes.

It wouldn't: You'd use a authenticated data structure so that all the input could come from an unverified domain. It could fail completely but it would know it... even a not found would be expected to prove non-membership by providing fragments for a search query that failed against the current root.  The UTXO doesn't even need to be committed in the blockchain for this to work, e.g. the bytecode machine could just keep track of the root itself.

(If you wanted to be especially paranoid you could use path-oram to make sure that there was no visible access pattern or data sequence outside of the sandbox thta was common on multiple hosts which could cause a systemic failure).

[hhanh00]

I agree that it would be a good idea to define a language agnostic spec for the communications protocol, and I haven't heard a single valid argument against it.

The communications are already speced out in reasonable enough detail that multiple people have implemented talking to it from just the documentation. (e.g. from he developer guide and Bitcoin Wiki/

I can attest that it's possible to write a client without looking at the reference code. @erik777 - what do you think a standardization process would add? Bear in mind that HTML 5 doesn't have an official spec and is still used by most of the internet.

[erik777]

gmaxwell, i was replying to one guy's statement, but didn't want to be too obvious by quoting him.  here's what he said that I didn't think the OP deserved:

"Any developer with basic knowledge of software engineering, c++, and cryptography concepts are all that is required to read and understand the vast majority of the reference code."

The documentation you linked to, on the other hand, is very useful, and reads a lot like a spec. 

Now that you pointed out the VM definition here, that is ironically what I have been thinking of developing for years for P2P, but with a broader context, a re-usable a service layer.  I'd like to create a P2P platform that can be used to create all kinds of novel things. 

I like the core functionality that retroshare has.  but, unfortunately, it's not spec'd to be a VM'd P2P platform, although they have tried to make it plug-able and have built good re-usable functionality like its cache services.  I'd like to help build a platform that can provide vast capabilities -- building blocks that can be used to create higher level craziness limited only by imagination.  A VM could provide standard access to the building blocks and simplify development of p2p apps, while protecting the machine running it.  To promote rapid adoption, I'd like a reward system for those running nodes dedicating resources, with perhaps a coin dedicated completely to machine to machine task interchange and resource sharing.   

[erik777]

I agree that it would be a good idea to define a language agnostic spec for the communications protocol, and I haven't heard a single valid argument against it.

The communications are already speced out in reasonable enough detail that multiple people have implemented talking to it from just the documentation. (e.g. from he developer guide and Bitcoin Wiki/

I can attest that it's possible to write a client without looking at the reference code. @erik777 - what do you think a standardization process would add? Bear in mind that HTML 5 doesn't have an official spec and is still used by most of the internet.

That's true.  Yet, good specs do mature to standardization and an open process independent of any one implementation. 

http://arstechnica.com/information-technology/2014/10/html5-specification-finalized-squabbling-over-who-writes-the-specs-continues/

Bitcoin has come a long way in 5 years.  Kudos to all the great innovators and hard laborers who made it happen.  The question is how best to objectively advance it over the next 10 years.  Should a standard drive a reference implementation, or a reference implementation drive the standard? 

That said, you could say there is some hair splitting in this discussion, as bitcoin has naturally taken on traits of an open standard through people's drive to improve implementations and the long-term health and viability of the protocol.  With open discussions, it has been behaving recently like the early days of the IETF.