Does a Request for Comments (RFC) for the Bitcoin protocol exist?

[casascius]

I apologize, I should not try to be funny.

I had initially misunderstood your post, thinking it meant my suggestion to split RFC had some element of self-interest involved with respect to my physical coins.  (and while I realize that it doesn't, I actually really do have an agenda when it comes to making physical bitcoins useful in the real world, and really would put effort into bending protocols to accommodate that, though mostly what I have in mind is the variety of physical notes you can print yourself, rather than my coins)

[1713583780]

1713583780

[1713583780]

1713583780

[1713583780]

1713583780

[1713583780]

1713583780

[1713583780]

1713583780

[1713583780]

1713583780

[gmaxwell]

The Bitcoin wire protocol is a cake that can be defined in plain text like RFCs, but the requirement that it has to converge to the same internal state at every node (that is ultimately the UTXO set) is a non-trivial algorithm that likely needs creative ways of standardizing including using programs to define.

An RFC would be perfectly capable of documenting the protocol rules, it would just be very hard to do, and very hard to gain confidence that you got it right.

In RFC6716 we went the route of using source code as the primary normative specification: "For the practical reasons of compatibility and testability, it would be advantageous to give the reference implementation priority in any disagreement.  The C language is also one of the most widely understood, human-readable symbolic representations for machine behavior.  For these reasons, this RFC uses the reference implementation as the sole symbolic representation of the codec."

Opinion's differ on this being the best thing to have done for RFC6716, but the argument applies much more strongly for Bitcoin— where even the smallest disagreements can break the software for everyone in the world, not just the users of the oddball software— than a multimedia codec.

It may be the case that future refactoring, cleanups, and modularization eventually end up segregating the protocol rules part of the reference client into isolated parts that seldom change (only when changing the rules!), and then they'd act (either internally or pulled out and stuck in a digital-dead-tree) as better specifications than we have today.

[proff]

The Bitcoin wire protocol is a cake that can be defined in plain text like RFCs, but the requirement that it has to converge to the same internal state at every node (that is ultimately the UTXO set) is a non-trivial algorithm that likely needs creative ways of standardizing including using programs to define.

An RFC would be perfectly capable of documenting the protocol rules, it would just be very hard to do, and very hard to gain confidence that you got it right.

In RFC6716 we went the route of using source code as the primary normative specification: "For the practical reasons of compatibility and testability, it would be advantageous to give the reference implementation priority in any disagreement.  The C language is also one of the most widely understood, human-readable symbolic representations for machine behavior.  For these reasons, this RFC uses the reference implementation as the sole symbolic representation of the codec."

Opinion's differ on this being the best thing to have done for RFC6716, but the argument applies much more strongly for Bitcoin— where even the smallest disagreements can break the software for everyone in the world, not just the users of the oddball software— than a multimedia codec.

It may be the case that future refactoring, cleanups, and modularization eventually end up segregating the protocol rules part of the reference client into isolated parts that seldom change (only when changing the rules!), and then they'd act (either internally or pulled out and stuck in a digital-dead-tree) as better specifications than we have today.

Algorithms and protocols are typically specified in machine-checkable form using languages such as +CAL/TLA+ and PROMELA. Not C code. The implementation should follow the specification: "a program is an algorithm plus an implementation of its data operations".

[2112]

The Bitcoin wire protocol is a cake that can be defined in plain text like RFCs, but the requirement that it has to converge to the same internal state at every node (that is ultimately the UTXO set) is a non-trivial algorithm that likely needs creative ways of standardizing including using programs to define.

An RFC would be perfectly capable of documenting the protocol rules, it would just be very hard to do, and very hard to gain confidence that you got it right.

In RFC6716 we went the route of using source code as the primary normative specification: "For the practical reasons of compatibility and testability, it would be advantageous to give the reference implementation priority in any disagreement.  The C language is also one of the most widely understood, human-readable symbolic representations for machine behavior.  For these reasons, this RFC uses the reference implementation as the sole symbolic representation of the codec."

Opinion's differ on this being the best thing to have done for RFC6716, but the argument applies much more strongly for Bitcoin— where even the smallest disagreements can break the software for everyone in the world, not just the users of the oddball software— than a multimedia codec.

It may be the case that future refactoring, cleanups, and modularization eventually end up segregating the protocol rules part of the reference client into isolated parts that seldom change (only when changing the rules!), and then they'd act (either internally or pulled out and stuck in a digital-dead-tree) as better specifications than we have today.

Quoted for future reference. I'm not going to bold the "eventually" in the last paragraph for emphasis. It is a raw quote.

I just wanted to point out that gmaxwell is again spreading FUD and false information.

Only the decoder portion of this software is normative, though a significant amount of code is shared by both the encoder and decoder.

Q: How are you?
A: SSDD.
Q: ?
A: Same shit, different day.

Almost any discussion where gmaxwell makes a statement takes the same trajectory. Here is the example of discussion about Stratum&Mining vs. getblocktemplate&longpoll:

https://bitcointalk.org/index.php?topic=131035.msg1403446#msg1403446

You may have missed it because it was hidden in a relatively less oftern read subforum about mining software.

[gmaxwell]

Quoted for future reference. I'm not going to bold the "eventually" in the last paragraph for emphasis. It is a raw quote.
I just wanted to point out that gmaxwell is again spreading FUD and false information.

Only the decoder portion of this software is normative, though a significant amount of code is shared by both the encoder and decoder.

You appear to be claiming that I misquoted RFC 6716. You do realize that I wrote (and revised in many subsequent revisions) that section of the RFC, right? It's not clear to me specifically how you're misunderstanding that sentence but what it's saying is no different there than what we might say about Bitcoin: only the code that defines how a node responds to a block is normative, not the code that actually authors a new block.  I'll consider accepting an apology— but after all the other times you've been a jerk spewing adhominem and vague allegation about subject matter you were totally wrong about you're going to have to phrase your apology very nicely.

And would you cut with the allegations that you need to quote me for any particular reason? If for some bizarre reason I wanted to edit my comments I could edit them right inside your posts. As far as I can tell you're doing that to imply that I'm dishonest in order to piss me off. Congratulations, you've been successful. I'm wondering if you behave like this in person, and if so— do you get a discount on cold-compresses for all the black eyes you must inspire people to give you?

Algorithms and protocols are typically specified in machine-checkable form using languages such as +CAL/TLA+ and PROMELA. Not C code. The implementation should follow the specification: "a program is an algorithm plus an implementation of its data operations".

An argument can be made for that, but I am very skeptical.  Bitcoin is a distributed consensus system. The deepest and most fundamental definition of correctness— above _all_ other considerations— is that they actually achieve global consistency.  It is more important the Bitcoin nodes achieve a consensus than behave in any accordance to an other particular definition of "correct" (though, other forms of correctness matter of course, but they is a dim flicker to the furnace of consistency).  A specification can say many things, but if the installed nodes behave differently from the specification but consistently with each other and its over a matter which would cause a failure to achieve a consensus then it is the specification which is wrong in any practical sense, not the nodes, and the specification which must be changed. In effect any not-practically-executable specification is a dead letter, pure fodder for rules-lawyers to debate, and not something that would exert influence on the real world.

A parallel formal description may be good for additional confidence, but unless it can be extracted (e.g. like the software to extract code from COQ proofs) into reasonably performant production grade code it cannot practically be the normative specification.  Sure, you could call it that— but the claim would be a lie. It's worth noting that suitably authored (e.g. ACSL annotated) C is machine checkable, and also runnable— allowing a who layer of additional tests (e.g. catching some cases where a checker can prove that the algorithm will complete, but failing to warn you that it would take 2^64 operations or words of memory to get there...)— and also allowing it to be an actual normative specification which could functionally define the behavior on the network both on rules-layer paper but also in practice by actually defining the behavior of real nodes which must be exactly matched.

[Peter Todd]

In any case, for everyone who thinks an RFC or similar specification document is such a great idea, nobody is stopping you from writing one yourself. If you write a good one that manages to address the issues raised by Mike Hearn and gmaxwell, and keep it updated, it has a chance of being adopted and in any case the process will teach you a lot about how Bitcoin really works.

[grau]

In any case, for everyone who thinks an RFC or similar specification document is such a great idea, nobody is stopping you from writing one yourself.

What stops us doing lots of useful stuff is scarcity of resources.

This could perhaps be addressed by assigning a bounty by those who think it is a good idea. I wish we had crowd funding for projects like this.

[proff]

What stops us doing lots of useful stuff is scarcity of resources.

This could perhaps be addressed by assigning a bounty by those who think it is a good idea. I wish we had crowd funding for projects like this.

Needs a bit more than a bounty; it would take thousands of coins per year to support even one student to work on this sort of thing (and for the forseeable future you would not be able to get away with giving someone coins, you would need to sell them). The Bitcoin Foundation apparently has some bounties for development, but there seems to be no interest by anyone to support pure research (even directly applicable to Bitcoin), which is understandable: the Bitcoin economy is too marginal. There are no billionaires to give away million-dollar endowments here and there.

Anyway, obviously the desirable state of affairs would be to generate production code automatically from a verified specification, or to verify suitably-annotated source code. If that cannot be done, however, it does not mean that specifications or model-checking are useless. Subtle bugs do get found that way. But for immediate practical purposes it may be more beneficial to publish the existing reference implementation using literate-programming techniques.

[2112]

You appear to be claiming that I misquoted RFC 6716. You do realize that I wrote (and revised in many subsequent revisions) that section of the RFC, right? It's not clear to me specifically how you're misunderstanding that sentence but what it's saying is no different there than what we might say about Bitcoin: only the code that defines how a node responds to a block is normative, not the code that actually authors a new block.  I'll consider accepting an apology— but after all the other times you've been a jerk spewing adhominem and vague allegation about subject matter you were totally wrong about you're going to have to phrase your apology very nicely.

Actually you are 100% right when you say that I'm "too vague". I definitely need to elaborate on how half-defined standards are used for the purpose of gaming benchmarks and competition.

Lets start with an example that turns out to be mostly good. TCP/IP is a reasonably well-defined protocol with an not-well-defined "congestion avoidance" algorithm. This opening is well-publicised and there are many choices available: Tahoe, Reno, Vegas, BIC, CUBIC, etc. Even Microsoft has its entry: Compound TCP that has been ported to Linux. http://en.wikipedia.org/wiki/TCP_congestion_avoidance_algorithm

In the above example there is a scope for gaming benchmarks by calling everything "the TCP/IP", but underneath using "TCP/IP X" or "TCP/IP Y" depending on the results you want to get. But probably close to 50% of benchmark readers are familiar with the fact that there's no single "TCP/IP" and will ask further questions.

Now lets step into the pits. The perceptual compression codecs are good candidate to "less-than-half-define": just standardize the decoding algorithm and leave the coding side open-ended. Ostensibly the open-endedness will allow further gains in efficiency. But in practice it will be used to confuse the users/developers and game the benchamrks. Someone wants to show "MPEG from vendor X is slow?" Crank up all the compressor settings! Someone wants to show "MPEG from vendor Y is low-quality?" Pull all the compressor sliders down! If a benchmark reviever proclaims: "That was unfair!" then the answer is: "It conforms to MPEG standard!" Market for MPEG encoders is particularly problematic in the way described. But many others have caught on their way and are getting good at gaming comparisons by using various "hidden features" in the "authoring" portion of the "standard-conforming" implementation. Please learn to watch for it.

So what is the moral of the story above? Standards are tools (like axes), they can be used for good and bad purposes. The good question to ask yourself is:

1) is the particular vendor interested in interoperability and promoting the market by encouraging diverse implementation?

or

2) is the particular vendor known for promoting exclusivity, discouraging alternatives and always hyping his own implementation?

Please take your time to review the posting history and make a smart choice.

[Peter Todd]

So what is the moral of the story above? Standards are tools (like axes), they can be used for good and bad purposes. The good question to ask yourself is:

1) is the particular vendor interested in interoperability and promoting the market by encouraging diverse implementation?

or

2) is the particular vendor known for promoting exclusivity, discouraging alternatives and always hyping his own implementation?

Please take your time to review the posting history and make a smart choice.

You just don't get it do you? Bitcoin is a consensus system and we don't yet know how to create diverse implementations that meet the incredibly strict requirement of every implementation acting exactly the same way. For now, specifying the "Bitcoin standard" as source code is unfortunately the best we can do, and even that is difficult. This is why the recommended way to create a Bitcoin-using service is to run one or more full nodes using the reference (Satoshi) implementation to have some trusted nodes to connect to, and then use either the reference client or an alternative implementation as the base for your actual business logic. The reference node insulates your custom code from the network, especially malicious actors, and ensures that the rules for what are valid blocks and valid transactions are followed correctly. Your code can assume that anything the reference node communicates to it is accurate. (though you do need to handle re-orgs)

Ultimately that the GUI and the Bitcoin node were implemented in the same codebase was a big design mistake by Satoshi, but we have to live with it. Satoshi should have written a very small, very simple, validating node core and then wrote a separate client library and UI/wallet app as a separate code-base using that core. I think all the developers want to work towards that goal, but doing so is a huge project with a lot of risks. Not worth it given there aren't many disadvantages to just running bitcoind.

I think it's notably that you don't see the core devs complaining about Mike Hearns bitcoinj or jgarzik's pynode, neither of which claims to be a full validating node. The former especially has been used as the basis for a lot of services - satoshidice is built on bitcoinj IIRC. I have working on pynode on my todo list myself; we don't have enough people working on libraries to make it easy to explore new types of transactions. What we do have is people wasting a lot of time and effort attempting to make fully validating node re-implementations that are downright dangerous to their users and the network as a whole, yet don't provide any benefits.

You know gmaxwell has asked me a few times about my progress on pynode - not exactly the actions of someone trying to clamp down on alternatives. But he knows I understand the consensus problem.

[2112]

You just don't get it do you? Bitcoin is a consensus system and we don't yet know how to create diverse implementations that meet the incredibly strict requirement of every implementation acting exactly the same way. For now, specifying the "Bitcoin standard" as source code is unfortunately the best we can do, and even that is difficult.

Oh, I assure you that I get the "old money" vs. "new money" split in the Bitcoin millieu really well. I can even commiserate with the plight of the "old money" people like you: you've staked everything on the support of the proof-of-concept quality code, and things aren't looking good.

https://bitcointalk.org/index.php?topic=101686.msg1113732#msg1113732
https://bitcointalk.org/index.php?topic=14382.msg1147050#msg1147050

 

Bitcoin Airlines 2012 Annual Report:

1) The founder and CEO had split and we are still unable to locate him. Interim CEO is in charge until then.

2) The original paper plane has been covered in many layers of varnish and we have declared it fit for the regular service.

3) We've sold many passenger tickets for our Airline and the regular scheduled flights have started.

4) Unfortunately the cruise speed of our planes is barely above stall speed. The engines are weak and we cannot upgrade to the professional state-of-the-art engines that other airlines use. None of our mechanics know how to maintain them and we don't have money to hire some help.

5) Moreover the oxygen supply on our planes isn't working right and the passengers are passing out while onboard. Frequently the conscious passengers rob the passed-out passengers. Sometimes even the plane crew takes into temptation and plunders the unconscious passengers.

6) Fortunately the in-flight entertainment systems on our planes are the best. No-one else is even getting close to what we have to offer: neither on the land nor on the sea nor in the air.

7) Unfortunately almost all the fuel and engine thrust goes into powering the in-flight entertainment systems.

 

So getting back to the question in the title: RFC protocol for Bitcoin does exist. But in order to operate effectively it has to gain the "economic majority". The current "economic majority" already has the title to over the half of the real estate in the Bitcoin-landia. Do you think that they will give the single bird in their hands for the promise of a share of two birds in the bush?

[Gyrsur]

^^epic!  

so come on guys let us change the current situation and let Bitcoin become an Internet standard like email...

EDIT: if you want... 

[Peter Todd]

Bitcoin Airlines 2012 Annual Report:

You can troll all you want, but fundamentally the reference client runs great on fairly modest computers and because of the 1MiB block size Moore's law this will continue to be true. As I've posted elsewhere raising that limit is not an option. Right now I can run a Bitcoin node just fine on a $5/month VPN server with very little CPU power or memory. Even if your RFC specification somehow resulted in a Bitcoin node that used 10x less resources, frankly I don't care about the difference between $5/month and $0.5/month. On the other hand, I do care about network splits, and using the Satoshi codebase for full validating nodes will be the best way to prevent them for the foreseeable future.

Talking about "old money" and "new money" as somehow having anything to do with the issue of what codebase you should us to run a validating node is bizzare. The only financial interest the "old money" has is to keep Bitcoins valuable, and the best way to do that is to keep the network secure and well-used. Other than validating nodes there are lots of reasons to promote alternative implementations - I personally use Armory as a wallet and am working on a project that would create ledger services to process transactions entirely off the blockchain, while providing for a mechanism where these services could be both anonymous and trusted.

Of course, I can't think of any projects you've actually created, so I don't have any reason to think you've actually run into any the supposed serious limitations inherent in the satoshi implementation that only a complete re-write can solve.

[Gyrsur]

Bitcoin Airlines 2012 Annual Report:

Of course, I can't think of any projects you've actually created, so I don't have any reason to think you've actually run into any the supposed serious limitations inherent in the satoshi implementation that only a complete re-write can solve.

haha you mean there is no one which is fully understand the "whole thing"?? and you expect that "the rest" will trust the "whole thing"??

so who is the "Linus Torvalds" of Bitcoin?? I don't want to hear "Satoshi Nakamoto" because he is disappear.

 

EDIT: so demand can go down now... everthing above 1-2$ per BTC is overpriced for such a "thing" without a clean foundation...

[Gyrsur]

modularity is the first step to control a complex system.

[2112]

You can troll all you want, but fundamentally the reference client runs great on fairly modest computers and because of the 1MiB block size Moore's law this will continue to be true. As I've posted elsewhere raising that limit is not an option. Right now I can run a Bitcoin node just fine on a $5/month VPN server with very little CPU power or memory. Even if your RFC specification somehow resulted in a Bitcoin node that used 10x less resources, frankly I don't care about the difference between $5/month and $0.5/month. On the other hand, I do care about network splits, and using the Satoshi codebase for full validating nodes will be the best way to prevent them for the foreseeable future.

Those arguments about block size are a classic example of strawman argument from the "old money" & "old code" side. The forward-difference state maintenance was good for the proof-of-concept. But in practice nearly everyone does reverse-differencing.

https://bitcointalk.org/index.php?topic=87763.msg965877#msg965877

It is really old and well researched problem and almost all practical solutions use reverse-delta or some variant involving reverse-differencing.

I fully understand that given current situation there are no human (and other) resources available to move the network protocol and the code base forward.

Just don't say that there's no way forward.

As much as you hate my Bitcoin Airlines story it is a decent analogy: adding "fuel fees" to the "seat ticket transaction price" isn't going to make the Airline more popular and safer.

[Peter Todd]

Of course, I can't think of any projects you've actually created, so I don't have any reason to think you've actually run into any the supposed serious limitations inherent in the satoshi implementation that only a complete re-write can solve.

haha you mean there is no one which is fully understand the "whole thing"?? and you expect that "the rest" will trust the "whole thing"??

That's exactly what I mean. Unfortunately it's true, and the solution has been to use the code Satoshi wrote, the reference client, as a module by itself to talk to the network, and then write other modules, such as your wallet code and business logic, that interfaces to the Satoshi client.

If my answer results in you not being able to trust Bitcoin, then sell your coins and stop using it.

Those arguments about block size are a classic example of strawman argument from the "old money" & "old code" side. The forward-difference state maintenance was good for the proof-of-concept. But in practice nearly everyone does reverse-differencing.

https://bitcointalk.org/index.php?topic=87763.msg965877#msg965877

It is really old and well researched problem and almost all practical solutions use reverse-delta or some variant involving reverse-differencing.

I fully understand that given current situation there are no human (and other) resources available to move the network protocol and the code base forward.

Alright, so you are talking about either one of two things:

1) The reference implementation should use a reverse-delta scheme to actually store transactions. But as I've said, the current system runs fine.

2) You're suggesting implementing the UTXO concept, probably as a hard-fork change. Funny enough though, I'm actually working on writing a prototype UTXO implementation right now based on TierNolan's suggestion to use Radix/PATRICIA trees. A UTXO set is definitely something the devs want to see in the reference client, although it's a long-term goal, and in any case no-one has even done a prototype yet. If it is adopted, yes, blocks may eventually be 100% reverse-delta, but that's a really long way off because you would need a consensus...

As much as you hate my Bitcoin Airlines story it is a decent analogy: adding "fuel fees" to the "seat ticket transaction price" isn't going to make the Airline more popular and safer.

...or maybe you're confusing reverse-delta schemes with the blocksize limit. Reverse-delta doesn't decrease the size of blocks, only the amount of data needed to prove the existence of a given transaction. (this is why I'm doing my prototype; I'll need it for fidelity bonded trusted ledgers eventually) Again, RFCs and alternate implementations have nothing to do with this issue.

Anyway, I've wasted my time enough and have real work to do.

[Gyrsur]

Of course, I can't think of any projects you've actually created, so I don't have any reason to think you've actually run into any the supposed serious limitations inherent in the satoshi implementation that only a complete re-write can solve.

haha you mean there is no one which is fully understand the "whole thing"?? and you expect that "the rest" will trust the "whole thing"??

That's exactly what I mean. Unfortunately it's true, and the solution has been to use the code Satoshi wrote, the reference client, as a module by itself to talk to the network, and then write other modules, such as your wallet code and business logic, that interfaces to the Satoshi client.

If my answer results in you not being able to trust Bitcoin, then sell your coins and stop using it.

thank you for your candidness! I'm in progress of doing my own intuitive risk analysis about to put some savings into Bitcoin but with this background I will not risk this. As I wrote everthing above 1-2$ per BTC is speculation at the moment in my opinion.

[2112]

...or maybe you're confusing reverse-delta schemes with the blocksize limit. Reverse-delta doesn't decrease the size of blocks, only the amount of data needed to prove the existence of a given transaction.

I think you are getting close to understanding the reverse-delta concept.
It is a postfix notation where forward-delta is an prefix notation.

Currently the "block" message consist of header, Merkle tree & array of transactions. Most of the bytes are used for "array of transactions" are aready a duplicates of what had been seen in the "tx" messages.

When doing reverse-delta the "array of transactions" is not send in the block, only the header and the Merkle tree. The transactions had been already sent in the "tx" messages.

I believe Microsoft Research folks have already discussed this change in their "red balloon" papers:

https://bitcointalk.org/index.php?topic=51712.0
https://bitcointalk.org/index.php?topic=64738.0

as one of the ways to remove the incentives to hide the transactions from the rest of the network.

In other words: You want to get your mined block accepted really fast? Send all the transactions ahead.

Or maybe another way: the reverse-delta concept affects both the network protocol and the transaction storage. They don't have to be taken both at the same time.

[Gyrsur]

Bitcoin core developers please start to refactor the original Satoshi code to get a modular and easy maintainable code base for Bitcoin and then do the documentation.