Creating an "official" protocol specification for the Bitcoin internet currency

[totaleclipseofthebank]

All the debate about the Maxblocksize limit, and the criticism of the dev team for using "magic numbers" in their code has brought up an important point, and that is that satoshi's white paper is not a full or detailed specification of how the bitcoin protocol is actually defined. It's really more of a proposal + example.

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

The creation of early internet protocols happened in much the same way as is happening with bitcoin, organized by enthusiastic volunteers. The difference is that the protocols ended up being defined in "official" protocol specifications, written in nice, clear, technical language. Here is an example, the ipv4 protocol specification. This can be used by developers as a reference, and lots of people can independently write software within the protocol specifications.

I think the idea of having a "reference implementation" is fundamentally flawed. The reason is that routine updates to this implementation are not separate from the definition of the protocol itself, which can result in the kind of catastrophes we saw on 3/11 (never forget). Imagine that IPv4 had been defined in this way: here is our "internet" open source project. Run this program to use the internet. You are free to write your own, but all disputes will be resolved by what this one piece of software decides. This would have been a disaster, if that program contained bugs.

tldr: I think a formal protocol specification is essential to making Bitcoin the rock-solid currency system that we want it to be. I'm willing to help by working on this. Post thoughts here.

edit: just found the other post about this

[1715393446]

1715393446

[1715393446]

1715393446

[kjj]

Heh, this topic has plenty of threads, not just the one you found.

Basically, Satoshi needed to write the whole thing before he could be sure it worked.  And then we all started using it.  And now bitcoins are downright valuable.

Meanwhile, there is still no formal specification.  And the damn thing is complicated.  If you've been in the source much and followed the discussions here and in IRC, you can start to believe the devs when they say that your choices for a specification are either 1) wrong, or 2) nearly as complex as the source code itself.

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.  And there are more and stranger things that any node has to faithfully reproduce to keep the network sane, and we have no way of knowing that we've found them all.

[r.willis]

It can be solved. I'd tackle it this way:
mark blocks 1-xxxx "specification v1" (satoshi client code being the one)
xxxx-yyyy "specification v2" ("written spec")
yyyy-zzzz, where zzzz is some point in the future - specification v3
zzzz-aaaa specification v4 (bitcoin-next)
and so on.
To assist it, all clients must follow deprecation policy: every release has life span, and refuse to work
past this set date. GUI applications should spew warnings in advance with instructions how to upgrade,
and daemons should send their admins e-mails (and refuse to start without valid mail configured).
In this way we can escape "satoshi code" deadlock.

[TierNolan]

Meanwhile, there is still no formal specification.  And the damn thing is complicated.  If you've been in the source much and followed the discussions here and in IRC, you can start to believe the devs when they say that your choices for a specification are either 1) wrong, or 2) nearly as complex as the source code itself.

As long as everything that the spec allows is acceptable by the reference client, then it won't cause a fork.

If > 50% of miners reject all blocks that aren't valid according to the spec, then the chain will remain consistent with the spec, even if the reference client would accept blocks that are invalid according to the spec.

Effectively, the spec must define a sub-set of blocks that the official client will accept and also must not define any blocks that are in the chain as illegal.  The second condition could be achieve with exceptions, if necessary.

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.

Not necessarily, you could make it so that the ref client won't mine those blocks, but would accept them, as long as they occur far enough into the future.

Another one is that apparently the acceptable encodings for the public keys is "whatever openSSL accepts".  The spec would have to have a clear definition of what encodings are acceptable.

[Gyrsur]

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.  And there are more and stranger things that any node has to faithfully reproduce to keep the network sane, and we have no way of knowing that we've found them all.

I understand this. if we have such "strange" things everybody should know about it because it is not wise to let the price go up to 1000$/BTC and then release such important facts and risk that the price will go down again. you understand what I'm trying to say?

EDIT: the testnet should be become an important part of the change process of new releases. also within testnet we have the ability to use different codebase (https://code.google.com/p/bitcoinj/wiki/FullVerification) for mining without risk the healthy of the mainnet.

[Ekaros]

So what could happen is that big institutions will write a new spec for alt coin and start it fresh. Or someone else might do this.

That is one option. The idea works, but current implementation seems to be lacking... For wide scale...

[Mashuri]

Has anyone posted a bounty for this yet?  I don't know enough about SW development to properly define the bounty but I am certainly interested in contributing to one.

[TierNolan]

Has anyone posted a bounty for this yet?  I don't know enough about SW development to properly define the bounty but I am certainly interested in contributing to one.

For a bounty, there needs to be a definition of exactly what is required.

Ideally, there would be a testing phase.  For example, blocks produced in accordance with the spec must be acceptable to all clients starting with version <some version>.  From here. it looks like version 4+ represents more than 90% of the client base.

Maybe, to claim the price the author would have to post 5-10% of the bounty as a reward for anyone who can create a spec compliant transaction or block that wouldn't be accepted by a version 4 or later client (or who finds a flaw with the spec).  If nobody manages to find such a block/transaction within 60 days, the author gets the bounty.  The author could slowly increase their bounty up to the required amount, since initially, there would likely be many flaws.

Also, the spec should be as compact as possible.  Ideally, it should cover what is needed to verify the block chain, as it currently is.  Exceptions should be used if a small number of blocks are only acceptable due to bugs.

[r.willis]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

[Ekaros]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

Probably only good solution. Define the spec. Test it with bounties after general testing and do hard-fork at certain point. Preferably at this point there is multiple implementations and pools use different implementations.

[Nancarrow]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

I agree, but what I think TierNolan was getting at, was just that the spec should count all blocks prior to (say) 250,000 as valid, even if some of them would not actually be under the new spec, i.e. they should be 'grandfathered in' even if the new spec would otherwise reject them. But the spec shouldn't allow future blocks to have the same 'deformities'. Assuming of course  there turn out to be any in the first ~250,000. So iron out the bugs going forward, but keep the blockchain, imperfect though it may be, up to a certain point. (Actually, not that I have much of a clue about these things, but isn't that kind of what the current system of checkpoints is for?)

[totaleclipseofthebank]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

[justusranvier]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

Blocks have version numbers.

[totaleclipseofthebank]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

Blocks have version numbers.

That isn't really the same thing, since the client version doesn't fully describe the protocol that defines a valid block.

[2112]

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

There isn't anything wrong with what you are proposing, aside from one thing: your proposal is completely skewed into the technical realm and into the total ignorance of marketing and game theoretic issues.

I'm an engineer by training, but I understand the language of business managers and marketing people. Unfortunately they rarely read this section of the forum, therefore you will normally not get the broad feedback that you desire.

What you are proposing is producing something akin to:

Code:

#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332

Nothing wrong with the above. But because it is open source somebody will soon transfer it to

Code:

#if defined(BITCOIN)
#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332
#elif defined(LITECOIN)
#define HASH(x) scrypt(x)
#define P2PPORT 9333
#define RPCPORT 9332
#elif defined(IXCOIN)
/* ... */
#elif defined(I0COIN)
/* ... */
#endif

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

[2112]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

For more detailed elaboration of this idea please see the first link in my signature.

[totaleclipseofthebank]

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

There isn't anything wrong with what you are proposing, aside from one thing: your proposal is completely skewed into the technical realm and into the total ignorance of marketing and game theoretic issues.

I'm an engineer by training, but I understand the language of business managers and marketing people. Unfortunately they rarely read this section of the forum, therefore you will normally not get the broad feedback that you desire.

What you are proposing is producing something akin to:

Code:

#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332

Nothing wrong with the above. But because it is open source somebody will soon transfer it to

Code:

#if defined(BITCOIN)
#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332
#elif defined(LITECOIN)
#define HASH(x) scrypt(x)
#define P2PPORT 9333
#define RPCPORT 9332
#elif defined(IXCOIN)
/* ... */
#elif defined(I0COIN)
/* ... */
#endif

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

this is about defining a standard not a brand name (do people use the internet because of its brand name? what a load of bs)

[jgarzik]

There is already plenty of work here:

     https://en.bitcoin.it/wiki/Protocol_specification

and here:

     https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

[2112]

this is about defining a standard not a brand name (do people use the internet because of its brand name? what a load of bs)

If "bs" stands for "bullshit" then you have a gc above your head, where "gc" stands for "glass ceiling". There isn't anything wrong with being focused on technological aspects of development.

But completely ignoring the finacial and game-theoretic aspects of development is a serious disadvantage in career advancement.

Yes, people do use and invest in Bitcoin because of the brand name and the brand mystique that was built around it. Just look at their reactions to any alternate cryptocurrency that is nearly identical from the technical point of view.

[totaleclipseofthebank]

There is already plenty of work here:

     https://en.bitcoin.it/wiki/Protocol_specification

and here:

     https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

Those are both excellent documents and places to start, but what I had in mind was something more set-in-stone than a wiki page, that would have formalized procedures and barriers to prevent any one person from altering the protocol in the future.