Creating an "official" protocol specification for the Bitcoin internet currency

[totaleclipseofthebank]

All the debate about the Maxblocksize limit, and the criticism of the dev team for using "magic numbers" in their code has brought up an important point, and that is that satoshi's white paper is not a full or detailed specification of how the bitcoin protocol is actually defined. It's really more of a proposal + example.

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

The creation of early internet protocols happened in much the same way as is happening with bitcoin, organized by enthusiastic volunteers. The difference is that the protocols ended up being defined in "official" protocol specifications, written in nice, clear, technical language. Here is an example, the ipv4 protocol specification. This can be used by developers as a reference, and lots of people can independently write software within the protocol specifications.

I think the idea of having a "reference implementation" is fundamentally flawed. The reason is that routine updates to this implementation are not separate from the definition of the protocol itself, which can result in the kind of catastrophes we saw on 3/11 (never forget). Imagine that IPv4 had been defined in this way: here is our "internet" open source project. Run this program to use the internet. You are free to write your own, but all disputes will be resolved by what this one piece of software decides. This would have been a disaster, if that program contained bugs.

tldr: I think a formal protocol specification is essential to making Bitcoin the rock-solid currency system that we want it to be. I'm willing to help by working on this. Post thoughts here.

edit: just found the other post about this

[1714973075]

1714973075

[1714973075]

1714973075

[1714973075]

1714973075

[kjj]

Heh, this topic has plenty of threads, not just the one you found.

Basically, Satoshi needed to write the whole thing before he could be sure it worked.  And then we all started using it.  And now bitcoins are downright valuable.

Meanwhile, there is still no formal specification.  And the damn thing is complicated.  If you've been in the source much and followed the discussions here and in IRC, you can start to believe the devs when they say that your choices for a specification are either 1) wrong, or 2) nearly as complex as the source code itself.

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.  And there are more and stranger things that any node has to faithfully reproduce to keep the network sane, and we have no way of knowing that we've found them all.

[r.willis]

It can be solved. I'd tackle it this way:
mark blocks 1-xxxx "specification v1" (satoshi client code being the one)
xxxx-yyyy "specification v2" ("written spec")
yyyy-zzzz, where zzzz is some point in the future - specification v3
zzzz-aaaa specification v4 (bitcoin-next)
and so on.
To assist it, all clients must follow deprecation policy: every release has life span, and refuse to work
past this set date. GUI applications should spew warnings in advance with instructions how to upgrade,
and daemons should send their admins e-mails (and refuse to start without valid mail configured).
In this way we can escape "satoshi code" deadlock.

[TierNolan]

Meanwhile, there is still no formal specification.  And the damn thing is complicated.  If you've been in the source much and followed the discussions here and in IRC, you can start to believe the devs when they say that your choices for a specification are either 1) wrong, or 2) nearly as complex as the source code itself.

As long as everything that the spec allows is acceptable by the reference client, then it won't cause a fork.

If > 50% of miners reject all blocks that aren't valid according to the spec, then the chain will remain consistent with the spec, even if the reference client would accept blocks that are invalid according to the spec.

Effectively, the spec must define a sub-set of blocks that the official client will accept and also must not define any blocks that are in the chain as illegal.  The second condition could be achieve with exceptions, if necessary.

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.

Not necessarily, you could make it so that the ref client won't mine those blocks, but would accept them, as long as they occur far enough into the future.

Another one is that apparently the acceptable encodings for the public keys is "whatever openSSL accepts".  The spec would have to have a clear definition of what encodings are acceptable.

[Gyrsur]

Let me give you one odd example.  The coinbase from the genesis block is unspendable.  At first, it was a bug that kept the txid out of the index, but now, we have to explicitly flag that transaction as unspendable.  If we don't, then Satoshi has the power to fork the network at will by redeeming it.  Basically, that bug has to be part of the specification, when one is written.  And there are more and stranger things that any node has to faithfully reproduce to keep the network sane, and we have no way of knowing that we've found them all.

I understand this. if we have such "strange" things everybody should know about it because it is not wise to let the price go up to 1000$/BTC and then release such important facts and risk that the price will go down again. you understand what I'm trying to say?

EDIT: the testnet should be become an important part of the change process of new releases. also within testnet we have the ability to use different codebase (https://code.google.com/p/bitcoinj/wiki/FullVerification) for mining without risk the healthy of the mainnet.

[Ekaros]

So what could happen is that big institutions will write a new spec for alt coin and start it fresh. Or someone else might do this.

That is one option. The idea works, but current implementation seems to be lacking... For wide scale...

[Mashuri]

Has anyone posted a bounty for this yet?  I don't know enough about SW development to properly define the bounty but I am certainly interested in contributing to one.

[TierNolan]

Has anyone posted a bounty for this yet?  I don't know enough about SW development to properly define the bounty but I am certainly interested in contributing to one.

For a bounty, there needs to be a definition of exactly what is required.

Ideally, there would be a testing phase.  For example, blocks produced in accordance with the spec must be acceptable to all clients starting with version <some version>.  From here. it looks like version 4+ represents more than 90% of the client base.

Maybe, to claim the price the author would have to post 5-10% of the bounty as a reward for anyone who can create a spec compliant transaction or block that wouldn't be accepted by a version 4 or later client (or who finds a flaw with the spec).  If nobody manages to find such a block/transaction within 60 days, the author gets the bounty.  The author could slowly increase their bounty up to the required amount, since initially, there would likely be many flaws.

Also, the spec should be as compact as possible.  Ideally, it should cover what is needed to verify the block chain, as it currently is.  Exceptions should be used if a small number of blocks are only acceptable due to bugs.

[r.willis]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

[Ekaros]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

Probably only good solution. Define the spec. Test it with bounties after general testing and do hard-fork at certain point. Preferably at this point there is multiple implementations and pools use different implementations.

[Nancarrow]

I think inclusion bugs and quirks of satoshi client in spec will be counter-productive.
Spec should be "clean" and will take effect at some point in the future. By that date, everyone should upgrade their clients.

I agree, but what I think TierNolan was getting at, was just that the spec should count all blocks prior to (say) 250,000 as valid, even if some of them would not actually be under the new spec, i.e. they should be 'grandfathered in' even if the new spec would otherwise reject them. But the spec shouldn't allow future blocks to have the same 'deformities'. Assuming of course  there turn out to be any in the first ~250,000. So iron out the bugs going forward, but keep the blockchain, imperfect though it may be, up to a certain point. (Actually, not that I have much of a clue about these things, but isn't that kind of what the current system of checkpoints is for?)

[totaleclipseofthebank]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

[justusranvier]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

Blocks have version numbers.

[totaleclipseofthebank]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

Blocks have version numbers.

That isn't really the same thing, since the client version doesn't fully describe the protocol that defines a valid block.

[2112]

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

There isn't anything wrong with what you are proposing, aside from one thing: your proposal is completely skewed into the technical realm and into the total ignorance of marketing and game theoretic issues.

I'm an engineer by training, but I understand the language of business managers and marketing people. Unfortunately they rarely read this section of the forum, therefore you will normally not get the broad feedback that you desire.

What you are proposing is producing something akin to:

Code:

#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332

Nothing wrong with the above. But because it is open source somebody will soon transfer it to

Code:

#if defined(BITCOIN)
#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332
#elif defined(LITECOIN)
#define HASH(x) scrypt(x)
#define P2PPORT 9333
#define RPCPORT 9332
#elif defined(IXCOIN)
/* ... */
#elif defined(I0COIN)
/* ... */
#endif

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

[2112]

An interesting idea occurred to me... what if you could embed a hash of the protocol spec itself along in each block to be accepted by the network? This would mean that the definitions themselves could be set in stone.

For more detailed elaboration of this idea please see the first link in my signature.

[totaleclipseofthebank]

For this reason, I think it would make sense to begin work (and it will be a lot of work!) on developing a working document, or some kind of mutually agreed upon standard that goes into greater depth than satoshi's paper on defining the nuts and bolts of what exactly Bitcoin is.

There isn't anything wrong with what you are proposing, aside from one thing: your proposal is completely skewed into the technical realm and into the total ignorance of marketing and game theoretic issues.

I'm an engineer by training, but I understand the language of business managers and marketing people. Unfortunately they rarely read this section of the forum, therefore you will normally not get the broad feedback that you desire.

What you are proposing is producing something akin to:

Code:

#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332

Nothing wrong with the above. But because it is open source somebody will soon transfer it to

Code:

#if defined(BITCOIN)
#define HASH(x) sha256(x)
#define P2PPORT 8333
#define RPCPORT 8332
#elif defined(LITECOIN)
#define HASH(x) scrypt(x)
#define P2PPORT 9333
#define RPCPORT 9332
#elif defined(IXCOIN)
/* ... */
#elif defined(I0COIN)
/* ... */
#endif

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

this is about defining a standard not a brand name (do people use the internet because of its brand name? what a load of bs)

[jgarzik]

There is already plenty of work here:

     https://en.bitcoin.it/wiki/Protocol_specification

and here:

     https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

[2112]

this is about defining a standard not a brand name (do people use the internet because of its brand name? what a load of bs)

If "bs" stands for "bullshit" then you have a gc above your head, where "gc" stands for "glass ceiling". There isn't anything wrong with being focused on technological aspects of development.

But completely ignoring the finacial and game-theoretic aspects of development is a serious disadvantage in career advancement.

Yes, people do use and invest in Bitcoin because of the brand name and the brand mystique that was built around it. Just look at their reactions to any alternate cryptocurrency that is nearly identical from the technical point of view.

[totaleclipseofthebank]

There is already plenty of work here:

     https://en.bitcoin.it/wiki/Protocol_specification

and here:

     https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

Those are both excellent documents and places to start, but what I had in mind was something more set-in-stone than a wiki page, that would have formalized procedures and barriers to prevent any one person from altering the protocol in the future.

[justusranvier]

Yes, people do use and invest in Bitcoin because of the brand name and the brand mystique that was built around it. Just look at their reactions to any alternate cryptocurrency that is nearly identical from the technical point of view.

What you call "brand name and the brand mystique" I would call "first mover advantage, network effect, and human capital".

[totaleclipseofthebank]

Yes, people do use and invest in Bitcoin because of the brand name and the brand mystique that was built around it. Just look at their reactions to any alternate cryptocurrency that is nearly identical from the technical point of view.

What you call "brand name and the brand mystique" I would call "first mover advantage, network effect, and human capital".

focusing on "brand names" you might as well just use US dollars since they are the king in that department. bitcoin is cool because its a superior technology. your business school paradigms just aren't going to be all that useful here, sorry.

[2112]

Oh well, I've tried. I've really tried to point that each coin has at least two sides, and that includes Bitcoin.

Time will show what had really mattered: exact standards, network effects, first mover advantage, human capital and other presumed highfalutin concepts.

Or just this:

Bitcoin Pizza Index = $801,896.65 .... 

http://www.ounce.me

[TierNolan]

I agree, but what I think TierNolan was getting at, was just that the spec should count all blocks prior to (say) 250,000 as valid, even if some of them would not actually be under the new spec, i.e. they should be 'grandfathered in' even if the new spec would otherwise reject them. But the spec shouldn't allow future blocks to have the same 'deformities'.

Right.  Most of the block chain is presumably pretty clean already.

Basically the spec would be a sub-set of what is already acceptable to the current reference clients and also a (small) list of exceptions so that the current block chain is spec compliant.  I don't think you need to specify a switch date exactly.  The number of "non-standard" elements in the block chain would be finite anyway.

Ideally, the spec writers would get the agreement from > 50% of the miners to reject transactions and blocks that violate the draft (at least while the draft is in progress).

The exceptions could be a list of scripts that are defined as valid, even though they wouldn't be according to the spec.

[r.willis]

No need to enumerate "exceptions" from spec. Apply spec only from certain block number, use checkpoint hash to set all old blocks as valid. New spec-compliant do not need to to verify this old blocks, but only correctly parse them.

[aantonop]

Exactly as r.willis said.

Ideally, we would also implement a very small and lightweight protocol validation engine in Lisp and use that as reference implementation the for the protocol, and for interoperability, quality and eventually security certification.

At some point, "Talks Bitcoin Protocol" needs to be a verifiable statement, as with other protocol implementations. That way, a developer can test and validate that their client implementation passes all interop tests and speaks official "bitcoin".

Just like you can do W3C validation, or test an SSL client for protocol completeness, we should be able to test a bitcoin protocol node against something.

[TierNolan]

I think there also needs to be a "core" spec, which describes how blocks are validated and then a separate network spec.

[doobadoo]

I"m sure no one wants to here this, but maybe we need to start again.  Accept bitcoin as a grand experiment.  We've learned a lot.  Think thru all these issues:  block size issues, pruning, bitdust, gpu mining/asics.  Build a new client, with a firm protocol.  Better, faster, smarter.  Then launch it all over again.  Of course all the people techinically capable and involved all have a substantial stake in THIS bitcoin blockchain. 

Part of me is hoping that satoshi is out there somewhere.  Maybe working on this new, better cryptocurrency.  And when he returns and publishes the signed code on the alt-currency board every one will dump btc and jump on the new better one. 

I think he is the only one with the technical prowess, and the passion to see the idea REALLY work to the point where he may be willing take all his founders coins, cash them the fuck out quietly and then hit us with the REAL solution.  Call it the final solution. (No Jews were harmed making this post.)

[doobadoo]

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

Dude i'm really having trouble understanding what in tar-nation you mean by this.  Are you suggesting that the dev team runs the bitcoin network bc, even though the client is open source, they have written it in a way that makes the spec amorphous, thus forcing every one to keep relying on them?  Solidifying Gavin & co's benevolent dictatorship?

[kjj]

In other words, you've magnamiously forfeited the advantage you had over the competition and commoditized your core product.

Maybe for the moment think of BTC as KFC (Kentucky Fried Coin) and your intention to publish a detailed information about the formula for the Colonel Sanders' secret seasoning and the process of how he arrived at it. You will immediately spawn numerous competitors peddling their version of Fried Chicken Coin.

If you don't like the above analogy then maybe think of how you would like to differentiate your Coca-Cola Coin from Pepsi-Cola Coin and RC-Cola Coin and whole slew of other immitators.

As an engineer you were probably trained to think that your goal is to minimize the number of defects in your software. Try for the moment to think like a CEO and maximize the stock price of your enterprise.

Do you see the problem now?

Dude i'm really having trouble understanding what in tar-nation you mean by this.  Are you suggesting that the dev team runs the bitcoin network bc, even though the client is open source, they have written it in a way that makes the spec amorphous, thus forcing every one to keep relying on them?  Solidifying Gavin & co's benevolent dictatorship?

You are new here.*  Over time, you'll learn to discern 2112's pearls from his effluent.  In this case, KFC's secret recipe is widely available for everyone to see, and both Coca-Cola Coin and RC-Cola Coin already exist, putting this post solidly in the latter category.

* Yes, I know you registered around the same date as me.  Not important.

[2112]

Dude i'm really having trouble understanding what in tar-nation you mean by this.  Are you suggesting that the dev team runs the bitcoin network bc, even though the client is open source, they have written it in a way that makes the spec amorphous, thus forcing every one to keep relying on them?  Solidifying Gavin & co's benevolent dictatorship?

Try to forget for a moment everything that you know about software engineering and think about yourself as a brand manager of a consumer product.

From now on the discussion isn't going to be about technical facts, it is going to be about the perception of facts amongs the non-technical people.

Do you think it possible to reverse engineer the recipe to the Kentucky Fried Chicken to such an accuracy that nobody will be able to distinguish it in a blind tasting? I think yes, but people will insist on sighted tasting and will claim that they are capable of distinguishing the true KFC from your Clone Fried Chicken.

If you don't like popular American consumer products then how about French wines? Why for example Veuve Clicquot is considered more valuable than many other "Appellation d'Origine Contrôlée, Méthode Champenoise" beverages? How did they achieved that value and how they maintained it? Do you think that it is impossible to produce a Californian sparkling wine that 99.9% drinkers will not be able to distinguish in a blind tasting from the original? I think that this will be relatively easy, but the clone wine will have much lower market price because everyone buys wine while looking at the labels, not by the taste.

For a non-engineer it is very easy to distinguish between say Bitcoin and Litecoin. For now everyone knows that the first Appelation d'Satoshi Controlee and the second is sovetskoye shampanskoye.

What if the market gets flooded with Bitcoin clones, e.g. Bitcoin over port 8888 (marketed to Chinese, with some Confucius' proverb in its block 0), etc. And every one of those clones will be able to show that they are produced exactly according to the original Satoshi method. Or they will be able to easily show how they improved the original recipe.

Bitcoin currently has a lot of very valuable brand equity, simply because it was first in its niche. The situation may however easily change once experienced brand managers decide to enter this niche with their clone products. Currently all the clones are marketed in a completely amateurish way. Or even much worse than amateurish, like the self-destructive marketing of SolidCoin.

[wumpus]

So, who dares to start writing a specification and actually getting it reviewed by everyone involved?

This has been the zillionth meta-discussion about this subject, debating whether it is possible or not, or a good idea or not, with other people trying to derail the thread into their own agenda. People will never be unanimous on those things, certainly not on a forum like this. So it is best to just try and see how far it gets.

There are two separate things to be written down, don't confuse them

(1) what is the current behavior of the (a) network and (b) block chain. This includes any satoshi-client specific warts. This is important for people that want to implement a client right now, and is mostly uncontroversial as it just has to be correct, complete and mostly self-contained.
This specification should be in the English language, written as clearly as possible and divided into sections, so that it is possible to write test cases against specific rules to validate the spec as well as the client.
The spec needs to be the form of human or mathematical descriptions, not "here's code" because code is not well-defined nor self-contained, we've seen this with the upgrade from bdb to leveldb.

(2) what would be the ideal behavior of the (a) network and (b) block chain. This is the above, but simplified, with strange warts and exceptions marked and removed. Through a process of staged upgrades it will eventually be possible to get here. This is very controversial, there will be a lot of disagreement, so it's better to start with (1).

You can start from the wiki document, but I would prefer something in git so that changes can be reviewed before they are pushed. Wikis are great for 'living documents', but not so much for specs.

Note that BlueMatt has already written a package of block chain and network handling tests (used by the pull tester on github, but not specific to the satoshi client), so it is best to synchronize with him.

I post a bounty of 10BTC (offer valid for 6 months) for the first person to take serious effort in the direction of (1). It is a huge, complex, messy job but it needs to be done. We really need a spec that documents what we have now.

[gmaxwell]

No need to enumerate "exceptions" from spec. Apply spec only from certain block number, use checkpoint hash to set all old blocks as valid. New spec-compliant do not need to to verify this old blocks, but only correctly parse them.

Even the minimal sane behavior is surprisingly complex. But beyond that, "lock old _invalid_ blocks in with checkpoints" is really pretty hideous. You'd still have to distribute code to validate them because part of the point of the system is that none of it depends on some magic values being trustworthy. Though perhaps its preferable to relegate the legacy support to a validation tool.

At the same time, there is really not that much legacy stuff that is really worth fixing.  Making some of the byte order more consistent— or what have you— is not worth the risk of a hardfork to get it.  A lot of the other complexity in the system rules is simply necessary...  a lot of things which are easy to specify when you don't need absolutely identical bit exact behavior become difficult when you do... and the nature of what we're doing implies bit exact behavior for almost everything that reads from the blockchain.

[TiagoTiago]

this is about defining a standard not a brand name (do people use the internet because of its brand name? what a load of bs)

If "bs" stands for "bullshit" then you have a gc above your head, where "gc" stands for "glass ceiling". There isn't anything wrong with being focused on technological aspects of development.

But completely ignoring the finacial and game-theoretic aspects of development is a serious disadvantage in career advancement.

Yes, people do use and invest in Bitcoin because of the brand name and the brand mystique that was built around it. Just look at their reactions to any alternate cryptocurrency that is nearly identical from the technical point of view.

The eggs are already scrambled, Bitcoin is open source. It is not the lack of proper specification that is gonna stop people from launching altcoins. If anything, not having a proper specification actually makes it easier for incompatible competitors to rise than for more development to be done on Bitcoin compliant software.



Btw, i hope the people working on this won't be stupid enough to just blindly accept any old block as valid; if you're gonna grandfather old blocks make sure they are the blocks you're looking for, don't accept just any old-looking block, otherwise an attacker could easily rewrite history.

[TiagoTiago]

...

At the same time, there is really not that much legacy stuff that is really worth fixing.  Making some of the byte order more consistent— or what have you— is not worth the risk of a hardfork to get it. 

If life-like tests are run on testnet, would there really still be a risk of unintentional hard-forking?

[r.willis]

gmaxwell, bitcoind, as I understand, uses checkpoints to speedup initial chain validation. I'm suggesting that this behavior will be documented in spec.

[gmaxwell]

gmaxwell, bitcoind, as I understand, uses checkpoints to speedup initial chain validation. I'm suggesting that this behavior will be documented in spec.

It does not, however, use it to skip the validation entirely. A chain cannot inflate the supply of coin regardless of what the checkpoints are set as. Checkpoints have absolutely no place in a Bitcoin specification, and as of right now they way they are used do not simplify the enforcement of the protocol rules.

Once we have header first syncing we will possibly eliminate checkpoints for any speed purpose.  (though may maintain a single one for a rewrite-doomsday security blanket).

[gmaxwell]

If life-like tests are run on testnet, would there really still be a risk of unintentional hard-forking?

uh. Testnet didn't prevent the hardforking we created in Bitcoin 0.8 even though we believe that the relevant cases were tested already. (there were already super large blocks there).

[r.willis]

It does not, however, use it to skip the validation entirely. A chain cannot inflate the supply of coin regardless of what the checkpoints are set as.

It's a fine check. I hope there is no quirks associated with this check?

Checkpoints have absolutely no place in a Bitcoin specification

Why not? It's just another way of limiting scope of some ruleset. It can be "no check", "relaxed check", "check this, do not check that", "check this way" etc. The other alternative is to limit scope with height. But checkpoint has additional benefit of making impossible creation of other spec-compliant chain going all the way to the root.

[justusranvier]

What I expect to happen is that somebody is eventually just going to start deploying an alternate implementation that will get significant usage. At some point there will be another chain-splitting bug and people running the alternate implementation, or the reference client, or both, will scramble to upgrade their nodes to fix the bug. We will probably have several more March 11th-style events until all the bugs are shaken out.

The goal of preventing unexpected forks at all cost is demonstrably futile. Better to just accept them as inevitable growing pains and prepare to deal with them as efficiently as possible when they happen.

[arklan]

so, i've been wondering something this thread brought up in my mind. armory is, or is not, a full client implementation? i don't run it, so i could be asking as dumb question.

[TierNolan]

It isn't not a full client implementation, but it requires use of the bitcoin-qt/bitcoind to act as a gateway into the network so in that sense it is a full client.

Uh, isn't it the opposite?  It is a light client that only handles your transactions. 

The combination of armour + the reference client is a full client, but the reference client is a full client on its own anyway.

[gweedo]

It isn't not a full client implementation, but it requires use of the bitcoin-qt/bitcoind to act as a gateway into the network so in that sense it is a full client.

Uh, isn't it the opposite?  It is a light client that only handles your transactions. 

The combination of armour + the reference client is a full client, but the reference client is a full client on its own anyway.

A light client means something else, so I wouldn't call armory a light client. Light client means it uses the bloom filter, or SPV part of the bitcoin protocol.

[TierNolan]

A light client means something else, so I wouldn't call armory a light client. Light client means it uses the bloom filter, or SPV part of the bitcoin protocol.

It is even lighter than the light client then.  It does a specific task and just focuses on that.

[TiagoTiago]

If life-like tests are run on testnet, would there really still be a risk of unintentional hard-forking?

uh. Testnet didn't prevent the hardforking we created in Bitcoin 0.8 even though we believe that the relevant cases were tested already. (there were already super large blocks there).

But did it had a similar ratio or outdated clients and miners as the live net?

[runeks]

No need to enumerate "exceptions" from spec. Apply spec only from certain block number, use checkpoint hash to set all old blocks as valid. New spec-compliant do not need to to verify this old blocks, but only correctly parse them.

Even the minimal sane behavior is surprisingly complex. But beyond that, "lock old _invalid_ blocks in with checkpoints" is really pretty hideous. You'd still have to distribute code to validate them because part of the point of the system is that none of it depends on some magic values being trustworthy. Though perhaps its preferable to relegate the legacy support to a validation tool.

I think the idea here is for third parties to develop their own Bitcoin-compatible clients. If this means that they can only verify the block of the previous two years, for example, I think that would be acceptable to most. If they want they can always use legacy code to verify all the blocks (bitcoin-qt). I think a clean specification that only verifies the blocks of the last n-years will be very valuable to Bitcoin.

I made a post outlining the same concept put forward by r.willis in another thread. I'll quote myself for further clarification:

The problem is that the rules (as defined by Satoshi's implementation) simply pass data directly to OpenSSL, so the network rule effectively is "whatever cryptographic data OpenSSL accepts", which is bad. OpenSSL has all reasons for trying to accept as much encodings as possible, but we don't want every client to need to replicate the behaviour of OpenSSL. In particular, if they add another encoding in the future (which, again, is not necessarily bad from their point of view), we might get a block chain fork.

Considering cases like these, it occurs to me that it might be desirable to - at some point - split the Satoshi code into three parts: "legacy", "current" and "next".

The "legacy" part would handle the odd corner cases described in the above quote. It would basically pull in all the relevant OpenSSL code into the legacy module (including the bugs in question), where it would stay untouched. This module would only be used to verify already-existing blocks in the chain; no new blocks would be verified with this code, as pulling in OpenSSL code into Bitcoin and managing future patches is next to impossible. This is the part that should be possible for future clients to not implement. They will miss the part of the block chain that follows the rules defined by this module, but I reckon that we really don't need 5+ year old blocks for more than archival purposes.

The "current" module would handle verifying current blocks, and be compatible with the "legacy" module. It would depend on OpenSSL still, and if changes are made to OpenSSL that break compatibility with "legacy", patches would need to be maintained against OpenSSL to work around this. This module cannot code-freeze OpenSSL, as vulnerabilities can become uncovered in OpenSSL, and no one must be able to produce blocks that exploit the uncovered attack vectors. Newly uncovered attack vectors aren't a problem for the "legacy" module, as it only verifies already-existing blocks, produced before the vulnerability in question was uncovered.

The "next" module would be backwards incompatible with the "legacy" and "current" module. This module changes verification rules to not accept, for example, the otherwise invalid signatures that OpenSSL accepts. The "next" module would have a block chain cut-off point into the future where, from that point on, a Bitcoin transaction would be considered invalid if, for example, it includes an invalid signature (was it a negative S-value?) even though it's accepted by the old OpenSSL code. It's sort of a staging module, where undesirable protocol behavior is weeded out. These protocol changes wouldn't take effect until some point in the future (a block number). The block number cut-off point would be advertised well in advance, and from this point on, the "next" module would become the "current" module, and the old "current" module would move into the "legacy" module (and no new blocks would be verified using this module). The new "next" module would then target fixing undesirable protocol behavior that was uncovered when running the previous "current" module, and would set a new cut-off time into the future, at which point new blocks would need to follow this improved protocol to get accepted.

Couldn this work? It would mean we could slowly (very slowly) clean up the protocol, while still maintaining backwards compatibility with clients not older than, say, 2 years, or however long into the future we choose the cut-off point for the "next" module's protocol to become mandatory.

[spartacusrex]

There are plenty of smart people who understand Bitcoin enough to knock up a new version..

Sure - John Carmack invented Quake, but others took it to the levels 3D shooters are now..

I too feel a new chain, with all the benefits of hindsight, may be the answer, but then again, they said the same about the only other massive peer-to-peer decentralised comms system.. email.

And it just keeps on ticking.. :-)

You can't force people off Bitcoin. Just as you can't force people to stop using email. That's the beauty of decentralisation..

I think in the end, as always, the answer will lie somewhere in the middle. BitEmail anyone ?

[wumpus]

As it appears to be consensus that deriving the spec is too complex and tricky for a human being, It would be an interesting computer science project to mechanically, automatically extract a specification from the source code, or even the binary. The resulting spec would be very detailed (perhaps over-complete, but pruning unnecessary details is easier than adding missing details).
Seeing what the s2e guys have done (https://s2e.epfl.ch/) with for example, automatic reverse engineering of network drivers, it should be possible.

[r.willis]

the only other massive peer-to-peer decentralised comms system.. email

email isn't decentralized - it's relying on DNS, which is centralized.

As it appears to be consensus that deriving the spec is too complex and tricky for a human being

No, it is not.
1) Given enough resources, it's possible
2) We can make saner spec and fix the client

[TiagoTiago]

the only other massive peer-to-peer decentralised comms system.. email

email isn't decentralized - it's relying on DNS, which is centralized.

...

You can't send an email to an address in the format username@255.255.255.255 ?

[r.willis]

You can't send an email to an address in the format username@255.255.255.255 ?

Actually, you can, using address literal. It's written like this: user@[111.111.111.111].
But this form IMO can't be considered "most massive" system.

[Zeilap]

As it appears to be consensus that deriving the spec is too complex and tricky for a human being

I really don't understand this sentiment, especially from one of the devs. I think that everyone who complains that it's too difficult have just not bothered to find out how anything works.
In fact, I think it speaks volumes about those complaining that every one of these threads talk about a 'protocol spec', when in fact the way that the network passes messages is largely irrelevant to the working of Bitcoin (aside from the fact that it's very well documented and simple). The only important part is the binary format of transactions and blocks, so that you can verify proof-of-work and merkle roots, other than that it really doesn't matter how you obtain them - you could get all the data you need from blockexplorer.com and bypass having to connect to the network at all if you really wanted to.

There's only one thing you actually need to worry about - block validation, which is explained here. If it's too much to implement in one go, start with SPV style checks and work your way up, it's better to accept bad blocks than to reject good ones. Just don't mine or relay blocks until you're sure you've got verification completed, or isolate yourself from the rest of the network by connecting to a single trusted reference client node.

There's also the question of why you actually want to implement a fully verifying node. The only reason I can think of is if you want to mine with it - so you don't waste your effort producing invalid blocks. Other than that, you can get all the functionality you need from SPV checks and a full blockchain (making sure to verify the merkle root) for significantly less development effort, less cpu+ram use, less likelihood of forking yourself into oblivion, for a tiny reduction in security.

Additionally, there is almost zero financial incentive for creating an alternative implementation - no-one will trust it if it's not open source, and no-one will buy it if it is open source.

The biggest problems in creating an alternative client are software engineering type problems, not implementing the network protocol. How do you guarantee database consistency or recover from corruption if the power goes out while you're processing a block? How do you deal with re-orgs? How do you store your blocks/headers for best performance? How do you minimize the expensive processing you perform on bad data before you realize it's bad? How do you properly deal with errors like failed verification or your DB crapping out so that you don't confuse one with the other and end up stuck on a fork? If you receive an invalid block a number of times from different peers, when do you decide that either you're under attack or maybe you're rejecting a block that's actually good, and what do you do? Etc.
I think people are just scared of having to deal with network sockets and processing binary data, if everything was dealt with as JSON there'd be no complaints (other than network usage).

[TierNolan]

In fact, I think it speaks volumes about those complaining that every one of these threads talk about a 'protocol spec', when in fact the way that the network passes messages is largely irrelevant to the working of Bitcoin (aside from the fact that it's very well documented and simple).

I think it is more accurate to say that the spec should have 2 parts, network and validation rules.

You are fundamentally right, what prevents forking is the rules for accepting blocks and transactions.  If a client messes up the protocol, then little harm is done, as long as the blocks and tx info is still propagated.

Even if a client failed to forward a block or something, as long as a reasonable portion of the network does it right, the blocks and tx's will get to the miners.

There's only one thing you actually need to worry about - block validation, which is explained here.

I disagree.  The core validation is a description of the rules for validating transactions.  This would include the script language and the rules for the encryption.

Much of what is in the above rules is, as the title says, protocol rules.  They are mostly to ensure efficient operation of the network and keep spam low.

There's also the question of why you actually want to implement a fully verifying node. The only reason I can think of is if you want to mine with it - so you don't waste your effort producing invalid blocks. Other than that, you can get all the functionality you need from SPV checks and a full blockchain (making sure to verify the merkle root) for significantly less development effort, less cpu+ram use, less likelihood of forking yourself into oblivion, for a tiny reduction in security.

It would be nice to be able to validate the system in a distributed way.  Also, the SPV system is vulnerable to double spends, since you have to trust the node you connect to.

At the moment, there are 2 extremes, either you validate everything or validate almost nothing.  A system for notifying invalid blocks which proof of invalidity attached would allow distributed validation.  If you have 10k nodes and each node validates a random 1% of the transactions, then the probability of an invalid tx been missed is (0.99)^10000 = 2 * 10^(-44).

[runeks]

As it appears to be consensus that deriving the spec is too complex and tricky for a human being

The only important part is the binary format of transactions and blocks, so that you can verify proof-of-work and merkle roots, other than that it really doesn't matter how you obtain them - you could get all the data you need from blockexplorer.com and bypass having to connect to the network at all if you really wanted to.

No, the binary format is not the only important part. It's a necessary part, but the really tricky part is defining how one should handle the data - what constitutes valid and invalid data? What should be rejected and what should be accepted?

I would claim that not even Gavin knows this completely. For example, the DER-encoded signature is not always valid according to the DER specification, because OpenSSL does some things differently. Link. This means that an implementation that rejects these invalid signatures would reject an otherwise, according to bitcoin-qt, valid block, and it would then be on a different chain than the main (bitcoin-qt) chain.

This is what is really hard to capture in the spec, and - when the spec is done and published - we could uncover additional quirks that would cause an implementation of the spec to disagree with bitcoin-qt on the longest chain, and thus become non-standard.

[justusranvier]

If I were to guess I'd say the process of arriving at a spec is going to be a trial by fire.

One alternate implementation (bits of proof) looks like it will be deployed shortly, and may come to represent a significant fraction of the network. Surely there will be others as well.

Undoubtedly these implementations are not fully compatible with each other, or what anyone thinks the spec is. Some of the incompatibilities will be discovered by code review and fixed preemptively, and others won't be found until the chain forks like it did on March 11th. When that happens, everybody involved will rapidly deploy a fix and correct the bugs in their respective implementations.

My prediction is that an attempt to turn the existing behavior into a spec will never be completed. Instead multiple implementations will evolve into a spec via a series of unexpected chain fork events.

[Zeilap]

There's only one thing you actually need to worry about - block validation, which is explained here.

I disagree.  The core validation is a description of the rules for validating transactions.  This would include the script language and the rules for the encryption.

Much of what is in the above rules is, as the title says, protocol rules.  They are mostly to ensure efficient operation of the network and keep spam low.

Actually that page is fairly detailed on what constitutes valid and invalid transactions and blocks. for example:

• 12. For each input, if the referenced output transaction is coinbase (i.e. only 1 input, with hash=0, n=-1), it must have at least COINBASE_MATURITY (100) confirmations; else reject this transaction
• 13. Verify crypto signatures for each input; reject if any are bad
• 14. For each input, if the referenced output has already been spent by a transaction in the main branch, reject this transaction

There's also the question of why you actually want to implement a fully verifying node. The only reason I can think of is if you want to mine with it - so you don't waste your effort producing invalid blocks. Other than that, you can get all the functionality you need from SPV checks and a full blockchain (making sure to verify the merkle root) for significantly less development effort, less cpu+ram use, less likelihood of forking yourself into oblivion, for a tiny reduction in security.

It would be nice to be able to validate the system in a distributed way.  Also, the SPV system is vulnerable to double spends, since you have to trust the node you connect to.

Not really, your attacker still has to craft a whole fake block for you which must pass the current proof-of-work - this is exactly as hard as creating a legitimate block. Additionally, if you are connected to more than one node, then the other nodes are going to be telling you about new blocks much faster, and the fake one you got is going to be orphaned. So the result is that an attacker has to give up what would be a 25BTC reward for solving a block in exchange for a window of a few minutes where he may be able to take advantage of a double spend against me.

At the moment, there are 2 extremes, either you validate everything or validate almost nothing.  A system for notifying invalid blocks which proof of invalidity attached would allow distributed validation.  If you have 10k nodes and each node validates a random 1% of the transactions, then the probability of an invalid tx been missed is (0.99)^10000 = 2 * 10^(-44).

This is a nice idea, but in practice you end up creating a DoS vulnerability: if I create an invalid transaction on purpose and send it to you, then you determine it is invalid and rush to announce the fact to the rest of the network, spamming everyone with an invalid transaction. The current way this is handled is that you determine my transaction is invalid and ban me for 24 hours. The rest of the network never have to hear about my transaction at all. The other problem is that you have to perform the attached proof in order to accept it, which results in the same amount of work as if you'd just performed the verification yourself, but with a lot more network traffic.

[TierNolan]

This is a nice idea, but in practice you end up creating a DoS vulnerability: if I create an invalid transaction on purpose and send it to you, then you determine it is invalid and rush to announce the fact to the rest of the network, spamming everyone with an invalid transaction.

I was thinking that you would only "warn" if it was part of the main chain.

For example, the first thing the client should do is download the block headers.  From this, you can work out the the depth for each block.

If, when you get the full block, you find an invalid transaction that is on the main chain and buried more than, say, 12 blocks deep, you would broadcast the proof that it is invalid.

Other nodes would check the proof, and if they also thought that it was on the main chain, then they would forward the proof.  Once they receive the proof, they would tag the block as invalid and exclude any chain built on it.

The big difficulty with the system is proving that information has gone missing.  A Markle root is useless if you don't have all the hashes associated with it.

[wumpus]

I would claim that not even Gavin knows this completely. For example, the DER-encoded signature is not always valid according to the DER specification, because OpenSSL does some things differently. Link. This means that an implementation that rejects these invalid signatures would reject an otherwise, according to bitcoin-qt, valid block, and it would then be on a different chain than the main (bitcoin-qt) chain.

Indeed, it's full of such small but significant pitfalls, that can't just be ignored. I'm still not convinced that a human can write a complete spec (of the block chain validation, not so much the network protocol, that's trivial in comparison) that catches all of these cases and completely matches the client. But I'm happy to be proven wrong, and look forward to a well-written block chain spec.

Anyway that's why I proposed creating the spec mechanically from the actual executable code. It could catch such subtleties, too, at least if dependent libraries are included in the scope. It would be an interesting (but nevertheless very difficult) project...

But yeah, there is no financial incentive in doing either, I have to agree with that. But that's not a technical issue and kind of besides the point.

[TierNolan]

Indeed, it's full of such small but significant pitfalls, that can't just be ignored. I'm still not convinced that a human can write a complete spec (of the block chain validation, not so much the network protocol, that's trivial in comparison) that catches all of these cases and completely matches the client. But I'm happy to be proven wrong, and look forward to a well-written block chain spec.

The way to do it is a "soft" fork.  As long as all transactions defined by the spec are accepted by the default client, then all you need is for the miners to reject all transactions/blocks that fail to meet the spec.  All clients would still accept the blocks since the spec is a subset of what is allowed.  Also, if a small number of miners don't do the check, then they will just generate some forks that will be orphaned quickly.

Looking at the other thread about the DER thing, it looks like there are loads of examples.  It means that it isn't just a theoretical problem.  This means putting in exceptions won't work, as the list would be very long.

The spec could just define a checkpoint and depth, and have the stricter rules apply to all blocks after that step.  Effectively, the checkpoint says that all blocks leading up to the checkpoint are considered correct and then the rules apply after that.  As long as the checkpoint has is correct, this secures the chain.

[jgarzik]

There's only one thing you actually need to worry about - block validation, which is explained here.

I disagree.  The core validation is a description of the rules for validating transactions.  This would include the script language and the rules for the encryption.

What is the core problem that bitcoin solves?  The distributed consensus problem.

There have been chains of hashes and chains of digital signatures before. What makes bitcoin different is that it is timestamping these digital messages, and protecting those timestamps against being reversed.  The currency aspect of bitcoin is simply a layer on top of the distributed timestamping service.  namecoin is an example of a non-currency use of distributed timestamping.

Thus, validation of blocks and transactions is important, but in a way misses (and ignores in testing) the part about bitcoin that makes bitcoin work.

[Peter Todd]

What is the core problem that bitcoin solves?  The distributed consensus problem.

There have been chains of hashes and chains of digital signatures before. What makes bitcoin different is that it is timestamping these digital messages, and protecting those timestamps against being reversed.  The currency aspect of bitcoin is simply a layer on top of the distributed timestamping service.  namecoin is an example of a non-currency use of distributed timestamping.

Actually I think you are oversimplifying a bit... just timestamping isn't enough. If all Bitcoin did was timestamp SHA256 hashes of transactions, it'd be useless because there would be still no consensus about double spends. The actual data itself must be public.

Better is to describe Bitcoin as an data store with inherently difficult to change order. Now it happens to be that Bitcoin is implemented in such a way that it will never store conflicting data twice, but if it wasn't, it would still work just fine - the second transaction would be useless garbage and ignored. Similarly actually having the time attached to the data is only an artifact of the PoW implementation. Again, the order is what matters, not the time. If the difficulty was fixed at the current value, rather than adjusting every two weeks, Bitcoin would work just fine strictly speaking, though you would have to wait months to get a block.(1)

I'm being a bit pedantic... but it's an important distinction in the case of alt-chains that try to bootstrap off the Bitcoin PoW function. For instance you could try to define an alt-chain where ordering was determined by getting the block header hash timestamped into Bitcoin - I've proposed the idea before on the bitcoin-dev email list - but solving the actual consensus problem of being sure everyone knows about all transactions is very tricky with such "pseudo-mining"

1) You know, Bitcoin could have been usefully implemented via telegraph and semaphores back in the late 1800's had the block interval been set to, say, 1 month or so. A small team of human calculators could probably calculate a SHA256 hash for a transaction in a few hours, especially with purpose built mechanical aids, and back then they could have used a much weaker hash function and gotten away with it. Though they'd have eventually had a rousing debate about Sir. Charles Babbage's "mechanical work engines", not to mention allowing more than a few dozen transactions per month...

[TierNolan]

What is the core problem that bitcoin solves?  The distributed consensus problem.

Forks are caused by 2 or more client implementations disagreeing about whether a block is valid or not.

This is the part that needs to be exactly right to prevent forking.

Slight differences in the protocol implementation are not as fatal.

The currency aspect of bitcoin is simply a layer on top of the distributed timestamping service.  namecoin is an example of a non-currency use of distributed timestamping.

The timestamping requires header verification.

The spec could have 2 parts, net protocol and validation rules.

The validation rules are more critical to get exactly right.  A full node requires 100% support for the validation system and all clients need to be the same.

An alt client which implements the 95% of the network rules would probably be functional.  Maybe it wouldn't protect against spam quite so well etc.

[Peter Todd]

An alt client which implements the 95% of the network rules would probably be functional.  Maybe it wouldn't protect against spam quite so well etc.

You're totally right, but we've known this for a long time; fundamentally my objections to large blocks on the basis that they can be used to flood miners on low bandwidth connections is an example of consensus failure due to network rules. It's just the "rules" in that case happen to be laws of physics rather than software.

My blockheaders over Twitter thing, while at one level an April Fools joke, is at another level not a joke at all. We do need alternate methods of block and transaction propagation on the network, and unlike validation having as much variety as possible is only a good thing. It's the fundamental security principle of Bitcoin: information is easy to spread and hard to stifle.

[TierNolan]

You're totally right, but we've known this for a long time; fundamentally my objections to large blocks on the basis that they can be used to flood miners on low bandwidth connections is an example of consensus failure due to network rules. It's just the "rules" in that case happen to be laws of physics rather than software.

What are your views on having nodes validate only 1% of the block-chain?

The system needs basically a distributed hash table.  The hash becomes the key.  It could resolve to children on the merkle tree and transactions.

If you ask for a merkle root, it will resolve to the 2 child hashes.  You can then resolve one of those hashes to follow the chain downwards.

The big problem is how to handle missing (or intentionally withheld) data.

My blockheaders over Twitter thing, while at one level an April Fools joke, is at another level not a joke at all. We do need alternate methods of block and transaction propagation on the network, and unlike validation having as much variety as possible is only a good thing. It's the fundamental security principle of Bitcoin: information is easy to spread and hard to stifle.

Cool, and yeah that is exactly the point.  The network is just a means to an end.

The big benefit of the exactness of a spec is validation of a block chain.

Having said that, the network spec should be relatively easy to define exactly too.