Bitcoin Forum
May 24, 2024, 01:49:33 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Bitcoin protocol spec  (Read 964 times)
coinrevo (OP)
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
December 29, 2013, 04:39:24 PM
 #1

To my knowledge these are the standard references for the bitcoin protocol specification

https://en.bitcoin.it/wiki/Protocol_specification
https://en.bitcoin.it/wiki/Protocol_rules

BIPs are more detailed, but are as their name implies about improvements to the protocol, not the protocol itself

https://github.com/bitcoin/bips/

Are there are any other attempts for a full spec or lengthy detailed technical descriptions to be aware of?

History trail:

Discussion as of 11/2010: https://bitcointalk.org/index.php?topic=1860.0
coinrevo (OP)
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
December 30, 2013, 10:12:36 AM
 #2

A good explanation of the block format in a more logical way

http://james.lab6.com/2012/01/12/bitcoin-285-bytes-that-changed-the-world/
theymos
Administrator
Legendary
*
Offline Offline

Activity: 5208
Merit: 13013


View Profile
December 30, 2013, 11:19:37 PM
 #3

The code of Bitcoin-Qt is the only complete protocol reference. If your protocol-related behavior doesn't exactly match the behavior of Bitcoin-Qt, even if you accurately follow the protocol specification on the wiki, then you are wrong.

1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD
coinrevo (OP)
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
December 30, 2013, 11:47:04 PM
 #4

The whole point of protocols is to get software independent descriptions, so that software can communicate over networks independently. At least that is how it works for TCP/IP, DNS, HTTP, HTML, SMTP, ECMAScript and so on. It seems Bitcoin as a network has quite a different relationship to protocols. After all the consensus build around the software, and protocols are essentially enforced consensus. At any rate, the view that bitcoind source should be the documentation is not very helpful for the future development of the network. And to say there should be no good documentation doesn't really hold up in argument. One could argue that Bitcoin should be the best documented piece of software on the planet.
DannyHamilton
Legendary
*
Offline Offline

Activity: 3402
Merit: 4656



View Profile
December 31, 2013, 12:30:16 AM
 #5

- snip -
At any rate, the view that bitcoind source should be the documentation is not very helpful for the future development of the network. And to say there should be no good documentation doesn't really hold up in argument. One could argue that Bitcoin should be the best documented piece of software on the planet.

You are welcome to write any documentation that you like.

The thing to keep in mind is that any written protocol is an attempt to describe what the reference client already does (including any "bugs" or unexpected behavior).  The reference client is not written to implement a specific written protocol.

As such, if there is any discrepancy at all between what is written in the documentation and what the reference client actually does, then the bug is in the written documentation, not in the reference client.  The written documentation will in that case have failed to properly describe the protocol.

This means that no developer can depend on the written documentation as a definitive description of intended (or actual) behavior.  As such, while the written documentation can be useful as a reference and as educational material for getting started, it is useless as a specification for exact behavior.
Crowex
Member
**
Offline Offline

Activity: 111
Merit: 10


View Profile
December 31, 2013, 12:49:22 AM
 #6

A protocol is a system of rules and procedure that should be followed in a certain situation. (other definitions are available Smiley).
One of the interesting things about bitcoin is how the protocol is established and how it is maintained.
I suppose software is a set of rules and procedures but I think it is more usual to establish the rules and procedures and then design the software to implement them rather than to base the protocol on the way that the software behaves.

TierNolan
Legendary
*
Offline Offline

Activity: 1232
Merit: 1083


View Profile
December 31, 2013, 12:49:47 AM
 #7

This means that no developer can depend on the written documentation as a definitive description of intended (or actual) behavior.  As such, while the written documentation can be useful as a reference and as educational material for getting started, it is useless as a specification for exact behavior.

That is a statement of the principle that "the reference client is the spec".  It doesn't necessarily justify it.

Is the long term plan that there will be one reference client forever?  Ideally, the protocol should include some kind of system for handling a situation where some of the miners are using a client that has forked.

For example, if there was customer and miner clients, then customer clients would accept a super-set of blocks relative to miners.

There could be a system where miners pass blocks and transactions through multiple clients and only accept blocks that are accepted by all.  The offending block would be logged for analysis and client authors would be expected to bring their software into compliance.

I also think that an official "verifier" library would be a big improvement.  Essentially, that would accept a series of blocks and verify that it is correct.

That would be a less complex piece of software relative to an entire client including networking and orphan handling.  It just says Yes/No to a blockchain.

1LxbG5cKXzTwZg9mjL3gaRE835uNQEteWF
DannyHamilton
Legendary
*
Offline Offline

Activity: 3402
Merit: 4656



View Profile
December 31, 2013, 02:08:20 AM
 #8

This means that no developer can depend on the written documentation as a definitive description of intended (or actual) behavior.  As such, while the written documentation can be useful as a reference and as educational material for getting started, it is useless as a specification for exact behavior.

That is a statement of the principle that "the reference client is the spec".  It doesn't necessarily justify it.

No it doesn't justify it, it simply explains the reality of the situation.

Is the long term plan that there will be one reference client forever?

I have no idea, but I suspect so.  That's sort of the definition of a "reference client", isn't it?  The "reference client" is the one client that is used to specify the details of exactly how the protocol works.  Perhaps instead of thinking of "the reference client software is the spec", it would be more acceptable to you to think of "the spec" as being a document that has been carefully and exactly written in a language other than English.  "the spec" is a document that just happens to have been written in a computer programming language instead of an specific country's "official language".  Since bitcoin doesn't belong to any particular country, it seems fitting that it's spec isn't written in English, or Spanish, or French, etc.

Ideally, the protocol should include some kind of system for handling a situation where some of the miners are using a client that has forked.

It does. Doesn't it?  That's why it's called "forked".  The protocol states that if a miner is using a client that doesn't exactly implement the necessary protocol rules, then the output of that miner is ignored and not accepted as "bitcoin" communication.

For example, if there was customer and miner clients, then customer clients would accept a super-set of blocks relative to miners.

And how would the "chain" be maintained then?  The idea of the blockchain is to force a consensus.  If a "customer client" is willing to hold multiple competing blocks as equally valid, then the concept of a consensus is lost.

There could be a system where miners pass blocks and transactions through multiple clients and only accept blocks that are accepted by all.

How would you know when "all" clients had accepted the blocks/transactions?  How would you handle a situation where an adversary creates a client that specifically rejects the very transactions or blocks that you believe should be included?

The offending block would be logged for analysis and client authors would be expected to bring their software into compliance.

If two clients disagree about a two different blocks, which block is the "offending" one, and which is the "acceptable" one.
coinrevo (OP)
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
December 31, 2013, 12:00:50 PM
 #9

Is the long term plan that there will be one reference client forever?  Ideally, the protocol should include some kind of system for handling a situation where some of the miners are using a client that has forked.

Interesting. The question is if it is reasonable to have several alternative implementations. Say one would have two node types, say 50% of miners would run bitcoinA and 50% bitcoinX. Now the developers of bitcoinX decide to introduce a new feature for making transactions more anonymous. bitcoinA devs oppose. And so the chain splits, based on the implementations.

What would be the incentive for miners to use bitcoinX? Pretty much none, because there is only risk involved and little benefit. Everyone will stick to bitcoinA.

Essentially it probably means that true alternative implementations will be Alt-Coins and Coins will be competing against each other. I don't know how much BTC the core devs hold, and to what extent it is public knowledge, but one can imagine that introduces some biases. The general attitude towards alternative implementations and Alt-Coins is mostly negative, for good and bad reasons. But the idea that there should not even be proper documentation is absolutely ludicrous. You might as well run the project as closed source then.

The control of the code is codified through the SSH access keys. In the end the references to "community" are very vague and not based on formal trust models. I assume that miners largely don't care.
Altoidnerd
Sr. Member
****
Offline Offline

Activity: 406
Merit: 251


http://altoidnerd.com


View Profile WWW
December 31, 2013, 03:36:22 PM
 #10

But the idea that there should not even be proper documentation is absolutely ludicrous. You might as well run the project as closed source then.

The control of the code is codified through the SSH access keys. In the end the references to "community" are very vague and not based on formal trust models. I assume that miners largely don't care.

This is not a choice made, this is a result of what the meaning of trust is. The openest source project is one which the code is the only documentation for the reasons already provided.  This is because any other interpretation is by definition not necessarily trustworthy.  Descriptions are free to exist, but none have special status as "correct" except the source code.

Bitcoin is a big "company" of people who have no mutual trust.  I've got no reason to trust your description over the actual source code, which is why others here see this as moot.

Do you even mine?
http://altoidnerd.com 
12gKRdrz7yy7erg5apUvSRGemypTUvBRuJ
TierNolan
Legendary
*
Offline Offline

Activity: 1232
Merit: 1083


View Profile
January 01, 2014, 01:11:42 AM
 #11

That's sort of the definition of a "reference client", isn't it? 

Well, the reference client could be the one that implements a written spec.

I can see source code being a valid way of doing the spec.  However, the bitcoin client isn't really written that way.

It would have to compromise efficiency if that helps clarity.

However, Gavin has commented that they are not likely to do a complete re-write of the reference client.  At the moment, keeping it functional is more important.

Quote
It does. Doesn't it?  That's why it's called "forked".  The protocol states that if a miner is using a client that doesn't exactly implement the necessary protocol rules, then the output of that miner is ignored and not accepted as "bitcoin" communication.

A way of doing this would be broadcasting block headers even if the blocks are rejected.  The allows all clients to build up the block header tree.  It would give them visibility of forks that have happened and an estimate of hashing power.

This would allow clients to display a warning that around 10% of the networks hashing power is being committed to a forked chain.

Similarly, the clients on fork could display a warning that they are on a chain that is only getting that 10%.

In a situation like that, clients could increase the confirm window and/or require confirms on both forks.

This gives an automatic/instant response to the forking of the chain and then the developers of the client that caused the fork can bring their client back into consensus.

The most recent fork was caused by a new release of the reference client.  Even if there is only one client, it will still have version numbers.  Being able to handle forks so that the network degrades more elegantly would be an improvement and it would allow alternative implementations.

Quote
And how would the "chain" be maintained then?  The idea of the blockchain is to force a consensus.  If a "customer client" is willing to hold multiple competing blocks as equally valid, then the concept of a consensus is lost.

Soft forks already operate on this principle.  It is ok for merchants and customers to have outdated clients.  They accept blocks based on the old and new rules. 

(A majority of) miners only accept blocks based on the new rules.  This means that the chain is guaranteed to only have blocks that operate on the new rules (barring temporary orphans).

Mining nodes need to be stricter than general usage nodes.

With mining pools, they could very easily verify all blocks they receive against multiple versions of the most popular clients.

Handling it with p2pool would be more difficult.

Quote
How would you know when "all" clients had accepted the blocks/transactions?  How would you handle a situation where an adversary creates a client that specifically rejects the very transactions or blocks that you believe should be included?

I meant all of the multiple clients that the miner is using to check blocks, not all possible clients.

Even with a reference client, this would be a good system.

Quote
The offending block would be logged for analysis and client authors would be expected to bring their software into compliance.

If two clients disagree about a two different blocks, which block is the "offending" one, and which is the "acceptable" one.

If there was 10 common clients, then for almost any block almost all of them would agree.  However, in that context, I would take the position that if any of them reject a block, then miners would be recommended to reject the block.

1LxbG5cKXzTwZg9mjL3gaRE835uNQEteWF
danneu
Newbie
*
Offline Offline

Activity: 32
Merit: 0



View Profile
January 01, 2014, 01:50:20 AM
 #12

The whole point of protocols is to get software independent descriptions, so that software can communicate over networks independently. At least that is how it works for TCP/IP, DNS, HTTP, HTML, SMTP, ECMAScript and so on. It seems Bitcoin as a network has quite a different relationship to protocols. After all the consensus build around the software, and protocols are essentially enforced consensus. At any rate, the view that bitcoind source should be the documentation is not very helpful for the future development of the network. And to say there should be no good documentation doesn't really hold up in argument. One could argue that Bitcoin should be the best documented piece of software on the planet.

Well, the reality is that we're forced to reverse-engineer the specs as with most software since it unfortunately wasn't pre-engineered by a liability-hardened steering committee but rather evolved from a buggy ad-hoc pseudonymous side project by a few volunteers.

Most of the topical work has been done, but we're left with edge-cases and idiosyncrasies and arbitrary implementation details with a high cost for reimplementing it wrong.

But what's new?

If anybody really cared to ever tune up the state of documentation in software projects, then documentation wouldn't be persistently relegated to a thankless chore dependent on the goodwill gruntwork of the few volunteers capable of writing it.
Altoidnerd
Sr. Member
****
Offline Offline

Activity: 406
Merit: 251


http://altoidnerd.com


View Profile WWW
January 01, 2014, 02:37:17 AM
 #13


If anybody really cared to ever tune up the state of documentation in software projects, then documentation wouldn't be persistently relegated to a thankless chore dependent on the goodwill gruntwork of the few volunteers capable of writing it.

This is a human problem in general that bitcoin micropayments can solve.  I'd micropay for this! 

Do you even mine?
http://altoidnerd.com 
12gKRdrz7yy7erg5apUvSRGemypTUvBRuJ
kjj
Legendary
*
Offline Offline

Activity: 1302
Merit: 1025



View Profile
January 01, 2014, 04:03:00 AM
 #14

Ugh.  Do we really need to start a brand new thread on this every few months?

At this point, is there even a slight chance of someone making a meaningful addition to the discussion that can be found in all of the many, many other threads on this topic?

17Np17BSrpnHCZ2pgtiMNnhjnsWJ2TMqq8
I routinely ignore posters with paid advertising in their sigs.  You should too.
coinrevo (OP)
Member
**
Offline Offline

Activity: 70
Merit: 10


View Profile
January 01, 2014, 09:45:42 AM
 #15

Ugh.  Do we really need to start a brand new thread on this every few months?

At this point, is there even a slight chance of someone making a meaningful addition to the discussion that can be found in all of the many, many other threads on this topic?

What does this comment add to the discussion? Absolutely nothing. It just shows that better efforts are needed, and that people who are in this on board for a long time, don't care about a lot of things. That's why so much things are broken and should be improved. This messageboard and the wiki are not properly searchable. Lets all go back to usenet please.

Quote
If anybody really cared to ever tune up the state of documentation in software projects,

As if Bitcoin was a regular software project. hilarious.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!