Bitcoin Forum
September 28, 2016, 06:52:22 AM *
News: Latest stable version of Bitcoin Core: 0.13.0 (New!) [Torrent]. Make sure you verify it.
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: Bitcoin Protocol Specification  (Read 10591 times)
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
November 20, 2010, 06:09:48 PM
 #1

Hi,

I'm surprised that there is no complete documentation of current protocol specification at this time. I found something on wiki http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1 and some fragments on pybitcoin pages https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol. But there are many blank pieces (unknown bytes in format etc).

I'm missing also some standard way for protocol proposals - everything is done here on the forum in some obscure process (at least obscure for me). I think it is because bitcoin community is still quite small, but we should define standard processes for that when we want to grow.

I would like to implement own library (in python) to support bitcoin protocol, but I realized that there is no easy way until I'm familiar with cpp and official client sources. There are also many 'hacks' like limited block size, which are related to protocol itself than on client implementation.

Also one dumb question - it is really needed to have binary protocol for our intentions? I think something more standardized should be more friendly to programmers in another languages (say java) and on another platform (I don't need to solve 32/64 bit problems on datatypes etc) when I defined protocol as (for example) gzipped xml (like other opensource data formats).

Currently there is big barrier to bring new clients with cool new features, but because I don't know lowlevel internals of bitcoin client, I don't know how to improve this situation for now :-(.

1475045542
Hero Member
*
Offline Offline

Posts: 1475045542

View Profile Personal Message (Offline)

Ignore
1475045542
Reply with quote  #2

1475045542
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1475045542
Hero Member
*
Offline Offline

Posts: 1475045542

View Profile Personal Message (Offline)

Ignore
1475045542
Reply with quote  #2

1475045542
Report to moderator
1475045542
Hero Member
*
Offline Offline

Posts: 1475045542

View Profile Personal Message (Offline)

Ignore
1475045542
Reply with quote  #2

1475045542
Report to moderator
RHorning
Full Member
***
Offline Offline

Activity: 210


View Profile
November 20, 2010, 07:00:42 PM
 #2

Hi,

I'm surprised that there is no complete documentation of current protocol specification at this time. I found something on wiki http://www.bitcoin.org/wiki/doku.php?id=bitcoins_draft_spec_0_0_1 and some fragments on pybitcoin pages https://code.google.com/p/pybitcoin/wiki/BitcoinProtocol. But there are many blank pieces (unknown bytes in format etc).


I, for one, would love to work with somebody or a group who is interested in helping put together a formal specification for Bitcoins.  I think it is something whose time has come and has been discussed on a number of other thread.  It isn't strictly necessary, but a formal specification rather than depending on the reference implementation (something also needed BTW!) is incredibly useful.

I've plowed through some of the source code in an attempt to try and find out how it all works, but it has proven to be a very tough nut to crack in terms of trying to "grok" the source code and figure out just what is happening.  I really could use a "guide" to at least help out in terms of navigating the source code and knowing what to look for.  The question is also how to organize the documentation effort for the protocol.

I think the wiki itself is sufficient in terms of the collaborative writing of the protocol document, but there may need to be some place for discussions beyond that in terms of organizing the content.  Any ideas on that would be appreciated.  Yes, there are a number of holes in the documentation and things that could be improved towards that end.

1FLK3uUT3Vup5JtkGJVXKHAoS3AZWPcKdv
Cdecker
Hero Member
*****
Offline Offline

Activity: 487



View Profile WWW
November 21, 2010, 04:34:29 PM
 #3

I'm all for a specification of some kind of the protocol. We gave it a first shot when trying to get a working Bitcoin Client to work with pybitcoin, but it turned out the protocol is both complex and hard to understand at times. Also a high level overview of how the different scenarios play out would be incredibly helpful.

We do need alternative clients for the Network, as proven by the huge interest in a native Android client, and for that to ever happen we need a specification of the underlying protocol.

Want to see what developers are chatting about? http://bitcoinstats.com/irc/bitcoin-dev/logs/
Bitcoin-OTC Rating
da2ce7
Legendary
*
Online Online

Activity: 1216


Live and Let Live


View Profile
November 23, 2010, 06:00:45 AM
 #4

I am not a professional programmer, or technical document writer.
How much work, (man hours), would it take to do a Formal Bitcoin Protocol Specification?
I would like to help organize to get this done.

One off NP-Hard.
RHorning
Full Member
***
Offline Offline

Activity: 210


View Profile
November 23, 2010, 06:34:47 AM
 #5

I am not a professional programmer, or technical document writer.
How much work, (man hours), would it take to do a Formal Bitcoin Protocol Specification?
I would like to help organize to get this done.

I've seen specifications take years to get completed, so much of it depends upon how much detail that you put into the specification document. Some of them are pretty good, some are abysmal in terms of how close they are to reality and even in terms of ease of re-implementation based off of the specification.

The one nice thing about having a clean spec to work off of, besides the fact that you can create a new implementation without having to copy software, is that you can also argue about specific parts of the specification as a common framework in a fashion like you argue about legislation.  In fact, the process is very similar in terms of how laws are passed in a legislative environment.  The difference is that engineers, not lawyers have to be able to use whatever it is that you come up with.

One of the best real specifications that I've ever read was the PNG image file specification:

http://www.libpng.org/pub/png/spec/1.2/

This is of course not similar to what we have here with Bitcoins, which is instead a networking protocol.  That would be more similar to the Internet RFCs which can be found here:

http://www.ietf.org/rfc.html

If you want to read a real specification and at the same time see a bit of nerd humor, I'd suggest that you read this particular spec here:

http://tools.ietf.org/html/rfc1149

I would expect this to take several man-months of effort at the least in terms of getting it prepared for one of these international bodies.  Preferably, I'd love to see this submitted as a formal network RFC once we get everything nailed down real good.  That would give Bitcoins some solid legitimacy and is a pretty high quality target to hit.

There are other standards bodies we need to work with as well, including the ECMA (European Computer Manufacturer's Association) and the W3C (Web standards committee).

1FLK3uUT3Vup5JtkGJVXKHAoS3AZWPcKdv
theymos
Administrator
Legendary
*
expert
Offline Offline

Activity: 2422


View Profile
November 23, 2010, 07:05:52 AM
 #6

Creating an IETF-ready specification would be a waste of time. The simple Bitcoin network used now is not what will be used in the future. Like Usenet or IP+BGP, different protocols will be used for generator-to-generator and generator-to-client connections. None of this is implemented yet, and no one knows how it will behave.

Bitcoin is incomplete. Getting the message format is easy compared to figuring out what something like this is supposed to mean:

Code:
// Subscription methods for the broadcast and subscription system.
// Channel numbers are message numbers, i.e. MSG_TABLE and MSG_PRODUCT.
//
// The subscription system uses a meet-in-the-middle strategy.
// With 100,000 nodes, if senders broadcast to 1000 random nodes and receivers
// subscribe to 1000 random nodes, 99.995% (1 - 0.99^1000) of messages will get through.

Guess what doesn't exist in the code? "Product" and "Table" appear nowhere in this sense. Subscriptions are only mentioned a few more times. This "broadcast and subscription system", whatever it is, is some feature that currently only exists in Satoshi's head. There are several other examples of this kind of thing in the code.

Describing how the system currently works in detail would be very useful, but there is no chance that a "specification" will still be complete in a year from now.

1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD
ribuck
Donator
Legendary
*
Offline Offline

Activity: 826


View Profile
November 23, 2010, 11:12:58 AM
 #7

I think an incremental approach is the only practical one here.

Specify first those parts that are (a) relatively easy to specify, and (b) for which it is particularly useful to have a specification. For everything else, just rely on the reference implementation for now.

For this incremental approach, the wiki is ideal. Some time in the future a more formal process might be considered, but for now it would just slow things down.
nanotube
Hero Member
*****
Offline Offline

Activity: 485


View Profile WWW
November 23, 2010, 02:51:16 PM
 #8

fwiw, artforz, on irc, claims to have some working python bitcoin node code... he may be a good one to hit up on help about understanding the protocol.

Join #bitcoin-market on freenode for real-time market updates.
Join #bitcoin-otc - an over-the-counter trading market. http://bitcoin-otc.com
OTC web of trust: http://bitcoin-otc.com/trust.php
My trust rating: http://bitcoin-otc.com/viewratingdetail.php?nick=nanotube
RHorning
Full Member
***
Offline Offline

Activity: 210


View Profile
November 23, 2010, 03:08:56 PM
 #9

Creating an IETF-ready specification would be a waste of time. The simple Bitcoin network used now is not what will be used in the future. Like Usenet or IP+BGP, different protocols will be used for generator-to-generator and generator-to-client connections. None of this is implemented yet, and no one knows how it will behave.

Bitcoin is incomplete. Getting the message format is easy compared to figuring out what something like this is supposed to mean:

I suppose it is my time to waste getting such an effort done.  The issue here is that the protocol is the network.  I know that some software developers hate to be hamstrung with a formal spec that they must work from, under the presumption that perhaps too much planning in advance will straight jacket their ability to code.

My own personal experience is considerably different, where the time spent in planning and documenting ahead of time is time very much worth the effort and makes the job of any sort of coding considerably easier.  It also tends to make for much, much cleaner software that is easier to maintain, easier to extend, and as an indirect result less dependent upon a single person to make all of the decisions.

This is essentially the difference between a mere computer programmer and a real software engineer too.  A programmer is somebody who pulls out the compiler/editor to figure out how to get a project working, where a software engineer starts with a word processor.  That coding needs to happen eventually is true, but setting out a roadmap for how the project is to be developed is usually a good start.

We could debate and discuss various models used for software development here too, but by its very nature Bitcoins is something that deserves to have some significant effort at bringing eyeballs into its development, and the more eyeballs that we have the better that the network will behave.  At the moment it is just a minor amount of money involved, but it soon may represent some significant economic activity.  To me, this deserves some solid software engineering principles which includes documenting the effort going on here.

I'm not expecting anybody to necessarily participate in this effort, but it seems strange that there is resistance to even start such an effort and potentially thwarting such an effort.  If Bitcoins is incomplete, let's make it more complete.  If there are gaps, those gaps need to be filled.

1FLK3uUT3Vup5JtkGJVXKHAoS3AZWPcKdv
Timo Y
Legendary
*
Offline Offline

Activity: 938


bitcoin - the aerogel of money


View Profile
November 23, 2010, 05:03:27 PM
 #10

My personal experience is that strict specification creates software that is extremely good at doing what the specification wants, but for all its elegance, it's not always what the user wants or needs.  We haven't fully figured out the details of how Bitcoin is supposed to behave yet (see discussion on transaction fee policy). We are finding out as we go along.  Nothing against specification, but isn't it a bit early right now?

GPG ID: FA868D77   bitcoin-otc:forever-d
Gavin Andresen
Legendary
*
qt
Offline Offline

Activity: 1652


Chief Scientist


View Profile WWW
November 23, 2010, 05:10:57 PM
 #11

As a veteran of the premature standardization trenches (I wrote most of the ISO/IEC 14772-1 "VRML97" specification before I changed my last name from "Bell" to "Andresen")... I agree with foreverdamaged.  It is too early to try to create a formal specification.

However, it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.


How often do you get the chance to work on a potentially world-changing project?
RHorning
Full Member
***
Offline Offline

Activity: 210


View Profile
November 23, 2010, 06:27:47 PM
 #12

As a veteran of the premature standardization trenches (I wrote most of the ISO/IEC 14772-1 "VRML97" specification before I changed my last name from "Bell" to "Andresen")... I agree with foreverdamaged.  It is too early to try to create a formal specification.

However, it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.


I've also seen software written without any sort of specification or for that matter any sort of planning at all turn out horribly too.  There obviously needs to be some sort of balance to the whole thing, and I'm not against people trying experiments out before formalizing the behavior.

Indeed, it might be useful to simply state that any sort of desired inclusion into a specification ought to be implemented in code first to see if the thing works at all before it is added to a formal specification for Bitcoins.  I wouldn't mind supporting such a notion too.  A semi-formal process or even informal process for writing the spec documents would especially be useful at the moment.

I remember Steve Wozniak complaining about a full day meeting he was involved with in terms of arguing about the placement of semi-colons in the implementation of Pascal on the Apple II computers.  He thought that effort was a wasted day in his life and certainly I'd like to avoid that kind of minutiae debates.  A wiki goes a long way to fix that kind of argument too.

1FLK3uUT3Vup5JtkGJVXKHAoS3AZWPcKdv
slush
Legendary
*
Offline Offline

Activity: 1358



View Profile WWW
November 23, 2010, 06:38:31 PM
 #13

it's not always what the user wants or needs

Well, there is no way how to implement unofficial clients for many users/programmers (like me), because they are not enough skilled in C++ and reverse engineering. But I'm capable to write alternate client with at least basic specification how whole thing works. Unfortunatelly because I'm not capable to write own client, I'm also not capable to help anybody with specs. At this time, I'm dependent to somebody else who starts specification process.

I'm absolutely not talking about any formal standard, wiki should help a lot in this stage.

The Madhatter
Hero Member
*****
Offline Offline

Activity: 490


My avatar pic says it all


View Profile
November 23, 2010, 07:15:24 PM
 #14

I'm still advocating for a few small changes to the protocol now before it becomes too much of a PITA to change later. (I mentioned this on the forum about 10 months ago.)

1. The handshake should be reversed. An open Bitcoin port shouldn't identify what it is. The connecting client should initiate the handshake. This improves privacy a lot. Think nmap. Think spies. Think any tool that can fingerprint (I use telnet) a service by simply connecting to an open port.

2. The connections should be SSL. We should try to emulate FF connecting to Apache or DPI will eventually become our worst enemy. We should take what the Tor developers learned the hard way into account early on.

3. The Bitcoin client should choose a random unused port to listen on when it is first installed. For a ISP or even a nation to block port 8333 is quite easy and is becoming easier all the time.

4. UPnP is a must. The Bitcoin client should automatically open up whatever port it decided on with UPnP. This will relive a lot of NAT problems and will extend the P2P network a lot better.

Smiley
MoonShadow
Legendary
*
Offline Offline

Activity: 1666



View Profile
November 23, 2010, 07:19:57 PM
 #15


#1 & #3 seem contradictory.  How can a connecting client initiate a handshake on an unknown port without spamming the target?  Wouldn't it still show up on a portscan that was specificly looking for a running bitcoin client?

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
The Madhatter
Hero Member
*****
Offline Offline

Activity: 490


My avatar pic says it all


View Profile
November 23, 2010, 07:24:04 PM
 #16

The port isn't unknown. The IP/port are published to the network once the client has seeded successfully. Every other node writes that to their addr.dat.

As far as I can tell the addr.dat contains IP/port already. Smiley
MoonShadow
Legendary
*
Offline Offline

Activity: 1666



View Profile
November 23, 2010, 07:39:50 PM
 #17

The port isn't unknown. The IP/port are published to the network once the client has seeded successfully. Every other node writes that to their addr.dat.

As far as I can tell the addr.dat contains IP/port already. Smiley


I see.  Well, I'm not a programmer, yet I cannot see the value in obscuring, encrypting or otherwise trying to hide the port.  The port can be blocked for those who wish to hide their client, and still work; while the data in transit is only openly coded transactions and blocks.  The only risks to the port being open is a sign to potential crackers that there is a running client (and therefore a wallet.dat) on the machine.  Just close the port if that is a concern.

"The powers of financial capitalism had another far-reaching aim, nothing less than to create a world system of financial control in private hands able to dominate the political system of each country and the economy of the world as a whole. This system was to be controlled in a feudalist fashion by the central banks of the world acting in concert, by secret agreements arrived at in frequent meetings and conferences. The apex of the systems was to be the Bank for International Settlements in Basel, Switzerland, a private bank owned and controlled by the world's central banks which were themselves private corporations. Each central bank...sought to dominate its government by its ability to control Treasury loans, to manipulate foreign exchanges, to influence the level of economic activity in the country, and to influence cooperative politicians by subsequent economic rewards in the business world."

- Carroll Quigley, CFR member, mentor to Bill Clinton, from 'Tragedy And Hope'
wumpus
Hero Member
*****
qt
Offline Offline

Activity: 798

No Maps for These Territories


View Profile
November 23, 2010, 07:56:35 PM
 #18

I'm still advocating for a few small changes to the protocol now before it becomes too much of a PITA to change later. (I mentioned this on the forum about 10 months ago.)

1. The handshake should be reversed. An open Bitcoin port shouldn't identify what it is. The connecting client should initiate the handshake. This improves privacy a lot. Think nmap. Think spies. Think any tool that can fingerprint (I use telnet) a service by simply connecting to an open port.
On the other hand, this would be a lot of trouble for existing clients. A more breaking protocol change is hard to think of.

2. The connections should be SSL. We should try to emulate FF connecting to Apache or DPI will eventually become our worst enemy. We should take what the Tor developers learned the hard way into account early on.

3. The Bitcoin client should choose a random unused port to listen on when it is first installed. For a ISP or even a nation to block port 8333 is quite easy and is becoming easier all the time.

4. UPnP is a must. The Bitcoin client should automatically open up whatever port it decided on with UPnP. This will relive a lot of NAT problems and will extend the P2P network a lot better.
Agreed.
1. This would counter simple port scan/identification attacks by script kiddies. Bitcoin (or any protocol) should not announce what it is. Let the connecter speak first. Just break the connection if it is not what it expected. It will not be impossible to identify the service, just a lot harder.

2. I'm all for this. SSL support is always a good addition. It would at least provide a level of security. Potential issue (specific to SSL) is key/certificate management.

3. Why not. The range in which to randomize should be configurable though, so that firewalls that only leave through a certain range can be used (same as with bittorrent)

4. Yep, that would help with a lot of home routers.


Bitcoin Core developer [PGP] Warning: For most, coin loss is a larger risk than coin theft. A disk can die any time. Regularly back up your wallet through FileBackup Wallet to an external storage or the (encrypted!) cloud. Use a separate offline wallet for storing larger amounts.
RHorning
Full Member
***
Offline Offline

Activity: 210


View Profile
November 23, 2010, 08:31:59 PM
 #19

it's not always what the user wants or needs

Well, there is no way how to implement unofficial clients for many users/programmers (like me), because they are not enough skilled in C++ and reverse engineering. But I'm capable to write alternate client with at least basic specification how whole thing works. Unfortunatelly because I'm not capable to write own client, I'm also not capable to help anybody with specs. At this time, I'm dependent to somebody else who starts specification process.

I'm absolutely not talking about any formal standard, wiki should help a lot in this stage.

I've thrown some additional information onto the wiki already, at least enough to start.  I've found at least some of the relevant sections in the source code for Bitcoins and will try to get some more information put on there, as well as some threads to look through as well.  More theory certainly should be put together, and perhaps an evaluation of some of the decisions already made... which can certainly be useful.

1FLK3uUT3Vup5JtkGJVXKHAoS3AZWPcKdv
da2ce7
Legendary
*
Online Online

Activity: 1216


Live and Let Live


View Profile
November 24, 2010, 08:28:24 AM
 #20

[... , I] it think writing informal specifications documenting how bitcoin works right now is a great idea, and will be really helpful when it is time to go through some standardization process.

This is the most important thing to happen, IMHO, doing so would dramatically lower the barriers of entry of creating 2nd generation bitcoin clients independent of the reference implementation.

So if it would take many man_months of work to develop a formal specification, then how long would it take to develop a 'good enough' informal specification?

One off NP-Hard.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!