Bitcoin Forum

(Apologies if there are historical errors in my recounting of the Gnutella story)

"If Gnutella Had A Protocol Specification, Why Can't Bitcoin?"

Gnutella was a peer to peer file sharing program written by Justin Frankel from Nullsoft and released in 2000 as an executable with accompanying source code. There was no documentation or written specification other than the source code and comments contained inside. Gnutella immediately became popular, and the company Clip2 was formed whose first opus was to write a quick protocol specification based on an analysis of the code:

Gnutella Protocol Specification v0.4 (http://www.stanford.edu/class/cs244b/gnutella_protocol_0.4.pdf)

Gnutella's popularity immediately spawned a number of clones. These clones offered various features but all interoperated together on the same peer to peer network. Commercial entities like BearShare and LimeWire were formed to produce their own Gnutella clients and build new features. As more companies appeared, it was in everyone's interests to have an up to date specification so that clients from different vendors could be compatible. Volunteers worked with the commercial and open source Gnutella client providers to write the next iteration of the Gnutella protocol:

Gnutella Protocol Specification v0.6 (http://rfc-gnutella.sourceforge.net/src/rfc-0_6-draft.html)

Nodes identified their vendor origin and Gnutella protocol version number on the initial handshake. Using this information, newer implementations would accomodate older implementations by behaving appropriately as per the different versions of the protocol specification. To distinguish their products, vendors would add features that were only enabled when both hosts were from the same vendor. This allowed vendors to improve their implementations while maintaining compatibility with other nodes.

Thanks to the availability of language-agnostic protocol specifications, academic research into Gnutella was made easy. Many papers were written analyzing the Gnutella protocol and suggesting improvements:

Improving Gnutella Protocol: Protocol Analysis and Research Proposals (http://www.cs.cornell.edu/people/egs/615/gnutella.pdf)

The largest entities developing Gnutella clients worked together in a vendor-neutral message board to discuss improvements to the protocol. In many ways this is similar to the Bitcoin development mailing list except that representatives of more than one client implementation were present instead of just one:

The Gnutella Developers Forum (http://groups.yahoo.com/group/the_gdf/)

The technical merits and details of each improvement were discussed until a consensus was reached. Documentation was written, and test code developed. After some iterations and refinement, the official version of the proposal was submitted and became part of the growing official protocol. Here's an example of the “ultrapeer” proposal from LimeWire, which re-structured the overlay network to make better use of bandwidth:

Ultrapeers: Another Step Towards Gnutella Scalability (http://rfc-gnutella.sourceforge.net/Proposals/Ultrapeer/Ultrapeers.htm)

No single vendor or programmer had control over the Gnutella specification. Changes to interoperability was always done through cooperation, discussion, and consensus. Vendors who wanted to create their own variations of these proposals were free to to so when two of their own nodes were talking to each other. But when talking to nodes from other vendors they could follow the official specification. The flood-fill nature of queries created bandwidth problems, and scaling issues were always at the forefront. In many ways, Gnutella faced the exact same problems that Bitcoin is facing. Bloom Filters were proposed for Gnutella, a specification hammered out and a document produced:

Query Routing for the Gnutella Network (http://www.cs.kent.edu/~javed/class-P2P12F/qrp.html)

Although LimeWire, LLC was a commercial entity, they had a strong desire to see Gnutella succeed as an open protocol and to this end they contributed resources and efforts to define the specification so that anyone could write their own implementation and interoperate on the Gnutella network. At the peak of Gnutella's adoption, there were many vendors and many different implementations. They all worked together on the same peer to peer network. A “monoculture” did not exist – Gnutella implementations were written in a variety of languages but they all had the Gnutella protocol specification to work from.

Bitcoin has many features in common with Gnutella, both technical and social. It uses a peer to peer overlay network. The reference implementation is open source. They both have the “commons”: shared bandwidth, storage, and computing resources. Unfortunately, the development process for these two software systems is completely different. Unlike Gnutella, Bitcoin has only one real reference implementation. The build process for this implementation is difficult and cumbersome, requiring a hodge-podge of separate dependencies to get built. There is no protocol specification for Bitcoin, and the developers have not expressed any interest in creating one. They make changes to the “wire protocol” (messages exchanged between peers) which would ordinarily break other vendors' implementations. But since there are no other vendors, there is little incentive to stop. The Bitcoin Foundation is nothing like the Gnutella Developers Forum and lacks the benevolence of the LimeWire employees. There is no altruistic vision, just a group of corporate individuals with partially overlapping financial interests.

Bitcoin would greatly benefit from having multiple robust node implementations, but this is hamstrung because there is no formal protocol specification, and developers feel free to make breaking changes on their whims. Every web standard has a specification. It has often been stated that "Bitcoin is different" because it transmits money. Doesn't that make the case for an official specification even stronger?

When I compare the development of Gnutella to the development of Bitcoin, the contrast is extreme. Whereas Gnutella had dedicated individuals who set the bar high, Bitcoin developers often make excuses:


Are we to believe that the programmers who hold the blockchain in their very hands are not capable of producing a document with a concise, language-agnostic description of a particular aspect of the Bitcoin protocol? Shouldn't we hold commits to the official repository to a higher standard than "just read the code?"

How can we expect any serious corporate or institutional involvement in Bitcoin if the developers' attitude is "If you can't keep up, don't step up?"


Then Bitcoin developers need to stop adding new features and document all of these obscurities fully. There are a finite number of them, and they can be enumerated. In the words of Mike Hearn, “If you can't keep up, don't step up.” If you can't document all the idiosyncrasies of every external dependency, the don't be adding new features. In fact, don't make any commits at all and leave it to someone more capable.

The argument that Bitcoin is “too complicated” or “too important” to have a fully documented specification and formal process for making “wire breaking changes” is shown to be false by the success of Gnutella. Multiple vendors, producing separate client implementations in different programming languages, working together on protocol improvements and adhering to a formal written specification agreed upon by all.

Why can't we have the same thing for Bitcoin? As users we must demand that Bitcoin development is held to a higher standard. The Bitcoin Foundation isn't interested in doing it, so we need to step up.

Reserved

False. Different Gnutella implementations had interconnectivity issues in past. Bitcoin did it only once and even then a minor one.

Bitcoin implementations with connectivity bugs would find themselves unable to participate effectively in the network. But this situation is preferable to the current situation, which is a monoculture of developers who lack the motivation to define and adhere to a standard.

My opinion is that the extent to which a project succeeds is tied to the attributes of the leader. Gnutella was strongly influenced by LimeWire and BearShare's participation with the community in creating a subset of open standards and encouraging interoperability. LimeWire in particular made a point of keeping accurate protocol specifications, submitting proposals in writing ahead of time, and soliciting feedback from the community.

If we want Bitcoin development to follow the same trajectory, the first step is to get people involved who share these values, or convince the developers to adopt these values. Here's a start:


* Develop a comprehensive protocol specification from the code and stick to it
* Feature freeze until a protocol has been written
* Eliminate external dependencies in the reference client
* Define formal process for commits to the official repository


Commits which change the wire protocol or behavior of the reference client with respect to other nodes should only after the changes are fully described in a proposal paper and agreed upon.

The problem wasn't the lack of a specification.  The problem was the new version met the specification and the old version had an unknown bug that limited it to a subset of the specification.  When the new version exercised this previously unused area of the specification space, it triggered the bug in the old version.

What I find somewhat disturbing that there is no real specification to adhere to. Just on reference implementation, which isn't obviously fully understood...

Right, you're referring to the most recent isolated incident. Recognize that this is a symptom of a much larger problem, that realistically speaking there is only one implementation of Bitcoin and that it is difficult or impossible to write and maintain a separate implementation. First of all because there is no specification, and second because the current developers have the freedom to change the protocol at their whim.

If we had more diversity in interoperable implementations, a bug like the recent database problem would have been less severe.


And even more disturbing is the complete apathy of both the current developers and the Bitcoin Foundation at changing the status quo.

The source code is the best specification. No documentation written in human language be complete and unanimous enough to be sure that everything is covered. You cannot compile Hemingway writings into executable code.

Bitcoin with connectivity difficulties would have problems with different Bitcoin clients, but will happily create disconnected network and all sorts of other nasty things.

Bitcoin have much more at stake than Gnutella warez download.

Source code which contains unknown cases isn't acceptable for system (to be) used as medium of exchange. Bitcoin does have much much more at stake than some filesharing network, which is why there should be spec to follow.

For something that my life depend on, I want certainty that it's well understood and tested.

An how would specification prevent what happened yesterday? Document the unknown bugs? Only thorough examination of source code could reveal hidden flaws.

I completely agree on the importance of Bitcoin.

I'm all for having a spec that allows an ecosystem of multiple compatible implementations to arise, but in this case it wouldn't have prevented this problem. No amount of examination of the Bitcoin source code would have revealed the limitation in BDB, and the resulting need for a configuration file to change the BDB default values.

Unit tests might not have even caught the problem because they said that not all 0.7 nodes rejected the large block. This implies that different BDB versions and/or platforms have different default values so consequently not all of them would have exhibited the problem, possibly including the platform used to run the unit tests.

Write a spec, get multiple implementations. Majority of them should be correct and those who disagree must be fixed. I believe it's the best long term strategy. It's really bad to base entire thing on one implementation in long run.

Agreed

If there is no specification, what is this?

https://en.bitcoin.it/wiki/Protocol_specification

That's what 0.8 followed but 0.7 didn't.

The situation with gnutella is also laughable when compared to how any even halfway respectable financial company handles these kind of issues.

The first thing that is done in all the companies I have worked in always is a REQUIREMENT SPECIFICATION. Without that you do not know which features must be written into the protocol specification and jumping to define that is just going to lead into fail. In case of gnutella it does not matter much. Planning bitcoin to take over the world and last for centuries as the king of the hill is not going to succeed in an adhoc way of working. Talking about how the source code is the best protocol spec is about the most horrendous adhoc way of working possible. Anybody talking about that is seriously fooling themselves.

For example the issue of how much capacity the network should have to process transactions should be first specified in the requirement spec. Now nobody knows and nobody agrees about what the capacity should be and that means that any attempt to fix the issue is futile. It might be so that the needed capacity is 100x of what there is now or the needed capacity should be 10000x of current capacity. I don't know, nobody knows. What I do know that the current attempts to increase it by 2x or 4x or whatever are just a bandaid and adding a lot of bandaid, bubblegum and duct-tape into the system now could be later found to cause huge problems when the next step needs to be taken and somebody might need to work with the multiple versions with varying amount of ducttape in them.

One thing that I have seen done in all companies I have worked in is test specification. And an comprehensive set of implemented testcases should be available in some way so that everybody could have them and run them against any client they want to test and then for their part add to that set of implemented testcases.

Many companies trying to implement an system of this magnitude and complexity could decide to build first is a simulation of the thing at the network level. That could be done so that it could be run inside one computer when in the simulation model the CPU-expensive hashing is replaced with something else much less CPU intensive but still showing the main characteristics of the hashing that are visible at the network level. This model could be later upgraded to be closer and closer to the protocol spec but sill retaining the low CPU usage so that it could be practically run in a realistically available HW.

The tests defined in the test specification could be first tested running them against this simulator. Many other financial institutions have I guess multiple complete datacentres across the globe as redundancy and I guess they can take some of them offline in quiet periods to test a new version of their SW in real HW and to be able to run tests to them at loads that are close to real capacity of the network. Similar is not possible with bitcoin and it is a huge problem.

Bitcoin is lacking in testing in so many areas. The test spec, there is no real environment to test. There are no automated test generators that can read the protocol spec and create random tests without human intervention. This is one of the ways I believe major financial institutions handle the issue whether all cornercases are tested or not. The automated testvector generator that reads in the computer readable protocol spec can be guaranteed given enough time to test all the cornercases of the protocol-spec. Just let it run and run.

Multiple implementations of the client written before most of the previously talked about stuff is done however makes things just worse, not better.

Some people talk about how the current issue is solved by telling everybody complaining to go write some C++ to fix the issue. This attitude could not be more wrong. I understand that the proper specs are not maybe going to emerge in the near future. What however should really be done first before coding begins is some whitepapers addressing the questions how the current problems are going to be solved. At least two subjects have become quite clear are needed.

One whitepaper that goes through all the issues related to capacity how much it is going to be needed and how that goal is going to be reached. From this whitepaper then can be seen how much the increase of blocksize makes actual sense or not.

Another whitepaper should be dedicated on how to deal with all the issues related to spam, both friendly spam like SD and also hostile spam from someone wanting to bring bitcoin down whether they are some botnet, fiat-world-banking institution or black hand of some government.

After these whitepapers exist and hopefully multiple attempts at them they can be discussed and some action planed based on them.

I am optimistic that the capacity issue is solvable somehow. However I am not sure the spam issue is solvable at all. It might be that since spam can be made to be totally similar to real traffic, there is nothing that could be used to even detect spam and much less something that could be used to deal with it without massive sideeffects. I hope to be proven wrong with maybe when the whitepapers emerge.

You've made me want to eat custard creams now, curse you

Bitcoin certainly needs all these things, and for the reasons that you gave. But we should be realistic about what is possible and/or probable given the current environment. A good "first step" on the road to rigor would be a tightening up of the development process, realignment of priorities for what is worked on, and a commitment to improve the process.

I refuse to say anything negative about Gavin or any of the other bitcoin devs but the above is missing and it is a must.

The bitcoin protocol is Implementation defined, which means that no-one can go write a client in a language of their choice. It's the one official client or it won't work right.

Also add robust testing to the above list. But don't underestimate the complexity of a fit for purpose testing setup for complex software that has to work flawlessly under all situations and with many historic versions of clients.

thats sounds more like a saying than his professional opinion

http://pinterest.com/ameliamitta/the-greatest-sayings-of-all-time/

Can't we just stick to the original Satoshi's implementation and build all other features on the top of it? Some guys decided they were clever than Satoshi and now we have a lot of problems.

PS: Of coz I'm only semi-serious.

OK, no offense but this sentiment keeps coming up over and over again from young zealous programmers who truly believe in the Church of Turing.

Let me just point this out, all successful internet protocols start out or end up with a human-readable specification that allows lots and lots of independent implementations in any programming language by  multiple independent teams of developers.


Here's a list of past successes:

IP
TPC/IP
SMTP
HTTP/1.1
SSL/TLS

If Bitcoin is ever to become a REAL INTERNET PROTOCOL and not just a large hairball of C++ which can talk nicely to its own clones, then it MUST have a wire-level message specification along with all the message semantics (must-tolerate past bugs included, etc), see the list of successful internet protocols above for how it's actually done, regardless of how many convoluted edge cases there are already out there in the wild ...

The only other alternative is to continue with the present mono-culture of genetically identical clones waiting for some clever hacker to discover the NEXT bug or the NEXT way to attack the entire network due to an "implementation" or "library" bug in the single source base, which can be triggered and activated to affect ALL the nodes, instead of just a fraction running a particular spec-compliant independent implementation.

Ponder that well ...

Cheers ...
 

Do you realize that most alternative independent implementations would never even use BDB and therefore would be completely unaffected by this particular issue ? That's the whole point of "genetic diversity"

The "spec" was not self-consistent, because on one hand it claimed blocks are a max of so many bytes yet it also claimed blocks of that many bytes only ever need a max of 10,000 BDB locks, which turns out to be false.

So, it was an incorrect / invalid / buggy / mis-specified / inconsistent "spec".

If we turn it into a human-readable spec, it should be read as blocks are at most one megabyte in size and the programmers working on the implementation should ensure that all details of the implementation are such that there are enough widgets / whatzits / whatevers provided (heap space, array size, whatever, locks if your database happens to use locks, etc etc etc) for blocks of that size to work.

Turns out that if you happen to use BDB for your persistence layer, you need more than 10,000 locks. Ooops.

-MarkM-

This "we need a spec!" thing has already been covered in a previous thread.

The consensus there was that no-one was willing to write it.

So, misterbigg, since you are so up in arms about there not being a spec, I humbly propose you start sifting the code for every skerrick of Bitcoin functionality, document it, produce unit tests for it, and update/maintain said spec. As Gavin has said in the past, he used to work on a big project doing these "specs" you wanted, and they spent so damn long going through committees, discussions and updates, the project died before it was ever realised. Gavin has a motto of if you want things around Bitcoin to happen then it is up to you to get the ball rolling. The onus is on YOU to get involved in Bitcoin.

So, gather some like-minded individuals, download the source, start contacting the main devs that are willing to spare time to help you, and make it happen. Appealing to the greater community to do it for you shows you aren't really serious about this and are just waving your arms hoping you can inflate discontent so others will do it for you.

And no, the devs don't "need a spanking", they need a cold beer and a pat on the back.

From what I've seen it's not willful malice but rather that the people involved, while good at the technical challenges of, are a bit rusty at the skills necessary to groom code and make it palatable for a large programmer audience. Admittedly this is boring work which is why the current implementation is hard to build and resembles a hairball. Everyone wants to work on cool stuff, and places no value in something that is easily maintained or understood.

Then go do it and get involved. I would love to see this also but don't have time to, which is why I won't complain about the current developers.

From my limited time lurking around the dev IRC channel, it seems this need is understood but it is a large undertaking that the current developers a) don't need themselves and b) don't have time to do.

It seems for an interested person new to the code, getting involved in protocol documentation would be a great way to get up to speed on it. You clearly know code and can write better than most coders, so go ahead.

I think you are right but you are understating the problem. Practically everything that has been implemented across multiple programing languages or on multiple platforms has a formal spec. Your list is way too limited, there are specs of just about everything that goes though the Internet on http://www.ietf.org/rfc.html (http://www.ietf.org/rfc.html)

If there is no formal spec it's guesswork getting anything working. Bitcoin should never rely on implementation specific details from openssl or anywhere else, as we saw these are not even understood by the devs so what chance does anyone else have? The need for a written protocol specification is critical along with the need for better testing of clients. All other development should be halted until a full and complete spec is in place, anything else is cowboy coding and it will cause more screwups like the chain split we recently saw. If the current devs can't handle this we should not blame them, we should assist them by collecting money to pay for full time people to handle nothing but getting the spec ready.

Once that spec is in place programmers will appear to code to it in their favorite languages. We will get clients and libraries for everything and the bitcoin world will be a hell of a lot better for it.

I, for one, would be happy to contribute to a fund to hire professional engineers to produce a spec.

+1

As Pieter wrote on bitcoin-development list,


Or restated:  The fundamental problem being solved by bitcoin at a technical level, on a daily basis, is the distributed consensus problem (link (http://en.wikipedia.org/wiki/Consensus_%28computer_science%29)).

We fully support the writing of specifications and documentation, which you can see here
    https://en.bitcoin.it/wiki/Protocol_specification (https://en.bitcoin.it/wiki/Protocol_specification)

And changes to the existing protocol are formally documented here,
    https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals (https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals)

Ultimately the operational definition of consensus comes from what the network accepts/expects, not a theoretical paper.  Specification practices are healthy as a manual, human-based method of achieving consensus on network protocol rules.  Alternate client implementations (c.f. heterogeneous environment (http://en.wikipedia.org/wiki/Heterogeneous_computing)) are another good practice.

But the collective software rules are always the final specification, by definition.  That is what bitcoin does, achieve consensus.

A few other observations:

Gnutella had a business and project environment with co-motivated individuals working on a few key codebases.  The reference codebase in bitcoin, in contrast, has one paid developer (Gavin@BF) and a few part time unpaid volunteers.

All the big bitcoin businesses seem to either (a) contribute to BF, (b) use bitcoind without contributing back any testing/dev/specification resources, or (c) do their own thing entirely, not contributing back any testing/dev/specification resources.

Bitcoin is a thing, an invention, not a funded project with a built-in set of professionals paid to ensure full spec/dev/test engineering effort.  If you want something, DO IT.  You cannot expect the engineering resources to do X to magically appear, just because you complained on an Internet forum.

In an unfunded open source project, arguing all day about the lack of full-engineering-team rigor is entirely wasted energy.  Blame the dev team if that is your favorite target, that will not magically create extra time in the day or extra manpower to accomplish these extra tasks being demanded by non-contributors.

The time spent whining about what an unfunded effort fails to do could be better spent, say, creating a test network of full nodes running all known bitcoind versions, 0.3 through present.  And test, test, test changes through that.

So I take it the Foundation is unable to do anything of above? Why?

I have suggested long ago that miners paying 1% of their income to core dev team, with today's BTC value, they will get reasonably paid

This thread OP is based upon a false premise.  Namely that Gnutella failed because of it's technical difficulties.  This is not true.  Gnutella failed because it wasn't distributed well enough, and was replaced by similar p2p networks that were better suited to same.  This is well documented in the book, The Starfish and the Spider (http://en.wikipedia.org/wiki/The_Starfish_and_the_Spider).  Bitcoin is, by design, as decentralized as is probably possible, as only cash transactions in meatspace are more decentralized.  When I read that book, I immediately thought that Satoshi must have already read it him/her self also. 

You completely missed the point. I wasn't referring to Gnutella's failure at all. I was highlighting the fact that Gnutella had a protocol and multiple competing implementations which all interoperated. Both commercial and non-commercial interests worked together for the health of the network and encouraged standardization. Documentation on the wire format and proper behavior was copious. Compare and contrast this with Bitcoin.

Gnutella failed for legal reasons, but that is irrelevant.

I consider Gnutella versus Bitcoin to be an apples to oranges comparison.  And Bitcoin does have the above, anyway.  A single, short lived, event such as happened yesterday is not evidence of the contray; it's evidence towards it's effectiveness.  Imagine if microsoft had been in charge of fixing the running code!

I guess there is something to the Apollo mission movie analogies.

Maybe its time to watch that movie again more carefully?

-MarkM-

Like TCP/IP you mean? Or SSH? Or SMTP? IMAP? LDAP? DNS?

Ultimately it doesn't matter what's in their theoretical papers if the implementations are different so why bother writing them? It's not like all the major people with their different platforms and different coding practices could ever implement compatible things. Oh wait, they did precisely that because there were specs to follow.

I'm not bashing the devs here. I'm suggesting that BitCoin right now has outgrown the development capacity provided by one full timer who hates specs and a bunch of volunteers. Explicitly not because these people are bad, but because these people are not enough. I would contribute some real money to help improve the situation but I'm not a miner. If some mining pools could be convinced to offer the option to donate a small percentage of mined coins to a bitcoin dev fund we might stand a chance of getting a few full timers on this. We can't prioritize work done by volunteers at all, but we ( the fund, whoever ) can prioritize work done by paid employees.

One of the biggest flaws of BTC is the tx fee.

As the price of BitCoin inevitably increases, Smaller and Smaller BitCoin amounts are going to become the norm. After which point, the only BTC Transactions worth even thinking about sending would be in the thousands of dollars value range!

I mean, operating a faucet like site myself. This is extremely clear to me.

Thank devo for devcoins, heck with those things you could buy a stick of gum and still have it come out to a whole coin or more!

-MarkM-

+1

The source code is the best documentation possible, and I dont wish to see few other implementation from third party, this may just brings problems..

If you only ever want one client, then who cares whether the source code provides good documentation or not? You'll use the software you're given because that's all there is. Your statement makes no sense.

We've reduced the minimum fee setting twice already since I've been here.  The transaction fee is a non-issue.

+1

The majority user of the network, SD pays miner and shareholder and does not care to reduce its footprint even though it would take few lines of code and cost nothing on their side. The miner could fund with 1% of their revenue 4 full time developer and are not even prominent member of the foundation that pays for one.

We seem to already suffer the tragedy of the commons.

I think this is where some of the disagreement is. I prefer to have multiple clients, for a long term(next 20 years and on) safety it's better and safer. And earlier it's done the better. If crypto-currency were to be anything very much larger than software toy of small geek population it needs to be in multiple hands...

Other argument is that is stability of system more important than it not being centralized in some way and one implemention ruling over all is centralised power in my mind.

Gavin and myself certainly support alternate client implementations (heterogeneous environment (http://en.wikipedia.org/wiki/Heterogeneous_computing)).

No, you are suffering the tragedy of arrogance.

So has anyone actually started a bounty for forming a BTC spec?  Complaining is a much poorer incentive than money.  C'mon complainers, put your money where your mouth is.  I might even chip in.  :)

A textual capture of it could be a good start, but clearly insufficient. Some aspects of the protocol will need to be captured as snippnets of code or formulae to be exact and purposeful, and the definition will have to be supplemented with an extensive set of test vectors and test scenarios. This is a major effort.

Once finished such definition should take over the role of reference even if a scheduled hard fork is needed to enforce to align existing implementations. I can not imagine this economy growing by magnitudes running a single sacred code maintained by a few who preserve its bugs and undefined behavior to eternity.

Should there be crowd funding the effort, I for one, would be contributing to the definition, provided:

Sources of funds declare the funds as irreversible donation to a project separate of their businesses and that they expect no return other than documents, data and programs that immediately become goods of the public domain. Furthermore contributor of work should be set free of claims referring to the result's incompleteness, consequences of its implementation or if the project is abadoned without implementation or influence to the network.

After reading all the responses to the issue, I think I would be satisfied if I could just build the thing on Windows using Visual Studio 2010 without fetching endless external dependencies and without a set of build instructions that rivals the size of the Bible (old testament).

Doesn't Visual Studio collect the dependencies for you automatically like maven and netbeans and eclipse and, I had somehow imagined, pretty much any GUI build-system?

-MarkM-

So that's how they trojan the enterprise resource planing systems! I've seen and heard of some deployments getting trojaned through the internal development departments. I thought that it was some sort of more advanced exploit. But all it takes is some lazy-ass Java code-monkeys that press the "collect dependencies" button in their IDE!

Thanks. Very usefull knowledge, that I was completely unaware of. I always thought that it requires some click-on-the-email-attachment social engineering trick. But it is through the click-some-button-in-the-IDE, no need for social engineering, the engineers are already conditioned for brainless clicking.

Thanks again.

I've compiled many, many C++ apps in my life. Bitcoin isn't particularly good or bad relative to the others. Manually installing dependencies is boring but is inherent in unmanaged app development.

If it makes you feel any bigger, sipa would love to ditch the OpenSSL dependency ;p

(and has been coding in that direction)

That is one thing that is nice about Gentoo. You don't have to do that manually.

Oh great, its not enough to change the world, now Satoshi has to retroactively learn "proper" C++ coding style before doing so! :)

Try to clean it up. Rumours hint doing so might have some lurking "gotchas" but maybe that is just idle superstitious scuttlebutt used as an excuse for doing fun stuff instead of code-monkey housecleaning chores.

-MarkM-

It is. Four years after the project is in production it still looks like a proof of concept code written by an academic genius.
Once you scratch the surface you see that it is actually high quality of its own style (of the early 90s).

I can only guess why refactoring did not happen as follows:

1. There were no unit tests, so people did not dare to change anything non-trivial. This is improving...
2. C++ is too complex and has too many possible side effects for refactoring tools.
3. People maintain it got used to it and only see the beauty that is well below the surface.
4. Is more sexy to add features, tune performance than clean up code, that only have downside of breaking things.

Yep, I noticed and it is definitely great to hear that.

The repositories in my sig (LayerEffects, SimpleDJ, and DSP Filters in particular) should serve as the model. All three are GUI apps, have several dependencies each, are self-contained repositories, and build and run on GNU/Linux with X-Windows, OS X, and Windows.

You can do a git clone on any one of those, and without having to pull any other dependencies build the executables and run them.

They also build really fast, being organized in the "unity build" style, and some of them make use of amalgamated source distributions.

My comment was not about the code but about the code development workflow. Anyone can make a mistake, c.f.
I was just not aware that the problem is so prevalent, that so many people developed such a dependency on build automation that they are incapable of building without it.

It used to be that Microsoft Outlook promised increased productivity by allowing office automation. It ended up with security consultants training the cubicle monkeys to avoid clicking on the e-mail attachments.

Then there was Visual Basic and certain VB programmers for whom any project could be improved by downloading some random ActiveX control from some random web site.

I only just yesterday realized that I've already encountered one such situatuation where the SAP/Netweaver/Java deployment was trojaned. I normally don't do sales calls, but in one unusual situation I've met with a prospective clients after a "security event". They asked if we use "Maven, Netbeans, Eclipse" and were very happy to hear that all our developers are comfortable working with "vi or emacs and our own tools". I didn't pay attention to their secuirity consultants talk about "advanced persistent threats" or some such.

So the summary is that Visual Studio, which I sometimes call Visual Straitjacket has one significant benefit: it makes somewhat more dificult to trojan a whole dev-team using automated dependency collection.

I'm sorry for the off-topic post: it has no relation to Gnutella, but it is related to the security of the build process.

tl;dr:
Build automation with dependency resolution is a major productivity tool, that I would not want to miss. Its use can be adapted to a security aware environment by self hosting artifact repositories.

personal remark:
Although I wrote my first professional programs punching paper cards for (a russian clone of) an IBM 360, still managed to move on to virtual punchcards on a terminal, than vi, than Emacs than to IDEs and recently using maven with Eclipse. I do see the advantage of all and fall back myself to vi and command line quite often. I am skeptical of new developments (since they are mostly not new or better at all), but am not outright rejecting them. The only constant in this industry is the exponent of change. I would be scared if I would not change my tools for years, since all those younger people can not be idiots, I must be old if that happens. BTW, now learning Scala

I don;t think TCP/IP, DNS or similar technologies are good analogies to bitcoin. All these technologies are very centralized (authorities assigning IP address ranges, root DNS servers etc.). I think bitcoin can be (to certain extent) compared to maybe HTML. There were always incompatibilities between browsers and having w3c as an authority didn't really help that much. So maybe having single or multiple reference implementations might be actually better than having a spec that no implementation fully follows. What I think should be done is to ask current developers to help complete one other implementation (maybe bitcoinj/multibit full verifying node might be a good candidate), not necessarily by coding, but by consultations etc. Then we would have two reference implementations in the wild and hopefully they would gain equal market share. That would be a much healthier environment compared to today's situation.

If there was 10 implementations of the spec, then the bug would be limited to 10% of the users.  They would find that their block rate drops to 10% and

Miners would be well advised to have all blocks verified by multiple implementations, so they can react to forking.

Users of bugged, i.e. non-compliant, nodes would have to upgrade or end up on a fork.

Same here - question is, if someone started such a fund, would BF and the volunteer developers even pay attention to it? That is to say, if we crowd-sourced money to get a few full-time engineers to run all the versions on a test network and develop a protocol spec - in the end, would anyone be remotely interested in it? Esp. since we have a main dev. who is opposed to having a protocol specification (or at least so I've read.) And other devs just don't seem too keen on the whole idea.

Note that I have nothing against the devs, not here to spank them or whatever; just being realistic about their opinions regarding developing specs. I know they work hard. I just want more direction, focus, a cleaner approach to development that encourages alternative clients operating via a well-defined protocol.

On the other hand, maybe BF would eventually hire more full-timers to do that - work on cleaning code, writing protocol, testing, etc. The boring stuff that no one wants to do but is essential to not doing this whole ad-hoc thing with spaghetti code filled with magic numbers. Or maybe someone will start their own foundation that will perform this function and work with BF and the current developers.

Oh yeah, and magic numbers, another pet peeve of mine. Someone posted an IRC paste of Gavin just yanking a projected yearly % increase in broadband out of his butt (based on past numbers, so of course will be accurate in the future) and suggesting basing one of our "magic numbers" in the code on that. Not picking on Gavin, but there's a bigger problem that is illustrated by that kind of thinking.

This thread has some great thoughts and ideas; we need to figure out what to do to make some of these things happen. I could donate more money to BF but for all I know, they aren't even remotely interested in developing a spec or de-tangling the code so who knows, maybe I'd just be contributing to the problem.

I don't like the sentiment that if you don't know how to code, don't complain - there are dozens of other ways one can take action. Organize a bounty/fund, donate to a fund, raise awareness via discussion, donate to BF, install & test patches and alt. client for people who do code, etc.

It would be great if I didn't have to run bitcoind behind armory... their reason for relying on bitcoind basically boils down to "It's spaghetti with no defined protocol spec, what the heck do you expect us to do?" although they explain it nicer than that.

"Standardizing Bitcoin" is the top item in BF's charter https://bitcoinfoundation.org/about/.

If that's not funding and hosting a protocol spec the same way that W3C consortium does, I don't know what that means.

So far, they are failing to make any movement in that direction.

Their comment is:


That suggests that "standardizing" just means improving the quality of the reference client?

Simplest explanation is that BF basically makes it up as they go along, there is no cohesive plan.

So far the dev. team, despite all its voluntary (except Gavin) and heroic efforts, only pays lip service to the idea of alternative implementations.  If they really wanted to see those, they would actually extract the spec from the code and target their own implementation to that spec, for say, 1.0 version, then it would be feasible to create alternative implementations in many diff. languages could that use the spec. as standard to interoperate.

Dev team's positions (usually expressed by sipa) is that "consensus of the running network is more important than any notion of correctness or spec compliance" (paraphrasing). In other words, Bitcoin is special because it's not a client-server style protocol but rather a live network that must maintain consensus at all costs, however, consensus could be defined as agreement on past view of history (the blockchain) or as identical behavior with respect to new messages occurring in the network, which "consensus" is mean or desired has been left unspecified so far...

It's a sad situation, because a few people have tried and given up to extract and re-implement the spec, but it's always a moving target (new commits change protocol message semantics) and also subject to as-yet not well understood bugs and edge cases in the existing Satoshi client. 

Cheers ...

The writing is already on the wall: Ripple

I would argue that Ripple in its current (and planned) implementation is more centralized than bitcoin, however, the concept behind ripple, if it were made truly P2P and decentralized, could be a superior alternative provided it solved these problems with the Satoshi client (impossible to make functioning alternative implementations because the code IS the spec and the spec is constantly changing, make it up as we go along hey)
Although, really, if any alt-coin decided to go the spec route and develop a spec that any client could adhere to, it'd have some potential.

I don't see how Ripple adds any value to Bitcoin at all, unless certain people are successful at blocking progress with regards to scalability. I do see how Ripple can make it easier to transfer national currencies.

Wrong.  I have written two alternate implementations:

Counter example #1: https://github.com/jgarzik/picocoin/

Counter example #2: https://github.com/jgarzik/python-bitcoinlib/ and https://github.com/jgarzik/pynode/

Thats great, hopefully the documentation devs will compare those to the Satoshi node and document what it is that they all do in common.

-MarkM-

These are just examples of alternative implementations created by an insider already intimately familiar with both C++ and the existing C++ code base.

The whole point of a good specification is to allow outsiders (whole other teams really) who program in, say, Scala, Erlang or Clojure, to create interoperable implementations based solely on contents of the human-readable spec. (plus test data could be shared in a separate test kit) without having to become both a C++ expert (and go through all the overhead of setting up a functional Bitcoin development environment) and without having to have extensive experience with the existing C++ code base to understand protocol semantics buried therein and without having to follow all the ongoing commits to the existing C++ codebase to make sure the semantics haven't been changed (i.e. recent Dead Puppies change).

Once there is a spec, then a spec evolution process (i.e. BIPs) would amend the spec. and all the implementations would target a particular future version of it.

Speaking of BIPs, they need to mark BIPs that have been included in the reference client.  Bloom filtering is in the client, but still has a status of accepted.

On spaghetti code, I would venture some advice that has worked for me over the decades. Upon "inheriting" some "old code", that "just has to work", I work on a "parallel" version, keeping the original for "current production" and the new for "eventual replacement".

The "parallel" version is not a "refactoring" or rearranging of the code in any fashion, at least not initially. I worked in C and before that assembler, but I see a lot of C in the bitcoind code!

First one must have all the code "in view" and as much in one's head as possible. And using the best IDE and tools one has, begin to first name things properly. This means everything, from class definitions to instance names, methods, member variables. And very important, no magic numbers, like method_name( argument_variable, 0 ). What is the 0? What does it represent?

For example, I am looking at db.h, ~line 117 in the CDB.Read template area. And there is a magic  ssKey.reserve(1000);
OK what does the 1000 represent? Is it important? Does it matter? Shouldn't it be a
const intergal_type Meaningfully_named_Value = 1000;

I know how it is when one is "in the zone", and coding. Documentation is superfluous and slows one down! And in the 1970s, especially, less so in the 1980s, and basically no more in the 1990s and on, variable names were no longer restricted to 8 or 16 characters, or 32 characters (of significance), etc. And [floppy] disc storage didn't force a lot of "vertical" compression (read K&R style here). Nor horizontal compression (read no spaces between operators and operands here). I see a lot of that "residue" in the code.

Just renaming things doesn't change the executable, AFAIK, in a release build. So I would globally, with a careful global search and replace, acknowledging each replacement, names that are obviously too terse, obscure or no longer even representative of what they are, with more meaningful names.

I have done this with some God awful C chess code, to good effect some time ago. One finds dead code, duplicate globals, etc. etc.

It is tedious but neccessary, and the result will be the same .exe file but the code will start to look readable.

I have had many an argument with "upper management" about spending the time to make a code base readable and where the names truly reflected what they are. Otherwise I am constantly translating with a symbol table in my head, what everything means. I hate that, and I hate having to remember things that I know I shouldn't have to!  Example, what is the instance bitdb, see line 29 in db.cpp rally a class of? Then what is the Class CDBEnv{} really? Is it a BerkeleyDB or a LevelDB, or a ? Of what? Wallet, block chain info? I keep digging, hoping to find a comment or some "action" that might nail it down to something definite. Then I can rename it from bitdb to something more meaningful.

I propose that style of renaming, rearranging, etc. that leaves the executable, byte for byte identical. Maybe then we can see the beauty of the forest thorough the ugly arrangement of the trees. I think the .h files especially need a lot of "work". I see hodge podge lines of extern this or that sprinkled in .h and .cpp files!

I would propose this sort of "one to one transformation" of the code to start with, and then it may become easy to: find the bugs that are there, add new "stuff", enhance old "stuff", etc. etc.

BTW, I am doing it myself, off to the side anyway, since I hate K&R style, and all the stuff mentioned above, and more (LOL)

Ron

Please do, then submit a pull request to this github project: https://github.com/bitcoin/bitcoin

Really, it says Ripple on the wall.  I promise to lift this cover off the wall to show you, sometime soon. 

Is there software tools for verifying equivalence ?


That would do it.  However, presumably, object code would have variable names.  Is the linker deterministic, I wonder.

Hash maps could end up in a different state if different keys are used (assuming the compiler sometimes iterates).

You can easily strip debugging symbols and just leave offsets/pointers.  The linker should be deterministic as long as you don't update any libraries in between.  It really depends on your compiler and language.  I know there are C compilers that will produce identical binaries regardless of var name, but I'm less sure about C++.

Consensus in Bitcoin. Who is it for? Who is it by? :  

A number of types of steakholder Come up when I think about Consensus, in the Context Of Bitcoin.  

Core devs of Bitcoind/QT
Devs of other implementations
Miners
Merchants
Users  

It would be helpfull to have clarification on this question of What is really meant by the devs themselves when they talk about consensus. What is the intended scope and how is it decided. I have a fair idea how it goes already, but it's not an explicit protocol itself.

This sugestion of the OP, kinda gets down to how important is it to the devs/BF, to foster consensus as broadly as possible. Should it be consensus of only those who do core code, or also those who can read code but only use & test the software?

A concise and clear protocol definition should help not only potential alternative implementation devs, but also people who are not actualy coders (or maybe neophytes) to understand the technicalities and also in dealing with more admin/systems-analysis type matters. That in turn, is critical to feilding informed consensus from the non-developer community.

That said, is there any reason why those of us with more peripheral skills shouldn't take the inititive and help ourselves to the problem/task? I think it would be a great spring-board into more serious coding for anybody with intermedite or less skills, who may be a bit apprehnsive of coding in the client itself. Also the work on profiling and style, sounds like a great student task.

Maybe we need more of an overt system for dev mentoring of newbie programers of whom they may co-opt into the less system critical and more mundane house keeping chores. Of course, the proceeds of a bounty or funding drive couldnt hurt either. I would happily throw a couple of btc into the hat to see a dev mentor system founded.

About the existing specs that have allready been mentioned here; what was the problem with them again?  

I might take a look at them and the source, just to see if a dunce amongst the geeks can gleen some differentiation between the interface code, the main body of the protocol and the rest.

It may help if the core devs could develop a habit of some infomal comment markup, for less invasive follow up work. If they are going over a part that could easily be abstracted (without busting it), they might put in some thing like:  

 // _dev_abst_&test @ 18333 .... lines=45

That could denote that the following 45 lines have been earmarked for abstraction chores and the resulting build should be tested on port 18333 or whatever the testnet port is. Then the student devs could use the  comment notations to pick out bits of work that are usefull and educational, but otherwise boring and mundane to the main devs. If these changes were held off the table and not released in the next one or two releases, then testing could ensure the changes were safe to merge into the main codebase and release into the wild.

I have the utmost respect for the devs however much they achive. They could easily be coding for their own bitcoin startups, mining, or just go fishing. But if the work load is accumulating, then the recruitment of new devs is a good option. When in doubt - mumble. When in trouble - delegate.  ;D

The poverty of the foundation/devs/community shouldn't even need to be raised, in a movement that is manifesting the future of money; one I might add, that's made some very wealthy people. Patrons shouldn't be hard at all to come by.