What Bitcoin Could Learn From Gnutella (or, why devs need a spanking)

[Bitcoinpro]

"If you can't keep up, don't step up?"

thats sounds more like a saying than his professional opinion

http://pinterest.com/ameliamitta/the-greatest-sayings-of-all-time/

[1714618805]

1714618805

[1714618805]

1714618805

[Come-from-Beyond]

Can't we just stick to the original Satoshi's implementation and build all other features on the top of it? Some guys decided they were clever than Satoshi and now we have a lot of problems.

PS: Of coz I'm only semi-serious.

[alexkravets]

The source code is the best specification. No documentation written in human language be complete and unanimous enough to be sure that everything is covered. You cannot compile Hemingway writings into executable code.

Bitcoin with connectivity difficulties would have problems with different Bitcoin clients, but will happily create disconnected network and all sorts of other nasty things.

Bitcoin have much more at stake than Gnutella warez download.

OK, no offense but this sentiment keeps coming up over and over again from young zealous programmers who truly believe in the Church of Turing.

Let me just point this out, all successful internet protocols start out or end up with a human-readable specification that allows lots and lots of independent implementations in any programming language by  multiple independent teams of developers.


Here's a list of past successes:

IP
TPC/IP
SMTP
HTTP/1.1
SSL/TLS

If Bitcoin is ever to become a REAL INTERNET PROTOCOL and not just a large hairball of C++ which can talk nicely to its own clones, then it MUST have a wire-level message specification along with all the message semantics (must-tolerate past bugs included, etc), see the list of successful internet protocols above for how it's actually done, regardless of how many convoluted edge cases there are already out there in the wild ...

The only other alternative is to continue with the present mono-culture of genetically identical clones waiting for some clever hacker to discover the NEXT bug or the NEXT way to attack the entire network due to an "implementation" or "library" bug in the single source base, which can be triggered and activated to affect ALL the nodes, instead of just a fraction running a particular spec-compliant independent implementation.

Ponder that well ...

Cheers ...
 

[alexkravets]

The source code is the best specification. No documentation written in human language be complete and unanimous enough to be sure that everything is covered. You cannot compile Hemingway writings into executable code.

Bitcoin with connectivity difficulties would have problems with different Bitcoin clients, but will happily create disconnected network and all sorts of other nasty things.

Bitcoin have much more at stake than Gnutella warez download.

Source code which contains unknown cases isn't acceptable for system (to be) used as medium of exchange. Bitcoin does have much much more at stake than some filesharing network, which is why there should be spec to follow.

For something that my life depend on, I want certainty that it's well understood and tested.

I'm all for having a spec that allows an ecosystem of multiple compatible implementations to arise, but in this case it wouldn't have prevented this problem. No amount of examination of the Bitcoin source code would have revealed the limitation in BDB, and the resulting need for a configuration file to change the BDB default values.

Unit tests might not have even caught the problem because they said that not all 0.7 nodes rejected the large block. This implies that different BDB versions and/or platforms have different default values so consequently not all of them would have exhibited the problem, possibly including the platform used to run the unit tests.

Do you realize that most alternative independent implementations would never even use BDB and therefore would be completely unaffected by this particular issue ? That's the whole point of "genetic diversity"

[markm]

The "spec" was not self-consistent, because on one hand it claimed blocks are a max of so many bytes yet it also claimed blocks of that many bytes only ever need a max of 10,000 BDB locks, which turns out to be false.

So, it was an incorrect / invalid / buggy / mis-specified / inconsistent "spec".

If we turn it into a human-readable spec, it should be read as blocks are at most one megabyte in size and the programmers working on the implementation should ensure that all details of the implementation are such that there are enough widgets / whatzits / whatevers provided (heap space, array size, whatever, locks if your database happens to use locks, etc etc etc) for blocks of that size to work.

Turns out that if you happen to use BDB for your persistence layer, you need more than 10,000 locks. Ooops.

-MarkM-

[SimonL]

This "we need a spec!" thing has already been covered in a previous thread.

The consensus there was that no-one was willing to write it.

So, misterbigg, since you are so up in arms about there not being a spec, I humbly propose you start sifting the code for every skerrick of Bitcoin functionality, document it, produce unit tests for it, and update/maintain said spec. As Gavin has said in the past, he used to work on a big project doing these "specs" you wanted, and they spent so damn long going through committees, discussions and updates, the project died before it was ever realised. Gavin has a motto of if you want things around Bitcoin to happen then it is up to you to get the ball rolling. The onus is on YOU to get involved in Bitcoin.

So, gather some like-minded individuals, download the source, start contacting the main devs that are willing to spare time to help you, and make it happen. Appealing to the greater community to do it for you shows you aren't really serious about this and are just waving your arms hoping you can inflate discontent so others will do it for you.

And no, the devs don't "need a spanking", they need a cold beer and a pat on the back.

[misterbigg]

what is not widely embraced, is competition. Or co-operation, or a healthy ecosystem of Bitcoin-based currencies all interacting with each other.

From what I've seen it's not willful malice but rather that the people involved, while good at the technical challenges of, are a bit rusty at the skills necessary to groom code and make it palatable for a large programmer audience. Admittedly this is boring work which is why the current implementation is hard to build and resembles a hairball. Everyone wants to work on cool stuff, and places no value in something that is easily maintained or understood.

[rocks]

* Develop a comprehensive protocol specification from the code and stick to it
* Feature freeze until a protocol has been written
* Eliminate external dependencies in the reference client
* Define formal process for commits to the official repository

Agreed

Then go do it and get involved. I would love to see this also but don't have time to, which is why I won't complain about the current developers.

From my limited time lurking around the dev IRC channel, it seems this need is understood but it is a large undertaking that the current developers a) don't need themselves and b) don't have time to do.

It seems for an interested person new to the code, getting involved in protocol documentation would be a great way to get up to speed on it. You clearly know code and can write better than most coders, so go ahead.

[sd]

Let me just point this out, all successful internet protocols start out or end up with a human-readable specification that allows lots and lots of independent implementations in any programming language by  multiple independent teams of developers.


Here's a list of past successes:

IP
TPC/IP
SMTP
HTTP/1.1
SSL/TLS

I think you are right but you are understating the problem. Practically everything that has been implemented across multiple programing languages or on multiple platforms has a formal spec. Your list is way too limited, there are specs of just about everything that goes though the Internet on http://www.ietf.org/rfc.html

If there is no formal spec it's guesswork getting anything working. Bitcoin should never rely on implementation specific details from openssl or anywhere else, as we saw these are not even understood by the devs so what chance does anyone else have? The need for a written protocol specification is critical along with the need for better testing of clients. All other development should be halted until a full and complete spec is in place, anything else is cowboy coding and it will cause more screwups like the chain split we recently saw. If the current devs can't handle this we should not blame them, we should assist them by collecting money to pay for full time people to handle nothing but getting the spec ready.

Once that spec is in place programmers will appear to code to it in their favorite languages. We will get clients and libraries for everything and the bitcoin world will be a hell of a lot better for it.

[coqui33]

I, for one, would be happy to contribute to a fund to hire professional engineers to produce a spec.

[gusti]

+1

[jgarzik]

As Pieter wrote on bitcoin-development list,

The protocol is whatever the network enforces - and that is some mix of versions of the reference client right now, but doesn't need to remain that way.

I would very much like to have a text book of rules that is authorative, and every client that follows it would be correct. Unfortunately, that is not how a consensus system works. All (full) clients validate all rules, and all must independently come to the same solution. Consensus is of utmost importance, more than some theoretical "correctness". If we'd have a specification document, and it was discovered that a lot of nodes on the network were doing something different than the document, those nodes would be buggy, but it would be the specification that is wrong.

Or restated:  The fundamental problem being solved by bitcoin at a technical level, on a daily basis, is the distributed consensus problem (link).

We fully support the writing of specifications and documentation, which you can see here
    https://en.bitcoin.it/wiki/Protocol_specification

And changes to the existing protocol are formally documented here,
    https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

Ultimately the operational definition of consensus comes from what the network accepts/expects, not a theoretical paper.  Specification practices are healthy as a manual, human-based method of achieving consensus on network protocol rules.  Alternate client implementations (c.f. heterogeneous environment) are another good practice.

But the collective software rules are always the final specification, by definition.  That is what bitcoin does, achieve consensus.

A few other observations:

Gnutella had a business and project environment with co-motivated individuals working on a few key codebases.  The reference codebase in bitcoin, in contrast, has one paid developer (Gavin@BF) and a few part time unpaid volunteers.

All the big bitcoin businesses seem to either (a) contribute to BF, (b) use bitcoind without contributing back any testing/dev/specification resources, or (c) do their own thing entirely, not contributing back any testing/dev/specification resources.

Bitcoin is a thing, an invention, not a funded project with a built-in set of professionals paid to ensure full spec/dev/test engineering effort.  If you want something, DO IT.  You cannot expect the engineering resources to do X to magically appear, just because you complained on an Internet forum.

In an unfunded open source project, arguing all day about the lack of full-engineering-team rigor is entirely wasted energy.  Blame the dev team if that is your favorite target, that will not magically create extra time in the day or extra manpower to accomplish these extra tasks being demanded by non-contributors.

The time spent whining about what an unfunded effort fails to do could be better spent, say, creating a test network of full nodes running all known bitcoind versions, 0.3 through present.  And test, test, test changes through that.

[rini17]

As Pieter wrote on bitcoin-development list,

The protocol is whatever the network enforces - and that is some mix of versions of the reference client right now, but doesn't need to remain that way.

I would very much like to have a text book of rules that is authorative, and every client that follows it would be correct. Unfortunately, that is not how a consensus system works. All (full) clients validate all rules, and all must independently come to the same solution. Consensus is of utmost importance, more than some theoretical "correctness". If we'd have a specification document, and it was discovered that a lot of nodes on the network were doing something different than the document, those nodes would be buggy, but it would be the specification that is wrong.

Or restated:  The fundamental problem being solved by bitcoin at a technical level, on a daily basis, is the distributed consensus problem (link).

We fully support the writing of specifications and documentation, which you can see here
    https://en.bitcoin.it/wiki/Protocol_specification

And changes to the existing protocol are formally documented here,
    https://en.bitcoin.it/wiki/Bitcoin_Improvement_Proposals

Ultimately the operational definition of consensus comes from what the network accepts/expects, not a theoretical paper.  Specification practices are healthy as a manual, human-based method of achieving consensus on network protocol rules.  Alternate client implementations (c.f. heterogeneous environment) are another good practice.

But the collective software rules are always the final specification, by definition.  That is what bitcoin does, achieve consensus.

A few other observations:

Gnutella had a business and project environment with co-motivated individuals working on a few key codebases.  The reference codebase in bitcoin, in contrast, has one paid developer (Gavin@BF) and a few part time unpaid volunteers.

All the big bitcoin businesses seem to either (a) contribute to BF, (b) use bitcoind without contributing back any testing/dev/specification resources, or (c) do their own thing entirely, not contributing back any testing/dev/specification resources.

Bitcoin is a thing, an invention, not a funded project with a built-in set of professionals paid to ensure full spec/dev/test engineering effort.  If you want something, DO IT.  You cannot expect the engineering resources to do X to magically appear, just because you complained on an Internet forum.

In an unfunded open source project, arguing all day about the lack of full-engineering-team rigor is entirely wasted energy.  Blame the dev team if that is your favorite target, that will not magically create extra time in the day or extra manpower to accomplish these extra tasks being demanded by non-contributors.

The time spent whining about what an unfunded effort fails to do could be better spent, say, creating a test network of full nodes running all known bitcoind versions, 0.3 through present.  And test, test, test changes through that.

So I take it the Foundation is unable to do anything of above? Why?

[johnyj]

In an unfunded open source project, arguing all day about the lack of full-engineering-team rigor is entirely wasted energy.  Blame the dev team if that is your favorite target, that will not magically create extra time in the day or extra manpower to accomplish these extra tasks being demanded by non-contributors.

The time spent whining about what an unfunded effort fails to do could be better spent, say, creating a test network of full nodes running all known bitcoind versions, 0.3 through present.  And test, test, test changes through that.

I have suggested long ago that miners paying 1% of their income to core dev team, with today's BTC value, they will get reasonably paid

[MoonShadow]

This thread OP is based upon a false premise.  Namely that Gnutella failed because of it's technical difficulties.  This is not true.  Gnutella failed because it wasn't distributed well enough, and was replaced by similar p2p networks that were better suited to same.  This is well documented in the book, The Starfish and the Spider (http://en.wikipedia.org/wiki/The_Starfish_and_the_Spider).  Bitcoin is, by design, as decentralized as is probably possible, as only cash transactions in meatspace are more decentralized.  When I read that book, I immediately thought that Satoshi must have already read it him/her self also. 

[misterbigg]

This thread...is based upon a false premise...that Gnutella failed because of it's technical difficulties.

You completely missed the point. I wasn't referring to Gnutella's failure at all. I was highlighting the fact that Gnutella had a protocol and multiple competing implementations which all interoperated. Both commercial and non-commercial interests worked together for the health of the network and encouraged standardization. Documentation on the wire format and proper behavior was copious. Compare and contrast this with Bitcoin.

Gnutella failed for legal reasons, but that is irrelevant.

[MoonShadow]

This thread...is based upon a false premise...that Gnutella failed because of it's technical difficulties.

You completely missed the point. I wasn't referring to Gnutella's failure at all. I was highlighting the fact that Gnutella had a protocol and multiple competing implementations which all interoperated. Both commercial and non-commercial interests worked together for the health of the network and encouraged standardization. Documentation on the wire format and proper behavior was copious. Compare and contrast this with Bitcoin.

Gnutella failed for legal reasons, but that is irrelevant.

I consider Gnutella versus Bitcoin to be an apples to oranges comparison.  And Bitcoin does have the above, anyway.  A single, short lived, event such as happened yesterday is not evidence of the contray; it's evidence towards it's effectiveness.  Imagine if microsoft had been in charge of fixing the running code!

[markm]

I guess there is something to the Apollo mission movie analogies.

Maybe its time to watch that movie again more carefully?

-MarkM-

[sd]

Ultimately the operational definition of consensus comes from what the network accepts/expects, not a theoretical paper.

Like TCP/IP you mean? Or SSH? Or SMTP? IMAP? LDAP? DNS?

Ultimately it doesn't matter what's in their theoretical papers if the implementations are different so why bother writing them? It's not like all the major people with their different platforms and different coding practices could ever implement compatible things. Oh wait, they did precisely that because there were specs to follow.

I'm not bashing the devs here. I'm suggesting that BitCoin right now has outgrown the development capacity provided by one full timer who hates specs and a bunch of volunteers. Explicitly not because these people are bad, but because these people are not enough. I would contribute some real money to help improve the situation but I'm not a miner. If some mining pools could be convinced to offer the option to donate a small percentage of mined coins to a bitcoin dev fund we might stand a chance of getting a few full timers on this. We can't prioritize work done by volunteers at all, but we ( the fund, whoever ) can prioritize work done by paid employees.

[BitHits]

One of the biggest flaws of BTC is the tx fee.

As the price of BitCoin inevitably increases, Smaller and Smaller BitCoin amounts are going to become the norm. After which point, the only BTC Transactions worth even thinking about sending would be in the thousands of dollars value range!

I mean, operating a faucet like site myself. This is extremely clear to me.