What Bitcoin Could Learn From Gnutella (or, why devs need a spanking)

[grau]

tl;dr:
Build automation with dependency resolution is a major productivity tool, that I would not want to miss. Its use can be adapted to a security aware environment by self hosting artifact repositories.

personal remark:
Although I wrote my first professional programs punching paper cards for (a russian clone of) an IBM 360, still managed to move on to virtual punchcards on a terminal, than vi, than Emacs than to IDEs and recently using maven with Eclipse. I do see the advantage of all and fall back myself to vi and command line quite often. I am skeptical of new developments (since they are mostly not new or better at all), but am not outright rejecting them. The only constant in this industry is the exponent of change. I would be scared if I would not change my tools for years, since all those younger people can not be idiots, I must be old if that happens. BTW, now learning Scala

[awkorama]

I don;t think TCP/IP, DNS or similar technologies are good analogies to bitcoin. All these technologies are very centralized (authorities assigning IP address ranges, root DNS servers etc.). I think bitcoin can be (to certain extent) compared to maybe HTML. There were always incompatibilities between browsers and having w3c as an authority didn't really help that much. So maybe having single or multiple reference implementations might be actually better than having a spec that no implementation fully follows. What I think should be done is to ask current developers to help complete one other implementation (maybe bitcoinj/multibit full verifying node might be a good candidate), not necessarily by coding, but by consultations etc. Then we would have two reference implementations in the wild and hopefully they would gain equal market share. That would be a much healthier environment compared to today's situation.

[TierNolan]

An how would specification prevent what happened yesterday? Document the unknown bugs? Only thorough examination of source code could reveal hidden flaws.

If there was 10 implementations of the spec, then the bug would be limited to 10% of the users.  They would find that their block rate drops to 10% and

Miners would be well advised to have all blocks verified by multiple implementations, so they can react to forking.

Users of bugged, i.e. non-compliant, nodes would have to upgrade or end up on a fork.

[BitcoinAshley]

I, for one, would be happy to contribute to a fund to hire professional engineers to produce a spec.

Same here - question is, if someone started such a fund, would BF and the volunteer developers even pay attention to it? That is to say, if we crowd-sourced money to get a few full-time engineers to run all the versions on a test network and develop a protocol spec - in the end, would anyone be remotely interested in it? Esp. since we have a main dev. who is opposed to having a protocol specification (or at least so I've read.) And other devs just don't seem too keen on the whole idea.

Note that I have nothing against the devs, not here to spank them or whatever; just being realistic about their opinions regarding developing specs. I know they work hard. I just want more direction, focus, a cleaner approach to development that encourages alternative clients operating via a well-defined protocol.

On the other hand, maybe BF would eventually hire more full-timers to do that - work on cleaning code, writing protocol, testing, etc. The boring stuff that no one wants to do but is essential to not doing this whole ad-hoc thing with spaghetti code filled with magic numbers. Or maybe someone will start their own foundation that will perform this function and work with BF and the current developers.

Oh yeah, and magic numbers, another pet peeve of mine. Someone posted an IRC paste of Gavin just yanking a projected yearly % increase in broadband out of his butt (based on past numbers, so of course will be accurate in the future) and suggesting basing one of our "magic numbers" in the code on that. Not picking on Gavin, but there's a bigger problem that is illustrated by that kind of thinking.

This thread has some great thoughts and ideas; we need to figure out what to do to make some of these things happen. I could donate more money to BF but for all I know, they aren't even remotely interested in developing a spec or de-tangling the code so who knows, maybe I'd just be contributing to the problem.

I don't like the sentiment that if you don't know how to code, don't complain - there are dozens of other ways one can take action. Organize a bounty/fund, donate to a fund, raise awareness via discussion, donate to BF, install & test patches and alt. client for people who do code, etc.

It would be great if I didn't have to run bitcoind behind armory... their reason for relying on bitcoind basically boils down to "It's spaghetti with no defined protocol spec, what the heck do you expect us to do?" although they explain it nicer than that.

[alexkravets]

"Standardizing Bitcoin" is the top item in BF's charter https://bitcoinfoundation.org/about/.

If that's not funding and hosting a protocol spec the same way that W3C consortium does, I don't know what that means.

So far, they are failing to make any movement in that direction.

[TierNolan]

"Standardizing Bitcoin" is the top item in BF's charter https://bitcoinfoundation.org/about/.

If that's not funding and hosting a protocol spec the same way that W3C consortium does, I don't know what that means.

Their comment is:

As a non-political online money, Bitcoin is backed exclusively by code. This means that—ultimately—it is only as good as its software design. By funding the Bitcoin infrastructure, including a core development team, we can make Bitcoin more respected, trusted and useful to people worldwide.

That suggests that "standardizing" just means improving the quality of the reference client?

[alexkravets]

That suggests that "standardizing" just means improving the quality of the reference client?

Simplest explanation is that BF basically makes it up as they go along, there is no cohesive plan.

So far the dev. team, despite all its voluntary (except Gavin) and heroic efforts, only pays lip service to the idea of alternative implementations.  If they really wanted to see those, they would actually extract the spec from the code and target their own implementation to that spec, for say, 1.0 version, then it would be feasible to create alternative implementations in many diff. languages could that use the spec. as standard to interoperate.

Dev team's positions (usually expressed by sipa) is that "consensus of the running network is more important than any notion of correctness or spec compliance" (paraphrasing). In other words, Bitcoin is special because it's not a client-server style protocol but rather a live network that must maintain consensus at all costs, however, consensus could be defined as agreement on past view of history (the blockchain) or as identical behavior with respect to new messages occurring in the network, which "consensus" is mean or desired has been left unspecified so far...

It's a sad situation, because a few people have tried and given up to extract and re-implement the spec, but it's always a moving target (new commits change protocol message semantics) and also subject to as-yet not well understood bugs and edge cases in the existing Satoshi client. 

Cheers ...

[alexkravets]

Test, code, test, code, test....

Test for what?
Bugs in the code? But didn't they say that the code is the spec?
So if the code is the spec, then who decides when the spec is wrong?

Decentralised my ass. I can't wait until someone beats them at their own game and dethrones those clowns.

The writing is already on the wall: Ripple

[BitcoinAshley]

The writing is already on the wall: Ripple

I would argue that Ripple in its current (and planned) implementation is more centralized than bitcoin, however, the concept behind ripple, if it were made truly P2P and decentralized, could be a superior alternative provided it solved these problems with the Satoshi client (impossible to make functioning alternative implementations because the code IS the spec and the spec is constantly changing, make it up as we go along hey)
Although, really, if any alt-coin decided to go the spec route and develop a spec that any client could adhere to, it'd have some potential.

[justusranvier]

I don't see how Ripple adds any value to Bitcoin at all, unless certain people are successful at blocking progress with regards to scalability. I do see how Ripple can make it easier to transfer national currencies.

[jgarzik]

So far the dev. team, despite all its voluntary (except Gavin) and heroic efforts, only pays lip service to the idea of alternative implementations.

Wrong.  I have written two alternate implementations:

Counter example #1: https://github.com/jgarzik/picocoin/

Counter example #2: https://github.com/jgarzik/python-bitcoinlib/ and https://github.com/jgarzik/pynode/

[markm]

Thats great, hopefully the documentation devs will compare those to the Satoshi node and document what it is that they all do in common.

-MarkM-

[alexkravets]

So far the dev. team, despite all its voluntary (except Gavin) and heroic efforts, only pays lip service to the idea of alternative implementations.

Wrong.  I have written two alternate implementations:

Counter example #1: https://github.com/jgarzik/picocoin/
Counter example #2: https://github.com/jgarzik/python-bitcoinlib/ and https://github.com/jgarzik/pynode/

These are just examples of alternative implementations created by an insider already intimately familiar with both C++ and the existing C++ code base.

The whole point of a good specification is to allow outsiders (whole other teams really) who program in, say, Scala, Erlang or Clojure, to create interoperable implementations based solely on contents of the human-readable spec. (plus test data could be shared in a separate test kit) without having to become both a C++ expert (and go through all the overhead of setting up a functional Bitcoin development environment) and without having to have extensive experience with the existing C++ code base to understand protocol semantics buried therein and without having to follow all the ongoing commits to the existing C++ codebase to make sure the semantics haven't been changed (i.e. recent Dead Puppies change).

Once there is a spec, then a spec evolution process (i.e. BIPs) would amend the spec. and all the implementations would target a particular future version of it.

[TierNolan]

Once there is a spec, then a spec evolution process (i.e. BIPs) would amend the spec. and all the implementations would target a particular future version of it.

Speaking of BIPs, they need to mark BIPs that have been included in the reference client.  Bloom filtering is in the client, but still has a status of accepted.

[old c coder]

I, for one, would be happy to contribute to a fund to hire professional engineers to produce a spec.

Same here - question is, if someone started such a fund, would BF and the volunteer developers even pay attention to it? That is to say, if we crowd-sourced money to get a few full-time engineers to run all the versions on a test network and develop a protocol spec - in the end, would anyone be remotely interested in it? Esp. since we have a main dev. who is opposed to having a protocol specification (or at least so I've read.) And other devs just don't seem too keen on the whole idea.

Note that I have nothing against the devs, not here to spank them or whatever; just being realistic about their opinions regarding developing specs. I know they work hard. I just want more direction, focus, a cleaner approach to development that encourages alternative clients operating via a well-defined protocol.

On the other hand, maybe BF would eventually hire more full-timers to do that - work on cleaning code, writing protocol, testing, etc. The boring stuff that no one wants to do but is essential to not doing this whole ad-hoc thing with spaghetti code filled with magic numbers. Or maybe someone will start their own foundation that will perform this function and work with BF and the current developers.

Oh yeah, and magic numbers, another pet peeve of mine. Someone posted an IRC paste of Gavin just yanking a projected yearly % increase in broadband out of his butt (based on past numbers, so of course will be accurate in the future) and suggesting basing one of our "magic numbers" in the code on that. Not picking on Gavin, but there's a bigger problem that is illustrated by that kind of thinking.

This thread has some great thoughts and ideas; we need to figure out what to do to make some of these things happen. I could donate more money to BF but for all I know, they aren't even remotely interested in developing a spec or de-tangling the code so who knows, maybe I'd just be contributing to the problem.

I don't like the sentiment that if you don't know how to code, don't complain - there are dozens of other ways one can take action. Organize a bounty/fund, donate to a fund, raise awareness via discussion, donate to BF, install & test patches and alt. client for people who do code, etc.

It would be great if I didn't have to run bitcoind behind armory... their reason for relying on bitcoind basically boils down to "It's spaghetti with no defined protocol spec, what the heck do you expect us to do?" although they explain it nicer than that.

On spaghetti code, I would venture some advice that has worked for me over the decades. Upon "inheriting" some "old code", that "just has to work", I work on a "parallel" version, keeping the original for "current production" and the new for "eventual replacement".

The "parallel" version is not a "refactoring" or rearranging of the code in any fashion, at least not initially. I worked in C and before that assembler, but I see a lot of C in the bitcoind code!

First one must have all the code "in view" and as much in one's head as possible. And using the best IDE and tools one has, begin to first name things properly. This means everything, from class definitions to instance names, methods, member variables. And very important, no magic numbers, like method_name( argument_variable, 0 ). What is the 0? What does it represent?

For example, I am looking at db.h, ~line 117 in the CDB.Read template area. And there is a magic  ssKey.reserve(1000);
OK what does the 1000 represent? Is it important? Does it matter? Shouldn't it be a
const intergal_type Meaningfully_named_Value = 1000;

I know how it is when one is "in the zone", and coding. Documentation is superfluous and slows one down! And in the 1970s, especially, less so in the 1980s, and basically no more in the 1990s and on, variable names were no longer restricted to 8 or 16 characters, or 32 characters (of significance), etc. And [floppy] disc storage didn't force a lot of "vertical" compression (read K&R style here). Nor horizontal compression (read no spaces between operators and operands here). I see a lot of that "residue" in the code.

Just renaming things doesn't change the executable, AFAIK, in a release build. So I would globally, with a careful global search and replace, acknowledging each replacement, names that are obviously too terse, obscure or no longer even representative of what they are, with more meaningful names.

I have done this with some God awful C chess code, to good effect some time ago. One finds dead code, duplicate globals, etc. etc.

It is tedious but neccessary, and the result will be the same .exe file but the code will start to look readable.

I have had many an argument with "upper management" about spending the time to make a code base readable and where the names truly reflected what they are. Otherwise I am constantly translating with a symbol table in my head, what everything means. I hate that, and I hate having to remember things that I know I shouldn't have to!  Example, what is the instance bitdb, see line 29 in db.cpp rally a class of? Then what is the Class CDBEnv{} really? Is it a BerkeleyDB or a LevelDB, or a ? Of what? Wallet, block chain info? I keep digging, hoping to find a comment or some "action" that might nail it down to something definite. Then I can rename it from bitdb to something more meaningful.

I propose that style of renaming, rearranging, etc. that leaves the executable, byte for byte identical. Maybe then we can see the beauty of the forest thorough the ugly arrangement of the trees. I think the .h files especially need a lot of "work". I see hodge podge lines of extern this or that sprinkled in .h and .cpp files!

I would propose this sort of "one to one transformation" of the code to start with, and then it may become easy to: find the bugs that are there, add new "stuff", enhance old "stuff", etc. etc.

BTW, I am doing it myself, off to the side anyway, since I hate K&R style, and all the stuff mentioned above, and more (LOL)

Ron

[notme]

I, for one, would be happy to contribute to a fund to hire professional engineers to produce a spec.

Same here - question is, if someone started such a fund, would BF and the volunteer developers even pay attention to it? That is to say, if we crowd-sourced money to get a few full-time engineers to run all the versions on a test network and develop a protocol spec - in the end, would anyone be remotely interested in it? Esp. since we have a main dev. who is opposed to having a protocol specification (or at least so I've read.) And other devs just don't seem too keen on the whole idea.

Note that I have nothing against the devs, not here to spank them or whatever; just being realistic about their opinions regarding developing specs. I know they work hard. I just want more direction, focus, a cleaner approach to development that encourages alternative clients operating via a well-defined protocol.

On the other hand, maybe BF would eventually hire more full-timers to do that - work on cleaning code, writing protocol, testing, etc. The boring stuff that no one wants to do but is essential to not doing this whole ad-hoc thing with spaghetti code filled with magic numbers. Or maybe someone will start their own foundation that will perform this function and work with BF and the current developers.

Oh yeah, and magic numbers, another pet peeve of mine. Someone posted an IRC paste of Gavin just yanking a projected yearly % increase in broadband out of his butt (based on past numbers, so of course will be accurate in the future) and suggesting basing one of our "magic numbers" in the code on that. Not picking on Gavin, but there's a bigger problem that is illustrated by that kind of thinking.

This thread has some great thoughts and ideas; we need to figure out what to do to make some of these things happen. I could donate more money to BF but for all I know, they aren't even remotely interested in developing a spec or de-tangling the code so who knows, maybe I'd just be contributing to the problem.

I don't like the sentiment that if you don't know how to code, don't complain - there are dozens of other ways one can take action. Organize a bounty/fund, donate to a fund, raise awareness via discussion, donate to BF, install & test patches and alt. client for people who do code, etc.

It would be great if I didn't have to run bitcoind behind armory... their reason for relying on bitcoind basically boils down to "It's spaghetti with no defined protocol spec, what the heck do you expect us to do?" although they explain it nicer than that.

On spaghetti code, I would venture some advice that has worked for me over the decades. Upon "inheriting" some "old code", that "just has to work", I work on a "parallel" version, keeping the original for "current production" and the new for "eventual replacement".

The "parallel" version is not a "refactoring" or rearranging of the code in any fashion, at least not initially. I worked in C and before that assembler, but I see a lot of C in the bitcoind code!

First one must have all the code "in view" and as much in one's head as possible. And using the best IDE and tools one has, begin to first name things properly. This means everything, from class definitions to instance names, methods, member variables. And very important, no magic numbers, like method_name( argument_variable, 0 ). What is the 0? What does it represent?

For example, I am looking at db.h, ~line 117 in the CDB.Read template area. And there is a magic  ssKey.reserve(1000);
OK what does the 1000 represent? Is it important? Does it matter? Shouldn't it be a
const intergal_type Meaningfully_named_Value = 1000;

I know how it is when one is "in the zone", and coding. Documentation is superfluous and slows one down! And in the 1970s, especially, less so in the 1980s, and basically no more in the 1990s and on, variable names were no longer restricted to 8 or 16 characters, or 32 characters (of significance), etc. And [floppy] disc storage didn't force a lot of "vertical" compression (read K&R style here). Nor horizontal compression (read no spaces between operators and operands here). I see a lot of that "residue" in the code.

Just renaming things doesn't change the executable, AFAIK, in a release build. So I would globally, with a careful global search and replace, acknowledging each replacement, names that are obviously too terse, obscure or no longer even representative of what they are, with more meaningful names.

I have done this with some God awful C chess code, to good effect some time ago. One finds dead code, duplicate globals, etc. etc.

It is tedious but neccessary, and the result will be the same .exe file but the code will start to look readable.

I have had many an argument with "upper management" about spending the time to make a code base readable and where the names truly reflected what they are. Otherwise I am constantly translating with a symbol table in my head, what everything means. I hate that, and I hate having to remember things that I know I shouldn't have to!  Example, what is the instance bitdb, see line 29 in db.cpp rally a class of? Then what is the Class CDBEnv{} really? Is it a BerkeleyDB or a LevelDB, or a ? Of what? Wallet, block chain info? I keep digging, hoping to find a comment or some "action" that might nail it down to something definite. Then I can rename it from bitdb to something more meaningful.

I propose that style of renaming, rearranging, etc. that leaves the executable, byte for byte identical. Maybe then we can see the beauty of the forest thorough the ugly arrangement of the trees. I think the .h files especially need a lot of "work". I see hodge podge lines of extern this or that sprinkled in .h and .cpp files!

I would propose this sort of "one to one transformation" of the code to start with, and then it may become easy to: find the bugs that are there, add new "stuff", enhance old "stuff", etc. etc.

BTW, I am doing it myself, off to the side anyway, since I hate K&R style, and all the stuff mentioned above, and more (LOL)

Ron

Please do, then submit a pull request to this github project: https://github.com/bitcoin/bitcoin

[hashman]

Test, code, test, code, test....

Test for what?
Bugs in the code? But didn't they say that the code is the spec?
So if the code is the spec, then who decides when the spec is wrong?

Decentralised my ass. I can't wait until someone beats them at their own game and dethrones those clowns.

The writing is already on the wall: Ripple

Really, it says Ripple on the wall.  I promise to lift this cover off the wall to show you, sometime soon. 

[TierNolan]

Just renaming things doesn't change the executable, AFAIK, in a release build. So I would globally, with a careful global search and replace, acknowledging each replacement, names that are obviously too terse, obscure or no longer even representative of what they are, with more meaningful names.

Is there software tools for verifying equivalence ?

I propose that style of renaming, rearranging, etc. that leaves the executable, byte for byte identical.

That would do it.  However, presumably, object code would have variable names.  Is the linker deterministic, I wonder.

Hash maps could end up in a different state if different keys are used (assuming the compiler sometimes iterates).

[notme]

Just renaming things doesn't change the executable, AFAIK, in a release build. So I would globally, with a careful global search and replace, acknowledging each replacement, names that are obviously too terse, obscure or no longer even representative of what they are, with more meaningful names.

Is there software tools for verifying equivalence ?

I propose that style of renaming, rearranging, etc. that leaves the executable, byte for byte identical.

That would do it.  However, presumably, object code would have variable names.  Is the linker deterministic, I wonder.

Hash maps could end up in a different state if different keys are used (assuming the compiler sometimes iterates).

You can easily strip debugging symbols and just leave offsets/pointers.  The linker should be deterministic as long as you don't update any libraries in between.  It really depends on your compiler and language.  I know there are C compilers that will produce identical binaries regardless of var name, but I'm less sure about C++.

[Karmicads]

Consensus in Bitcoin. Who is it for? Who is it by? :  

A number of types of steakholder Come up when I think about Consensus, in the Context Of Bitcoin.  

Core devs of Bitcoind/QT
Devs of other implementations
Miners
Merchants
Users  

It would be helpfull to have clarification on this question of What is really meant by the devs themselves when they talk about consensus. What is the intended scope and how is it decided. I have a fair idea how it goes already, but it's not an explicit protocol itself.

This sugestion of the OP, kinda gets down to how important is it to the devs/BF, to foster consensus as broadly as possible. Should it be consensus of only those who do core code, or also those who can read code but only use & test the software?

A concise and clear protocol definition should help not only potential alternative implementation devs, but also people who are not actualy coders (or maybe neophytes) to understand the technicalities and also in dealing with more admin/systems-analysis type matters. That in turn, is critical to feilding informed consensus from the non-developer community.

That said, is there any reason why those of us with more peripheral skills shouldn't take the inititive and help ourselves to the problem/task? I think it would be a great spring-board into more serious coding for anybody with intermedite or less skills, who may be a bit apprehnsive of coding in the client itself. Also the work on profiling and style, sounds like a great student task.

Maybe we need more of an overt system for dev mentoring of newbie programers of whom they may co-opt into the less system critical and more mundane house keeping chores. Of course, the proceeds of a bounty or funding drive couldnt hurt either. I would happily throw a couple of btc into the hat to see a dev mentor system founded.

About the existing specs that have allready been mentioned here; what was the problem with them again?  

I might take a look at them and the source, just to see if a dunce amongst the geeks can gleen some differentiation between the interface code, the main body of the protocol and the rest.

It may help if the core devs could develop a habit of some infomal comment markup, for less invasive follow up work. If they are going over a part that could easily be abstracted (without busting it), they might put in some thing like:  

 // _dev_abst_&test @ 18333 .... lines=45

That could denote that the following 45 lines have been earmarked for abstraction chores and the resulting build should be tested on port 18333 or whatever the testnet port is. Then the student devs could use the  comment notations to pick out bits of work that are usefull and educational, but otherwise boring and mundane to the main devs. If these changes were held off the table and not released in the next one or two releases, then testing could ensure the changes were safe to merge into the main codebase and release into the wild.

I have the utmost respect for the devs however much they achive. They could easily be coding for their own bitcoin startups, mining, or just go fishing. But if the work load is accumulating, then the recruitment of new devs is a good option. When in doubt - mumble. When in trouble - delegate. 

The poverty of the foundation/devs/community shouldn't even need to be raised, in a movement that is manifesting the future of money; one I might add, that's made some very wealthy people. Patrons shouldn't be hard at all to come by.