In re Bitcoin Devs are idiots

[alexkravets]

The spec should simply describe exact format and formal-enough semantics of all valid network messages.

That's it.  

Need examples of such specs ?

1. TCP/IP
2. HTTP/1.1 (with extensions)
3. TLS

etc.

[1714046922]

1714046922

[1714046922]

1714046922

[1714046922]

1714046922

[1714046922]

1714046922

[shamaniotastook]

This matter was apparently for the first time discussed here, which is in itself ridiculously late, but the recent events illustrate the need of having the various issues much more clearly delineated.

Recently Bitcoin came close to unmitigated disaster, in the following way: Gavin diplomatically suggested that miners increase their block size, from the previous magic number of "250k" to something they themselves pick. This approach is flawed: the solution to the problem of having a magic number in the code is not passing the responsibility of choosing it to a larger group. It may work politically, in the sense that where large, vague groups are responsible for a bad move nobody will ever be hung. It does not work practically.

This point does not begin to get sufficient emphasis: stop thinking politically, stick to thinking practically. The political importance, usefulness or competence of a dev is nil. This is not your job, and more importantly this is one of the things you suck at the most. A casual skim through the -dev sessions is ample proof for this, more ridiculous dickwad posturing and knowshitism has never before been seen (outside of the mailing lists of some meanwhile failed open source projects). Snap out of it. Stick to writing code.

But we digress: as a result of a number of miners implementing their own version of a magic miner, a number of large blocks were created and mined by them, as long as they ran 0.8. Miners running 0.7 failed to mine these same blocks, and a fork developed.

The reason is that Bitcoin code sucks. It's not that "the blocksize", it's not that "the database", it's not that "nobody could have foreseen their using a plane like a rocket". That shit does not belong in this discussion, passing the buck is not and cannot be accepted in Bitcoin. The reason is that Bitcoin code sucks, and Bitcoin code sucks because people want to be Bitcoin devs, people want to call each other Bitcoin devs, people want to participate in idle irc chatter as if they in fact were Bitcoin devs, but those same people do not have either the ability or intellectual resources to write dependable, usable, good, clean code.

This is a problem, and this problem needs to be resolved, preferably by the people who are causing it. You know yourselves, I won't name and shame. Fix your heads. You won't be getting much more warning.

Today will go down in history as the day when Bitcoin nearly died, and its fate depended on BTC-Guild staying online. Stop and think for a minute. What are you doing here? Why are you here, really?

With all due respect, MPOE, i do realize i am a newbie here, but my conscience forces me to make two observations of fact, and one suggestion.

Fact 1: The developers are NOT idiots. If they were, you and 80,000 other's would not be so engrossed in the going's on here.

Fact 2: You do have some valid points, but they are overshadowed by your lack of constructive criticism and insistence on personal character attacks on the developers intelligence. They are human! What they have accomplished should be praised.

One suggestion: from experience, i have learned that if you're smart enough to identify a problem, then your'e smart enough to identify at least ONE possible solution--not necessarily, THE solution, but A solution. And depending upon your experience, understanding of the problem domain and ability to organize effectively, you actually have the responsibility to move the initiative forward. leadership is not born into someone, or granted by top-down organizations--where everyone fits nicely in a box on some org-chart hanging on the wall--but simply belongs to she/he who is running the fastest. so my suggestion would be to let your frustration and passion turn into something positive. put together some testing framework suggestions, work with developers to setup qa and staging environments, write detailed explanations of required unit tests and work with developers to push those changes..whatever it takes to realize your vision! I'ld wager that if you took up such an initiative with the same spirit you carry on this petty attack, you would be a significant part of what is sure to be the coming bitcoin revolution we all know is inevitable at some point. united, that time is now. divided, forget it!

Shaman

[Atruk]

BDB shouldn't even be part of the spec, the spec should abstract the actual database implementation details and just maybe talk about whether it should have atomic transactions, whether we should be able to roll those back, under what circumstances to roll than back and so on.

Enshrining some bug in some particular implementation of "database back end" should be anathema, our spec should basically be telling us "BDB isn't up to spec, don't use versions of it that aren't able to perform as we need them to".

This. So much this. Dublin Core and RDF/RDA container formats are excellent examples of standards that in all of their lenitive documentation provide an example of moving the database details out of the software implementation.

[jgarzik]

It is always entertaining to watch non-contributors opine about completely obvious solutions that the devs are silly to have overlooked.

The interesting thing about bitcoin is its organic nature.  The bitcoin codebase, warts and all, was dumped into the Internet's collective lap.  Reality does not give anyone a chance to pause, wait for a specification to be polished, to wait for every single edge case to be tested (if that were even possible), etc.

Almost half a billion dollars in market cap, and the dev team is still largely unpaid volunteers, trailing behind events, cleaning up the messes reality leaves behind.

[jgarzik]

BDB shouldn't even be part of the spec, the spec should abstract the actual database implementation details and just maybe talk about whether it should have atomic transactions, whether we should be able to roll those back, under what circumstances to roll than back and so on.

Enshrining some bug in some particular implementation of "database back end" should be anathema, our spec should basically be telling us "BDB isn't up to spec, don't use versions of it that aren't able to perform as we need them to".

This. So much this. Dublin Core and RDF/RDA container formats are excellent examples of standards that in all of their lenitive documentation provide an example of moving the database details out of the software implementation.

This idea is from an alternate world where you don't have to worry about breaking existing users' software and data to the point that they cannot spend their own money.

Bug-for-bug compatibility is not a choice made joyfully and willingly.

Engineers always prefer a clean slate, a clean interface.  That works here, only with a little help from science fiction's time machines.

[MPOE-PR]

Another way to state the real problem: There is no Bitcoin Protocol Spec, most semantics buried in the hairball of the C++ reference implementation

That would be correct. It's a huge issue.

It is also a huge issue that those suggesting we need the protocol spec are brushed off with the "you don't get it" nonsense by elitist devs.
Bits and pieces, some expired, are scattered around Wiki. Is that the best we can do?
@MP: would you be willing to lead the effort of formulating the current protocol spec?

Sez MP:

I am willing to help, both by providing funding and by consulting pro bono on things I might have a clue about. These often enough are not coding, as I'm not a programmer. Nevertheless I would gladly strive to help avoid the sorts of strategic mistakes we've seen at work so far here.

I don't think this would necessarily constitute a lead role, and moreover if we focus on getting things right rather than propping egos and stroking issues leading might not even be so desperately needed. Let sense lead, not common but actual sense.

Maybe it's time to do a headcount, start a project... specified Bitcoin is certainly better than code-centralized Bitcoin.

[shamaniotastook]

BDB shouldn't even be part of the spec, the spec should abstract the actual database implementation details and just maybe talk about whether it should have atomic transactions, whether we should be able to roll those back, under what circumstances to roll than back and so on.

Enshrining some bug in some particular implementation of "database back end" should be anathema, our spec should basically be telling us "BDB isn't up to spec, don't use versions of it that aren't able to perform as we need them to".

This. So much this. Dublin Core and RDF/RDA container formats are excellent examples of standards that in all of their lenitive documentation provide an example of moving the database details out of the software implementation.

This idea is from an alternate world where you don't have to worry about breaking existing users' software and data to the point that they cannot spend their own money.

Bug-for-bug compatibility is not a choice made joyfully and willingly.

Engineers always prefer a clean slate, a clean interface.  That works here, only with a little help from science fiction's time machines.

jgarzik, i completely agree...i would be more inclined to take their criticisms seriously if they were actual contributors. but then again, i am not a contributor either (yet). Nevertheless, the core developers at least deserve the respect they've worked so hard to earn! None of this would even matter without them. i guess now is the time for the 'rats' to abandon ship, while the 'crew' works to right the ship and keep her afloat!

[MPOE-PR]

This idea is from an alternate world where you don't have to worry about breaking existing users' software and data to the point that they cannot spend their own money.

Bug-for-bug compatibility is not a choice made joyfully and willingly.

Engineers always prefer a clean slate, a clean interface.  That works here, only with a little help from science fiction's time machines.

It's ok Jeff, if this new thing will have to be the Real Bitcoin then this new thing WILL be the Real Bitcoin.

Point blank: money will never tolerate the sort of crap that has been pouring out of devtroop for the past while. I don't care you(*)'ve been dorking around with it for years and nobody told you it's wrong, that doesn't make it right, that just makes the little coven irrelevant enough.

If current Bitcoin can't be made into good Bitcoin then sucks for current Bitcoin, it is no different from Novacoin or whatever flavor of the week failed altchain.

(*) Impersonal you. Very, very impersonal you. As in, I'm not talking about you (and I'm not talking about Gavin - in fact, you know who I'm talking about).

[markm]

It's ok [you], if this new thing will have to be the Real Bitcoin then this new thing WILL be the Real Bitcoin.

[...SNIP...]

(*) Impersonal you. Very, very impersonal you. As in, I'm not talking about you (and I'm not talking about Gavin - in fact, you know who I'm talking about).

Thank you. I do not think I am actually far afield from Gavin, as I am pretty sure I have read him and/or other core devs discussing blockchain validator tools each of which would validate a certain range (by "height") of blocks.

In other words, no time machine going back and changing the historical past, just no repeating of the errors of the past in the future.

Basically the spec that tells us certain configurations, versions or implementations of BDB are not up to spec would be the spec for the new thing, the Real Bitcoin. The fact that the Satoshi node wasn't up to that spec gets enshrined not in the historically non-normative block-creation spec/code going forward but, rather, in the normative "validate a certain period of blockchain history's blocks" blockchain-checkers that I read some core devs discuss once upon a time.

An RFC could comment on details where certain builds of the Satoshi client on certain platforms failed to meet the new thing spec, and block heights could be targetted for the block height at which such builds will be orphaned, unsupported, deprecated, unable to function as part of "the Real Bitcoin" aka the new thing from that point on.

This is all old news / old ideas to Gavin et al I am pretty darn sure, which is why I am not in a panic over block sizes nor was in a panic yesterday evening over certain builds of Berkely DB on certain platforms in certain builds of the Satoshi node failing to perform to [ex]spec[tation].

So nyah nyah nyah circle jerk time group hugs for the geeks and techies, sorry if the PR folk aren't into that.

-MarkM-

[🏰 TradeFortress 🏰]

SOMEHOW? Private keys? They're all on your client and are never transferred anywhere else.

Read up on http://en.wikipedia.org/wiki/Side_channel_attack ...

It's practically impossible for that to happen unless you have access to the hardware. In which case, your wallet is screwed.

Bitcoin does not transfer the private keys anywhere, you also do not know when someone is making a transaction, only when someone pushes a transaction.

[makomk]

It is also a huge issue that those suggesting we need the protocol spec are brushed off with the "you don't get it" nonsense by elitist devs.
Bits and pieces, some expired, are scattered around Wiki. Is that the best we can do?
@MP: would you be willing to lead the effort of formulating the current protocol spec?

You really, really don't get it. We can't have a protocol spec becasue the existing client has a whole bunch of unspecified and unintended behaviour that no-one knows about, and any divergence from that behaviour by any major implementation will result in fiascos like this one. This problem could just as easily have been caused by a independent third-party implementation of the client. The developers have been finding and documenting all the corner cases as thoroughly as possible, but some of them are - like this one - really subtle and hard to spot. I'm not sure anyone's managed to fully work out the circumstances under which 0.7 will fail to accept a block due to this bug.

[Severian]

The interesting thing about bitcoin is its organic nature.  The bitcoin codebase, warts and all, was dumped into the Internet's collective lap.  Reality does not give anyone a chance to pause, wait for a specification to be polished, to wait for every single edge case to be tested (if that were even possible), etc.

I'm not a coder but even I know this. It is what it is. Whoever wrote the code wrote it as proof of concept and then they went back and explained what they did. That's Bitcoin, like or not.

I have to say you all are very patient in the face of the spoiled brat brigade that are yelling for you all to modify their hot rod while the mechanics marvel at how its staying on the road in the first place.

[coqui33]

MPOE-PR: Please do think from what I posted above that I defend the decision to forcibly accept the 0.7 fork and abandon the 0.8 fork, in violation of the bitcoin protocol's founding principle that greater hashing power always wins a dispute. Nobody asked me, of course, since I am neither developer nor mining pool manager. But as I see it, the final decision had two strong drawbacks and one weak upside.

The first strong drawback is that the decision forced a patch on clean efficient code (0.8 ) to conform to buggy slow code (0.7). In my experience this is never a good idea over the long pull.

Second, the decision burned up developers' political influence. A handful of developers used their deservedly high community respect to force a decision on the 0.8 miners. So be it. But the next emergency will see developer influence greatly reduced in consequence.

The weak upside? The decision prevented thousands of 0.7 users suffering crashes as their wallets choked on the offending block. This would have forced users around the world to stop dithering and upgrade to 0.8 immediately. So? What would have been wrong with that?

[behindtext]

Another way to state the real problem: There is no Bitcoin Protocol Spec, most semantics buried in the hairball of the C++ reference implementation

This was actually a good example of something a spec wouldn't help (though would still be good to have). The inconsistent behavior here arose out of 0.8 not faithfully implementing some implicit behavior in BDB. The behavior in question is not anywhere in the Bitcoin code itself, nor is it visible at the interface of BDB and Bitcoin.  If there were a spec it would make no mention of it— and yet the network would be broken all the same.

The key thing is that in a distributed consensus system the most important definition of right is "consistent", this is paramount to all other concerns. A spec is helpful to the extent that it makes it easier to achieve a bit identical consistent behavior in the validation of blocks across all nodes, but because the spec itself can't be executed a spec can never guarantee consistency, at least not in the real world. (well, unless the spec is code— which is effectively what we have, for worse or better)

I respectfully disagree.  

A real protocol specification would enable a whole ecosystem of alternative and interoperable implementations to appear in Scala, Java, Go, C, etc.  

These implementations would NOT use the same flawed 3rd party libraries and there would be a real diversity in Bitcoin land, instead of the monoculture of C++ with all of its "magic" and "folklore".

Once no individual implementation had large enough fraction of the hash rate, then such forks would NOT occur, instead, bugs in a given implementation would only affect some miners, merchants & users but the chain would be just fine.

Let us hope and pray that eventually there will be some effort ( funded perhaps by the Bitcoin Foundation ) to extract and publish such a spec from the C++ hairball formal enough for viable alternative implementations to appear and to interoperate.


Cheers ...

suffice it to say there are hella layering violations in bitcoind and that is *bad* for bugs. can't really blame the devs tho, unless they helped write the original client from whence they inherited said issues.

[jgarzik]

Second, the decision burned up developers' political influence. A handful of developers used their deservedly high community respect to force a decision on the 0.8 miners. So be it. But the next emergency will see developer influence greatly reduced in consequence.

Read the chat logs before speaking about topics of which you know little.

The miners were not "forced" to do anything.  They could have chosen to stay on the 0.8 side of the fork.

Each miner (really, pool operator) votes with their collective hash power, deciding whether or not to support a mining-related decision like this.

[DeathAndTaxes]

in violation of the bitcoin protocol's founding principle that greater hashing power always wins a dispute.

It was never a founding principal for greater hashing power to resolve hard forks.  Hard forks by definition can't EVER be resolved by hashing power.  If a single node remains on an incompatible fork it exists as a parallel implementation of Bitcoin.  Creating a precedent for using hashing power to fork the network is horribly dangerous and will lead to intentionally putting bugs into the codebase to force a fork for profit.

The decision prevented thousands of 0.7 users suffering crashes as their wallets choked on the offending block. This would have forced users around the world to stop dithering and upgrade to 0.8 immediately. So? What would have been wrong with that?

You are completely uninformed.  v0.7 nodes didn't crash they simply rejected the block.  The risk is that v0.7 nodes would be vulnerable to attacks by double spends, 51% attacks and accepting newly generated coins from the incompatible v0.7 generation blocks.

The rational for rolling back to v0.7 was to prevent massive chaos, and loss of confidence and scammers and fraudsters looted the users of running older nodes.  Nobody is saying we will remain on BDB forever but the transistion can be better managed.

A v0.81 can be released which uses the new db format but prevents incompatible blocks.  older version users can be strongly encouraged to upgrade to the new version and warned of potential risks in staying with incompatible older version.  Where a vast super majority (not of miners but of ALL stakeholders, users, exchanges, merchants, service providers) are on the new platform a final critical warning can be released notifying users of older version they face significant risk of being forked away if they don't upgrade and then the incompatible block rules can be introduced.

TL/DR a planned transition instead of a "fork it and if people get fucked in the chaos, well too bad" attitude.  Which do you think will destroy trust in the Bitcoin network?

[bullioner]

Another way to state the real problem: There is no Bitcoin Protocol Spec, most semantics buried in the hairball of the C++ reference implementation

This was actually a good example of something a spec wouldn't help (though would still be good to have). The inconsistent behavior here arose out of 0.8 not faithfully implementing some implicit behavior in BDB. The behavior in question is not anywhere in the Bitcoin code itself, nor is it visible at the interface of BDB and Bitcoin.  If there were a spec it would make no mention of it— and yet the network would be broken all the same.

The key thing is that in a distributed consensus system the most important definition of right is "consistent", this is paramount to all other concerns. A spec is helpful to the extent that it makes it easier to achieve a bit identical consistent behavior in the validation of blocks across all nodes, but because the spec itself can't be executed a spec can never guarantee consistency, at least not in the real world. (well, unless the spec is code— which is effectively what we have, for worse or better)

I respectfully disagree.  

A real protocol specification would enable a whole ecosystem of alternative and interoperable implementations to appear in Scala, Java, Go, C, etc.  

These implementations would NOT use the same flawed 3rd party libraries and there would be a real diversity in Bitcoin land, instead of the monoculture of C++ with all of its "magic" and "folklore".

Once no individual implementation had large enough fraction of the hash rate, then such forks would NOT occur, instead, bugs in a given implementation would only affect some miners, merchants & users but the chain would be just fine.

Let us hope and pray that eventually there will be some effort ( funded perhaps by the Bitcoin Foundation ) to extract and publish such a spec from the C++ hairball formal enough for viable alternative implementations to appear and to interoperate.
Cheers ...

A agree with you, but based on statements by Jeff, Gavin, and Mike, this will not happen.  If we want a well specified crypto currency with a wealth of implementations, we are going to have to create it.  Bitcoin is not that, and it is not going to become that.

[bullioner]

v0.7 reliance on BDB caused it to be fundamentally broken/flawed. Its actions weren't consistent with either the documented protocol, the higher level source code, or anyone's understanding/expectation of what should have happened.  It was a landmine. 

Does this not argue for a diversity of implementations with different underlying 3rd party libraries ? Clearly it does. But this cannot happen until and unless there is actually a formal-enough spec to enable those to be written w/o ever "groking" all that C++ folklore, which is a moving target anyways ...

Not going to happen within Bitcoin.  If we want such a thing, we are going to have to create it, and it is not Bitcoin.

[bullioner]

I would like to also point out that the alexkravets idea re specification is sound. The only way forward for the dev team, provided they wish to preserve a shred of dignity, is a. immediately start work on specification, which is the only priority, and release no further versions until such specification is complete and b. immediately start work on removing all magic numbers and assorted code smelliness from their UNreference implementation of crap, and not release any further version before they've released a clean one.

In that order.

Whoa... All of this... Agreement...

Nope, not going to happen in Bitcoin.  And that will be its downfall.  The well specified crypto currency will have long term advantages that could well ensure it wins in the market, despite Bitcoin's first mover advantage.

[bullioner]

It is always entertaining to watch non-contributors opine about completely obvious solutions that the devs are silly to have overlooked.

The interesting thing about bitcoin is its organic nature.  The bitcoin codebase, warts and all, was dumped into the Internet's collective lap.  Reality does not give anyone a chance to pause, wait for a specification to be polished, to wait for every single edge case to be tested (if that were even possible), etc.

Almost half a billion dollars in market cap, and the dev team is still largely unpaid volunteers, trailing behind events, cleaning up the messes reality leaves behind.

There are many examples of protocols which only became specified after an implementation as in widespread use.  Look at HTTP, look at ssh, look at OpenPGP.  These all have organic ecosystems of implementations, and were all specified after an implementation went into widespread use.  It is simply part of the maturation process.  Crypto currency is no different.  Lacking a spec actually prevents the "organic" nature you speak of, the same way that any other single-implementation unspecified "protocol" does.