The bug fixed in Bitcoin Core 0.16.3 was really bad. IMO it was the worst bug since 2010. If it had been exploited in a 0-day fashion, significant & widespread losses (due to acceptance of counterfeit BTC) would've been likely, and Bitcoin's reputation would've long been tarnished. Furthermore, since a ton of altcoins are based on Bitcoin Core, this would've affected a huge swath of the crypto space all at once.

Everyone's human, and secure software engineering is largely an unsolved problem. The Bitcoin Core devs have done a remarkably good job over the years; in fact, in this case they were able to recognize that a bug report for a DoS attack was actually a critical consensus bug, and then they managed to roll out a fix in a way which ending up protecting Bitcoin. I am thankful for their work and diligence. However, the fact that this bug was introduced and then allowed to exist from 0.14.0 to 0.16.2 was undeniably a major failure, and if all of Bitcoin Core's policies/practices are kept the same, then it's inevitable that a similar failure will eventually happen again, and we might not be so lucky with how it turns out that time.

Finger-pointing would not be constructive, but neither would it be sufficient to say "we just need more eyes on the code" and move on. This bug was very subtle, and I doubt that anyone would've ever found it by actually looking at the code. Indeed, the person who found it did so when they were doing something else and ended up tripping the assertion. Furthermore, this bug probably wouldn't have been found through standard unit testing, since this was a higher-level logic error. (By my count, something like 18% of the entire Bitcoin Core repository is tests, but that still didn't catch it.)

Perhaps all large Bitcoin companies should be expected by the community to assign skilled testing specialists to Core. This vulnerability could've been detected through more sophisticated testing methods, and currently a lot of companies don't contribute anything to Core development.

Perhaps the Core release schedule is too fast. Even though it sometimes already feels painfully slow, there's no particular "need to ship", so it could be slowed down arbitrarily in order to quadruple the amount of testing code or whatever.

Perhaps there should be more support and acceptance for running older versions, or a LTS branch, or a software fork focused on stability. The official maintenance policy says that the current and previous major release is supported, but that doesn't seem to be closely followed. In this bug, backports were written for 0.14.x and 0.15.x, but as of this writing no binaries have been released for those, even days after 0.16.3's release. SegWit didn't have any backports when it was released, even though users would've needed to enforce SegWit in order to achieve full security if SegWit had activated as quickly as originally hoped. Sometimes fixes are backported to an old version's git branch but no release is actually made. If 0.13.x is not currently supported, then there was no supported version without the vulnerability. This might indicate that the maintenance period isn't long enough; there are a few hundred people still using 0.13.2, and they were the only Bitcoin users completely safe from this vulnerability.

I do not think that it would be constructive to turn to any of the full node total-reimplementations like btcd, which are very amateur in comparison to Bitcoin Core.

I don't know exactly how this can be prevented from happening again, but I do know that it would be a mistake for the community to brush off this bug just because it ended up being mostly harmless this time.

I know this is probably the last argument most people want to hear, but is this not a case where more independent implementations would result in less risk?  If you maintain that one particular client should form the "backbone of the network", you have to consider what happens if that backbone breaks.  If there were a wider variety of clients being run, there may have been less of a threat to the network overall?

Core have done exceptional work, but at the end of the day, they're still only human.  Assigning more people to keep an eye on one codebase might help mitigate faults, but if there's only one main codebase, there's still going to be an issue if an error slips through the net.  Hence my belief that more codebases would create a stronger safeguard.

I agree with DooMad completely. Diversity is the only real solution to network security.

We have to deal with the fact that bugs are found in absolutely ALL software.
Only thing we can hope is that it is: A) rare that bugs are found in all software at the same time and B) that no single entity will likely have all of these bugs at that time.

You can put million top tier devs on the code and they will still let a bug go through every few decades,
Security of the entire Internet isn't that there is low level code that is perfectly secure, but that there are many different systems and implementations running it. You need to have more implementations to have any chance of long term survival.

I am weary of this assertion. Many bitcoin related businesses use custom implementations of Bitcoin that are based on the underlying consensus rules. The same is true for the miners today (even if they broadcast they are using other implementations), so I doubt a single malicious actor could have gotten counterfeit BTC more than a small number of confirmations.

I would echo what DooMAD said in that the solution is to encourage more implementations, and for each implementation to not have a high percentage of overall nodes.

At the end of the day, the Bitcoin network is nothing more than a bunch of consensus rules that everyone follows.  

The solution to this is simple, and it is that the blockchain (whose tip contains the most cumulative work) that follows all of the consensus rules is the Bitcoin blockchain, and any fork of this is not (in many cases, it would be an altcoin intentionally created).

The incentive to attack an implementation that is used by 10%-20% of the Bitcoin network is much smaller than the incentive to attack an implementation that affects 90% of the network. The further would be a minor hiccup, while the later has the potential to actually steal large amounts of money, and cause serious disruptions.

The complexity comes in ensuring that the network is no longer partitioned and that everyone has received the blockchain with the most cumulative work.

I disagree.

Suppose there is an exchange that happens to be connected to some nodes that are vulnerable to some kind of attack. Or perhaps they aren't connected directly to those nodes, but connected to node which are connected to those nodes. Suppose this attack causes those nodes to go offline or otherwise become disconnected from the network. Even if there are a very small number of these vulnerable nodes, if they happen to form a ring around the exchange, an attacker can attack those nodes and cause the network to partition. It would break into at least two pieces: the chunk containing the exchange, and the rest of the network. The attacker, if he has some hashrate, can now be mining a fork of the blockchain specifically created so that he can attack this exchange. Since this is a fork for a part of the network that is no longer receiving the rest of the blockchain, this attacker has 100% of the hashrate for that fork and can do everything with it that anyone with >51% of the hashrate can do. This kind of attack does not need a large number of nodes to be vulnerable, it just needs enough so that an attacker can partition the network.

But you just described Sybil Attack which can happen with or without multiple implementations of bitcoin. It doesn't even need a vulnerability or hashrate to happen.

I do say we need more implementations of bitcoin from scratch. It won't be a perfect solution and it will take time for the software to reach maturity of bitcoin core but the benefits of it are more.

It doesn't have to be the same person.
Like in this case, if such an event happened that there was a block that was trying to spend the same input twice, my node would just not let it through and got stuck on one block, which would make me to analyze why, which would maybe trigger an alarm.
It's obviously better if more people run the node like mine. Because sometimes I'm busy

Yeah, what are the chances of this never happening again?
Very small, we need to accept it. Better a fork then a complete failure.

The way I see it we need to start doing 2 things as a community:
1) Operators that are running full nodes that want to contribute more to the network should start running more nodes with different implementations.
2) We should connect our nodes to nodes running different implementations and have warnings pop up in case of a chain-split.
3) We need more good quality implementations.

Even this will not protect us 100% from a potential network-wide vulnerability, but it is a hell of a lot better.

I think that I am misunderstanding what exactly this means.


Are they talking about the double spend input, that uses a previously created UTXO? Or are the talking about the newly created UTXO that had two double spend inputs and now has a block built on top of it?

Edit: I understand this better now. This answer on slack helped clear things up for me.  https://bitcoin.stackexchange.com/questions/79481/how-does-the-most-recently-found-critical-vulnerability-cve-2018-17144-work

It is very encouraging to see a prominent figure like OP starts a topic about this event admitting that there are lessons to be learned and measures to be taken. kudos theymos.  

I understand there are many people with various incentives mostly not in the best interests of bitcoin who may find such an event a good opportunity to take advantage in an irresponsible and unproductive way. Although I have a reputation on this forum to be somewhat discontented with Core devs, I hope my statements here other than being helpful in the context of this topic, would show how committed I'm to bitcoin as a liberating movement instead of blindly opposing and fighting with specific parties.

Seemingly, the "multiple implementations proposal" is trending right now in this thread. I don't know to what extent it could be considered a healthy proposal with good intentions but I'm sure it is neither correct nor practical to be considered an effective measure against vulnerabilities to implementation bugs. On the contrary it seems to be more dangerous rather than a helpful strategy.

Being decentralized does not change the fact that bitcoin is an integrated meta-system. Running multiple implementations of code in such a system is not recommended because it adds another complexity factor: Multiple implementations means more lines of code and hence more bugs. It will cause frequent chain splits and encourages a new range of sybil attacks.

Suggesting that no implementation should be used by a majority, besides its impracticality (who is in charge of imposing this rule?) does not seem to be of much help with this idea:
Firstly, it implies a new kind of consensus system, with no theoretical support. I'm not aware of any serious work covering a decentralized consensus based system that uses heterogeneity of implementations as a main or auxiliary security measure e.g. for immunity against software bugs.

Secondly, it is very unlikely that such an evenly distributed heterogeneity might help in damage control by localizing bugs. Nodes have to validate blocks independently and they don't apply longest/heaviest chain rule unless they have validated both chains beforehand. It is impossible for a wallet to feel something is wrong with its own code and decide to follow the majority blindly. One possible solution for such a schema would be running multiple implementations by a single federated node that uses a BFT algorithm to make final decisions. It has been suggested up-thread somehow and is infeasible because of  exaggerated costs and complexities involved.

Thirdly, adversaries would find it feasible to identify flaws in a single implementation and targeting its users (which typically are a minority) without compromising the whole ecosystem explicitly. In this scenario, although bitcoin won't collapse dramatically but the scammers might consider it an advantage as the funds they steal, preserve their value.

Conclusively, I would say this proposal, encouraging development and hiring multiple implementations of bitcoin protocols, shows negative indications and should be dropped as a serious proposal.

Hence the issue remains open: What happened and what measures should be taken to mitigate such disruptive problems?

I have something to suggest in this regard but for now I think it is more convenient to close the case with the multiple implementations proposal before proceeding any more.