Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: PowerGlove on July 20, 2022, 09:06:49 AM



Title: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 20, 2022, 09:06:49 AM
@franky1 posted (https://bitcointalk.org/index.php?topic=5406677.msg60597353#msg60597353) something interesting (to me) regarding how alternative implementations of Bitcoin are regarded, here's the relevant snip:

for anyone that has tried to make a REFERENCE client of a full node thats not part of the 'core' dev branding, they get treated like outsiders, competition, threat. thus everyone ends up relying on core.. like zombies

I find that really unfortunate. I think alternative implementations should be encouraged.

I've been slowly working my way through old posts in "Development & Technical" (to understand the history of Bitcoin better). These snips by @satoshi surprised me a little:

I don't believe a second, compatible implementation of Bitcoin will ever be a good idea.  So much of the design depends on all nodes getting exactly identical results in lockstep that a second implementation would be a menace to the network.  The MIT license is compatible with all other licenses and commercial uses, so there is no need to rewrite it from a licensing standpoint.

A second version would be a massive development and maintenance hassle for me.  It's hard enough maintaining backward compatibility while upgrading the network without a second version locking things in.  If the second version screwed up, the user experience would reflect badly on both, although it would at least reinforce to users the importance of staying with the official version.  If someone was getting ready to fork a second version, I would have to air a lot of disclaimers about the risks of using a minority version.  This is a design where the majority version wins if there's any disagreement, and that can be pretty ugly for the minority version and I'd rather not go into it, and I don't have to as long as there's only one version.

I guess I can empathize with this stance a little, but I can't help but think this whole approach is kind of backwards. Bitcoin offers unprecedented freedoms (compared to fiat) to users of the network, and that should be loudly applauded. But why don't developers get to freely participate in this brave new world?

Now, I'm aware that nothing prevents anyone from attempting an alternative implementation and that someone could conceivably read the source code and proceed from there. But if the reference implementation is hostile to that idea and the core devs don't properly document what they're doing and work amicably with outsiders then any such attempt is surely doomed to fail?

Take PDF for example. I think it being an ISO standard is a great thing and I would be pretty bummed if Adobe had arranged things so that no one but them could provide a working implementation.

Shouldn't Bitcoin, first and foremost, be a documented standard?

Shouldn't the reference implementation be one of many?

Isn't that more in the spirit of decentralization?


Title: Re: Why aren't alternative implementations encouraged?
Post by: pooya87 on July 20, 2022, 09:27:02 AM
I am personally a supporter of alternative implementation of Bitcoin because of this reason:
If the second version screwed up, the user experience would reflect badly on both,
If the first version screwed up (eg. value overflow in 2010 and in 2018), the user experience would reflect badly on Bitcoin.

With multiple implementations of the protocol, the chain's survival is ensured because for example in 2010 when the chain started growing with invalid blocks the alternative implementation would have rejected those blocks and any miner running that full node or multiple nodes would have built the correct chain ensuring that bitcoin network doesn't halt and could even prevent fraud such as double spends when the invalid blocks were removed from the chain.

Now, I'm aware that nothing prevents anyone from attempting an alternative implementation and that someone could conceivably read the source code and proceed from there.
Alternative implementations do exist though. Like BTCD (https://github.com/btcsuite/btcd).

~ and the core devs don't properly document what they're doing and work amicably with outsiders then any such attempt is surely doomed to fail?
That's true about a lot of other open source projects too. I rarely see any that is well documented. Most of the code in bitcoin core is straight forward though and if you know c++ you'll have an easy time understanding it all, and if you know other languages you should still be able to figure most of it out.

Shouldn't Bitcoin, first and foremost, be a documented standard?
I agree with this but more like the documentation should be improved because it already exists on bitcoin.org, bitcoin wiki and among BIPs but some parts aren't good enough.

Shouldn't the reference implementation be one of many?
There should be only one reference implementation but other alternative options for people to run as full node.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 20, 2022, 09:54:12 AM
Alternative implementations do exist though. Like BTCD (https://github.com/btcsuite/btcd).

I'm aware, and the following question is not rhetorical: Do any other implementations see significant use by miners?

Shouldn't the reference implementation be one of many?
There should be only one reference implementation but other alternative options for people to run as full node.

Obviously, not the clearest sentence construction on my part, but that is what I meant.


Title: Re: Why aren't alternative implementations encouraged?
Post by: pooya87 on July 20, 2022, 10:07:36 AM
I'm aware, and the following question is not rhetorical: Do any other implementations see significant use by miners?
A good question but only mining pool operators can answer this.


Title: Re: Why aren't alternative implementations encouraged?
Post by: HeRetiK on July 20, 2022, 10:15:25 AM
It's a tricky matter as the way I see it there's good arguments both for and against alternative Bitcoin node implementations.

On a smaller scale we're seeing this play out on the Lightning Network: On the one hand it's great that there are multiple implementations out there as it helps with the proliferation of new ideas and people get to choose which LN node to run based on which featureset they prefer. On the other hand there have been compatibility issues between LN nodes implementations in the past [1], which in similar scenarios may be more problematic on the baselayer than on layer 2. What we also see on LN is that people unfortunately tend to flock towards single implementations [2] anyway. While not an argument against alternative implementations, in the case of the Bitcoin baselayer such flocking would make alternative implementations largely irrelevant.

[1] https://github.com/lightningnetwork/lnd/issues/6023
[2] https://www.bitrefill.com/lightning-toplist


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 20, 2022, 10:25:19 AM
I am personally a supporter of alternative implementation of Bitcoin because of this reason:
If the second version screwed up, the user experience would reflect badly on both,
If the first version screwed up (eg. value overflow in 2010 and in 2018), the user experience would reflect badly on Bitcoin.

With multiple implementations of the protocol, the chain's survival is ensured because for example in 2010 when the chain started growing with invalid blocks the alternative implementation would have rejected those blocks and any miner running that full node or multiple nodes would have built the correct chain ensuring that bitcoin network doesn't halt and could even prevent fraud such as double spends when the invalid blocks were removed from the chain.

@pooya87: That's a great point. The hashing power being distributed amongst a set of independently developed implementations would make bugs in a specific implementation pretty harmless.


Title: Re: Why aren't alternative implementations encouraged?
Post by: garlonicon on July 20, 2022, 10:27:48 AM
Quote
Do any other implementations see significant use by miners?
There is no way to check that, because if you have your own client, then it can still advertise as "Satoshi" with some version, so nobody would notice, that in practice, you use your own code. It is the same case as "User-Agent" field in web browsers, where the Tor Browser is simply "Firefox + Windows".

Also, even if some messages are fully removed from the protocol, some nodes could still handle that (for example, P2P marketplace from version 0.1.0), or they could allow more services than they advertise (for example P2P messaging, it is possible if some node accepts free or low-fee transactions).


Title: Re: Why aren't alternative implementations encouraged?
Post by: DaveF on July 20, 2022, 11:19:21 AM
...
Take PDF for example. I think it being an ISO standard is a great thing and I would be pretty bummed if Adobe had arranged things so that no one but them could provide a working implementation....


At least the last time I was dealing with it, in the financial markets it's actually closer to the BTC way. Several places had their own 'pretty' front end but the back end was all the same. The corporate users saw something different depending on where you worked, the consumers saw something different. The back end coding was from one place and it really did not change. BUT if you logged into your account from "A" brokerage and a different account from "B" brokerage you would think it was 100% different.

As for big pools and exchanges we are ASSUMING that they are on core we don't know if it is 100% the reference client or some heavily modified version of it or something else they put together. And for security they probably would not and should not tell us the truth.

-Dave


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotATether on July 20, 2022, 11:47:47 AM
I find that really unfortunate. I think alternative implementations should be encouraged.

I enthusiastically disagree.

If there are two independent Bitcoin clients with roughly equal market share, then the protocol becomes the intersection of supported features on both clients. The protocol is no longer singularly developed ahead of the reference implementation. It will be possible for some feature to be added to the protocol (by people who also happen to be devs of the first clients), but the second client torpedoes the feature by not implementing it. This is the exact hell that is facing LN at this very moment.

It's the reason why you don't see any major improvements to the PDF format. Because ISO standards have a tendancy of staying put.


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotFuzzyWarm on July 20, 2022, 03:54:08 PM
I find that really unfortunate. I think alternative implementations should be encouraged.
I enthusiastically disagree.

If there are two independent Bitcoin clients with roughly equal market share, then the protocol becomes the intersection of supported features on both clients. The protocol is no longer singularly developed ahead of the reference implementation. It will be possible for some feature to be added to the protocol (by people who also happen to be devs of the first clients), but the second client torpedoes the feature by not implementing it. This is the exact hell that is facing LN at this very moment.

It's the reason why you don't see any major improvements to the PDF format. Because ISO standards have a tendency of staying put.
NAT hit the nail on the head.
The whole point of a Standard is that no single entity - eg Adobe or Microsoft et al - can futz with it by adding 'features'  to make it more their own format - and in turn therefore attempt to make any other versions less desirable to use - based on their own particular marketing plans. Adobe in the past attempted to do exactly that every time a 3rd party put out a pdf reader/writer like Foxit Reader and initially MS's implementations of it tried to add 'Features' geared specifically  to the 'Doze OS. The whole point of pdf was to have a universal document format that is not tied to 1 specific software vendor or OS and to have documents produced using it to be able to be easily accessed for decades without having to find an old copy of the software that produced it.

Ja the bitcoin protocol is not managed by an outside entity like ISO but until (if ever) it is, having Core as the sole official implementation of it is the best we can have.

The developers involved with Core bend over backwards to ensure that any changes made to the BTC protocol have a REAL and useful functional purpose and do not break older versions. In the case of LN obviously the various developers could care less about that because they have their own agendas.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 20, 2022, 05:48:40 PM
If there are two independent Bitcoin clients with roughly equal market share, then the protocol becomes the intersection of supported features on both clients. The protocol is no longer singularly developed ahead of the reference implementation.

@NotATether: You see, I think that's a good thing. With N independent implementations and a reasonable distribution of hashing power between them, then as N gets bigger that "intersection" means a more bug-resilient network.

Of course, it also means a network that is much harder to implement changes on, but I'm not convinced that Bitcoin should be messed with all that often anyway. Personally, I would prefer it if Bitcoin was even more conservative about protocol changes than it already is.

It will be possible for some feature to be added to the protocol (by people who also happen to be devs of the first clients), but the second client torpedoes the feature by not implementing it. This is the exact hell that is facing LN at this very moment.

@NotATether: Maybe I'm a masochist, but I think that's a good thing too. It's painful, but that's how "real" distributed consensus should work, IMHO.

The whole point of a Standard is that no single entity - eg Adobe or Microsoft et al - can futz with it by adding 'features'  to make it more their own format based on their own particular marketing plans.

@NotFuzzyWarm: On first reading, I struggled to reconcile that I very much agree with this, but you very much agree with NAT, but NAT very much disagrees with me. :D


Title: Re: Why aren't alternative implementations encouraged?
Post by: garlonicon on July 20, 2022, 06:31:26 PM
Quote
it also means a network that is much harder to implement changes on
True, and some people seems to support stronger decentralization, without understanding this drawback: soft-forks like Segwit or Taproot were possible, because huge mining pools supported those changes. But imagine that we would have truly decentralized mining, with, I don't know, 10k miners with much lower difficulty, or even more, where each of them would share their 80-byte block headers, that would be later claimed by publishing some data to some nodes. Then, forming a consensus around any change would be harder. And the same would be about many clients: if you want decentralization, you have to accept that drawback, and you have to accept the fact that people would say "no", and then you should know, how to handle it correctly (for example, one possible solution is to follow the heaviest chain of Proof of Work, and allow claiming only valid shares, and burning/locking the rest, so if the coinbase reward is 1 BTC, but the honest nodes only have 1% of the total hashrate, they should take only 0.01 BTC, and the rest should be burned/locked).


Title: Re: Why aren't alternative implementations encouraged?
Post by: achow101 on July 20, 2022, 11:01:55 PM
It comes down to what tradeoffs you are willing to accept. Alternative implementations does not necessarily mean the network is more resilient.

Suppose there is some bug in one implementation that others do not have, and this bug causes that implementation to reject blocks that other implementations accept, thereby causing a chain split. This could be very dangerous and result in people losing money. And so people use this as a reason to advocate for homogenous implementation because chain splits are generally bad.

On the other hand, there could be a bug that causes the node to stop accepting blocks altogether. If only one implementation has this bug, they would stop accepting new blocks but the network would still continue operating as other implementations are not affected by the bug. And so people use this as a reason to advocate for alternative implementations because the network functioning is better than it being at a standstill.

IMO the reason is just a combination of the first mover effect and people taking Satoshi's word as gospel. Bitcoin Core is a direct descendant of the original Satoshi client and so people prefer to use it. Writing another implementation is also a major pain in the ass - there's a lot of stuff that needs to be implemented to be consensus compatible. Additionally, as you point out, Satoshi stated that he did not thing alternative implementations are a good idea, so many people have latched onto that.

Regardless, I would argue that we have alternative implementations already through people running different versions of Bitcoin Core. The major versions can be substantially different, especially once you get a few versions apart. Different versions have had consensus bugs that resulted in the same issues that alternative implementations pose. In terms of the features argument, people are encouraged to not upgrade until the accept the changes proposed in the new version. Soft forks are now done in minor versions for this reason - to allow people to get the non-consensus features of the major version without necessarily accepting the consensus changes. Just as an alternative implementation could just not implement a proposed soft fork, users can also just not upgrade to the version the deploys the soft fork.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 21, 2022, 12:43:03 AM
@achow101: I agree with basically your whole post, but could use some clarification on this bit:

Suppose there is some bug in one implementation that others do not have, and this bug causes that implementation to reject blocks that other implementations accept, thereby causing a chain split. This could be very dangerous and result in people losing money. And so people use this as a reason to advocate for homogenous implementation because chain splits are generally bad.

I'm not yet far enough along my path to understanding Bitcoin to have much confidence in the argument I'm about to make, but I'll try it anyway :P

Doesn't the chain split regularly[1], because of occasional mining "races"? And aren't these little splits resolved by simply following the longest chain?

In a very heterogeneous network with many different implementations and roughly evenly divided hashing power, wouldn't splits caused by implementation bugs resolve themselves in the same way?

[1] I mean regularly in the "as a matter of course" sense, not in the "frequently" sense.


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotFuzzyWarm on July 21, 2022, 12:54:30 AM
@achow101: I agree with basically your whole post, but could use some clarification on this bit:
Suppose there is some bug in one implementation that others do not have, and this bug causes that implementation to reject blocks that other implementations accept, thereby causing a chain split. This could be very dangerous and result in people losing money. And so people use this as a reason to advocate for homogenous implementation because chain splits are generally bad.
I'm not yet far enough along my path to understanding Bitcoin to have much confidence in the argument I'm about to make, but I'll try it anyway :P

Doesn't the chain split regularly, because of occasional mining "races"? And aren't these little splits resolved by simply following the longest chain?
...
For one, orphan races are not a regular event. For another, the average 10 min between blocks ensures that the losing chain is never more than at most 2-3 (very fast but not fast enough) blocks long so transactions in blocks built on the losing split are regenerated and soon re-confirmed in the longer chain. The Txs still exist in the mempool so no double-spends can happen, it's just that they have to be reconfirmed.


Title: Re: Why aren't alternative implementations encouraged?
Post by: achow101 on July 21, 2022, 02:06:39 AM
Doesn't the chain split regularly[1], because of occasional mining "races"? And aren't these little splits resolved by simply following the longest chain?

In a very heterogeneous network with many different implementations and roughly evenly divided hashing power, wouldn't splits caused by implementation bugs resolve themselves in the same way?

[1] I mean regularly in the "as a matter of course" sense, not in the "frequently" sense.
Not quite.

Nodes will follow the valid chain with the most work (most work usually mean longest, but not always). The key word here is valid. If the node receives a block that it determines to be invalid, it doesn't matter how much work is done on top of that block, the node will not accept that chain. That's the problem with implementation bugs. If one implementation determines a block is valid, and a different implementation finds the same block to be invalid, that's a consensus failure and potentially a chain split.


Title: Re: Why aren't alternative implementations encouraged?
Post by: pooya87 on July 21, 2022, 03:24:45 AM
One thing that should be clarified is the definition of "implementation". In my opinion if you are translating code from another project/language into your own project/language then you are not "implementing" anything. If any bugs exist in the original project it will be transferred into your project and additional bugs will be introduced during translation since languages are different.

For example some implementations of BIP32 that copied each other without understanding the protocol had a bug where they didn't pad the keys properly when hashing them. It happened because they were copying each other instead of implementing the protocol. Other stand alone projects that actually implemented the protocol like Electrum never had such a bug.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 21, 2022, 04:02:00 AM
One thing that should be clarified is the definition of "implementation". In my opinion if you are translating code from another project/language into your own project/language then you are not "implementing" anything. If any bugs exist in the original project it will be transferred into your project and additional bugs will be introduced during translation since languages are different.

For example some implementations of BIP32 that copied each other without understanding the protocol had a bug where they didn't pad the keys properly when hashing them. It happened because they were copying each other instead of implementing the protocol. Other stand alone projects that actually implemented the protocol like Electrum never had such a bug.

Very much agree. Different implementations should use absolutely nothing but documentation for reference. Unfortunately, most modern programmers only know how to solve problems by reaching for libraries so very few would be capable of producing a from-scratch clean-room implementation.


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotATether on July 21, 2022, 04:38:55 AM
The whole point of a Standard is that no single entity - eg Adobe or Microsoft et al - can futz with it by adding 'features'  to make it more their own format based on their own particular marketing plans.

In my opinion, there is nothing wrong with creating an ISO standard for some format or protocol if you do not plan to implement any major revisions for it, minor ones only. So on the contrary to what might be thought after my first reply, I don't see PDF's "staying put" as a bad thing, and as a file format its better that it stays like that.


If there are two independent Bitcoin clients with roughly equal market share, then the protocol becomes the intersection of supported features on both clients. The protocol is no longer singularly developed ahead of the reference implementation.

@NotATether: You see, I think that's a good thing. With N independent implementations and a reasonable distribution of hashing power between them, then as N gets bigger that "intersection" means a more bug-resilient network.

Of course, it also means a network that is much harder to implement changes on, but I'm not convinced that Bitcoin should be messed with all that often anyway. Personally, I would prefer it if Bitcoin was even more conservative about protocol changes than it already is.

Bug-resilience is only achieved if the N independent implementations all review every pull request that is submitted, only allow direct commits from a small group of people, and are tested thoroughly at a protocol level to stamp out regressions and accidential forks.

Can Bitcoin Core afford to do this? Yes, because it's manned by dozens of developers. Whether alternative clients could afford to do so however is doubtful. If you have an alternative client maintained by just 5 people, they might struggle to completely test and review all changes, let alone implement new protocol additions.

The hard fact is, most people do not want to maintain 200,000 lines of reference client code alone, which is the only reason we have a single client anyway. Otherwise this space would be buzzing with reference clients. Many promising OSS projects have been started by a group of a few people but fizzled out a few years later simply because the devs are burned out.

To understand how much 200000 (edit) lines is, I give this comparison: I myself am managing a codebase about six times smaller than this at work [nearly all of it written by yours truly], and even that is exhausting when done alone. I don't see how 5 or even 10 people can update a 200K-line codebase in a timely manner. So the question of having alternative implementations of a program at this scale has already been answered through the efforts of other developers: "No". (It's also the reason why, at 6 million lines of code, you don't see forks of the Linux kernel floating around.)

It effectively means that most clients would not styme progress by rejecting feature upgrades in Bitcoin Core, but they would quite literally not have enough manpower to implement them in the timespan of a few weeks. Thus the implementation takes several months or even years to develop, if at all. This unnecessarilly bogs down the schedule for rolling out the features to users at a protocol level, as there would be many users of the less-maintained client, but no work is being done to actively develop the required support for it although it could be desired by the alternative maintainers. In fact, it is very common for users to be using slow codebase, examples include every exchange whose wallet engine does not support native segwit addresses in 2022.

Or another example - Although some SPV wallets still don't support taproot, imagine if some of them were connecting to the protocol using an alternative blockchain, which itself has not yet implemented the taproot support code for lack of manpower. This would have had a downstream effect for those wallets which wanted to support Taproot addresses, but can't because it depends on support from the upstream reference client.


Title: Re: Why aren't alternative implementations encouraged?
Post by: witcher_sense on July 21, 2022, 08:14:31 AM
Regardless, I would argue that we have alternative implementations already through people running different versions of Bitcoin Core. The major versions can be substantially different, especially once you get a few versions apart. Different versions have had consensus bugs that resulted in the same issues that alternative implementations pose. In terms of the features argument, people are encouraged to not upgrade until the accept the changes proposed in the new version. Soft forks are now done in minor versions for this reason - to allow people to get the non-consensus features of the major version without necessarily accepting the consensus changes. Just as an alternative implementation could just not implement a proposed soft fork, users can also just not upgrade to the version the deploys the soft fork.
I find it useful to elaborate on why we have the diversity of Bitcoin Core versions or, more precisely, why it has been possible so far to preserve successfully such a diversity. The answer is there are no hardforks in the Bitcoin protocol. Traditionally, hardforks are considered highly undesirable because they are backward-incompatible, they make consensus rules softer, causing an inevitable chain split and weakening of the bitcoin community. Part of the reason hardforks are discouraged is that they are forcing everyone to upgrade simultaneously and in a coercive manner, which is against not only bitcoin core principles but also principles of libertarianism and the concept of private property. Moreover, the lack of backward compatibility means that old versions are no longer enforcing anything (at least in a new chain), and anyone who wants to be a part of a new chain (perhaps the one with the majority of hashrate) is forced to upgrade to exactly the same version of bitcoin full node software as everyone else. In short, a hardforked chain will have only one implementation of the full node, which may cause not only an inevitable decrease in a network and development process decentralization but also a complete failure of the whole network.

TL;DR Alternative implementations are encouraged, albeit implicitly, via the universal agreement that hardforks, in general, are taboo.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 21, 2022, 01:31:28 PM
Bug-resilience is only achieved if the N independent implementations all review every pull request that is submitted, only allow direct commits from a small group of people, and are tested thoroughly at a protocol level to stamp out regressions and accidential forks.

Can Bitcoin Core afford to do this? Yes, because it's manned by dozens of developers. Whether alternative clients could afford to do so however is doubtful. If you have an alternative client maintained by just 5 people, they might struggle to completely test and review all changes, let alone implement new protocol additions.

The hard fact is, most people do not want to maintain 200,000 lines of reference client code alone, which is the only reason we have a single client anyway. Otherwise this space would be buzzing with reference clients. Many promising OSS projects have been started by a group of a few people but fizzled out a few years later simply because the devs are burned out.

To understand how much 200000 (edit) lines is, I give this comparison: I myself am managing a codebase about six times smaller than this at work [nearly all of it written by yours truly], and even that is exhausting when done alone. I don't see how 5 or even 10 people can update a 200K-line codebase in a timely manner. So the question of having alternative implementations of a program at this scale has already been answered through the efforts of other developers: "No". (It's also the reason why, at 6 million lines of code, you don't see forks of the Linux kernel floating around.)

It effectively means that most clients would not styme progress by rejecting feature upgrades in Bitcoin Core, but they would quite literally not have enough manpower to implement them in the timespan of a few weeks. Thus the implementation takes several months or even years to develop, if at all. This unnecessarilly bogs down the schedule for rolling out the features to users at a protocol level, as there would be many users of the less-maintained client, but no work is being done to actively develop the required support for it although it could be desired by the alternative maintainers. In fact, it is very common for users to be using slow codebase, examples include every exchange whose wallet engine does not support native segwit addresses in 2022.

Or another example - Although some SPV wallets still don't support taproot, imagine if some of them were connecting to the protocol using an alternative blockchain, which itself has not yet implemented the taproot support code for lack of manpower. This would have had a downstream effect for those wallets which wanted to support Taproot addresses, but can't because it depends on support from the upstream reference client.

I enjoyed reading your post, but I think that what you've mostly addressed with it is why it's not possible (in practice) and not why it's not encouraged (the point of the OP). Of course, one could infer that it's not encouraged because it's not possible, but I don't think that's what's going on.

After digesting some of the answers above, I suppose it's no great mystery why the Bitcoin Core devs don't actively encourage outside implementations. It would make their lives more difficult and give them less control over the protocol and that's probably all there is to it.

I'm not entirely comfortable having a "de facto" majority implementation. I think that's too much power to concentrate in one team of programmers.

Right now, changes to the consensus rules can be blocked by lack of miner agreement. I think it would be better if they could also be blocked by lack of programmer agreement (i.e. an outside team refusing to update their implementation).

I'm aware that this would make implementing new features (especially controversial ones) nightmarishly difficult, but (in principle) I believe that's a good thing.

Obviously, any idea that leads to a glacial pace of feature development is not going to be a popular one, and I'm mostly just fantasizing out loud at this point :D


Title: Re: Why aren't alternative implementations encouraged?
Post by: HeRetiK on July 21, 2022, 02:29:21 PM
Right now, changes to the consensus rules can be blocked by lack of miner agreement. I think it would be better if they could also be blocked by lack of programmer agreement (i.e. an outside team refusing to update their implementation).

Arguably that's pretty much what happened during the hardforks of 2017. The scenario you describe is less extreme, of course, but in the end that would be the logical conclusion.



I'm aware that this would make implementing new features (especially controversial ones) nightmarishly difficult, but (in principle) I believe that's a good thing.

Obviously, any idea that leads to a glacial pace of feature development is not going to be a popular one, and I'm mostly just fantasizing out loud at this point :D

Compared to most alts Bitcoin's development already is moving at a glacial pace ;) And for good reason! But I fear increasing the political overhead would pretty much halt development and possibly even increase developer attrition.


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotATether on July 21, 2022, 03:09:24 PM
I enjoyed reading your post, but I think that what you've mostly addressed with it is why it's not possible (in practice) and not why it's not encouraged (the point of the OP).

I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 21, 2022, 07:35:37 PM
Compared to most alts Bitcoin's development already is moving at a glacial pace ;) And for good reason! But I fear increasing the political overhead would pretty much halt development and possibly even increase developer attrition.

@HeRetiK: I agree with you, and I'm beating a dead horse at this point, but I have a different definition of "glacial" and would much prefer it if the design was basically set in stone, never to be tweaked again. I don't have the same appetite for improvements that most people seem to.

Of course, that's blithely ignoring real issues like quantum cryptography, but I suspect that even once Bitcoin is "post-quantum" there will still be many more upgrades than are strictly necessary, with each one being an opportunity to get something wrong, with possibly disastrous consequences.

I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).

@NotATether: Yup, can't argue with that.


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotFuzzyWarm on July 21, 2022, 11:47:00 PM
Compared to most alts Bitcoin's development already is moving at a glacial pace ;) And for good reason! But I fear increasing the political overhead would pretty much halt development and possibly even increase developer attrition.
@HeRetiK: I agree with you, and I'm beating a dead horse at this point, but I have a different definition of "glacial" and would much prefer it if the design was basically set in stone, never to be tweaked again. I don't have the same appetite for improvements that most people seem to.

Of course, that's blithely ignoring real issues like quantum cryptography, but I suspect that even once Bitcoin is "post-quantum" there will still be many more upgrades than are strictly necessary, with each one being an opportunity to get something wrong, with possibly disastrous consequences.
I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).

@NotATether: Yup, can't argue with that.
Agreed. :)
Considering that BTC is a Financial System, it is admirable that so far the Core dev team have taken what in financial circles is called a Fiduciary approach to making sure the BTC network works securely as intended including an immutable and easily accessible record of all TX's back to the Genesis Block and nothing that is added breaks its intended purpose.

In lay terms it translates to, "Once something Works - DO NOT SCREW WITH IT unless there is a more than compelling reason to change it/add a 'Feature' to it". When there is a non-critical but valid reason, get a feeling how the major parties affected feel about the idea. If it is worth pursing then review the hell out of the code followed by tests the hell out of it with edge-cases before release. Rule #1 to that is Thou Shalt NOT Break Code! Everything created since day-1 (TX's) MUST be maintained and unchanged and seamlessly accessible by the new code.

I'm more than happy to let the altcoin folks experiment with interesting ideas for using blockchains and I'm certain that the Core developers watch what they do as well to see if what the alt folks come up with could *possibly* fit in well with "The BTC Manifesto" so to speak.

edited for text flow


Title: Re: Why aren't alternative implementations encouraged?
Post by: tiltedIceCream on July 22, 2022, 12:44:11 AM
I enjoyed reading your post, but I think that what you've mostly addressed with it is why it's not possible (in practice) and not why it's not encouraged (the point of the OP).

I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).


Having the protocol be the least common denominator is exactly what the protocol architects should be striving for. Everything else is an implementation detail. To keep things truly decentralized the knowledge of how Bitcoin works should be spec driven. This allows every technical person to weigh in on the high level challenges of the protocol through documents like bips. A friendly environment towards more implementations also helps ensure a system of checks and balances where core devs can't dictate the direction of the protocol against the majority of user wishes just because they have market share.

A real-world case study of this approach can be showcased through web browsers. Sure most browsers are based on chromium but browsers like Safari and Firefox (and spin-offs like the Tor browser) have unique approaches to protocol implementations that many internet users value.


Title: Re: Why aren't alternative implementations encouraged?
Post by: PowerGlove on July 22, 2022, 01:19:31 AM
I enjoyed reading your post, but I think that what you've mostly addressed with it is why it's not possible (in practice) and not why it's not encouraged (the point of the OP).

I intentionally worded it like that, because I already addressed why its not encouraged in my first reply (i.e. the protocol becomes the least common demoninator, which is exactly what the protocol architects do not want).


Having the protocol be the least common denominator is exactly what the protocol architects should be striving for. Everything else is an implementation detail. To keep things truly decentralized the knowledge of how Bitcoin works should be spec driven. This allows every technical person to weigh in on the high level challenges of the protocol through documents like bips. A friendly environment towards more implementations also helps ensure a system of checks and balances where core devs can't dictate the direction of the protocol against the majority of user wishes just because they have market share.

A real-world case study of this approach can be showcased through web browsers. Sure most browsers are based on chromium but browsers like Safari and Firefox (and spin-offs like the Tor browser) have unique approaches to protocol implementations that many internet users value.

That's very well put, and exactly how I feel. Have a merit and welcome to the forum!


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotFuzzyWarm on July 22, 2022, 01:36:08 AM
Quote
Having the protocol be the least common denominator is exactly what the protocol architects should be striving for. Everything else is an implementation detail.
What you describe already exists: Sidechains such as LN that at their base level are tied to Core and the main BTC network.

Sidechains lets folks explore a myriad of possibilities as long as in the end it's compatible with the BTC protocol while (hopefully) having zero or at worst only a minuscule chance of undesired impact to the main BTC blockchain. Any risk in using the side-chain MUST be only to the operators of the side-chain service and its users and not anyone else using BTC. Again, if what those side-chain folks come up with truly serves a truly needed purpose then ja, the Core devs should consider it provided it does NOT break anything or otherwise extensively modify the basics of how the BTC network operates.


Title: Re: Why aren't alternative implementations encouraged?
Post by: pooya87 on July 22, 2022, 03:13:54 AM
Let me disagree with two statements that were posted here which I call "assumptions".

1. Alternative implementations would disagree with any protocol change and cause problems
First of all this is not a bad thing, Bitcoin is not a dictatorship to fear any kind of disagreement with or rejection of proposals by others. In fact other opinions contribute to improvement of the proposal and help see the mistakes and come up with better proposal.
Secondly other implementations doesn't mean the devs can't come together to discuss proposals in one place. In fact alternative projects brings developers from other programming languages together.
Finally, if the proposal is any good, there shouldn't be any disagreements or rejections.
In the end the community (ie. miners + nodes + economy) should decide not the developers.

2. Alternative implementations would have inadequate number of developers
This is a false assumption in my opinion. Any serious project would get enough contributors to keep it going. BTCD example I posted earlier is a good example of a project with enough serious developers, Electrum is another example (SPV implementation of the protocol).


Title: Re: Why aren't alternative implementations encouraged?
Post by: NotATether on July 22, 2022, 04:41:45 AM
Having the protocol be the least common denominator is exactly what the protocol architects should be striving for. Everything else is an implementation detail. To keep things truly decentralized the knowledge of how Bitcoin works should be spec driven. This allows every technical person to weigh in on the high level challenges of the protocol through documents like bips.

Well yeah, as someone who is busy drafing a BIP right now (about standardizing the message verification algo - but that's an entirely different topic) - I am very happy that I can spearhead an improvement without having to maintain entire modules of BTC code first. And from what I know, the BIP merely has to reach a consensus with other people like me i.e. other mailing list posters - in order to be generally accepted. Here is a lot of power in the community's hand already, and pretty much explains, in a practical sense, why nobody has attempted a serious alternate implementation - because the BIP system for improving Bitcoin already lets them vote on changes they like and reject the ones that they don't like, so that's what they do. Read below for an explination for why this pevents protocol divergence.

Quote
A friendly environment towards more implementations also helps ensure a system of checks and balances where core devs can't dictate the direction of the protocol against the majority of user wishes just because they have market share.

When I was talking about developers, I was referring to anybody who'd be interested in contributing code to Bitcoin Core, lest my position be misunderstood.

I'd certainly not be happy if Core code contribution was entirely done by achow and a small group of people, because then the little devs would be shut out (and I'm already not happy with Electrum devs, for leaving my bugfix (https://github.com/spesmilo/electrum/pull/7584) PR dangling for months without even a comment about it <- this is what you get when implementations or protocols are managed entirely by 3 or 4 people{1}).

Having said that, most of this discussion has been very theoretical, because intra-protocol LCM has already been achieved due to the BIP voting system you mentioned, which is open to developers of all clients. Mathematically speaking, consensus converges to some number.

Whereas inter-protocol LCM is pretty much non-existant because attempts to create the conditions necessary for one, via the opposing developers' own client plus its own system of proposals that do not carry over to BIPs, eventually lead to a chain split anyway [like Bcash]. A consensus divergence to infinity basically.

Let me repeat, that creating an alternative client does not in itself create the necessary conditions for divergent protocol improvements between them. (I have not seen libbitcoin server make a single part of the protocol different from Core, for example). I think we can all agree on this.

Also, implementing some BIPs but not others does not create the necessary conditions for divergent protocols either, as most of them can be implemented independently from each other.

What *does* create divergent protocols is when the other alternative client has a different system for making improvements to the protocol (ie something other than BIPs) or if it has no improvement system at all and merely adds protocol features at a user's or developer's whim. Unfortunately this is what most people who don't like the protocol do - they fork not only the client, but also the BIP system. That is to say, from that point BIPs for one protocol will not apply for the other (unless they are proposed to both protocols), and vice versa.

And the reason why they don't make alternative clients in the first place has been written in my second post. The few tough lot like Libbitcoin who have succeeded in making an alternative client, do not burden themselves with maintaining a clone proposal system - which is necessary for the concept of divergent protocols to exist.

There is generally a reason why people make alternative implementations in the first place - it's usually not to cause protocol divergence (and since most people make clients exactly for this purpose I have been specifically warning about this type of client earlier), but to implement non-protocol features that are not present inside Bitcoin Core. For example, RPC calls and command-line options. Maybe a better way of selecting peers, or a way to kick out spam & bot peers which does not exist for Core. They are implementaiton details, after all.

Very few people make a client for this purpose anyway, because it's cruel, menial work. They very much perfer to make their own protocol so they can be more glamorous and advertise how their protocol (which they usually refer to as a "coin" - they are doing this as a hardfork anyway) is so much better than the first one.

edit {1}: I call it a bug, because it is impossible to view transaction history & balance on a restored wallet via the RPC otherwise.
edit {2} bbcode formatting kludge
edit {3}:


2. Alternative implementations would have inadequate number of developers
This is a false assumption in my opinion. Any serious project would get enough contributors to keep it going. BTCD example I posted earlier is a good example of a project with enough serious developers, Electrum is another example (SPV implementation of the protocol).

Seriousness can't be measured in terms of developers. Because developers can quit but the project does not become any less serious. The quit factor will always be there, since someone started the project which means there is at least one dev for it.

They might quit because the code is perfect (in their opinion) and nothing more needs to be done. Or they might quit due to outside factors, entirely unrelated to code - especially because most of them work for free. Many an abandoned linux project falls in the second category of this thinking, by the way. We can't say that those are not serious projects.


Title: Re: Why aren't alternative implementations encouraged?
Post by: Wind_FURY on July 22, 2022, 12:23:45 PM
Now, I'm aware that nothing prevents anyone from attempting an alternative implementation and that someone could conceivably read the source code and proceed from there.
Alternative implementations do exist though. Like BTCD (https://github.com/btcsuite/btcd).


Shouldn't the reference implementation be one of many?
There should be only one reference implementation but other alternative options for people to run as full node.

..
BTCD is not a unique node that allows people an alternative path to make protocol Bips or upgrades features from.
read their comment on the link YOU provided


Debatable. Because the BTCD developers are outside the political process of the Core developers. It can also be debated that IF BTCD starts gaining a larger following, it can create their own political process and compete with the Core developers.

In that hypothetical situation, it's probably A "good" for decentralization, but it's also debatable IF the "other implementation/reimplementation" is "still" the Bitcoin protocol.


Title: Re: Why aren't alternative implementations encouraged?
Post by: pooya87 on July 22, 2022, 12:29:05 PM
Let's not change the discussion from the title into whether BTCD was a good example or not.


Title: Re: Why aren't alternative implementations encouraged?
Post by: ABCbits on July 22, 2022, 01:01:14 PM
A real-world case study of this approach can be showcased through web browsers. Sure most browsers are based on chromium but browsers like Safari and Firefox (and spin-offs like the Tor browser) have unique approaches to protocol implementations that many internet users value.

It's great example. Now i'm inspired to try running alternate Bitcoin full node implementation and will write short review/guide about it within next few days.

--snip--

as for saying 'its just following consensus'. which you forget, whom is in control of consensus changes. yep one dev team, and who is that dev team.. (rhetorical)

Just wondering, do you think are any implementation which do more than following most/all consensus? If yes, do you currently run such implementation (even if it's not running 24/7)?


Title: Re: Why aren't alternative implementations encouraged?
Post by: garlonicon on July 22, 2022, 02:50:29 PM
Quote
Just wondering, do you think are any implementation which do more than following most/all consensus?
Yes.

Quote
If yes, do you currently run such implementation (even if it's not running 24/7)?
Yes, 24/7, with some minor halts for maintenance.

For example, there are nodes that accept free transactions, and they are used for P2P messaging. As long as free transactions are rejected by miners, it is safe enough. If those transactions will be accepted, then it is possible to switch to negative fee transactions with proper sighashes, then it is doubtful that any miner would attach its own coins to mine that.


Title: Re: Why aren't alternative implementations encouraged?
Post by: aliashraf on July 22, 2022, 06:37:20 PM
Hello everyone, my apologies for not quoting/following a specific post, I've read and enjoyed all, now I have something to share:

A tribute to redundancy

Any  software is susceptible to containing bugs, lurking in the shadows, waiting for the show-time.
However, bugs are not magic, they can be identified and fixed through code review and testing, yet there is a trade-off between 1)the resources one dedicates to this process and the wasted opportunities resulting from delays in launching the code, and 2)the actual risks that are involved and have to be addressed. An infinite resource and time allocation, theoretically can address almost all the risks, but it is not how we develop software in real world.

So, what alternatives do we have to deal with mission-critical software where failure is not tolerable? It is established practice: redundancy. Unlike what is trended here, I'm in support of redundant alternative clients.  8)

As of the fears, uncertainties and doubts which Satoshi has triggered :D about the possible chain splits, I'm not proposing it as a naive scenario in which competing clients are exposing the network to unintentional forks, etc. On the contrary, I think it was Satoshi who put bitcoin in bug-related risks by discouraging alternate implementations. Hence, CVE-2018-17144 (https://bitcoinops.org/en/topics/cve-2018-17144/) was his fault and not Peter Wuille's. Devs are entitled to make mistakes once or twice a year, it is the architect who is in charge of fault tolerance and risk management.

Obviously, for bitcoin as a new class of distributed systems, naively recommending multiple implementations is not reasonable, the same as unbelievable recommendation of one god, one client! As a more detailed assessment of the risks would reveal, there are 3 class of untolerable, high priority failures due to bug where non of them is truly addressed by neither of the two architectures under consideration. Let's take a look:

Risk #1: Unintentional forks happen with disastrous consequences, because of different nodes, implementing different clients or different versions.

Risk #2: Nodes explicitly and widely compromise official protocol specs, yet consensus is reached by a considerable portion of the network.

Risk #3: Healthy, protocol compliant nodes are not able to bootstrap as a result of mild versions of the above case, where the breach is considered to be tolerable, hence no roll-back or even bug-fix is applied.


Single (dominating) client architecture:
#1- Is not fully adequate, even for risk #1 because of the versioning and slow adoption process. Despite huge dominance in the network, bitcoin development and versioning follows an exaggerated conservatism and undergoes an exhaustive dedication of time and resource in the upgrade process, a clear deviation of state of the art agile development methodologies with huge resource and opportunity lost consequences.

#2- Fails to meet any measure against this risk. The CVE-2018-17144 mentioned above, is the nearest case (yet not an exemplary one) where a bug can stay in shadows for long times, being inherited version by version, getting enough dominance in the network waiting for the show-time, usually the worst time ever.

#3- Can enforce circumventing measures like what is called "consensus bugs" by Core devs and BIPs are required to comply, a reasonable approach, I've to admit.

Naive multiple clients architecture: ( where we have different implementations installed by the nodes deliberately)
It suffers from the same problems as the single client model with different degrees of severeness, better for risk #2 and worse for #1, to be specific.


I'd suggest a third model: an Enhanced Heterogeneous Multiple Clients Architecture, as follows:

1-We classify nodes into 2 categories, high-end nodes with more critical mission with regard to the owners' expectations and resource allocation (we expect them to be properly aligned, i.e., more expectations ---> stronger nodes installed).

2-We encourage/support multiple alternative clients to be developed by different parties, with diverse platforms.

3-We certify, introduce the client software after passing the appropriate checklists.

4-Implementations are supposed to meet a well-defined integration spec via RPC/API/..., for supporting a robust voting module, called before  changing the state of the blockchain and (optionally) the mempool.

5-We recommend first class users to install 3 different clients of their choice, ordinary less critical users are free to choose one client of their choice,

As a result, bugs cause disagreements in high-end nodes which are resolved by voting, just triggering bug alerts, while the affected low-end clients are not supposed to cause catastrophic damages. Other than very high fault tolerance, it has an astonishingly huge boosting effect on development process.

AFAICT, there is no other comparable architecture for a system such as bitcoin while, interestingly, huge impacts on everything discussed here will be in the horizon, positive impacts, for sure. Bests of both worlds are achieved, and the network is guaranteed to have an astronomical level of fatal failure resistance. Use your own imagination  8)



Title: Re: Why aren't alternative implementations encouraged?
Post by: Wind_FURY on July 23, 2022, 12:12:12 PM
Let's not change the discussion from the title into whether BTCD was a good example or not.


That wasn't the point. The point was, in my opinion, reimplementations should be discouraged because of the reasons that I posted in reply to franky1's post. That was only the hypothetical situation. BUT the actual situation is, there's not enough users that matter who run those other implementations/reimplementations, which could also be debated that it can only pull contributors away from Core, and could make Core's developer community smaller, and therefore possibly making its development process more centralized.