Why soft fork is a very bad idea and should be avoided at all costs

[johnyj]

I was surprised by the latest announcement of Pieter that a SegWit implementation which changes pretty much everything in bitcoin can be implemented via a soft fork, where it does not require all the nodes to upgrade to be compatible

After a bit research, this is possible because you can always give old data new meaning in a new implementation, while the old nodes simply do not know how to parse that data

In principle, with this method, you can move all the transaction data out of the block, not only signature data, so that old nodes always see empty blocks

As a result, a new block will be accepted by the old nodes because they appear to be valid blocks, but in the new implementation, they are just a small subset of the new data structure, all the transaction data is in another related block


More importantly, after further analysis, it shows that although normal nodes can still run old client, all the mining nodes will have to upgrade, otherwise the blocks mined by them will simply be orphaned, because the new nodes do not accept old blocks, and new nodes have majority of hash power

As a result, such a soft fork is to force 100% of the mining nodes move to the new implementation so that there is no slightest chance of the old mining nodes forking into their own chain: Their fork will always be orphaned since they accept the longest chain


What this means in practical?

If one type of nodes, Classic for example, controls over 51% of hash power, they can implement whatever change they like by simply use this soft fork trick: Let all the old Core nodes accept their new format blocks while Classic nodes will reject Core blocks, so that Core blocks all get orphaned due to less hash power

The most deadly part is that Classic nodes do not need to ask for permission of Core nodes, they simply enforce it by orphaning blocks mined by those Core nodes. And Core nodes have no way to fork into another chain by rejecting Classic blocks, since they can not tell which block is Core block and which block is Classic block

As a result, a miner running core will simply lose money, he either trash his miners and quit the game, or bite the bullet and upgrade to Classic so that his hash power can still mine him some coins. I guess majority of the miners will upgrade. And they might start another round of similar soft fork to regain control when they accumulated more than 51% of hash power


This is very bad, since it totally removed the freedom of choice for those old mining nodes, you can call it a forceful take over of bitcoin network rules by 51% hash power. It totally break the major consensus rule and the spirit of bitcoin, so that with only 51% of hash power, you can implement whatever you want and force it upon rest of the miners and their hash power will be captured

Sure, Core nodes can change to another PoW fork, but since the value of the coin is always roughly the same as the mining cost due to arbitraging, that coin will not get more value than litecoin and will be forgotten quickly


Unfortunately, it is almost impossible to prevent such thing from happening by current design, so it is very important in mining community to widely spread this information so that miners are fully aware of the deadly effect of a soft fork, and reject such attempt as much as possible. At mean time, trying to find a solution to fix this security vulnerability

[achow101]

I was surprised by the latest announcement of Pieter that a SegWit implementation which changes pretty much everything in bitcoin can be implemented via a soft fork, where it does not require all the nodes to upgrade to be compatible

After a bit research, this is possible because you can always give old data new meaning in a new implementation, while the old nodes simply do not know how to parse that data

In principle, with this method, you can move all the transaction data out of the block, not only signature data, so that old nodes always see empty blocks

As a result, a new block will be accepted by the old nodes because they appear to be valid blocks, but in the new implementation, they are just a small subset of the new data structure, all the transaction data is in another related block


More importantly, after further analysis, it shows that although normal nodes can still run old client, all the mining nodes will have to upgrade, otherwise the blocks mined by them will simply be orphaned, because the new nodes do not accept old blocks, and new nodes have majority of hash power

As a result, such a soft fork is to force 100% of the mining nodes move to the new implementation so that there is no slightest chance of the old mining nodes forking into their own chain: Their fork will always be orphaned since they accept the longest chain

The same happens with hard forks as well. Deployment requires consensus.

What this means in practical?

If one type of nodes, Classic for example, controls over 51% of hash power, they can implement whatever change they like by simply use this soft fork trick: Let all the old Core nodes accept their new format blocks while Classic nodes will reject Core blocks, so that Core blocks all get orphaned due to less hash power

Again, deployment requires consensus, as it does with hard forks.

The most deadly part is that Classic nodes do not need to ask for permission of Core nodes, they simply enforce it by orphaning blocks mined by those Core nodes. And Core nodes have no way to fork into another chain by rejecting Classic blocks, since they can not tell which block is Core block and which block is Classic block

Anyone can do that now. You can orphan blocks mined by other miners now, it makes no difference. Again, consensus. Same thing happens with hard forks.

As a result, a miner running core will simply lose money, he either trash his miners and quit the game, or bite the bullet and upgrade to Classic so that his hash power can still mine him some coins. I guess majority of the miners will upgrade. And they might start another round of similar soft fork to regain control when they accumulated more than 51% of hash power

They will probably upgrade. As I said before, and I will say it again, consensus.

This is very bad, since it totally removed the freedom of choice for those old mining nodes, you can call it a forceful take over of bitcoin network rules by 51% hash power. It totally break the major consensus rule and the spirit of bitcoin, so that with only 51% of hash power, you can implement whatever you want and force it upon rest of the miners and their hash power will be captured

So really what is the difference in the forking between a hard fork and a soft fork? The Core nodes can proceed as they did because their chain is orphaned. Any blocks produced by the core nodes are ignored by the Classic nodes. The core nodes, if they are able to build extend their own chain, or at least build a block on top of a core block, results in their chain completely separating from the classic chain since they have a different block history. It splits, much like a hard fork.

Sure, Core nodes can change to another PoW fork, but since the value of the coin is always roughly the same as the mining cost due to arbitraging, that coin will not get more value than litecoin and will be forgotten quickly


Unfortunately, it is almost impossible to prevent such thing from happening by current design, so it is very important in mining community to widely spread this information so that miners are fully aware of the deadly effect of a soft fork, and reject such attempt as much as possible. At mean time, trying to find a solution to fix this security vulnerability

And how would you propose to fix this? It isn't a security vulnerability, it is an issue with forking that is not able to be resolved due to the nature of forking. It isn't really any worse than hard forks are, but with the backwards compatible nature of soft forks, not all nodes are required to upgrade, only mining nodes. Hard forks require that all nodes upgrade.

Again, the way that both hard forks and soft forks are deployed are with consensus. Neither one is deployed unless a supermajority of the blocks indicate support for the fork. The deployment is the same with both forks. I don't see how a soft fork is worse than a hard fork.

tl;dr All forks are dangerous and all forks must be deployed with consensus otherwise problems happen. Applies for both hard and soft forks.

[johnyj]

Anyone can do that now. You can orphan blocks mined by other miners now, it makes no difference. Again, consensus. Same thing happens with hard forks.
.
.
They will probably upgrade. As I said before, and I will say it again, consensus.

So really what is the difference in the forking between a hard fork and a soft fork? The Core nodes can proceed as they did because their chain is orphaned. Any blocks produced by the core nodes are ignored by the Classic nodes. The core nodes, if they are able to build extend their own chain, or at least build a block on top of a core block, results in their chain completely separating from the classic chain since they have a different block history. It splits, much like a hard fork.

It seems you still have not get the fundamental difference here, and it is indeed difficult, because you have to first understand the most technical intensive part, e.g. the old nodes do not know at all that there is a new chain forming

It is dangerous to do a hard fork with only 51% hash power, because it will end up with 2 chains and you can not guarantee that your chain will win eventually. So you must seek major consensus before you fork.

However, you don't need major consensus to do a soft fork, only 51% hash power. In fact the name is misleading, there will be no fork in a "soft fork". You can be rest assured that there will never be a competing chain, e.g. even the other 49% of mining hash power disagree with the new chain, they can not fork their own chain, because all their nodes regard new chain to be the longest valid chain, or to say, old nodes do not know it is already a new chain, they thought that they are still working on the old chain, the only difference they will notice is that suddenly all their mined blocks are orphaned one by one

And how would you propose to fix this? It isn't a security vulnerability, it is an issue with forking that is not able to be resolved due to the nature of forking. It isn't really any worse than hard forks are, but with the backwards compatible nature of soft forks, not all nodes are required to upgrade, only mining nodes. Hard forks require that all nodes upgrade.

Of course the final result will be similar, but a hard fork with major consensus will create two chains, so that those disagree with major consensus could run their fork as a small alt-coin with much less hash power and value, and they will see later that chain is not practical both economically and security wise but they can still run that chain forever for hobby, or even as a test network

With a soft fork, there is no such possibility, if you disagree, you can not fork your own chain, either you upgrade to new chain, or you node become the slave of the new chain, you don't have a choice

Technically, using a soft fork can make sure that there will never be a fork, since all those mining nodes that disagree with new implementation will be captured and just mine useless blocks that will never be accepted into the blockchain. So I can understand it is a powerful hack to keep the highest level of integrity of the network. But everything has two sides, since it is such a devastating weapon that can overtake the network without major consensus, what if your political enemy get it and fire it up on you?

[franky1]

to: johnyj
a slight correction

soft fork attack, does not produce a different chain. it just manipulates the information within the chain to make older clients no longer understand much of what they are passing along. basically giving them cataracts and told everything is fine just pass it on, blindly.

the older clients would still pass on the information(thus being "compatible"), but without checking it. making old nodes no longer "full" and just "compatible"
imagine it like downgrading bitcoin-qt's* network responsibility down to being multibit's network responsibility, without the user choosing to be downgraded or ultimately informed that its a downgrade, using wishy washy words that everything will be fine..
*i used QT as just an example of a full node instead of core, purely to avoid confusion of blockstreams debated future implementations of the name core

people need to realise that once segwit hits.. all other full node implementations are no longer "full nodes" but "compatible, but impaired" nodes
there is not a 51% numberic flag..
basically
if 1000 people make a transaction.. and one of them is a segwit implementation
99.9% of transactions the network will understand and 0.1% of transactions wont be understood. but that transaction would still be handed over blindly
if 1000 people make a transaction.. and 250 of them is a segwit implementation
25% of the network will understand 100% of transactions. and 75% wont understand a quarter of transactions they handle. but the transactions would still be handed over blindly

once the segwit debate is over, the next generation of softfork is weakblocks. which would again reduce the verifiable data inside the main block and push it out into secondary blocks, changing from having cateracts, to complete blindness



this is the theorized idea of what would be happening between now and 2017


and blockstreams rhetoric is... dont worry, you wont be affected.
but when pointing out that full nodes are made blind,, they reply ..if you want to be a full node. you have to upgrade to our agenda

[Carlton Banks]

and blockstreams rhetoric is... dont worry, you wont be affected.
but when pointing out that full nodes are made blind,, they reply ..if you want to be a full node. you have to upgrade to our agenda

Upgrading your node would be the answer. What's the problem with that? Keep it concise, please

[achow101]

Anyone can do that now. You can orphan blocks mined by other miners now, it makes no difference. Again, consensus. Same thing happens with hard forks.
.
.
They will probably upgrade. As I said before, and I will say it again, consensus.

So really what is the difference in the forking between a hard fork and a soft fork? The Core nodes can proceed as they did because their chain is orphaned. Any blocks produced by the core nodes are ignored by the Classic nodes. The core nodes, if they are able to build extend their own chain, or at least build a block on top of a core block, results in their chain completely separating from the classic chain since they have a different block history. It splits, much like a hard fork.

It seems you still have not get the fundamental difference here, and it is indeed difficult, because you have to first understand the most technical intensive part, e.g. the old nodes do not know at all that there is a new chain forming

It is dangerous to do a hard fork with only 51% hash power, because it will end up with 2 chains and you can not guarantee that your chain will win eventually. So you must seek major consensus before you fork.

However, you don't need major consensus to do a soft fork, only 51% hash power. In fact the name is misleading, there will be no fork in a "soft fork". You can be rest assured that there will never be a competing chain, e.g. even the other 49% of mining hash power disagree with the new chain, they can not fork their own chain, because all their nodes regard new chain to be the longest valid chain, or to say, old nodes do not know it is already a new chain, they thought that they are still working on the old chain, the only difference they will notice is that suddenly all their mined blocks are orphaned one by one

Not true, just look at the July 4th fork for an example of a fork that happens with soft forks.

Blocks that are mined by the nodes with the old rules are invalid by the new rules. But if there are enough miners mining old blocks, then they could still be extending the old chain. In fact, if they were able to extend the old chain, you would end up with two different blockchains because the histories are different. Once an old block is inserted into the blockchain and miners extended on that, it would fork since the blockchain of new blocks is invalid since it has a different history. Likewise, the chain of old blocks is invalid to the other miners because they are old blocks.

[johnyj]

Anyone can do that now. You can orphan blocks mined by other miners now, it makes no difference. Again, consensus. Same thing happens with hard forks.
.
.
They will probably upgrade. As I said before, and I will say it again, consensus.

So really what is the difference in the forking between a hard fork and a soft fork? The Core nodes can proceed as they did because their chain is orphaned. Any blocks produced by the core nodes are ignored by the Classic nodes. The core nodes, if they are able to build extend their own chain, or at least build a block on top of a core block, results in their chain completely separating from the classic chain since they have a different block history. It splits, much like a hard fork.

It seems you still have not get the fundamental difference here, and it is indeed difficult, because you have to first understand the most technical intensive part, e.g. the old nodes do not know at all that there is a new chain forming

It is dangerous to do a hard fork with only 51% hash power, because it will end up with 2 chains and you can not guarantee that your chain will win eventually. So you must seek major consensus before you fork.

However, you don't need major consensus to do a soft fork, only 51% hash power. In fact the name is misleading, there will be no fork in a "soft fork". You can be rest assured that there will never be a competing chain, e.g. even the other 49% of mining hash power disagree with the new chain, they can not fork their own chain, because all their nodes regard new chain to be the longest valid chain, or to say, old nodes do not know it is already a new chain, they thought that they are still working on the old chain, the only difference they will notice is that suddenly all their mined blocks are orphaned one by one

Not true, just look at the July 4th fork for an example of a fork that happens with soft forks.

Blocks that are mined by the nodes with the old rules are invalid by the new rules. But if there are enough miners mining old blocks, then they could still be extending the old chain. In fact, if they were able to extend the old chain, you would end up with two different blockchains because the histories are different. Once an old block is inserted into the blockchain and miners extended on that, it would fork since the blockchain of new blocks is invalid since it has a different history. Likewise, the chain of old blocks is invalid to the other miners because they are old blocks.

Yes, the July 4th fork is possible because more than 50% of the hash power was doing SPV mining thus they could fork an old chain with major hash power. But in my example, the new chain owns more than 51% or even higher amount of hash power, which makes this kind of fork impossible

This is also another dangerous aspect of the soft fork, since some pools who have more than 30% of hash power might first support the new implementation, and after the soft fork is triggered, they changed idea and reverse to the old version thus make the old hash power larger than 55% again, thus the new implementation will stay in a new fork with minor hash power and possibly die

[johnyj]

and blockstreams rhetoric is... dont worry, you wont be affected.
but when pointing out that full nodes are made blind,, they reply ..if you want to be a full node. you have to upgrade to our agenda

Upgrading your node would be the answer. What's the problem with that? Keep it concise, please

The question is upgrade to which version: SW? Classic? BU? XT? They all can be implemented use this soft fork trick, and it seems Classic currently have more than 51% hash power backing

[achow101]

Yes, the July 4th fork is possible because more than 50% of the hash power was doing SPV mining thus they could fork an old chain with major hash power. But in my example, the new chain owns more than 51% or even higher amount of hash power, which makes this kind of fork impossible

How does it make it impossible? If your 51% of the miners are also SPV mining, they can just as likely fork the blockchain. Even if they are not, just one miner with the old rules getting a few blocks onto the blockchain can result in a fork.

This is also another dangerous aspect of the soft fork, since some pools who have more than 30% of hash power might first support the new implementation, and after the soft fork is triggered, they changed idea and reverse to the old version thus make the old hash power larger than 55% again, thus the new implementation will stay in a new fork with minor hash power and possibly die

The same can happen with a hard fork.

[johnyj]

to: johnyj
a slight correction

soft fork attack, does not produce a different chain. it just manipulates the information within the chain to make older clients no longer understand much of what they are passing along. basically giving them cataracts and told everything is fine just pass it on, blindly.

the older clients would still pass on the information(thus being "compatible"), but without checking it. making old nodes no longer "full" and just "compatible"
imagine it like downgrading bitcoin-qt's* network responsibility down to being multibit's network responsibility, without the user choosing to be downgraded or ultimately informed that its a downgrade, using wishy washy words that everything will be fine..
*i used QT as just an example of a full node instead of core, purely to avoid confusion of blockstreams debated future implementations of the name core

Exactly, I'm against this hacky soft fork practice since this is pure scam, it is possible because bitcoin nodes are so stupid, they can not understand something that is not in their existing knowledge base, they can not sense the change in environment

I recall this trick was originally brought out by Mastercoin, by encoding information into bitcoin transactions, they could use blockchain to securely transfer other information. This parasitic behavior caused some panic among devs, Peter Todd even joined them to see what was happening there. But it seems it costs too much to transport information using blockchain, so I did not hear from them any more. But the idea is widely spread and I think the side chain concept is also spawned from this thought

Happened read this days ago

https://www.cryptocoinsnews.com/davos-elites-talk-bitcoin/

“We know that bitcoin itself is a complete failure and shows the number one law of programming and software: that anything that can be programmed can be hacked. So nothing is completely secure,” said Willem Buiter, Citi’s chief economist.

This guy is smart, he really get some fundamental philosophy here. But he forget about that without hash power hackers don't have write permission to bitcoin blockchain. It seems the hash power is the only force to prevent bitcoin from being hacked. It is very important to make sure the mining nodes are not infected by this kind of virus, otherwise one day you wake up just find out that all your blocks are orphaned and you have to upgrade to a new version with the amount of total coin supply extended to 42 million  

[johnyj]

Yes, the July 4th fork is possible because more than 50% of the hash power was doing SPV mining thus they could fork an old chain with major hash power. But in my example, the new chain owns more than 51% or even higher amount of hash power, which makes this kind of fork impossible

How does it make it impossible? If your 51% of the miners are also SPV mining, they can just as likely fork the blockchain. Even if they are not, just one miner with the old rules getting a few blocks onto the blockchain can result in a fork.

This is also another dangerous aspect of the soft fork, since some pools who have more than 30% of hash power might first support the new implementation, and after the soft fork is triggered, they changed idea and reverse to the old version thus make the old hash power larger than 55% again, thus the new implementation will stay in a new fork with minor hash power and possibly die

The same can happen with a hard fork.

Indeed, the new chain might also have lots of SPV miners, but I think if they are aiming for a hostile take over with 51% hash power, they will be well prepared so that all their mining nodes are not doing SPV mining

A hard fork does not have so many uncertain risky scenario that can go wrong. A hard fork is a well informed situation so that everyone knows what will happen and be prepared long before it happens

[gmaxwell]

I'm sorry you disagree with the primary process by which the Bitcoin system evolves which has been used since the very beginning.

Perhaps you should take this to the altcoin subforum. No one is going to buy into the suicide pact Hearn and you seem to wish for-- and even if they did, they can't enforce it: soft forks can't be prevented.

[achow101]

Indeed, the new chain might also have lots of SPV miners, but I think if they are aiming for a hostile take over with 51% hash power, they will be well prepared so that all their mining nodes are not doing SPV mining

True, but a soft fork can still result in forks even when not SPV mining, as I have explained above.

A hard fork does not have so many uncertain risky scenario that can go wrong. A hard fork is a well informed situation so that everyone knows what will happen and be prepared long before it happens

And so are soft forks. When soft forks are planned to happen, everyone knows what will happen and they are known ahead of time. Like hard forks, they still require consensus in order to fork smoothly.

[johnyj]

I'm sorry you disagree with the primary process by which the Bitcoin system evolves which has been used since the very beginning.

Perhaps you should take this to the altcoin subforum. No one is going to buy into the suicide pact Hearn and you seem to wish for-- and even if they did, they can't enforce it: soft forks can't be prevented.

I know that soft fork eliminate the risk of fork by forcing nodes to upgrade from old version, but this can also work against you when some one with a different political interest gathered enough hash power, just want to list the fact

And I am not aware of this is the primary process since the very beginning, where can you point me to for a detailed documentation?

[Carlton Banks]

And I am not aware of this is the primary process since the very beginning, where can you point me to for a detailed documentation?

It was almost certainly here on bitcointalk, Jonny

[Indianacoin]

Hardforks are much safer. When people claim softforks are safer, they are only referring to the software developers pushing the changes.
A softfork can be shipped with little to no consensus.  Keep in mind that Bitcoin is about what people run.  
So what node you run and which SPV client you run decides what you think is Bitcoin.
The problem with soft-forks is that that the software people no longer need your vote. And so they get full power over the protocol.

So its not safe to end users or node operators, its safe for the new policy makers of Bitcoin.

You can read more about why soft fork is unwelcome here: https://medium.com/@octskyward/on-consensus-and-forks-c6a050c792e7#.6c7yhmrg2

In a soft fork, a protocol change is carefully constructed to essentially trick old nodes into believing that something is valid when it actually might not be.

[Mashuri]

What matters most to the user is that their own bitcoin transactions are verified. Anyone running these nodes are not compatible with many soft forks currently running.  They don't care because it's not any transactions that involve their own bitcoin.  Soft forks preserve every users' choice whether or not to opt in, whereas hard forks break this capability -- along with introducing other unknowable systemic risks, but that's for another post.

[rizzlarolla]

OP Are you saying "Why soft fork TO SEGWIT is a bad idea?" I agree. Good post.

Segwit just ain't Bitcoin.

[achow101]

OP Are you saying "Why soft fork TO SEGWIT is a bad idea?" I agree. Good post.

He is saying that all soft forks are bad, but it is interesting that he only brings it up now when SegWit is being prepared to be deployed. We have had multiple soft forks already to deploy new features. It seems to just be an attack at SegWit while simultaneously attacking soft forks.

Segwit just ain't Bitcoin.

How so?

[sturle]

If one type of nodes, Classic for example, controls over 51% of hash power, they can implement whatever change they like by simply use this soft fork trick: Let all the old Core nodes accept their new format blocks while Classic nodes will reject Core blocks, so that Core blocks all get orphaned due to less hash power

This is nothing but a classic 51% attack by cooperating miners, and has nothing whatsoever to do with soft forks.  If you manage to control 51% of the hashpower, you can ignore everyone else and always have the best chain.  Bitcoin 101.