Bitcoin Forum

Introduction

It is generally assumed that increasing the blocksize limit requires a
hardfork.  Instead we show that a increasing the limit can be achieved using a
"generalized" softfork.  Like standard softforks, generalized softforks need a
mere miner majority (>50% hashpower) rather than global consensus.

Standard Softforks

After a softfork two potential chains exist:

* The new chain B3,B4,B5,... valid under the new rules and old rules.
* The old chain B3',B4',B5',... valid under the old rules only.

E.g.


Assuming that >50% of the hashpower follow the new rules, the old chain is
doomed to be orphaned:


Hardforks may result in two chains that can co-exist indefinitely:


Generalized Softforks

A *generalized* softfork introduces a transform function f(B)=B' that maps a
block B valid under the new rules to a block B' valid under the old rules.

After a generalized softfork three chains may exist:

* The new chain B3,B4,B5,... valid under the new rules only.
* The mapped chain f(B3),f(B4),f(B5),... valid under the old rules.
* The old chain B3',B4',B5',... valid under the old rules only.

E.g.


This is "generalized" softfork since defining f(B)=B (identity function)
reduces to the standard softfork case above.

As with standard softforks, if the majority of the hashpower follow the new
rules then the old chain B3',B4',B5',... is doomed to be orphaned:


Example:

Segregated Witness can be thought of as an example of a generalized softfork.
Here the new block format consists of the combined old block and witness data.
The function f() simply strips the witness data to reveal a valid block under
the old rules:


An Arbitrary Block-size Increase Via a Generalized Softfork

Segregated Witness only allows for a modest effective blocksize increase
(although there can be other motivations for SW, but that is off-topic).

Instead we engineer a generalized softfork that allows an arbitrarily increase
of the blocksize limit.  The proposal consists of two parts: (a) new rules for
valid blocks, and (b) a transformation function f().

The new block rules are very similar to the old block rules but with some
small changes.  In summary the changes are:

* The MAX_BLOCK_SIZE limit is raised to some new limit
  (e.g. 8Mb, BIP101, 2-4-8, BIP202, etc., or some other limit)
* The MerkleRoot field in the header has been re-interpreted.
* The CoinBaseTx must obey some additional new rules.

As with old blocks, a block under the new rules consists of a block header
followed by a vector of transactions [CoinBaseTx, Tx1, .., Txn], i.e.


The block header format is the same as under the old rules defined as follows:


Under the old rules MerkleRoot is the Merkle root of the hashes of all
transactions included in the block, i.e.


Under the new rules we instead define:


That is, under the new rules, MerkleRoot is the Merkle root of a singleton
vector containing the CoinBaseTx hash only.

In order to preserve the security properties of Bitcoin we additionally
require that the CoinBaseTx somehow encodes the Merkle root of the remaining
transactions [Tx1, .., Txn].  For example, this could be achieved by requiring
a mandatory OP_RETURN output that encodes this information, e.g.


Alternatively the Merkle root could be encoded in the coinbase itself.  This
ensures that new transactions cannot be added/deleted from the block without
altering the MerkleRoot field in the header.

Aside from these changes and the increased MAX_BLOCK_SIZE, the new block must
obey all the rules of the old block format, e.g. valid PoW, have valid block
reward, contain valid transactions, etc., etc.

In order to be a generalized softfork we also need to define a mapping f()
from valid new blocks to valid blocks under the old rules.  We can define this
as follows:


That is, function f() simply truncates the block so that it contains the
coinbase transaction only.  After truncation, the MerkleRoot field of the
block header is valid under the old rules.

The proposed new rules combined with the transformation f() comprise a
generalized softfork.  After the fork a new chain B3,B4,B5,... will be
generated under the new rules defined above, including an increased blocksize
limit.  This new chain can be mapped to a valid chain f(B3),f(B4),f(B5),...
under the old rules.  Assuming that >50% of the hashpower has adopted the new
rules, the mapped chain will orphan any competing chain under the old rules,
just like a standard softfork.

An interesting consequence of this design is that, since all mapped blocks are
empty, old clients will never see transactions confirming.  This is be a
strong incentive for users to update their clients.

Conclusion

Conventional wisdoms suggests that increasing the blocksize limit requires a
hardfork.  We show that it can instead be achieved using a generalized
softfork.  Like with a standard softfork, a generalized softfork merely
requires a majority (>50%) of hash power rather than global consensus.
Experience has shown that the former is significantly easier to achieve.

Future Work:

Investigate other kinds of hardforks that can instead be implemented as
generalized softforks, and the security implications of such...

7943a2934d0be2f96589fdef2b2e00a2a7d8c3b782546bb37625d1669accb9b1
72f018588572ca2786168cb531d10e79b81b86d3fada92298225a0f950eed3a5

This looks interesting to me, it basically extracts the weird part of segwit, but minus the signature-specific details.

Kanzure points out this looks possibly similar to "aux blocks" https://bitcointalk.org/index.php?topic=283746.0
Do you see a difference?

I see a few problems with those proposal

1) You are proposing to fork the blockchain without consensus. This in itself is a problem, we don't want to fork without consensus (even with a soft fork) because there is still the possibility of the old chain surviving, and that can lead to some major problems.

2) Your generalized soft fork idea breaks one of the basic principles of Bitcoin, storing all of the transaction history in the blockchain. It is not possible to know what the transaction data is if you use your idea to transform blocks under new rules to blocks under old rules. It isn't backwards compatible so it isn't a soft fork. If I ran a client with the old rules, then I would not be able to know what the transactions were in that block without having to download the fork. That is completely pointless.

A generalized softfork has the same minimum requirements of a standard softfork, i.e. >50% of mining hashpower.  In practice it would wiser to shoot for something beyond the bare minimum, such as the standard 75% or 95% of hash power as with other forks.  The point is that this proposal is no different than other softforks in terms of requirements.


This proposal is an alternative to hardforks which completely replace the blockchain with a new version under different rules.  My proposal does exactly the same thing, but does so essentially as a softfork.  The new chain keeps all the the transaction history on the chain under the new rules.

I have seen a similar proposal labeled as "extended blocks".  It is also possible to view these ideas as a "generalized softforks" if one so desires.

The main difference is that there are no different types of blocks.  In fact, the new block format is essentially the same as the old block format sans for a new blocksize limit.

ZoomT, I would assert that your definition of a generalized softfork is incomplete. You want to make sure that old nodes can:

+ send transactions
+ receive transactions
+ receive transactions from the extended blocks

Though the definition is incomplete, though it only affects the implementation a little. When you add coins to the extended blocks, they get stuck in 'anyonecanspend' outputs as recognized by the old nodes. In your transformation, you add some specification that money sent back to old nodes be pulled out of the most sensible set of the anyonecanspend outputs (for some strict definition of sensible). You would also need to add a rule that the new blockchain can't spend so many outputs to old addresses that the transformation results in an old block greater than 1mb. Finally, I think it's worth highlighting that the transformation is somehow performed in a way that the POW is still valid on the old chain (ie using the coinbase txn to store a hash), though I think that this is implicitly clear.


I would refute both of those problems, especially if you maintain the three properties listed above. The old chain surviving is only problematic for miners that do not upgrade. This is the same as a 51% attack, and in fact all softforks are basically 51% attacks on non-upgraded miners, it's just that they are not typically deployed until much more than 51% hashpower is reached. Additionally, all nodes still have a full transaction history that they recognize as valid, even the old nodes. Their history will also contain a bunch of 'anyonecanspend' transactions that miner's for some reason aren't mining, but they can do full trustless verification of the coins that they own, and can spend them if they are willing to pay the fee.

Hard forks don't need consensus, we just want consensus to actually do one. It only requires the same amount of hashpower to execute. The difference between soft forks and hard forks is not the hashpower required, it is whether the new rules are backwards compatible (soft fork) or not (hard fork)

Nothing about your proposal makes it backwards compatible (a soft fork). It is not possible to still use Bitcoin if you are running an old version of software, as is the case with hard forks. The point of a soft fork is that I don't need to upgrade but I can still send transactions, see confirmations, see other transactions, and consider other transactions and blocks as still valid even if they run under the new rules. Your proposal makes it so that I can't see confirmations (so I can't spend the Bitcoin because the transaction will remain in the mempool and then disappear when no one continues to relay it after it has been confirmed), and it won't consider new blocks containing actual block data as valid since they are too big. It is still a hard fork.

Actually, using this proposal, it looks like the only way the old chain could survive is if the users that don't want the new chain create a hard fork to reject the blocks created by those that do want the new chain.


If I'm understanding this proposal correctly, it could be described as a hard fork with a simultaneous majority hash power attack on the old fork.
 
The OP has indicated a method by which the hash power that hashes the blocks on the new format side of the hard fork can simultaneously hash empty blocks on the old format side.

That way, once the hashing participants on the new format side have enough hash power, the old format side will no longer be able to confirm any transactions ever again.  They will be forced to either hard fork to keep their old format, or they will have to join the new format.

So it really would just be discouraging any use of the old chain and force people to switch to the new one.

The way I understood it was that it is supposed to be that people on the old chain can still use Bitcoin normally. As I said in an earlier post, this is not possible unless somehow all of the UTXOs in the new block are presented in the old block format. But I think in order to do that, a software upgrade is still needed so that the client actually knows what to do with that data because it cannot be in the same format that is currently used. Either way, I don't think the idea that this is a soft fork works, it is still a hard fork.

No, a hardfork requires everyone to upgrade to a new client.  After a hardfork old clients will outright reject the new chain.

If that does not happen, and if a significant portion of the hashpower refuses to upgrade, then Bitcoin will split into two completing cryptocurrenies that can co-exist forever.  This is the worst-case scenario.

Which is why this is a hard fork.  Old clients will outright reject the new chain with the larger blocks.

They will accept the majority hash power attacks against their own chain, but that is a separate chain with separate rules.  They would have to deal with this the same way they would deal with any majority hash power attack that refuses to confirm transactions.


Correct, and this is what your proposal does.

If a "significant portion of the hashpower refuses to upgrade" to your proposal, then all they have to do is had code in a rejection of one of your blocks and "Bitcoin will split into two completing cryptocurrenies that can co-exist forever".

I'm curious, how does your intended solution expect to deal with the situation where the old format has a longer chain?  This would mean that the old network would reject the transformed chain, but how would your system be aware that the transformed chain was rejected and what would it do with all those transactions that had been confirmed in it's big block chain?

Technically this can be done via a standard softfork.

The same is true for any softfork.  For example, if some portion of the hashpower were passionately against the OP_CLOCKTIMEVERIFY fork, they could create a competing softfork that makes any block including a OP_CLOCKTIMEVERIFY transaction invalid.  If this occurred, then Bitcoin will split into two competing cryptocurrencies (both softforks of the original chain).


Yes, in the same way a standard softfork uses the majority hash power to attack any chain under the old rules.

No it cannot. Clients running the old rules are not able to use Bitcoin as they have normally. They are unable to use Bitcoin at all because they cannot see the transaction data in the blockchain and thus won't know about confirmations. This proposal is not backwards compatible, and backwards compatibility is the key point of a soft fork.

They could, but that is also why we wait for a supermajority of miners say that they are willing to enforce the new rules before actually enforcing.

No it does not. A soft fork simply lets the old chain die out if people are still mining there. They aren't actively attacking the old chain as your proposal suggests. A normal soft fork would still allow miners to mine on and extend the old chain. This causes problems, see the July 4th forking incident for an example of this.

This is why I called it a "generalized" softfork, since it has properties of softforks but has the power of a hardfork (in this case).  It is sort of an "in-between" kind of fork.


I do not disagree but the same it true for all softforks (e.g. OP_CLOCKTIMEVERIFY).


Since this is a (generalized) softfork, there is a very strong incentive for miners to upgrade to the new rules (orphaned blocks).  So this should not happen provided the majority hash power are on board when the fork occurs.

However, if it *did* happen, then this is a disaster no matter how the fork is designed.  Either the old (longer) chain is rejected as invalid, or there is a very large reorg undoing many multi-confirmation transactions.

A softfork is the same as a 51% attack against the old rules.  The generalized softfork is no different.  I think DannyHamilton understands this.

How does this have properties of a soft fork? It is missing the key point of a soft fork, old clients are still able to use Bitcoin after the fork. This is a hard fork since I cannot run current software after this fork and still be able to send and receive transactions and use it as I do now.

There is already a strong incentive to upgrade with a normal soft fork. Miner's blocks will be orphaned if they don't upgrade.

And such forks have happened before, see the July 4th forking incident: https://bitcoin.org/en/alert/2015-07-04-spv-mining

I think this is different. What you propose is simultaneously mine on two different blockchains, the old and new one. This is a 51% attack on the old chain as it just produces a bunch of empty blocks thus preventing confirmations. In a normal soft fork, this doesn't happen and other miners can extend that old chain, they just don't have any incentive to do so. Again, see the July 4th forking incident for an example of miners extending the old chain. Your proposal wouldn't allow that to happen, but old clients would only be able to use the old chain, so it is a hard fork.

If Danny understood it, then I invite him to explain it to me since he is much better at explaining than you are.

I think your proposal sounds similar to "aux" blocks from above.

There is nothing wrong with these kinds of proposals except that they are arguably more complex for users.

My proposal is more like a simple blocksize hardfork, except can be deployed like a softfork.


Quoted.

Who lost money in July 4th and 5th fork? I guess they will hate the BIP66 roll out which caused the fork, similar to they hate Gavin's 2013 fork  ;D

For the 4th July fork the old chain was nothing but empty blocks (a bit like the above proposal).  So they cannot lose money unless they were accepting zero-conf txs.

The miners lost money. The miners who mined on the invalid chain lost money in the form of wasted time and lost block rewards.

I was also thinking it may be possible to redesign the idea such that the old chain is not empty.  This could be achieved by introducing a new transaction version number, e.g. version=2, and all version 2 transactions cannot appear in the legacy transformed f(B) blocks.  However, old version=1 transactions can appear subject to the 1MB blocksize limit.  However, once a coin has been used in a version=2 transaction, it and all child transactions can never again appear in the legacy block.

So:
And the MerkleRoot in the header corresponds to the transformed block.

This is more like extension/aux blocks, except:

* There is no need to explicitly move coins to the extension.  Instead they are moved "automatically" when an new client issues a version=2 transaction.
* Coins can never be moved back out from the extension.  There is no need to do this anyway, since there is no aim to support old clients indefinitely (remember this is a proposed alternative to a hardfork, which is immediately incompatible with old clients anyway).

This means old clients can still send coins, but may not be able to see received coins unless they upgrade.

An implementation of BIP102 as a softfork using this idea:

https://github.com/ZoomT/bitcoin/tree/2015_2mb_blocksize
https://github.com/jgarzik/bitcoin/compare/2015_2mb_blocksize...ZoomT:2015_2mb_blocksize?diff=split&name=2015_2mb_blocksize

generalized soft-forks:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012073.html

bip102 forced soft-fork:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012153.html

auxiliary blocks and evil soft-forks or forced soft-forks:
https://bitcointalk.org/index.php?topic=283746.0
https://bitcointalk.org/index.php?topic=874313.0

soft-fork block size increase using extension blocks:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-May/008356.html

extension blocks were also discussed in this interview:
http://diyhpl.us/wiki/transcripts/bitcoin-sidechains-unchained-epicenter-adam3us-gmaxwell/
.... which includes something very similar to the idea of a soft-hard fork (something I was informed about on 2015-06-13 ..... not sure if I should mention this?)

some discussion from today re: origin of the term evil fork, evil soft-fork, forced soft-fork:
https://www.reddit.com/r/Bitcoin/comments/3yrsxt/bitcoindev_an_implementation_of_bip102_as_a/cyg2g7q

some much older discussion about extension blocks and sidechains:
http://gnusha.org/bitcoin-wizards/2015-01-01.log

some discussion about "generalized soft-forks" and extension blocks and evil soft-forks:
http://gnusha.org/bitcoin-wizards/2015-12-20.log

segwit soft-fork makes use of a similar idea:
http://diyhpl.us/wiki/transcripts/scalingbitcoin/hong-kong/segregated-witness-and-its-impact-on-scalability/

discussion about evil forks and evil soft-forks and extension blocks:
http://gnusha.org/bitcoin-wizards/2015-12-30.log

fork types:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012172.html

these links are also mentioned here:
https://www.reddit.com/r/Bitcoin/comments/3yrsxt/bitcoindev_an_implementation_of_bip102_as_a/cyg7mdr

and also mentioned here:
http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/012173.html

This seems to be the same idea, but using an "extension record" approach instead of a separate blokchain:

Original comment by /u/seweso on reddit (https://np.reddit.com/r/btc/comments/40arwh/you_should_realise_that_anything_can_be_changed/)

My understanding of the proposal (https://www.reddit.com/r/Buttcoin/comments/40ayz8/perplexed_butters_realize_that_the_same_extension/cytfvfm)

Comparison to the hard fork approach (https://www.reddit.com/r/Buttcoin/comments/40ayz8/perplexed_butters_realize_that_the_same_extension/cytlb08)

Slightly off-topic...
Isn't the Note concerning 4 July 2015 problem (https://bitcoin.org/en/alert/2015-07-04-spv-mining) a bit obsolete?

"of course" you dont need a hard fork. "of course" everybody wants an increase.
but it is not blockstream approved. end.  :-\

This is a great proposal. This gets over the problem that even if consensus (say 75%) is reached how do we ensure transactions are not lost before everyone (100%) has upgraded or risk being on a losing chain forever.

Under this proposal we no longer need as much as 75%, 55% would probably do and the rest can safely upgrade with all transactions ending up on the winning chain because, by the time we reach 55%, we will know which chain will win.  

I propose the old chain actually keeps a record of legacy transactions, and that the new chain effectively becomes a sidechain (of sorts).

Do two-way peg using spend-all coins (like SW) in the original chain. So new transactions can actually send coins to legacy addresses.

I assume SW is not yet implemented, so that's why I simply copy some SW tricks.

1. A transaction is send to a spend-all address (legacy chain), segregated witness data is added to the segregated block (like SW).
1. Transactions to/from segregated addresses only go into the segregated chain
1. When creating a transactions to legacy addresses a transaction is created from any spend-all transaction (legacy chain) and coins are destroyed (on the segregated chain)

Miners would then simply check whether all spend-all transactions in the legacy actually came from "destroyed" coins in the new chain.

As mentioned on reddit, this idea seems very similar to "extension/auxiliary blocks", although maybe there are some differences (?).

There is nothing wrong with this kind of idea other than it adds more complexity for the sake of better backwards compatibility (i.e. it's a trade-off).  Personally I prefer the simpler solution as backwards compatibility is only a short-term problem, whereas any complexity it introduces can remain forever.

I want to say softfork is best rather than hardfork, as the bitcoin node base has been huge and expanding rapidly. It will help quick updation in no time.

Hi ZoomT, I attempted to provide a non-technical explanation of the principal for how a generalised softfork could be achieved, is this a fair representation of your basic principle?

http://pondpolitics.com/2016/01/hard-vs-soft-fork-is-there-a-third-way-to-increase-the-bitcoin-block-size/

Thanks for writing this.  In terms of analogies, I was thinking something like this:

Bitcoin = flower picking game
Example Softfork = everyone must pick only yellow flowers
Example Hardfork = everyone must pick mushrooms instead of flowers
Example "Generalized" Softfork = everyone must pick flowering mushrooms

The example softfork does not violate the original rules of the game, i.e. everyone is still picking flowers, albeit yellow flowers.

The example hardfork changes the rules to a new game (mushroom picking).

The example generalized softfork effectively achieves the same thing as a hardfork (upgrading everyone to mushroom picking) but as a softfork (technically everyone is still picking flowers as well).

I do not think there is such thing as a flowering mushroom however.