Sidechains, Treechains, the TL;DR, welcome to join discussion.

[provenceday]

Sidechains, Treechains, the TL;DR


（i am not the author of this article, here is the source: http://blog.greenaddress.it/2014/06/13/sidechains-treechains-the-tldr/）





This document is aimed at technical readers and is simply a brief explanation of sidechains and treechains as far as I understand them, based on public information.  Both are obviously still in very preliminary development, but this document is just to introduce the broad concepts, and their consequences. Some people have been asking for something like this, might as well see if this helps.

With GHash is getting nearly 50% of hashing power of the network, this discussion is more timely than ever.

I’ll start with sidechains, since treechains are essentially a specific form of sidechains.

Sidechains:

In the most general, sidechains will use “SPV Proofs” to send satoshis from the regular Bitcoin chain to the sidechain, and allows the sidechain to eventually send the satoshis back to the main chain once the owner of said coin is finished utilizing the sidechain. While in the sidechain, the main chain knows nothing of what’s happening to the coin, the sidechain is the one tracking who owns what at what time.

The side chain can basically have any rules it likes for what a valid block is, block times, etc. Typically the idea is that these chains will be merge mined with the Bitcoin network, to ensure that a reasonable amount of hashing power is protecting the sidechain network from DoS, and outright theft of coins by miners which is possible due to the limitations of the SPV proofs. It’s important to note however, that it has been suggested that the outright theft of coins by miners may be protected against using zk-SNARKs.(https://eprint.iacr.org/2013/507.pdf)

The pros of sidechains appear to be:


1 You don’t need permission to start a new chain with new validation rules, block times, whatever. You could fairly trivially add Zerocash, Ethereum rules, and still have them pegged in satoshis. Also would be a great way to test out new opcodes/communication protocols for the base protocol and codebase.

2 The sidechains would be backed by the hashing power of the Bitcoin network, so given certain conditions(detailed below) it can’t be trivially attacked.


The cons are as far as we know(not counting new zk-SNARK moon math that hasn’t been given to the public):


1 Merge mining also means two things: There is no inherent block reward. Security will most likely be only be from transaction fees. more importantly, you need to convince the large pools to manually activate the merged mining of these chains, otherwise a 51% attack is essentially free. You also have to trust the pools aren’t faking downtime, while secretly mining the chain.

2 Long-term it can contribute to centralization of mining, just in the same way that increasing the block size would. It would be optional to mine these sidechains yes, but if it becomes a sizeable fraction of transaction fees, the economics work in the favor of more centralization.

3 Sending satoshis back and forth  between chains will take days, to ensure that satoshis aren’t being stolen by miners, again due to the aforementioned SPV proofs, which is something that simply can’t happen in vanilla Bitcoin. Most going back and forth will be done using atomic swaps in between users to reduce this waiting period.


Treechains:

I think of treechains as tighter-coupled sidechains. The difference in chain structure is larger than between sidechains and the vanilla Bitcoin protocol, so I’m tackling them in broad brush-strokes.

1 Miners are not required to validate blocks, outside of the PoW difficulty being low enough, and being a proper hash of the block+previous block. If the block header looks legit, miners can start to build on top of this.


2 Starting from the main Bitcoin chain, each chain will have a left and right descendant chain. This builds a binary tree of chains, hence “treechains”. Each chain level has 2^(numlevels-1) chains, doubling the number of the previous level. Each difficulty threshold is also halved. Based on the hash of the transactions, they can only be mined in in specific paths of the tree structure(starting from the first bit of the hash from the root of the tree, ‘0’ means left subtree, ‘1’ means right). Each time satoshis are spent, it will get sent to another chain in the same level based on the previous transaction’s hash(ignoring up/down movement for clarity).
In addition, each path is merge mined, allowing miners to mine one and only one path of the tree using the same hashing work. So for example, 3 layers down, there should on average only be an 8th of the total transactions on any specific chain, as well as only an 8th of the total mining power, resulting in roughly the same block time as higher chains!

3 The chains are linked together more strongly than sidechains to enforce a total ordering of transactions. Every time a miner gets a PoW high enough for a certain level, it “links” that block with all the blocks being mined below together. This enforces the total ordering we want. Transactions on let’s say level 16 will have a higher chance of getting orphaned, but eventually once they “percolate” up to the main chain, they are just as secure as the main chain. The linking also determines when you can spend your satoshis, meaning lower chains will take longer to spend the same outputs again compared to higher chains. To spend your satoshis from chain A to chain B at level C, the previously mined transaction’s block in A must be linked to B’s nearest common ancestor chain, with the only valid paths being forward/up the chains, not backwards.

4 Last important thing to note about the tree structure: Parent chain always wins. If the child chain is in conflict with the parent chain(the links are inconsistent, making total ordering inconsistent), those blocks child blocks are orphaned. Therefore, re-organizations at higher chains can cause reorganizations at lower chains, but not vice versa.


And their consequences/caveats:

1 Since miners aren’t required to validate anything outside of basic PoW, this breaks the need to beg miners for protocol changes. Granted, there will be a base BTC layer that allows things like “miner gets block reward” and “pay .0001 BTC to miner for transaction fee” to incentivize the mining, but outside of this, it allows fairly arbitrary protocols. One could even imagine paying a miner colored coins to get it included in a block, if the miner wanted equity! One thing this can’t do versus sidechains is initialize chains with arbitrary block times. However you might be able to get away with much faster block times than vanilla Bitcoin due to #2. Overall, this will let innovation at the edges happen, without having to agree on everything with Core Devs, or mining pools, or industry, etc. SPV clients won’t be possible, at least in their current form, due to SPV’s assumption that mined blocks are validated by the miners.

2 Proving who owns what when will be more complicated for the client, as they can’t assume miners are validating a certain protocol. Clients will have to hold data outside of their private keys, proving to the payee that these coins exist and control them. This will be more complicated than our SPV clients we have today, but will make running a node with “full node” security tractable, as you don’t care what the contents of most blocks are, just the blocks that prove to you that you own the satoshis you own(a small sample of blocks compared to the whole tree of chains). These proofs will be “compact”, although it remains to be seen how much more compact than linear in block sizes we can get(insert zk-SNARK moon math for sublinear performance?).

3 Combining with consequences from #1, miners will be able to mine as little or as much as they like, with only paying attention to block headers, and block payloads that again, prove to him that they’re actually being paid to mine by fees. A miner could simply keep track of all headers in the treechains, which is trivial, and solo mine 16 levels down, where their variance is 2^(-16) less than the vanilla blockchain mining, due to the sparsity of miners that far down in a branch. If a user is willing to wait a while for the ability to re-spend their outputs, they can approach a solo miner, pay a smaller fee than usual, and wait for the block to get linked higher in the tree.This opens up a true marketplace for fees, as well as allows small pools/solo miners to make a real difference when it comes to block creation. Lastly, this system appears to scale to an infinite amount of transactions, without hurting decentralization.

4 The linking scheme ultimately means that orphan rates will be higher at lower levels, and re-spending outputs will take longer, and will be based on where the next transaction will end up in the tree structure. However, for your coffee money, it enables you to get in a block, and for the merchant to not worry too much that you’ll try and 51% attack 5 levels down as it won’t make economic sense.

In summary(TL;DR’s TL;DR):


A Sidechain, at its most general, is a loosely coupled chain that, in general, uses merged mining to protect the network. These chains are “backed” by BTC from the Bitcoin network, rather than minting their own coin and diluting scarcity. There are some questions about security guarantees versus the Bitcoin network.


A Treechain is a structure of more-tightly coupled sidechains. This structure, in theory, allows miners to mine at arbitrary variance without pooling, scaling of the system far beyond 7tps without asking permission, and other innovation at the edges, all with the same protections of the main Bitcoin network. With the huge caveat that the idea is still half-baked, has no known SPV client support, and is much more complicated than a vanilla blockchain.


Both ideas are interesting ways of tackling some of the important problems that all cryptocurrencies face. We should know more about the actual implementation of sidechains within 3 months, as the company Blockstream will be releasing a white paper and source code. Many of these ideas that aren’t published will be directly applicable to treechains, as they are kin in many ways, including how they will be rolled out initially.

I’m personally biased towards treechains in that I believe the de-coupling of miners and policy is a huge step forward, even just for new fancy opcodes without permission. It may also enable us to be free of begging MegaPool#9 not to 51% attack us, which is already happening. I for one would like to solo-mine on a USB ASIC!

Unfortunately due to its complexity and fundamental difference with Bitcoin proper, it will almost certainly take more time to flesh out and convince others that radical steps need to be taken to keep cryptocurrency decentralized. I look forward to its development.

If you have time on your hands to check out more of the details of treechains,
here is Peter Todd’s initial writeup of many of the ideas: http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg04388.html


As well as the Let’s Talk Bitcoin podcast where he goes into much of this detail: here:http://letstalkbitcoin.com/ltb104-tree-chains-with-peter-todd/
(thanks to /u/_Mr_e)

Hope someone finds this helpful,

Greg Sanders
Contributor to Bitcoin.org’s Bitcoin Developer Guide
gsanders87@gmail.com


Peter Todd (https://twitter.com/petertoddbtc)sent us the following:

FWIW there are some concerns raised re: how tree chains handles data
loss at the lowest levels; I’m not sure yet that those concerns can be
resolved. Also Adam Back raised some potential issues re: incentives in
some edge cases. Of course, you did quite correctly describe the idea as
half baked.

[provenceday]

welcome to join discussion.

[provenceday]

More details on treechains by pter todd

1 http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg04388.html

2 http://www.mail-archive.com/bitcoin-development%40lists.sourceforge.net/msg03307.html

[neuroMode]

Interesting. I'll be linking these on myriadplatform.org.

Thanks for making me aware of them!

[bitwho]

great topic!!

[provenceday]

Sidechains or Treechains are not really a good idea because they are Centralizing the idea of a decentral system

no, it's not, check my post:

anybody want to know more details of TreeChains?

here is a article by peter todd:

http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg04388.html


Tree-chains preliminary summary

Peter Todd Tue, 25 Mar 2014 05:39:15 -0700

On Sat, Mar 22, 2014 at 12:43:34PM -0700, Mark Friedenbach wrote:
> Btw, any chance we could get a summary description of tree-chains
> posted to bitcoin-development?
sure

1
Introduction
============

Bitcoin doesn't scale. There's a lot of issues at hand here, but the
most fundemental of them is that to create a block you need to update
the state of the UTXO set, and the way Bitcoin is designed means that
updating that state requires bandwidth equal to all the transaction
volume to keep up with the changes to what set. Long story short, we get
O(n^2) scaling, which is just plain infeasible.

So let's split up the transaction volume so every individual miner only
needs to keep up with some portion. In a rough sense that's what
alt-coins do - all the tipping microtransactions on Doge never have to
hit the Bitcoin blockchain for instance, reducing pressure on the
latter. But moving value between chains is inconvenient; right now
moving value requires trusted third parties. Two-way atomic chain
transfers does help here, but as recent discussions on the topic showed
there's all sorts of edge cases with reorganizations that are tricky to
handle; at worst they could lead to inflation.

So what's the underlying issue there? The chains are too independent.
Even with merge-mining there's no real link between one chain and
another with regard to the order of transactions. Secondly merge-mining
suffers from 51% attacks if the chain being merge-mined doesn't have a
majority of total hashing power... which kinda defeats the point if
we're worried about miner scalability.

2 Blocks and the TXO set as a binary radix tree
=============================================

So how can we do better? Start with the "big picture" idea and take the
linear blockchain and turn it into a tree:


Obviously if we could somehow split up the UTXO set such that individual
miners/full nodes only had to deal with subsets of this tree we could
significantly reduce the bandwidth that any one miner would need to
process. Every transaction output would get a unique identifier, say
txoutid=H(txout) and we put those outputs in blocks appropriately.

We can't just wave a magic wand and say that every block has the above
structure and all miners co-ordinate to generate all blocks in one go.
Instead we'll do something akin to merge mining. Start with a linear
blockchain with ten blocks. Arrows indicate hashing:

    a0 ⇽ a1 ⇽ a2 ⇽ a3 ⇽ a4 ⇽ a5 ⇽ a6 ⇽ a7 ⇽ a8 ⇽ a9

The following data structure could be the block header in this scheme.
We'll simplify things a bit and make up our own; obviously with some
more effort the standard Satoshi structures can be used too:

    struct BlockHeader:
        uint256 prevBlockHash
        uint256 blockContentsHash
        uint256 target
        uint256 nonce
        uint time

For now we'll say this is a pure-proof-of-publication chain, so our
block contents are very simple:

    struct BlockContents:
        uint256 merkleRoot

As usual the PoW is valid if H(blockHeader) < blockHeader.target. Every
block creates new txouts, and the union of all such txouts is the txout
set. As shown previously(1) this basic proof-of-publication
functionality is sufficient to build a crypto-currency even without
actually validating the contents of the so-called transaction outputs.

The scalability of this sucks, so let's add two more chains below the
root to start forming a tree. For fairness we'll only allow miners to
either mine a, a+b, or a+c; attempting to mine a block with both the b
and c chains simultaneously is not allowed.

    struct BlockContents:
        uint256 childBlockHash # may be null
        bool childSide # left or right
        uint256 merkleRoot

Furthermore we shard the TXO space by defining txoid = H(txout) and
allowing any txout in chain a, and only txouts with LSB=0 in b, LSB=1 in
c; the beginning of a binary radix tree. With some variance thrown in we
get the following:




We now have three different versions of the TXO set: ∑a, ∑a + ∑b, and
∑a+∑c. Each of these versions is consistent in that for a given txoutid
prefix we can achieve consensus over the contents of the TXO set. Of
course, this definition is recursive:




Unicode unfortunately lacks 3D box drawing at present, so I've only
shown left-sided child chains.


3 Herding the child-chains
========================



If all we were doing was publishing data, this would suffice. But what
if we want to syncronize our actions? For instance, we may want a new
txout to only be published in one chain if the corresponding txout in
another is marked spent. What we want is a reasonable rule for
child-chains to be invalidated when their parents are invalidated so as
to co-ordinate actions across distant child chains by relying on the
existance of their parents.

We start by removing the per-chain difficulties, leaving only a single
master proof-of-work target. Solutions less than target itself are
considered valid in the root chain, less than the target << 1 in the
root's children, << 2 in the children's children etc. In children that
means the header no longer contains a time, nonce, or target; the values
in the root block header are used instead:

    struct ChildBlockHeader:
        uint256 prevChildBlockHash
        uint256 blockContentsHash

For a given chain we always choose the one with the most total work. But
to get our ordering primitive we'll add a second, somewhat brutal, rule:
Parent always wins.

We achieve this moving the child block header into the parent block
itself:

    struct BlockContents:
       ChildBlockHeader childHeader # may be null (zeroed out)
       bool childSide # left or right
       bytes txout
Let's look at how this works. We start with a parent and a child chain:




to



This behavior is easier to understand if you say instead that the node
learned about block b2', which had more total work than b2 as the sum
total of work done in the parent chain in blocks specifying the that
particular child chain is considered before comparing the total work
done in only the child chain.

It's important to remember that the parent blockchain has and validates
both childrens' block headers; it is not possible to mine a block with
an invalid secret of child headers. For instance with the following:



I can't mine block a5 that says following b2 is b2' in an attempt to
kill off b2 through b7.

4 Token transfer with tree-chains
===============================

How can we make use of this? Lets start with a simple discrete token
transfer system. Transactions are simply:

    struct Transaction:
        uint256 prevTxHash
        script prevPubKey
        script scriptSig
        uint256 scriptPubKeyHash

We'll say transactions go in the tree-chain according to their
prevTxHash, with the depth in the tree equal to the depth of the
previous output. This means that you can prove an output was created by
the existance of that transaction in the block with prefix matching
H(tx.prevTxHash), and you can prove the transaction output is unspent by
the non-existance of a transaction in the block with prefix matching
H(tx).

With our above re-organization rule everything is consistent too: if
block b_i contains tx1, then the corresponding block c_j can contain a
valid tx2 spending tx1 provided that c_j depends on a_p and there is a
path from a_p to b_(i+k). Here's an example, starting with tx1 in c2:



Now that a3 exists, block c2 can only be killed if a3 is, which would
also kill b3 and thus destroy tx2.


5 Proving transaction output validity in a token transfer system
==============================================================
How cheap is it to prove the entire history of a token is valid from
genesis?  Perhaps surprisingly, without any cryptographic moon-math the
cost is only linear!

Remember that a transaction in a given chain has committed to the chain
that it can be spent in. If Alice is to prove to Bob that the output she
gave him is valid, she simply needs to prove that for every transaction
in the histroy of the token the token was created, remained unspent,
then finally was spent. Proving a token remained unspent between blocks
b_n and b_m is trivially possible in linear size. Once the token is
spent nothing about blocks beyond b_m is required. Even if miners do not
validate transactions at all the proof size remains linear provided
blocks themselves have a maximum size - at worst the proof contains some
invalid transactions that can be shown to be false spends.

While certainly inconvenient, it is interesting how such a simple system
appears to in theory scale to unlimited numbers of transactions and with
an appropriate exchange rate move unlimited amounts of value. A possible
model would be for the the tokens themselves to have power of two
values, and be split and combined as required.

6 The lost data problem
=====================

There is however a catch: What happens when blocks get lost? Parent
blocks only contain their childrens' headers, not the block contents.
At some point the difficulty of producing a block will drop sufficiently
for malicious or accidental data loss to be possible. With the "parent
chain wins" rule it must be possible to recover from that event for
mining on the child to continue.

Concretely, suppose you have tx1 in block c2, which can be spent on
chain b. The contents of chain a is known to you, but the full contents
of chain b are unavailable:



The proof of now shows that while a3 and a4 has b-side blocks, by the
time you reach b6 those two lost blocks were in the minority. Of course
a real system needs to be careful that mining blocks and then discarding
them isn't a profitably way to create coins out of thin air - ratios
well in excess of 1:1 are likely to be required.

7 Challenge-response resolution
=============================

Another idea is to say if the parent blockchain's contents are known we
can insert a challenge into it specifying that a particular child block
be published verbatim in the parent. Once the challenge is published
further parent blocks may not reference that children on that side until
either the desired block is re-republished or some timeout is reached.
If the timeout is reached, mining backtracks to some previously known
child specified in the challenge. In the typical case the block is known
to a majority of miners, and is published, essentially allowing new
miners to force the existing ones to "cough up" blocks they aren't
publishing and allow the new ones to continue mining. (obviously some
care needs to be taken with regard to incentives here)

While an attractive idea, this is our first foray into moon math.
Suppose such a challenge was issued in block a2, asking for the contents
of b1 to be published. Meanwhile tx1 is created in block c3, and can
only be spent on a b-side chain:



A proof of tx2 as valid payment would entirely miss fact that the
challenge was published and thus not know that b1' was invalid. While
I'm sure the reader can come up with all kinds of complex and fragile
way of proving fraud to cause chain a to be somehow re-organized, what
we really want is some sub-linear proof of honest computation.  Without
getting into details, this is probably possible via some flavor of
sub-linear moon-math proof-of-execution. But this paper is too long
already to start getting snarky.


8 Beyond token transfer systems
=============================

We can extend our simple one txin, one txout token transfer transactions
with merkle (sum) trees. Here's a rough sketch of the concept:



Where previously a transaction committed to a specific transaction
output, we can make our transactions commit to a merkle-sum-tree of
transaction outputs. To then redeem a transaction output you prove that
enough prior outputs were spend to add up to the new output's value. The
entire process can happen incrementally without any specific
co-operation between miners on different parts of the chain, and inputs
and outputs can come from any depth in the tree provided that care is
taken to ensure that reorganization is not profitable.

Like the token transfer system proving a given output is valid has cost
linear with history. However we can improve on that using
non-interactive proof techniques. For instance in the linear token
transfer example the history only needs to be proven to a point where
the transaction fees are higher than the value of the output. (easiest
where the work required to spend a txout of a given value is well
defined) A similar approach can be easily taken with the
directed-acyclic-graph of mutliple-input-output transactions. Secondly
non-interactive proof techniques can also be used, again out of the
scope of this already long preliminary paper.

1) "Disentangling Crypto-Coin Mining: Timestamping,
   Proof-of-Publication, and Validation",
http://www.mail-archive.com/bitcoin-development%40lists.sourceforge.net/msg03307.html



 

[provenceday]

if you have any question, just leave a message on this thread.
but it seems few people care about the real edge technology.  

[TrangLee]

This does go against the concept of decentralized currency. In this form the side chains and tree chains are tied to dear old mom. Also security may be a concern. If dear old mom suffers a catastropic failure the side and tree chains go down with it. it's like having your eggs in one basket. If it drops you loose it all.

[riclas]

did you even read the intro to tree chains? It's just a more scalable way of building the blockchain, simplifying merged mining is just a bonus.

[instagibbs]

The GA.it writeup is mine, nice to see it being useful again.

FWIW there was a great discussion on twitter yesterday/today on the subject between Adam Back, Peter Todd and Andrew Miller:

https://twitter.com/adam3us/status/495679452211580928

I usually am in agreement with Peter Todd about subjects, but Adam Back raised some great points.
To me this just demonstrates the difficulty in evaluating the pros and cons of these extensions. They aren't black/white.  

This does go against the concept of decentralized currency. In this form the side chains and tree chains are tied to dear old mom. Also security may be a concern. If dear old mom suffers a catastropic failure the side and tree chains go down with it. it's like having your eggs in one basket. If it drops you loose it all.

It's either this or you use an alternative PoW that is easily bot-able/51% attacked. I think both sides in this early debate agree that we want to use some form of merged mining, as it's easily the most secure.

[Yoghurt114]

I'm having some trouble fully understanding treechains and I'll be spending some more time researching its full extent, but I think I get the gist of it. It feels like the better way forward. Anyway, here's my thoughts.

--

It's important to realize one of the most important reasons that make bitcoin work, is the fact nodes and miners must be aware of 'at least most things' that are happening in the network.

It's also important to realize one of the most important reasons bitcoin is hard to scale and (dare I say) 'fat', is the fact nodes and miners must, again, be aware of 'at least most things' that are happening in the network......

It's kind of crappy both those reason are the same.

I venture to say treechains are the superior scaling solution. Ie. allowing nodes and miners to be unaware of 'at least some things' that are happening in the network, without making it less secure and at the same time making it more flexible, more scalable and less fat? Sounds good to me. Sounds like the internet.

Now, I'm not saying bitcoin with just a vanilla blockchain cannot scale to a transaction frequency of say, 250k/sec, I'm saying that would be very hard. And yes, 250k/s may be orders of magnitude greater than Visa's puny (4k/s?) fiat network, and greater still than what we could even hope to require at this point in time, but that's exactly the magnitude the bitcoin community should be concerned with scaling to in the future. I think that may have been a key difference of opinion in the discussion between PT and GA following PT's initial tree chain description in the mailing list.

Don't be the guy that goes down in history saying:

"4k/sec transactions ought to be enough for everybody."

[maaku]

In the most general, sidechains will use “SPV Proofs” to send satoshis from the regular Bitcoin chain to the sidechain, and allows the sidechain to eventually send the satoshis back to the main chain once the owner of said coin is finished utilizing the sidechain. While in the sidechain, the main chain knows nothing of what’s happening to the coin, the sidechain is the one tracking who owns what at what time.

No, a side chain is simply any chain with an asset that supports some form of 2-way pegging against BTC. There is no reason that side chains have to use the SPV proof mechanism for accomplishing this peg, or merged mining for validation. These are simply aspects of one example implementation pathway that was presented as a proof of concept of how a side chain could be implemented. The design space is actually quite large, however.

[ex0du5]

Do sidechains solve any real problem not already solved by altcoin exchanges?

Also, I am confused by:

It’s important to note however, that it has been suggested that the outright theft of coins by miners may be protected against using zk-SNARKs.(https://eprint.iacr.org/2013/507.pdf)

I know that zk-SNARKs can resolve transaction verification anonymity issues, providing true security.  I've read the zerocash proposals, and am familiar with the mathematical basis.  But the information I have seen is that this actually makes 51% attacks easier (because it slows down propagation through the network, wasting node cycles and effectively decreasing the computational power of the network).  The link is to one of the original program execution verification papers and is unrelated to cryptocurrency except in the general way that execution verification is the point.  So eating newly minted coins is actually not stopped, and I imagine that is the security risk mentioned in passing here.

Is there a new algorithm being mentioned that goes beyond the zerocash proposals here?  Does this refer to something else?

[maaku]

ex0du5, the point is that the bitcoin validators can do full validation of the side chain via a constant-time SNARK validation, even one whose rules they don't know.

[provenceday]

nice we have some serious discussions in this thread.

[provenceday]

Disentangling Crypto-Coin Mining: Timestamping, Proof-of-Publication, and Validation

Peter Todd Tue, 19 Nov 2013 03:03:38 -0800

http://www.mail-archive.com/bitcoin-development%40lists.sourceforge.net/msg03307.html


In the design of Bitcoin mining serves two fundemental purposes:
proof-of-publication and order consensus.  Bitcoin's design entangles
these fundemental purposes with other goals, such as validation and
initial coin distribution. This leads to a design that is fundementally
unscalable, albeit effective on a small scale. Here we show how these
purposes do not need to be entangled together, and how by disentangling
them we can achieve better scalability and validation of the system as a
whole.

Let's first look at what role each of those purposes plays:

* Proof-of-publication

The fundemental problem Bitcoin solves is the double-spend problem.
Alice has some Bitcoins, and she wants to give them to Bob. She does
this by signing a digital message, a transaction, authorizing her coins
to be assigned to Bob. However, Bob has no way of knowing if Alice has
signed a conflicting digital message assigning her coins to Charlie
instead.

Bitcoin solves this problem by providing a way for Alice and Bob to
agree on a common place where *all* transactions will be published, the
blockchain. Because the definition of a valid transaction is that it has
been published in the blockchain, Bob can examine the contents of it,
and be confident that no conflicting transaction exists.


* Order consensus

Due to the constraints of physics no decentralized system can provide
instantaneous and reliable proof of publication; for a non-ideal
proof-of-publication system to be useful to solve the double-spend
problem we need to come to a consensus about the order in which data was
published. Once an order has been established, subsequent
double-spending transactions can be declared invalid.

Note that time itself isn't directly required, only the order of
transactions needs to be agreed upon.


* Why validation is an optional optimization

Given only proof-of-publication, and a consensus on the order of
transactions, can we make a succesful crypto-coin system? Surprisingly,
the answere is yes!

Suppose the rules of Bitcoin allowed blocks to contain invalid
transactions, in fact, suppose miners did no verification what-so-ever
of the contents of the blocks they mined. Could Bob still be confident
in the coins he received? Absolutely. There is consensus that the
transaction sending coins to Bob's came first and all prior transactions
can be verified as valid by checking the entire blockchain. In Bitcoin
all full nodes do this and Bitcoin could succesfully operate on that
model.

What can't be supported in this model is SPV clients: the existance of a
transaction in a block tells you nothing about its validity, so no
compact proof can be made.

Real-world examples of this issue can be found in the parasitic
consensus system Mastercoin, and to a lesser extent Colored Coins: the
former uses Bitcoin as a proof-of-publication, applying it's own
independent set of rules to that published data. The latter tracks the
transfer of assets in a way that takes advantage of the Bitcoin
validation rules, but any given txout can only be proven to represent a
particular asset with a full chain of transfers back to the asset
genesis. It's notable that proponents of colored coins have proposed
that rules to validate colored coins be added to Bitcoin to make such
lengthy proofs not required.(1)


* What is the minimum domain for anti-double-spend proof-of-publication?

Answer: a single txout.

So what do we mean by "domain" here? In the existing Bitcoin system,
modulo validation, what Alice has proven to Bob is that an entire
transaction has been published. But that's not actually what Bob wants
to know: he only wants to be sure that no transaction inputs, that is
the CTxIn data structure containing a valid scriptSig and reference to a
previous output, have been published that spend outputs of the
transaction he is accepting from Alice. Put more simply, he doesn't care
where a double-spending transaction sends the money, he only cares that
it exists at all.

Suppose the blockchain consisted of blocks that only contained
information on the transaction outputs spent by that block; essentially
a block is a list of CTxIn's. We also, add a third field to the existing
CTxIn structure, hashTx, which commits to the rest of the transaction
spending that txout.

If we sort the CTxIn's in each block by the hash of the *transaction
output being spent* and commit to them with a merkle tree, Bob can now
determine if Alice's transaction is valid by checking the blockchain for
blocks that contain a conflicting spend of any of the inputs to that
transaction. For each block the proof that the block does not contain a
given spend is log2(n) in size.

Put another way, Bob needs proof that some data, a valid CTxIn spending
some CTxOut, has never been published before. He only cares about that
particular CTxOut, so the "publication domain" he is interested in is
that single CTxOut. (note that we are considering a CTxIn as valid if
its scriptSig satisfies the prevout's scriptPubKey; the rest of the
transaction may be invalid for other reasons)

Conversely a transaction is only considered to be valid if all CTxIn's
in that transaction have been succesfully committed to the blockchain
proper; there must be proof that every CTxIn has been published.

Note the parallels to the authors TXO commitments proposal: where TXO
commitments commit to the outputs of every transaction in each block,
here we are committing to the inputs of all transactions.


* Transaction validation

Miners still are doing almost no validation in this scheme, other than
the fact that a block is only valid if the data in it follows some
order. Bob still needs to examine the chain of of all transactions to
determine if Alice's payment was valid. However, the information he
needs to do this is greatly diminished: log(n) * m per txout in that
history, with n as the average number of spends in a block, and m the
number of blocks each txout was in existance for.

Of course, a practical implementation of this concept will have to rely
heavily on direct transfer of proof data from payor to payee.


** Privacy

The increased validation effort required on the part of Bob has an
important privacy advantage: whole transactions need never appear in the
blockchain at all. By incorporating a simple nonce into every
transaction blinding the miners have no way of linking CTxIn's to
CTxOut's. This achieves the end goal of Adam Back's blind symmetric
commitments(3) but by leaving data out of the blockchain entirely rather
than blinding it.


* The incentive to share blockchain data

What is the incentive for miners have in the Bitcoin system to share
their blocks? Why not just share the block header? Of course, the
incentive is that unless they share their block data, all other miners
in the system won't build upon their blocks because they have no idea if
they are valid or not.

But here there is no such thing as an invalid block! Blocks are just
arbitrary data with no specific meaning; whether or not the data is
valid in some sense is of no importance to the miner.

We can re-introduce this incentive by using a proof-of-work scheme that
has the requirement of posession of blockchain data. For instance we
could make the underlying computation be simply H(header + all previous
blocks) - without the entire blockchain you would be unable to mine, or
even validate the work done.

Of course this is impractical for a number of reasons. But it's
important to recognize that this simple scheme doesn't make any
compromises about the continual availability of blockchain data, and
thus the ability for users to validate history. Any lesser scheme will
be a trade-off between that guarantee and other objectives.


** Full TxIn set commitments

Since we have to require miners to posess blockchain data, we might as
well make a simple optimization: rather than commit to the CTxIn's in a
single block, commit to multiple blocks.

First, let's require that every CTxIn present in a block be have a valid
scriptSig for the corresponding scriptPubKey. To do this we need for
CTxIn's to commit to the H(txout) they are spending, and include the
CTxOut itself alongside the CTxIn in the block. Our hash commitments are
now chained as follows:

    CTxIn -> CTxOut -> <merkle path> -> CTransaction -> <merkle path> -> CTxIn

Now that we have valid and invalid CTxIn's, we might as well state that
only one valid CTxIn is allowed for a given CTxOut per block; proof that
a transaction is valid now doesn't have to take into account the problem
of an *invalid* CTxIn that you need to prove is invalid and thus can be
ignored. This validation is stateless, requiring only local data, and
still provides for strong privacy.(a) A fraud proof in this scheme is
simply the CTxIn and CTxOut and merkle path, and the code required to
evaluate it is the same code required to evaluate the data in a block.

a) Remember the mention of a per transaction nonce? It can be used
   between the CTxOut and the rest of the CTransaction so that even if
   every CTxIn and CTxOut is known, the actual transactions can't be
   derived.

Now that we have a definition of a valid CTxIn, we can naturally extend
this to define the set of all valid *oldest* CTxIn's. That is for any
given CTxOut, we include the first valid CTxIn found in any block in
this set. This is analogous to the concept of the UTXO set, except that
items can only ever be added to the TxIn set.

As with UTXO commitments we can commit to the state of the TxIn set
using a merkelized radix tree whose tip is committed to by the block
header.

Of course because a block can manipulate the contents of this set in an
invalid way, we've strongly reintroduced the notion of an invalid block,
we've re-introduced the incentive to share blockchain data, and we've
re-introduced the requirement to have the full set of blockchain data to
mine.


*** Mining with incomplete blockchain data

Or have we? This requirement isn't particularly strong as all: if other
miners are usually honest we'll get away with just trusting them to mine
only valid blocks. Meanwhile the TxIn set in merkelized radix tree form
can have items added to it with only the subset of internal nodes
modified by your additions. A miner can easily produce blocks only
containing CTxIn's spending CTxOuts from a subset of the possible
values. Multiple such miners can even co-operate to produce blocks, with
each handling a specific subset, as multiple radix trees are easily
composed.(b)

Note that Bitcoin is even worse in this regard: you don't need any
previous blockchain data at all to create a new block. For instance the
authors proof-of-tx-propagation concept(5) has the serious flaw that
unscrupulous miners can use the proof that other miners are mining
certain transactions as a way to avoid doing any validation themselves.


*** The deletion problem

What happens if a copy of some of the txin set can't be found? With
Bitcoin this isn't an issue in theory - the miners are supposed to never
extend blocks they haven't verified in full and they are supposed to
distribute blocks freely. Not necessarily a perfect assumption(6) but it
mostly holds true.

With any type of sharded blockchain, it is easy to see that assumption
may not hold true. Now rather than a 51% attack in terms of total
hashing power, you could have a "local" attack on some portion of the
commitment set. On the other hand, with the right set of incentives, the
existance of such an attack can be made to imply actual consent by those
owning the coins involved, e.g. through proof-of-stake combined with the
proof-of-work. (perhaps better described as proof-of-consent with
proof-of-work)


1) OP_CHECKCOLORVERIFY: soft-fork for native color coin support,
   https://bitcointalk.org/index.php?topic=253385.0,
   jl2012

2) Merkle tree of open transactions for lite mode?
   https://bitcointalk.org/index.php?topic=21995.0,
   Gregory Maxwell

3) Ultimate blockchain compression w/ trust-free lite nodes
   https://bitcointalk.org/index.php?topic=88208.0
   Alan C. Reiner

4) blind symmetric commitment for stronger byzantine voting resilience,
   
http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02184.html,
   Adam Back

5) Near-block broadcasts for proof of tx propagation,
   
http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg02868.html,
   Peter Todd

6) Perverse incentives to withhold blocks
   
http://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03200.html
   Peter Todd

[provenceday]

 Indeed, maybe a solution doesn't exist. Point is we already know the proposed MM #sidechains isn't a solution.


https://twitter.com/adam3us/status/495679452211580928

[ex0du5]

ex0du5, the point is that the bitcoin validators can do full validation of the side chain via a constant-time SNARK validation, even one whose rules they don't know.

Thanks.  I thought there was some greater guarantee being implied (with later mentions of moon math). I see this doesn't actually prevent all ways of taking coins from others, and others have expressed the same possibility.  This is an area I'm trying to get a better understanding of, as I've been analyzing other algorithms, and I keep seeing these discussions pop up.

I still don't see what this solves that altcoin exchanges don't.  You have an entity that takes value into it's store and gives out value in an alternate blockchain that may or may not follow similar rules (some altcoin, maybe a Bitcoin clone or maybe something quite different).  The exchange can hold onto that value until the altcoin work is done and exchange back the value.  Validation is done in whatever currency is chosen for the transfer, which can obviously have any of a variety of zero-knowledge transaction and block validation schemes.

I've always expected altcoins are the natural transaction scaling mechanism, so I'm probably biasing myself, though.

[adam3us]

ex0du5, the point is that the bitcoin validators can do full validation of the side chain via a constant-time SNARK validation, even one whose rules they don't know.

Thanks.  I thought there was some greater guarantee being implied (with later mentions of moon math). I see this doesn't actually prevent all ways of taking coins from others, and others have expressed the same possibility.  This is an area I'm trying to get a better understanding of, as I've been analyzing other algorithms, and I keep seeing these discussions pop up.

Wanted to clarify: the point in my view is the snark proof proves that the validation program was run on the sidechain and the rules were followed.  Then the bitcoin chain can validate the proof without needing to understand the validation program.  Other than the risks coming from bleeding edge crypto, it thereby allows the sidechain to offer the same security properties as the main bitcoin blockchain.

gmaxwell wrote about this concept first here https://bitcointalk.org/index.php?topic=277389.0

I still don't see what this solves that altcoin exchanges don't.

alt-coins as usually defined have floating market prices, pegged coins do not.  

Sidechains give a way to extend the parent chain (eg a bitcoin sidechain allows you to experiment with and deploy new features like scripting, issued assets, different block intervals etc).

You have an entity that takes value into it's store and gives out value in an alternate blockchain that may or may not follow similar rules (some altcoin, maybe a Bitcoin clone or maybe something quite different).  The exchange can hold onto that value until the altcoin work is done and exchange back the value.  Validation is done in whatever currency is chosen for the transfer, which can obviously have any of a variety of zero-knowledge transaction and block validation schemes.

that sounds rather centralised?  Dont forget MtGox also operated like that - they exchanged your bitcoins for gox iou entries in their database.  Then some stuff happened (you trade etc) and finally you ask for repayment of the iou.  If its central there is a central point of failure that can lose or steal the backing funds.

You can think of a sidechain as a decentralised escrow agent where the sidechain economic majority (hashrate etc) controls and fairly administers the backing.

Adam

[ex0du5]

You have an entity that takes value into it's store and gives out value in an alternate blockchain that may or may not follow similar rules (some altcoin, maybe a Bitcoin clone or maybe something quite different).  The exchange can hold onto that value until the altcoin work is done and exchange back the value.  Validation is done in whatever currency is chosen for the transfer, which can obviously have any of a variety of zero-knowledge transaction and block validation schemes.

that sounds rather centralised?  Dont forget MtGox also operated like that - they exchanged your bitcoins for gox iou entries in their database.  Then some stuff happened (you trade etc) and finally you ask for repayment of the iou.  If its central there is a central point of failure that can lose or steal the backing funds.

You can think of a sidechain as a decentralised escrow agent where the sidechain economic majority (hashrate etc) controls and fairly administers the backing.

Adam

I didn't think of it as centralized because anyone can be an exchange and whoever wants the branch can choose the exchange.  Also, the choice on whether to peg the conversion or let it float is an option of the exchange and it's store.  But I understand that this forces openness on a not-necessarily-open process and standardizes for a particular use.  And I can certainly see the benefits of distributed administration of the backing (though would still point out that such an implementation of an exchange is not prohibited either).

I think, though, it's important for me to better understand the proposals here before appearing to suggest that existing practices solve the use cases.  I was just trying to get a feel for what they were targeted at solving, and your answer is a good start!