Segregated witness - The solution to Scalability (short term)?

[BitUsher]

But what amount of transaction per second would this deliver?

"Size = 4*BaseSize + WitnessSize <= 4MB. For normal transaction load, it means 1.75 MB, but more for multisig."

More than 1.75 but less than 4 effective. Thus between 6 to 28 tps vs current 3-7tps.

can this really compete against LN in terms of being able to be at the same level of VISA and the like?

It isn't meant to compete but merely be one piece of the puzzle that solves many other concerns as well. Sipa is positive towards other BIPs and LN. This softfork is not meant to replace a future hardfork to address scalability.

This looks good on the surface but I would wait for someone like gmaxwell to comment on it to see potential flaws. Did any of the core devs speak on this?

All of the core devs are familiar as this has been discussed since 2011. The only real objection is from the fact that previous to luke-jrs brilliant discovery they assumed a hard fork would be needed which would be very difficult to accomplish. I know of no core devs against this implementation because if you understand it, it is a no -brainer.

[Carlton Banks]

Sounds alright to me so far, but I will need to familiarise myself better with it even still.

Video of great work done-

https://www.youtube.com/watch?v=fst1IK_mrng#t=36m

The presentation directly after Pieter Wuille's is a good watch. Demonstrates the (exemplary) capacity of:

• Basic Lightning (CLTV only)
• Intermediate Lightning (CLTV + CSV)
• Advanced Lightning (CLTV + CSV + Segregrated Witness)

Looks like the kind of transaction rate scaling we're after (100's of millions of users supported under the Advanced scheme)

[franky1]

1.But what amount of transaction per second would this deliver?
2.can this really compete against LN in terms of being able to be at the same level of VISA and the like?

1. no, its not about tx per second. its about relaying speeds of solved blocks to clients imagine it this way.
a pizza is 1mb and oneside has pepperoni, the other has anchovies.. the proposal is to cut the pizza in half but you as a full and hard worker(full node miner) still eat the same 1mb whole pizza.. thats right the full 1mb limit nothing has changed.

but because its in 2 halves, its now easier for your witness to CLONE(relay) just the pepperoni half of the pizza because he doesnt like anchovies(signatures). and thus he as a separate thing holds onto just 500kb pepperoni of his copy of the pizza

so each 10 minutes you(full node/miner) are still eating a whole 1mb pizza and your witness has cloned a copy of the pepperoni side meaning he is only eating 500kb per blocktime.

for the full nodes mining the real bitcoin blockchain. nothing has changed, the data totals are the same and thus no extra transactions can fit into bitcoins blocks. the only difference is that the miners data has a split in the middle to make it easier to relay a copy of a half to the network of lite clients..

its not changing the transactions per second. and is only reducing the data per block for the witness(liteclients) who dont want to download everything.

2.
witness wont scale anything.. its just a tool for SPV wallets to not be as bloated.. nothing to do with the real blockchain that gets mined properly
LN has more potential to scale however i have seen a few limitations that dont make it perfect.

[QuestionAuthority]

Ok, that doesn't seem so bad at all. It's just a sidechain solution.

[brg444]

Ok, that doesn't seem so bad at all. It's just a sidechain solution.

It's not?

[BitUsher]

1.But what amount of transaction per second would this deliver?
2.can this really compete against LN in terms of being able to be at the same level of VISA and the like?

1. no, its not about tx per second. its about relaying speeds of solved blocks to clients imagine it this way.
a pizza is 1mb and oneside has pepperoni, the other has anchovies.. the proposal is to cut the pizza in half but you as a full and hard worker(full node miner) still eat the same 1mb whole pizza.. thats right the full 1mb limit nothing has changed.

but because its in 2 halves, its now easier for your witness to CLONE(relay) just the pepperoni half of the pizza because he doesnt like anchovies(signatures). and thus he as a separate thing holds onto just 500kb pepperoni of his copy of the pizza

so each 10 minutes you(full node/miner) are still eating a whole 1mb pizza and your witness has cloned a copy of the pepperoni side meaning he is only eating 500kb per blocktime.

for the full nodes mining the real bitcoin blockchain. nothing has changed, the data totals are the same and thus no extra transactions can fit into bitcoins blocks. the only difference is that the miners data has a split in the middle to make it easier to relay a copy of a half to the network of lite clients..

its not changing the transactions per second. and is only reducing the data per block for the witness(liteclients) who dont want to download everything.

2.
witness wont scale anything.. its just a tool for SPV wallets to not be as bloated.. nothing to do with the real blockchain that gets mined properly
LN has more potential to scale however i have seen a few limitations that dont make it perfect.

This is all 100% true in itself as Segregating the signature isn't intended to increase TPS by itself, but with a soft fork core can increase the witness data set limit to 4MB thus indirectly increasing TPS to 6 to 28 tps. It isn't the Segregation in itself that increases the TPS but the partial and limited blocksize growth that only applies to part of the data set. Thus after the softfork there will be a 1 and 4MB limit concurrently depending upon which dataset.

Full nodes will take on more data, but lite nodes will have new features like fraud proofing (although this isn't as thorough as using a full node) and take on as much data as the blocksize limit never increased and have 6 to 28 tps.

There will than be 3 categories of nodes-
1) Traditional and the least secure SPV nodes
2) Lite / Prune nodes with SW nodes that have some fraud proofs
3) Full nodes

Ok, that doesn't seem so bad at all. It's just a sidechain solution.

Well it was tested as a sidechain in elements alpha , but this is actually intended to be an immediate soft fork on the main chain.

[franky1]

Ok, that doesn't seem so bad at all. It's just a sidechain solution.

well its not a side chain in itself.. but yea if you have 20 blockchains linked together.. some lite clients may want a reduced history of all chains.
although the full nodes/miners still have to deal with 20 full blockchains. lite clients can have a basic summerised copy.

i think the solution to lite clients. is to not send out a command to the network with a block height attached asking for all new blocks since X.. (to resync)
but instead send out a command to the network with a public key attached asking for all tx data of that key.

that way liteclients only store the data relevant to the keys it holds

[franky1]

This is all 100% true in itself as Segregating the signature isn't intended to increase TPS by itself, but with a soft fork core can increase the witness data set limit to 4MB thus indirectly increasing TPS to 6 to 28 tps. It isn't the Segregation in itself that increases the TPS but the partial and limited blocksize growth that only applies to part of the data set. Thus after the softfork there will be a 1 and 4MB limit concurrently depending upon which dataset.

if you increased the tx data to 4mb but kept signature to 1mb... thats just the same as having a 5mb block.. and full nodes will still have to mine upto 5mb of data whether its split or together..

so witness is not itself causing more tx per sec.. it is however usingits ninja skills to just up the block limit to 5mb under the subterfuge that its something special.

so without meandering..
explain the benefits to miners and bitcoin full node users who (imagine last week) agreed to a simple 5mb block limit increase.. and today you are trying to sell them the idea of witness..
forget about lite user benefits.. explain the benefits to the miners/full node network

[Lauda]

it is however usingits ninja skills to just up the block limit to 5mb under the subterfuge that its something special.

so without meandering..
explain the benefits to miners and bitcoin full node users who (imagine last week) agreed to a simple 5mb block limit increase.. and today you are trying to sell them the idea of witness..
forget about lite user benefits.. explain the benefits to the miners/full node network

You know, I'm quite disappointed. Have you even looked at the picture shown in the original post? Of course you have not, else you would not be asking the ignorant question of why this is something special. All the side-effects of this are presented on that picture and they are just pure pros. Unintentional transaction malleability gets fixed, script upgrades become simpler, fraud proofs, etc.

[QuestionAuthority]

Ok, that doesn't seem so bad at all. It's just a sidechain solution.

well its not a side chain in itself.. but yea if you have 20 blockchains linked together.. some lite clients may want a reduced history of all chains.
although the full nodes/miners still have to deal with 20 full blockchains. lite clients can have a basic summerised copy.

i think the solution to lite clients. is to not send out a command to the network with a block height attached asking for all new blocks since X.. (to resync)
but instead send out a command to the network with a public key attached asking for all tx data of that key.

that way liteclients only store the data relevant to the keys it holds

Yeah, I can understand that. Those of us that primarily use SPV clients will still be secure. Miners will hash reduced size blocks and full nodes will be the only ones burdened with an even larger carrying capacity. Works for me. I only use lite clients now and haven't run a full node in over a year.

[BitUsher]

if you increased the tx data to 4mb but kept signature to 1mb... thats just the same as having a 5mb block.. and full nodes will still have to mine upto 5mb of data whether its split or together..

so witness is not itself causing more tx per sec.. it is however usingits ninja skills to just up the block limit to 5mb under the subterfuge that its something special.

Close, its actually more complicated than this because of multisig and is essentially reflecting miners having an effective increase between 1.75MB limit but less than 4 MB limit . Thus if blocks are 100% full expect ~3MB of bloat per 10 min avg for full nodes.

so without meandering..
explain the benefits to miners and bitcoin full node users who (imagine last week) agreed to a simple 5mb block limit increase.. and today you are trying to sell them the idea of witness..
forget about lite user benefits.. explain the benefits to the miners/full node network

Many user benefits but to address your point concerning miners and full nodes:

Negatives vs status quo-
Increased Disk bloat and bandwidth requirements (Just like rasing the rate to 1.75- 3.5MB limit)

Nuetral-
UTXO unaffected

Positives-
Increased TPS capacity thus more mining fees and Better job security as profits are now diversifies from tx volume instead of reward alone which will be quickly dropping

There are other positives to miners from benefiting the overall health of the bitcoin ecosystem but I think you are trying to focus on the short term selfish incentives of miners alone absent from external motivations.

it is however usingits ninja skills to just up the block limit to 5mb under the subterfuge that its something special.

so without meandering..
explain the benefits to miners and bitcoin full node users who (imagine last week) agreed to a simple 5mb block limit increase.. and today you are trying to sell them the idea of witness..
forget about lite user benefits.. explain the benefits to the miners/full node network

You know, I'm quite disappointed. Have you even looked at the picture shown in the original post? Of course you have not, else you would not be asking the ignorant question of why this is something special. All the side-effects of this are presented on that picture and they are just pure pros. Unintentional transaction malleability gets fixed, script upgrades become simpler, fraud proofs, etc.

I think franky is just trying to focus on the incentive structure for miners to see how well this will work. It is a fair question to make and try and remove any external goodwill of miners and consider worst possible outcomes. Sometimes some skepticism is healthy and we shouldn't rely on the goodwill of miners to secure us into the future.... that is merely another layer of security that involves humans I appreciate.

[BitUsher]

Yes this was a very good explanation!

So another question, why don't we implement this if it's so effective and it works as advertised?!

Well there has been 6 months of testing and Pieter Wuille did suggest we ASAP implement this improvement so I would expect a few weeks to allow outside criticsm to be taken into account but it is going to happen quickly as his statement from the transcript is -

"This is my proposal that we do right now. We implement segregated witness right now, soon. "

[Mickeyb]

Yes this was a very good explanation!

So another question, why don't we implement this if it's so effective and it works as advertised?!

Well there has been 6 months of testing and Pieter Wuille did suggest we ASAP implement this improvement so I would expect a few weeks to allow outside criticsm to be taken into account but it is going to happen quickly as his statement from the transcript is -

"This is my proposal that we do right now. We implement segregated witness right now, soon. "

I see. Thanks for responding. I have deleted my post since I am in the process of reading more about it. I didn't want to clog up the thread too much.

I must say that this is a very interesting concept. This also gives us time, if implemented, to find another long-term solution since I doubt that even a 4 times increase will be enough in the future!

[BitcoinNewsMagazine]

I see the commits from Pieter Wuille (sipa) for Segregated Witness at https://github.com/sipa/bitcoin/commits/segwit

Appears that this is a done deal, am I correct in that statement? If so when will the soft fork take place. Thanks.

[BitUsher]

Lauda, explain Segregated Witness to me like I'm five.

And to me as if I'm just born

An Airline only allows you to bring a limited amount of weight and bags onboard a plane for your luggage. They have to be fair and have a consistent policy for their passengers while at the same time recognize real life physical limitations and risks. There is a careful balance to be made where they make their clients happy with allowing them more luggage but at the same time keep the weight considerations reasonable for speed , limiting fuel costs and safety.

They previously had a policy of allowing 2 carry on bags weighing 10 Kg in total, but overtime with a growing economy their clients grew more wealthy and needed to travel with more bags because they were flying to Aruba for 2 week vacations, instead of 1 week. The clients were demanding more bags and up to 40 Kg of luggage weight. They had a choice to increase the size of the plane but the accountants and engineers were concerned because the fuel costs would increase and the safety of the planes may be compromised as well with so much weight. The sales department also voiced an objection that larger planes would cut down on their flying routes leading to less possible destinations clients could visit.

It was known since 2011 a safe and effective solution would be to separate the less critical luggage on a train or boat and the important luggage could go with the client to solve these problems but the airline kept delaying these changes because of distribution and contract problems with the railroad/shipyard companies. Redirecting every railroad or shipping route would be a logistical nightmare. Another concern is passengers didn't want their luggage being mixed or misplaced within the trains/boats as they were keen on some of the new quick luggage check-in and fraud protection processes the airlines were developing and there is no way they could get any of that with dumping their luggage on a train or boat.

One day a bright young engineer recommended a novel approach to solve these concerns. He indicated, "Look, we already have shipping agreements with Fedex/DHL and lease out a certain amount of space on their cargo planes that flyout from the same airports. We would have to increase capacity on them but they are efficient and we won't have to reroute all the trains or deal with the nasty railroad companies and our passengers could still bring their important luggage through our new fraud protection and quick check-in processes. Our passengers won't have to pay more and get everything they want. We will have to secure larger contracts with the airfreight companies but ultimately they will benefit from more business and we will benefit from more flight routes and clients." ... followed by an applause from the board of directors and some quick recommendations to roll out the long awaited changes immediately.



***edited to better explain the complexities and tradeoffs in the analogy.***

Ok, I needed to update my ELI5 analogy to more accurately reflect Segregated witness.

I see the commits from Pieter Wuille (sipa) for Segregated Witness at https://github.com/sipa/bitcoin/commits/segwit

Appears that this is a done deal, am I correct in that statement? If so when will the soft fork take place. Thanks.

"This is my proposal that we do right now. We implement segregated witness right now, soon. "  --  Pieter Wuille

[johnyj]

I have watched this video twice but still don't really get it

So far as I understand Pieter's proposal, a spending transaction is first validated by the mining nodes by checking its signature, then after it is validated, the node will include the transaction in the next mined block, but without that signature

However, this will create a security question for the other nodes receiving such a block: Without signature, how do they know the transaction is valid? Is it possible for a rougue node to forge a block with lots of invalid transctions but appears to be all validated by that node?

Normally all the adjacent nodes have similar mempool which contains similar transactions. If a transaction is already in their mempool they will know it is valid or not. However, if a new block arrives from far apart which contains many transactions not in their mempool, they need all the data of the transaction, especially the signature to validate it

Otherwise you would be able to spend bitcoin at any address without the signature

I'm not very sure how it works today, e.g. what kind of check other nodes do to make sure it is a valid block with all valid transactions, have to check it back


A side thought: I suppose the original Satoshi client design tried his best to be both secure and efficient. If there is such a large room of efficient increase without impacting security, why it has not been implemented before?

I remember 2-3 years ago there was a talk about the separation of transaction data using two chains to get rid of the forever growing block size pain, then after some deeper discussion it turns out you can not secure the separated part from being tampered, so it seems everything in a transaction is a whole and need to be included and secured by hash power altogether, which is the most efficient way

[johnyj]

"What are the advantages of this? It allows you to drop the signatures from relay whenever you are relaying to a node that is not actually doing full-validation at the time. "

So this solution still does not help full-validation nodes, it only improve the relay speed for SPV nodes. But SPV nodes could just connect to the latest full node to get faster access, they are not the bottleneck for the whole network

[Lauda]

"What are the advantages of this? It allows you to drop the signatures from relay whenever you are relaying to a node that is not actually doing full-validation at the time. "

So this solution still does not help full-validation nodes, it only improve the relay speed for SPV nodes. But SPV nodes could just connect to the latest full node to get faster access, they are not the bottleneck for the whole network

What are you talking about? Advantages: 4x increase in traffic that is possible via a soft fork, fix to malleability, simpler script upgrades (quite important as well), fraud proof, less bandwidth for light nodes and historical sync. I think that it could help full nodes (not sure yet) by not syncing all of the historical data (i.e. signatures), because it isn't really needed. More discussion about this is definitely needed.

[BitUsher]

"What are the advantages of this? It allows you to drop the signatures from relay whenever you are relaying to a node that is not actually doing full-validation at the time. "

So this solution still does not help full-validation nodes, it only improve the relay speed for SPV nodes. But SPV nodes could just connect to the latest full node to get faster access, they are not the bottleneck for the whole network

https://www.reddit.com/r/Bitcoin/comments/3vurqp/greg_maxwell_capacity_increases_for_the_bitcoin/

Segregated witness does several things; fixing malleability, improving upgradability; improving scaleability, and increasing capacity.

The improved scalablity comes from the new security models it makes available.. lite nodes with full node security (under specific conditions), fractional ('sharded' verification), quick bootstrapping by not fetching history data you're not going to verify and are only going to prune, and reduced data sent to lite clients.

Increased capacity comes from the fact that it takes roughly 2/3rds of the transaction data from participating transaction out of the part of the block that the blocksize limit counts and moves them to the witness, where they're counted (by the implementation) as 1/4th the size. The result for typical transactions is a better than 2x increase in capacity (more if multisig is heavily used). In the worst case, an strategically behaving miner might produce a 4MB bloat-block; since thats the largest size you can get if you were to make an all witness block. The typical increase in blocksize would be more like 2MB, but expressing it that way would underplay the worst case behavior..

Here is an exact breakdown of an example of the savings in overhead and why capacity can increase-

Yea, the exact impact depend on usage patterns.

If your case is a counting one input, one output, pay to hash transactions the sizes work out to

4 (version) + 1 (vin count) + 32 (input id) + 4 (input index) + 4 (sequence no) + 1 (sig len) + 0 (sig) + 1 (output count) + 1 (output len) + 36 + (32 byte witness program hash, push overhead, OP_SEGWIT) + 8 (value) + 4 (nlocktime) = 96 non-witness bytes

1 (witness program length) + 1 (witness program type) + 33 (pubkey) + 1 (checksig) + 1 (witness length) + 73 (signature) = 110.

96x + 0.25*110x = 1000000; x = 8097 or 13.5 TPS for 600 second blocks; (this is without the code in front of me, so I may well have slightly miscounted an overhead; but it's roughly that)... which is around double if you were assuming 7 tps as your baseline. Which is why I said double the capacity in my post... but YMMV.

[Zarathustra]

All in all, seems to be even less than a x2 capacity increase:

http://lists.linuxfoundation.org/pipermail/bitcoin-dev/2015-December/011869.html

For how many month will it be 'the solution' in 2016, the year of the Great Halvening? April or even May?