Segregated witness - The solution to Scalability (short term)?

[RoadTrain]

Also a good summary by StephenM347
http://bitcoin.stackexchange.com/a/41771

Segregated witness splits up transactions into different parts that can be handled separately instead of the single chunk of data as they are now. Specifically, it takes the digital signatures out of transactions and puts them in a separate merkle tree that has the same structure as the transaction merkle tree. So, if fully implemented, to check that an input legally spends its previous output, you would get the signature from the signature tree, instead of the standard scriptSig field.

These are a few of the benefits of this idea:

• Since signature data (witness data) is stored outside the transaction (and outside the standard block), it means that that data doesn't have to be counted towards the block size. Pieter Wuille is proposing a 75% discount on space taken up by signature data, meaning that you can fit 4x as much signature data into blocks. This effectively results in a soft fork increase to the block size.
• Completely solves malleability issues. Using transactions with the signature data outside the transaction means that TXIDs don't hash the signature data, which means that they're not malleable (assuming you're using the standard SIGHASH flag). Technically, the signatures are still malleable, it's just that modifying them doesn't invalidate chains of transactions because the signatures don't sign the modifiable parts.
• Allows for a slow upgrade. Software has to opt in to using Segregated Witness after it has been fully deployed to the network, but in the meantime (and afterward) transactions can still be made as usual without segregated witness.
• All future Script updates become soft forks. When segregated witness gets fully implemented, it will have a version byte in outputs for what version of Script it is using. And the behaviour for clients that see a script with a non recognized version number is that they treat it as an 'anyone can spend' output.
• Signatures only prove that a transaction is authorized, it doesn't describe where funds are going or where they came from. So, after they're checked they can be discarded. Putting the signatures in a separate data structure makes it much easier to prune that data, which results in much less blockchain data needing to be stored on your hard drive.

However, this doesn't completely increase the block size, it just increases the amount of signature data that a block can store. Since transactions are roughly 60% made up of signature data, this is still a pretty big gain.

[1715125706]

1715125706

[1715125706]

1715125706

[funkenstein]

Also a good summary by StephenM347
http://bitcoin.stackexchange.com/a/41771

Segregated witness splits up transactions into different parts that can be handled separately instead of the single chunk of data as they are now.

Wait a minute.  Lets just stop right there shall we?  Because I was under the impression that a transaction, and in fact any information at all, could already be divided up in to ones and zeros and handled any way we liked. 

Isn't a "single chunk of data" actually a bunch of ones and zeros that any client, miner, observer, bot, or individual, and do whatever they want with - put in tables, trees, hash them, store them.. whatever?  That is, already? 

What am I missing here?  How is this a thing to be discussed or debated?  Go, split your transactions up any way you like.  Nobody is stopping you. 

Personally I have some code that only keeps the first 20 base58 digits of btc addresses, dropping the rest, to save space.  I didn't make a post on a forum about it or a talk at a conference though.  You don't have to ask permission to manipulate public data. 

[spartacusrex]

I think SegWit brings up an interesting point.

Namely - What data is essential to Bitcoin ?

What do we need to store and what can we through away ?

Which bits matter for the security and integrity of the system ?

..

I'm of the opinion that the inputs and outputs could also be put into an external disposable merkle tree (maybe the same one..).

The only bits that matter are the block headers, to show the POW, and the UTXO set. (this would probably aid anonymity as well)

It would of course require storing the UTXO in some fashion, ( the root hash of that tree would be mined into the block headers ) but certainly not beyond the capabilities of 'bitcoin developers'.. 

Those are the only bits that matter IMHO and then we could seriously talk about shrinking bitcoin 100 fold (in storage requirements).

[franky1]

Also a good summary by StephenM347
http://bitcoin.stackexchange.com/a/41771

Segregated witness splits up transactions into different parts that can be handled separately instead of the single chunk of data as they are now. Specifically, it takes the digital signatures out of transactions and puts them in a separate merkle tree that has the same structure as the transaction merkle tree. So, if fully implemented, to check that an input legally spends its previous output, you would get the signature from the signature tree, instead of the standard scriptSig field.

These are a few of the benefits of this idea:

[1]Since signature data (witness data) is stored outside the transaction (and outside the standard block), it means that that data doesn't have to be counted towards the block size. Pieter Wuille is proposing a 75% discount on space taken up by signature data, meaning that you can fit 4x as much signature data into blocks. This effectively results in a soft fork increase to the block size.
[2]Signatures only prove that a transaction is authorized, it doesn't describe where funds are going or where they came from. So, after they're checked they can be discarded. Putting the signatures in a separate data structure makes it much easier to prune that data, which results in much less blockchain data needing to be stored on your hard drive.

However, this doesn't completely increase the block size, it just increases the amount of signature data that a block can store. Since transactions are roughly 60% made up of signature data, this is still a pretty big gain.

1. imagine you have a pair of pants(fullnode) and in the pockets(chain) you carry receipts for all your spending..
your girlfriend(segwit) loves wearing skinny jeans hates filling her pockets up.. she now wants you to keep the transaction part of the receipt in your left pocket and the part with the shops logo of the receipt in your right pocket.. the pants still weight the same as the total paper equals the same amount but she wants you to only think about the left pocket. and although you pretend the right pocket doesnt exist, you know its still weighing you down..

the only benefit of splitting up the receipts. is so that your girlfriend(segwit) can quickly grab receipts from your left pocket and read data without having to see useless logo's or "thank you for shopping with us" footers. she cant tell where you shopped or if the receipt is real.. but she can see that money has been spent and naively does accounts trusting you.

2. the girlfriend can photocopy(be relayed) the receipts and she can bin the logo part of the receipt.. but the guy in the pants (fullnode) will still need the right pocket of store logo's(signatures), because he needs to be a seeder for anyone else who may ask for the receipts. so they can do proper checks that the receipts came from proper and real shops..

in short. splitting the receipts does not benefit fullnodes and only benefits lazy lite clients. that can easily split the chain themselves by deciding what details they dont want to save to file.

yes some say this means the left pocket is only 40% full, but with each new receipt, part goes into the left part goes into the right.. meaning the pants still get just as heavy as they would if they just made bigger pants.. instead of messing with the pockets.

so now the guy has to not only cut up the transactions to put into each pocket. not only give the girlfriend the 40% half copy of the receipt so they can fit into her smaller skinny jeans.. but if she questions where the receipt comes from, she questions the guy and he then has to check the other pocket. read which store it came from and tell her if its legit. causing more arguments (bandwidth bloat)

and also the shop logo receipt part(signature), the guy has to write in the receipt number and timestamp(hash) to it so that he can link the parts together (data bloat by having extra data on 2 parts instead of 1 data on one part

id much prefer a girlfriend who didnt ask me to re-arrange my pants, and that girlfriend simply didnt wear any pants at all and simply asked me about a particular receipt when she needs to know it.. rather then asking for every receipt for the last 7 years cut up and handed to her. where in 99.99% of the time she wont need to look at as the receipts in only 0.01% has relevance to her.

after all its not like her copies of the receipts can be of any use, especially if i have to be the one to tell her that one of the receipts has an error.. thus there is no point her handing the receipts onto anyone else either. as she would need to inform them of issues too and they would then need to check with me. which would get annoying

[RoadTrain]

Also a good summary by StephenM347
http://bitcoin.stackexchange.com/a/41771

Segregated witness splits up transactions into different parts that can be handled separately instead of the single chunk of data as they are now.

Wait a minute.  Lets just stop right there shall we?  Because I was under the impression that a transaction, and in fact any information at all, could already be divided up in to ones and zeros and handled any way we liked.  

Isn't a "single chunk of data" actually a bunch of ones and zeros that any client, miner, observer, bot, or individual, and do whatever they want with - put in tables, trees, hash them, store them.. whatever?  That is, already?  

What am I missing here?  How is this a thing to be discussed or debated?  Go, split your transactions up any way you like.  Nobody is stopping you.  

Personally I have some code that only keeps the first 20 base58 digits of btc addresses, dropping the rest, to save space.  I didn't make a post on a forum about it or a talk at a conference though.  You don't have to ask permission to manipulate public data.  

It makes no sense to stop right there. You can't just split a transaction without it losing its integrity, because we need the whole transaction to calculate its txid. With SW we can do that by introducing a new consensus rule (witness merkle hash), and now we will have two hashes representing a transaction: txid and witness hash. The benefit is that only txid is used as a reference in the blockchain, and this way we solve the malleability issue introduced by scriptSig.

[RoadTrain]

after all its not like her copies of the receipts can be of any use, especially if i have to be the one to tell her that one of the receipts has an error.. thus there is no point her handing the receipts onto anyone else either. as she would need to inform them of issues too and they would then need to check with me. which would get annoying

You are painting it as if you would need to manually do all this stuff. It's all casually done by the software, you likely won't notice any changes.

You are arguing against a semi-full-node architecture that is better than SPV. Why would you do that? Are you against SPV as well? Do you understand flaws in its current design? Don't you see how it could get much better with segwit? Do you think we should all run full nodes? Do you think it's realistic?

[funkenstein]

It makes no sense to stop right there. You can't just split a transaction without it losing its integrity, because we need the whole transaction to calculate its txid. With SW we can do that by introducing a new consensus rule (witness merkle hash), and now we will have two hashes representing a transaction: txid and witness hash. The benefit is that only txid is used as a reference in the blockchain, and this way we solve the malleability issue introduced by scriptSig.

Thanks for your reply.  Well certainly we don't want to lose integrity!  That is important emphasis thank you.  

One can split data apart, move it around, stack it in different piles, but if one has thrown out the instructions for how to put it back.. well, one is royally fucked at that point or at least out the original data.

So a couple things come to mind:

1) Malleability of TXID has never been an issue or a problem

2) The fundamental data we need for integrity is still the fundamental data we need for integrity.  Nothing has really changed.  If you come up with a clever way to store data that helps miners be efficient, that's great.  Not that any of them would bother looking here anyway for such ideas.  Nor would they care if pull requests are "approved".  Just grab the code and run it if you like.  
    

[brg444]

It makes no sense to stop right there. You can't just split a transaction without it losing its integrity, because we need the whole transaction to calculate its txid. With SW we can do that by introducing a new consensus rule (witness merkle hash), and now we will have two hashes representing a transaction: txid and witness hash. The benefit is that only txid is used as a reference in the blockchain, and this way we solve the malleability issue introduced by scriptSig.

Thanks for your reply.  Well certainly we don't want to lose integrity!  That is important emphasis thank you.  

One can split data apart, move it around, stack it in different piles, but if one has thrown out the instructions for how to put it back.. well, one is royally fucked at that point or at least out the original data.

So a couple things come to mind:

1) Malleability of TXID has never been an issue or a problem

2) The fundamental data we need for integrity is still the fundamental data we need for integrity.  Nothing has really changed.  If you come up with a clever way to store data that helps miners be efficient, that's great.  Not that any of them would bother looking here anyway for such ideas.  Nor would they care if pull requests are "approved".  Just grab the code and run it if you like.  

Here? No but they certainly look up to the Core devs, right or wrong.

"Just grab the code and run it" is pretty much the definition of a miners enforced soft fork, no?

[RoadTrain]

1) Malleability of TXID has never been an issue or a problem

For you? Because if we're to implement full Lightning Network and possibly other sophisticated solutions, we need malleability solved.

[funkenstein]

1) Malleability of TXID has never been an issue or a problem

For you? Because if we're to implement full Lightning Network and possibly other sophisticated solutions, we need malleability solved.

I'm not sure that's the case, but of course I could be wrong.  Can't you build it to actually check receipt of funds?  I mean, that should in theory be the important part of making a payment anyway, rather than thepayment ID.  Please let me know what I am missing here.     
 

[BitUsher]

I'm not sure that's the case, but of course I could be wrong.  Can't you build it to actually check receipt of funds?  I mean, that should in theory be the important part of making a payment anyway, rather than thepayment ID.  Please let me know what I am missing here.    

Lightning Network can work just fine with tx Malleability but wont be as efficient and cannot scale as long as it remains. This is why solving it is one of the requirements to proceed with the project.

https://blog.bitgo.com/malevolent-malleability/
https://lightning.network/lightning-network-paper-DRAFT-0.5.pdf

[funkenstein]

I'm not sure that's the case, but of course I could be wrong.  Can't you build it to actually check receipt of funds?  I mean, that should in theory be the important part of making a payment anyway, rather than thepayment ID.  Please let me know what I am missing here.    

Lightning Network can work just fine with tx Malleability but wont be as efficient and cannot scale as long as it remains. This is why solving it is one of the requirements to proceed with the project.

https://blog.bitgo.com/malevolent-malleability/
https://lightning.network/lightning-network-paper-DRAFT-0.5.pdf

Sorry I am slow.  What won't be as efficient?  I guess I don't understand the problem exactly.

https://blog.bitgo.com/malevolent-malleability/  <-- I see here a list of how folks who didn't know what they were doing could make mistakes, and a conclusion: "The general consensus up to this point has been that malleability is an annoying but not critical system-wide problem. "

Anyway, if for some reason you can't figure out a way to actually check that funds have been sent and arrived at the proper address, or to not reference any TXs by number, you can always pay a miner a few extra milliies to put your TX with exactly the bit order you like into the block just where you want it. 

[BitUsher]

Sorry I am slow.  What won't be as efficient?  I guess I don't understand the problem exactly.

https://blog.bitgo.com/malevolent-malleability/  <-- I see here a list of how folks who didn't know what they were doing could make mistakes, and a conclusion: "The general consensus up to this point has been that malleability is an annoying but not critical system-wide problem. "

Anyway, if for some reason you can't figure out a way to actually check that funds have been sent and arrived at the proper address, or to not reference any TXs by number, you can always pay a miner a few extra milliies to put your TX with exactly the bit order you like into the block just where you want it.  

The whitepaper details it a bit more. In laymans terms, tx malleability allows interesting and complex attack vectors on non-confirmed transactions. Since the Lightning network is a caching layer where contracts are made between lightning nodes before confirmations appear than one has to assume all implications in a hostile environment. Fixing malleability allows for decentralized and untrusted parties to cache these tx's. In order to cache tx with malleability than one has to have centralized sources of trust which basically amounts to a coinbase/circle model of off the chain txs. Centralized off the chain solutions do provide a valuable service to our ecosystem but have heavy inherent regulatory, human , and insurance overhead. If bitcoin is to compete with other payment systems and fullfil its true vision it must eliminate these inefficient and corruptible sources of security.

[johnyj]

Sorry I am slow.  What won't be as efficient?  I guess I don't understand the problem exactly.

https://blog.bitgo.com/malevolent-malleability/  <-- I see here a list of how folks who didn't know what they were doing could make mistakes, and a conclusion: "The general consensus up to this point has been that malleability is an annoying but not critical system-wide problem. "

Anyway, if for some reason you can't figure out a way to actually check that funds have been sent and arrived at the proper address, or to not reference any TXs by number, you can always pay a miner a few extra milliies to put your TX with exactly the bit order you like into the block just where you want it.  

The whitepaper details it a bit more. In laymans terms, tx malleability allows interesting and complex attack vectors on non-confirmed transactions. Since the Lightning network is a caching layer where contracts are made between lightning nodes before confirmations appear than one has to assume all implications in a hostile environment. Fixing malleability allows for decentralized and untrusted parties to cache these tx's. In order to cache tx with malleability than one has to have centralized sources of trust which basically amounts to a coinbase/circle model of off the chain txs. Centralized off the chain solutions do provide a valuable service to our ecosystem but have heavy inherent regulatory, human , and insurance overhead. If bitcoin is to compete with other payment systems and fullfil its true vision it must eliminate these inefficient and corruptible sources of security.

It seems that's the main reason here, so let's drop Lightning Network

If a clearing based solution, require so much change to bitcoin's architecture, then it must be able to provide lots of benefit to worth the risk. Currently I don't really see a big difference between lightning network and traditional clearing solutions, which require no changes for bitcoin at all

BTW, I just heard that Adam Back said that you need insurance for lightning network to work properly (https://www.reddit.com/r/bitcoinxt/comments/3wty7s/dr_adam_back_believes_that_insurance_may_be/)

Ok, if lightning network need insurance to work, and traditional clearing solution also works perfect given insurance, then why don't just use existing mature clearing based solutions? I thought the biggest benefit Lightning network has against traditional clearing solution is that it requires no trust, but it seems not the case. If you need insurance to be trustworthy, then there must be some fundamental weakness with the design of lightning network. I have not looked into details about this statement, too low signal to noise ratio there, but it is very natural that LN like any new system have many security problem which only time will tell if it is a robust design

In my not-so-deep understanding of LN, they are using a similar design as NashX exchange's mutually assured destruction model to keep it trustless, however, that model does not work well under certain circumstances. That's also the reason those so called P2P exchanges can not gain any momentum against localbitcoins: You eventually need an authority to solve a complex dispute, blockchain can not be this authority since it lacks judgement

Ok, everyone can go home and sleep, no work needs to be done, bitcoin is perfect, just raise the block size limit to 2MB for the time being  

[RoadTrain]

Sorry I am slow.  What won't be as efficient?  I guess I don't understand the problem exactly.

https://blog.bitgo.com/malevolent-malleability/  <-- I see here a list of how folks who didn't know what they were doing could make mistakes, and a conclusion: "The general consensus up to this point has been that malleability is an annoying but not critical system-wide problem. "

Anyway, if for some reason you can't figure out a way to actually check that funds have been sent and arrived at the proper address, or to not reference any TXs by number, you can always pay a miner a few extra milliies to put your TX with exactly the bit order you like into the block just where you want it.  

The whitepaper details it a bit more. In laymans terms, tx malleability allows interesting and complex attack vectors on non-confirmed transactions. Since the Lightning network is a caching layer where contracts are made between lightning nodes before confirmations appear than one has to assume all implications in a hostile environment. Fixing malleability allows for decentralized and untrusted parties to cache these tx's. In order to cache tx with malleability than one has to have centralized sources of trust which basically amounts to a coinbase/circle model of off the chain txs. Centralized off the chain solutions do provide a valuable service to our ecosystem but have heavy inherent regulatory, human , and insurance overhead. If bitcoin is to compete with other payment systems and fullfil its true vision it must eliminate these inefficient and corruptible sources of security.

It seems that's the main reason here, so let's drop Lightning Network

If a clearing based solution, require so much change to bitcoin's architecture, then it must be able to provide lots of benefit to worth the risk. Currently I don't really see a big difference between lightning network and traditional clearing solutions, which require no changes for bitcoin at all

BTW, I just heard that Adam Back said that you need insurance for lightning network to work properly (https://www.reddit.com/r/bitcoinxt/comments/3wty7s/dr_adam_back_believes_that_insurance_may_be/)

Ok, if lightning network need insurance to work, and traditional clearing solution also works perfect given insurance, then why don't just use existing mature clearing based solutions? I thought the biggest benefit Lightning network has against traditional clearing solution is that it requires no trust, but it seems not the case. If you need insurance to be trustworthy, then there must be some fundamental weakness with the design of lightning network. I have not looked into details about this statement, too low signal to noise ratio there, but it is very natural that LN like any new system have many security problem which only time will tell if it is a robust design

In my not-so-deep understanding of LN, they are using a similar design as NashX exchange's mutually assured destruction model to keep it trustless, however, that model does not work well under certain circumstances. That's also the reason those so called P2P exchanges can not gain any momentum against localbitcoins: You eventually need an authority to solve a complex dispute, blockchain can not be this authority since it lacks judgement

Ok, everyone can go home and sleep, no work needs to be done, bitcoin is perfect, just raise the block size limit to 2MB for the time being  

With that kind of attitude you might have just continued to use fiat instead of Bitcoin. But you've managed to comprehend Bitcoin. You can do the same with segwit or LN. It just means you must spend time on that, and refrain from making judgements until you're finished.

Mark Friedenbach: https://www.reddit.com/r/btc/comments/3woin3/to_adam_back_we_are_hereby_officially_requesting/cxzpcpw

There is absolutely no reason for lightning nodes to require insurance or even reputation. The block chain can be used to settle all disputes, with the non-cooperative party paying fees.

[johnyj]

With that kind of attitude you might have just continued to use fiat instead of Bitcoin. But you've managed to comprehend Bitcoin. You can do the same with segwit or LN. It just means you must spend time on that, and refrain from making judgements until you're finished.

Mark Friedenbach: https://www.reddit.com/r/btc/comments/3woin3/to_adam_back_we_are_hereby_officially_requesting/cxzpcpw

There is absolutely no reason for lightning nodes to require insurance or even reputation. The block chain can be used to settle all disputes, with the non-cooperative party paying fees.

The problem is that time is the only precious resource that is constantly reduce for anyone. If you can not present a system that is simple enough for others to fully understand in a couple of days, they will just ignore it

In a centralized organization you don't need other's understanding to enforce a change, but here ...

[BitUsher]

BTW, I just heard that Adam Back said that you need insurance for lightning network to work properly (https://www.reddit.com/r/bitcoinxt/comments/3wty7s/dr_adam_back_believes_that_insurance_may_be/)

He was discussing a hypothetical where insurance might possibly be necessary with certain conditions like many more tx and extremely cheap rates.

https://www.reddit.com/r/Bitcoin/comments/3siyff/z/cwxp1oi

The way you phrased it sounds like it would be required for the LN to even work which is very misleading.

[funkenstein]

Sorry I am slow.  What won't be as efficient?  I guess I don't understand the problem exactly.

https://blog.bitgo.com/malevolent-malleability/  <-- I see here a list of how folks who didn't know what they were doing could make mistakes, and a conclusion: "The general consensus up to this point has been that malleability is an annoying but not critical system-wide problem. "

Anyway, if for some reason you can't figure out a way to actually check that funds have been sent and arrived at the proper address, or to not reference any TXs by number, you can always pay a miner a few extra milliies to put your TX with exactly the bit order you like into the block just where you want it.  

The whitepaper details it a bit more. In laymans terms, tx malleability allows interesting and complex attack vectors on non-confirmed transactions. Since the Lightning network is a caching layer where contracts are made between lightning nodes before confirmations appear than one has to assume all implications in a hostile environment. Fixing malleability allows for decentralized and untrusted parties to cache these tx's. In order to cache tx with malleability than one has to have centralized sources of trust which basically amounts to a coinbase/circle model of off the chain txs. Centralized off the chain solutions do provide a valuable service to our ecosystem but have heavy inherent regulatory, human , and insurance overhead. If bitcoin is to compete with other payment systems and fullfil its true vision it must eliminate these inefficient and corruptible sources of security.

Thanks for your reply.  I think the details are important here.  "I have a way to make a secure lightning network if TXs were not malleable" is not the same as "I have a proof that lightning networks will only work securely if TXs are not malleable".   Sadly I'm not qualified to tell you at the moment whether one, both, or neither of these are true.

/begin rant

This "if bitcoin is to compete with other payment systems" stuff is all nonsense.  Bitcoin is a unit of account.  A public, verifiable, noncounterfeitable unit of account.  Visa, Swift, and ACH don't offer that.  There's no competition. All of these work on top of bitcoin just as well (or better) than they did on top of funbux.  .    ,

/end rant

Anyway, even if someone were to come up with a proof that certain forms of payment channels are only possible with a hardfork level protocol change..  it still wouldn't happen.  I'm still interested though!     

[pereira4]

https://www.youtube.com/watch?v=NOYNZB5BCHM

Peter Wuille: Segregated witness and its impact on scalability at SF Meetup with more time for questions and clarifications.

I would love to listen to this since the latest lecture was very limited in time as there was a bunch of other speakers in queue, but I just can't stand having to hear a voice from only one of my headphones...

Is there a way to make the sound mono? It's not that hard to do it before you upload it, so much for tech geniuses that fail to duplicate the left audio channel into the right one before uploading

[jbreher]

https://www.youtube.com/watch?v=NOYNZB5BCHM

Peter Wuille: Segregated witness and its impact on scalability at SF Meetup with more time for questions and clarifications.

I would love to listen to this since the latest lecture was very limited in time as there was a bunch of other speakers in queue, but I just can't stand having to hear a voice from only one of my headphones...

Is there a way to make the sound mono? It's not that hard to do it before you upload it, so much for tech geniuses that fail to duplicate the left audio channel into the right one before uploading

First world problems. Just sayin'.

You could always just short the left hot to the right hot. An adapter could be soldered up in five minutes tops, and that would include the time spent rooting around for the soldering iron. Or go ghetto and slice into the leads, twisting the wires together.

^(partially) in jest.