Segwit details? N + 2*numtxids + numvins > N, segwit uses more space than 2MB HF

[xyzzy099]

[EDIT]How does it not help scaling, if it increases the number of transactions that can be included in each block?

Block size is easy to change. There's an arguably-popular client (Bitcoin Classic) that solves that problem today. To help scaling you need to invent tech to make running a full node easier, such as thin-blocks or IBLT. Shameless plug: I recently produced a video on Xtreme Thin Blocks.

That may be true, but you didn't answer the question I asked (See above).  I don't think segwit is being proposed as the solution to scaling...  I don't think it was really meant to be a scaling solution at all, really - the increased transaction capacity is just a side-effect, right?

[1715065045]

1715065045

[xyzzy099]

My point, perhaps poorly expressed, was that if you think these problems are 'not hard', you must have solutions in mind, no?  I'd be interested in hearing your ideas.  I am genuinely interested, not being sarcastic here.

It wasn't only me that had those solutions in mind. In fact they are already included in the "segregated witness" proposal, but without the "segregation" part. The "segregation" just splits the transaction in two parts. In fact one could come up with a deficient "segregated witness" proposal that wouldn't fix the discussed problems. They are orthogonal concepts.
 

Which solutions are you referring to here?

[sgbett]

I asked some of these questions 3 months ago.  Never got a decent answer.

Blockstream wants soft-forked SegWit to fix the malleability problems (that would be needed for the LN, if they ever get it to work), and to force ordinary p2p bitcoin users subsidize the costs of complicated multisig transactions (ditto).  But these reasons do not seem explain the urgency and energy that they are putting on the SegWit soft fork.  Maybe they have other undeclared reasons?  Perhaps they intend to stuff more data into the extension records, which they would not have to justify or explain since, being in the extension part, "ordinary users can ignore it anyway"?

As for SegWit being a soft fork, that is technically true; but a soft fork can do some quite radical changes, like imposing a negative interest (demurrage) tax, or raising the 21 million limit.  One could also raise the block size limit that way.  These tricks would all let old clients work for a while, but eventually everybody will be forced to upgrade to use coins sent by the new verson.

A hard fork based consensus mechanism, far from being dangerous, is actually the solution to centralised control over consensus.

Script versioning is essentially about changing this consensus mechanism so that any change can be made without any consensus. Giving this control to anyone, even satoshi himself, entirely undermines the whole idea of bitcoin. *Decentralised* something something.

[bold]Script versioning[/bold]
Changes to Bitcoin’s script allow for both improved security and improved functionality. However, the design of script only allows backwards-compatible (soft-forking) changes to be implemented by replacing one of the ten extra OP_NOP opcodes with a new opcode that can conditionally fail the script, but which otherwise does nothing. This is sufficient for many changes – such as introducing a new signature method or a feature like OP_CLTV, but it is both slightly hacky (for example, OP_CLTV usually has to be accompanied by an OP_DROP) and cannot be used to enable even features as simple as joining two strings.

Segwit resolves this by including a version number for scripts, so that additional opcodes that would have required a hard-fork to be used in non-segwit transactions can instead be supported by simply increasing the script version.

It doesn't matter where you stand on the blocksize debate, which dev team you support, or any of the myriad disagreements. As Gregory Maxwell himself states:

"Anyone who /understood/ it would [shut down bitcoin], if somehow control of it were turned over to them."

[rizzlarolla]

my analysis so far is that it creates a much more complicated error prone system with potential attack vectors that is not peer reviewed that reduces the ability to scale. Maybe my problem is that I am just not smart enough to understand it well enough to appreciate it?

but in some weeks it will be softforked, so its ok, there is no need to worry about it.

All I see is that segwit tx requires more work, more space, more confusion, but we do end up where there are tx in the blockchain that need to be trusted. bitcoin becomes partly a trusted ledger, but ripple is doing fine, so why not

[snipped]

Great posts. I think your smart enough to understand segwit. If anyone can.

In "a few weeks" Bitcoin will be fundamentally changed by segwit. (soft forked by core, Bitcoin guardians)

I agree with your earlier comment. Segwit must be abandoned postponed.

2mb blocks first, soon, then reassess segwit. At least hard fork.
(core could do this, 2mb blocks are road mapped in core?)

Segwit is not my bitcoin. Not at this point in time at least.
"a much more complicated error prone system with potential attack vectors that is not peer reviewed"

[gmaxwell]

Wow. The deceptive misinformation in this thread is really astonishing.

Contrary to the claims here, segwit doesn't increase transaction sizes (as was noted, it adds a single coinbase commitment per block).

all this seems to be above and beyond what would be needed for the normal, plus the nVersion (4 bytes) and nLockTime (4 bytes) are duplicated. To a simple C programmer like me it sure looks like instead of reducing the net amount as required by anything claiming to save space, it is increasing the size by approx 50 bytes.

Maybe its 32 + 4 + 1 + 1 + 4, so 42 bytes?

jl777, to be blunt, and offer some unsolicited advice: You have almost no chance of actually writing that bitcoin full node you say you want to be working on when you are so unwilling to spend more than a second reading or take any time at all to understand how existing Bitcoin software works.  Virtually every post of yours contains one or another fundamental misunderstanding of the existing system/software-- and your abrasive an accusatory approach leave other people disinterested in spending their time educating you. Even here, I am not responding to for your benefit-- as I would otherwise-- but because other people are repeating the misinformation you've unintentionally generated due to your ignorance. Please take a step back: Bitcoin is not "bitcoin dark", "nxt", or the other altcoins you've worked on in the past where an abusive/armwaving style that leans heavily on native intelligence while eschewing study will itself establish you as an expert. Bitcoin is full of really remarkably intelligent people, so simply being smarter than average doesn't make you a shining star as it may in some places.

The text you are quoting is instructions on computing a hash. None of the data involved in it is stored, not any more than the tens of times the transaction size of data used for the sighashes on a large transaction is stored.

If the carefully constructed, peer reviewed specifications are not to your liking; you could also spend some time studying the public segnet testnet. Given that there are both specifications and a running public network, the continued inquisitory "needs to be answered" conspiracy theory nonsense-- even after being given a _direct_ and specific answer ("segwit adds one coinbase output that contains the root hash of the Merkle tree that commits to the witness transaction ids. It uses 47 extra bytes per block")-- is highly inappropriate. Please do not subject other contributors to this forum to that kind of hostility.  

My assumption is that for a segwit compatible full relaying node to be able to relay the full blockchain it would need to have ALL the data, original blockchain and witness data.

Your lack of understanding about how Bitcoin is structured and existed today works against you. A full node does not need to store "ALL the data", and in Bitcoin Core today you can set an option and run a full node with only about 2GB storage. Configured in this pruning manner, the node relays transactions, blocks, fully validates everything, etc.  This is the state _today_.

Segwit improves scaling in several ways as was already explained in this thread:

• The extra space used by witnesses is more prunable than normal block space, as it's not needed by non-validating clients.
• Has less effect on bandwidth, as light clients don't need the witness data.
• Has no effect on the UTXO set, so does not contribute to database growth and/or churn.
• Enables script versions, which will make the introduction of Schnorr signatures much easier later on, which are more space efficient than what we have now (even for simple single-key outputs/inputs).

For example, all existing full node software that I'm aware of widely used on the current network does not validate signatures in the far past chain. They just download them, and if pruning is enabled, throw them out. They can't verify the transaction hashes, make sure no inflation or other non-signature validation rule violations happened, and build their UTXO set without downloading them... but the download is pure waste. Segwit makes it possible for a node which isn't going to verify all the signatures in the fast past to skip downloading them.  Segwit reduces greatly the bandwidth required to service lite nodes for a given amount of transactions, segwit increases the capacity (in terms of transactions per block) without increasing the amount of UTXO growth per block... and all this on top of the non-scaling related improvements it brings.

This is why the technical space around Bitcoin is overwhelmingly in favor of it

Script versioning is essentially about changing this consensus mechanism so that any change can be made without any consensus. Giving this control to anyone, even satoshi himself, entirely undermines the whole idea of bitcoin. *Decentralised* something something.

The content of your scriptpubkey, beyond the resource costs to the network, is a private contract between the sender of the funds and the receiver of the funds. It is only the business of these parties, no one else. Ideally, it would not be subject to "consensus", in any way/shape/form-- it is a _private matter_. It is not any of your business how I spend my Bitcoins; but unfortunately, script enhancing softforks do require consensus of at least the network hashpower.

Bitcoin Script was specifically designed because how the users contract with it isn't the network's business-- though it has limitations. And, fundamentally, even with those limitations it already, at least theoretically, impossible to prevent users from contracting however they want. For example, Bitcoin has no Sudoku implementation in Script, and yet I can pay someone conditionally on them solving one (or any other arbitrary program).

Bitcoin originally had an OP_VER to enable versioned script upgrades. Unfortunately, the design of this opcode was deeply flawed-- it allowed any user of the network, at their unannounced whim, to hardfork the network between different released versions of Bitcoin.  Bitcoin's creator, removed it and in its place put in facilities for softforks. Softforks have been used many times to compatibly extend the system-- first by Bitcoin's creator, and later by the community. The segwit script versioning brings back OP_VER but with a design that isn't broken---- it makes it faster and safer to design and deploy smart contracting/script improvements (for example, a recently proposed one will reduce transaction sizes by ~30% with effectively no costs once deployed); but doesn't change the level of network consensus required to deploy softforks; only perhaps the ease of achieving the required consensus because the resulting improvements are safer.

If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011 (though it's unclear how any blockchain cryptocurrency could _prevent_ this from happening).  Your views, if not just based on simple misunderstandings, are totally disjoint with how Bitcoin works. I don't begrudge you the freedom to want weird or even harmful things-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--, but Bitcoin isn't the place for them, and the restrictions you're asking for appear to be deeply disjoint with Bitcoin's day-one and every-day-since design, which has a huge amount of complexity in the original design for user (not consensus) determined smart contracting and where softforks (hashpower consensus) have been frequently used to extend the system.

[achow101]

Oh, the amount of misinformation in this thread!!

jl777 you have misunderstood the segwit BIPs.

Transaction ID

A new data structure, witness, is defined. Each transaction will have 2 IDs.

Definition of txid remains unchanged: the double SHA256 of the traditional serialization format:

  [nVersion][txins][txouts][nLockTime]
  
A new wtxid is defined: the double SHA256 of the new serialization with witness data:

  [nVersion][marker][flag][txins][txouts][witness][nLockTime]

from the BIP...

the wtxid is based on all of the original, plus marker (1 byte?) flag (1 byte) and witness, which appears to be:

 1-byte - OP_RETURN (0x6a)
   1-byte - Push the following 36 bytes (0x24)
   4-byte - Commitment header (0xaa21a9ed)
  32-byte - Commitment hash: Double-SHA256(witness root hash|witness nonce)

all this seems to be above and beyond what would be needed for the normal, plus the nVersion (4 bytes) and nLockTime (4 bytes) are duplicated. To a simple C programmer like me it sure looks like instead of reducing the net amount as required by anything claiming to save space, it is increasing the size by approx 50 bytes.

NO NO NO!!!!. AND PLEASE STOP SPREADING THIS!! IT IS WRONG.

The above with the OP_RETURN is how the witness root hash (the hash of the wtxids where the wtxids are leaves on a hash tree). This is added to the Coinbase transaction of the block. The point of doing this is to commit the witnesses to the blockchain without adding all of the witnesses to the size calculation of the blockchain except for these 38 bytes.

The wtxid (also called witness hash) is the hash of the transaction with all of the witness data and the transaction data. The transaction format if witness serialization is specified is 4 bytes for the transaction version (currently used), 1 byte for a marker (new, is always 0), 1 byte for a flag (new), the input count (currently used), the inputs (currently used), the output count (currently used), the outputs (currently used), the witnesses (new), and the locktime (currently used). All of this is what is hashed to get the wtxid. However, the regular txid is just the hash of everything that is currently used (so it contains none of the new stuff) so as to maintain backwards compatibility. If witness serialization is not specified then only the currently used stuff is sent in the tx message.

Likewise, with blocks, from what I understand, the block sent will contain the new transaction format if witness serialization is specified. Otherwise it will just include the transaction data that is currently used so that it maintains backwards compatibility. The only "bloat" caused by segwit is the 38 bytes in a second output of the Coinbase to commit the wtxids in a similar manner to the way that regular txids are committed and 2 extra bytes per transaction. Only upgraded nodes would get the witness data and those 2 extra bytes.

P.S. So my understanding is that you need a special segwit address (that is somehow determined to be a segwit address using what mechanism?) so both sender and receiver need to already have the segwit version. I guess just ignoring all the existing nodes is at least some level of backward compatibility. But are you sure all users will quickly get used to having to deal with two types of addresses for every transaction and they will make sure they know what version the other party is running. Doesnt this bifurcate the bitcoin universe? maybe the name should be "bifurcating softfork"

The addresses are different. An upgraded node will use p2sh addresses, which we currently use. Those p2sh addresses can be spent to using the current methods so non-upgraded users can still spend to those addresses. To spend from those addresses requires segwit, and the way that the scriptsig is set up, non-upgraded nodes will always validate those transactions as valid even if the witness (which those nodes cannot see) is invalid. Those segwit transactions still create outputs the regular way, so they can still send to p2pkh or p2pk outputs which non-upgraded users can still receive and spend from. Segwit transactions are considered by old nodes as transactions which spent an anyonecanspend output and thus are treated with a grain of salt. The best course of action is to of course wait for confirmations as we already should still be doing now.

Whoa, whoa, wait...

From the point of view of old clients, segwit adds one coinbase output that contains the root hash of the Merkle tree that commits to the witness transaction ids. It uses 47 extra bytes per block, so technically, yes, it "wastes precious blockchain space".

So, 47 bytes per block. That's not too unreasonable. But...

This then gets us to my question that is not being answered. On average, how many bytes in the blockchain will be needed for a standard payment sent via segwit?

Is this ever less than it would be now?
Is this ever the same as it is now?
Is this usually about 50 bytes more per tx?

50 bytes per transaction for fully-validating nodes? This needs to be answered.

I would like to see an answer to this too, it seems everyone is avoiding this question. Fully validating nodes are very important for those of us that want to verify the blockchain ourselves and are required for bootstrapping new nodes.

01000000000102fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f0 0000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5c dd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eef fffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000 ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9 093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac000247304402 203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4 518331561406f90300e8f3358f51928d43c212a8caed02de67eebee0121025476c2e83188368da1 ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee635711000000

the above is from https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki
it is a 2 input 2 output tx in the witness space.

In addition to the above, the much smaller anyonecan spend tx is needed too. I think it will be about 100 bytes?

so we have a combined space of around 800 bytes against the 1000 bytes the usual 2 input/2 output tx occupies. Or was that 400 bytes that the 2input/2output tx takes?

Again, NO. See above.

Wow. The deceptive misinformation in this thread is really astonishing.

Ah *sigh of relief*; here comes somebody who actually knows what they are talking about.

Could you also let me know if I presented any misinformation? I have been trying my best to not and to make jl777 understand why he is wrong but I may have accidentally (either due to misunderstanding the BIPs or just really bad typing) given him false information.

If the carefully constructed, peer reviewed specifications are not to your liking; you could also spend some time studying the public segnet testnet.

Since Pieter and no one on the IRC reponded either, I will ask this again. Will there be a full write up (preferably before segwit's release) of all of the changes that segwit entails so that wallet developers can get working on implementing segwit? AFAIK the segwit implementation contains omissions and changes from what was specified in the BIPs.

[jl777]

I apologize for my technical ineptitudes.

I am trying to understand how segwit saves blockchain space as that is what it is being marketed as and with a softfork in the upcoming weeks.

So i look at the BIP, I find each segwit tx needs a commitment hash, plus more. this is 32 bytes per tx that to my ignorant simple minded thinking is in addition to what would be needed if it was a normal tx.

Now I am being told that we dont need to count the space used by the witness data. This confuses me. I like to count all permanently required data as the permanent space cost.

I apologize for asking questions like this, but I am just a simple C programmer trying to allocate space for the segwit and noticing it seems to take more space for every tx. I await to be properly learned about how the commitment hash is not actually needed to be stored anywhere and how that would still allow a full node to properly validate all the transactions.

Or did I misunderstand? Does segwit mean that it is no business of full nodes to verify all the transactions?

James

****
OK, it was made clear that the commitment hash is just for the wtxid calculation so it doesnt really exist anywhere. The cost is 2 bytes per tx and 1 byte per vin. Still it is increasing HDD space used permanently, which is what confused me as segwit was marketed as saving HDD space and helping with scaling

[jl777]

Oh, the amount of misinformation in this thread!!

jl777 you have misunderstood the segwit BIPs.

Transaction ID

A new data structure, witness, is defined. Each transaction will have 2 IDs.

Definition of txid remains unchanged: the double SHA256 of the traditional serialization format:

  [nVersion][txins][txouts][nLockTime]
  
A new wtxid is defined: the double SHA256 of the new serialization with witness data:

  [nVersion][marker][flag][txins][txouts][witness][nLockTime]

from the BIP...

the wtxid is based on all of the original, plus marker (1 byte?) flag (1 byte) and witness, which appears to be:

 1-byte - OP_RETURN (0x6a)
   1-byte - Push the following 36 bytes (0x24)
   4-byte - Commitment header (0xaa21a9ed)
  32-byte - Commitment hash: Double-SHA256(witness root hash|witness nonce)

all this seems to be above and beyond what would be needed for the normal, plus the nVersion (4 bytes) and nLockTime (4 bytes) are duplicated. To a simple C programmer like me it sure looks like instead of reducing the net amount as required by anything claiming to save space, it is increasing the size by approx 50 bytes.

NO NO NO!!!!. AND PLEASE STOP SPREADING THIS!! IT IS WRONG.

The above with the OP_RETURN is how the witness root hash (the hash of the wtxids where the wtxids are leaves on a hash tree). This is added to the Coinbase transaction of the block. The point of doing this is to commit the witnesses to the blockchain without adding all of the witnesses to the size calculation of the blockchain except for these 38 bytes.

The wtxid (also called witness hash) is the hash of the transaction with all of the witness data and the transaction data. The transaction format if witness serialization is specified is 4 bytes for the transaction version (currently used), 1 byte for a marker (new, is always 0), 1 byte for a flag (new), the input count (currently used), the inputs (currently used), the output count (currently used), the outputs (currently used), the witnesses (new), and the locktime (currently used). All of this is what is hashed to get the wtxid. However, the regular txid is just the hash of everything that is currently used (so it contains none of the new stuff) so as to maintain backwards compatibility. If witness serialization is not specified then only the currently used stuff is sent in the tx message.

Likewise, with blocks, from what I understand, the block sent will contain the new transaction format if witness serialization is specified. Otherwise it will just include the transaction data that is currently used so that it maintains backwards compatibility. The only "bloat" caused by segwit is the 38 bytes in a second output of the Coinbase to commit the wtxids in a similar manner to the way that regular txids are committed and 2 extra bytes per transaction. Only upgraded nodes would get the witness data and those 2 extra bytes.

P.S. So my understanding is that you need a special segwit address (that is somehow determined to be a segwit address using what mechanism?) so both sender and receiver need to already have the segwit version. I guess just ignoring all the existing nodes is at least some level of backward compatibility. But are you sure all users will quickly get used to having to deal with two types of addresses for every transaction and they will make sure they know what version the other party is running. Doesnt this bifurcate the bitcoin universe? maybe the name should be "bifurcating softfork"

The addresses are different. An upgraded node will use p2sh addresses, which we currently use. Those p2sh addresses can be spent to using the current methods so non-upgraded users can still spend to those addresses. To spend from those addresses requires segwit, and the way that the scriptsig is set up, non-upgraded nodes will always validate those transactions as valid even if the witness (which those nodes cannot see) is invalid. Those segwit transactions still create outputs the regular way, so they can still send to p2pkh or p2pk outputs which non-upgraded users can still receive and spend from. Segwit transactions are considered by old nodes as transactions which spent an anyonecanspend output and thus are treated with a grain of salt. The best course of action is to of course wait for confirmations as we already should still be doing now.

Whoa, whoa, wait...

From the point of view of old clients, segwit adds one coinbase output that contains the root hash of the Merkle tree that commits to the witness transaction ids. It uses 47 extra bytes per block, so technically, yes, it "wastes precious blockchain space".

So, 47 bytes per block. That's not too unreasonable. But...

This then gets us to my question that is not being answered. On average, how many bytes in the blockchain will be needed for a standard payment sent via segwit?

Is this ever less than it would be now?
Is this ever the same as it is now?
Is this usually about 50 bytes more per tx?

50 bytes per transaction for fully-validating nodes? This needs to be answered.

I would like to see an answer to this too, it seems everyone is avoiding this question. Fully validating nodes are very important for those of us that want to verify the blockchain ourselves and are required for bootstrapping new nodes.

01000000000102fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f0 0000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5c dd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eef fffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000 ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9 093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac000247304402 203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4 518331561406f90300e8f3358f51928d43c212a8caed02de67eebee0121025476c2e83188368da1 ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee635711000000

the above is from https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki
it is a 2 input 2 output tx in the witness space.

In addition to the above, the much smaller anyonecan spend tx is needed too. I think it will be about 100 bytes?

so we have a combined space of around 800 bytes against the 1000 bytes the usual 2 input/2 output tx occupies. Or was that 400 bytes that the 2input/2output tx takes?

Again, NO. See above.

OK, I apologize again for misunderstanding the BIPs
that is why I am posting questions.

can you show me a specific rawtxbytes for a simple tx as it is now and what the required bytes are if it was a segwit? and for the segwit case it would need to be two sets of bytes, I think I understand that part well enough.

If the combined overhead is as small as 3 bytes, per tx, then it is amazing. but I dont understand how it is done.

But still even if it is only 3 bytes more, it is more data required permanently, so it is reducing the total tx capacity per MB and it anti-scaling from that standpoint. I cannot speak to the scaling impact of optimizd signature handling, etc.

Also, it seems a 2MB hardfork is much, much simpler and provides the same or more tx capacity. Why isnt that done before segwit?

[achow101]

I apologize for my technical ineptitudes.

I am trying to understand how segwit saves blockchain space as that is what it is being marketed as and with a softfork in the upcoming weeks.

It saves space in that it does not require that the signatures be downloaded. Since clients that are syncing do not need verify the signatures of old transactions, the witness data is then not needed. This would mean that they do not need to download the witnesses and can thus save bandwidth and space. This is only for several years in the future after segwit has been used for a while.

So i look at the BIP, I find each segwit tx needs a commitment hash, plus more. this is 32 bytes per tx that to my ignorant simple minded thinking is in addition to what would be needed if it was a normal tx.

Not every transaction needs that hash, just like how not every transaction now has its txid stored in the blockchain. Just like it is done today, the merkle root of the wtxids is stored, and that is it.

Now I am being told that we dont need to count the space used by the witness data. This confuses me. I like to count all permanently required data as the permanent space cost.

The witnesses are not permanent and can be deleted. After a few years, they won't even need to be downloaded.

I apologize for asking questions like this, but I am just a simple C programmer trying to allocate space for the segwit and noticing it seems to take more space for every tx. I await to be properly learned about how the commitment hash is not actually needed to be stored anywhere and how that would still allow a full node to properly validate all the transactions.

Checking the transaction hash is not part of the validation of transactions. Rather the signature is validated with the witness data and checks that the signature is valid for that transaction. The block contains a merkle root. This merkle root is calculated by hashing all of the transaction hashes. The same is done for the witness root hash which is just the hash of all of the wtxids.

Or did I misunderstand? Does segwit mean that it is no business of full nodes to verify all the transactions?

See above.

OK, I apologize again for misunderstanding the BIPs
that is why I am posting questions.

It is okay to ask questions, but when you spread around false information (like the title of this thread) then it is not okay. I admit, sometimes I do say false things, but I usually do say things with stuff like "I think" or "from what I understand" to indicate that what I say may not be the truth.

can you show me a specific rawtxbytes for a simple tx as it is now and what the required bytes are if it was a segwit? and for the segwit case it would need to be two sets of bytes, I think I understand that part well enough.

Kind of, I can't build one right now and I don't have a segnet node set up yet so I can't pull one from there. I will give you a similar example, pulled from the BIP. In this example there are two inputs, one from p2pk (currently used) and one from a p2wpkh (segwit but this format will probably not be used).

Take for example this transaction:

Code:

01000000000102fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f00000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eeffffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac000247304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee0121025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee635711000000

Here it is broken down:

Code:

nVersion:  01000000
    marker:    00
    flag:      01
    txin:      02 fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f 00000000 494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01 eeffffff
                  ef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a 01000000 00 ffffffff
    txout:     02 202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac
                  9093510d00000000 1976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac
    witness    00
               02 47304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee01 21025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee6357
    nLockTime: 11000000

An upgraded node would request and receive this transaction like this. It would contain all of the witness data.

But a non-upgraded node (and a node that doesn't want witness data) would only receive

Code:

0100000002fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f00000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eeffffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac11000000

which is considerably smaller than the witness serialization because it doesn't include the witness.

If the combined overhead is as small as 3 bytes, per tx, then it is amazing. but I dont understand how it is done.

But still even if it is only 3 bytes more, it is more data required permanently, so it is reducing the total tx capacity per MB and it anti-scaling from that standpoint. I cannot speak to the scaling impact of optimizd signature handling, etc.

Those three bytes (and whatever other overhead because I am fairly sure there is other overhead in the witness data) are not part of the transaction that goes into the count of the size of a block.

Also, it seems a 2MB hardfork is much, much simpler and provides the same or more tx capacity. Why isnt that done before segwit?

Because Segwit adds more useful stuff and 2 Mb hard fork doesn't solve the O(n^2) hashing problem. The Classic 2 Mb hard fork had some weird hackish workaround for the O(n^2) hashing problem. In terms of lines of code, I believe that there was an analysis done on segwit and the classic hard fork that found that the amount of code lines added was roughly the same. This is because the hashing problem needed to be fixed with that hard fork.

I suppose you could also say that segwit is more "efficient". Segwit, with roughly the same amount of code as the classic hard fork, brings much more functionality to Bitcoin than a 2 Mb hard fork does.

[rizzlarolla]

Wow. The deceptive misinformation in this thread is really astonishing.

snip

If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011 (though it's unclear how any blockchain cryptocurrency could _prevent_ this from happening).  Your views, if not just based on simple misunderstandings, are totally disjoint with how Bitcoin works. I don't begrudge you the freedom to want weird or even harmful things-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--, but Bitcoin isn't the place for them,

That is so wrong.

"If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011"

I don't see jl777 sgbett arguing that. You want hashpower, consencus or not, to blind existing nodes. Introduce trust or obscurity. A hard fork in disguise.


"-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--"

What has jl777 sgbett done to harm Bitcoin?
Segwit however could destroy it. (unproven obviously. just opinion)

Something better will come in due course.
Segwit needs more thought.

Segwit needs to be hard forked.


edited - gmax does appear to be responding to sgbett. apologies for wrong accreditation. 

[jl777]

I apologize for my technical ineptitudes.

I am trying to understand how segwit saves blockchain space as that is what it is being marketed as and with a softfork in the upcoming weeks.

It saves space in that it does not require that the signatures be downloaded. Since clients that are syncing do not need verify the signatures of old transactions, the witness data is then not needed. This would mean that they do not need to download the witnesses and can thus save bandwidth and space. This is only for several years in the future after segwit has been used for a while.

So i look at the BIP, I find each segwit tx needs a commitment hash, plus more. this is 32 bytes per tx that to my ignorant simple minded thinking is in addition to what would be needed if it was a normal tx.

Not every transaction needs that hash, just like how not every transaction now has its txid stored in the blockchain. Just like it is done today, the merkle root of the wtxids is stored, and that is it.

Now I am being told that we dont need to count the space used by the witness data. This confuses me. I like to count all permanently required data as the permanent space cost.

The witnesses are not permanent and can be deleted. After a few years, they won't even need to be downloaded.

I apologize for asking questions like this, but I am just a simple C programmer trying to allocate space for the segwit and noticing it seems to take more space for every tx. I await to be properly learned about how the commitment hash is not actually needed to be stored anywhere and how that would still allow a full node to properly validate all the transactions.

Checking the transaction hash is not part of the validation of transactions. Rather the signature is validated with the witness data and checks that the signature is valid for that transaction. The block contains a merkle root. This merkle root is calculated by hashing all of the transaction hashes. The same is done for the witness root hash which is just the hash of all of the wtxids.

Or did I misunderstand? Does segwit mean that it is no business of full nodes to verify all the transactions?

See above.

OK, I apologize again for misunderstanding the BIPs
that is why I am posting questions.

It is okay to ask questions, but when you spread around false information (like the title of this thread) then it is not okay. I admit, sometimes I do say false things, but I usually do say things with stuff like "I think" or "from what I understand" to indicate that what I say may not be the truth.

can you show me a specific rawtxbytes for a simple tx as it is now and what the required bytes are if it was a segwit? and for the segwit case it would need to be two sets of bytes, I think I understand that part well enough.

Kind of, I can't build one right now and I don't have a segnet node set up yet so I can't pull one from there. I will give you a similar example, pulled from the BIP. In this example there are two inputs, one from p2pk (currently used) and one from a p2wpkh (segwit but this format will probably not be used).

Take for example this transaction:

Code:

01000000000102fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f00000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eeffffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac000247304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee0121025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee635711000000

Here it is broken down:

Code:

nVersion:  01000000
    marker:    00
    flag:      01
    txin:      02 fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f 00000000 494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01 eeffffff
                  ef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a 01000000 00 ffffffff
    txout:     02 202cb20600000000 1976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac
                  9093510d00000000 1976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac
    witness    00
               02 47304402203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4518331561406f90300e8f3358f51928d43c212a8caed02de67eebee01 21025476c2e83188368da1ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee6357
    nLockTime: 11000000

An upgraded node would request and receive this transaction like this. It would contain all of the witness data.

But a non-upgraded node (and a node that doesn't want witness data) would only receive

Code:

0100000002fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f00000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5cdd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eeffffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac11000000

which is considerably smaller than the witness serialization because it doesn't include the witness.

If the combined overhead is as small as 3 bytes, per tx, then it is amazing. but I dont understand how it is done.

But still even if it is only 3 bytes more, it is more data required permanently, so it is reducing the total tx capacity per MB and it anti-scaling from that standpoint. I cannot speak to the scaling impact of optimizd signature handling, etc.

Those three bytes (and whatever other overhead because I am fairly sure there is other overhead in the witness data) are not part of the transaction that goes into the count of the size of a block.

Also, it seems a 2MB hardfork is much, much simpler and provides the same or more tx capacity. Why isnt that done before segwit?

Because Segwit adds more useful stuff and 2 Mb hard fork doesn't solve the O(n^2) hashing problem. The Classic 2 Mb hard fork had some weird hackish workaround for the O(n^2) hashing problem. In terms of lines of code, I believe that there was an analysis done on segwit and the classic hard fork that found that the amount of code lines added was roughly the same. This is because the hashing problem needed to be fixed with that hard fork.

I suppose you could also say that segwit is more "efficient". Segwit, with roughly the same amount of code as the classic hard fork, brings much more functionality to Bitcoin than a 2 Mb hard fork does.

my reaction was based on the answers I was getting and clearly it is a complex issue. segwit is arguably more changes to bitcoin than all prior BIP's combined. I dont think anybody would say otherwise.

Now please ignore the space savings for nodes that are not full nodes. I am assuming that to bootstrap a node it will need to get the witness data from somewhere, right? so it is needed permanently and thus part of the permanent HDD requirement.

I still dont fully understand how the size of the truncated tx+ witness data is as small as 2 bytes per tx + 1 byte per vin. But even if that is the case, my OP title is accurate. as N+2*numtx+numvins is more than N

that is the math I see.

Also I made the mistake of making sure the transaction hash matches for a transaction. I had assumed that if the transaction hash doesnt match, it is invalid rawbytes. Are you saying that we dont need to verify that the transaction hashes match? As you know verifying signatures is very time consuming compared to verifying txid. So if verifying txid is not available anymore, that would dramatically increase the CPU load for any validating node.

Before I go making new threads about that, let us wait for some clarity on this issue.

I think if the witness data is assumed to be there permanently, then we dont increase the CPU load 10x or more to have to validate sigs vs validate txid, so it would be a moot point.

but it negates your point that the witness data can just go away

James

[achow101]

my reaction was based on the answers I was getting and clearly it is a complex issue. segwit is arguably more changes to bitcoin than all prior BIP's combined. I dont think anybody would say otherwise.

I agree with that, after all, there are 6 separate BIPs for it of which 4 (or 5?) are being implemented in one go.

Now please ignore the space savings for nodes that are not full nodes. I am assuming that to bootstrap a node it will need to get the witness data from somewhere, right? so it is needed permanently and thus part of the permanent HDD requirement.

Again, no. Full nodes don't have to validate the signatures of transactions in super old blocks. This is something that Bitcoin Core currently does and will continue to do. Since it doesn't check the signatures of transactions in historical blocks those witnesses don't need to be downloaded, although I suppose they could. There will of course be people who run nodes that store all of that data (I will probably be one of those people) in case someone wants it.

I still dont fully understand how the size of the truncated tx+ witness data is as small as 2 bytes per tx + 1 byte per vin. But even if that is the case, my OP title is accurate. as N+2*numtx+numvins is more than N

that is the math I see.

From the view point of a new node some years in the future, that data isn't needed and it most certainly won't be downloaded or needed. But for now, yes, you will be storing that data but it can be pruned away like the majority of the blockchain can be now. Keep in mind that a pruned node is still a full node. It still validates and relays every transaction and block it receives. Full Nodes do not have to store the entire blockchain, they just need to download it. That extra data is also not counted as what is officially defined as the blockchain.

Also I made the mistake of making sure the transaction hash matches for a transaction. I had assumed that if the transaction hash doesnt match, it is invalid rawbytes. Are you saying that we dont need to verify that the transaction hashes match? As you know verifying signatures is very time consuming compared to verifying txid. So if verifying txid is not available anymore, that would dramatically increase the CPU load for any validating node.

Anymore? It was never done in the first place. Verifying the transaction has always been checking the signatures because the creating and verifying signatures involve the hash of the transaction.

Before I go making new threads about that, let us wait for some clarity on this issue.

I think if the witness data is assumed to be there permanently, then we dont increase the CPU load 10x or more to have to validate sigs vs validate txid, so it would be a moot point.

It won't increase the CPU load because that is what is currently being done and has always been done. In fact, signature validation in Bitcoin Core 0.12+ is significantly faster than in previous versions due to the use of libsecp256k1 which dramatically increased the performance of the validation.

[molecular]

Hold up. I'd like to hear from Wuille (one of the creators of segwit) about the size difference between a standard 2-input, 2-output transaction and its equivalent using segwit, for a fully-validating node. No need to attack with sarcasm.

BTW, I am also curious if the O(n^2) sigops issue can be solved in a much more simple way.

That leads me to another question I've been having that hasn't been answered as far as I know. If segregating the signatures out of the tx leads to a stable txid (malleability fixed), then why can't we simply fix malleability independantly by simply ingoring the signatures when hashing the txid?

[gmaxwell]

Segwit transactions are considered by old nodes as transactions which spent an anyonecanspend output and thus are treated with a grain of salt. The best course of action is to of course wait for confirmations as we already should still be doing now.

The segwit transactions are non-standard to old nodes. This means that old nodes/wallets ignore them until they are confirmed-- they don't show them in the wallet, they don't relay them, they don't mine them, so even confusion about unconfirmed transactions is avoided.

Ah *sigh of relief*; here comes somebody who actually knows what they are talking about.

Could you also let me know if I presented any misinformation? I have been trying my best to not and to make jl777 understand why he is wrong but I may have accidentally (either due to misunderstanding the BIPs or just really bad typing) given him false information.

At least the above was the only minor correction I've seen so far.

Since Pieter and no one on the IRC reponded either, I will ask this again. Will there be a full write up (preferably before segwit's release) of all of the changes that segwit entails so that wallet developers can get working on implementing segwit? AFAIK the segwit implementation contains omissions and changes from what was specified in the BIPs.

If that was you asking in #bitcoin-dev earlier, you need to wait around a bit for an answer on IRC-- I went to answer but the person who asked was gone.  BIPs are living documents and will be periodically updated as the functionality evolves. I thought they were currently up to date but haven't checked recently; make sure to look for pull reqs against them that haven't been merged yet.

my reaction was based on the answers I was getting and clearly it is a complex issue. segwit is arguably more changes to bitcoin than all prior BIP's combined. I dont think anybody would say otherwise.

I'll happily say otherwise.  It's a change of somewhat more complexity than P2SH; certainly less than all combined. The implementation, however is smaller than the BIP101 implementation (comparing with tests removed). The Bitcoin community is getting better at documenting changes, so there is more documentation written about this than many prior ones.  Conceptually segwit's changes are very simple; based on signaling in the scriptPubkey, scriptsigs can be moved to the ends of transactions, where they are not included in the txid. An additional hashtree is added to the coinbase transaction to commit to the signatures. The new scriptsigs begin with a version byte that describes how the scripts are interperted, two kinds are defined now, the rest are treated as "return true".

Now please ignore the space savings for nodes that are not full nodes. I am assuming that to bootstrap a node it will need to get the witness data from somewhere, right? so it is needed permanently and thus part of the permanent HDD requirement.

You can't "please ignore" major parts of the system scalability and hope to pose a discussion worth reading, if one is willing to ignore all the facts that disagree with them they can prove anything.  None the less, no-- right now existing full nodes do not verify signatures in the far past, but currently have to download them. Under segwit they could skip downloading them.  If you're not going to check it, there is no reason to download it-- but the legacy transaction hashing structure forces you to do so anyways; segwit fixes that.

I still dont fully understand how the size of the truncated tx+ witness data is as small as 2 bytes per tx + 1 byte per vin. But even if that is the case, my OP title is accurate. as N+2*numtx+numvins is more than N

There is no such thing as "size"-- size is always a product of how you serialize it.  An idiotic implementation could store non-segwit transactions by prepending them with a megabyte of zeros-- would I argue that segwit saves a megabyte per transaction? No.  

It's likely that implementations will end up using an extra byte per scriptsig to code the script version, though they could do that more efficiently some other way... but who cares about a byte per input? It certainly doesn't deserve an ALL CAPS forum post title-- you can make some strained argument that you're pedantically correct; that doesn't make you any less responsible for deceiving people, quite the opposite because now it's intentional. And even that byte per input exists only for implementations that don't want to do extra work to compress it (and end up with ~1 bit per transaction).

Meanwhile, that version byte makes it easy to safely deploy upgrades that reduce transaction sizes by ~30%.  What a joke that you attack this. God forbid that 'inefficient' implementations might store a byte for functionality that makes the system much more flexible and will allow saving hundreds of bytes.

Also I made the mistake of making sure the transaction hash matches for a transaction. I had assumed that if the transaction hash doesnt match, it is invalid rawbytes. Are you saying that we dont need to verify that the transaction hashes match? As you know verifying signatures is very time consuming compared to verifying txid. So if verifying txid is not available anymore, that would dramatically increase the CPU load for any validating node.
Before I go making new threads about that, let us wait for some clarity on this issue.

I think if the witness data is assumed to be there permanently, then we dont increase the CPU load 10x or more to have to validate sigs vs validate txid, so it would be a moot point.

You are still deeply confused. With segwit the witnesses-- the part containing the signature-- are not part of the transaction ID. They _must_ not be for malleability to be strongly fixed, and they really shouldn't be to optimal scalability.  This is no way increases the amount of signature validation anyone does.

(Nor does it decrease the amount of signature validation anyone does, though while you've been ranting here-- the people you're continually insulting went and shipped code that makes signature validation more than 5x faster.)

That leads me to another question I've been having that hasn't been answered as far as I know. If segregating the signatures out of the tx leads to a stable txid (malleability fixed), then why can't we simply fix malleability independantly by simply ingoring the signatures when hashing the txid?

This is what segwit effectively does, among other improvements. The first version of segwit that was created for elements alpha does _EXACTLY_ that, but there was no way to deploy that design in bitcoin because it would deeply break every piece of Bitcoin software ever written-- all nodes, all lite wallets, all thin clients, all hardware wallets, all web front ends, all block explorers, all pre-signed nlocked timed transactions, even many pieces of mining hardware; we learned about how impactful doing that was with elements alpha when it was very difficult getting existing software working with it... and for a while we didn't see any realistic way to deploy it short of rebooting the whole blockchain in a great big flag day (which would inevitably end up unintentionally confiscating some peoples' coins)-- not just a hard fork but an effective _rewrite_.  The clever part of segwit was reorganizing things a bit-- the signature field is still part of the txid but we don't use it for signatures anymore, we use a separate set of fields stapled onto the end to achieve exactly the same effect; but without blowing everything up.

[achow101]

Hold up. I'd like to hear from Wuille (one of the creators of segwit) about the size difference between a standard 2-input, 2-output transaction and its equivalent using segwit, for a fully-validating node. No need to attack with sarcasm.

BTW, I am also curious if the O(n^2) sigops issue can be solved in a much more simple way.

That leads me to another question I've been having that hasn't been answered as far as I know. If segregating the signatures out of the tx leads to a stable txid (malleability fixed), then why can't we simply fix malleability independantly by simply ingoring the signatures when hashing the txid?

I am pretty sure that the idea of segwit grew from this idea to ignore signatures in the txid calculation. This was proposed in BIP 140: https://github.com/bitcoin/bips/blob/master/bip-0140.mediawiki. It basically proposed a normalized txid which was the txid but ignoring the signatures. The other stuff in that BIP was because the author wanted it deployed as a soft fork rather than a hard fork. Otherwise a hard fork would be needed.

[molecular]

The above with the OP_RETURN is how the witness root hash (the hash of the wtxids where the wtxids are leaves on a hash tree). This is added to the Coinbase transaction of the block. The point of doing this is to commit the witnesses to the blockchain without adding all of the witnesses to the size calculation of the blockchain except for these 38 bytes.

So this is kind-of like a merge-mined witness chain?

[jl777]

my reaction was based on the answers I was getting and clearly it is a complex issue. segwit is arguably more changes to bitcoin than all prior BIP's combined. I dont think anybody would say otherwise.

I agree with that, after all, there are 6 separate BIPs for it of which 4 (or 5?) are being implemented in one go.

Now please ignore the space savings for nodes that are not full nodes. I am assuming that to bootstrap a node it will need to get the witness data from somewhere, right? so it is needed permanently and thus part of the permanent HDD requirement.

Again, no. Full nodes don't have to validate the signatures of transactions in super old blocks. This is something that Bitcoin Core currently does and will continue to do. Since it doesn't check the signatures of transactions in historical blocks those witnesses don't need to be downloaded, although I suppose they could. There will of course be people who run nodes that store all of that data (I will probably be one of those people) in case someone wants it.

I still dont fully understand how the size of the truncated tx+ witness data is as small as 2 bytes per tx + 1 byte per vin. But even if that is the case, my OP title is accurate. as N+2*numtx+numvins is more than N

that is the math I see.

From the view point of a new node some years in the future, that data isn't needed and it most certainly won't be downloaded or needed. But for now, yes, you will be storing that data but it can be pruned away like the majority of the blockchain can be now. Keep in mind that a pruned node is still a full node. It still validates and relays every transaction and block it receives. Full Nodes do not have to store the entire blockchain, they just need to download it. That extra data is also not counted as what is officially defined as the blockchain.

Also I made the mistake of making sure the transaction hash matches for a transaction. I had assumed that if the transaction hash doesnt match, it is invalid rawbytes. Are you saying that we dont need to verify that the transaction hashes match? As you know verifying signatures is very time consuming compared to verifying txid. So if verifying txid is not available anymore, that would dramatically increase the CPU load for any validating node.

Anymore? It was never done in the first place. Verifying the transaction has always been checking the signatures because the creating and verifying signatures involve the hash of the transaction.

Before I go making new threads about that, let us wait for some clarity on this issue.

I think if the witness data is assumed to be there permanently, then we dont increase the CPU load 10x or more to have to validate sigs vs validate txid, so it would be a moot point.

It won't increase the CPU load because that is what is currently being done and has always been done. In fact, signature validation in Bitcoin Core 0.12+ is significantly faster than in previous versions due to the use of libsecp256k1 which dramatically increased the performance of the validation.

I was told by gmax himself that a node that doesnt validate all signatures should call itself a fully validating node.

Also, I am making an optimized bitcoin core and one of these optimizations is rejecting a tx whose contents doesnt match the txid. The thinking being that if the hashes dont match, there is no point in wasting time calculating the signature

not sure what libsecp256k1's speed has anything to do with the fact that it is still much slower to calculate than SHA256.

So my point again, is that all witness data needs to be stored permanently for a full node that RELAYS historical blocks to a bootstrapping node. If we are to lose this, then we might as well make bitcoin PoS as that is the one weakness for PoS vs PoW. So if you are saying that we need to view bitcoin as fully SPV all the time with PoS level security for bootstrapping nodes, ok, with those assumptions lots and lots of space is saved.

However, with such drastic assumptions I can (and have) already saved lots more space without adding a giant amount of new protocol and processing.

So this controversy has at least clarified that segwit INCREASES the size of the permanently needed data for fully validating and relaying node. Of course for SPV nodes things are much improved, but my discussion is not about SPV nodes.

So the powers that be can call me whatever names they want. I still claim that:

N + 2*numtx + numvins > N

And as such segwit as way to save permanent blockchain space is an invalid claim.Now the cost of 2*numtx+numvins is not that big, so maybe it is worth the cost for all the benefits we get.

However on the benefits claims, one of them is the utxo dataset is becoming a lot more manageable. this is irrelevant as that is a local inefficiency that can be optimized without any external effects. I have it down to 4 bytes of RAM per utxo, but I could make it smaller if needed

It just seems a lot of unsupported (or plain wrong) claims are made to justify the segwit softfork. And the most massive change by far is being slipped in as a minor softfork update?

[rizzlarolla]

... and for a while we didn't see any realistic way to deploy it short of rebooting the whole blockchain in a great big flag day (which would inevitably end up unintentionally confiscating some peoples' coins)-- not just a hard fork but an effective _rewrite_.  The clever part of segwit was reorganizing things a bit-- the signature field is still part of the txid but we don't use it for signatures anymore, we use a separate set of fields stapled onto the end to achieve exactly the same effect; but without blowing everything up...

Yet.
Coming soon via core soft fork.

[jl777]

There is no such thing as "size"-- size is always a product of how you serialize it.  An idiotic implementation could store non-segwit transactions by prepending them with a megabyte of zeros-- would I argue that segwit saves a megabyte per transaction? No.  

It's likely that implementations will end up using an extra byte per scriptsig to code the script version, though they could do that more efficiently some other way... but who cares about a byte per input? It certainly doesn't deserve an ALL CAPS forum post title-- you can make some strained argument that you're pedantically correct; that doesn't make you any less responsible for deceiving people, quite the opposite because now it's intentional. And even that byte per input exists only for implementations that don't want to do extra work to compress it (and end up with ~1 bit per transaction).

Since you feel I am deceiving people by using uppercase, I changed it to lower case. The first responses I got did not make it clear the overhead was 2 bytes per tx and 1 byte per vin.

I was under the misunderstanding that segwit saved blockchain space, you know cuz I am idiotic and believed stuff on the internet.

I am glad that now we all know that a 2MB hardfork would be more space efficient than segwit as far as permanent blockchain space.

What I still dont understand is how things will work when a segwit tx is sent to a non-segwit node and that is spent to another non-segwit node. How will the existing wallets deal with that? What happens if an attacker created segwit rawtransactions and sent them to non-segwit nodes? there are no attack vectors? what about in zeroconf environments? how does a full relaying node mine a block with segwit inputs? or do existing full nodes cease to be able to mine blocks after segwit softfork?

And even a simpleton like me can understand how to increase blocksizes with a hardfork, so why not do that before adding massive new changes like segwit? especially since it is more space efficient and not prone to misunderstandings

[sgbett]

Script versioning is essentially about changing this consensus mechanism so that any change can be made without any consensus. Giving this control to anyone, even satoshi himself, entirely undermines the whole idea of bitcoin. *Decentralised* something something.

The content of your scriptpubkey, beyond the resource costs to the network, is a private contract between the sender of the funds and the receiver of the funds. It is only the business of these parties, no one else. Ideally, it would not be subject to "consensus", in any way/shape/form-- it is a _private matter_. It is not any of your business how I spend my Bitcoins; but unfortunately, script enhancing softforks do require consensus of at least the network hashpower.

Bitcoin Script was specifically designed because how the users contract with it isn't the network's business-- though it has limitations. And, fundamentally, even with those limitations it already, at least theoretically, impossible to prevent users from contracting however they want. For example, Bitcoin has no Sudoku implementation in Script, and yet I can pay someone conditionally on them solving one (or any other arbitrary program).

Bitcoin originally had an OP_VER to enable versioned script upgrades. Unfortunately, the design of this opcode was deeply flawed-- it allowed any user of the network, at their unannounced whim, to hardfork the network between different released versions of Bitcoin.  Bitcoin's creator, removed it and in its place put in facilities for softforks. Softforks have been used many times to compatibly extend the system-- first by Bitcoin's creator, and later by the community. The segwit script versioning brings back OP_VER but with a design that isn't broken---- it makes it faster and safer to design and deploy smart contracting/script improvements (for example, a recently proposed one will reduce transaction sizes by ~30% with effectively no costs once deployed); but doesn't change the level of network consensus required to deploy softforks; only perhaps the ease of achieving the required consensus because the resulting improvements are safer.

This is a really good explanation, thanks for taking the time to write it up. My understanding of Bitcoin doesn't come direct from the code (yet!) I have to rely on second hand information. The information you just provided has really deepened my understanding of the purpose of the scripting system over and above "it exists, and it makes the transactions work herp" which probably helps address your final paragraph...

If you're going to argue that you don't want a system where hashpower consensus allows new script rules for users to use to voluntarily contract with themselves, you should have left Bitcoin in 2010 or 2011 (though it's unclear how any blockchain cryptocurrency could _prevent_ this from happening).  Your views, if not just based on simple misunderstandings, are totally disjoint with how Bitcoin works. I don't begrudge you the freedom to want weird or even harmful things-- and I would call denying users the ability to choose whatever contract terms they want out of principle rather than considerations like resource usage both weird and harmful--, but Bitcoin isn't the place for them, and the restrictions you're asking for appear to be deeply disjoint with Bitcoin's day-one and every-day-since design, which has a huge amount of complexity in the original design for user (not consensus) determined smart contracting and where softforks (hashpower consensus) have been frequently used to extend the system.

As we have established my understanding was, lets say limited , then I don't think its far to say I am arguing against what it is for. I was arguing against what I though it meant. Quite the opposite to wanting weird or harmful things, I was very much arguing that we shouldn't be allowing a harmful thing! If as may be the case, that harmful thing is not an issue, then I have nothing to worry about!

I'm trying not to get (too) sucked into the conspiracy theories on either side, I'm only human though so sometimes I do end up with five when adding together two and two.

A question that still niggles me is segwit as a soft fork. I know that just dredges up the same old discussion about pros and cons of soft vs hard but for a simpleton such as me it seems that if the benefits of segwit are so clear, then compromising on the elegance of implementation in order to make it a soft fork seems a strange decision.