ah, the crosspost.

I am just so confused how being a softfork makes fundamentally changing (breaking) things ok

I did not make this a political thing.
segwit is marketed as a way to enable scaling, when it is no such thing.

my analysis so far is that it creates a much more complicated error prone system with potential attack vectors that is not peer reviewed that reduces the ability to scale. Maybe my problem is that I am just not smart enough to understand it well enough to appreciate it?

but in some weeks it will be softforked, so its ok, there is no need to worry about it.

so if the bitcoin supply is increased to 1 billion with a softfork, that's ok?

All I see is that segwit tx requires more work, more space, more confusion, but we do end up where there are tx in the blockchain that need to be trusted. bitcoin becomes partly a trusted ledger, but ripple is doing fine, so why not

I would feel much better for independent validation. Otherwise it is me verifying my own data with another dataset that I generate.

It seems that such would require that others trust that I didnt make any errors.

could you generate a dataset for balances before block 400,000, ie 0 to 399,999 processed? I can pay a reasonable bounty for it. How do you deal with multisig address balances and non-standard outputs?

James

01000000000102fff7f7881a8099afa6940d42d1e7f6362bec38171ea3edf433541db4e4ad969f0 0000000494830450221008b9d1dc26ba6a9cb62127b02742fa9d754cd3bebf337f7a55d114c8e5c dd30be022040529b194ba3f9281a99f2b1c0a19c0489bc22ede944ccf4ecbab4cc618ef3ed01eef fffffef51e1b804cc89d182d279655c3aa89e815b1b309fe287d9b2b55d57b90ec68a0100000000 ffffffff02202cb206000000001976a9148280b37df378db99f66f85c95a783a76ac7a6d5988ac9 093510d000000001976a9143bde42dbee7e4dbe6a21b2d50ce2f0167faa815988ac000247304402 203609e17b84f6a7d30c80bfa610b5b4542f32a8a0d5447a12fb1366d7f01cc44a0220573a954c4 518331561406f90300e8f3358f51928d43c212a8caed02de67eebee0121025476c2e83188368da1 ff3e292e7acafcdb3566bb0ad253f62fc70f07aeee635711000000

the above is from https://github.com/bitcoin/bips/blob/master/bip-0143.mediawiki
it is a 2 input 2 output tx in the witness space.

In addition to the above, the much smaller anyonecan spend tx is needed too. I think it will be about 100 bytes?

so we have a combined space of around 800 bytes against the 1000 bytes the usual 2 input/2 output tx occupies. Or was that 400 bytes that the 2input/2output tx takes?

I was told that all nodes are expected to be pruning nodes anyway, so you dont have to worry about any full node requirements. They will make sure all the archive copies will forever be kept safe and not tampered with. you can trust them. it is better for bitcoin to require trust

Isnt it nice to have all the hard choices made for you. We can trust in the math done by the central planners. Dont worry, be happy.

James

It is answered in the BIP:


all this seems to be above and beyond what would be needed for the normal, plus the nVersion (4 bytes) and nLockTime (4 bytes) are duplicated. To a simple C programmer like me it sure looks like instead of reducing the net amount as required by anything claiming to save space, it is increasing the size by approx 50 bytes.

Maybe its 32 + 4 + 1 + 1 + 4, so 42 bytes?

I am trying to understand enough to implement this, but unless the original tx is reduced by more than the witness data uses, it will cost more per tx.

But dont worry, I was told that it is likely that 100% of nodes will be pruning nodes in the future and all that matters is the size of the utxo. I still await how any new node can bootstrap if all nodes are pruning nodes...

James

how many bytes total does a segwit tx permanently occupy
just pick any standard normal tx.

is that number bigger or smaller than the size of the normal tx.

if the number is bigger, is this the case where "more" means "less"?
am I a troll because I am confused why something that is 10x more complicated, has potential attack vectors and requires adding trust to the bitcoin is marketed as helping with scaling by using MORE bytes.

I am just a simple C programmer and I dont understand complicated things like using more bytes to help scaling when that seems to be you end up with less tx that fit in the same amount of space.

please help me understand. Is there a new quantum zero knowledge phased space bit multiplexer now? do I need to find my flux capacitor from the attic?

dark side? I am asking questions I need answered to be able to implement things. The more I find out about segwit, the more it appears to be a LOT of work needed that can be avoided just with a 2MB hardfork. And compared to a 2MB hardfork segwit wastes space as far as I know at this moment and I await to be corrected.

FYI I am on no side other than the side of truth. The other side has politicized the RBF and claims it is the devil's spawn. I keep saying the reason RBF breaks zeroconf is that zeroconf is totally broken when the blocks are full. So adding RBF still leaves zeroconf broken when the blocks are full.

Now, one difference is that the other side doesnt start calling me a troll when I point out that their technical analysis is incorrect.

my agenda is to implement a scalable onchain bitcoin. If I see something that doesnt make sense, I will ask about it. If I get nonsense answers, I will point that out.

If using logic and asking pointed questions makes me a troll, then I guess I am a troll. Convince me with the math, then I will be segwit's strongest advocate. It the math doesnt add up, I will call it what it is

Let us ignore pruning nodes or anything about SPV nodes. My concern is about full relaying nodes. My assumption is that for a segwit compatible full relaying node to be able to relay the full blockchain it would need to have ALL the data, original blockchain and witness data.

How can ANY of that data be pruned and still be able to be a full  relaying node?

If all such data is needed, I want to call the combined size the size of the blockchain. Regardless of whether it is one or 2 different datasets.

So unless there is magic involved that allows relaying data that has been pruned, pruning is IRRELEVANT.

This then gets us to my question that is not being answered. On average, how many bytes in the blockchain will be needed for a standard payment sent via segwit?

Is this ever less than it would be now?
Is this ever the same as it is now?
Is this usually about 50 bytes more per tx?

Unless there is actual savings of blockchain space, then it would be a failure as far as reducing blockchain usage. What am I missing?

James

maybe you didnt say it, but the official claims are here: https://bitcoincore.org/en/2016/01/26/segwit-benefits/#block-capacitysize-increase

maybe its just me misinterpreting the english as my second language and the above isnt claiming that it will increase the block capacity. I avoid political stuff so maybe I am just not understanding the nuances of the english. recently I found out that "sick" meant "cool", but cool wasnt about the temperature, but something else. So I guess it just matters what the meaning of the words "size increase" means.

you know that segwit is being marketed as the magic solution to allow blocksize increase without hardfork. that is the issue here, disingenuous marketing

it might be that the "only" thing that is lost is the ability to verify the tx, but hey, what's wrong with needing to trust things, right? and the if the receiving node for the p2sh spend also doesnt run segwit, he is able to validate the tx because it is an anyone can spend signature? If anyone can spend it, what prevents somebody else from spending it? I am really confused as to how older nodes without the signature witness data can spend it to another node that doesnt have the witness data, without also allowing any random person from spending. What magic am I missing? Is that what the zeroknowlege commitment stuff is?

If you can only spend a wtxid to a segwit node, that is considered good enough? It seems segwit demotes all older nodes, even if they are fully validating and relaying nodes, into SPV nodes. This is acceptable?

I am not convinced at all that there is any space savings and pretty sure that it will take more space per tx if you count the witness data and we really need to count that data too.

This needs to be stopped being marketed as the softfork that increases tx capacity as that is not proven at all and likely the exact opposite when you make the reasonable assumptions that people will have to update to segwit and thus get all the witness data

ok, it seems the wtxid is not included in the witness data, however I cannot imagine how it can be encoded such that the space taken in the original blockspace + space in witness blockspace is smaller than just using 2MB of ordinary blockspace.

and if it doesnt REDUCE the total space, then it has no net gain and is a failure for increasing tx capacity. So where is the proof that it will reduce the total space used? We still trust in math around here, dont we?

and if the details about the total space used is not known by you, then the question arises about who has peer reviewed this. Using this for scalability has negative effect unless the combined space is reduced and in almost all cases when you have a single reference to something else, you cant save space as the something else needs to exist and also the reference to it. The best that would be possible is to have the position in the witness space be the implicit reference and that is probably how it is done.

however, there is still the issue of:

Transaction ID

A new data structure, witness, is defined. Each transaction will have 2 IDs.

Definition of txid remains unchanged: the double SHA256 of the traditional serialization format:

  [nVersion][txins][txouts][nLockTime]
  
A new wtxid is defined: the double SHA256 of the new serialization with witness data:

  [nVersion][marker][flag][txins][txouts][witness][nLockTime]

from the BIP...

the wtxid is based on all of the original, plus marker (1 byte?) flag (1 byte) and witness, which appears to be:

 1-byte - OP_RETURN (0x6a)
   1-byte - Push the following 36 bytes (0x24)
   4-byte - Commitment header (0xaa21a9ed)
  32-byte - Commitment hash: Double-SHA256(witness root hash|witness nonce)

all this seems to be above and beyond what would be needed for the normal, plus the nVersion (4 bytes) and nLockTime (4 bytes) are duplicated. To a simple C programmer like me it sure looks like instead of reducing the net amount as required by anything claiming to save space, it is increasing the size by approx 50 bytes.

So even if we say the cost for all the work in all the project across the bitcoin world is ZERO, it still reduces the overall tx capacity of bitcoin permanently. The fact that such a anti-space saving mechanism is marketed at all, let alone as a space saving "softfork", well you see my concerns about the technical reputations of anybody that supports the segwit for increased tx capacity.

I dont want to take away from the brilliance of the tech to solve the malleability and increase the potential usecases. The problem is that it is being backdoored through the softfork mechanism and being marketed without objective peer review.

James

P.S. So my understanding is that you need a special segwit address (that is somehow determined to be a segwit address using what mechanism?) so both sender and receiver need to already have the segwit version. I guess just ignoring all the existing nodes is at least some level of backward compatibility. But are you sure all users will quickly get used to having to deal with two types of addresses for every transaction and they will make sure they know what version the other party is running. Doesnt this bifurcate the bitcoin universe? maybe the name should be "bifurcating softfork"

OK, so you agree that if segwit achieves the activation level, all nodes will have to update and download the 2MB of data, which contains 300kb+ of wtxids that otherwise wouldnt be needed in a straight 2MB hardfork.

so it is a total fail from a "increasing tx capacity without requiring a hardfork" point of view. Let us not quibble if it technically is a softfork or hardfork, the reality is users will have to update or not be  able to spend. it looks like a hardfork, walks like a hardfork, quacks like a hardfork.

It sounds like it is possible to make it less of a problem, but it will be possible for segwit to be used to make unspendable payments to old nodes, so this creates an attack vector where the attacker simply sends to thousands of users some small amount of unspendable segwit wtxids. Once users get the bitcoins, they wont care about whether it is softfork or whatever, they will want to spend the bitcoins.

So, in the case where segwit is adopted, then all the nodes must update and get the full 2MB blocks that are bloated with needless wtxids. Now I just briefly looked at segwit details for the first time today, so maybe there is some super magic negative knowlege antimatter spacetime warping data compression that allows the segwit to actually save blockchain space. but calling the witness data not the blockchain since it is separate, again it becomes the type of stuff politicians do and not what technical guys should be doing. So if you are a politico, then fine, but I had always seen your posts as from an objective technical guy and was totally shocked at what you wrote. I am assuming the witness data is treated the same as the normal blockchain data, so it is in the same category and thus the statement that segwit is a total fail for increasing tx capacity without hardfork is fully justifed

Since segwit was started to fix malleability, maybe it should stick to that and not try to solve a problem it cannot solve. Officially claiming that it solves this is damaging to bitcoin's technical credibility and the other coins will take FULL advantage of this.

You cannot claim to be intelligent while advocating idiotic things, right?

James

I update balances based on the multisig (or p2sh ) address in that case. So the address that is credited is the multisig address and not the pubkeys that compose the multisig

this is a practicaly way of doing things and is what blockchain.info does

I think what is needed is a way to select out data from insight database, but I never used insight and not sure how best to do that

I already did that.
How do I verify independently from a known accurate source

James

P.S. my "database" is not actually a database, but for this purpose you can consider it a database

Are you trying to claim that having the entire existing installed base not being able to validate the wtxids they get is acceptable? And that not being able to spend it, is acceptable? The customer support issues it is guaranteed to cost, is acceptable? That the lost reputation for backward compatibility and reliability, is acceptable?

fixing malleability, great!
But to be able to spend the wtxids dont you need to get the extra witness data? Is that data part of the segwit blockchain? If it is part of the segwit blockchain, isnt there the wtxid for each txid that wouldnt otherwise be needed?

What am I missing?

To spend any received wtxid, you need to update to segwit chain, which is increased in size and includes the wtxid's for EVERY segwit tx, not just the merkle root. And this wtxid wouldnt be needed if we just hardforked to 2MB. So segwit as a space saver, actually loses space. Segwit as a softfork might be technically true, but it forces everyone to update to a sole sourced wallet or not be able to spend the coins received. And when they update, the wtxids are sitting there in their blockchain that wouldnt have been needed otherwise.

So, for fixing malleability and other things, great. no problems with that. but to claim it is increasing tx capacity without a hardfork is disingenuous at best. Most people would like to be able to spend the bitcoins they get. If you can agree with that, then you must agree that they will need to load the segwit blockchain, which is bloated with wtxids that would not be needed in a simple 2MB hardfork

I know you must have some sort of marching orders to follow the party line, but please, let us not make silly claims like "wallets still function perfectly fine with the old system. They can still receive segwit transactions, they just can't spend from them" I dont want you to lose your credibility

James

it is a hardfork pretending to be a softfork that increases tx capacity without doing a hardfork, but it actually wastes space permanently.

Of course it only wastes space on nodes running the segwit "softfork"

but if you received any wtxid and want to spend it, you need to run an updated wallet, which I am sure will be available within 3 days from when you get such a wtxid. After all the changes required are quite simple: https://github.com/bitcoin/bips/blob/master/bip-0141.mediawiki

Anybody that has written a bitcoin core from scratch (like me) should be able to implement the dozen or so changes in a month or two, and we can ignore any issues about the customer support as they can just do manual rawtx manipulations with all the great tools available for that if they really want to and dont want to update to the only wallet that supports segwit

So it breaks the installed base
Creates customer support and field update issues
And permanently wastes 30%+ of the new space as opposed to a simple 2MB hardfork (or 4MB)
But it will single source wallets for the months it takes for everybody to "fix" their software

Dont get me wrong, I think the segwit tech is pretty cool, but instead of pushing it into BTC mainnet under the innocent sounding "softfork" and exposing the entire installed base to pain and suffering, it seems this magnitude change should start in a side chain, get field tested and then if it makes sense to do a HARDFORK for it. DO NOT BREAK THE INSTALLED BASE

James

considering the obstacles it faces, I will not implement segwit support into initial iguana versions. If a bunch of users end up with unspendable funds, then I guess I will  be forced to massively complicate the blockchain handling.

does anybody know the BIP that defines the new network message(s)? I assume each block would need a getsegwitdata equivalent so existing nodes just do the existing getdata and the segwit softforks (ha ha) do the additional call, process the new data format, update the internal dataset, validate the signatures and enable spending. Seems like a LOT of work to get a 60% increased capacity.

Since this is basically a hardfork (let us not kid ourselves that creating INCOMPATIBLE bitcoins is in any way backward compatible!!), and it requires the additional data, then just hardfork 2MB.

That would not require wtxid wasting precious space in the blockchain.

The logic used to justify wtxid permanently using up space that otherwise wouldnt be needed is that it allows a softfork. But this is a fake softfork, as existing nodes wont be able to validate or spend any segwit payments it gets. HOW ON EARTH IS THAT BACKWARD COMPATIBLE?? which is the industry's understanding of softfork behavior (I know technically that is not what softfork is, I speak of what users are thinking)

So, if segwit gets 75%, then it forces all the nodes to update. How is that not a hardfork?

The logic is segwit allows 60% increase in capacity without a hardfork, but this premise is wrong. It is effectively a hardfork, actually I think it is worse. If it was a hardfork, then users who know about such things will understand that they have to update. Saying it is a softfork will make users think they dont have to upgrade. Then they find out that they cant spend the bitcoins they got. So we end up with two incompatible bitcoins. this is not a good plan at all

Let us call a hardfork a hardfork. segwit is a hardfork from the most important aspect that is it NOT BACKWARD COMPATIBLE with existing wallets and independent cores.

OK, so that then means that segwit is WASTING precious blockchain space and not avoid a hardfork. It makes no sense to me that a defacto hardfork that breaks ALL EXISTING wallets and also requires all independent cores to make significant changes, testing, field updates, customer support, and it is all to permanently waste space with the redundant wtxids that would not be needed if we just hardforked 2MB

Please tell me there is some sanity here. There is no logical justification for segwit and plenty of risk factors of creating the impression that bitcoin is broken, and if you dont consider the existing installed base not being able to validate or spend bitcoins as not broken, then something about how you evaluate brokenness is broken.

James

P.S. Technically segwit is very clever and I can see plenty of use cases it enables. However, positioning it as a softfork is a disaster. But I guess bitcoin has nothing to worry about, after all there are no altcoins out there that have any chance at all against bitcoin. So we can just do whatever we want and it wont matter since there are no alternatives out there for users. Who is afraid of LTC anyway, right?

I figured out how to get iguana to generate the balance for ANY address as of ANY block, with about 2 milliseconds compute time using a 1.3Ghz i5 CPU. With faster CPU and 8 cores, this would be able to be sped up to 100 microseconds. As I have written extensively, I dont use any DB, I use readonly parallel datasets.

Also generating a full dataset would be much faster as I can batch process all addresses in the same pass.

My issue is how do I verify that I have the right data? So, assume I have a file with (address, balance) as of the start of block 400,000, how can I verify that it is 100% correct

when you say "receive" but not spend, it is received and unverifiable and unspendable, right?

is it just me, or does it seem like calling this a softfork might be technically accurate, the market confusion and incompatibility it will cause is pretty much like a hardfork

Does anybody know how to query balances for all addresses as of a specific block?

I would think some sort of query layer on top of insight block explorer?

My use case is that I want to do a brute force verification of all addresses that I have generated with the iguana parallel sync. Now that I have this data, I cant find any easy way to verify it...

James