Segwit effective eliminates the old size limit and replaces it with a backwards compatible "weight" limit.
The outputs of a transaction have to be maintained by all nodes in low latency storage so that future blocks can be validated. By contrast, the other data (mostly signatures) is processed once by nodes and then can be forgotten by most nodes (as is described in section 7 of the Bitcoin whitepaper). So signature data is much less costly to the Bitcoin system long term. Because of this, it makes sense for the block capacity limit to work closer in terms to the cost of the system, which is what weight is intended to do.
Given the long term costs to the network it would be pretty easy to argue that the output data ought to consume a hundred or a thousand times more of the limit than signature data. But the instantaneous burden of forwarding new blocks-- under which both signature data and output data are about the same-- isn't irrelevant. If the ratio between outputs and signatures were 100 then someone could gum up forwarding with a block that had 100x the signature data. Also, because all transactions have both output and signature data Amdahl's law applies: the capacity increase from allowing more signature data has a diminishing return, and a block that allowed 1000x signature data would hardly allow more transactions than one that allowed 2x the signature data.
At the time segwit was created we'd recently figured out how to reduce the bandwidth associated with relaying transaction and block data by a factor of 2 on average and we deployed that improvement at the same time, so it seemed likely that allowing blocks to have a couple times more size was unlikely to cause severe problems for block relay (since the resource increase was paid for by the improvement). And it was also clear that going much further, even though it would be justified by the long term costs, wouldn't add a lot more capacity even though it would increase the risk of trouble for block relay.
As far as your thoughts on 'more bytes'-- the encoding of transactions on disk and sent over the network isn't a normative part of Bitcoin consensus rules. For storage a node can store the data however it wants, and for network transmission it's sufficient that the peer you're talking to understands the format you want to use.
So implementations are free to use different encodings and e.g. can just use a single bit to signal if a transaction is entirely segwit or not. There have been several such implementation (
an example) and one is in use e.g. for satellite bitcoin broadcast. A single bit per transaction (or even less!) is easily paid for by other savings like the elimination of DER encoding of signatures, or the 32-byte only pubkey or the elimination of the 20 byte pubkeyhash in taproot.
Of course, right now most nodes don't bother with alternative encodings to shave all the bytes because the differences aren't substantial enough to catch people's interest.
As far as hardforks go-- Since a hardfork can rewrite the rules completely arbitrarily, it would have been possible to change to the segwit style signatures inside the existing scriptsigs (as BCH and BSV did) and then apply the weight rules uniformly so that even spends of old coins consumed the limits in the same way new ones did. But other than the short term effect of users gradually opting into the new style this wouldn't be any better. And it would be vastly inferior by creating a huge coercive flag day change that everyone would have to be coordinated on, and that anyone who wasn't ready or didn't want to participate would be forced off the network. Nodes aren't relieved of having to process the prior formats and rules since the past chain still exists, so making an incompatible change create more technical debt and cruft.
It's important to realize that segwit was first invented and implemented as an incompatible hardfork change and only after was it realized that if the signature was put in another field that it could be made backwards compatible. So we've actually seen what doing it that way looked like, and the ultimate softfork version was simpler. Most of the differences, like the witness commitment being in the coinbase txn rather than in a branch right off the merkle tree are easily argued to be better in the final segwit version. (e.g. in that case because the softfork way makes each spv input proof 32-bytes smaller, and isn't worse for any user since anyone validating witnesses needs the coinbase txn regardless).
As a matter of principal people should have the greatest freedom to use whatever software they want to use and that means that compatibility with old, even outdated, software ought to be preserved if at all possible, even at some considerable cost. Various Bitcoin forks once bragged about their 'multiple implementations' but their culture of hardforks soon drove every other implementation off their networks as no group but the largest one could keep up with all the mandatory changes.
Some argued that making the change compatibly was inferior because the benefits came on slowly as users opted in to use the new functionality and there is some merit to that argument. But OTOH, the gradual change made the effective block size increase slowly rather than immediately creating a lot of empty space for people to fill with spam or other abusive traffic, so I think it's probably better that the effect phased in slowly as people have adopted new address types. I also find the argument that a slow phase in was bad unconvincing in the light of the year or so that miners took to adopt it, had it truly been that urgent it could have been activated more quickly. In any case, even if you disagree that the slow phase in was a benefit, I hope you agree that disrupting people's choice of software and creating a network wide flag day would be an unprincipled act that created harm that was well worth some delay to avoid.
