The wallet interface and design is setup to show *all* transactions, sent and received. The only way to do that is by some form of rescanning. There is no way for the wallet to indicate "stuff might have happened before X but I don't know about it", so without that failing to scan the history could get you into trouble (for example, you accidentally defraud business partners or customers by not crediting them for payments you received outside of your wallet's view, or you unintentionally file false tax reports, or you accidentally make a payment again which you already made, etc). Beyond confusing the user, the wallet UI expects the inputs for all spends to exist, otherwise it has cosmetic problems like displaying "negative fees".
You can see more about this in this issue I opened in 2016:
https://github.com/bitcoin/bitcoin/issues/9409AFAIK that is the only real reason that it is the way it is. Changing the interface in such a way that a partial wallet could be displayed isn't easy, and doing so in a way that won't break the interface for existing users may not be possible.
Re, achow's comment about technical debt-- I don't really agree. Yes the wallet contains transactions, but if that were the only issue, less than two hours of actual development(*) would be required to make a wallet centric rescan that used the existing scantxoutset functionality to quickly find a list of heights to scan. The whole process could be done in minutes rather than the 8 hours a rescan currently takes. Even if the blocks had to be fetched from the network it (due to pruning) it wouldn't be too big a deal. But with that done, you'd still be left with the wallet interface being wrong and confusing and sorting that out would be a massive change because it would have to change how the user interface works, presumably break API compatibility, etc.
The original reason the wallet contains transactions is almost certainly because of how Satoshi expected SPV to work: Rather than scanning blocks like multibit/bitcoinj work which has terrible performance (which has mostly caused the death of these clients) or expecting bloated centralized servers to have huge indexes (resulting in most working electrum servers probably being run by spy companies) satoshi expected they payees would get handed evidence of the confirmed payment from payer. The copy of the txn in the wallet in Bitcoin used to have a copy of the merkle proof in it, presumably for this purpose. The proof was removed because it was bloating up wallet files from large users and wasn't used, I don't even think we really understood what purpose it would be useful for when we removed it. Payment protocols took a different route than one which would have made it useful in any case.
But even ignoring the unused SPV use case, possession of the complete transactions is needed: E.g. For non-segwit utxo, the whole transactions are needed by untrusting parties in order to validate the amount of fee paid. This comes up with offline signers and hardware wallets. (In fact, because segwit's sighasher is still somewhat flawed, there are still some vulnerabilities to fee overpayment if you don't look at the inputs, even with segwit in use.)
Given that, you can't simply just go and eliminate the wallet storing transactions as some refactoring job against the codebase. To drop transactions entirely you'd have to either drop support for non-segwit outputs or at least the ability to export them for non-trusting signers-- real functionality the actually gets used. It isn't technical debt. But the propensity to call any behaviour that developers don't understand or like technical debt is as strong as ever.
(*) Actual development meaning the time it takes for a knowledgeable person to actually write a correctly working feature. Getting new functionality merged into Bitcoin Core is another matter entirely, and would probably take months of effort now. As a result very little new user visible functionality is added to the software.
