Not all wallets need to download and scan the entire blockchain to display your correct balance and transaction history.
When you use Bitcoin core, you actually run a node, and you verify the transactions and blocks yourself. That's why you need to download the entire blockchain.
By running a node, you can increase your privacy and you also help the network be more decentralized.
Most of Bitcoin users don't need to run a node, and instead they use SPV wallets.
In a SPV wallet, you don't need to download the entire blockchain. You connect to a server run by a third party and you only get the data you need from it. If you have too many transactions associated with your addresses, it may take longer to retrieve the required data from the server, but it should still take only a tiny fraction of the time it would take to download or scan the entire blockchain.
I've been thinking recently about why a Bitcoin wallet would want to rescan the blockchain, in particular after recovering the wallet from a backup, or loading the wallet using the seed phrase.
Which wallet required you to scan the entire blockchain after you recovered your wallet using your seed phrase?
I don't know of any wallet that gives you a seed phrase and requires downloading or scanning the entire blockchain.