This is in my opinion not enough to check, especially when you omit a portion to be checked somewhere else than at the start and end of an address.
As OP said and it's what I practice: I check 6 to 7 symbols (excluding the address type prefix) at the front, at the end and a few at the same position around the middle. If those three spots match, it's extremely unlikely that anything else doesn't match too.
The crucial thing is, you have to verify any of the transaction's outputs always, without exception. Best of course on the independent display of a hardware wallet.
I agree with you. I recommended checking the 4 digits at the beginning and end of the address as a minimum, but I check
ALL characters regardless of the amount to be sent.
I usually start by checking 4, but instinctively I check more than 4, and before I know it, I've already checked all the characters.

A malware that alters the transaction details before it's passed to be signed on a hardware wallet is not science fiction. That's why you always have to check all transaction details before you sign and broadcast a transaction. Every time!
A hardware wallet protects your private keys from malware. Your private keys and your mnemonic recovery words should never touch an online device. Your hardware wallet creates your mnemonic recovery words in a secure environment. You write an analog paper backup (or better multiple redundant copies of it) which never go online. You create maybe also an analog metal backup for fire protection. This again never becomes online stuff. This is Bitcoin safety 101.
That's what I always say: never act as if you're completely bulletproof, even if you have a hardware wallet. What guarantees security are the user's own practices and how diligent they're about the security of their hardware.
I think you're referring to
Dark-Skippy. In this case, the risk is real. Users typically become infected with this malware if they import a compromised firmware version or purchase a counterfeit device. In this case, the chances of it going unnoticed by the user, even an advanced and veteran user, are very high.