It is extreme or at least I think it is. But Honest How extreme should I be? When I add the passphrase
Since the passphrase is 'just' an additional layer and not the only thing which secures your wallet, it doesn't have to be too secure.
However, whenever choosing a password remember that
length beats complexity.
Instead of adding those non-standard characters which are very error prone, especially since they can't be entered using a keyboard, just add a few more characters while not using such characters or not even special characters and your password will be more secure than a very complex - but a few chars shorter - one.
The overall strength needed for your password depends on your threat model.
If no person will ever be with your hardware wallet for more than a few minutes and you can be sure that you will always detect whether your device got stolen within a short timeframe with the ability to recover your funds using a backup, it doesn't have to bee too strong.
If - on the other hand - it always will take a few months or even longer until you realize that your HW got stolen or to be able to recover your funds using a backup, i'd rather use a strong password instead of a semi-strong one.