The downside to using a hardware wallet is just one. And it is obvious. Under certain circumstances, it is very easy to lose your savings. Despite the increased security and exclusion of theft of funds, the wallet can be easily lost - it is small and simply dropped from your pocket without even noticing.
The device is just one way to access these coins.
Noone should use any wallet without a backup. And this also applies to a hardware wallet.
Simply backup the seed and you are fine when losing the device.
You can also just forget the access password.
Same applies here.. Simply reset the device and use your backup.
The downside is that the wallet becomes independent and the entire responsibility lies solely with the owner, which is why no one can help you get the money back if the wallet or password is lost.
This applies to every (proper) wallet.
If somehow you are not entirely responsible for your wallet (means: someone else can access it), you are using a really bad wallet and should immediately abandon it.
The only solution to this problem is the backup function, but this action significantly reduces the security of the wallet.
Everyone should always have a backup. Devices can break. Especially electronic devices (e.g. lightning strike when plugged in and running).
Using a wallet (doesn't matter whether hardware-, desktop-, mobile-, etc..) without a backup is negligent and will - sooner or later - lead to a loss of funds.