It is good to have the hardware wallet, but there are also some risks associated with that as well, imagine that something happened and the hardware wallet is lost, maybe in fire, flood, or stolen or something...Just like the last comment said, it is good to store your funds in different places (wallets you have the key) and not just in one place, i find this more secure than just hardware wallet.
Of course, you can never be 100% safe. But you can always hook it to a key that you ALWAYS carry around with you all the time. Even then that too can be lost, of course, but still. Seems like the best option if you're paranoid.