If I were you and lost the device, I would restore my wallet to Electrum or any other client and then move funds to a new hardware wallet. Although from the lost device funds wouldn't move without a password, and three-time entering the wrong password will reset the device. But who knows if anyone guesses the password and matches. So I feel this way is safer to secure my funds.
Better safe than sorry,right? Hardware wallets doesnt really cost much and wont be mind boggling specially if those funds stored are significant ones then it is really just sensible on making a new wallet
and trying out to restore and transfer all the stuffs and just like on what been said above that you could never have a good time sleep if you do know that someone is trying to crack or
access your HW but basing up on security it cant really be that easy but somewhat not impossible.
When it comes to security then it isnt really that bad to be that paranoid most of the times.