I have been using Ledger, which allowed me to generate the seed offline and hide your private keys. So an attacker wouldn't able to steal funds without the device or seed.
If we are talking about an PPT adversary without any known exploits and/or huge monetary resources, even stealing the device won't leak the private keys.
You'd need the device and the PIN to steal funds. The device itself is not sufficient (given that no 0-day exploits exist).
Note that there still might are evil maid attacks where funds could be stolen (i.e. manipulating the nano s).