With a stateless hardware wallet, you remove the risk of somebody stealing your device and extracting your keys. That's not possible with a stateless device since your keys aren't saved on the device.
A really paranoid person would want proof of that. The keys disappear from the memory of a stateless device quickly I think, so by turning it off, you should be safe. Physical manipulation is still possible to some extent. Just like with the Ledger implants, the hardware for stateless signers could also be tampered with. It's much more unlikely, though because the hardware isn't exclusively developed for bitcoin and crypto.
The person using the stateless device would need to know how the Pi Zero, for example looks like and what it's not supposed to look like. That would make it easier to recognize components that aren't supposed to be there, like an antenna, an eSIM, a secret storage medium, etc.
Some time ago there was some talk about malicious firmware being able to slowly leak keys through signed transactions that scammers could then get access to through blockchain data. I don't remember the specifics. You would need to install a malicious firmware for that, though.
The possibility for an attack is there, but it's much, much smaller.