I had also come up with a similar idea - although to make sure the hand device does not get hacked I had envisioned a QR code being scanned back from the device's screen to then complete the login (via web cam) with your authenticator being being a "completely offline" device (i.e. safe from any potential online threats).
Also the Trezor may be able to provide similar sort of OTP authentication down the track.