I use a RPi for signing myself, though the carrier is a USB stick (the RPi is never directly connected to my online machine, and it never was connected to internet in the first place).
There has been proposals to either add an audio modem or an animated QR code codec as extra transfer channels. In general smart cards are better than USB keys because the USB protocol and handshaking is significantly richer than that of smart cards, which are only ever storage devices. There's arguable minor benefits to be had from DMA access to the storage device vs going through the driver.
For the more paranoid, you could write the data back and forth by hand, or burn CDs.
I think for the "21 millions club" you have more than right to be paranoiac, it is your core skill ...
I remember in the early 90's some countries got their top level DNS and then failed miserably on keep their TSL root keys physically safe. (if I'm not mistake the term is "safe room" for a complete isolated computational system)