If I wanted to spend bitcoins I could just put the sim card back in.....Or would there even be a security risk in connecting for a few minutes?
In theory? Yes. Malware could be on the phone waiting for a network connection to broadcast your private key. In reality, the odds of this happening are extremely small, but any device that goes online, even briefly, is in principle a security risk.