I think your machine is broken. I've NEVER had Bitcoin-Qt do that, across several machines.
Well I have a pretty new (late 2012) iMac, and I'm pretty sure that it's not broken. But thanks for the useful response
In other news, as World has found, 0.8.4 seems to have fixed these problems for me now