Running a ubuntu live CD, install the bitcoin client.
Then copy the wallet.dat on a storage device, keep this safe and don't use this on another operating system.
Then, write down the bitcoin address and use that address for storing your bitcoins, you don't want to keep checking if they have been received via your ubuntu setup, because that will take a long time.
When you want to withdraw any bitcoins, then boot up your live cd once again, then withdraw like normal.
However don't use the wallet.dat on your computer without running a live cd.
I found that it might not the best alternative.
If you don't want restrictions / hassle, create a persistent live linux on a USB (knoppix supports it out of the box).
You can encrypt the file system too for added security.
Make sure to use a 16-32GB pendrive to have enough space for the blockchain.