Well Ubuntu can be booted from a USB and as such you would be able to use it anywhere - even on computers with no bitcoin client and which do not normally allow the installation of one.
Theoretically this would allow you to use restricted library or university or weird internet bar computers with IPs that can't be tied to you at anytime.
But if you just want to load and go windows is more widespread.
It should be possible to keep a linux BTC client and windows both on the same stick though, although I only think Ubuntu could RUN from the stick.
As for encrypting the wallet and backup this is my plan (only setup one wallet yet, I'll make another soon):
1. Using 0.52 client choose "encrypt wallet".
2. Create NEW keys/addresses during LOCKED mode and use only these henceforth. (Long pass I don't use often of course)
3. Copy wallet and compress it with RAR and choose encrypt with another different password.
4. Put this double secured wallet in email, sugarsync, phone and harddisk.
5. There can apparently be leaks between addresses in a wallet due to optimizations by the client - keep public addresses in one wallet and your private savings addresses in another wallet.
1. Tor sounds nice, but I would not trust it, not alone anyway.
2. Instead ONLY sign with the private keys from IP addresses that cannot be easily tied to you.
3. Some of your addresses will be publicly tied to you as your mtgox account or what not, do not send directly from these to your savings addresses.
4. The only truly anonymous pattern would be, for example: Mt.Gox BTC -> Pub. addr. -> 1 NEW addr. -> Many new address -> -||- -> STOP.
The money now looks like it was spent, re-spent and got stuck/saved up by someone in the natural market though the "tree branches" should be a bit more irregular/longer in length than this.
(Perhaps have a receive/spend wallet, an intermediary wallet and a secret store wallet for this).
5. Also think about timing here, imagine: X, Y and Z amounts of BTC separately leave your public account J2 on day 1-3 and EXACTLY amounts X, Y and Z end up in accounts A2-G2 the SAME separate days 1-3 without delay.
Many transactions -> some fees.
Complicated -> a few hours.
Getting away with tax frauding a bad government -> Priceless.
Eh obfuscator services rerouting/mixing coins a lot also exist, might be better, though I don't know much about them.
Hope that helps anyway; IP, transaction path and transaction timing.