I installed Ubuntu 11.04 on a 8G USB key on my rig. This is working great. Looks like there is a limitation to the size of the partition to 4G. I desinstalled libre office, java and other stuff I don't need on the machine but I have only about 200M of space left on the device. I also created another 3G partition to install the AMD SDK and mining stuff but I'm still running out of space on this 4G partition.
How do you cope with the 4G limitation ? Did you install the regular Ubuntu distribution ?