Protecting from host, protecting from users, protecting from outside attackers are all considerations. It isn't something to take lightly, cold storage is a consideration for a majority of the coins.
VPS can be a problem too. One host wasn't erasing the disk space between VPS sessions for example. You have to worry about hypervisor issues etc. A lot also depends on what attacks you are trying to prevent. E.g. if this is on a coin with a total market cap of $1.00 then it really doesn't matter much.
Here are a few more threads, there have been a number of discussions which should help to start. Experience will be key though:
https://bitcointalk.org/index.php?topic=106339.0https://bitcointalk.org/index.php?topic=27250.0