I am especially interested in hearing from anyone with alot of experience in network security about how to improve the anonymity of the site beyond running it as a tor hidden service. How would YOU do it? What are some worse case scenarios?
Well, as was previously mentioned in this thread, you might want to look into TrueCrypt. It's not network security, per sae, but it is pretty much the best hard-drive encryption method currently out there. You can also encrypt your entire OS. Plus there's always hidden recessed volumes. Basically what it means that if the server (god forbid) ever gets seized the feds/etc have virtually zero* chance of reading any of the stored data. It's well worth looking into IMO. I'll leave the network security discussion up to others, as it's not really my forte. Just make sure you keep all your software, expecially databases, php installs, etc etc etc up-to-date as that should protect you against the most common inject attempts, etc.
*Yes, I know cold boot attacks are theoretically an issue. That said, they've had problems doing them in labratory settings never mind in the real world. Not a big risk, IMO