Right, the site can't seize the deposit - that's exactly the point. The site can require a fairly large deposit and it's OK because you know you'll eventually get it back even if the site disappears.
* You open site and charge 1 Satoshi
There's your problem. Charge more.
* You raise fee to 1 BTC but pay back the botters + have problems with moderation etc.
The botted accounts won't get their funds back for the max amount of time. If you set it to six months, unless they're willing to only spam you twice a year, they need more funds at this point. You can tune these parameters to make it too expensive.
Also, if I'm getting the money back anyways, I would't care how much it costs, since in the end I will as a bot always fall under the regulations of a completely unknown new user. You cannot tell them that you'll keep their coins in limbo for years
Yes you can. The rule is, everyone is required to make a deposit, that may be lowered or raised according to some risk analysis (eg, if you have a Facebook/Google account perhaps it's lower). If you don't break the ToS you can withdraw that deposit after a short time and the account will be closed. If you do break the ToS then you have to wait the max amount of time to recover the deposit.
See it this way - spamming results in locking up funds such that they are unspendable. This does
have a cost, even if you get the funds back eventually, because it's opportunity costs. If I told you that I would give you half a million dollars the day you turn 70, would you feel richer today? You'd feel the same, because those funds aren't accessible to you for the things you want to buy right now.