Do you think it's enough to have the bitcoin wallets in a server you physically control or should also the website be located inhouse to prevent thieves from stealing bitcoins?
Server you control physically is sufficient in most cases. Most companies don't have the resources to build a datacenter to house their server. Going with a tier 1 datacenter, purchasing a locked cage or cabinet, and starting with bare metal server(s) provides a high barrier. Now if you service grows to the point you are processing billions a year well then moving servers (or at least hot wallet hardware) "in house" might be something to consider.
We use a private locked cabinet with access control in a major datacenter. No datacenter employees have need or ability to login to our hardware. The OS was installed clean onto bare metal we own so there are no "super admin" accounts that we don't know about. IPMI and power cycle PDUs have made it possible to do a lot more remotely these days (even BIOS access and remote media for installing OS is possible). Good secure chassis with intrusion detection are a good secondary line of defense to ensure the employees don't have access to the hardware internals. We disable USB in BIOS. Since disks are designed to be hotswapped, encrypted disks (and backups) are a requirement to ensure information isn't physically stolen by datacenter employee. A good datacenter should have no problems shipping replaced/dead disks back to you to verify serial numbers against inventory control.
The one bad thing about IPMI, is it is usually very poorly implemented from a security standpoint. It doesn't really matter the vendor, most have dozens of long running vulnerabilities. The IPMI ports should never be public facing and instead be behind a dedicated vpn hardware firewall (i.e establish vpn tunnel to firewall, authenticate, and then gain access to the IPMI network).
The web server is going to be the most vulnerable point of any system; it is by definition public facing with open access. For that reason that server should only be used as a webserver. The database, bitcoind connectivity (even for just listening wallets), remote WAN login access, backups, etc should be on a different server which has no public access. Most datacenter can provide a VLAN on a switch for private connectivity but switches are cheap so I like to buy and install our own switch in the cabinet. Of course all this is just the outer wall, intrusion detection software, monitoring, and vulnerability scanning should be part of the picture too.
If all that sounds hard well that is why the service is operating for a profit. Users should start to demand more from their bitcoin service companies and not accept that they are uncompensated investors (if exchange does good real owners profit, if exchange does bad depositors lose everything).
