[Emergency ANN] Bitcoinica site is taken offline for security investigation

[DeathAndTaxes]

2) If you are serious about security, don't advertise your genius ideas.

On the contraire, if your solution is proven and tested, you shouldn't be afraid of sharing it with the public. "Security by obscurity" is just laughable, and the source of problems like the one Bitcoinica is going through perhaps. Let's bet that nobody knows where our server is, or how we secure it. Let's just bet!

+1.

WEP was a closely guarded secret and busted wide open.  The implementation details of SHA-256 are public knowledge and the algorithm remains secure today.  If your "security" relies on the attacker not knowing "how it works" you have already failed.  Eventually the attacker will know and the house of cards will come crumbling down.   

Obviously some common sense applies.  Certain elements (ports, keys, certs, API commands, db schemas, etc) should remain secret but if you feel the need to hide the general concept well you are doing something wrong.

[coinft]

I am honestly flabbergasted how financial companies holding hot reserves keep using Rackspace, Linode, Amazon EC2, etc...

These "web 2.0 geeks" need to learn how to secure your system old school, like old bags like me do.

We have constantly 6 figures on our API enabled auto exchanger. Maybe because the reserves are actually ours, and do not hold client's funds we actually give a shit about keeping it secure?

Kids, here is how you secure a financial server:

- Get a BARE METAL dedicated server.
- Install Debian 6 on an ENCRYPTED LVM through KVM. Don't get your server at a hoster that does not allow you to install bare metal.
- update/safe-upgrade
- Generate SSH key pair. Leave only public on server....
- Disable SSH login with password
- Install firewall, only leave absolutely necessary ports open.
- Keep backups of your SSH key on encrypted drives, wualla, etc. This should be the only access to your server.

And there you have it ... break in on that ... not even the idiots working at the hosting company (or authorities getting your drives for that matter) will be able to get your info, your client's info, or your money for that matter.

Rackspace and the "cloud" is not there for servers holding 6 figures. It is unbelievable to me that after the break in on linode (which shouldn't have happened in the first place) neither Bitcoinica or Intersango took this as their absolute first priority.


The "cloud" is fine for your latest playing around project. But for financial matters, it is nothing more than glorified shared hosting.

Attack vectors:

- The idiots at the hosting company, or anyone getting access, can try a cold boot attack against your server. Bare metal isn't enough by far for physical security.

- You call them idiots, but they can at least take your site down easily, for damaging rep.

- Those guys can also reroute your traffic for man in the middle attacks vs your clients -- do you believe they will not click through SSL security warnings?

- If you login via ssh, I hope you got the server key and enabled strict checking, to avoid man in the middle attacks. You should consider something stronger than mere software keys too. Require 2 persons, or at least std 2 factor auth. And you need remote logging to another secured system for a verifyable audit trail.

- I hope you do debian security updates more often than you update your sig, which still advertises transfers to bitcoinica.

- Bragging about your server on the net while calling ISP employees idiots is a bad idea.

Free advice.

-coinft.

[kokjo]

WEP was a closely guarded secret and busted wide open.

NO IT WAS NOT.
the RC4 algoritm was until 1994. WEP was first used 1999. 5 years after the inner working of RC4 was discovered.
It was first in 2001 that the FMS-attack was done, that made it possible to crack RC4, and therefor WEP.

[tvbcof]

If you don't have a solution which obscures necessary weaknesses behind solutions which provide some degree of opacity, you probably have a somewhat primitive solution.  Of course you lose this advantage if you blather on about how you've done things.  Just sayin...

Well I will respectfully disagree with that , but to each its own I guess. If you apply the tools properly, no obscurity is needed. Case in point: Bitcoin. The cryptographical principles that make bitcoin secure are completely open source, and published everywhere, for everybody to see, tinker, and to break.

Can I steal your Bitcoins by breaking the cryptographic mechanism that protect your private key? NO.

I agree with your assessment of how to make a rational choice of tools (and realized that in doing so, I've already neglected to practice what I preach to some extent.)  But there is a big difference between choosing tools and choosing how, exactly, to glue them together.

I don't really need to paint a neon sign on my deployments to enjoy the benefit have having skilled attackers work on the tools I have chosen.

[muyuu]

DeathAndTaxes and aurumxchange both seem to be bright chaps but my main concern is different:

The course of action absolutely doesn't follow logically from what appear to be the reasons behind the hacks, both alleged and inferred.

"So, we've totally fixed the codebase, we have fantastic security standards in place. We just got our mail p0wned. Therefore... we are going to rebuild the codebase?"

So... not just there's nothing said about changing that mail policy instead of "improving the mail server", why are you rebuilding at all if the code-base and server are so solid and well secured?

Not that I'd put a Satoshi in Bitcoinica again, but wtf?

[padrino]

I've experienced much the same in my professional life and seen many hundred of thousands, if not millions of dollars being invested to "secure" a system (using the term in the abstract) without a comprehensive understanding of the system from end-to-end. I make a great living helping commercial and govt. customers address issues and generally the breakdown isn't that one person is or isn't an expert at everything but rather a lack of understanding with respect to what they do and do not know.

Many people, including those that "secured" Bitcoinica have a certain focus area that does not seem to extend into the OS, physical security, access mechanisms, etc. and while quite valuable as a component of a comprehensive solution they are not a solution in themselves and that is the fundamental breakdown.  I don't think this latest theft and the subsequent work to address will ultimately fix anything unless they bring in additional consultants that can provide a end-to-end plan to access and address the system, software, processes, etc...

[Bitcoinica Consultancy]

the claims process should be finished tonight. It took long because we did not want to use a 3rd party service such as Wufoo for obvious security reasons.

[phorensic]

I am honestly flabbergasted how financial companies holding hot reserves keep using Rackspace, Linode, Amazon EC2, etc...

These "web 2.0 geeks" need to learn how to secure your system old school, like old bags like me do.

We have constantly 6 figures on our API enabled auto exchanger. Maybe because the reserves are actually ours, and do not hold client's funds we actually give a shit about keeping it secure?

Kids, here is how you secure a financial server:

- Get a BARE METAL dedicated server.
- Install Debian 6 on an ENCRYPTED LVM through KVM. Don't get your server at a hoster that does not allow you to install bare metal.
- update/safe-upgrade
- Generate SSH key pair. Leave only public on server....
- Disable SSH login with password
- Install firewall, only leave absolutely necessary ports open.
- Keep backups of your SSH key on encrypted drives, wualla, etc. This should be the only access to your server.

And there you have it ... break in on that ... not even the idiots working at the hosting company (or authorities getting your drives for that matter) will be able to get your info, your client's info, or your money for that matter.

Rackspace and the "cloud" is not there for servers holding 6 figures. It is unbelievable to me that after the break in on linode (which shouldn't have happened in the first place) neither Bitcoinica or Intersango took this as their absolute first priority.


The "cloud" is fine for your latest playing around project. But for financial matters, it is nothing more than glorified shared hosting.

I like the cut of your jib 

[marcus_of_augustus]

Attack vectors:

- The idiots at the hosting company, or anyone getting access, can try a cold boot attack against your server. Bare metal isn't enough by far for physical security.

- You call them idiots, but they can at least take your site down easily, for damaging rep.

- Those guys can also reroute your traffic for man in the middle attacks vs your clients -- do you believe they will not click through SSL security warnings?

- If you login via ssh, I hope you got the server key and enabled strict checking, to avoid man in the middle attacks. You should consider something stronger than mere software keys too. Require 2 persons, or at least std 2 factor auth. And you need remote logging to another secured system for a verifyable audit trail.

- I hope you do debian security updates more often than you update your sig, which still advertises transfers to bitcoinica.

- Bragging about your server on the net while calling ISP employees idiots is a bad idea.

Free advice.

-coinft.

Of course no system can be 100% secure unless you have sole physical access to the server, and then there are other worries as well. I am just saying. From having a Rackspace VM holding 100K to my solution, there is a whole lot more security, and as you point on your post, the attack vectors need to be substantially more sophisticated than breaking into an IMAP account.

I wasn't calling my ISP guys idiots by the way, I was just talking generically. Our app server is neither hosted nor collocated. The rest of our infrastructure is.

aurumXchange ... I'm sure that live security testing of your set-up can be arranged if needs be ...

[btcgoldsilver]

Bitcoinica is still in its infancy, its not that long ago that it was a small project developed by a 17 year old programmer. I take my hat off to Zhoutong coming up with this idea, I think Bitcoin is better for it.  

Personally I think a bit of leeway can be given and bitcoinica are (in my opinion) being fair with customers, assuming they do what they have stated they will do.

The security concerns are valid though if we are to take bitcoinica seriously going forward it has to prove that it is taking security very seriously itself. A compromised email server leading to all that theft seems extremely amateurish at best.

The person who described how to lock down and secure a financial server made good points, I hope bitcoinica will listen to the advice being offered by the community

[defxor]

Regarding "omg it's been four days you have my money i will contact the authorities":

15 odd years or so someone abused my VISA to buy software on the Internet. Back then my main card was still a debit card, connected to the bank account I used for paying bills and where I got my pay check automatically deposited.

They cleaned me out. When I contacted the bank they were very understanding, they investigated the purchases and found it was quite obvious I hadn't done them.

The estimate they gave me until I'd have the money back in my account was 3-6 months. No interest.

Perspective.

[datafish]

The estimate they gave me until I'd have the money back in my account was 3-6 months. No interest.

Perspective.

Also, where are these whiners planning to put their money where they stand to safely earn a significant amount of interest in a few days or weeks?  Tell me, I'd like to invest there myself.

[teflone]

Regarding "omg it's been four days you have my money i will contact the authorities":

15 odd years or so someone abused my VISA to buy software on the Internet. Back then my main card was still a debit card, connected to the bank account I used for paying bills and where I got my pay check automatically deposited.

They cleaned me out. When I contacted the bank they were very understanding, they investigated the purchases and found it was quite obvious I hadn't done them.

The estimate they gave me until I'd have the money back in my account was 3-6 months. No interest.

Perspective.

I dont think the people give a shit what happened to you..


Perspective...

[ArticMine]

2) If you are serious about security, don't advertise your genius ideas.

On the contraire, if your solution is proven and tested, you shouldn't be afraid of sharing it with the public. "Security by obscurity" is just laughable, and the source of problems like the one Bitcoinica is going through perhaps. Let's bet that nobody knows where our server is, or how we secure it. Let's just bet!

How very true. The main strength of Bitcoin is that all the security is FLOSS (Free Libre Open Source Software) and yet is an extremely secure system. Now compare this with the multi billion dollar DRM industry, which is entirely based on security by obscurity using propriety software. With DRM the "security" is usually cracked by some teenager who in the process also makes an utter fool out of some multi billion dollar corporation or trade group. This however does not prevent many very deep pockets from paying for the security by obscurity snake oil.

[disclaimer201]

Regarding "omg it's been four days you have my money i will contact the authorities":

15 odd years or so someone abused my VISA to buy software on the Internet. Back then my main card was still a debit card, connected to the bank account I used for paying bills and where I got my pay check automatically deposited.

They cleaned me out. When I contacted the bank they were very understanding, they investigated the purchases and found it was quite obvious I hadn't done them.

The estimate they gave me until I'd have the money back in my account was 3-6 months. No interest.

Perspective.

Don't give them any ideas. If coins and money are returned to us I believe the matter will turn out to be good for bitcoin. -If- You guys are already discussing about securing other btc ventures this way and are helping each other. Since trust is everything online the issue has to be resolved if any bitcoin enterprise is to have a future. Trust is time-sensitive online, much more than with a credit card company. So, you have days at most to return funds and calm customers if you ever want to have a chance at milking that cash-cow again. Don't get me wrong, Bitcoinica could still be a scam IMO as in their business model didn't work out and they threw the anchor with a staged theft before it'd be too late to hold the status quo. Of course, this is all speculation since I can hardly prove it. It would be a semi-moral solution though rather than having shit really hit the fan and becoming totally insolvent. Let's hope I'm wrong.

[tvbcof]

2) If you are serious about security, don't advertise your genius ideas.

On the contraire, if your solution is proven and tested, you shouldn't be afraid of sharing it with the public. "Security by obscurity" is just laughable, and the source of problems like the one Bitcoinica is going through perhaps. Let's bet that nobody knows where our server is, or how we secure it. Let's just bet!

How very true. The main strength of Bitcoin is that all the security is FLOSS (Free Libre Open Source Software) and yet is an extremely secure system. Now compare this with the multi billion dollar DRM industry, which is entirely based on security by obscurity using propriety software. With DRM the "security" is usually cracked by some teenager who in the process also makes an utter fool out of some multi billion dollar corporation or trade group. This however does not prevent many very deep pockets from paying for the security by obscurity snake oil.

+1

It's kind of difficult to tell if we are looking more at a 'reading' or more at a 'comprehension' problem here.  But it's hard to care that much either.

[DiabloD3]

...
Kids, here is how you secure a financial server:
...

Here's another couple of ideas:

1) Don't follow anyone's cookie-cutter solutions...particularly if they are good ones (because if they are, they will be popular.)

2) If you are serious about security, don't advertise your genius ideas.

No, virtually every secure server uses those concepts.

This is what I do: I disable root login locally AND through ssh (local root disable is borderline security through obscurity though, you have to know the l/p of people who can su before you can cause any real damage), and I disable ssh login with passwords (key only), and I also use keys with passphrases only (so even if they steal the key from me physically, they can't use it).

Installing a firewall (ie, a handful of iptables rules) is somewhat overkill, you should only be running software that can be told to listen on specific IPs (ie, listen locally only). iptables kills network performance, so only use it if you must.

Running the actual httpd/db/maild/etc on a vm that itself is stored on an encrypted image is rather popular. You can boot the machine and do basic maintenance on the VM (ie, back up everything), but you can't start it or read the image unless you have the passphrase. Its basically a ghetto way of doing LOM cheaply, especially if your VM has its own IP and the host isn't masquerading for the VM.

Bare metal over VPS only stops one single attack: host reading VM memory. Which, for financial things, is worth it for some people.

Having your own hardware coloed instead of leasing a dedi is not going to increase security: others still have causal physical access to your rack, so you'll need to rent your own locked cage if you want to take paranoia the whole 9 yards.

[casascius]

How about just ... Not keeping bitcoins on the server?

How bad would it be if all non-trivial withdraws needed up to 24h to be done manually?  Where the platform issued pgp signed and encrypted withdrawal requests that were reviewed and performed manually, offsite?

[DiabloD3]

How about just ... Not keeping bitcoins on the server?

How bad would it be if all non-trivial withdraws needed up to 24h to be done manually?  Where the platform issued pgp signed and encrypted withdrawal requests that were reviewed and performed manually, offsite?

Well, with any site that needs to send bitcoins back out you need whats known as a hot wallet, ie, a wallet that ONLY has enough to do day to day business... you setup your software to send excess coins to a cold wallet (offline or otherwise hidden on another machine), and message you if you need to manually transfer from cold to hot.

No one has $90k worth of coins in their hot wallet.

[imsaguy]

No one has $90k worth of coins in their hot wallet.

*anymore