[Emergency ANN] Bitcoinica site is taken offline for security investigation

[Raoul Duke]

I'm only an employee at Acme Coins (acmecoins.com) and when I woke this morning and logged on, I noticed some strange transactions. Since everybody else was still sleeping, I called Rackspace and had the website shut down. Then I went to AcmeCoinTalk, the forum for said company, and posted what I did. I have yet to post on my official blog or put up a page on the official website because...

You forgot step 2: Post the drama thread, divert attention from the real problem and walk away with almost everybody patting you on the back and telling you what a great guy you are.

[1715253795]

1715253795

[1715253795]

1715253795

[1715253795]

1715253795

[bulanula]

I'm only an employee at Acme Coins (acmecoins.com) and when I woke this morning and logged on, I noticed some strange transactions. Since everybody else was still sleeping, I called Rackspace and had the website shut down. Then I went to AcmeCoinTalk, the forum for said company, and posted what I did. I have yet to post on my official blog or put up a page on the official website because...

You forgot step 2: Post the drama thread, divert attention from the real problem and walk away with almost everybody patting you on the back and telling you what a great guy you are.

Are you implying this was STAGED

No way !

Still no official word from Intersango. Gotta love denying responsibility ...

[Clipse]

Still waiting for the logs.................................................

Hacker deleted all the evidence mate. The only evidence we have is the 18K TX. What more do you want

Are we getting payed the interest on our USD while this is going ? Can they afford to really buy 18K BTC again to compensate ? I doubt it.

Ever since they started baiting users into interest rates if coins were left there I felt this turned into a huge collapsing ponzi scheme. The lack of urgency by those in control/ownership of bitcoinica is also very "charming".

[Raoul Duke]

I'm only an employee at Acme Coins (acmecoins.com) and when I woke this morning and logged on, I noticed some strange transactions. Since everybody else was still sleeping, I called Rackspace and had the website shut down. Then I went to AcmeCoinTalk, the forum for said company, and posted what I did. I have yet to post on my official blog or put up a page on the official website because...

You forgot step 2: Post the drama thread, divert attention from the real problem and walk away with almost everybody patting you on the back and telling you what a great guy you are.

Are you implying this was STAGED

No way !

Still no official word from Intersango. Gotta love denying responsibility ...

Not implying shit, dude. Was just filling the elipse in Phinn's post, nothing more.

[chsados]

so wtf is going on... is bitcoinca going back online monday or not?  am i missing any official announcement other than that blogpost a few days ago?

[muyuu]

so wtf is going on... is bitcoinca going back online monday or not?  am i missing any official announcement other than that blogpost a few days ago?

Apparently not. Looks like they'll rebuild. Maybe you're missing the part that everybody will be force-liquidated and reimbursed.

See here:
https://bitcointalk.org/index.php?topic=81045.msg897900#msg897900

[MemoryDealers]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

[BadBitcoin (James Sutton)]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

as long as I get the 4700 USD I had sitting in bitcoinica that all of a sudden got locked out from me, then I will be angry I'd certainly join the fight against the hackers, if not, there will be hell to pay by the person whos site I entrusted with my money.

[shockD]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

Anger towards the hackers is certainly appropriate but (assuming here the bank is not insured) if your bank had a back door made only of plate glass and left the vault open, allowing a thief to run off with your money, some anger towards said bank would also be appropriate. Especially if they allowed it to happen twice.

One could certainly also lay blame on those who re-deposited money they couldn't afford to lose in said bank after the first incident (even if the bank did reimburse them the first time).

 

[MemoryDealers]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

Anger towards the hackers is certainly appropriate but (assuming here the bank is not insured) if your bank had a back door made only of plate glass and left the vault open, allowing a thief to run off with your money, some anger towards said bank would also be appropriate. Especially if they allowed it to happen twice.

One could certainly also lay blame on those who re-deposited money they couldn't afford to lose in said bank after the first incident (even if the bank did reimburse them the first time).

 

Because no money was lost by customers last time,  and no money will be lost by customers this time,  there is NO REASON to be angry at Bitcoinica.

[geebus]

<level headed speech>
I don't want to sound like I'm overly defending anyone here, because I'm merely trying to calm the tension here...

It seems the websites (Bitcoinica, the blog, BitcoinConsultancy) are down because they were all hosted with RackSpace (?) and as Zhou said in the OP, they had the servers shut down.

It's part of the process. The MtGox hack, Tradehill Closing, and now this, are all giant clusterfuck situations, but they get dealt with in time. I know it's not an assurance, and no warm-fuzzy feelings are being generated by this, but it's only been a matter of days and I'm sure everyone involved is still trying to get a grip on exactly what may have been compromised, especially with the ominous threat of a "mass leak" overhead, and presumably, far more USD at risk than the BTC that was stolen (Admittedly, my own assumption.) Look at the bright side though, they have stated they are working on methods to reimburse their customers... this is leagues better than something like the MyBitcoin fiasco.

Personally, I still have accounts at Gox that were never recovered.
I still have an open balance at TH that was never recovered.
I probably wont get my Bitcoinica balance back...

...but the reality of the situation is, you can't blame the chieftain of the village you live in if raiders loot and pillage everything in the middle of the night. You can, however, realize that you made the conscious choice to have your BTC/USD held with a 3rd-party, and be cooperative, understanding and patient when it comes to getting it back.

Honestly, I hope everyone gets their money back, myself included, but pitchforks and torches don't help.

My condolences go out to Zhou his team.
</level headed speech>

Alright... commence with the "OMGWTFBBQ?!?!?"

[shockD]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

Anger towards the hackers is certainly appropriate but (assuming here the bank is not insured) if your bank had a back door made only of plate glass and left the vault open, allowing a thief to run off with your money, some anger towards said bank would also be appropriate. Especially if they allowed it to happen twice.

One could certainly also lay blame on those who re-deposited money they couldn't afford to lose in said bank after the first incident (even if the bank did reimburse them the first time).

 

Because no money was lost by customers last time,  and no money will be lost by customers this time,  there is NO REASON to be angry at Bitcoinica.

I'd agree that people up in arms are being irrational and also feel they were foolish to leave a large amount of money in a service that was just recently hacked. I believe that bitcoinica will reimburse people on the terms that they've stated. To say no money is lost to those with open positions is a bit naive though and to call the abilities of the bitcoinica ops/devs into question is certainly rational. I have no personal stake in any of this. In a perfect world, none of us would need passwords on our systems or accounts or locks on our doors. The fact is that we do need those things.

I would like to see the people running financial applications take security a bit more seriously though and I think all the attention focused on this incident is a very good thing in that respect.

[proudhon]

I think everyone should keep in mind that the real person/group to be angry with is the hackers, not Bitcoinica.
Anyone who is smart enough to figure out how to steal 18K BTC from Bitcoinica is more than smart enough to do honest work.  I hope Zhou goes on to have a long successful career while the hackers and other thieves burn in hell.

Direct your anger towards the hackers!

Anger towards the hackers is certainly appropriate but (assuming here the bank is not insured) if your bank had a back door made only of plate glass and left the vault open, allowing a thief to run off with your money, some anger towards said bank would also be appropriate. Especially if they allowed it to happen twice.

One could certainly also lay blame on those who re-deposited money they couldn't afford to lose in said bank after the first incident (even if the bank did reimburse them the first time).

 

Because no money was lost by customers last time,  and no money will be lost by customers this time,  there is NO REASON to be angry at Bitcoinica.

Bitcoinica is dealing with this in a stand up way and bravo to everyone involved. However to say there is no reason to be upset for that data loss and disrupted service I respectfully disagree.

I'm not upset about this, and yeah things happen but to call out the people who are upset as being irrational is a bit extreme.

Sorry, I really haven't followed this whole event closely enough.  What data was lost, besides the bitcoins?

[geebus]

Sorry, I really haven't followed this whole event closely enough.  What data was lost, besides the bitcoins?

Don't know yet. The hacker left a binary message in bitcoin transactions that said, "expect mass leak" or something to that effect... so, we can speculate and assume that more than just BTC was taken, but no one knows for sure yet, or if they do, they aren't saying.

[muyuu]

All you with significant balances in bitcoinica can feel lucky that there hasn't been any remarkable swing in the market price, because if that happened either way it could make the situation untenable to bitcoinica who could be unable to reimburse everybody.

That's why I think they should be settling this force-liquidation business ASAP.

Imagine there was a swing in price in any direction, and they'd be looking to dozens of claims by people with heavy positions settled either way. How over-leveraged are they if that happens? who knows. People are complaining about being force-liquidated now, imagine if they were looking at potential loses several times bigger from not having their positions honoured.

If you all want your money back it's in your best interest that those with ridiculous claims don't get them

That all said, I'm still willing to believe many people will continue entrusting big quantities to the site. Because this time it will be safe for real and expect a very one-sided ToS doc coming up when they're back, too. Just look at MtGox for reference (on both accounts, the ToS and having a massive number of users despite the ridiculous ToS and the history of failures).

[Krakonos]

Don't know yet. The hacker left a binary message in bitcoin transactions that said, "expect mass leak" or something to that effect... so, we can speculate and assume that more than just BTC was taken, but no one knows for sure yet, or if they do, they aren't saying.

Yeah, or he's just playing us. I think he is, but it's just as probable that I'm wrong and he isn't and has all of the verification documents decrypted already...

[arby]

In the drama thread, Mr. Tong carefully mentioned ' I continued to become the sole operator until Team Intersango took over two weeks ago', basically denying that he is responsible for the 'hack'.

How do we know the website was not 'hacked' before and now that the guy saw some professionals are actually taking over decided to make his move?
How do we know that it was not one of your old employees mad because someone else is taking over his job?

I dont know much facts about this, but it was very lame mentioning that they took over 2 weeks ago, and you have nothing to do with it anymore, Mr. Tong.

Personally I do not know why the intersango team decided to collaborate with this website, knowing they have reputation and security issues, but I may have an idea, money.

Anyway this story is really suspicious, also about the sad story about leaving the bitcoin world, and all those people crying on that thread, he is leaving probably with millions in his pocket, so I would not feel
that sad that 'he needs to leave now, so long my good friends I love you all, see you in Australia'.

Also the story about the binary message, I mean how cool is that from the 'hacker', he thinks he probably hacked the Pentagon's computers and he controls nuclear weapons, not some poor administered rackspace server.
Bragging 'hackers' are the worse kind, they only remind me of those defaced websites saying 'XXX WAS HERE', coloured in red. The good thing about it, is that they get caught first.

And on another side note, maybe Mr. Tong can explain in more detail how the 'hacker' got in, that would add credibility to his facts, and also would help others prevent such things from happening.
 

[muyuu]

Personally I do not know why the intersango team decided to collaborate with this website, knowing they have reputation and security issues, but I may have an idea, money.

Intersango have already admitted to having an ownership share in the new Bitcoinica, IIRC.

And on another side note, maybe Mr. Tong can explain in more detail how the 'hacker' got in, that would add credibility to his facts, and also would help others prevent such things from happening.

Won't happen with Bitcoinica out of operation, will it.

[zhoutong]

Update: How the hacker hacked Bitcoinica

I don't think this should be a secret, so I would just share my version of the story.

- I received several emails regarding password reset and finding out the username for our Rackspace account.
- I initially thought it was Patrick, because he did a password reset a few days ago, but I became suspicious when I realized that someone forgets the username of the account! (So it must not be Bitcoinica team member.)
- I immediately set the password back, and log in to the account. I SSH'd into the Bitcoin wallet server and found that everything is gone.
- This thread was posted and I tried to contact Rackspace the lock down the account.
- They suspended all servers, so that the hacker couldn't log in. However, despite two password changes and server suspension, the hacker is still in the session. I asked Rackspace to terminate his session but it seems that they don't know how to do it.
- The hacker recreated the server using our database backup, and possibly got the database successfully.
- Later we found out that Patrick's email server was compromised, and since he is in our mailing list, all emails sent to info@bitcoinica.com were delivered to his compromised email account.
- We are now working on a settlement plan. Patrick is in charge of the claim page.

If anything of the following happened this would be prevented:

- Patrick's email was not added to the mailing list, and he used Bitcoinica email instead.
- Rackspace should just terminate the sessions then at least the database would be safe.
- We should not use the official Bitcoin client because it's very hard to secure it without large investments and affecting instant withdrawals in large amounts.

I hope this insight can help some of you understand our situation right now.

[davout]

Update: How the hacker hacked Bitcoinica

I don't think this should be a secret, so I would just share my version of the story.

[...]

- Patrick's email was not added to the mailing list, and he used Bitcoinica email instead.
- Rackspace should just terminate the sessions then at least the database would be safe.
- We should not use the official Bitcoin client because it's very hard to secure it without large investments and affecting instant withdrawals in large amounts.

I hope this insight can help some of you understand our situation right now.

It's good that you're publishing all this.
I have a question, do you intend to publish the source code in one way or another ?
I'd love to take a look at it   (as a pro Rails developer)
Your app looked really good, a pity that some much trouble comes from infrastructure/admin issues.