[Payout Updates] Bitcoinica site is taken offline for security investigation

[rjk]

Some reputations could have been salvaged, maybe, a long time back with some proper communication and some action. At this point, all is pretty much lost.

This includes the reputation of any investor(s) that may have selfishly delayed the entire process just to be sure it could have been completed all in one big lump.

Taking a month to prepare to start paying out in one batch is a huge failure. Better would have been to start the payouts as soon as possible, even if it took a month to work through them, as long as some were going out each day.

And as ssaCEO said, how did all the USD go missing? This is absolutely inconceivable. Same goes for the cold wallets. What in the world is actually going on?

[zhoutong]

We have off-site backups in a different DC. It's managed by Rackspace.

If the server crashed, we have no problems of recovering. There are a lot of backups of all our main servers. It's just that these backups were deleted by the hacker.

I meant data center of a different company (a different admin panel with a different password, and an append-only configuration). That's what provides a level of actual redudancy. Using a cloud service from one company protects you if a truck crashes into one of their data centers, but offers zero protection if someone gets your admin password!

I can't afford a scalable solution that gives me the same reliability for a bootstrapped startup.

Not anymore you can't!

You can make backups for cheap (backups aren't accessed, so they don't scale with the rest of the site).

All financial transactions will be handled by payment gateways who will be responsible for their own security. Apart from that, no money is involved so I just want to consider scalability, performance, availability and cost. For me, I think AWS's EC2 instance with Load Balancer handling SSL termination (can't be DDOS'd) + RDS with snapshots and binary logs recoverable to 5 minutes ago are more than enough for me.

Most people choose to outsource security, it's just that in Bitcoin world everything is DIY.

I'm not going to use your domain service or any other zhoutong "cloud" service because its clear that you don't have plan B (contingencies). What happens when someone deletes your AWS instances and its snapshots and logs? Has the thought even crossed your mind to try and see if its possible on the AWS admin panel?

Its a truism that at the basic level, security can't be outsourced. You have to trust someone eventually (unfortunately for us, you happened to trust bitcoin consultancy), but catastrophes can only be averted by good planning.

I don't think I have to mention the common sense. Of course the database backups are going to be downloaded on a constant basis. Bitcoinica has accounting records, but they are not current enough to resume trading.

The requirement for currency isn't very important for a non-financial project if someone else (such as a domain registry) will keep the records anyway.

What I'm saying is that it's unreasonable to expect a mission-critical configuration for a project with limited financial interest involved. All possible efforts will be put into security for sure. The use of cloud services shouldn't be the argument here.

If I were to build another Bitcoin project, I will definitely consider the locked cages. But until I have the funds and the security expertise, I should probably stay away from money itself.

I recently got PCI Compliance for storing credit cards on the servers, and the certification has been recognized by a large business bank in Australia. AWS datacenters all have passed the physical security checks and required audits. But I still choose not to store any critical financial data at the moment. Instead I will use the vault services provided by reputable payment gateways.

For any of my commercial projects: Minute-interval binlogs and at least daily backups with weekly downloads. All firewalls configured properly. Two-factor auth on AWS accounts. No injection, XSS or CSRF possibilities.

No system is 100% secure. The uncompromised systems are the ones not being targeted. Outsourcing security to a responsible party will avoid the possibility of being a target as much as possible. We can already prove that Linode was not secure, but why only 8 accounts (all related to Bitcoin) were hacked? Bitcoin attracts cyber criminals and it's reasonable to expect a disproportionately frequent security attacks on Bitcoin-related projects.

However, I feel unfair for the reputation damage that wasn't even triggered by me. I'm always serious about security and even though I'm not a specialised security expert, I do have some knowledge and experience of maintaining a secure system that's enough for a SaaS project. You can criticise me for trusting 3rd parties too much, but it's still my belief that the so called 3rd parties have better security skills than me. It's just that they are being targeted. (Even Sony got hacked so many times.)

[LoupGaroux]

Okay, nice statement of outrage at being judged. How about answering the question regarding where are the USD funds? How about responding to user's emails since you are no longer busy running the service?

[zhoutong]

Okay, nice statement of outrage at being judged. How about answering the question regarding where are the USD funds? How about responding to user's emails since you are no longer busy running the service?

I have all your emails in record, and Patrick has received a copy as well. I'm not supposed to give any official replies. All I can do is give you advice on how to fill the claim form properly.

I have not yet received any instructions from Bitcoinica Consultancy.

[HorseRider]

We don't hold the funds.

Then WHO HAS ALL THE USD?

And when will you respond to emails? I want to know if my account is among the claims being processed. It's not like I say to my customers "oh sorry, can't give you your money back, Bitcoinica stole it". No. I have to pay it out of my own pocket. But I can tell them, "Bitcoinica stole your money. I was dumb enough to leave your USD with them. That would be Patrick, Zhou, Tihan, and everyone else related to that project. They say they got robbed for 18K BTC, and then all their USD magically disappeared at the same time. If you ever see anything else they do, make sure to avoid it like the plague. Now I'm paying this money personally back to you on their behalf. Hopefully they'll reimburse me, but I doubt it since they haven't responded to a single one of my emails."

+1

Bitcoinica hold the users away from their fund for too long, and this has brought inconvenience to the users.

[repentance]

Okay, nice statement of outrage at being judged. How about answering the question regarding where are the USD funds? How about responding to user's emails since you are no longer busy running the service?

To be fair, it's Bitcoinica Consultancy which needs to answer any questions regarding the USD, MtGox deposits and BTC in cold storage - they're the ones who are legally responsible for the management of the business and Zhoutong shouldn't be asked to speak on their behalf about such matters.  Who is actually managing user deposits is certainly a relevant question as it's central to the question of legal liability for Bitcoinica's debts. 

[bitcoinBull]

I don't think I have to mention the common sense. Of course the database backups are going to be downloaded on a constant basis. Bitcoinica has accounting records, but they are not current enough to resume trading.

Of course, now! Glad to hear you are learning. We're lucky some bitcoinica accounting records survived. But what a shame the user database is gone, or this quagmire of a claims process would've been much, much easier (this claims process is doing more to damage reputations of everyone involved than a reported theft ever would have).

However, I feel unfair for the reputation damage that wasn't even triggered by me.

You weren't the immediate trigger (the hacker was). But it was assumptions you made which allowed the situation to cascade into a catastrophe. (one assumption: hosting emergency-backup as a vps instance under the same rackspace account would be sufficient in the case of an emergency. another one: rackspace support guy is correct. why trust the word of a support guy? never should've had to.) That lack of preparedness is what damaged your reputation, and that's fair.

To think that you were building a new domain service while simultaneously operating bitcoinica! I'll trust you to build stuff, but not to operate it. (and thanks for reading).

[dooglus]

Anyone heard from Patrick Strateman since this happened?  Anyone know him personally?  It would be interesting to hear his take on how it went down.

Funny you should ask.  I added him on Skype maybe a month ago to tell him about a bug in the Intersango code.  The interaction was brief and to the point, and that was that.

Then a couple of days ago he messaged me out of the blue on Skype to ask me what I was wearing.

Very odd.

[zhoutong]

I don't think I have to mention the common sense. Of course the database backups are going to be downloaded on a constant basis. Bitcoinica has accounting records, but they are not current enough to resume trading.

Of course, now! Glad to hear you are learning. We're lucky some bitcoinica accounting records survived. But what a shame the user database is gone, or this quagmire of a claims process would've been much, much easier (this claims process is doing more to damage reputations of everyone involved than a reported theft ever would have).

However, I feel unfair for the reputation damage that wasn't even triggered by me.

You weren't the immediate trigger (the hacker was). But it was assumptions you made which allowed the situation to cascade into a catastrophe. (one assumption: hosting emergency-backup as a vps instance under the same rackspace account would be sufficient in the case of an emergency. another one: rackspace support guy is correct. why trust the word of a support guy? never should've had to.) That lack of preparedness is what damaged your reputation, and that's fair.

To think that you were building a new domain service while simultaneously operating bitcoinica! I'll trust you to build stuff, but not to operate it. (and thanks for reading).

I'll take the responsibility of not having proper backup plans. But no, it's not my fault to delay the process for weeks.

The thing is, people in Bitcoinica Consultancy don't trust me with money and I have no agreement with them at all. So technically I can't be responsible if I screw up the refunding process.

They are perfectly qualified to handle the whole process. But it's important for me to state a fact here: I wouldn't let everyone wait for so long. I have moderated only a few users, but they are perfectly legitimate records and they represent more than 50% of total deposits of the whole user base. It makes perfect sense to refund them immediately. I could reach this conclusion in a few days since the claim process.

I take the responsibility of the lack of records, but not the responsibility of Bitcoinica Consultancy's delay in processing refunds. They are the owners of the company since April and I had no control since then (technically I have to get approval from them if I need to change the code).

[repentance]

Anyone heard from Patrick Strateman since this happened?  Anyone know him personally?  It would be interesting to hear his take on how it went down.

Funny you should ask.  I added him on Skype maybe a month ago to tell him about a bug in the Intersango code.  The interaction was brief and to the point, and that was that.

Then a couple of days ago he messaged me out of the blue on Skype to ask me what I was wearing.

Very odd.

Maybe his Skype password was the same as his email password and the hacker is having fun with all his social networking accounts.

[S3052]

03 June 2012 23:20: We're adding extra fields to the claims database (should be finished soon), we have received the funds from Tihan to make the initial payouts. Then once that's done, the first round of payments can be finished.

That's a bit confusing.  If all that was lost was 20% of Bitcoins on hand it should have been possible to pay everyone out 80% (the initial round of payouts) without receiving funds from Tihan - additional capital should only have been required to replace the lost Bitcoins.  You should have still been in possession of 100% of USD and 100% of Mt Gox deposits.  Or were you still waiting on capital to enable you to replace Bitcoins which were lost in the Linode intrusion as well as additional funds to cover the most recent loss?

We don't hold the funds.

03 June 2012 23:20: We're adding extra fields to the claims database (should be finished soon), we have received the funds from Tihan to make the initial payouts. Then once that's done, the first round of payments can be finished.

Do you mean there will be another claim form to fill out, or are you just speaking about finalizing the current one?

Finalising the current one for internal (staff) usage. We need to track the payments we make more accurately for book keeping.

sounds very good. thanks for the update

[bitlane]

It's kind of funny that the Bitcoinica Consultancy is trying to recover their losses on the back of the interest currently being earned from their customer's/depositor's USD that is being held, but what will happen once users DEMAND interest for the funds for the 1 month of being 'invested' ? Wouldn't this pretty much negate the entire Bitcoinica Consultancy master plan ?

....and if you guys try to claim that not a single cent of interest has been generated from funds currently locked down, then it will prove once and for all that you truly believe that we are all a bunch of retards and show just what kind of regard you have for your customers.

This scam is so transparent that one would have to be a complete idiot to not see what you guys are trying to do.

My suspicions will be proven true when the USD is the LAST to get returned....taking advantage of every last day of interest, as BTC, itself, generates NONE.

[disclaimer201]

Almost one month and nothing but excuses and promises. The last posts were again, nothing but delaying tactics.

[Vod]

Subbed for updates.

Can't you just click on the "notify" link?

[dooglus]

taking advantage of every last day of interest, as BTC, itself, generates NONE.

I heard that there are ways of generating lots of interest on BTC investments.

[repentance]

It's kind of funny that the Bitcoinica Consultancy is trying to recover their losses on the back of the interest currently being earned from their customer's/depositor's USD that is being held, but what will happen once users DEMAND interest for the funds for the 1 month of being 'invested' ? Wouldn't this pretty much negate the entire Bitcoinica Consultancy master plan ?

Users can demand whatever they like but it doesn't mean the business is obliged to pay it.

[Transisto]

This is what you've missed if you haven't read this page. :

Maybe his Skype password was the same as his email password and the hacker is having fun with all his social networking accounts.

sounds very good. thanks for the update

This user is currently ignored.

Almost one month and nothing but excuses and promises. The last posts were again, nothing but delaying tactics.

Subbed for updates.

Can't you just click on the "notify" link?

only if he checks his e-mail :/

I heard that there are ways of generating lots of interest on BTC investments.

Users can demand whatever they like but it doesn't mean the business is obliged to pay it.

For F sake, THIS IS NOT A CHATROOM

[snoleo]

There are dollars and bitcoins in my Bitcoinica account.
The following 5 refunding solutions , which Bitcoinica will choose? :

1. refund dollars AND bitcoins
2. refund dollars ONLY
3. refund bitcoins ONLY
4. exchange bitcoins to dollars at SOME rate, and refund dollars
5. exchange dollars to bitcoins at SOME rate, and refund bitcoins

[Transisto]

There are dollars and bitcoins in my Bitcoinica account.
The following 5 refunding solutions , which Bitcoinica will choose? :

1. refund dollars AND bitcoins
2. refund dollars ONLY
3. refund bitcoins ONLY
4. exchange bitcoins to dollars at SOME rate, and refund dollars
5. exchange dollars to bitcoins at SOME rate, and refund bitcoins

All options but 1 and 5 are ridiculous.

The real question is HOW and WHEN,  If you aren't aware yet, they have no way to know precisely who owns what.

[LoupGaroux]

Dare we give voice to the likely Sixth Option?

6.  Continue to pay lip-service to their customers, while continuing to earn interest on their deposits, maintain a steady flow of denials, contradictory statements, and keep shifting the blame off onto others- hackers, unknown shadow-men who gain root access, strikingly inept service hosts, unsecured (mea culpa, mea maxima culpa) servers, youthful ignorance, and "it's the other guys that are deciding...", and when the world can no longer tolerate these stories a few paltry pennies will be paid out on the dollar, and the rest is lost because it A: took months to add a couple of fields to a database; B: unknown malfeasants broke into the server where the refunds were being calculated and stole everything else; C: it is Lithuanian New Year and the banks are closed, so we regret to inform you but we cannot make good on this scam, err, I mean terrible thing.