[Payout Updates] Bitcoinica site is taken offline for security investigation

[repentance]

Doesn't look like this is going well...

Whatever else might seem suspect, I'm inclined to believe that there is legal wrangling going on.  If I was the limited partner, I'd want everything about the claims and disbursement process reviewed by my lawyer and accountant to ensure that I was immune to any further (assuming I had any to begin with - which isn't certain in this case) liability once that process was completed.  Likewise, the general partners need to ensure that the processes they use aren't open to later legal challenge as they are responsible for all of Bitcoinica's debts and liabilities.

Was it, or was it not, possible to recover lost data from Rackspaces servers during that first 48 hour window? I don't understand how Rackspace is able to recover data from their servers following a "disaster" yet unable to after a phone call is made to them about data being erased by other means

.

I think they're saying that they can't recover information at the individual client level unless you buy specialised back-up services but if their whole server farm suffers a catastrophic event they have backups from which they can restore.  So they can restore the whole thing if needed but they can't restore selectively because they're backing up their data rather than that of individual clients is the way I'm reading it.

[Phinnaeus Gage]

Doesn't look like this is going well...

Whatever else might seem suspect, I'm inclined to believe that there is legal wrangling going on.  If I was the limited partner, I'd want everything about the claims and disbursement process reviewed by my lawyer and accountant to ensure that I was immune to any further (assuming I had any to begin with - which isn't certain in this case) liability once that process was completed.  Likewise, the general partners need to ensure that the processes they use aren't open to later legal challenge.

I keep forgetting about that Angel Investor. I've sure that not only is he concern about his investment but, moreover, that his good name is not attached to any rogue entity, hence him probably having his lawyer(s) look into this issue, with letterheads already certifiably mailed. Only if we were so lucky during the MyBitCoin episode, its one year anniversary fast approaching.

~Bruno~

[Serge]

The entire Cloud Sites FTP structure is backed up every four hours, which totals six daily backups. Those backups are rolled into a nightly backup, which are retained for two days. However, these backups are for disaster recovery on the server side. If for any reason a storage node on our side were to crash, our backups will be there to replace any lost data.

That said, we recommend that you make periodic backups of your site and data to your local computer since we are unable to extract an individual site's data from the nightly backups.

Was it, or was it not, possible to recover lost data from Rackspaces servers during that first 48 hour window? I don't understand how Rackspace is able to recover data from their servers following a "disaster" yet unable to after a phone call is made to them about data being erased by other means.

I'm puzzled!

~Bruno~

it means they do backup entire cloud cluster with all their cloud customer's sites in one snapshot, which in case of disaster would be more or less easy to restore, the whole cloud structure. problem is they cannot extract data of any individual client from it. the answer to your question is no, it wasn't possible by the sound of that citation.

[rjk]

The entire Cloud Sites FTP structure is backed up every four hours, which totals six daily backups. Those backups are rolled into a nightly backup, which are retained for two days. However, these backups are for disaster recovery on the server side. If for any reason a storage node on our side were to crash, our backups will be there to replace any lost data.

That said, we recommend that you make periodic backups of your site and data to your local computer since we are unable to extract an individual site's data from the nightly backups.

Was it, or was it not, possible to recover lost data from Rackspaces servers during that first 48 hour window? I don't understand how Rackspace is able to recover data from their servers following a "disaster" yet unable to after a phone call is made to them about data being erased by other means.

I'm puzzled!

~Bruno~

it means they do backup entire cloud cluster with all their cloud customer's sites in one snapshot, which in case of disaster would be more or less easy to restore, the whole cloud structure. problem is they cannot extract data of any individual client from it. the answer to your question is no, it wasn't possible by the sound of that citation.

Sooo... the extra cost additional backups are just more snapshots on the LVM, essentially? If so, that's lame. Paying extra for something that is already happening is stupid. However, if the extra paid option is actually a full disk clone, I would suspect that it would therefore be trivial to restore deleted data from backup in that case.

[repentance]

Sooo... the extra cost additional backups are just more snapshots on the LVM, essentially? If so, that's lame. Paying extra for something that is already happening is stupid. However, if the extra paid option is actually a full disk clone, I would suspect that it would therefore be trivial to restore deleted data from backup in that case.

One of the options includes back up to other media including rotated tapes.  You have a choice about whether to have absolutely everything including your OS backed up or just specific items.  They actually discuss which backup processes are best for what types of businesses and how the various options affect restoration time.

[repentance]

How else would you end an operation like Bitcoinica if you find out it doesn't work out? Certainly not announcing something like: "Dear customers, due to general shortcomings of the system we have to close our doors for good. We are very sorry for your loss."

Businesses cease trading due to insolvency every day of the week.  You don't need to invent a theft in order to justify closing down a business which is running at a loss (or even one which is running at profit).

[rjk]

Sooo... the extra cost additional backups are just more snapshots on the LVM, essentially? If so, that's lame. Paying extra for something that is already happening is stupid. However, if the extra paid option is actually a full disk clone, I would suspect that it would therefore be trivial to restore deleted data from backup in that case.

One of the options includes back up to other media including rotated tapes.  You have a choice about whether to have absolutely everything including your OS backed up or just specific items.  They actually discuss which backup processes are best for what types of businesses and how the various options affect restoration time.

OK, that makes sense. So the "default" backups are at server level, backing up all the hosted virtual machines simultaneously, and are only used to restore in the event of a catastrophic storage server failure. (Most "clouds" have all their storage abstracted onto dedicated SAN or DAS devices.)

And the paid version is the same as the dedicated servers I assume - they install a software agent running as its own user, which then backs up whatever directories and databases you request. That's how it works on our dedis, anyway.

[Serge]

Sooo... the extra cost additional backups are just more snapshots on the LVM, essentially? If so, that's lame. Paying extra for something that is already happening is stupid. However, if the extra paid option is actually a full disk clone, I would suspect that it would therefore be trivial to restore deleted data from backup in that case.

i'm no expert on backups but know there are solutions (probably most of them) where you can't extract specific data from a backup, only everything at once, and it must be slow as hell. i can understand their position - they would not nuke all their cloud clients to last backed-up snapshot just because of a single client request, doesn't make sense. and that's why they offer optional backup services  for clients who do require it

[zhoutong]

I almost forgot to mention, I know people who work at rackspace and I've talked to them.  Your logs & database being deleted is effectively a non-issue, it's a pain to recover but you can bet that they have the capability of recovering every last byte of missing information.

This has bugged me as well.  Until this episode I had the impression that Rackspace was a serious hosting provider.  Not some garage with a couple of racks on UPS and a fat ADSL line.  A serious hosting provider keep multiple backups of customer data off-site, because losing a lot of customer data due to some catastrophic event means losing their business.  Unlinking it from a web page just makes the data a bit more inconvenient to get to.  Impossible for the customer, but in no way impossible for Rackspace.  The data may be older than current, but I find it hard to believe that off-site backups were instantly deleted along with the servers.  Backup systems just aren't built for easy deletion.

Perhaps someone from Bitoinica can comment on how they have worked with Rackspace to rescue data?

We have talked to a manager and he confirms that no data can be recovered. We have even offered a $10,000 tip for any information recovered, but later they got the bad news again.

Rackspace shouldn't be used for serious applications, because of the following "features":

- You can own all servers in an account with an email.
- You can't force someone to log out, not even any Rackspace employee.
- You can suspend the servers through customer support. They will say it's safe. But anyone can delete the servers.
- When you delete something, even in Cloud Files, it's permanent.
- When the thief is in your account, you can't do anything to prevent him from doing anything destructive.

For these reasons, I personally will never use Rackspace Cloud again unless they address all of these issues. AWS is way more secure than them.

[zhoutong]

The entire Cloud Sites FTP structure is backed up every four hours, which totals six daily backups. Those backups are rolled into a nightly backup, which are retained for two days. However, these backups are for disaster recovery on the server side. If for any reason a storage node on our side were to crash, our backups will be there to replace any lost data.

That said, we recommend that you make periodic backups of your site and data to your local computer since we are unable to extract an individual site's data from the nightly backups.

Was it, or was it not, possible to recover lost data from Rackspaces servers during that first 48 hour window? I don't understand how Rackspace is able to recover data from their servers following a "disaster" yet unable to after a phone call is made to them about data being erased by other means.

I'm puzzled!

~Bruno~

According to an unknown source, the data retention for deleted servers is 12 hours. However, Rackspace suspended the servers made me feel that they are "safe", because I couldn't do anything against the servers. No one, not even the Rackspace manager knew that the servers can be deleted.

When I was ready to re-start the servers and continue Bitcoinica operations I found everything gone. And it's 17 hours since deletion.

[M4v3R]

The week is coming to end and I believe you have said that by the end of the week some people will get refunded. So how is it?

[zhoutong]

The week is coming to end and I believe you have said that by the end of the week some people will get refunded. So how is it?

Even though I'm in their mailing list, I don't have any official progress information.

I wasn't involved in the process except for very little moderation work. I think they are almost ready now. Hope someone from Bitcoinica Consultancy can post an update soon.

[repentance]

The week is coming to end and I believe you have said that by the end of the week some people will get refunded. So how is it?

This update was posted just under 12 hours ago.

02 June 2012 21:26: The process is at an impasse because of some legal wrangling.

https://bitcointalk.org/index.php?topic=84042.msg937203#msg937203

It's currently just after 8am Sunday in London, so perhaps there'll be some progress later in the day.

[ssaCEO]

I've sent numerous emails to verify@bitcoinica.com and still never received one single reply.

No answer to my question on the status of my claim, which is completely in USD. No reply whatsoever. Is anyone actually reading the mail there? Because it seems like no one else has received a response either.

[BadBitcoin (James Sutton)]

I've sent numerous emails to verify@bitcoinica.com and still never received one single reply.

No answer to my question on the status of my claim, which is completely in USD. No reply whatsoever. Is anyone actually reading the mail there? Because it seems like no one else has received a response either.

I don't think thats a reply to inbox, iirc they use it to match email+claim and nothing else.

[naima53]

I sent several e-mail, including screenshots of all transfers Gox (the only source of funding), forwarded to the email-confirmation Bitcoinica No answer. Nothing at all.  

I'm beginning to think that cracking did not have ... (was - no hacking. Just a way to prevent washout. (drain) )

If so - I hope that well paid. Why do you tighten the process? Return the money (it's a penny) - start the service on a reliable host, and earn more .. The trust has a habit of recovering. Moreover, everyone here understands that it's "growing pains" ..

edit. it`s penny, compared to how much can give a "cash cow" in the future

[bitcoinBull]

For these reasons, I personally will never use Rackspace Cloud again unless they address all of these issues. AWS is way more secure than them.

But that's still no excuse for not having offline backups. If you weren't online to notice the unauthorized rackspace session, the Rackspace admin "delete servers" bug (unable to disable) would still be an unknown bug/feature.

As for AWS, remember last year when bitomat.pl lost 17k BTC (iirc) in the blink of an eye when their AWS VPS was rebooted? MtGox bought them out and gauranteed depositor funds.

Don't trust a "cloud". (this is opposed to: first I trusted Linode, then I trusted Rackspace, and after getting burnt by Rackspace I finally decided to trust Amazon Web Services). Live and learn.

[Raoul Duke]

For these reasons, I personally will never use Rackspace Cloud again unless they address all of these issues. AWS is way more secure than them.

Some guys have the fate of repeating the same mistakes over and over and over again.

[ssaCEO]

For these reasons, I personally will never use Rackspace Cloud again unless they address all of these issues. AWS is way more secure than them.

Some guys have the fate of repeating the same mistakes over and over and over again.

Yeah. Seriously? The lesson from this should be DEDICATED HARDWARE IN A LOCKED CAGE, WITH NO ROOT PASSWORDS SENT BY EMAIL, AND NO EMAIL HOSTED BY A THIRD PARTY. It doesn't matter if it's 2x as expensive, or even 10x as expensive. It's called being "insurance poor". When you try to cut as many corners as possible to save a dollar, you end up losing $90,000.

Don't be cheap with other people's safety. No one in their right mind would use a fucking VPS for anything, let alone leave the keys to the castle on one.

And where the hell is my response email?

[Phinnaeus Gage]

WITH NO ROOT PASSWORDS SENT BY EMAIL

Herein lies the weakest link.

~Bruno~


Welcome to Fort Knox. The key is in the mailbox.