A public plea for help regarding Bitcoinica and my 24,841 BTC

[Serge]

This is what strikes me as odd.

When the Bitcoinica servers were deleted, I wonder... "What's Rackspace's response to Bitcoinica's queries of 'we really need your help, surely you have a backup?"  Nowhere did I read the results of them pushing Rackspace for the backups.  You would think that if Rackspace doesn't keep backups for their own business continuity and that once someone pushes delete, data is gone forever, that that would be a significant public relations fiasco for Rackspace.  I find it hard to believe the data is truly gone... unless of course nobody asked.

Even at my payroll business, we take backups and we don't really offer restorations when people delete their own data because it's to save our ass not theirs, but if someone REALLY needed a restore because it meant hundreds of thousands of dollars to their customers and they were willing to pay a grand or two in fees to prove it, I'd flip over backwards for them and restore whatever the hell they want.  I am highly suspicious of the notion that the data was gone the moment it was deleted.  And the fact that screenshots were taken of the deletion and paraded around - just in case virtual servers being deleted sounded too stupid to be believable - makes it even more ridiculous to me.

When 40k BTC and 40k USD moves somewhere, I am amazed that we hear nothing of where it went - no BTC withdrawal address so we can track the coins - and I'm genuinely interested to hear how anybody got 40k USD out of MtGox instantly when the usual wait time is much longer.  Why isn't anybody telling us "our history shows it went out as a wire" or "our history shows it went out as a MtGox code and MtGox has frozen whoever redeemed it"...

Rackspace's cloud keeps backup of all the could, it can't be used to recover data of a single client. That backup is kept in the event of emergency if anything drastic happens to Rackspace's infrastructure, that's why they offer optional offsite backup at extra if a customer requires it on individual level
Zhou has said before that they offered Rackspace $10k to help recover deleted serves; Rackspace simply couldn't help. RS would have to nuke all their cloud client's servers and restore them back (which also can be assumed isn't instantaneous process)

[1714118489]

1714118489

[1714118489]

1714118489

[1714118489]

1714118489

[1714118489]

1714118489

[casascius]

Rackspace's cloud keeps backup of all the could, it can't be used to recover data of a single client. That backup is kept in the event of emergency if anything drastic happens to Rackspace's infrastructure, that's why they offer optional offsite backup at extra if a customer requires it on individual level
Zhou has said before that they offered Rackspace $10k to help recover deleted serves; Rackspace simply couldn't help. RS would have to nuke all their cloud client's servers and restore them back (which also can be assumed isn't instantaneous process)

It's hard for me to picture a scenario where restoring a backup would "require" them to completely disrupt their clients' servers.  Surely they can restore it to other equipment that's not being used?  I mean, when I say I can "flip over backwards" and restore my backups for somebody, it doesn't mean I"m going to shut down all my services because I'm pretending that the backup can only be restored to the same hardware it was taken from - it means I'm going to grab some unused machine and restore to that instead.

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

At least if they have made the claim that they offered $10k and it wasn't accepted, that's better than what I had thought before, which was that no significant efforts were made (I don't follow every post on this topic).

[Clipse]

Rackspace's cloud keeps backup of all the could, it can't be used to recover data of a single client. That backup is kept in the event of emergency if anything drastic happens to Rackspace's infrastructure, that's why they offer optional offsite backup at extra if a customer requires it on individual level
Zhou has said before that they offered Rackspace $10k to help recover deleted serves; Rackspace simply couldn't help. RS would have to nuke all their cloud client's servers and restore them back (which also can be assumed isn't instantaneous process)

It's hard for me to picture a scenario where restoring a backup would "require" them to completely disrupt their clients' servers.  Surely they can restore it to other equipment that's not being used?  I mean, when I say I can "flip over backwards" and restore my backups for somebody, it doesn't mean I"m going to shut down all my services because I'm pretending that the backup can only be restored to the same hardware it was taken from - it means I'm going to grab some unused machine and restore to that instead.

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

At least if they have made the claim that they offered $10k and it wasn't accepted, that's better than what I had thought before, which was that no significant efforts were made (I don't follow every post on this topic).

I call bullshit, no half arse company(and mind you rackspace is fairly competent) would reject $10k for such a recovery job which they should have procedures in place to do.

[Serge]

I almost forgot to mention, I know people who work at rackspace and I've talked to them.  Your logs & database being deleted is effectively a non-issue, it's a pain to recover but you can bet that they have the capability of recovering every last byte of missing information.

This has bugged me as well.  Until this episode I had the impression that Rackspace was a serious hosting provider.  Not some garage with a couple of racks on UPS and a fat ADSL line.  A serious hosting provider keep multiple backups of customer data off-site, because losing a lot of customer data due to some catastrophic event means losing their business.  Unlinking it from a web page just makes the data a bit more inconvenient to get to.  Impossible for the customer, but in no way impossible for Rackspace.  The data may be older than current, but I find it hard to believe that off-site backups were instantly deleted along with the servers.  Backup systems just aren't built for easy deletion.

Perhaps someone from Bitoinica can comment on how they have worked with Rackspace to rescue data?

We have talked to a manager and he confirms that no data can be recovered. We have even offered a $10,000 tip for any information recovered, but later they got the bad news again.

Rackspace shouldn't be used for serious applications, because of the following "features":

- You can own all servers in an account with an email.
- You can't force someone to log out, not even any Rackspace employee.
- You can suspend the servers through customer support. They will say it's safe. But anyone can delete the servers.
- When you delete something, even in Cloud Files, it's permanent.
- When the thief is in your account, you can't do anything to prevent him from doing anything destructive.

For these reasons, I personally will never use Rackspace Cloud again unless they address all of these issues. AWS is way more secure than them.

more discussion re this following in the thread after that post https://bitcointalk.org/index.php?topic=84042.msg937829#msg937829

[Bitcoin Oz]

Rackspace's cloud keeps backup of all the could, it can't be used to recover data of a single client. That backup is kept in the event of emergency if anything drastic happens to Rackspace's infrastructure, that's why they offer optional offsite backup at extra if a customer requires it on individual level
Zhou has said before that they offered Rackspace $10k to help recover deleted serves; Rackspace simply couldn't help. RS would have to nuke all their cloud client's servers and restore them back (which also can be assumed isn't instantaneous process)

It's hard for me to picture a scenario where restoring a backup would "require" them to completely disrupt their clients' servers.  Surely they can restore it to other equipment that's not being used?  I mean, when I say I can "flip over backwards" and restore my backups for somebody, it doesn't mean I"m going to shut down all my services because I'm pretending that the backup can only be restored to the same hardware it was taken from - it means I'm going to grab some unused machine and restore to that instead.

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

At least if they have made the claim that they offered $10k and it wasn't accepted, that's better than what I had thought before, which was that no significant efforts were made (I don't follow every post on this topic).

Maybe there are other reasons they are refusing to restore the data. They are in the US and can be pressured by the feds and we know how much the US government loves bitcoin.....

I imagine a series of massive hacks would destroy confidence dont you ?

[teek]

Rackspace's cloud keeps backup of all the could, it can't be used to recover data of a single client. That backup is kept in the event of emergency if anything drastic happens to Rackspace's infrastructure, that's why they offer optional offsite backup at extra if a customer requires it on individual level
Zhou has said before that they offered Rackspace $10k to help recover deleted serves; Rackspace simply couldn't help. RS would have to nuke all their cloud client's servers and restore them back (which also can be assumed isn't instantaneous process)

It's hard for me to picture a scenario where restoring a backup would "require" them to completely disrupt their clients' servers.  Surely they can restore it to other equipment that's not being used?  I mean, when I say I can "flip over backwards" and restore my backups for somebody, it doesn't mean I"m going to shut down all my services because I'm pretending that the backup can only be restored to the same hardware it was taken from - it means I'm going to grab some unused machine and restore to that instead.

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

At least if they have made the claim that they offered $10k and it wasn't accepted, that's better than what I had thought before, which was that no significant efforts were made (I don't follow every post on this topic).

I call bullshit, no half arse company(and mind you rackspace is fairly competent) would reject $10k for such a recovery job which they should have procedures in place to do.

Yeah, none of that adds up at all.. They could restore the VM's if they really wanted too, there is no technical barrier for them to do so that money can't fix, that's for certain.  I mean even if they were doing LUN level backup and had to recover 10, 50, 100 customers or something in order to pull the VM's out of the LUN, they could have restored it to some scratch space took what they needed and scrapped the rest.

We use ZFS for all our VM's,  if something like that happened on our systems, we mount the earlier snapshot and boot the VM right back up again in a sandbox, take whatever we need, and that's it.

Rackspace probably has similar technologies, worst case scenario they have to restore 100 customers to a sandbox,  10k, 20k, 50k, 100k,  any of these amounts would be trivial in this case.  I'm sure if you got escalated to the right level of management it would have happened, had they acted QUICKLY and DECISIVELY, which unfortunately, we know is NOT what they do..

Which is why this unfortunately does all look like the long con, none of this shit adds up, its just one amateur hour shit show that just keeps going and going and going..

[mtminer]

Please help me.
I have 24,841 bitcoins on deposit with Bitcoinica.
I have provided them with my full and complete account history including every transaction since day one.
There is no question as to exactly how much I am owed. (I'm happy to provide proof to other trusted entities)
I have patiently been asking for them for my money back for months, but they have refused to return even a single Bitcoin to me.
// snip...

I will post another update once I know more. I'm guessing that payments will have to take a forced 30% cut. This has cost everyone a lot of money, time and stress dealing with this mess. We are actively losing money from dealing with the payouts.

In the main thread he updated it to they still have 2/3 of the claim money. so ~ another 80,000 BTC.

Good luck!

[2112]

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

I don't know Rackspace's procedures nor your procedures; but I can offer you a clean explanation of not being able to restore the individual backups. I'm kinda familiar how other kinda-cloud provider does backups.

All the backups are encrypted with a transactional key. There are separate keys for each VM and for each incremental backup. The keys are stored only as long as the accounting record for the VM is open. Once the VM is deleted and summarized for the purpose of charging all transactional keys are deleted zeroized (I forgot to use the proper lingo).

The encryption key deletion zeroization is a security precaution against many inside attacks as well as against compromised backup media.

The above procedure is actually an important selling point. If you don't do that internaly for the 3rd party data then you should start doing this as soon as practical to lower your own liability.

http://en.wikipedia.org/wiki/Hardware-based_full_disk_encryption
http://www.hgst.com/internal-drives/self-encrypting-drives/

[Serge]

I almost forgot to mention, I know people who work at rackspace and I've talked to them.  Your logs & database being deleted is effectively a non-issue, it's a pain to recover but you can bet that they have the capability of recovering every last byte of missing information.

This has bugged me as well.  Until this episode I had the impression that Rackspace was a serious hosting provider.  Not some garage with a couple of racks on UPS and a fat ADSL line.  A serious hosting provider keep multiple backups of customer data off-site, because losing a lot of customer data due to some catastrophic event means losing their business.  Unlinking it from a web page just makes the data a bit more inconvenient to get to.  Impossible for the customer, but in no way impossible for Rackspace.  The data may be older than current, but I find it hard to believe that off-site backups were instantly deleted along with the servers.  Backup systems just aren't built for easy deletion.

Perhaps someone from Bitoinica can comment on how they have worked with Rackspace to rescue data?

Rackspace offers lots of different hosting options with varying levels of security and varying cost.  At least on some plans, scheduled and on-demand backups are an optional service for which Rackspace charges.

Rackspace Cloud Servers include both Scheduled and On-Demand snapshots.  This is an optional service that will incur storage and bandwidth charges on Cloud Files, but the convenience of easily restoring from saved images is extremely valuable.

Does Rackspace back up my Cloud Server?
No, your Cloud Server does not get backed up until you configure and schedule backups. To learn how, please visit the knowledge center article here.

The entire Cloud Sites FTP structure is backed up every four hours, which totals six daily backups. Those backups are rolled into a nightly backup, which are retained for two days. However, these backups are for disaster recovery on the server side. If for any reason a storage node on our side were to crash, our backups will be there to replace any lost data.

That said, we recommend that you make periodic backups of your site and data to your local computer since we are unable to extract an individual site's data from the nightly backups.

And yes, Rackspace does offer fully managed backup and recovery services - at a price.

http://www.rackspace.com/managed_hosting/services/proservices/disasterrecovery/

http://www.rackspace.com/managed_hosting/services/storage/managedbackup/

[Bitcoin Oz]

Probably mtgox codes used at other exchanges.

How does that get past AMl ans KYC laws then ? You cant just withdraw 40 000 of anything without them having your government ID.

[teek]

Even the whole idea that their backups are only useful for restoring the "entire cloud", I just don't buy that.  The news recently reported they hit 100k customers - let's say each customer has an average of 30 gigs of data.  Do they back up their entire cloud to a single 3-petabyte file and that's why it can only be restored on an all-or-nothing basis?  What kind of media do they use to store a file this big?

I don't know Rackspace's procedures nor your procedures; but I can offer you a clean explanation of not being able to restore the individual backups. I'm kinda familiar how other kinda-cloud provider does backups.

All the backups are encrypted with a transactional key. There are separate keys for each VM and for each incremental backup. The keys are stored only as long as the accounting record for the VM is open. Once the VM is deleted and summarized for the purpose of charging all transactional keys are deleted.

The encryption key deletion is a security precaution against many inside attacks as well as against compromised backup media.

The above procedure is actually an important selling point. If you don't do that internaly for the 3rd party data then you should start doing this as soon as practical to lower your own liability.

Hmm yeah I have come across something similar now that I think about it..  the service provider only has the keys as long as the customer has the service.   If the customer was subscribing to their own backup, only they would have the keys, and not the provider..

There may still be a way around it, but by design, money wouldn't necessarily fix this problem.  Guess i'm wrong.

[MysteryMiner]

I read this saga and again I see proof of my theory that hosting servers in some Crackspace is insecure. I host my servers from my home. They are under my sole control, both the software and hardware. No need to trust someone else to be both competent in securing the server and not to snoop at my data or steal coins residing on server.

40k of BTC, if I was a techie in such datacenter I will steal the coins on first opportunity and quit working as a monkey for rest of my life. 40K of BTC is worth about 25 years of salary in my country. I could kill anybody for quarter of that.

[REF]

I read this saga and again I see proof of my theory that hosting servers in some Crackspace is insecure. I host my servers from my home. They are under my sole control, both the software and hardware. No need to trust someone else to be both competent in securing the server and not to snoop at my data or steal coins residing on server.

40k of BTC, if I was a techie in such datacenter I will steal the coins on first opportunity and quit working as a monkey for rest of my life. 40K of BTC is worth about 25 years of salary in my country. I could kill anybody for quarter of that.

Good to see you would destroy another individuals life for your own gain. do you have no morals or soul?

Well until the recent hack and all the new information coming out I was on Bitcoinicas side and thought payouts were coming. I dont see that happening anymore. A lawsuit would be the best way to go now especially if you dont want to take a 30% cut in what you are owed.

Does anyone have a VERY detailed timeline of events of what happened with Bitcoinica? One that goes as far back to the launch on Bitcoinica would be best.

[justusranvier]

40k of BTC, if I was a techie in such datacenter I will steal the coins on first opportunity and quit working as a monkey for rest of my life. 40K of BTC is worth about 25 years of salary in my country. I could kill anybody for quarter of that.

I offer computer security services...

For their sake I hope your customers know not to hire you if the value of what they want you to protect is more than BTC10K.

[smoothie]

Can I ask you why you have such large amount of money in a third party site?

I was attracted by the high interest rates and the money I thought I could earn by depositing bitcoins with them.
I never made a single trade on Bitcoinica.
I simply held a balance and collected interest.

This right here sounds like what is going on with pirate's deal...

Roger, I have one honest question:

After all of the hacks and stolen bitcoin stories, why would you deposit so much bitcoins with anyone?

I truly feel sorry for you and I give you credit for your tremendous contribution to the bitcoin community, but bitcoin is supposed to give the people 100% control of their money as you have said yourself many times over (even in the recent porc fest video).

Vladimir is right when he said that the ONLY way to keep your bitcoins safe is to heavily enrcypt your wallet, put it in a locked room/safe/, or have a brain wallet or paper wallet kept in a very secure place.

Let this once again be a lesson to everyone (not as a troll) but sincerely a lesson to keep your bitcoins in your possession and do everything it takes to keep it from being stolen.

[makomk]

I don't know Rackspace's procedures nor your procedures; but I can offer you a clean explanation of not being able to restore the individual backups. I'm kinda familiar how other kinda-cloud provider does backups.

All the backups are encrypted with a transactional key. There are separate keys for each VM and for each incremental backup. The keys are stored only as long as the accounting record for the VM is open. Once the VM is deleted and summarized for the purpose of charging all transactional keys are deleted zeroized (I forgot to use the proper lingo).

The encryption key deletion zeroization is a security precaution against many inside attacks as well as against compromised backup media.

The above procedure is actually an important selling point. If you don't do that internaly for the 3rd party data then you should start doing this as soon as practical to lower your own liability.

http://en.wikipedia.org/wiki/Hardware-based_full_disk_encryption
http://www.hgst.com/internal-drives/self-encrypting-drives/

Rackspace also supposedly allowed automated password reset via e-mail and didn't have any way to lock an attacker out of their management console once they'd authenticated, if we believe what Bitcoinica. Using this kind of protection against VM or backup compromise would be the equivalent of having an elaborately-boobytrapped steel front door which randomly maimed people who opened it wrong and then, around the back, having another access route though a rotted garden gate and a doorway secured with a single-lever lock. It'd be completely nuts as an approach to security.

[2112]

Rackspace also supposedly allowed automated password reset via e-mail and didn't have any way to lock an attacker out of their management console once they'd authenticated, if we believe what Bitcoinica. Using this kind of protection against VM or backup compromise would be the equivalent of having an elaborately-boobytrapped steel front door which randomly maimed people who opened it wrong and then, around the back, having another access route though a rotted garden gate and a doorway secured with a single-lever lock. It'd be completely nuts as an approach to security.

From your post I gather that you have never worked, consulted or even repeatedly visited any large publicly-traded company. Neither you've served in armed forces of any country.

They are all completely nuts, exactly like that. Top-level security and armed guards at the front door, but an elephant door to the warehouse in the back secured with a piece of plastic.

http://en.wikipedia.org/wiki/Mathias_Rust
http://en.wikipedia.org/wiki/Buffalo_Soldiers_(film)

[Bitcoin Oz]

Sorry to hear about this Roger.

[Phinnaeus Gage]

This is just a hunch but maybe try posting this plea in he Russian forums.

Reading between the line, I like how you think. I'm not advocating anything here, just stating that I like how you think. Did I mention I wasn't advocating...

Don't say this lightheartedly, Roger, but I feel your pain. I have an idea. Let's switch position. I'll pay what's owed you and you pay to have now two toxic sites cleaned up due to the storage of antique painted barn wood plus the forthcoming daily fines. My troubles stem, albeit indirectly, from the same debacle (I must be in good in spirits to have opted for that word).

I'm not sure what I would do if I discover that Satoshi Nakamoto is still in his teens.

~Bruno~

Edit: I continued reading from the point where I left off, and want to be very clear on something. I'm not seeking any sympathy for what I'm currently going through. Just stating facts and showing an example of the butterfly effect.

[Phinnaeus Gage]

I'm truly sorry for Roger
bitcoiners and accounts like these should priority one.

what's going on behind the scenes Tihan,  Zhou,  Patrick, Donald, Amir, or anyone else with access to Bitcoinica money ??

I have an idea who the mysterious Wendon is, but need more time to connect the dots. This was a tough one! I'll leave you with a one word clue: Slice.

~Bruno~