Bitfloor service outage postmortem - 8/31/2012

[shtylman]

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 31st 2012 for those interested.

Our hosting provider experienced a power outage in one of their facilities. The facility happened to be the one where we have some of our primary boxes and thus were were affected by the power outage. The website and other auxiliary services were offline as a result. As soon as the power was restored and our boxes brought back online I began restoring the various services (website, order gateways, matching engine, etc). Once those were restored, I restored the bitcoin services and wallet (this takes longer and is more manual due to the security in place for the wallet and private keys).

In the end it was nothing more than a power outage that caused the disruption. I can re-assure everyone that all financials (USD and BTC are in order) and as of now all exchange operations are back online.

Going forward I will be looking at using multiple data center locations and/or a separate status page to indicate the current situation and not keep you (our users) in the dark about what is going on. Our goal is to be as fault tolerant with our systems as possible and I will be evaluating how to make that happen as we continue to grow.

[bittenbob]

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

(Edit)
It doesn't look as bad with the new date. Glad to hear things are still running well.

[shtylman]

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case

http://dictionary.reference.com/browse/postmortem

[Stephen Gornick]

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested

It that explaining today's outage, or a prior one?

[shtylman]

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested

It that explaining today's outage, or a prior one?

Sorry. Updated the original post. The date in the title is correct but I mistyped 8 instead of 31 (was thinking about the month not the day). Obviously it has been a long day It is indeed about today's outage.

[Bitcoin Oz]

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case

http://dictionary.reference.com/browse/postmortem

I thought your site had died too 

Maybe "Debriefing" is a better term.

[Ocean6]

I like to be "de-briefed"! 

[BCB]

For the idiots who can't read I'm glad your vocabulary has been expanded today.  For Stephen the "first responder" on all things bitcoin thanks for fixing the typo.

For Roman, thanks for responding so quickly to the outage.  From the time of the first post in another thread announcing the outage and your post announcing the restoration (that means turning it back on guys, not a play about  England)  it was all of 1 hour and about 6 minutes. 

And thanks for voluntarily posting this postmortem.  I'll say it again -  transparency and responsiveness to clients/customers is sorely lacking in too many bitcoin companies so it is refreshing to continue to see great support and service from bitfloor.com.

[Stephen Gornick]

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

[Stephen Gornick]

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

Apparently, that was not the truth.  Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen:

No coins were kept in cold storage, and all coins from the hot wallet are now spent.  

As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.

 - http://bitcointalk.org/index.php?topic=105818.0
 - http://bitcointalk.org/index.php?topic=105819.0

[Shagnasty]

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

Apparently, that was not the truth.  Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen:

No coins were kept in cold storage, and all coins from the hot wallet are now spent.  

As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.

 - http://bitcointalk.org/index.php?topic=105818.0
 - http://bitcointalk.org/index.php?topic=105819.0

Why would you not have an offline machine just for storing reserves?

[Vod]

How about you do the job you are legally required to do as CEO, and call the police? 

Bitcoinica all over again, and you guys will accept the loss all over again....

[DannyHamilton]

. . .How about you do the job you are legally required to do as CEO, and call the police?. . .

What makes you think they didn't contact the police/FBI?

. . .I have filed reports with the FBI and the IC3 regarding the theft. . .