|
Title: Bitfloor service outage postmortem - 8/31/2012 Post by: shtylman on August 31, 2012, 11:11:34 PM I wanted to do a quick writeup on the Bitfloor outage that occurred on August 31st 2012 for those interested.
Our hosting provider experienced a power outage in one of their facilities. The facility happened to be the one where we have some of our primary boxes and thus were were affected by the power outage. The website and other auxiliary services were offline as a result. As soon as the power was restored and our boxes brought back online I began restoring the various services (website, order gateways, matching engine, etc). Once those were restored, I restored the bitcoin services and wallet (this takes longer and is more manual due to the security in place for the wallet and private keys). In the end it was nothing more than a power outage that caused the disruption. I can re-assure everyone that all financials (USD and BTC are in order) and as of now all exchange operations are back online. Going forward I will be looking at using multiple data center locations and/or a separate status page to indicate the current situation and not keep you (our users) in the dark about what is going on. Our goal is to be as fault tolerant with our systems as possible and I will be evaluating how to make that happen as we continue to grow. Title: Re: Bitfloor Postmortem - 8/31/2012 Post by: bittenbob on August 31, 2012, 11:33:57 PM Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.
(Edit) It doesn't look as bad with the new date. Glad to hear things are still running well. Title: Re: Bitfloor Postmortem - 8/31/2012 Post by: shtylman on August 31, 2012, 11:38:55 PM Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock. Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case :) http://dictionary.reference.com/browse/postmortem Title: Re: Bitfloor Postmortem - 8/31/2012 Post by: Stephen Gornick on August 31, 2012, 11:51:25 PM I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested It that explaining today's outage, or a prior one? Title: Re: Bitfloor Postmortem - 8/31/2012 Post by: shtylman on August 31, 2012, 11:52:58 PM I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested It that explaining today's outage, or a prior one? Sorry. Updated the original post. The date in the title is correct but I mistyped 8 instead of 31 (was thinking about the month not the day). Obviously it has been a long day :) It is indeed about today's outage. Title: Re: Bitfloor Postmortem - 8/31/2012 Post by: Bitcoin Oz on August 31, 2012, 11:54:10 PM Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock. Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case :) http://dictionary.reference.com/browse/postmortem I thought your site had died too :( Maybe "Debriefing" is a better term. Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: Ocean6 on September 01, 2012, 12:42:59 AM I like to be "de-briefed"! ;D
Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: BCB on September 01, 2012, 01:08:08 AM For the idiots who can't read I'm glad your vocabulary has been expanded today. For Stephen the "first responder" on all things bitcoin thanks for fixing the typo.
For Roman, thanks for responding so quickly to the outage. From the time of the first post in another thread announcing the outage and your post announcing the restoration (that means turning it back on guys, not a play about England) it was all of 1 hour and about 6 minutes. And thanks for voluntarily posting this postmortem. I'll say it again - transparency and responsiveness to clients/customers is sorely lacking in too many bitcoin companies so it is refreshing to continue to see great support and service from bitfloor.com. Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: Stephen Gornick on September 04, 2012, 04:32:33 PM A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:
- http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003 Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: Stephen Gornick on September 04, 2012, 06:49:53 PM A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread: - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003 Apparently, that was not the truth. Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen: No coins were kept in cold storage, and all coins from the hot wallet are now spent. As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack. - http://bitcointalk.org/index.php?topic=105818.0 - http://bitcointalk.org/index.php?topic=105819.0 Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: Shagnasty on September 05, 2012, 01:14:26 AM A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread: - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003 Apparently, that was not the truth. Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen: No coins were kept in cold storage, and all coins from the hot wallet are now spent. As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack. - http://bitcointalk.org/index.php?topic=105818.0 - http://bitcointalk.org/index.php?topic=105819.0 Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: Vod on September 06, 2012, 07:09:51 PM How about you do the job you are legally required to do as CEO, and call the police?
Bitcoinica all over again, and you guys will accept the loss all over again.... Title: Re: Bitfloor service outage postmortem - 8/31/2012 Post by: DannyHamilton on September 07, 2012, 04:50:15 AM . . .How about you do the job you are legally required to do as CEO, and call the police?. . . What makes you think they didn't contact the police/FBI?. . .I have filed reports with the FBI and the IC3 regarding the theft. . . |