|
shtylman (OP)
|
 |
August 31, 2012, 11:11:34 PM
Last edit: August 31, 2012, 11:55:23 PM by shtylman |
|
I wanted to do a quick writeup on the Bitfloor outage that occurred on August 31st 2012 for those interested.
Our hosting provider experienced a power outage in one of their facilities. The facility happened to be the one where we have some of our primary boxes and thus were were affected by the power outage. The website and other auxiliary services were offline as a result. As soon as the power was restored and our boxes brought back online I began restoring the various services (website, order gateways, matching engine, etc). Once those were restored, I restored the bitcoin services and wallet (this takes longer and is more manual due to the security in place for the wallet and private keys).
In the end it was nothing more than a power outage that caused the disruption. I can re-assure everyone that all financials (USD and BTC are in order) and as of now all exchange operations are back online.
Going forward I will be looking at using multiple data center locations and/or a separate status page to indicate the current situation and not keep you (our users) in the dark about what is going on. Our goal is to be as fault tolerant with our systems as possible and I will be evaluating how to make that happen as we continue to grow.
|
|
|
|
|
|
|
|
|
|
|
|
The grue lurks in the darkest places of the earth. Its favorite diet is adventurers, but its insatiable appetite is tempered by its fear of light. No grue has ever been seen by the light of day, and few have survived its fearsome jaws to tell the tale.
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
|
|
|
|
|
bittenbob
|
|
August 31, 2012, 11:33:57 PM
Last edit: September 01, 2012, 01:01:08 AM by bittenbob |
|
Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.
(Edit)
It doesn't look as bad with the new date. Glad to hear things are still running well.
|
|
|
|
|
|
shtylman (OP)
|
|
August 31, 2012, 11:38:55 PM |
|
Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.
Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case  http://dictionary.reference.com/browse/postmortem |
|
|
|
|
Stephen Gornick
Legendary
 Offline
Activity: 2506
Merit: 1010
|
|
August 31, 2012, 11:51:25 PM |
|
I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested
It that explaining today's outage, or a prior one? |
|
|
|
|
shtylman (OP)
|
|
August 31, 2012, 11:52:58 PM |
|
I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested
It that explaining today's outage, or a prior one? Sorry. Updated the original post. The date in the title is correct but I mistyped 8 instead of 31 (was thinking about the month not the day). Obviously it has been a long day It is indeed about today's outage. |
|
|
|
|
|
Bitcoin Oz
|
|
August 31, 2012, 11:54:10 PM |
|
Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.
Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case http://dictionary.reference.com/browse/postmortemI thought your site had died too  Maybe "Debriefing" is a better term. |
|
|
|
Ocean6
Member
Offline
Activity: 103
Merit: 10
|
|
September 01, 2012, 12:42:59 AM |
|
I like to be "de-briefed"!  |
Life is What you Make it!
|
|
|
BCB
CTG
VIP
Legendary
Offline
Activity: 1078
Merit: 1002
BCJ
|
|
September 01, 2012, 01:08:08 AM |
|
For the idiots who can't read I'm glad your vocabulary has been expanded today. For Stephen the "first responder" on all things bitcoin thanks for fixing the typo.
For Roman, thanks for responding so quickly to the outage. From the time of the first post in another thread announcing the outage and your post announcing the restoration (that means turning it back on guys, not a play about England) it was all of 1 hour and about 6 minutes.
And thanks for voluntarily posting this postmortem. I'll say it again - transparency and responsiveness to clients/customers is sorely lacking in too many bitcoin companies so it is refreshing to continue to see great support and service from bitfloor.com.
|
|
|
|
|
|
|
Stephen Gornick
Legendary
Offline
Activity: 2506
Merit: 1010
|
|
September 04, 2012, 06:49:53 PM |
|
Apparently, that was not the truth. Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen: No coins were kept in cold storage, and all coins from the hot wallet are now spent. As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.
- http://bitcointalk.org/index.php?topic=105818.0 - http://bitcointalk.org/index.php?topic=105819.0 |
|
|
|
Shagnasty
Member
Offline
Activity: 148
Merit: 10
|
|
September 05, 2012, 01:14:26 AM |
|
Apparently, that was not the truth. Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen: No coins were kept in cold storage, and all coins from the hot wallet are now spent. As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.
- http://bitcointalk.org/index.php?topic=105818.0 - http://bitcointalk.org/index.php?topic=105819.0Why would you not have an offline machine just for storing reserves? |
|
|
|
Vod
Legendary
Offline
Activity: 3696
Merit: 3070
Licking my boob since 1970
|
|
September 06, 2012, 07:09:51 PM |
|
How about you do the job you are legally required to do as CEO, and call the police?
Bitcoinica all over again, and you guys will accept the loss all over again....
|
https://nastyscam.com - landing page up https://vod.fan - advanced image hosting - coming soon! OGNasty has early onset dementia; keep this in mind when discussing his past actions.
|
|
|
DannyHamilton
Legendary
Offline
Activity: 3388
Merit: 4616
|
|
September 07, 2012, 04:50:15 AM |
|
. . .How about you do the job you are legally required to do as CEO, and call the police?. . . What makes you think they didn't contact the police/FBI? . . .I have filed reports with the FBI and the IC3 regarding the theft. . . |
|
|
|
|
|
