Bitcoin Forum
August 25, 2019, 12:42:05 PM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Bitfloor service outage postmortem - 8/31/2012  (Read 2170 times)
shtylman
Sr. Member
****
Offline Offline

Activity: 243
Merit: 250



View Profile
August 31, 2012, 11:11:34 PM
Last edit: August 31, 2012, 11:55:23 PM by shtylman
 #1

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 31st 2012 for those interested.

Our hosting provider experienced a power outage in one of their facilities. The facility happened to be the one where we have some of our primary boxes and thus were were affected by the power outage. The website and other auxiliary services were offline as a result. As soon as the power was restored and our boxes brought back online I began restoring the various services (website, order gateways, matching engine, etc). Once those were restored, I restored the bitcoin services and wallet (this takes longer and is more manual due to the security in place for the wallet and private keys).

In the end it was nothing more than a power outage that caused the disruption. I can re-assure everyone that all financials (USD and BTC are in order) and as of now all exchange operations are back online.

Going forward I will be looking at using multiple data center locations and/or a separate status page to indicate the current situation and not keep you (our users) in the dark about what is going on. Our goal is to be as fault tolerant with our systems as possible and I will be evaluating how to make that happen as we continue to grow.
1566736925
Hero Member
*
Offline Offline

Posts: 1566736925

View Profile Personal Message (Offline)

Ignore
1566736925
Reply with quote  #2

1566736925
Report to moderator
1566736925
Hero Member
*
Offline Offline

Posts: 1566736925

View Profile Personal Message (Offline)

Ignore
1566736925
Reply with quote  #2

1566736925
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
bittenbob
Hero Member
*****
Offline Offline

Activity: 560
Merit: 500


View Profile
August 31, 2012, 11:33:57 PM
Last edit: September 01, 2012, 01:01:08 AM by bittenbob
 #2

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

(Edit)
It doesn't look as bad with the new date. Glad to hear things are still running well.
shtylman
Sr. Member
****
Offline Offline

Activity: 243
Merit: 250



View Profile
August 31, 2012, 11:38:55 PM
 #3

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case Smiley

http://dictionary.reference.com/browse/postmortem
Stephen Gornick
Legendary
*
Offline Offline

Activity: 2506
Merit: 1001


View Profile
August 31, 2012, 11:51:25 PM
 #4

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested

It that explaining today's outage, or a prior one?

Unichange.me

            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █


shtylman
Sr. Member
****
Offline Offline

Activity: 243
Merit: 250



View Profile
August 31, 2012, 11:52:58 PM
 #5

I wanted to do a quick writeup on the Bitfloor outage that occurred on August 8th 2012 for those interested

It that explaining today's outage, or a prior one?

Sorry. Updated the original post. The date in the title is correct but I mistyped 8 instead of 31 (was thinking about the month not the day). Obviously it has been a long day Smiley It is indeed about today's outage.
Bitcoin Oz
Hero Member
*****
Offline Offline

Activity: 686
Merit: 500


Wat


View Profile WWW
August 31, 2012, 11:54:10 PM
 #6

Postmortem is probably not the best title for this thread. It gives the impression that the site is dead (offline and not coming back) but after reading your post that is clearly not what happened. Unless of course your point is to draw attention to the thread from shock.

Postmortem can also mean the breakdown or description of what happened after a particular event. I am not going for shock factor, it is a legitimate use of the word in this case Smiley

http://dictionary.reference.com/browse/postmortem

I thought your site had died too  Sad

Maybe "Debriefing" is a better term.

Ocean6
Member
**
Offline Offline

Activity: 103
Merit: 10



View Profile
September 01, 2012, 12:42:59 AM
 #7

I like to be "de-briefed"!  Grin

Life is What you Make it!
BCB
CTG
VIP
Legendary
*
Offline Offline

Activity: 1064
Merit: 1000


BCJ


View Profile
September 01, 2012, 01:08:08 AM
 #8

For the idiots who can't read I'm glad your vocabulary has been expanded today.  For Stephen the "first responder" on all things bitcoin thanks for fixing the typo.

For Roman, thanks for responding so quickly to the outage.  From the time of the first post in another thread announcing the outage and your post announcing the restoration (that means turning it back on guys, not a play about  England)  it was all of 1 hour and about 6 minutes. 

And thanks for voluntarily posting this postmortem.  I'll say it again -  transparency and responsiveness to clients/customers is sorely lacking in too many bitcoin companies so it is refreshing to continue to see great support and service from bitfloor.com.

Stephen Gornick
Legendary
*
Offline Offline

Activity: 2506
Merit: 1001


View Profile
September 04, 2012, 04:32:33 PM
 #9

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

Unichange.me

            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █


Stephen Gornick
Legendary
*
Offline Offline

Activity: 2506
Merit: 1001


View Profile
September 04, 2012, 06:49:53 PM
 #10

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

Apparently, that was not the truth.  Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen:

No coins were kept in cold storage, and all coins from the hot wallet are now spent.  

As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.

 - http://bitcointalk.org/index.php?topic=105818.0
 - http://bitcointalk.org/index.php?topic=105819.0



Unichange.me

            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █
            █


Shagnasty
Member
**
Offline Offline

Activity: 61
Merit: 10

ICOForums.net - A Cryptocurrency & ICO Forum


View Profile
September 05, 2012, 01:14:26 AM
 #11

A follow on to this, there was an e-mail from BitFloor reportedly sent out to some accountholders, apparently those who had created API keys. Here's a link to that thread:

 - http://bitcointalk.org/index.php?topic=105079.msg1159003#msg1159003

Apparently, that was not the truth.  Now the operator of BitFloor claims the wallet has been compromised and all BTC funds stolen:

No coins were kept in cold storage, and all coins from the hot wallet are now spent.  

As a last resort, I will be forced to fully shut BitFloor down and initiate account repayment using current available funds. I still have all of the logs for accounts, trades, transfers. I know exactly how much each user currently has in their account for both USD and BTC. No records were lost in this attack.

 - http://bitcointalk.org/index.php?topic=105818.0
 - http://bitcointalk.org/index.php?topic=105819.0



Why would you not have an offline machine just for storing reserves?

Activity Contest  \_____◢  ICO Forums  ◣____/  Promotion Contest
New Forum for Cryptocurrency Discussion
ICO Review Contest  / ̅ ̅ ̅ ̅ ̅ ̅ ◥  ANN Thread  ◤ ̅ ̅ ̅ ̅ ̅ ̅ \  Signature Campaign
Vod
Legendary
*
Offline Offline

Activity: 2842
Merit: 2284


Licking my boob since 1970


View Profile WWW
September 06, 2012, 07:09:51 PM
 #12

How about you do the job you are legally required to do as CEO, and call the police? 

Bitcoinica all over again, and you guys will accept the loss all over again....

I'm into creating universes, smiting people, writing holy books and listening to Prayer Messages (PMs).
BitcoinTalk Public Information Project (BPIP)  - BPIP Reports
"Masturbation makes you feel good but doesn't do anything for the person you're thinking of.  Just like prayer."
DannyHamilton
Legendary
*
Offline Offline

Activity: 2198
Merit: 1429



View Profile
September 07, 2012, 04:50:15 AM
 #13

. . .How about you do the job you are legally required to do as CEO, and call the police?. . .
What makes you think they didn't contact the police/FBI?

. . .I have filed reports with the FBI and the IC3 regarding the theft. . .

Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!