[ANN] Mt.Gox Outage Report - 2012/9/6

[Mt.Gox_Alex]

Dear Mt.Gox users and the Bitcoin community:

Yesterday, Mt.Gox suffered two different outages, preventing our customers to access the Mt.Gox website and making it impossible for people to trade via Mt.Gox user interface.

While these outages have been handle swiftly, we would like to reassure our customers and the community that we did not suffer any breach and that no data or funds have been stolen on our server. Also, please find below the details on each outage and what has been done to resolve this matter.

1. Prolexic Network Trouble.

While we have received all of the details on this matter, Prolexic suffered yesterday night (JST) due to some technical difficulties on its network, making it impossible for some people to access Mt.Gox website.



We hope to provide you with more detailed information as soon as they are made available to us.


2. SYN flood DOS Attack.

This morning around 6am to 7:15am (JST), Mt.Gox suffered another unrelated attack, preventing people to access and use Mt.Gox services. 



The person or group that targeted Mt.Gox used a SYN flood denial-of-service attack by sending a huge succession of SYN request on our servers in an attempt to consume all of our server resources. 



While our servers are equipped with adequate countermeasures for such attack, thanks to SYN cookies, our ISP networking infrastructure, including firewalls and routers, were not able to handle the sheer amount of packet (50,000 packets/sec) sent to our infrastructure, resulting in our system unresponsiveness to legitimate users.

Once again, we would like to reassure everyone that our server did not suffer any breach and that all our customers' data and funds are safe and accounted for.

On Behalf of the entire Mt.Gox team, please accept our apologizes for any inconvenience that these two outages may have caused you.

[1715308333]

1715308333

[Mt.Gox_Alex]

1. Prolexic Network Trouble.

While we have received all of the details on this matter, Prolexic suffered yesterday night (JST) due to some technical difficulties on its network, making it impossible for some people to access Mt.Gox website.



We hope to provide you with more detailed information as soon as they are made available to us.


Update on what Happened with Prolexic :

"The following report addresses the service degradation event during the time envelope exclusive from 14:20 UTC to 16:06 UTC on September 5th, 2012.

During the event envelope one of our upstream providers (TATA) experienced a faulty linecard failure in their Asburn router that peers directly with our Border router in the DCA Data Center. This failure caused all traffic that was being advertised through the Tata link to be dropped. The outage affected services both our Proxy and GRE solution.

Upon confirming the issue with TATA, the Prolexic SOC immediately escalated this to Network Engineering, and at 16:06 UTC the announcement of all routes through the TATA link was withdrawn. The removal of the TATA announcement resulted in all services returning to normal.

The hardware failure that was addressed by the TATA support team required them to issue remote hands to reset the card. After confirmation that the links were stable, Proexic Network Engineering restored all routes and announcements back to its default configuration. Traffic to Hong Kong, San Jose, and London Data Centers were unaffected during the DCA outage."