[ANN Mt.Gox] It’s been an epic few days: What happened?

[Mt.Gox_Alex]

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!

However, there are many who will try to take advantage of the system. The past few days were a reminder of this sad truth.

Mt.Gox has been suffering from its worst trading lag ever, 502 errors, and at one point some users were not able to log in their account. The culprit is a major DDoS attack against Mt.Gox.

Since yesterday, we are continuing to experience a DDoS attack like we have never seen. While we are being protected by companies like Prolexic, the sheer volume of this DDoS left us scrambling to fine-tune the system every few hours to make sure that things don’t go beyond a few 502 error pages and trading lag.

Why has Mt.Gox become the target of a DDoS attack?
It is not yet clear who is behind this DDoS and we may never know, but these actions seem to have two major purposes:

1.   Destabilize Bitcoin in general.
It is not a secret Mt.Gox is the largest Bitcoin exchange with more than 80% of all USD trades and more than 70% of all currencies. Mt.Gox is an easy target for anyone that wants to hurt Bitcoin in general.

2.   Abuse the system for profit.
Attackers wait until the price of Bitcoins reaches a certain value, sell, destabilize the exchange, wait for everybody to panic-sell their Bitcoins, wait for the price to drop to a certain amount, then stop the attack and start buying as much as they can. Repeat this two or three times like we saw over the past few days and they profit.

What can be done?
Believe it or not, there is pretty much nothing that can be done. Large companies are frequently victims of these kinds of attacks. Even though we are using one of the best companies to help us fight against these DDoS attacks, we are still being affected.

There are a few things that we can implement to help fight the attacks, such as disconnecting the trade engine backend from the Internet. By separating the data center from the Mt.Gox website, we will continue to be able to trade.

What can you do?
Like our favorite author here at Tibanne says… Don’t Panic!

“Panic-selling is a wide-scale selling of an investment which causes a sharp decline in prices. Specifically, an investor wants to get out of an investment with little regard of the price obtained. The selling activity is problematic because the investor is selling in reaction to emotion and fear, rather than evaluating the fundamentals.” (Source: Wikipedia)

I understand that many of you have a lot at stake here, but remember that Bitcoin, despite being designed to have its value increase over time, will always be the victim of people trying to abuse the system, or even the value of Bitcoin decreasing occasionally. These are not new phenomena and have been present since the beginning of time when humans first started trading.

Trade Engine Lags
Lag affects everyone, not only us, but also major, world-renowned exchanges like the NASDAQ and NYSE. We can fix lag, but we cannot eradicate lag. Only small exchanges with low volume and liquidity are immune to lag.

Does this mean that we are giving up fighting lag? Hell, no. We are working on it by creating a new trade engine that will solve many problems, but it’s not a magic bullet. We can always try to scale our servers, but we cannot predict what happens from external sources: DDoS, panic selling, immediate increase of buyers, etc. Lag will always be there, but our mission is to make lag as small as possible.

Account Verification
As if a major DDoS attack was not enough, we at Mt.Gox are victim of our own success!

Last year, Mt.Gox saw an average of 9,000 to 10,000 new accounts created every month. This number doubled in January, tripled in February, and sextupled in March. In this month alone, March, over 57,000 new accounts were created!

Our support and account verification team went from four people in January 2012 to twenty-two people working every day of the week. We are now hiring even more people to solve this problem by finalizing some deals with external companies.

Remember that even if you are waiting for your account to be verified, you can still deposit or withdraw funds via our Japanese account and make your trades! (Only accounts that we pro-actively required to be verified are limited to deposits and trade only.)

Finally
We have seen a significant amount of comments on the web (various forums, Reddit, etc.) that portray Mt.Gox as a company held by “idiots” and other rather rude words, complaining about inability to deal with lag and other system issues, without understanding the magnitude of work and attacks we are facing every day.

I understand the frustration many of you feel. We hate this situation as well. Since we took over Mt.Gox, we have been through Hell and back and we are still here. We are still the largest exchange with over 420,000 trades per month and  USD $121 million monthly trade volume. We have worked our way through all the requirements needed to run our exchange legally.

Now, there are some things we can improve, but so far we are doing an incredible job that no other exchange has been able to do so far. While I understand a certain amount of frustration, realize what we have accomplished. I appreciate all the work you are doing everyday to push things forward and to help secure the future of Bitcoin

And to all of you who are supporting us on a daily basis, thank you! We could not have done any of this without your help!

[1714600301]

1714600301

[1714600301]

1714600301

[anzy]

Thanks for the update

[justusranvier]

Finally
We have seen a significant amount of comments on the web (various forums, Reddit, etc.) that portray Mt.Gox as a company held by “idiots” and other rather rude words, complaining about inability to deal with lag and other system issues, without understanding the magnitude of work and attacks we are facing every day.

This thread is a great example of the communication that goes a long way towards defusing those kinds of reactions.

When your customers don't know what's going on they tend to assume the worst.

[TheButterZone]

Do you have enough reserves to suspend trading and meet payroll/vendor obligations until the new trade engine is ready for prime time (and maybe the verification queue is beaten to within an inch of its life, as well)? Not sure how many strikes against Gox there are going to have to be, before most smart people say "you're out!" and look somewhere else.

Personally, Namecheap is being denied 4 years of domain transfer/renewals of mine because of this insane high-low. I was just about to execute my bitcoin deposit to them before the latest of seemingly countless DDOSs. Bit-Pay loses out on its fee as well.

[casascius]

Thanks for the excellent feedback.

There is one thing you can do that's relatively easy.  You guys should consider doing a UDP blast of ticker information to perhaps a dozen volunteer sites who can use this information to relay it into an independently-provided web socket feed so that it doesn't have to go through you.  By UDP, just in case this isn't clear, I mean User Datagram Protocol, the sessionless protocol supported by TCP/IP, the kind of one-way outbound packets you can send from your facilities that shouldn't be affected by an inbound DDoS flood.

[SomeWhere]

I applaud you for this update and must admit that I was one of the people to go a little overboard.

This kind of communication goes a long way and is something you should do a lot more.

I however maintain that you had a *lot* of time over the past months to get a more efficient trading engine up and - given that it should be *the* top priority - the progress on that front is simply not satisfactory.

I, for one, will be moving to another exchange until Mt.Gox meets the standards that I expect from the leader of the industry.

P.S. Just to have some basis to my claims: I have a strong background in computer engineering, especially when it comes to big & performance intensive systems. I can assure you, I know the challenges that come with hosting infrastructure, a good trading engine, massively parallel architectures. Still, if prioritized the right way, results should have been there a long time ago.

[repentance]

Alex, you mentioned the verification issue.

It's really unclear to me what is going to happen with this when US/Canadian users get transferred to CoinLab.  Will the accounts which are currently awaiting verification still be verified by MtGox or will US/Canadian customers be verified by CoinLab following the transition (ie, will those US/Canadian customers currently awaiting verification by MtGox need to start the verification process again with CoinLab)?  

The lack of recent information about the transition is worrying as customers were initially told that they needed to agree to CoinLab's ToS if they wanted to use MtGox following the transition but the information on the CoinLab site says that the funds of US/Canadian customers will automatically be transferred to the US.  

This implies that people will need to withdraw their funds from MtGox before the transition if they don't want those funds transferred to the US bank (which may present difficulties for unverified customers given the backlog on verification) or they'll have to register as a CoinLab user whether they want to or not in order to withdraw their funds.

It was previously stated that MtGox user data would not be transferred to CoinLab without user agreement (ie, MtGox users accepting CoinLab's ToS), but if the funds are being transferred automatically then how will users be able to access them without using CoinLab?

I know you have a lot going on, but the transition to CoinLab has to the potential to be disastrous if you don't keep your users fully informed and - quite frankly - you don't need any more things happening right now which undermine people's confidence in you.

[Bitobsessed]

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!

Thought the top was $147?

[Mt.Gox_Alex]

Dear Mt.Gox users and Bitcoiners,

It’s been an epic few days on Bitcoin, with prices going up as high as $142 per BTC. We all hope that this is just the beginning!

Thought the top was $147?

You are right (last 24hrs), I was stuck on the past 12hrs data

[moni3z]

Prolexic sucks, they are resellers. Go with Black Lotus and make them agree to guaranteed uptime protection they'll do it. You could also clone a backup elastic cloud image on Amazon that ssh's to your secure db and switch to it whenever you get hammered beyond 100Gbps

[Mt.Gox_Alex]

Alex, you mentioned the verification issue.

It's really unclear to me what is going to happen with this when US/Canadian users get transferred to CoinLab.  Will the accounts which are currently awaiting verification still be verified by MtGox or will US/Canadian customers be verified by CoinLab following the transition (ie, will those US/Canadian customers currently awaiting verification by MtGox need to start the verification process again with CoinLab)?  

The lack of recent information about the transition is worrying as customers were initially told that they needed to agree to CoinLab's ToS if they wanted to use MtGox following the transition but the information on the CoinLab site says that the funds of US/Canadian customers will automatically be transferred to the US.  

This implies that people will need to withdraw their funds from MtGox before the transition if they don't want those funds transferred to the US bank (which may present difficulties for unverified customers given the backlog on verification) or they'll have to register as a CoinLab user whether they want to or not in order to withdraw their funds.

It was previously stated that MtGox user data would not be transferred to CoinLab without user agreement (ie, MtGox users accepting CoinLab's ToS), but if the funds are being transferred automatically then how will users be able to access them without using CoinLab?

I know you have a lot going on, but the transition to CoinLab has to the potential to be disastrous if you don't keep your users fully informed and - quite frankly - you don't need any more things happening right now which undermine people's confidence in you.

We, Coinlab & Mt.Gox, will announce something on this matter soon.

[Mt.Gox_Alex]

Prolexic sucks, they are resellers. Go with Black Lotus and make them agree to guaranteed uptime protection they'll do it. You could also clone a backup elastic cloud image on Amazon that ssh's to your secure db and switch to it whenever you get hammered beyond 10Gbps

We were using Black Lotus and runaway from them... And 10Gbps is pretty much nothing for us, we have this on weekly basis and EC2 doesn't have enough CPU/memory to handle our db on a single instance

[repentance]

We, Coinlab & Mt.Gox, will announce something on this matter soon.

I really hope so because to date both services have fallen well short of the mark in keeping users informed.

[Mt.Gox_Alex]

We, Coinlab & Mt.Gox, will announce something on this matter soon.

I really hope so because to date both services have fallen well short of the mark in keeping users informed.

I agree with you, but let's say that the FinCEN announcement delayed a few things.

[Wekkel]

Thanks for updating the community 

[MPOE-PR]

Thanks for the excellent feedback.

If this is excellent I bet nobody eats what you cook.

If you were to make the list of "events which might have killed Bitcoin", MtGox's hack is indisputably at #1, above the Pirate heist, above the recent unexpected fork, above everything else. This is MtGox's legacy: they're a historical threat to Bitcoin' continued existence and a permanent nuisance in the day to day life of it. To further illustrate this point:

Mar 04 19:15:17 blu3gr1ffon ;;goxlag
Mar 04 19:15:22 gribble 106.133771 seconds

Mar 06 04:46:15 Ukto ;;goxlag
Mar 06 04:46:15 gribble 78.10124 seconds

Mar 06 05:17:03 Chaang-Noi ;;goxlag
Mar 06 05:17:05 gribble 456.476647 seconds

Mar 06 11:41:29 dub ;;goxlag
Mar 06 11:41:46 gribble 245.37453 seconds

Mar 07 01:58:34 kakobrekla ;;goxlag
Mar 07 01:58:37 gribble 471.638897 seconds

Mar 11 09:35:00 jurov ;;goxlag
Mar 11 09:35:01 gribble 2025.557307 seconds
Mar 11 09:35:10 jurov what is the record?

Mar 12 03:53:17 dub ;;goxlag
Mar 12 03:53:18 gribble 38.016469 seconds

Mar 12 04:12:36 smickles ;;goxlag
Mar 12 04:12:36 gribble 157.044926 seconds

Mar 12 04:23:10 gesell ;;goxlag
Mar 12 04:23:11 gribble 199.519792 seconds

Let's skip to recent times:

Mar 29 00:34:07 Bitesaak ;;goxlag
Mar 29 00:34:07 gribble 288.72481 seconds

Mar 29 00:40:41 thestringpuller ;;goxlag
Mar 29 00:40:41 gribble 167.450794 seconds

It's so bad we have a special command to query it! And it's so funny people have taken to embellishing it:

Apr 03 09:24:23 ThickAsThieves ;;goxlag
Apr 03 09:24:24 gribble MtGox lag is 219.524796 seconds. During this time, light travels 0.439925233262 AU. You could have sent a bitcoin from the Sun to Mercury (0.39 AU).

Apr 03 11:50:16 dub ;;goxlag
Apr 03 11:50:16 gribble MtGox lag is 468.536678 seconds. During this time, light travels 0.938942256714 AU. You could have sent a bitcoin from the Sun to Earth (1 AU).

Apr 03 17:52:42 TomServo ;;goxlag
Apr 03 17:52:42 gribble MtGox lag is 6048.679827 seconds. During this time, light travels 12.1214866489 AU. You could have sent a bitcoin from the Sun to Saturn (9.54 AU).

Apr 03 18:03:07 thestringpuller ;;goxlag
Apr 03 18:03:07 gribble MtGox lag is 5455.751923 seconds. During this time, light travels 10.933265768 AU. You could have sent a bitcoin from the Sun to Saturn (9.54 AU).

The notion that you're running an exchange with multisecond lag is ridiculous on its face. I'm not even sure why this has to be spelled out, it's beyond ridiculous. It's like Monty Python's cheese shop, it's like Monty Python's "self defense classes", it's like a comedy routine. This is what we use you for, MtGox, comedic relief. You're not an exchange, okay? You're Bitcoin's very own Comedy Central.

So now, armed with this basic understanding of what's what and where we're standing, let us dissect MtGox's most recent load of bullcrap (continued).

[Zooey]

Mt.Gox has been suffering from its worst trading lag ever, 502 errors, and at one point some users were not able to log in their account.

Have to admit, I too am a sucker for folks who fess up to their failings, so props to you for doing that head-on without any crap; it affords you a level of credibility that ought to be a sober example for others who lack it in lumps, like the bullshitting BFL brigade.

[Isokivi]

One thing Mt.Gox could do is release what information they have on the attack(s). We all know it's a botnet of infected computers, but I would not count out the bitcoin community out from doing detective work and more.

[DougTanner]

I didn't see anything in there about adjusting the trade fee schedule.

It used to be < $160,000 to get to 0.3% (and probably a lot lower), now it's upwards of $1,370,000, a factor of ten. When will you be adjusting it to make it in line with current prices?

[casascius]

Also MtGox could take a position on my UDP streams idea, which could be any of the following without commitment:

a) Great idea, we haven't thought of it, and you're right, it would totally get information out immune to DDoS, we'll consider it but like anything else will take time
b) Great idea, but we don't agree it would work as well as you think it will, or for (specific technical reason) won't work on our platform
c) We haven't got a clue as to what this means
d) I don't have a clue what this means because I'm not a developer or tech guy myself, but I have relayed your suggestion to someone more technical, and he says (response).  (Hopefully this suggestion is more valuable than to merely forward it blindly like the latest facebook meme, since MtGox's reputation is suffering and this will actually solve the claimed issue at hand)

Just to be clear, using UDP to broadcast ticker data would be, for all intents and purposes, IMMUNE from DDoS attacks, because such a stream consists solely of outbound traffic which is not influenced by inbound traffic.  Unlike a normal stream, there is no inbound overhead for packets to acknowledge or to keep the connection in sync, packets which can be drowned out in a DDoS attack.  UDP is much more like a point-to-point radio broadcast: the signal gets sent from point A to B even if nobody's listening