We are victims of our own success

[virtualmaster]

We are victims of our own success.
https://mtgox.com/press_release_20130411.html
More and more people will run to get a fragment of bitcoin.

[proudhon]

We are victims of our own success.
https://mtgox.com/press_release_20130411.html
More and more people will run to get a fragment of bitcoin.

Yep, no DDOS, just the trading engine lagged to almost the length of a feature film when one guy decided to sell a few thousand.  Success!

[virtualmaster]

I re-post the Mt Gox declaration because most of the time is still not available.
Hopefully I will get no copyright problems:

"      TOKYO - JAPAN - April 11, 2013

      Hi everyone, just a quick update on the situation and what happened last night.

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the number of trade made a huge impact on the overall system that started to lag. As expected in such situation people started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze the trade engine!

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys:
- The number of trades executed tripled in the last 24hrs.
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new servers to our system.

Thank you for your understanding and continuous support!
      
   

Regards
Mt.Gox Co. Ltd Team.

Mt.Gox Contact press@mtgox.com"

[aantonop]

No we are victims of incompetent mtgox who don't know how to run a trading engine, are making shit up as they go and keep blaming everything else except their own incompetence.

They have no plan because they have no clue...

RUN to another exchange.

[lolhonk]

MtGox.com ‏@MtGox Twitter:

Maintenance Over however we are now under a DDoS attack.

[proudhon]

MtGox.com ‏@MtGox Twitter:

Maintenance Over however we are now under a DDoS attack.

DDoS, success, you can never be sure with these things...

[tosku]

Is Mt.Gox considering the lag caused by a few trades per second a success?
Is it still a success if the trades are generated by bots buying and selling 0.0001 BTC back and forth?
Is it still a success if the trading bots are doing these trades with the intention of causing more lag, or is it suddenly a DOS then?

[proudhon]

Is Mt.Gox considering the lag caused by a few trades per second a success?
Is is still a success if the trades are generated by bots buying and selling 0.0001 BTC back and forth?
Is it still a success if the trading bots are doing these trades with the intention of causing more lag, or is it suddenly a DOS then?

YES

[MPOE-PR]

So basically this is not MtGox lying, this is MtGox doing the spin cycle. "We stepped in poo, we're great ecologists".

Basically this is a bunch of teenage dorks caught deer-in-headlights during an attempt to parlay everything they've learned about how to "seduce womenz" into the business of money exchanging. It's working about on par with what you'd expect.

[Operatr]

The only weakness of Bitcoin currently is just a lack of exchanges to diversify the network.

And who knows what kind of hardware they are running this on. A professional op would not have gone down like that. There should have been a failover ready to roll in such a scenario, whatever it is it is not fault tolerant in any way which isn't good.

I imagine I will begin one of my own someday, though setting up an exchange isn't all that documented yet.

[aantonop]

According to the IRC conversation with Magicaltux, the exchange engine and front end are running on the SAME SERVER

If that is true, it points to a level of amateurish hackery which is unbelievable.

A system like this should have multiple geographically redundant load balancers, multi-tier hierarchy of workers, web engines, trade engines, market data and charting engines. The DNS should be run by an independent DNS provider and they should have DDoS protection, regional failover, graceful degradation, etc etc.

It seems they have none of that, nor any clue how to do any of it. It's not even expensive nowadays, it just requires knowledge of scalable distributed systems.

[proudhon]

According to the IRC conversation with Magicaltux, the exchange engine and front end are running on the SAME SERVER

If that is true, it points to a level of amateurish hackery which is unbelievable.

A system like this should have multiple geographically redundant load balancers, multi-tier hierarchy of workers, web engines, trade engines, market data and charting engines. The DNS should be run by an independent DNS provider and they should have DDoS protection, regional failover, graceful degradation, etc etc.

It seems they have none of that, nor any clue how to do any of it. It's not even expensive nowadays, it just requires knowledge of scalable distributed systems.

They do have DDoS protection.  Cost them a pretty penny in 2011.  I'm assuming they still have it.  All that other stuff, yeah, you're probably right.

[justusranvier]

According to the IRC conversation with Magicaltux, the exchange engine and front end are running on the SAME SERVER

If that is true, it points to a level of amateurish hackery which is unbelievable.

A system like this should have multiple geographically redundant load balancers, multi-tier hierarchy of workers, web engines, trade engines, market data and charting engines. The DNS should be run by an independent DNS provider and they should have DDoS protection, regional failover, graceful degradation, etc etc.

It seems they have none of that, nor any clue how to do any of it. It's not even expensive nowadays, it just requires knowledge of scalable distributed systems.

The worst thing is that what you've pointed out has been obvious since 2011. They've had two years between when these problems were identified and now to implement proper solutions, and either did nothing or did something which has not been effective.

[Ekaros]

According to the IRC conversation with Magicaltux, the exchange engine and front end are running on the SAME SERVER

If that is true, it points to a level of amateurish hackery which is unbelievable.

A system like this should have multiple geographically redundant load balancers, multi-tier hierarchy of workers, web engines, trade engines, market data and charting engines. The DNS should be run by an independent DNS provider and they should have DDoS protection, regional failover, graceful degradation, etc etc.

It seems they have none of that, nor any clue how to do any of it. It's not even expensive nowadays, it just requires knowledge of scalable distributed systems.

One server? With amount of money I heard they have around?

At minimum the web services should on different servers than the trading...

Truly amateurs, and I'm a amateur...

[IIOII]

What I do not understand from a rational perspective (well, I can understand this from a mass psychological one):

It's a free market. Why does everyone use Mt.Gox?
There are quite a few alternatives.

[aantonop]

Because there's a feedback loop:

Existing volume creates a good environment for trading, which attracts volume. It's a virtuous cycle that is only broken when they fuck up (more and more, as the volume increases).

There's a lot of 'inertia" from people who are not willing to try another exchange because of the hassle of verification and signup (weeks), wire transfers (weeks) etc.

Which is why people need to use this opportunity to STAY ANGRY and remember why they are angry. Use that anger to motivate them to sign up to new exchanges today. So that next time this happens (and it will, sooner rather than later), they can dump gox.

[IIOII]

Because there's a feedback loop:

Existing volume creates a good environment for trading, which attracts volume. It's a virtuous cycle that is only broken when they fuck up (more and more, as the volume increases).

There's a lot of 'inertia" from people who are not willing to try another exchange because of the hassle of verification and signup (weeks), wire transfers (weeks) etc.

Which is why people need to use this opportunity to STAY ANGRY and remember why they are angry. Use that anger to motivate them to sign up to new exchanges today. So that next time this happens (and it will, sooner rather than later), they can dump gox.

Yeah.

Maybe we need an even more catastrophic event at Mt.Gox before people start to rethink using it.

[n8rwJeTt8TrrLKPa55eU]

They do have DDoS protection.  Cost them a pretty penny in 2011.  I'm assuming they still have it.  All that other stuff, yeah, you're probably right.

DDOS protection against external traffic is logical...but simultaneously they have a giant "DDOS welcome" by allowing arbitrarily small trades and bots to flood their crappy engine wihtout restrictions.  Thus any registered user can cause massive disruption through an atttack via penny trades.

They have known for ages that their engine has flaws.  Don't want to fix that, or it's too complicated in the near term?  At least mitigate overloading scenarios by imposing a minimum trade size in BTC or USD, or raise fees to make HFT difficult, or restrict the trading API to only trusted businesses and payment proccessors, until you can get your scalability fixed.

[aantonop]

Yeah.

Maybe we need an even more catastrophic event at Mt.Gox before people start to rethink using it.

Wiping out a BILLION DOLLARS wasn't enough for you?

[kiko]

lol, from bitcoinity:

last trade: 4 minutes ago.
Mtgox lag: 8 seconds.

wat.