[ANN Mt.Gox] Hi everyone, just a quick update on the situation and what happened

[Mt.Gox_Alex]

https://mtgox.com/press_release_20130411.html

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the number of trade made a huge impact on the overall system that started to lag. As expected in such situation people started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze the trade engine!

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys:
- The number of trades executed tripled in the last 24hrs.
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new servers to our system.

Thank you for your understanding and continuous support!
      
Regards
Mt.Gox Co. Ltd Team.

Mt.Gox Contact press@mtgox.com

[jojo69]

I would have more sympathy if this was not part of a long history of lackadaisical malfeasance on the part of MtGox.  You have no incentive to fix your steaming pile of crap and I do not believe you will.

[nebulus]

That's a lot of traffic, good luck and thanks for the update!

[Mt.Gox_Alex]

I would have more sympathy if this was not part of a long history of lackadaisical malfeasance on the part of MtGox.  You have no incentive to fix your steaming pile of crap and I do not believe you will.

You are obviously no ideas of what you are talking about, come and work with us you will see how things are done and what we have to deal with on a daily basis. As I write this we have two concurrent DDoS attack with NO lags and still online. Things always seems easy at the other end of the keyboard.

[Adrian-x]

https://mtgox.com/press_release_20130411.html

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the number of trade made a huge impact on the overall system that started to lag. As expected in such situation people started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze the trade engine!

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys:
- The number of trades executed tripled in the last 24hrs.
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new servers to our system.

Thank you for your understanding and continuous support!
      
Regards
Mt.Gox Co. Ltd Team.

Mt.Gox Contact press@mtgox.com

Sure thing enjoy the fruits of all those buy orders executed 20% 30% higher simultaneously with the sell orders.

I think an apology would have been more fitting than a simple Thank you.

[Inedible]

Wait, so are you or are you not being DDoS'd?

[🏰 TradeFortress 🏰]

Wait, so are you or are you not being DDoS'd?

I think are but the lag wasn't due to the DDoS.

[repentance]

I would have more sympathy if this was not part of a long history of lackadaisical malfeasance on the part of MtGox.  You have no incentive to fix your steaming pile of crap and I do not believe you will.

You are obviously no ideas of what you are talking about, come and work with us you will see how things are done and what we have to deal with on a daily basis. As I write this we have two concurrent DDoS attack with NO lags and still online. Things always seems easy at the other end of the keyboard.

Alex, please don't do this.  To date MtGox has refrained from insulting customers and I'd like to see it stay that way.  At the end of the day "what you have to deal with" should not be your customers' problem.  You need to explain what's happening without being defensive or aggressive.

Can I also implore you to please be more communicative about other issues such as the cessation of non-BTC MtGox vouchers and the CoinLab transition?  It's not enough to simply announce these things.  As they come closer to being implemented you need to be actively reminding your customers that they're about to happen.

[TheButterZone]

FTFY

Shut down trading NOW and DO NOT resume until your trade engine is made NYSE-grade DDOS, lag, and all other forms of bullshit, resistant.

That's directed at ALL exchanges, BTW.

STOP allowing Bitcoin's market value to be bent over and gang raped bloody OVER AND OVER AND OVER AND OVER AND OVER AGAIN.

[bitsalame]

Please don't be pulling another "force majeure" crap again.
With more than half million dollars a month in revenue, I would expect no excuses in performance.
How on Earth are you going to survive the deal with Coinlabs?

[aantonop]

Instead of coinlab getting partnered with the "expert", they got saddled with the broken exchange whose owners don't have a clue how to run a high-performance and scalable infrastructure.

Magic The Gathering Online Exhange is a systemic risk to bitcoin, a death trap for traders and a business run by the clueless.

I'm not only leaving mtgox, I'm not going to touch coinlab with a 10ft pole.

This press release shows a complete disregard for customers, is full of excuses and no plan to fix anything. Throw more servers at it, see if that makes your non-scalable software suck less. It won't.

I'm willing to try others, after all what can be worse than an HOUR lag?

Campbx.com, bitfloor, coinbase, anything but gox or their new victim/partner coinlab.

[SomeWhere]

So much truth in this thread.

[repentance]

FTFY

Shut down trading NOW and DO NOT resume until your trade engine is made NYSE-grade DDOS, lag, and all other forms of bullshit, resistant.

That's directed at ALL exchanges, BTW.

STOP allowing Bitcoin's market value to be bent over and gang raped bloody OVER AND OVER AND OVER AND OVER AND OVER AGAIN.

In the reddit thread, MtGox are blaming HFT.  Of course it's totally within their control to set a minimum order limit and/or a minimum time between trades but instead of doing that until they have the infrastructure to actually handle HFT, they're going to keep accepting more customers (many of whom are new to Bitcoin and are going to get pissed off when they realise they wont be able to withdraw their profits without getting verified and who have no idea about the proposed CoinLab transition) and just throw on some more servers.

Bitcoin services who have been in businesses for as long as MtGox have no excuse whatsoever for being unable to scale.  The capacity to handle an explosion of new users should have been put in place long ago.

If I had to guess, I suspect that MtGox doesn't have the capital to install the type of infrastructure which is needed right now to deal with the explosive growth in use.  If you cannot scale, you will fail.  Count on it.

[aantonop]

You are obviously no ideas of what you are talking about, come and work with us you will see how things are done and what we have to deal with on a daily basis. As I write this we have two concurrent DDoS attack with NO lags and still online. Things always seems easy at the other end of the keyboard.

This really takes the price for EPIC FAIL followed by tone deaf attack on your own customers.

Wow, just wow!

Since you decided anyone calling your bullshit with you doesn't know how hard it is on your end of the keyboard, I'll offer you this:



That's me on the left. I'll give you three guesses as to the location and why I was invited to this by the CEO.

So as someone who does know what they're talking about, SCREW YOU Alex and your company. First you fail, then you insult your customers for having the audacity to call you on your BS.

Everyone reading this: Open an account on bitfloor, campbx, btc-e or any (ALL) of the other exchanges. Close your MTGOX account.

[Scrat Acorns]

This isn't new. Your engine has been lagging for the past 3 months whenever there is so much as a hint of volume.

Anything more than a few seconds of lag in the real financial world is UNACCEPTABLE. You wasted a lot of people's money yesterday. These people will be taking their business elsewhere.

[Isokivi]

Mt.Gox_Alex thank you for the update, some news earlyer would of have been good too. Keep up the good work your doing, atleast a large portion of the more seasoned bitcoiners can fully understand the circumstances you are coping with at the moment and can clearly see you are handling things supprisingly well.

Couch critics, if Mt. Gox is doing such a bad job, please start competing with them.. the market should be wide open by your reasoning.

[codro]

If you look at the trade feed, there are barely a few trades per minute in times of high lag. I still don't understand how they can capitulate under what is essentially so little volume.

The engine isn't even capable of >one trade per second most times, Nasdaq does at least 35,000 per second and is tested up to 60,000.

http://www.nasdaq.com/services/homw.stm

The order of magnitude difference doesn't make sense to me. Sure, some is to be expected, but not that much.

[Liquid]

I knew it! i watched it booming yesterday and this morning was a shock when it was at $150, $100 less but i knew what happened you got overloaded well its all good now we can watch bitcoin go to $250 + again  

[codro]

[12:29] <keymone_> MagicalTux tell me why are you not using more mature exchange system?
[12:29] <keymone_> like milleniumit or something
[12:30] <keymone_> the one that is behind london stock exchange
[12:30] <keymone_> why reinventing the wheel especially when you see the incoming exponential usage growth
[12:30] <@MagicalTux> keymone_: does it handle ddos attacks well?
[12:31] <keymone_> yesterday press release said it was not due to ddos
[12:31] <keymone_> it's 2 separate problems, solving 1 is already a huge progress
[12:31] <@MagicalTux> keymone_: we are already solving this
[12:31] <bit14> @MagicalTux: why exactly would a ddos affect the matching and execution engine?
[12:31] <@MagicalTux> bit14: overloading system cpu
[12:31] <@MagicalTux> ddos are not all about cpu
[12:32] <dandate2> um did rebooting mtgox cause already deleted recent bids to come into play?
[12:32] <bit14> @MagicalTux: wouldn't the front end and execution engine be on different machines?
[12:32] <@MagicalTux> dandate2: rebooting?
[12:32] <keymone_> are you trying to battle trading bots? with all these 0.1 buys and sells?
[12:32] <@MagicalTux> bit14: not yet
[12:32] <@MagicalTux> keymone_: unrelated
[12:32] <bit14> @MagicalTux: wow, well, then that explains everything

I don't mean to add even more insult to injury, but as a software engineer myself - I can't even comprehend why someone would create such an abomination.

[TheButterZone]

The back patting will continue until morale "improves" to $0.00000001/BTC.