Bitcoin Forum

We are victims of our own success.
https://mtgox.com/press_release_20130411.html
More and more people will run to get a fragment of bitcoin.

Yep, no DDOS, just the trading engine lagged to almost the length of a feature film when one guy decided to sell a few thousand.  Success!

I re-post the Mt Gox declaration because most of the time is still not available.
Hopefully I will get no copyright problems:

"      TOKYO - JAPAN - April 11, 2013

      Hi everyone, just a quick update on the situation and what happened last night.

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the number of trade made a huge impact on the overall system that started to lag. As expected in such situation people started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze the trade engine!

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys:
- The number of trades executed tripled in the last 24hrs.
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new servers to our system.

Thank you for your understanding and continuous support!
      
   

Regards
Mt.Gox Co. Ltd Team.

Mt.Gox Contact press@mtgox.com"

No we are victims of incompetent mtgox who don't know how to run a trading engine, are making shit up as they go and keep blaming everything else except their own incompetence.

They have no plan because they have no clue...

RUN to another exchange.

MtGox.com ‏@MtGox Twitter:

DDoS, success, you can never be sure with these things...

http://www.reactiongifs.com/wp-content/uploads/2013/03/coy-dunno.gif

Is Mt.Gox considering the lag caused by a few trades per second a success?
Is it still a success if the trades are generated by bots buying and selling 0.0001 BTC back and forth?
Is it still a success if the trading bots are doing these trades with the intention of causing more lag, or is it suddenly a DOS then?

YES

So basically this is not MtGox lying, this is MtGox doing the spin cycle. "We stepped in poo, we're great ecologists".

Basically this is a bunch of teenage dorks caught deer-in-headlights during an attempt to parlay everything they've learned about how to "seduce womenz" into the business of money exchanging. It's working about on par with what you'd expect.

The only weakness of Bitcoin currently is just a lack of exchanges to diversify the network.

And who knows what kind of hardware they are running this on. A professional op would not have gone down like that. There should have been a failover ready to roll in such a scenario, whatever it is it is not fault tolerant in any way which isn't good.

I imagine I will begin one of my own someday, though setting up an exchange isn't all that documented yet.

According to the IRC conversation with Magicaltux, the exchange engine and front end are running on the SAME SERVER

If that is true, it points to a level of amateurish hackery which is unbelievable.

A system like this should have multiple geographically redundant load balancers, multi-tier hierarchy of workers, web engines, trade engines, market data and charting engines. The DNS should be run by an independent DNS provider and they should have DDoS protection, regional failover, graceful degradation, etc etc.

It seems they have none of that, nor any clue how to do any of it. It's not even expensive nowadays, it just requires knowledge of scalable distributed systems.

They do have DDoS protection.  Cost them a pretty penny in 2011.  I'm assuming they still have it.  All that other stuff, yeah, you're probably right.

The worst thing is that what you've pointed out has been obvious since 2011. They've had two years between when these problems were identified and now to implement proper solutions, and either did nothing or did something which has not been effective.

One server? With amount of money I heard they have around?

At minimum the web services should on different servers than the trading...

Truly amateurs, and I'm a amateur...

What I do not understand from a rational perspective (well, I can understand this from a mass psychological one):

It's a free market. Why does everyone use Mt.Gox?
There are quite a few alternatives.

Because there's a feedback loop:

Existing volume creates a good environment for trading, which attracts volume. It's a virtuous cycle that is only broken when they fuck up (more and more, as the volume increases).

There's a lot of 'inertia" from people who are not willing to try another exchange because of the hassle of verification and signup (weeks), wire transfers (weeks) etc.

Which is why people need to use this opportunity to STAY ANGRY and remember why they are angry. Use that anger to motivate them to sign up to new exchanges today. So that next time this happens (and it will, sooner rather than later), they can dump gox.

Yeah.

Maybe we need an even more catastrophic event at Mt.Gox before people start to rethink using it.

DDOS protection against external traffic is logical...but simultaneously they have a giant "DDOS welcome" by allowing arbitrarily small trades and bots to flood their crappy engine wihtout restrictions.  Thus any registered user can cause massive disruption through an atttack via penny trades.

They have known for ages that their engine has flaws.  Don't want to fix that, or it's too complicated in the near term?  At least mitigate overloading scenarios by imposing a minimum trade size in BTC or USD, or raise fees to make HFT difficult, or restrict the trading API to only trusted businesses and payment proccessors, until you can get your scalability fixed.

Wiping out a BILLION DOLLARS wasn't enough for you?

lol, from bitcoinity:

last trade: 4 minutes ago.
Mtgox lag: 8 seconds.

wat.

I do not use Mt.Gox. But I doubt it will be enough for the majority of users.

MtGox was always PHP/mysql, including the trading engine. Do you know if it is still the same?

<shudder>

The more I learn about mtgox, the more incompetent they seem. PHP/MySQL? For a billion dollar market?

I'm revising my opinion. Negligence is a much more appropriate word than incompetence.

Umm, isn't using PHP at trading engine bit incompetent? For web stuff yes, but trading engine?

Haven't they really not hired any professionals? With industry experience?

If they plan on closing the exchange for 2-4 hours in the next 12 to 24,  how about a heads up and notification on specifically when this might be.  If I had to guess, they will do this around 3pm EST with no notification at the very moment the price becomes most volatile.  
Clowns.

And what has Coinlab done to help the situation at Mt.Gox?  When Peter Veseness announced the partnership with Mt. Gox, it was described as all kitten and rainbows.  I can't figure out what advantages that partnership has brought.

PLAN?

You think they PLAN?

HAHAHAHAHA!

It's pretty standard for Bitcoin services.  They find themselves growing before they have the infrastructure in place to support that growth.  MtGox has had two years to prepare for this, though, so it's pretty unforgivable.  With Bitcoin prices so high, they're going to be a bigger target than ever so lets hope they've spent an enormous amount on hardening their security over the last two years because no-one's going to forgive any significant loss of user funds at this stage in the game and they don't have an unlimited capacity to absorb losses.

Two years, tons of revenue, funding... They have everything they need to fix this except for the skills and competence.

Unforgivable. This is not some little startup that just showed up. This is 80% volume of a $2 billion dollar market, with revenues in the multi-million per year.

We need to stop making excuses and accept that mtgox will not fix itself because they dont know how.

I think now would be the right time that Mark does an Eleuthria https://bitcointalk.org/index.php?topic=168108.0 (https://bitcointalk.org/index.php?topic=168108.0)

Enough with the a hissin' 'n moanin' ... we've known this for ages ... get your sorry asses out there and build somethiing better.

It's a wide open free market and there is enormous greenfields potential for new exchanges, tons of moolah to be made ... show us all what ya got besides big ideas and big yaps  :D

marcus, with respect, you're missing the point.

There are 5-6 other exchanges. They don't have volume, even though they have good-enough or better-than mtgox trading engines and interfaces.

They don't have volume because of the inertia of people giving mtgox a second chance. Then a third, fourth, fifth and sixth chance, even while the fuckups keep getting bigger.

Hissin' and moanin' is what gets newbies to notice that mtgox is associated with a bunch of hissin and moaning and hopefully pick a different exchange to join.

So do some productive moaning too and spread the word: Boycott MTGOX. They're incorrigible.

i cannot habeeb that the lag was caused solely by traffic, and if it was, then they obviously are very behind on hardware and security for the amount the have invested.

Can't say I ever endorsed them after the infamous "I was Goxxed!" rollback fiasco and the onerous requirements on customer interrogations (suspect data-mining motives) ... I've never used them and don't have intention to unless I see a massive change in operations. ... but damn they get some volume, there is no telling the stupidity of most people.

Gox are incorrigible, don't use them people!

How was that?

Superb.

I especially liked the part where you stretched out the word incorrigible so it sounded like "They can take our lives but they can never take our FREEEEEEDDOOOOOM". Definitely the right level of passion for the job.

For a funded op MTGOX seems to have zero in the way of failover redundancy, which is server room 101 when you have systems that need to mush on without stopping. I smell the stink of Gox being run by a bunch of inexperienced kids.

Eleuthria is 'da man

I am astonished.   What a crap  all over around  (all forums, portals, FB, twitter etc) ?  A "massive crash"  ?!

What ???!   A crash ???   
But we are simply where we all been only ONE (1!) week ago.  Just think of it !  ONE week ago.
How many days has passed since we all were  sreaming   "look !  BTC is already $100+  today !

I deeply believe that all these events has nothing to do with trust/believe in BTC, with the fundamentals of BTC

From the other side - WHAT can you expect, when MtGox, the largest exchange (70-80% market share) has an operation  time lag of 4600-4800 seconds ! 
Grab you calculator and think again of what you got - that is a HUGE 1.2 - 1.5 hours  LAG  !!!

What kind of trade (logical?) can you expect on such a crap system???
And compare this to a advanced FOREX, NYSE automatic trading systems, where   serious contenders would not even try their chances, unless their access delay (ping, lag, what ever you call it) is less then 1-2 ms ( mili-seconds, that is)

Think of it. 
While you can ( & while exchanges are stopped for a cool-off)

I can not speak to the security but the servers they would need ( if they bought them ) would be less then $100,000
For gox that would be a few days or even less profit.

DDOS protection in this case might be making things worse.

Mt. GOX will be the DEMISE of bitcoin!  Mark my words.

I guess mtgox got their hands in this somehow...

mtgox has sucked right from the very beginning (ugly screenshot here: https://i.imgur.com/vlKihqr.jpg)

it's incredible that people just keep on using it  :o

Impressive.  :o

However in the last years they've built something more complex based on this competence...  ;D

Fucking idiots!

A billion wiped is a success? What company in their right mind would accept such a currency?? Bitcoin from here is a ponzi scheme until prices go to where they should be, circa 15$ and stabilize.

you do know that btc is to be used for trade and not speculation..

store of value will come with supply and demand.

Work to secure the network is going to be needed even if BTC is worth $1 or $1000.

I'm new to it, my impressions are it is a casino. Only thing I can see you can buy which is worthwhile is gold/silver coins & other bitcoins or alt coins. LONG way to go. And if it can be manipulated so easily by dumping a small amount of coins how is it going to survive when the central banks come for it, which they will, if they have not already.