Mtgox statement on facebook

[Thursday]

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

[1715629327]

1715629327

[1715629327]

1715629327

[wonkytonky]

From mtgox facebook: https://www.facebook.com/MtGox/posts/455962117821534

Code:

Hi everyone, just a quick update on the situation and what happened last night.

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own 
success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the 
number of trade made a huge impact on the overall system that started to lag. As expected in such situation people 
started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze 
the trade engine! 

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys: 
- The number of trades executed triple in the last 24hrs. 
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days
of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working 
around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today 
and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new 
servers to our system.

Thank you for your understanding and continuous support!

Holy Crap! 20,000 Accounts a day?!?!?!

that number does not make much sense..    to me;. gox queue in total can be this big. but not on a daily basis..   I know people waiting for days now so and they are around number 10k..      20k a day.. sounds a bit far stretched...  to me..  

[caveden]

that number does not make much sense..    to me;. gox queue in total can be this big. but not on a daily basis..   I know people waiting for days now so and they are around number 10k..      20k a day.. sounds a bit far stretched...  to me..  

Aren't people in US allowed to start trading small volumes without verifying?
I think verification is only mandatory for Europeans, but I'm not sure.

[codro]

If you look at their trade feed, there are barely a few trades per minute in times of high lag. I still don't understand how they can capitulate under what is essentially so little volume.

The engine isn't even capable of >one trade per second most times, Nasdaq does at least 35,000 per second and is tested up to 60,000.

http://www.nasdaq.com/services/homw.stm

The order of magnitude difference doesn't make sense to me. Sure, some is to be expected, but not that much.

[ruski]

If you look at their trade feed, there are barely a few trades per minute in times of high lag. I still don't understand how they can capitulate under what is essentially so little volume.

The engine isn't even capable of >one trade per second most times, Nasdaq does at least 35,000 per second and is tested up to 60,000.

http://www.nasdaq.com/services/homw.stm

The order of magnitude difference doesn't make sense to me. Sure, some is to be expected, but not that much.

Have you considered the glut of orders is preventing execution, thus our noticing it in the first place? I'd say that's the root of the whole problem. If they had all their trades going through we wouldn't be worried.

[crazy_rabbit]

From mtgox facebook: https://www.facebook.com/MtGox/posts/455962117821534

Code:

Hi everyone, just a quick update on the situation and what happened last night.

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own 
success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the 
number of trade made a huge impact on the overall system that started to lag. As expected in such situation people 
started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze 
the trade engine! 

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys: 
- The number of trades executed triple in the last 24hrs. 
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days
of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working 
around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today 
and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new 
servers to our system.

Thank you for your understanding and continuous support!

Holy Crap! 20,000 Accounts a day?!?!?!

that number does not make much sense..    to me;. gox queue in total can be this big. but not on a daily basis..   I know people waiting for days now so and they are around number 10k..      20k a day.. sounds a bit far stretched...  to me..   

You don't have to get verefied to sign up and play with small amounts. 20K new users would be possible, I'm sure that verification que will skyrocket soon.

[minimalB]

I hope a lot of Mtgox users saw this FB message and will move to another exchange... more balanced traffic on different exchanges would be healthier for Bitcoin.

[jago25_98]

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

Agreed. It's good for Bitcoin.

The micro trading bots spamming are interesting. Very important not to react like the USSA and ban such things but to react to them in a fluid market response. Established markets also have this problem of spamming - I expect this community to be better than those trillion pound markets in the response.

If it's 20k accounts a day then we have to presume that KYC is outsourced to a specialist company for this? Or is it in someone's garage handling passport scans and bank statements?

[tpantlik]

Why yes, they have. They purchased Bitomat after it collapsed because the servers crashed and Amazon couldn't do data recovery because it was in the cloud. So yes, they are well aware of cloud services.

I'm not even sure where to start.

You think that if a server crashes 'in the cloud' that the data is unrecoverable?

You don't think that the particular people at Bitomat might have just not known what they were doing?

'In the cloud' isn't some magical meta-space where things don't exist and the data is unrecoverable, it just means the machines are somewhere else and you can programatically request more of them, for you know, like, proper load balancing.

This stuff has been figured out like 10 years ago buddy MtGox just don't know what they are doing.

LOL, I think that Maged surely know this stuff  But anyway did you heard of Linode?

[Sage]

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

What's it going to take before the Bitcoin community gets together and creates a viable decentralized exchange to replace Mtgox?

How loud do we need to scream it!  Get a decentralized exchange platform going.  Please, please, those with the technical know-how, lets get the discussion started.

[briannguyen]

I am sure mtgox earning a decent amount of money since the bitcoin price rise. They should spend them all to upgrade the infrastructure and trading platform to prevent any problem. Bitcoin trades 24/7 and requires a high end reliable system. I was very disappointed with the service yesterday

[samson]

They should get rid of the tiny trades or charge more for anything less than a certain limit.

Thousands of tiny orders are not required anymore and they're annoying.

[Francesco]

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

Does anyone know of a good exchange? BitStamp lagged like hell as well, and I heard it's not the only one. (well, even the forum and bitcoincharts did)

Really, we read every day about the wonders offered to today's computing by ever progressing cloud/virtualization/whatnot technologies -and the only ones that know not how to properly use them are the ones managing our money?

[zoolander]

Have they not heard of Amazon EC2? Google Compute Engine? Heroku? Azure?!

"Adding more servers" - what is it, the 90s? Amateurs..

We've learnt that you can never add enough servers to cope with an unpredictable and quickly-changing demand curve. That's why they invented elastic computing. They add 3 servers now, next month those 3 servers can't handle. So they add 3 more, and so on and so on.

As much as I think Gox has some very serious issues and is currently unfit for purpose (clearly they have some horrible legacy and crazy architecture to content with - e.g. I heard someone mention the web front is not decoupled from the trading engine - WTF?!) it just not that simple.

Anybody who knows about how to build fast highly secure exchanges that have to guarantee perfect consistency with regards to state/transactions would think twice about using any sort of public virtualised infrastructure. Physical hardware is still the norm in this domain because raw single threaded clock speed and fast IO (e.g. solid state drives) often matters a great deal for exchanges and trading platforms. To give you to example I have direct knowledge of: I worked at Betfair in the past and they were considering building bespoke hardware chips to run the code of their core exchange engine to ensure they could scale their platform for the future and I know a friend at Merrill Lynch who told me about their servers which ran in oil baths to enable over-clocking raw CPU speed matters. There are many other example of custom hardware in use by banks and the like. To think anyone would use Heroku for an exchange is laughable. And I speak as someone who choose to build an app on Heroku or any other PaaS/IaaS provider 99% of the time. And I didn't even mention security.

What I do hold Gox accountable for though is not investing in their platform a long time ago. I can't believe they haven't had the money to build an entirely new trading engine and hire a really good team to build and support it. They have been very short-sighted. Any good start-up has a plan in place to scale if they start to experience exponential growth.

I think we should be using some of our Bitcoins to fund the building worthy alternative - I certainty don't hold much faith in any of the others so far. An exchange for bitcoiners, built by bitcoiners.

[briannguyen]

elastic computing service might be too expensive for mt.gox?

[w1R903]

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

What's it going to take before the Bitcoin community gets together and creates a viable decentralized exchange to replace Mtgox?

How loud do we need to scream it!  Get a decentralized exchange platform going.  Please, please, those with the technical know-how, lets get the discussion started.

There are some excellent alternatives to Mt. Gox that already exist: bitstamp, campbx, bitfloor.  But most of you all won't switch or strongly suggest that your friends/family go to one of these alternatives.  Why in the world would we with technical know-how want to spend time building yet another exchange when it's clear most people just go with the best-known name.

[Amitabh S]

Let me take a guess... PHP and MySQL??

[Capitan]

As I said in another thread, they need to get 10 times more servers than they think they need. Gox never seems to be up to par, its like they are ALWAYS playing catch up.

They did. In fact, they had much more than that. As they said, in just the last day the number of trades tripled! It's truly amazing that they scaled as well as they did.

Seems like they may have a bottleneck somewhere that adding servers won't fix. High probability that it's in the database, which is pretty hard to scale by just adding servers. But i don't see how the amount of traffic they get should be this much of a problem for them. They should be  able buy a powerful enough database server with one or two day's worth of the profits they make (assuming the $100K+ per day profit I saw posted in another thread is true). I've worked on websites that got 1M+ hits per day in traffic spikes, not quite the same type of app as an online exchange, but I know a little bit about performance and scaling.

For the amount of money they make, and the amount of traffic they do, there is simply no excuse. Their public statements don't instill confidence. First they said "you guys have no idea what it's like, come work with us for a day, we can handle DoS attacks, where withstood two at the same time without any lag", then the next day they are down after the maintenance because of a DoS attack.

[deadweasel]

As I said in another thread, they need to get 10 times more servers than they think they need. Gox never seems to be up to par, its like they are ALWAYS playing catch up.

+1

[Capitan]

So, a few sentences about this issue.

* A drop from 266 to 105 will make headlines all over! Result: Bitcoin publicity
* Knowing that the issue was cause by a huge amount of new accounts Result: BULL
* Crashing the market. Result: MtGox making a shit ton of money

It's a Win-Win for everyone except the weak hands!

The drop and the publicity about it could also dampen the public interest.