Mtgox statement on facebook

[wamatt]

Lol, I think they need more than just a "few" new servers.

Remember that during GoxLag, people panic and sell massive amounts of coin - resulting in higher profits for Gox than a normal day of trading. Their incentive to improve capacity is limited to the threat of people leaving Gox due to the lag. And it looks like not very many people are leaving... big resounding LOL. Everyone's used to the granny nanny socialist state where regulatory bodies attempt to protect them from poorly performing businesses. They don't do their research on gox... it takes like 2 seconds of lurking here to realize that Gox Sux.

Get out of Gox! They're going to be playing "catch up" forever until we send them a powerful message.

Right on comrade! Bitstamp here we come, really blown away by the professionalism of that operation compared to Mt Gox

[farfiman]

Gox should move out of the garage and start becoming professional. They still have enormous potential despite being the largest already.

Even Apple had to do that eventually

[deathcode]

From http://www.pcworld.com/article/2033809/bitcoin-volatility-caused-by-surge-in-demand-slow-software.html

Bitcoin's roller-coaster price swings on Wednesday were caused by an influx of new buyers and software that couldn't keep up, according to the largest exchange, Mt. Gox.

People trading bitcoin on the Tokyo-based exchange saw its price hit US$266 at one point, then fall to as low as $105 with high lag times for trades. The surge in interest in bitcoin has overwhelmed Mt. Gox.

"We are running like crazy," said Gonzague Gay-Bouchery, who heads marketing for Mt. Gox, on Thursday in an interview. "We are trying to do everything by the book and trying to make everything work smoothly."

Mt. Gox and other exchanges that allow people to buy bitcoin all appeared to have varying degrees of problems on Wednesday, illustrating how small Internet companies often grapple with scaling their services amid rising interest.

Bitcoin, a virtual currency that is transferred by a peer-to-peer network, has been propelled in recent weeks by mainstream media coverage, although the number of transactions remains very small compared to traditional financial networks.

In just two days earlier this month, Mt. Gox gained 75,000 new users, and some 20,000 new ones in the last three days, Gay-Bouchery said. As soon as Mt. Gox credits their accounts with cash, those users jump into trading. Mt. Gox's system has had trouble processing those trades quickly, which has lead to vast price swings as users try to quickly complete trades and anticipate price moves.

"Bitcoin will go up and down," Gay-Bouchery said. "If there is a lag, don't panic."

Last week, Mt. Gox said it fought a massive distributed denial-of-service (DDoS) attack intended to manipulate the price of bitcoin. Although Mt. Gox's exchange is nearly always under a DDoS attack, Gay-Bouchery said Wednesday's issues were not related to DDoS.

Mt. Gox has spent an "enormous" amount of money on new servers for its bitcoin trading engine. Already, the exchange uses servers with 64GB of RAM each, but it hasn't been enough to keep the site running perfectly. The new servers, which have solid-state drives for faster data input and output, are due to arrive from the U.S. soon, he said.

The exchange has also been recently contacted by major server manufacturers who supply technology to the largest banks in the world, Gay-Bouchery said. The companies said they can solve Mt. Gox's trading woes.

The code for Mt. Gox's trading platform would have to be changed to accommodate new hardware, but it would give the exchange a trading platform comparable to that of a bank, lending more stability, Gay-Bouchery said.

Other exchanges that sell bitcoin were also groaning under the strain. Bit Innovate in Australia wrote on Twitter that is website was under stress due to increased demand. On Wednesday, it appeared at one point to simply run out of bitcoins to sell.

"Due to extremely high demand, we have had to temporarily close our Buy Bitcoins service," Bit Innovate wrote. "Sourcing more. It will be open as soon possible."

BitFloor, the largest U.S. bitcoin exchange based in New York, warned of delays and technical issues with its website. Roman Shtylman, the founder of BitFloor, said his exchange saw its highest trading volume and highest number of users ever.

Shtylman said he was working on some changes to ensure smooth trading following Wednesday's frenzy.

[jojo69]

Mt. Gox has spent an "enormous" amount of money on new servers

yeah, right, enormous 

[Amitabh S]

This is not good for bitcoin. What will gox do about causing the "correction".

What a huge central point of failure for bitcoin!

[sedeki]

Overall, I think this is a good thing: people have gotten an interest in bitcoin.

[threeip]

Overall, I think this is a good thing: people have gotten an interest in bitcoin.

60k for March alone to 75k new account created for the first few days of April! We now have roughly 20,000 new accounts created each day.

At what cost?

[deathcode]

So, a few sentences about this issue.

* A drop from 266 to 105 will make headlines all over! Result: Bitcoin publicity
* Knowing that the issue was cause by a huge amount of new accounts Result: BULL
* Crashing the market. Result: MtGox making a shit ton of money

It's a Win-Win for everyone except the weak hands!

[HappyBitCoinUser]

MtGox makes money during both panic buy and panic sell, cause of fee for each transaction. The more buying and selling, regardless of reason, is fee profits!

MtGox wins!

[deathcode]

MtGox makes money during both panic buy and panic sell, cause of fee for each transaction. The more buying and selling, regardless of reason, is fee profits!

MtGox wins!

Exactly, but during these kinds of sell-offs... the volume goes to the roof!

[piramida]

Have they not heard of Amazon EC2? Google Compute Engine? Heroku? Azure?!

"Adding more servers" - what is it, the 90s? Amateurs..

We've learnt that you can never add enough servers to cope with an unpredictable and quickly-changing demand curve. That's why they invented elastic computing. They add 3 servers now, next month those 3 servers can't handle. So they add 3 more, and so on and so on.

+1 that; either that, or, in case their regulations insist they run in a closed-doors DC, then they MUST make sure that their normal trading volume is not loading any part of the system higher than 10%. 3x increase in volume should be absolutely normal and should not cause even a slightest delay.

Amateurs indeed. Bitstamp has heavily lagged as well, the only exchange operating normally was btc-e.

[simplydt]

As I said in another thread, they need to get 10 times more servers than they think they need. Gox never seems to be up to par, its like they are ALWAYS playing catch up.

They have some weird policy in place of we add more servers when we need them. Reacting to market demand rather than as you suggest predicting it and playing ahead of it. Sucks. It just doesn't work.

[zedicus]

ask em about that!!! Macro order flooding  via bots that trap out regular customer but computer systems dont get trapped out in a panic and they can drive it down via 50 a pop or smaller!! happens every-time is there is a panic! Like someone is manipulating !! Doesn't care about $ its micro manipulate and watch it tank! (look at the dam pics!!)

 Gox needs to filter out micro or macro trades or do something! Thats whats cause's the wild drops and panic !! Probably banks with some bots and endless fiat money!  Needs to be addressed!


Drive it down via those small trades and watch the herd jump ship ! Since there is no cooling period or auto shut off its disastrous! !

Thats whats gonna kill it!  We need a % slide rule and cooling period put in like the real wall-street trading platforms do!

If it falls by x amount they stop trading for x amount of time! "Cooling period"  Hours .. till the next day! Whatever .. something is better than nothing!




At this rate in this setup.. a foreign entity with negative intention's towards bitcoin could micro trades us to our deaths!

[HappyBitCoinUser]

How about anyone caught using a bot for macro ordering gets frozen/suspending pending investigation?

[Ghostofkobra]

The basic problem with clouds are that they are quite expensive when you have a high volume of traffic and that the servers are not that powerfull.
You end up renting more cloudservers than you would if you used physical servers. Which means that you have to scale your operation wider, and as
always with scaling as the number of servers grow the extra power you get from adding an extra server decreases.

For some compute intensive systems (like mining/hashing) the power decrease very slowly, however for something where data has to be shared the curve can be quite drastic.

Some software (and im not talking hobby coders but software from the 10 original software companies) that has negative scaling after a certain amount of parallelization. I.e. the
time INCREASES when you add more servers. It always has to do with the basic algorithm and then there are 2 options, redesign algorithm or hardware scaleup.
Algorithm design is extremely expensive and not always even possible (?). Hardware scaleup is much easier when you have your own machines and you DEFINATELY do not want to
run the stuff in virtual machines!!!

There is a reason why "real" banks use specialized hardware.


Well now i have to get back to work.
//GoK

I'm not even sure where to start.

You think that if a server crashes 'in the cloud' that the data is unrecoverable?

You don't think that the particular people at Bitomat might have just not known what they were doing?

'In the cloud' isn't some magical meta-space where things don't exist and the data is unrecoverable, it just means the machines are somewhere else and you can programatically request more of them, for you know, like, proper load balancing.

This stuff has been figured out like 10 years ago buddy MtGox just don't know what they are doing.

[piramida]

How about anyone caught using a bot for macro ordering gets frozen/suspending pending investigation?

Are you guys insane? Just checking.

MtGox has a trading API just for that purpose. The problem is not automated trading, the problem is crappy trading engine and insufficient resources.

[sunblaster]

What! no SSDs yet, no wonder they crashing every-time someone wants to make a 14k sale!

Coming soon from US, just do OVERNIGHT EMS with no tracking!

[piramida]

The basic problem with clouds are that they are quite expensive when you have a high volume of traffic and that the servers are not that powerfull.

You have no idea what you are talking about. Seriously. You may have heard something on the internet but that does not qualify you for the opinion because you didn't understand anything from what you heard.

Computing resources allocated in the cloud can be much more powerful than anything you could ever build yourself. The cost is a variable between providers, but as a rule of thumb, the cost of running your own commercial datacenter eclipses by several orders of magnitude any cloud on-demand scaling application. You can of course continue running your servers from your basement as MtGox does for a fraction of a cost, but that has no properties required in a commercial application. Like, backup uplinks, reserve power, controlled access, staff available at all times, etc.

Banks and many large instituions use their own hardware setups for none of the reasons you mention, but because of the requirement to keep all customer data on-site.

And to non-linear scaling - yeah sure, that has been figured out 10 years ago as well. Of course it requires redesign for horizontal scalability, but it is always possible.

[crazy_rabbit]

From mtgox facebook: https://www.facebook.com/MtGox/posts/455962117821534

Code:

Hi everyone, just a quick update on the situation and what happened last night.

First of all we would like to reassure you but no we were not last night victim of a DDoS but instead victim of our own 
success!

Indeed the rather astonishing amount of new account opened in the last few days added to the existing one plus the 
number of trade made a huge impact on the overall system that started to lag. As expected in such situation people 
started to panic, started to sell Bitcoin in mass (Panic Sale) resulting in an increase of trade that ultimately froze 
the trade engine! 

To give you an idea of how impressive things were here are some numbers that we would love to share with you guys: 
- The number of trades executed triple in the last 24hrs. 
- The number of new account opened went from 60k for March alone to 75k new account created for the first few days
of April! We now have roughly 20,000 new accounts created each day.

Due to these facts we have been busy working on improving things since last week and our team has been working 
around the clock to improve Mt.Gox to catch up with the demand. We will continue to release several updates today 
and in the coming few days to improve our system overall performance.

Also please note that we may have to close the exchange for two hours in the next 12 to 24hrs to add several new 
servers to our system.

Thank you for your understanding and continuous support!

Holy Crap! 20,000 Accounts a day?!?!?!

[simplydt]

This doesn't make me feel any better about trading today tbh, just imagine another big hoarder deciding they will take $180 today because all the bear threads scare him. He/they sell of 20,000 bitcoins and we are back in panic sell mode. The next panic sale could easily cause another huge downward spiral.