MTGOX was down on Jun 23. Now its back up and running.

[bb113]

Where is some acknowledgement from empty-gox?

Yes, they need to start explaining what the "techinical problem" was so that we can trust them.

[genuise]

and here is it real time water fall

[chsados]

<MagicalTux> someone called order/add without passing a currency couple in the new api (was not possible before, but since we standardized calls it became possible)
<MagicalTux> and order/add didn't check if it had a currency couple, since it was not possible to not pass one when it was created
<MagicalTux> not possible anymore, anyway
<MagicalTux> I'll get a clean announce made explaining what happened and how it won't happen again monday
<MagicalTux> (and maybe I should get myself a really loud pager for these specific cases)
<MagicalTux> We'll get an emergency update posted, main priority is to ensure there was no bigger issue than that and resume trading

[bb113]

passing a currency couple

What does this mean?

[chsados]

passing a currency couple

What does this mean?

from what i can tell it seems like there was some sort of an error in the system that set off a manual intervention....somehow someone set a buy order without any BTC amount attached to it.  very odd and am awaiting an official statement for gox

MagicalTux> looks like someone managed somehow to place an order to buy a couple of nothing
MagicalTux> there was a buy order without "BTC" set

[bitcoinBull]

<MagicalTux> someone called order/add without passing a currency couple in the new api (was not possible before, but since we standardized calls it became possible)
<MagicalTux> and order/add didn't check if it had a currency couple, since it was not possible to not pass one when it was created
<MagicalTux> not possible anymore, anyway
<MagicalTux> I'll get a clean announce made explaining what happened and how it won't happen again monday
<MagicalTux> (and maybe I should get myself a really loud pager for these specific cases)
<MagicalTux> We'll get an emergency update posted, main priority is to ensure there was no bigger issue than that and resume trading

Next time, launch new API's after waking up. Not before going to sleep!

[Herodes]

<MagicalTux> someone called order/add without passing a currency couple in the new api (was not possible before, but since we standardized calls it became possible)
<MagicalTux> and order/add didn't check if it had a currency couple, since it was not possible to not pass one when it was created
<MagicalTux> not possible anymore, anyway
<MagicalTux> I'll get a clean announce made explaining what happened and how it won't happen again monday
<MagicalTux> (and maybe I should get myself a really loud pager for these specific cases)
<MagicalTux> We'll get an emergency update posted, main priority is to ensure there was no bigger issue than that and resume trading

Next time, launch new API's after waking up. Not before going to sleep!

There's something called software testing as well, unit tests etc. You don't launch something and 'hope it works'.

[bpd]

Cut him a little slack. It's not like he botched the entire Facebook IPO like Nasdaq.

[Mt.Gox Support]

Hi everyone! Sorry for the little Glitch but everything is back in order. We will also would like to thanks everyone who contacted us to warm us in order to fix it.

Here you are the Official announcement regarding the freeze.

https://support.mtgox.com/entries/21616122-resolved-outage-30254-trading-unavailalable


[OUTAGE-30254 ] Trading Unavailalable
Hello Mt.Gox Users,

At approximately 23:00JST on June 23, 2012, trading became unavailable due to an invalid trade order which was not checked correctly by the Add Order API. The trading system however, detected this and halted itself to wait for further  instructions,  resulting in this outage. 
 
Trading is now back to normal.  We apologize for the inconvenience our users have suffered because of this outage and would like to thank you for your patience through this issue. If you have any further questions regarding this matter, please do not hesitate to contact Mt.Gox Support.

Mt.Gox Team

[mollison]

HUGE props to MtGox on the transparency here. I am really thankful that they were willing to explain the exact nature of the problem, so that we can all see what a big deal it is not.

Sure, downtime like this should not happen, and will only help competitors. Sure, somebody should be "on call" to fix these kinds of things. However, mistakes happen, we all know that, and the important thing is how one reacts to them.

[streblo]

Cut him a little slack. It's not like he botched the entire Facebook IPO like Nasdaq.

Should slack cutting be based on the fact that there are bigger fuckups than Mt Gox's downtime or on the fact that the downtime was easily preventable with the standard practice of unit tests? I think the latter.

[Astro]

Cut him a little slack. It's not like he botched the entire Facebook IPO like Nasdaq.

His slack was all used up when he leaked all of our emails and passwords.

[mollison]

Cut him a little slack. It's not like he botched the entire Facebook IPO like Nasdaq.

Should slack cutting be based on the fact that there are bigger fuckups than Mt Gox's downtime or on the fact that the downtime was easily preventable with the standard practice of unit tests? I think the latter.

For all you know, there could have been a subtle bug in the unit tests.

There are people who spend their lives on software verification and guess what: so far, there is no silver bullet.

[Bit_Happy]

Yeah, it's currently 05:31 in Japan.

So Mark's probably sleeping.

However, I have some viewpoints:

1. Mark need to trust 1 or two others to have admin access to MtGox. What if he gets hit by a bus ? Where's the backup solution ?
2. As MtGox is the biggest exchange, they should take great care to monitor all systems 24/7, if not they lose business, and trust is eroded.
3. There's a lot of businesses depending on MtGox being available, and therefore point 2.) is extra important.

MtGox has already clearly shown that they are not good at communication with the customers and PR work overall, so they should actually get help from somebody that is good at this.

In the end, I don't think there's anything to worry about, once Mark gets back, he'll probably kick all in working order again.

HOWEVER, this is simply not good enough in the corporate world of today. When Gox decided to run their exchange 24/7, that's also what they need to do in a serious way, then 'glitches' are not acceptable. And if it occurs, it should be fixed, and the community should get an honest and candid explanation of what happend, and it should not happen again.

If I were Mark, I'd let external experts review the code base to avoid future "glitches" and also have a financial audit of MtGox to regain trust in the community.

I have no doubt that Mark believes in bitcoin and work hard to make MtGox a good exchange, but he needs to see his own limitations, and to delegate work, he can't do it all himself.

Because of all the unrest these days (delayed payments etc), personally I don't hold any funds at MtGox, be it USD or BTC at the moment.

I would love to see MtGox succeed, but they have to up their game, as other professional players WILL move into this market.

One thing that I absolutely would love to see was better follow up of withdrawals. Customers actually invest their funds, and this is money that the customers own. When withdrawals take 30 days or more, these monies are being held 'hostage' and the customers needs to be notified about the status of their withdrawal in a much better way.

There should be absolutely no need for any customers coming running to this forum to resolve a withdrawal request.

Many people are too proud to admit to weaknesses and problems. IF there's USD liquidity problems at Gox, which many customers seems to believe, then it would be better if MtGox was honest about it, and told the community about the current situation, and made a plan to get out of that situation, perhaps raising fees temporarily and doing other measures to balance the books.

If it is so that only the customers that complains the loudest are the ones getting their cases solved, so that the Gox crew can avoid the fire, while the rest who wait in patience are the ones having to endure the painful waiting of their funds, it is a very unfortunate situation and while Gox is perhaps scared of telling how the situation is, perhaps they're afraid of losing customers. Not telling how things really is, could backfire pretty hard in the end. As we all know, the road to "hell" is paved with good intentions.

It's also very dangerous not to care about the customers, because once there are viable alternatives, customers will move to other exchanges. Being biggest today, doesn't mean you'll stay biggest forever.

Swift handling of customer funds is the single most important thing.

I've been away for several months and was expecting better. The BTC economy needs quality service from it's leading exchange.

[HorseRider]

It seems that the mtgox get some automated circuits, that's good sign.

[M4v3R]

It would be better if the automated circuits actually displayed a message on the site, like on Heroku:

"Our automated systems detected a potential problem and all trading has been halted. Our technicans have been notified and we're looking into this problem. We will provide more information as it arrives".

This way, there would be no panic and people would just patiently wait for the situation to be resolved.

[phelix]

a bug resulting in a halt is always possible in a system like this.

+1 for transparency.

-1 for taking too long to post a statement.

[Technomage]

Well, I told you so. The issue smelled like a technical glitch and that it was. Mt. Gox is still not the biggest company in the world and it happened at an unfortunate time, so I think it's fair to cut them some slack. However I hope they learn from this and make sure that in the future there is someone on hand to handle it without a 4 hour delay.

Testing might be an issue as well but testing complex systems in a fool proof way is not easy. Bugs can always happen, the most important thing to learn from this is that it just can't all be up to Mark. There needs to be someone online, with access, when he's asleep.

[casascius]

I take no issue with this problem or the way it was handled.

[TheHarbinger]

I take issue with having to wait to get money out of Mt.Gox.  There is ZERO reason a $200 withdraw should take a week or more.