Bitcoin Forum

From mtgox facebook: https://www.facebook.com/MtGox/posts/455962117821534 (https://www.facebook.com/MtGox/posts/455962117821534)

I think this is good news. Not a DOS and them adding servers will alleviate fears that this could happen again.

As I said in another thread, they need to get 10 times more servers than they think they need. Gox never seems to be up to par, its like they are ALWAYS playing catch up.

Great News.

Let the Rally Begin (again) !

HAHAH!

Lol, I think they need more than just a "few" new servers.

Remember that during GoxLag, people panic and sell massive amounts of coin - resulting in higher profits for Gox than a normal day of trading. Their incentive to improve capacity is limited to the threat of people leaving Gox due to the lag. And it looks like not very many people are leaving... big resounding LOL. Everyone's used to the granny nanny socialist state where regulatory bodies attempt to protect them from poorly performing businesses. They don't do their research on gox... it takes like 2 seconds of lurking here to realize that Gox Sux.

Get out of Gox! They're going to be playing "catch up" forever until we send them a powerful message.

I've noticed the lags and price drops every day about the same time. Perhaps it would be a good idea to go offline for an hour of maintenance every day to refresh and update everything.

They did. In fact, they had much more than that. As they said, in just the last day the number of trades tripled! It's truly amazing that they scaled as well as they did.

So they got 10 times more servers than then think they need? And then ten times more after that?

You're telling me THAT happened because of normal trading volume?

Geezus.

Quite Bullish I'd say.The Gox also has 19,000 more accounts to verify so I think this is going to be an interesting ride.

Have they not heard of Amazon EC2? Google Compute Engine? Heroku? Azure?!

"Adding more servers" - what is it, the 90s? Amateurs..

We've learnt that you can never add enough servers to cope with an unpredictable and quickly-changing demand curve. That's why they invented elastic computing. They add 3 servers now, next month those 3 servers can't handle. So they add 3 more, and so on and so on.

It is so ironic, bitcoin is supposed to be the new new thing, but some infrastructure is so "old"(No flexible cloud based server)
But since bitcoin is about security, they may have some concern about the cloud.

I would have more sympathy if this was not part of a historic pattern of lackadaisical malfeasance on their part.  Gox has sucked for as long as I can remember and has absolutely NO incentive to improve.

Why yes, they have. They purchased Bitomat after it collapsed because the servers crashed and Amazon couldn't do data recovery because it was in the cloud. So yes, they are well aware of cloud services.

I'm not even sure where to start.

You think that if a server crashes 'in the cloud' that the data is unrecoverable?

You don't think that the particular people at Bitomat might have just not known what they were doing?

'In the cloud' isn't some magical meta-space where things don't exist and the data is unrecoverable, it just means the machines are somewhere else and you can programatically request more of them, for you know, like, proper load balancing.

This stuff has been figured out like 10 years ago buddy MtGox just don't know what they are doing.

I think DoS'ing them would be for their own good, since they clearly can't understand how to stress test themselves.

Gox should move out of the garage and start becoming professional. They still have enormous potential despite being the largest already.

https://bitcointalk.org/index.php?topic=172991.0;topicseen

Right on comrade! Bitstamp here we come, really blown away by the professionalism of that operation compared to Mt Gox

Even Apple had to do that eventually :)
http://johnsciacca.webs.com/Epcot%20Apple%20Garage.jpg

From http://www.pcworld.com/article/2033809/bitcoin-volatility-caused-by-surge-in-demand-slow-software.html (http://www.pcworld.com/article/2033809/bitcoin-volatility-caused-by-surge-in-demand-slow-software.html)

Bitcoin's roller-coaster price swings on Wednesday were caused by an influx of new buyers and software that couldn't keep up, according to the largest exchange, Mt. Gox.

People trading bitcoin on the Tokyo-based exchange saw its price hit US$266 at one point, then fall to as low as $105 with high lag times for trades. The surge in interest in bitcoin has overwhelmed Mt. Gox.

"We are running like crazy," said Gonzague Gay-Bouchery, who heads marketing for Mt. Gox, on Thursday in an interview. "We are trying to do everything by the book and trying to make everything work smoothly."

Mt. Gox and other exchanges that allow people to buy bitcoin all appeared to have varying degrees of problems on Wednesday, illustrating how small Internet companies often grapple with scaling their services amid rising interest.

Bitcoin, a virtual currency that is transferred by a peer-to-peer network, has been propelled in recent weeks by mainstream media coverage, although the number of transactions remains very small compared to traditional financial networks.

In just two days earlier this month, Mt. Gox gained 75,000 new users, and some 20,000 new ones in the last three days, Gay-Bouchery said. As soon as Mt. Gox credits their accounts with cash, those users jump into trading. Mt. Gox's system has had trouble processing those trades quickly, which has lead to vast price swings as users try to quickly complete trades and anticipate price moves.

"Bitcoin will go up and down," Gay-Bouchery said. "If there is a lag, don't panic."

Last week, Mt. Gox said it fought a massive distributed denial-of-service (DDoS) attack intended to manipulate the price of bitcoin. Although Mt. Gox's exchange is nearly always under a DDoS attack, Gay-Bouchery said Wednesday's issues were not related to DDoS.

Mt. Gox has spent an "enormous" amount of money on new servers for its bitcoin trading engine. Already, the exchange uses servers with 64GB of RAM each, but it hasn't been enough to keep the site running perfectly. The new servers, which have solid-state drives for faster data input and output, are due to arrive from the U.S. soon, he said.

The exchange has also been recently contacted by major server manufacturers who supply technology to the largest banks in the world, Gay-Bouchery said. The companies said they can solve Mt. Gox's trading woes.

The code for Mt. Gox's trading platform would have to be changed to accommodate new hardware, but it would give the exchange a trading platform comparable to that of a bank, lending more stability, Gay-Bouchery said.

Other exchanges that sell bitcoin were also groaning under the strain. Bit Innovate in Australia wrote on Twitter that is website was under stress due to increased demand. On Wednesday, it appeared at one point to simply run out of bitcoins to sell.

"Due to extremely high demand, we have had to temporarily close our Buy Bitcoins service," Bit Innovate wrote. "Sourcing more. It will be open as soon possible."

BitFloor, the largest U.S. bitcoin exchange based in New York, warned of delays and technical issues with its website. Roman Shtylman, the founder of BitFloor, said his exchange saw its highest trading volume and highest number of users ever.

Shtylman said he was working on some changes to ensure smooth trading following Wednesday's frenzy.

yeah, right, enormous  ::)

This is not good for bitcoin. What will gox do about causing the "correction".

What a huge central point of failure for bitcoin!

Overall, I think this is a good thing: people have gotten an interest in bitcoin.

At what cost?

So, a few sentences about this issue.

* A drop from 266 to 105 will make headlines all over! Result: Bitcoin publicity
* Knowing that the issue was cause by a huge amount of new accounts Result: BULL
* Crashing the market. Result: MtGox making a shit ton of money

It's a Win-Win for everyone except the weak hands!

MtGox makes money during both panic buy and panic sell, cause of fee for each transaction. The more buying and selling, regardless of reason, is fee profits!

MtGox wins!

Exactly, but during these kinds of sell-offs... the volume goes to the roof!

+1 that; either that, or, in case their regulations insist they run in a closed-doors DC, then they MUST make sure that their normal trading volume is not loading any part of the system higher than 10%. 3x increase in volume should be absolutely normal and should not cause even a slightest delay.

Amateurs indeed. Bitstamp has heavily lagged as well, the only exchange operating normally was btc-e.

They have some weird policy in place of we add more servers when we need them. Reacting to market demand rather than as you suggest predicting it and playing ahead of it. Sucks. It just doesn't work.

https://i.imgur.com/rJOKsK8.jpg
https://i.imgur.com/TTfPVyd.jpg

ask em about that!!! Macro order flooding  via bots that trap out regular customer but computer systems dont get trapped out in a panic and they can drive it down via 50 a pop or smaller!! happens every-time is there is a panic! Like someone is manipulating !! Doesn't care about $ its micro manipulate and watch it tank! (look at the dam pics!!)

 Gox needs to filter out micro or macro trades or do something! Thats whats cause's the wild drops and panic !! Probably banks with some bots and endless fiat money!  Needs to be addressed!


Drive it down via those small trades and watch the herd jump ship ! Since there is no cooling period or auto shut off its disastrous! !

Thats whats gonna kill it!  We need a % slide rule and cooling period put in like the real wall-street trading platforms do!

If it falls by x amount they stop trading for x amount of time! "Cooling period"  Hours .. till the next day! Whatever .. something is better than nothing!




At this rate in this setup.. a foreign entity with negative intention's towards bitcoin could micro trades us to our deaths!

How about anyone caught using a bot for macro ordering gets frozen/suspending pending investigation?

The basic problem with clouds are that they are quite expensive when you have a high volume of traffic and that the servers are not that powerfull.
You end up renting more cloudservers than you would if you used physical servers. Which means that you have to scale your operation wider, and as
always with scaling as the number of servers grow the extra power you get from adding an extra server decreases.

For some compute intensive systems (like mining/hashing) the power decrease very slowly, however for something where data has to be shared the curve can be quite drastic.

Some software (and im not talking hobby coders but software from the 10 original software companies) that has negative scaling after a certain amount of parallelization. I.e. the
time INCREASES when you add more servers. It always has to do with the basic algorithm and then there are 2 options, redesign algorithm or hardware scaleup.
Algorithm design is extremely expensive and not always even possible (?). Hardware scaleup is much easier when you have your own machines and you DEFINATELY do not want to
run the stuff in virtual machines!!!

There is a reason why "real" banks use specialized hardware.


Well now i have to get back to work.
//GoK

Are you guys insane? Just checking.

MtGox has a trading API just for that purpose. The problem is not automated trading, the problem is crappy trading engine and insufficient resources.

What! no SSDs yet, no wonder they crashing every-time someone wants to make a 14k sale!

Coming soon from US, just do OVERNIGHT EMS with no tracking!

You have no idea what you are talking about. Seriously. You may have heard something on the internet but that does not qualify you for the opinion because you didn't understand anything from what you heard.

Computing resources allocated in the cloud can be much more powerful than anything you could ever build yourself. The cost is a variable between providers, but as a rule of thumb, the cost of running your own commercial datacenter eclipses by several orders of magnitude any cloud on-demand scaling application. You can of course continue running your servers from your basement as MtGox does for a fraction of a cost, but that has no properties required in a commercial application. Like, backup uplinks, reserve power, controlled access, staff available at all times, etc.

Banks and many large instituions use their own hardware setups for none of the reasons you mention, but because of the requirement to keep all customer data on-site.

And to non-linear scaling - yeah sure, that has been figured out 10 years ago as well. Of course it requires redesign for horizontal scalability, but it is always possible.

Holy Crap! 20,000 Accounts a day?!?!?!

This doesn't make me feel any better about trading today tbh, just imagine another big hoarder deciding they will take $180 today because all the bear threads scare him. He/they sell of 20,000 bitcoins and we are back in panic sell mode. The next panic sale could easily cause another huge downward spiral.

Mt. Gox is the achilles heel of bitcoin.  Unfortunately it seems the only thing that will wake Mt. Gox and it's users up to this is a lot of bad tasting medicine.

that number does not make much sense..    to me;. gox queue in total can be this big. but not on a daily basis..   I know people waiting for days now so and they are around number 10k..      20k a day.. sounds a bit far stretched...  to me..  

Aren't people in US allowed to start trading small volumes without verifying?
I think verification is only mandatory for Europeans, but I'm not sure.

If you look at their trade feed, there are barely a few trades per minute in times of high lag. I still don't understand how they can capitulate under what is essentially so little volume.

The engine isn't even capable of >one trade per second most times, Nasdaq does at least 35,000 per second and is tested up to 60,000.

http://www.nasdaq.com/services/homw.stm

The order of magnitude difference doesn't make sense to me. Sure, some is to be expected, but not that much.

Have you considered the glut of orders is preventing execution, thus our noticing it in the first place? I'd say that's the root of the whole problem. If they had all their trades going through we wouldn't be worried.

You don't have to get verefied to sign up and play with small amounts. 20K new users would be possible, I'm sure that verification que will skyrocket soon.

I hope a lot of Mtgox users saw this FB message and will move to another exchange... more balanced traffic on different exchanges would be healthier for Bitcoin.

Agreed. It's good for Bitcoin.

The micro trading bots spamming are interesting. Very important not to react like the USSA and ban such things but to react to them in a fluid market response. Established markets also have this problem of spamming - I expect this community to be better than those trillion pound markets in the response.

If it's 20k accounts a day then we have to presume that KYC is outsourced to a specialist company for this? Or is it in someone's garage handling passport scans and bank statements?

LOL, I think that Maged surely know this stuff  :D But anyway did you heard of Linode?

What's it going to take before the Bitcoin community gets together and creates a viable decentralized exchange to replace Mtgox?

How loud do we need to scream it!  Get a decentralized exchange platform going.  Please, please, those with the technical know-how, lets get the discussion started.

I am sure mtgox earning a decent amount of money since the bitcoin price rise. They should spend them all to upgrade the infrastructure and trading platform to prevent any problem. Bitcoin trades 24/7 and requires a high end reliable system. I was very disappointed with the service yesterday

They should get rid of the tiny trades or charge more for anything less than a certain limit.

Thousands of tiny orders are not required anymore and they're annoying.

Does anyone know of a good exchange? BitStamp lagged like hell as well, and I heard it's not the only one. (well, even the forum and bitcoincharts did)

Really, we read every day about the wonders offered to today's computing by ever progressing cloud/virtualization/whatnot technologies -and the only ones that know not how to properly use them are the ones managing our money?

As much as I think Gox has some very serious issues and is currently unfit for purpose (clearly they have some horrible legacy and crazy architecture to content with - e.g. I heard someone mention the web front is not decoupled from the trading engine - WTF?!) it just not that simple.

Anybody who knows about how to build fast highly secure exchanges that have to guarantee perfect consistency with regards to state/transactions would think twice about using any sort of public virtualised infrastructure. Physical hardware is still the norm in this domain because raw single threaded clock speed and fast IO (e.g. solid state drives) often matters a great deal for exchanges and trading platforms. To give you to example I have direct knowledge of: I worked at Betfair in the past and they were considering building bespoke hardware chips to run the code of their core exchange engine to ensure they could scale their platform for the future and I know a friend at Merrill Lynch who told me about their servers which ran in oil baths to enable over-clocking raw CPU speed matters. There are many other example of custom hardware in use by banks and the like. To think anyone would use Heroku for an exchange is laughable. And I speak as someone who choose to build an app on Heroku or any other PaaS/IaaS provider 99% of the time. And I didn't even mention security.

What I do hold Gox accountable for though is not investing in their platform a long time ago. I can't believe they haven't had the money to build an entirely new trading engine and hire a really good team to build and support it. They have been very short-sighted. Any good start-up has a plan in place to scale if they start to experience exponential growth.

I think we should be using some of our Bitcoins to fund the building worthy alternative - I certainty don't hold much faith in any of the others so far. An exchange for bitcoiners, built by bitcoiners.

elastic computing service might be too expensive for mt.gox?

There are some excellent alternatives to Mt. Gox that already exist: bitstamp, campbx, bitfloor.  But most of you all won't switch or strongly suggest that your friends/family go to one of these alternatives.  Why in the world would we with technical know-how want to spend time building yet another exchange when it's clear most people just go with the best-known name.

Let me take a guess... PHP and MySQL??

Seems like they may have a bottleneck somewhere that adding servers won't fix. High probability that it's in the database, which is pretty hard to scale by just adding servers. But i don't see how the amount of traffic they get should be this much of a problem for them. They should be  able buy a powerful enough database server with one or two day's worth of the profits they make (assuming the $100K+ per day profit I saw posted in another thread is true). I've worked on websites that got 1M+ hits per day in traffic spikes, not quite the same type of app as an online exchange, but I know a little bit about performance and scaling.

For the amount of money they make, and the amount of traffic they do, there is simply no excuse. Their public statements don't instill confidence. First they said "you guys have no idea what it's like, come work with us for a day, we can handle DoS attacks, where withstood two at the same time without any lag", then the next day they are down after the maintenance because of a DoS attack.

+1

The drop and the publicity about it could also dampen the public interest.

No, the panic selling began well before the lag. This is an absurd statement. "As expected"?? No, I do not expect that new accounts on a trading platform would devastate a currency. And I don't believe it did. They use this lag as an excuse for any sell-off. We were going down well before this sell-off started. The lag actually slowed the sell-off down.

IMO This is an absurd pump....basically saying 'Don't worry, we have tons of new buyers coming'. Can you imagine a broker in any other asset class making a statement like this?

In another statement during the previous minor sell-off (not from yesterday but April 3rd) -- 'I understand that many of you have a lot at stake here, but remember that Bitcoin, despite being designed to have its value increase over time, will always be the victim of people trying to abuse the system, or even the value of Bitcoin decreasing occasionally. These are not new phenomena and have been present since the beginning of time when humans first started trading'.

Can you imagine a brokerage in the US being able to say that the underlying is 'designed to increase in value' or that it's value 'decreases occasionally'? They couldn't because it would be illegal.

Twitter: 'Trading is suspended until 2013-04-12 02:00am UTC for market cooldown. Once back trading will be also faster.'

And now we're halted. I question their timing. We were in a slow sell-off at a key psychological support level (yesterday's low). We were not in a panic sell today, so why halt? They could have halted yesterday. Yes, they need to upgrade their servers....but I think it's interesting that they chose a key psychological support level to halt and not when it was trading higher today.

It is not in Mtgox interest to stop a sell off.  They make a more in a few hours of panic selling then they do in weeks of regular trading.

You really think they would try to stop a sell off? 

Their infer-structure simply cannot handle the demand.  Plain and simple.

The market carried on without gox. When it reopened the adjustment was in line with the other exchanges - a good sign?