Seriously what is behind the trading lag.

[ElectricMucus]

What are the technical reasons for it? And don't come with systematic overload.
This problem wasn't there last year or a few months ago for that matter. Neither of the other exchanges are suffering it and neither are any Forex sites and other Trading platforms in the world of professionals.

The size of the orderbook probably is in the magnitude of a few megabytes and the rate at which orders are processed stands in no relation to the lag. Please correct me if I am wrong.

Oh and one thing please: Hamsters, there I said it can we now get some (technical) debate going?

[0x11]

You may be willing to contact MagicalTux if you are looking for technical reasons.  Some of them are posted here: https://bitcointalk.org/index.php?topic=160666.0

[cedivad]

 It would be great if they published the faulty part of their trading engine, or a pseudo code of their trading engine.
So that we could try to replicate the problem.

This could have funny results.

[ElectricMucus]

Yeah I've read the "under funded orders" explanation. But it just doesn't make sense.

My problem is: Forex and Stock trading platforms operate in the sub-millisecond range, I don't know if you've seen the ted talk of how they drilled a tunnel through a mountain to shorten the path the signal has to travel.
So while the professional have systems performant enough that the speed of light becomes relevant we have issues to even keep up with price fluctuations.

[Capitan]

It's possible that they use an inefficient algo to process orders. Then, even if the queue is only a few megs, if the algo run time takes longer and longer depending on how many orders are in the queue, that could explain why the lag takes place. They upgraded to a server with SSD's, so they might have been bottlenecked by IO. If a part of their order processing algo relies on disk IO (obviously it does, since it needs to transactional and talk to a database), it could be that things were backing up.

The other exchanges aren't getting the same amount of traffic, and I'm not sure if they are also getting the same amount of fake micro-transactions that were allegedly bogging down MtGox.

Either way, I'm curious about this too, but I think the only people who can answer these Q's are at MtGox, and they would not divulge this info.

[ElectricMucus]

It's possible that they use an inefficient algo to process orders. Then, even if the queue is only a few megs, if the algo run time takes longer and longer depending on how many orders are in the queue, that could explain why the lag takes place. They upgraded to a server with SSD's, so they might have been bottlenecked by IO. If a part of their order processing algo relies on disk IO (obviously it does, since it needs to transactional and talk to a database), it could be that things were backing up.

The other exchanges aren't getting the same amount of traffic, and I'm not sure if they are also getting the same amount of fake micro-transactions that were allegedly bogging down MtGox.

Either way, I'm curious about this too, but I think the only people who can answer these Q's are at MtGox, and they would not divulge this info.

What data are they storing? Orders what else? How can that ever amount to any significant amount of I/O bandwidth?
They would have to read and rewrite the entire data in the orderbook on each order to come even close to this amount. And it would make absolutely no sense to do it.

[repentance]

Yeah I've read the "under funded orders" explanation. But it just doesn't make sense.

My problem is: Forex and Stock trading platforms operate in the sub-millisecond range, I don't know if you've seen the ted talk of how they drilled a tunnel through a mountain to shorten the path the signal has to travel.
So while the professional have systems performant enough that the speed of light becomes relevant we have issues to even keep up with price fluctuations.

You just answered your own question.  "The professionals" have hundreds of millions of dollars invested in the infrastructure which makes real time trading possible.  Not only can no individual Bitcoin exchange afford such an investment, all of the Bitcoin exchanges combined would not have the resources to put such high level infrastructure in place at the moment. I doubt they could even afford to rent someone else's infrastructure.

[ElectricMucus]

Yeah I've read the "under funded orders" explanation. But it just doesn't make sense.

My problem is: Forex and Stock trading platforms operate in the sub-millisecond range, I don't know if you've seen the ted talk of how they drilled a tunnel through a mountain to shorten the path the signal has to travel.
So while the professional have systems performant enough that the speed of light becomes relevant we have issues to even keep up with price fluctuations.

You just answered your own question.  "The professionals" have hundreds of millions of dollars invested in the infrastructure which makes real time trading possible.  Not only can no individual Bitcoin exchange afford such an investment, all of the Bitcoin exchanges combined would not have the resources to put such high level infrastructure in place at the moment. I doubt they could even afford to rent someone else's infrastructure.

What the fuck?

They also cook with water. The cost of computer hardware is negligible, both in the world of finance and the amount of profit mtgox had during the last year.
What performance? There were computerized exchanges in the 90s dammit.

Second: Look at the issues I raised above: Which part do you think requires high performance computing by todays standards?

[Capitan]

I agree with you that they should be able to do it with a reasonably powerful computer -- one that could be purchased for under $20K. This is why I think their system is just inefficient. as I said in the above post, you can easily write an inefficient algorithm that just process orders really slowly, and gets exponentially slower as the order book grows.

I don't know for sure that this is what their problem was though. Just speculating. If you are interested in how to implement a high performance one, look here, these guys are talking about creating an open source high performance engine that should be able to process thousands to millions of orders per second: http://www.reddit.com/r/Bitcoin/comments/1c7v6z/buttercoin_open_source_highperformance_bitcoin/

[ElectricMucus]

I agree with you that they should be able to do it with a reasonably powerful computer -- one that could be purchased for under $20K. This is why I think their system is just inefficient. as I said in the above post, you can easily write an inefficient algorithm that just process orders really slowly, and gets exponentially slower as the order book grows.

I don't know for sure that this is what their problem was though. Just speculating. If you are interested in how to implement a high performance one, look here, these guys are talking about creating an open source high performance engine that should be able to process thousands to millions of orders per second: http://www.reddit.com/r/Bitcoin/comments/1c7v6z/buttercoin_open_source_highperformance_bitcoin/

Yeah inefficient code is the obvious conclusion.

Thanks for the link, lol I know this guy the world is small  

[repentance]

What the fuck?

They also cook with water. The cost of computer hardware is negligible, both in the world of finance and the amount of profit mtgox had during the last year.
What performance? There were computerized exchanges in the 90s dammit.

Second: Look at the issues I raised above: Which part do you think requires high performance computing by todays standards?

This isn't just a hardware problem, though.

There's a poster in one of the other threads who has worked on the systems used by real world stock exchanges.  It would be interesting to see if he can estimate what it would cost MtGox to create the infrastructure to solve this problem in a way which allows them to scale and become a serious trading platform.

MtGox was never designed to be a serious trading platform and they need to make up their minds whether they want to become one or whether it's more profitable for them to serve a niche market than to try to be all things to all people.  They're not designed to service serious traders (who want all kinds of trading options MtGox doesn't offer) and there are a couple of other exchanges close to launch who are pitching at that market. 

MtGox needs to decide what it wants to be and then re-invent itself to serve its chosen market.  Slapping on bandaids without any clear sense of direction isn't going to help MtGox or its customers.

[moni3z]

Why reinvent the wheel. When you send bank wires into these forex companies this is what happens: bank notification money is there now internal USD balance is credited. Why can't a trusted 3rd party company act as the bitcoin bank. It handles the transaction and notifies when bitcoins are there, FX company credits your internal BTC balance. They just need a partner to handle the safe multisig cold wallet for them. Coinlab has enough clout to do this, or go make a startup.

The forex multi million dollar trading platform admins then go into settings and add 'BTC' and it's done. Traded just like USD, EUR, TRY ect. When you withdraw, instead of asking the bank to send you a wire they ask the bitcoin cold wallet operator to send the coins.

[TimJBenham]

What the fuck?

They also cook with water. The cost of computer hardware is negligible, both in the world of finance and the amount of profit mtgox had during the last year.
What performance? There were computerized exchanges in the 90s dammit.

Second: Look at the issues I raised above: Which part do you think requires high performance computing by todays standards?

This isn't just a hardware problem, though.

There's a poster in one of the other threads who has worked on the systems used by real world stock exchanges.  It would be interesting to see if he can estimate what it would cost MtGox to create the infrastructure to solve this problem in a way which allows them to scale and become a serious trading platform.

MtGox was never designed to be a serious trading platform and they need to make up their minds whether they want to become one or whether it's more profitable for them to serve a niche market than to try to be all things to all people.  They're not designed to service serious traders (who want all kinds of trading options MtGox doesn't offer) and there are a couple of other exchanges close to launch who are pitching at that market. 

MtGox needs to decide what it wants to be and then re-invent itself to serve its chosen market.  Slapping on bandaids without any clear sense of direction isn't going to help MtGox or its customers.

If it wants to call itself an exchange then it should act like one. The core business of an exchange is reliably matching bids and offers. At the moment it is not doing this. Exotic trading options are not necessary. They can be added by third party providers via an API. Trading platforms are a layer above exchanges.

TBF to MtGox none of the exchanges I have seen looks properly done. Mostly they look like something thrown together by a programmer who knows little about markets and trading.

The trading volume at MtGox is trivial compared to Betfair. Betfair's infrastructure can handle one million transactions per second (tps). They are a gambling market, not a zillion dollar international fraud jew bankster cartel illuminati operation. You could handle 100 tps easy on a PC with decent software.

If robot traders spamming the orderbook is a problem then the solution is easy. Give each user a number of free bids and offers per day (say 100) and charge a small fee for each one thereafter. Spamming goes away as soon as it costs the spammer something.