Although the complexity involved with real time order matching is higher than the that of what an average programmer has to deal with, it is a solved problem. It is hard to scale and if this volume had been hitting a completely new exchange, this would probably have been forgiven. The issue is that MtGox has been running this exchange for a while with quite the revenue and liquidity to hire programmers that has been dealing with such algorithms for decades. The volume is not high enough to sink an in-memory algorithm of order matching, far from it.
Having said that, I do feel with MtGox as I believe they are trying to solve this issue. It might just be too late.
The main problem could be that AFAIK the trading engine is written in PHP. Now PHP is a fine language, as long as you don't care about speed or readable code