Bitcoin Forum

i was just looking at this:

https://mtgox.com/img/pdf/20120831/Transparency.008.jpg

does anyone else think that this server setup is way overkill for gox?

maybe i underestimate its traffic.

Computer nerds building a financnial service business?  Buys cool uber computers?

No such thing as overkill when it comes to servers!

No such thing as overkill when it comes to PHP!

Raid1 is strange... this should be raid5 or 10

2 Gb nic's isn't really overkill, pretty standard.

Some SSD's would speed the whole thing up.

I wonder what % of utilization all that hardware is at?

$5000 monthly cost?  For the backup service?  Seems..  bit taxed.

There's got to be some backbone to MtGox's servers; they're always getting DDOS'ed.  If I'm not wrong MagicalTux runs Kalyhost too, so he probably could get more bang for his buck.

Well, figure they should be using at least two layers of encryption (database and Bitcoin keys), so it's probably more taxing then you think.

Yeah if anything MtGox needs even better servers, or at least better ddos protection. They seem to get knocked off far too often

It's not.

If you are not looking for HW on ebay and for the cheapest possible reseller of server housing services, then the price seems to be ok. :)

5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!

RAM prices have really fallen in the last 18 months (more than the usually 50% drop every 2 years).

It is strange they wouldn't buy and colocate though.  Especially for that high end.  Paying $5K to $10K for the hardware and then cutting your monthly cost to <$1K per month would for a private cabinet would have paid off a long time ago.  Then again maybe Japan has some insanely expensive datacenter costs no matter which way you go?

The specs don't look overkill except maybe the 32 cores.  Nothing they do is that CPU intensive.     Lots of memory and some fast disks makes sense given the db centric nature.  I am surprised they didn't seperate it out into db server and web/front end server though.  Still I doubt they are getting much utilization of CPU side.  If they are it would be cheaper to go with some high end NICs (TCP/IP offloading) and a HSA then trying to bang it all out with expensive CPU cores.

isn't bitcoin itself rather cpu intensive?

Not really.  I mean not 32 high end server cores intensive.   We aren't talking mining.  Bitcoind's (i.e. running a node for the backend) bottlenecks are I/O & memory.  You got me curious so I checked.  Peak CPU time on our bitcoind process is ~2% (excluding when it was bootstrapping which was more like 4%) and that is on a Single CPU (quad core xeon).

Even if bitcoind was CPU intensive the current codebase is single threaded so you aren't going to use more than 1 core anyways (so the 2% is more like 8% of a single core).   If you are running on a atom based netbook or a oversubscribed VPS instance you might have problems but any dedicated hardware won't blink.

It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month.

I want to ignore you, but every once in a while, you do post something moderately useful.

I agree.  2GB/s of WAN data is spendy, no matter where you are.  It could very well be the bulk of the cost.

I tough that to be a backend database server, inaccessible from the outside, behind a pool of frontend servers... Still, 2Gbps of dedicated bandwidth to handle DDOS? It's not gonna scale, after all...
But who am i to criticize their setup?

Edit, the 2Gbps uplink might be internal; they do have an external DDOS protection service. :)

is it really for the DDOS protection?

So...  does it seem to anyone that their hardware is extremely underutilized and their software is inefficient?

4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second. 

LOL Raid 1 on their backup data server?  WTF?  Raid 1???  Even Raid 5 is no good...   Spend less on equipment, and more on someone that knows what they are doing.

Yes it likely should be sufficient to handle 100x that, assuming the single box was used as dedicated trading engine with other servers used to offload other tasks (API access, webserver, support, charting, etc).

You can't run an exchange as a single-stack application. No way it will scale.

The fact that they are still trying to throw hardware at this means they simply do not understand scaling.

They will not solve these problems because they don't have a clue how to solve them


RUN AWAY, don't just walk away.

hmm looked under kill to me by far

The database on the main server should be entirely on SSD.  It's small enough.

RAID1 is appropriate, as disks are cheap, and they really aren't using much space.  Downtime is more expensive than disks.

There should be a secondary server with a copy of the database replicated in real time just for handling read only transactions, like clarkmoody.com, bitcoin itty.org, etc.  And a backup for it, too.  The main server is obviously still overloaded.

They are underconfigured andneed to stop thinking smalltime!  

But most important, they need to stop th spambots from choking the system with .001 BTC orders.  An easy way to do that is set a minimum fee per trade of .006 BTC.  This would only impact the nuisance orders, since any reasonable person is trading at least 1 BTC.

If your systems are behind Nic cards that are 2gbps and you are being hit by a DDos of 10gbps.. how are you supposed to protect from that?  That size DDos will just halt all traffic just from the bottleneck effect.

Actually what I meant to say is that their main line into their server room(s) has to be a 1000gbps  capable line.. and they need a minimum of 10gbps bandwidth to survive . their servers behind whatever protection they come up with can be 2gbps  .. no problem.