mtgox servers

[paulie_w]

i was just looking at this:

https://mtgox.com/img/pdf/20120831/Transparency.008.jpg

does anyone else think that this server setup is way overkill for gox?

maybe i underestimate its traffic.

[DeathAndTaxes]

Computer nerds building a financnial service business?  Buys cool uber computers?

[mccorvic]

No such thing as overkill when it comes to servers!

[slush]

No such thing as overkill when it comes to PHP!

[server]

Raid1 is strange... this should be raid5 or 10

2 Gb nic's isn't really overkill, pretty standard.

Some SSD's would speed the whole thing up.

[mc_lovin]

I wonder what % of utilization all that hardware is at?

$5000 monthly cost?  For the backup service?  Seems..  bit taxed.

[John (John K.)]

i was just looking at this:

https://mtgox.com/img/pdf/20120831/Transparency.008.jpg

does anyone else think that this server setup is way overkill for gox?

maybe i underestimate its traffic.

There's got to be some backbone to MtGox's servers; they're always getting DDOS'ed.  If I'm not wrong MagicalTux runs Kalyhost too, so he probably could get more bang for his buck.

[thebaron]

Well, figure they should be using at least two layers of encryption (database and Bitcoin keys), so it's probably more taxing then you think.

[Bendur]

Yeah if anything MtGox needs even better servers, or at least better ddos protection. They seem to get knocked off far too often

[MPOE-PR]

i was just looking at this:

https://mtgox.com/img/pdf/20120831/Transparency.008.jpg

does anyone else think that this server setup is way overkill for gox?

maybe i underestimate its traffic.

It's not.

[Deafboy]

If you are not looking for HW on ebay and for the cheapest possible reseller of server housing services, then the price seems to be ok.

[cedivad]

5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!

[DeathAndTaxes]

5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!

RAM prices have really fallen in the last 18 months (more than the usually 50% drop every 2 years).

It is strange they wouldn't buy and colocate though.  Especially for that high end.  Paying $5K to $10K for the hardware and then cutting your monthly cost to <$1K per month would for a private cabinet would have paid off a long time ago.  Then again maybe Japan has some insanely expensive datacenter costs no matter which way you go?

The specs don't look overkill except maybe the 32 cores.  Nothing they do is that CPU intensive.     Lots of memory and some fast disks makes sense given the db centric nature.  I am surprised they didn't seperate it out into db server and web/front end server though.  Still I doubt they are getting much utilization of CPU side.  If they are it would be cheaper to go with some high end NICs (TCP/IP offloading) and a HSA then trying to bang it all out with expensive CPU cores.

[paulie_w]

isn't bitcoin itself rather cpu intensive?

[TangibleCryptography]

isn't bitcoin itself rather cpu intensive?

Not really.  I mean not 32 high end server cores intensive.   We aren't talking mining.  Bitcoind's (i.e. running a node for the backend) bottlenecks are I/O & memory.  You got me curious so I checked.  Peak CPU time on our bitcoind process is ~2% (excluding when it was bootstrapping which was more like 4%) and that is on a Single CPU (quad core xeon).

Even if bitcoind was CPU intensive the current codebase is single threaded so you aren't going to use more than 1 core anyways (so the 2% is more like 8% of a single core).   If you are running on a atom based netbook or a oversubscribed VPS instance you might have problems but any dedicated hardware won't blink.

[MPOE-PR]

5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!

It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month.

[SgtSpike]

5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus!

It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month.

I want to ignore you, but every once in a while, you do post something moderately useful.

I agree.  2GB/s of WAN data is spendy, no matter where you are.  It could very well be the bulk of the cost.

[cedivad]

I tough that to be a backend database server, inaccessible from the outside, behind a pool of frontend servers... Still, 2Gbps of dedicated bandwidth to handle DDOS? It's not gonna scale, after all...
But who am i to criticize their setup?

Edit, the 2Gbps uplink might be internal; they do have an external DDOS protection service.

traceroute to mtgox.com (72.52.5.67), 64 hops max, 52 byte packets
 1  10.0.0.10 (10.0.0.10)  0.468 ms  0.383 ms  0.248 ms
 2  *  15.757 ms  15.461 ms  15.779 ms
 3  wf2.mi.sw-gw1.seflow.it (158.58.168.1)  14.648 ms  19.250 ms  16.862 ms
 4  caldera.dc2.hsr1a.seflow.it (95.141.47.254)  45.370 ms  172.757 ms  223.565 ms
 5  ge4-12.mil01-1.eu.as5580.net (78.152.32.201)  34.848 ms  17.582 ms  21.405 ms
 6  tge1-3.par02-1.fr.as5580.net (78.152.34.109)  36.162 ms  41.075 ms  34.076 ms
 7  tge1-1.lon01-1.uk.as5580.net (78.152.34.74)  50.626 ms  42.321 ms  47.633 ms
 8  blackhole.prolexic.com (195.66.224.31)  43.664 ms  44.109 ms  43.942 ms
 9  unknown.prolexic.com (209.200.156.34)  43.583 ms  43.484 ms  43.788 ms
10  unknown.prolexic.com (72.52.5.67)  43.222 ms  43.045 ms  43.080 ms

[paulie_w]

is it really for the DDOS protection?

[mc_lovin]

So...  does it seem to anyone that their hardware is extremely underutilized and their software is inefficient?

4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second.