Title: mtgox servers Post by: paulie_w on December 03, 2012, 04:02:45 PM i was just looking at this:
https://mtgox.com/img/pdf/20120831/Transparency.008.jpg does anyone else think that this server setup is way overkill for gox? maybe i underestimate its traffic. Title: Re: mtgox servers Post by: DeathAndTaxes on December 03, 2012, 04:03:48 PM Computer nerds building a financnial service business? Buys cool uber computers?
Title: Re: mtgox servers Post by: mccorvic on December 03, 2012, 04:08:08 PM No such thing as overkill when it comes to servers!
Title: Re: mtgox servers Post by: slush on December 03, 2012, 04:18:47 PM No such thing as overkill when it comes to PHP!
Title: Re: mtgox servers Post by: server on December 03, 2012, 04:52:44 PM Raid1 is strange... this should be raid5 or 10
2 Gb nic's isn't really overkill, pretty standard. Some SSD's would speed the whole thing up. Title: Re: mtgox servers Post by: mc_lovin on December 20, 2012, 08:50:09 AM I wonder what % of utilization all that hardware is at?
$5000 monthly cost? For the backup service? Seems.. bit taxed. Title: Re: mtgox servers Post by: John (John K.) on December 20, 2012, 08:56:11 AM i was just looking at this: There's got to be some backbone to MtGox's servers; they're always getting DDOS'ed. If I'm not wrong MagicalTux runs Kalyhost too, so he probably could get more bang for his buck.https://mtgox.com/img/pdf/20120831/Transparency.008.jpg does anyone else think that this server setup is way overkill for gox? maybe i underestimate its traffic. Title: Re: mtgox servers Post by: thebaron on December 20, 2012, 08:58:10 AM Well, figure they should be using at least two layers of encryption (database and Bitcoin keys), so it's probably more taxing then you think.
Title: Re: mtgox servers Post by: Bendur on December 21, 2012, 09:43:35 AM Yeah if anything MtGox needs even better servers, or at least better ddos protection. They seem to get knocked off far too often
Title: Re: mtgox servers Post by: MPOE-PR on December 21, 2012, 12:17:37 PM i was just looking at this: https://mtgox.com/img/pdf/20120831/Transparency.008.jpg does anyone else think that this server setup is way overkill for gox? maybe i underestimate its traffic. It's not. Title: Re: mtgox servers Post by: Deafboy on December 21, 2012, 12:26:21 PM If you are not looking for HW on ebay and for the cheapest possible reseller of server housing services, then the price seems to be ok. :)
Title: Re: mtgox servers Post by: cedivad on December 21, 2012, 12:44:23 PM 5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer?
Suggest them to buy and colocate, Jesus! Title: Re: mtgox servers Post by: DeathAndTaxes on December 21, 2012, 02:13:15 PM 5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer? Suggest them to buy and colocate, Jesus! RAM prices have really fallen in the last 18 months (more than the usually 50% drop every 2 years). It is strange they wouldn't buy and colocate though. Especially for that high end. Paying $5K to $10K for the hardware and then cutting your monthly cost to <$1K per month would for a private cabinet would have paid off a long time ago. Then again maybe Japan has some insanely expensive datacenter costs no matter which way you go? The specs don't look overkill except maybe the 32 cores. Nothing they do is that CPU intensive. Lots of memory and some fast disks makes sense given the db centric nature. I am surprised they didn't seperate it out into db server and web/front end server though. Still I doubt they are getting much utilization of CPU side. If they are it would be cheaper to go with some high end NICs (TCP/IP offloading) and a HSA then trying to bang it all out with expensive CPU cores. Title: Re: mtgox servers Post by: paulie_w on December 21, 2012, 03:35:09 PM isn't bitcoin itself rather cpu intensive?
Title: Re: mtgox servers Post by: TangibleCryptography on December 21, 2012, 03:58:44 PM isn't bitcoin itself rather cpu intensive? Not really. I mean not 32 high end server cores intensive. We aren't talking mining. Bitcoind's (i.e. running a node for the backend) bottlenecks are I/O & memory. You got me curious so I checked. Peak CPU time on our bitcoind process is ~2% (excluding when it was bootstrapping which was more like 4%) and that is on a Single CPU (quad core xeon). Even if bitcoind was CPU intensive the current codebase is single threaded so you aren't going to use more than 1 core anyways (so the 2% is more like 8% of a single core). If you are running on a atom based netbook or a oversubscribed VPS instance you might have problems but any dedicated hardware won't blink. Title: Re: mtgox servers Post by: MPOE-PR on December 21, 2012, 05:04:21 PM 5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer? Suggest them to buy and colocate, Jesus! It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month. Title: Re: mtgox servers Post by: SgtSpike on December 21, 2012, 07:00:56 PM 5000$ for 2 servers with less than 100GB of ram each? Who is their provider, rackspace, softlayer? Suggest them to buy and colocate, Jesus! It's probably the double gbit links that are pricey. In some locations 100mbit dedicated can run you up 1k per month. I agree. 2GB/s of WAN data is spendy, no matter where you are. It could very well be the bulk of the cost. Title: Re: mtgox servers Post by: cedivad on December 21, 2012, 08:57:49 PM I tough that to be a backend database server, inaccessible from the outside, behind a pool of frontend servers... Still, 2Gbps of dedicated bandwidth to handle DDOS? It's not gonna scale, after all...
But who am i to criticize their setup? Edit, the 2Gbps uplink might be internal; they do have an external DDOS protection service. :) Quote traceroute to mtgox.com (72.52.5.67), 64 hops max, 52 byte packets 1 10.0.0.10 (10.0.0.10) 0.468 ms 0.383 ms 0.248 ms 2 * 15.757 ms 15.461 ms 15.779 ms 3 wf2.mi.sw-gw1.seflow.it (158.58.168.1) 14.648 ms 19.250 ms 16.862 ms 4 caldera.dc2.hsr1a.seflow.it (95.141.47.254) 45.370 ms 172.757 ms 223.565 ms 5 ge4-12.mil01-1.eu.as5580.net (78.152.32.201) 34.848 ms 17.582 ms 21.405 ms 6 tge1-3.par02-1.fr.as5580.net (78.152.34.109) 36.162 ms 41.075 ms 34.076 ms 7 tge1-1.lon01-1.uk.as5580.net (78.152.34.74) 50.626 ms 42.321 ms 47.633 ms 8 blackhole.prolexic.com (195.66.224.31) 43.664 ms 44.109 ms 43.942 ms 9 unknown.prolexic.com (209.200.156.34) 43.583 ms 43.484 ms 43.788 ms 10 unknown.prolexic.com (72.52.5.67) 43.222 ms 43.045 ms 43.080 ms Title: Re: mtgox servers Post by: paulie_w on January 23, 2013, 07:24:29 AM is it really for the DDOS protection?
Title: Re: mtgox servers Post by: mc_lovin on April 14, 2013, 03:46:22 AM So... does it seem to anyone that their hardware is extremely underutilized and their software is inefficient?
4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second. Title: Re: mtgox servers Post by: cosmicone on April 14, 2013, 04:15:15 AM LOL Raid 1 on their backup data server? WTF? Raid 1??? Even Raid 5 is no good... Spend less on equipment, and more on someone that knows what they are doing.
Title: Re: mtgox servers Post by: DeathAndTaxes on April 14, 2013, 04:54:26 AM So... does it seem to anyone that their hardware is extremely underutilized and their software is inefficient? 4x 8 core CPUs and shit tons of RAM should be able to handle more than 1 transaction per second. Yes it likely should be sufficient to handle 100x that, assuming the single box was used as dedicated trading engine with other servers used to offload other tasks (API access, webserver, support, charting, etc). Title: Re: mtgox servers Post by: aantonop on April 14, 2013, 05:03:57 AM You can't run an exchange as a single-stack application. No way it will scale.
The fact that they are still trying to throw hardware at this means they simply do not understand scaling. They will not solve these problems because they don't have a clue how to solve them RUN AWAY, don't just walk away. Title: Re: mtgox servers Post by: jubalix on April 14, 2013, 05:07:34 AM hmm looked under kill to me by far
Title: Re: mtgox servers Post by: jzcjca00 on April 14, 2013, 05:14:03 AM The database on the main server should be entirely on SSD. It's small enough.
RAID1 is appropriate, as disks are cheap, and they really aren't using much space. Downtime is more expensive than disks. There should be a secondary server with a copy of the database replicated in real time just for handling read only transactions, like clarkmoody.com, bitcoin itty.org, etc. And a backup for it, too. The main server is obviously still overloaded. They are underconfigured andneed to stop thinking smalltime! But most important, they need to stop th spambots from choking the system with .001 BTC orders. An easy way to do that is set a minimum fee per trade of .006 BTC. This would only impact the nuisance orders, since any reasonable person is trading at least 1 BTC. Title: Re: mtgox servers Post by: MarKusRomanus on April 14, 2013, 05:43:01 AM If your systems are behind Nic cards that are 2gbps and you are being hit by a DDos of 10gbps.. how are you supposed to protect from that? That size DDos will just halt all traffic just from the bottleneck effect.
Title: Re: mtgox servers Post by: MarKusRomanus on April 14, 2013, 05:49:45 AM Actually what I meant to say is that their main line into their server room(s) has to be a 1000gbps capable line.. and they need a minimum of 10gbps bandwidth to survive . their servers behind whatever protection they come up with can be 2gbps .. no problem.
|