A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[phorensic]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

[DeathAndTaxes]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

Not to late.  Haven't pulled the trigger on the pump.  Was doing work on the heat exchanger and sweating npt connections this weekend.

Show me a pump with
a) similar curve as a Iwaki MD-40
b) has 1" NPT intake and discharge
c) has similar heat dump
d) has significantly lower price

and I am interested.  I have looked any haven't found it.

Everything I have found either has horrible head (most aquarium pumps) or is horribly inefficient (most sump dumps) dropping 300W to 500W of heat into the loop or is more expensive than the Iwaki.

If you found a good pump that can get similar flow as the Iwaki MD-40 at 10ft to 15ft of head let me know.

[DeathAndTaxes]

I`m having strange problem here with system 6GPU+
made 2 systems yesterday, and bouth got it.
GPU`s are shutting down randomly....
I was testing few OS`s: xubuntu, linuxcoin, W7(WTF? - i was desperat)
  

1 rig - 3x5970+5870  - OCZ ZX 1250 - all GPU down after 7h - i was sleeping
2 rig - 3x5870+2x5970 - ENERMAX 1250 + sommething 550W - 3 GPU down after 7H

Both rigs has 890FXA-GD70 and GPUs are watercooled.
Cheked watteg with KILL-O-WAT but only rig no.1 - takes 980W from the wall - 5870 - 950, 150 and 5970 - 750,150 - so not a big deal. temps - max 54. Cheked vrm tmp under W7 - each GPU - higher by 10*C so max ~65...cool and nice, so why down ??
any ideas ??

Entire rigs are crashed?  Can't SSH into them and see what happened?  That is weird.  I would turn on logging in cgminer and try to see what happened.  If you need help on logging in cgminer use the cgminer thread.  As a test I would take every card to stock and with a higher memclock of say 280.   Get them running 48 hours+ before you start playing around with overclocking.

My 4x5970 rackmount chassis has been rock solid.  Significantly overclocked it hasn't had any downtime since I started this thread.  No crashes, no failures, no noise just a continual supply of new shares.  The longer that test rigs runs solid, cool, and quiet the more convinced I am of this project despite other setbacks (like radiator arriving damaged, and trying to find a cheap manifold, etc).

[phorensic]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

Not to late.  Haven't pulled the trigger on the pump.  Was doing work on the heat exchanger and sweating npt connections this weekend.

Show me a pump with
a) similar curve as a Iwaki MD-40
b) has 1" NPT intake and discharge
c) has similar heat dump
d) has significantly lower price

and I am interested.  I have looked any haven't found it.

Everything I have found either has horrible head (most aquarium pumps) or is horribly inefficient (most sump dumps) dropping 300W to 500W of heat into the loop or is more expensive than the Iwaki.

If you found a good pump that can get similar flow as the Iwaki MD-40 at 10ft to 15ft of head let me know.

Oh cool, you are still deciding on the pump.

I'm going to be fairly blunt about this.  What makes you think you need to pump ~15ft of head at 450+ GPH?  Have you calculated your total head based on real height plus all your fittings and doo-dads?  There are calculators out there, btw.

Based on my experience with even shitty aquarium pumps, they are all underrated on what they will flow at their "supposed" maximum head height.  I think you would be surprised what a ~50-75W submersed pump can do even "choked" down to 1/2" tubing.  Don't get caught up too much in the system designs for water cooling in the overclocking computer world.  They tend to grossly oversize EVERYTHING in search of dropping that last 1 degree.  I was doing computer watercooling before you could just go out and buy a kit in a box.  Some of the setups the enthusiasts started making in the early 2000's up to now are downright hilarious on how oversized everything is.  Some of the more commercial application are more realistic, with way smaller tubing, pumps, and radiators.  I think if you shoot for 50-60C temps at the GPU itself, instead of 2C above ambient like some guys, you would be surprised how cheap your initial investment gets on the equipment, host fast you get your ROI, and how long everything would last.

If you live in a dry environment you could even look into evap cooling to dump the heat instead of a radiator.  You know, how nuclear power plants are cooled?  A while back people designed PVC "water bongs" to blow air up over water that was showering down into a sump.  I made what I called a "bucket bong" that did the same thing in a 5 gallon bucket, just a shower head and an 80mm fan.  The advantage?  Your water coming out of the "bong" was 1-2C *colder* than ambient.  The disadvantage?  You have a bit of humidity to worry about in the room you dump the heat in.  You could either plumb it outside, or just dump some of the air outside with another window fan again, lol.

[DeathAndTaxes]

I do believe I need 10gpm @ 15ft of hea.  For expansion and some margin of safety in flow rates (due to balancing) I would prefer 20gpm or as high as economically possible @ 15 feet of head.   Remember each rig is in parallel so 10gpm across the mainline is only 1.5gpm across the server if the minimum of 6 servers are used.  If that is expanded to 10 servers (~32 GH/s) for a full rack it is only 10 gpm mainline = 1 gpm per server.

I already used a flow/pressure calculator:
The server has 4.2ft of head loss (@ 2gpm) measured by me.
remember 8 GPUs worth of blocks in serial is rather restrictive this isn't your "normal" loop.

As I don't have a manifold yet I am reserving 1.0 ft of head loss for the manifold (and various fittings).
The heat exchanger has 2.2ft of head loss at 20gpm (thankfully it has 1" inlets and 8x 3/8" copper branches).
The mainline as planned is 70ft of 1" PEX.  At 20gpm that is ~8ft of head loss.

4.2 + 1.0 + 2.2 + 8 ~= 15ft.

So options:
Options 1) Could use a pair of radiators in parallel to reduce the flow across each and the corresponding head loss.  At 10 gpm the head loss is only ~ 1 ft. Still I doubt that saves much money when you consider the additional cost of 2nd radiator and fan assembly.  

Option 2) Use a less restrictive mainline.   1" PEX is just so cheap and easy to use.  Going to 1.25" CVPC is an option but a lot more work installing.  

Option 3) I could just accept lower flow across each rig but I worry that if I go that route flow isn't going to be perfectly balanced and one rig ends up getting too low of a flow.  That would require either putting in expensive balancing valves on the manifold or buying a larger pump.

Given those options I would rather not skimp on the pump and end up needing to waste time and money putting a second radiator in parallel, dropping in balancing valves, or re plumbing a larger mainline (1" PEX = $0.80 per foot.  1.5" PEX = $3 per foot ).  Going "cheap" on the pump may save $100 or so but it may end up costing three times that in extra work and modifications.

On evaporative cooler:
I don't think any home grown evaporative cooler is going to handle 7KW of heatload.  Regardless the summers in southern VA approach 90% humidity so I think a forced fan heat exchanger is going to be superior for year round operation.

I don't mind looking at alternative brands I just haven't found anything that gives me what I need for less.  No doubt it likely exists so if you have a pump model in mind I am all ears.  I have found some industrial pumps but they tend to be lower efficiency and dump as much as 500w into the loop which make them useless.

Your thoughts?

[DeathAndTaxes]

I think if you shoot for 50-60C temps at the GPU itself, instead of 2C above ambient like some guys, you would be surprised how cheap your initial investment gets on the equipment, host fast you get your ROI, and how long everything would last.

I am shooting for 55C to 60C temps.  Remember though this is nearly 7KW in heatload and outside temp peaks at 35C.  To keep 7KW of heatload at 25C over ambient requires more cfm, flow, and surface area than you may think.  I did some heat flow calculaitons and to avoid needing 2000 cfm+ requires pretty significant gpm on the liquid side.  Sure cooling in the winter is easier but we need to make it through August in Virginia Beach first.

On edit (found this looking for info on flow rates):
http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphad/watercool9125-F2A.htm

Looks like IBM recommends 22gpm per server rack and 33 psi (~14 ft of head) of pressure.  Granted no exactly apples to apples (they are also using water chiller to lower input temp of the coolant to <20C) but thought it was interesting comparison.

[Roadhog2k5]

You should be looking at flow rate instead of static head pressure. Static head doesn't matter in a closed loop.

Look at the pressure drop (friction head) of each of your components to pick a proper pump.

EDIT: Also if you want a nice pump, get this. http://www.pexsupply.com/Grundfos-59896773-UPS15-55SFC-Stainless-Steel-3-Speed-Flanged-Circulator-Pump-1-12-HP-115-V

Have three of them for my radiant floor heating, one for each zone. They have no problem pushing through hundreds of feet of pex without skipping a beat.

[phorensic]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

[DeathAndTaxes]

You should be looking at flow rate instead of static head pressure. Static head doesn't matter in a closed loop.

Look at the pressure drop (friction head) of each of your components to pick a proper pump.

I am not sure what you are trying to say and I am not sure you do either? 

The flow rate at given resistance is what we are discussing.  I am not sure what you are advocating differently?

[DeathAndTaxes]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

[Roadhog2k5]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

Have you considered running two pumps in series? You would gain additional flow, and redundancy in case of pump failure.

[siggy]

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

Do a search for "Panworld pump" .. they're the "inexpensive iwaki's" ..  I've got 2 of them running on my saltwater tank that have been running nonstop for 7 years now.

Sigg.

[phorensic]

Does this satisfy you need to have a massive ass pump? - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html  hehehe, but seriously the price is way better for the same quality.

Even though this doesn't meet your head requirements, I would really like to see what it would do in your actual finalized setup - http://www.marinedepot.com/Aquarium_Systems_Premium_Maxi_Jet_3000_Utility_Pump_775_gph_Up_to_500_Gallons_Per_Hour_Submersible_Water_Pumps-Marineland-AS1311-FIWPSBUF-AS1319-vi.html .  You could sell it to the aquarium nerds if it doesn't work =P  Take into account also that your plumbing on the pump gets reduced to just a pipe that dumps into the sump, and then a hose straight off the outlet of the pump...instead of all the fittings required to get an external hooked up properly, plus union valves to make it easier to service, etc.

[phorensic]

Oops, sorry, the 100PX is considered comparable to the MD40 - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html .  I was looking at flow curves instead of the feature briefs.

[DeathAndTaxes]

Even though this doesn't meet your head requirements, I would really like to see what it would do in your actual finalized setup - http://www.marinedepot.com/Aquarium_Systems_Premium_Maxi_Jet_3000_Utility_Pump_775_gph_Up_to_500_Gallons_Per_Hour_Submersible_Water_Pumps-Marineland-AS1311-FIWPSBUF-AS1319-vi.html .  You could sell it to the aquarium nerds if it doesn't work =P  Take into account also that your plumbing on the pump gets reduced to just a pipe that dumps into the sump, and then a hose straight off the outlet of the pump...instead of all the fittings required to get an external hooked up properly, plus union valves to make it easier to service, etc.

Even if we take the specs at face value it is 8 gpm at 4 ft of head.  No curve or max head is provided but I imagine that is the sweet spot.  Likely sub 4gpm at 8ft of head and 0gpm at <15 ft.  I don't even see that being in the same league of the kind of pump necessary to cool 8 servers in parallel w/ 8 GPU each in series, through 70ft of tubing to a massive radiator with 64 ft of 3/8" copper tubing. 

[DeathAndTaxes]

The 100PX is interesting.  In searches for Pan World the brand "BlueLine" keeps coming up.  Looks very similar to Iwaki.  

HD 40
http://www.aquacave.com/Blueline-40-HD-ExternalAquarium-Water-Pump-by-Pan-World-P488.aspx

HD 55
http://www.aquacave.com/Blueline-55-HD-External-Aquarium-Water-Pump-by-Pan-World-P490C762.aspx

Lots of merchant indicate Blueline is produced by Pan World but Pan world website has no reference no Blueline.



If these specs are to be believed then BlueLine might be the pump I need.

[DeathAndTaxes]

Finally pulled the trigger on a Pan World 150 PS.  Found one at a good price (cheaper than other places are selling 100 PS)



Looks like 12 gpm @ 14 feet of head.  Should handle the resistence of 1" PEX mainline just fine (saving me the cost and tedious work of assembling 70 ft of CPVC).  I am hoping it gets here my Friday but it might not arrive till next week.   Come on Fedex lottery.

[rjk]

Man it would be cool to have a 3 phase pump on a PID loop with a VFD. Then you could add and remove rigs with no need to worry about balancing anything out (for the most part). The pump would just slow down as more resistance is added (closing off valves to swap a rig, or whatever) or speed up if you add more rigs to the loop.

[phorensic]

OK, you found the other brand that is a clone of the Iwaki.  I knew from my saltwater aquarium days there were 2 Iwaki clones, one was Pan World and the other I couldn't think of for the life of me so I didn't list it.  I would consider a BlueLine a contender also.  They all share excellent the same engineering specs, with different people actually making them.  Fake edit: I see you bought a used PanWorld 150.  Badass!  As long as the price isn't outrageous it is still cool that you got an overkill pump.  And yes, rjk, VFD would be the ultimate luxury on a setup like this.  Totally unnecessary, but we are nerds, so every improvement is cool!

Should put one of these in the system: http://www.marinedepot.com/Digital_Aquatics_ReefKeeper_Lite_Controller_Multi_Item_Monitors_Controllers_for_Saltwater_Aquariums-Digital_Aquatics-DA1131-FITEMOMI-vi.html . I mean, wouldn't it be cool to monitor the pH, temp, and *salinity* of your *freshwater* cooling loop? Im just kidding man, seriously, just kidding.  I want to see this project grow.

[ummas]

Since my last post i discovered, that my sempron was unlocked to Atholon II and it maight be problem of the all GPU crush. seccond rig is stable after changing intensity to d (jumps 3-6) - aprox 20mhash loss :/ but stable for last 24h.

rig was simple to get on with ssh, but all gpus was down and i had to reeboot it. xorg 99.9% of cpu bug
thats why i was digging in the bios and i figured the unloced core.

thats haw i become 2x2.5Ghs 100% Watercooled rigs owner

I`m using 12V laing`s with modifed electrics - pulls 20-30W (depands on Voltage) and makes low noise and preaty nice flow.
My temps are aprox 51-55 with just 3x120Fans on old 3x120x2 (2 radiators puted toghether).
My system was build from scraches. I paid really low prize for it, for example: 1 block cost me 60$, pump 60$, 2 new blocks costs me 180$ each, block for 5870 costed 30$. 2 Radiators and other accesiores i had for a long time working in my old systems.