A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[DeathAndTaxes]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

Yup that is the one. 

[DeathAndTaxes]

Controlling a farm from single server.

Most farms are run as a set of independent rigs.  With watercooling and a shared outer loop that presents some challenges and risks.

The farm needs to shutdown in the event of a LOCA (to borrow reactor terminology http://en.wikipedia.org/wiki/Loss-of-coolant_accident).   Now including software on the rig which shuts down the rig when its temp rises is possible but not a solid failsafe.  Rising rig temps can cause processes to hang which could keep rig online cooking itself without cooling.  Having each rig monitor sensors creates a lot of duplication.

So my plan is to have a daemon on each rig monitoring GPU temps and if they rise above a failsafe to start a shutdown of the rig.

Backing that up a non-mining server "watchdog1" will provide analog monitoring of the entire farm.  It will also hold copies of all rig config files, collect stats from cgminer API, host p2pool & bitcoind daemons (and possibly act as a PXE server).

I plan to have:
4 outside temperature sensors (water before radiator, water after radiator, air intake, air exhaust)
1 ambient air sensor
1 flow sensor
1 reservoir level sensor (contact only)
1 pump current sensor

Now when watchdog detects something "abnormal" it could send a "emergency power off command" to each rig but we run into the same risk if a rig fails to shutdown.  So instead I am building a power control board which will allow the watchdog to power on, power off or power cycle all the rigs.


Multiple "dumb" quad 5970 water cooled rigs (initially 6).
&
"Watchdog" supervisor to keep them safe.

[shakaru]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

Yup that is the one. 

Not to tread-jack to much. But I have 2 of these (more in storage) for sale for WAAAAAY less than that price.

[DeathAndTaxes]

Not to tread-jack to much. But I have 2 of these (more in storage) for sale for WAAAAAY less than that price.

Not a thread jack at all.  If it is that exact one (RM41300-fs81) I am interested, PM me. 

Chenbro makes other RM41300 models but they only have 7 expansion slots.

[JinTu]

I have been contemplating water cooling for my 4RU 19" rack mounted miners as well. Noise is a big issue for me due to the WAF, so I have been looking at passive radiators like these. They take up a lot more room, but are totally silent. My current thought is to mount an array of these to the side of the 19" rack so it doesn't take up additional space.

[DeathAndTaxes]

I have been contemplating water cooling for my 4RU 19" rack mounted miners as well. Noise is a big issue for me due to the WAF, so I have been looking at passive radiators like these. They take up a lot more room, but are totally silent. My current thought is to mount an array of these to the side of the 19" rack so it doesn't take up additional space.

That is a quiet option. Even a "normal" radiator is very quiet.  My "test radiator" has 4 low speed 120mm fans.  It is really no more noisy than a power supply (a quiet "woosh" instead of any buzz of whine from high RPM fans).  Even a small amount of airflow can significantly improve the cooling power of a radiator.   4x120mm radiator is insufficient for 4x5970s. 

My goal though it to dump the heat outside thus saving all the AC costs.

[check_status]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer.  

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

Will do.  I have a crimping tool from some home remodeling work already.  I will likely make a test loop involving just pump, radiator, and heat exchanger and leave it running.  The reason I would like to use PEX is I could take an off the shelf radiant heating manifold and use that for splitting the run to each rig.  

That radiator is rad. Same price range as units that are sold for PC's but the advantage of dumping more heat. Schweet!

With PEX you can get a 3/8" inside diameter hose on to a 1/2" diameter PEX fitting (not compression). Fit should be very tight.

[The-Real-Link]

Very nice job, Death!  While I don't have a giant rack for my rigs, it's cool to see that those can be used.  I've not done water cooling before though so yeah while it is tempting for even lower temps, that's a lot of extra parts for even a handful of 5970s. 

[DeathAndTaxes]

An update (if the updates seem a little rambling they are.  Just trying to get my thought down on paper as I go along).

Heat Exchanger and coolant lines:
So I bought the radiator water to air heat exchanger.  I talked on the phone with supplier, a nice guy who answered all my questions despite me using the HE in a non conventional manner.  It is designed for transferring heat from a wood burning furnace to air handler.





I also picked up some rolls of 3/8" (branch line) and 1" PEX (mainline).  I am still deciding on the pump.  I need a pretty "beefy" pump.  Something which can push 8-12gpm with 10 feet of head.  My first thought was aquarium pump but they are expensive and seemingly under powered.  Next I thought about radiant heating pumps but I would need a stainless steel model and they aren't cheap.  Someone recommended Iwaki water pumps.



Seems interesting.  The smaller ones are used by extreme watercoolers but I need more extreme than extreme.  Nice thing about these is the build quality is second to none.  I would be confident with it running 24/7.  Any thoughts on a pump?

Changes to the plan:
After some thinking I decided to modify my plan a little.  Rather than use a flat plate heat exchanger (water to water) for each rig I am going to run the coolant in one giant loop.  The main reason is cost.  Having a closed loop system in each rig transferring the heat but not water (via heat exchanger) to an outer loop is nice but expensive.  We are talking $200 per rig for internal pump, reservoir, and heat exchanger.  6 rigs = $1200.  Removing that significantly cuts the cost with a little more work when adding and removing a rig.  


So new design will be:
heat exchanger -> 1" PEX -> pump -> 1" PEX -> 8 cold line port manifold -> 3/8" PEX branch lines in parallel -> each of the 6 rigs -> 3/8" PEX branch lines ->  8 port hot line manifold -> back to heat exchanger.

Each rig will connect to the cold and hot side manifolds with no-spill quick disconnects and flexible tubing (likely Tygon).

Code:

                        3/8" PEX branch lines
from heat-exchanger
                         |--------{rig #1}--------| 
1" PEX cold line         |                        | 1" PEX hot line 
-------------------------|--------{rig #2}--------|-----------------    to pump/reservoir/heat-exchanger   ---> 
                         |                        |
                         |--------{rig #3}--------| 

          cold side manifold                hot side manifold

                -----(flow direction)--------->

Only 3 rigs show for simplicity.  Total of 6 rigs will be connected.

Bleeding the system will be a little more challenging but $1200 in cost savings makes it worthwhile.  Having a large (say 1 gallon+) reservoir higher than the highest rig.  That should allow water pressure to aid in bleeding.  Possibly 2 reservoirs?

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

[rjk]

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

There was a folder on another forum that used a 55 gallon plastic barrel, and a saltwater-rated aquarium pump for longevity. EDIT: Link to forum thread
Do you have your manifolds yet? Here is a good one:


http://www.ebay.com/itm/190550567439

[johnyj]

better places to put the mining rig:

1. In mountains of Tibet
2. water radiator sunk into a lake, or even better a river

[stevegee58]

During the winter I just position my PC next to a slightly open window.  Works great.

[nedbert9]

Impressive job, DandT.  Yeah, get that heat outside.

[tonto]

Jesus.  This idea is just full of win.
 
It's making my idea to water-cool my 7970 through my aquarium seem like weak sauce. 
 
I haven't done it yet, 'cause it's only 55gal aquarium, and I'm still unsure if the heat exchange in my basement would dissipate that heat fast enough before creating fish-soup. 
 
Should I come across a nice sum of money that my wife doesn't find out about, I think I'm going to steal your idea, DeathAndTaxes.

[DeathAndTaxes]

I don't own an RV but this looks like a nice resivour to me

http://www.tank-depot.com/productdetails.aspx?part=85-1859WH



Turn it so the fittings are on top.  
1" in,  1" out, 1/4" vent line, and 1/4" for installing a float sensor.  

I was going to make a cube out of plexi, bond the edges, and tap threads into it but man it is like they made this for extreme watercooling.

[DeathAndTaxes]

An update:

Talking w/ some watercooling guys I am deciding on a different pump.



Likely an Iwaki MD series pump.  These are some monster pumps and have a good rep for requirements of large water cooling projects.  Likely I will need a model larger than what most people consider.  I am thinking of the Iwaki WMD-40RLT.

[rjk]

An update:

Talking w/ some watercooling guys I am deciding on a different pump.

http://i11.photobucket.com/albums/a154/eastcoasthandle/WATER%20COOLING/Iwaki_MD30RT.jpg

Likely an Iwaki MD series pump.  These are some monster pumps and have a good rep for requirements of large water cooling projects.  Likely I will need a model larger than what most people consider.  I am thinking of the Iwaki WMD-40RLT.

You might also consider a bore pump for use in submersible well applications. However that might have a head pressure that is too great.

[DeathAndTaxes]

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

There was a folder on another forum that used a 55 gallon plastic barrel, and a saltwater-rated aquarium pump for longevity. EDIT: Link to forum thread
Do you have your manifolds yet? Here is a good one:


http://www.ebay.com/itm/190550567439

I haven't found a good cheap manifold yet.  

The problem is it is aluminum.  copper + aluminum in same loop of circulating water =

If they made that exact item in copper it would be perfect.  

I have been looking for something which is copper with 1" supply and 6 to 10 branches outlets w/ 3/8" or 1/2" npt threads that doesn't cost a fortune.  That doesn't seem to exist.  Lots of radiant heating manifolds but they come with high end pex connectors and cost a fortune.  I may have to break out the torch and make my own.  

[P_Shep]

How do you make sure flow though each of the splits is the same? Or rather, that one or more isn't so slow it causes problems?

[DeathAndTaxes]

How do you make sure flow though each of the splits is the same? Or rather, that one or more isn't so slow it causes problems?

Each split has the same resistance because it uses the same components.  That keeps the flow roughly the same though each split.  You couldn't however have 4x5970 in one rig and 3x7970 is another rig.  As long as all the "downstream components" are identical resistance should be roughly the same.  As water velocity increases the back pressure increases and that prevents one loop from flowing faster than another one.

If you did have dissimilar branches you could use a "booster pump" on each branch to boost flows to the one with higher resistance.  As an alternative you could use something called a balancing valve which increases pressure to the lower pressure segment to balance the flows.  I hope to avoid that though (more cost, more complexity).