A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[Red Emerald]

So this thread is mostly about the hardware, but I'm curious what you use for the software.

I know you've mentioned that part of your farm is mining at p2pool.  Do you use BAMT? Linuxcoin? Your own installs?

[1714835572]

1714835572

[1714835572]

1714835572

[1714835572]

1714835572

[1714835572]

1714835572

[DeathAndTaxes]

I was/am using BAMT for the 3x5970 air cooled rigs  but it is x86 only which means it won't work w/ 8 GPU for the watercooled conversion.  For the first watercooled rig I am using a modified version of LinuxCoin.   I may consider a custom distro at some point.

The farm is on p2pool (well part of it is right now), but p2pool is not installed on each rig.  The "watchdog server" runs bitcoind and p2ool.  My workstation runs a backup copy of bitcoind & p2pool.  Each rig has first p2pool instance as primary, the second p2pool instance as secondary and bitminter as tertiary mining pools using failover.  Something that is a low priority but I would like to look into, is using a static distro which is reconfigured on each boot.  There would be no USB drives.  The rigs would simply boot from LAN and pull the distro from a PXE server (watchdog again).   Once booted the rig would pull a cgminer config file from the server.

My goal is to make the actual rigs as "dumb" as possible.  Turning them into 3GH/s hashing appliances with control/config/monitoring done by the watchdog.  It looks like my pump might arrive today.

[Spotswood]

I'm curious as to how much heat that pump is going to add to the loop. 

[siggy]

I'm curious as to how much heat that pump is going to add to the loop. 

less than 1/2 of 1 GPU equivelant.   These things are used a lot for salt-water fish tanks.. they are very efficient pumps. 

Sigg

[DeathAndTaxes]

I'm curious as to how much heat that pump is going to add to the loop.  

About 90W.  In a "normal" water cooling system that would be excessive.  Given I have a heat load of ~6KW, 90 more isn't much.  One way to look at it is I am replacing 24 blower fans @ 8W ea = 192W with a 90W pump and 40W fan.

The pump did arrive today.  UPS ground and it got here in 2 days.     It is impressive.  Everything about it screams build quality and performance.  Did some testing and ended up soaking myself.  Even at 10ft of head it moves a ton of water.   Hopefully I can get the first rig hooked up to the main loop this weekend and then order parts for the rest of the rigs.    No time to waste we already have 80F weather.   

[rjk]

The pump did arrive today.  UPS ground and it got here in 2 days.     It is impressive.  Everything about it screams build quality and performance.  Did some testing and ended up soaking myself.  Even at 10ft of head it moves a ton of water.

Lol, I bet the sequence of events was something similar to the following:

Hell yes, the pump is here! *rips open the box like a kid at christmas*
Lets see if it works - *plugs in for only 3 seconds with no primer water, cause 3 seconds won't kill it, right?*
Sweet, it works. *runs off to the garage to attach some hose to the intake, and maybe a little bit to the output*
*fills up tub of water, primes pump and plugs in again*
Oh shit son, water everywhere hopefully none got on my rigs shitshitshit
*stupid grin while cleaning up the mess*

[DeathAndTaxes]

Something like that except I never run pumps without priming and I did have the common sense to not test it around the rigs.

Actually everything went fine in first two tests.  Then I figured I could test @ 10ft of head by putting the pump downstairs with a 5 gallon bucket, run discharge hose up the stairs to bathtub on the second floor.  Great plan but how to start and stop the pump.  So I give my wife instructions and warning about not letting intake run dry.  Everything is going good but I start thinking man this is a lot of water is the bucket almost empty.  I turn my head to yell down to her to make sure the bucket doesn't run dry (it didn't) and when I did I dropped the discharge hose.   Of course then it starts snaking around the bathroom and me trying to direct it towards the bathtub catch 20gpm of water to the face.

Short story is the pump "should" be more than powerful enough. 

[johnyj]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

[DeathAndTaxes]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

Yeah I thought about that.  I even tried looking for smaller commercial ones as inspiration.  Seems they don't really exist for anything <20KW.  Even the 20KW units look pretty compact about the size of say 3 or 4 outdoor AC units side by side. 

The other option would be to bury 100m or so of tubing below the front line and have it act as a giant geothermal heatsink.

[cuz0882]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

Yeah I thought about that.  I even tried looking for smaller commercial ones as inspiration.  Seems they don't really exist for anything <20KW.  Even the 20KW units look pretty compact about the size of say 3 or 4 outdoor AC units side by side. 

The other option would be to bury 100m or so of tubing below the front line and have it act as a giant geothermal heatsink.

That's a interesting idea, wonder how effective it would be. I'm a little skeptical about running everything in one loop. Repairs and downtime could be a major problem. Maybe one large loop that cools everything with each pc having its own smaller loop. Using heat exchangers to connect them.

[DeathAndTaxes]

I'm a little skeptical about running everything in one loop. Repairs and downtime could be a major problem. Maybe one large loop that cools everything with each pc having its own smaller loop. Using heat exchangers to connect them.

That was my original idea the major problem is that you now need n+1 pumps, radiators, and n heat exchangers.  Add in the additional cost of more fittings and the cost is simply not viable.

Each rig can be seperated from the main loop without affecting other rigs.  If the main loop is down then heat exchanger or not all the rigs will need to be idle anyways. 

[DeathAndTaxes]

So an update.

The manifold:
I found a manifold for my rigs.  It is a radiant heating manifold.  I was at local plumbing supply house and after explaining what I needed  one of the employees found a returned open box manifold.  This is a $250+ part but was marked down to $120 due to some damaged.  Got them to drop it to $70 with no returns unless not water tight.    I will post some images tonight but here is a stock photos.  The blue caps at the top are balancing valves and the red tubes at the bottom are flow meters. 



One of the balancing valves is jammed, and two of the crimp rings are warped, one of the temp guages doesn't go above 80C.  It also has some cosmetic damages and tooling marks.  No idea what idiot tried to use this and why the store accepted the return but I got a lot of nice features for very little cost. 

Now it is designed for connecting PEX piping and the adapters for each zone are proprietary which makes it a challenge to repurpose.  Through trial and error I found that 1/2" ID, 5/8" OD tubing is very close to the nominal dimensions of 1/2" PEX and remains water tight even under pressure using the compression fittings which work with this manifold. I ordered some 7/16" ID tubing which I think will provide an even tighter seal.  Another option is changing the fittings on each zone to ones designed for 5/8" PEX which is a good fit for 1/2" ID, 3/4" OD tubing.  Since the tubing will be outside the server (connects quick disconnects on front of server to manifold) having a thicker sidewall may be better.

In testing I got 5 loops (no rigs connected) running @ > 1.6 gpm (the max on flow meters).    The balancing valves will make is very easy to ensure each rig gets at least 1 gpm to ensure turbulent flow.  Honestly despite being damaged this is a total score.

The pump and loop:
Hooked the pump up to using some PVC Unions.  At this point I am using 3/4" ID tubing (fits tightly over 1" PEX barbs) for the mainline because PEX isn't exactly flexible or easy to work with when testing.  Purging air from the system was a challenge.  I made a PVC "T" to allow a line for adding coolant and purging air. It only worked marginally.   It took hours to purge air from the loops and I don't think I got it all.  Went back to Lowes and picked up a 2" PVC T with some 1" threaded adapters.  The larger diameter means water will slow down as it enters the T allows the air to separate easier.  

The heat exchanger:


The good news.  It is massive and well built with a ton of surface area both the fins and the internal tubing.  The surface area of copper tubing is important because it is how heat is transferred from the water to the fins.  You can have a giant heat sink but if it only has a small amount of tubing it isn't going to be very efficient.

To put it into perspective your average 3x120mm water cooling radiator has 1/8" copper tubing usually 4 parallel lines which make one pass.  
Surface area of the copper tubing on 2x120mm radiator  = 2 * Pi * (1/8) * 14 * 2 * 4 = 88 in^2

This heat exchanger using 8 parallel lines of 3/8" copper tubing and makes 3 round trip passes.  
Surface area of copper tubing on this water to air heat exchanger = 2 * Pi * (3/8) * 20 *6 * 8 = 2261 in^2  

The bad news.  I bought it from an ebay seller and some fins were damaged.  I can fix them but I need a "rake".  Worse for some reason rather than capping the 1" copper tubes with plastic or rubber plugs they sweated bronze caps (using what looks like two gallons of solder).  In the process it looks like they knocked the tubes out of round.  I had a nightmare of a time sweating new fittings on.   I ended up cutting the caps off and having to bend the tubes back round.  It look a lot longer than I had planned.  Should have spent a little more because I certainly paid for any discount in manhours.

Disclaimer:
I hope these posts have shown that there is nothing "standard" about cooling a rack of servers.  Everything requires adjustment and modification as you go.  Luckily I own a lot of tools.  If I didn't I likely would have had to spent $1000 in tools and supplies (wrenches, channel locks, pipe cutters, soldering torch, mapp gas, tube cutters, pipe dope, thread sealant, etc).

What's next:
Will get some photos tonight.  Hopefully (work dependent) in the next couple days I can get the time to hook the test rig to the new cooling loop.  I have enough waterblocks to test a second 3x5970 rig maybe this weekend.  If everything goes good I need to pull the trigger on 20x 5970s waterblocks.

[cuz0882]

I was thinking of adding a fourth card on one of my water cooled rigs. There would be no slot spacing between 2 of the cards though. I was thinking of using male to male adapter. You know if that would work? http://www.amazon.com/Enzotech-Rotary-Adapter-BRMM-G14-Black/dp/B007AIXE1M/ref=sr_1_36?s=electronics&ie=UTF8&qid=1332286492&sr=1-36

[DeathAndTaxes]

it probably will work but I am not positive.  That is why I used MSI 890FXA-GD70 motherboard.  It has 4 16x slots 2 apart. 

Likely you may need to do some trial and error to find something which works with a single slot.

[DeathAndTaxes]

Some photos

The manifold.  5 test loops.  The balancing valves make it easy to keep flow even through all loops even if I create artificial restriction in one of the loops.  No rigs attached yet but hopefully I can get the first rig hooked up tomorrow and maybe second partial rig by this weekend.  If all goes well then I need to order 20 more waterblocks and for on the next 4-6 rigs.


The pump assembly.  Used two unions and the assembly to the left is the bubble eliminator MK II.  The larger pipe size slows the water velocity down which makes it easier to separate air.  Air flows up the T line which is where additional coolant is added. 


The radiator.  Also 4 of my existing aircooled 3x5970s rigs.  Will be nice to have some peace and quiet in the garage again.


The entire loop.  I am currently using 3/4" tubing but hopefully this weekend I can mount manifold properly to a sheet of plywood (which will mount to side of server rack).  Once everything has been tested in their final position I intend to replace the tubing with 1" PVC.

[rjk]

Beautiful. Those test loops look a little kinked, hope you will be using unkinkable hose for the final run. BTW, how much flow restriction should I expect from a waterblock? I'm sure normal pressure shouldn't be more than 1-2 bar, but when sizing everything up, the pump expert wants to know how much actual restriction there is, and I can't find anything relevant in the tech specs.

[DeathAndTaxes]

Yeah the tubing is the Lowes chepo vinyl stuff.  I got better tubing but no sense in cutting it up for test loops.  It was hard to keep it from getting kinked but I intentionally kinked some of them to increase resistance to simulate a server loop.  Not very scientific but luckily this pump seems have more than enough juice.

Most watercoolers measure resistance in terms of ft of head loss.  It varies by waterblock and finding exact specs can be tough but resistance is usually on the order of 1 to 2 ft of head loss per block at 1 gpm.  If the blocks are in series then the head loss if cumulative.  If the blocks are in parallel the head loss for the loop is the most restrictive.

There isn't a lot of information because really modern blocks aren't that restrictive and modern pumps have more than enough head.  This can make it tough to find good stats.  I have found reviews on pumps tend to have better stats on loop resistance than review on the blocks themselves.

Restriction (head loss) increases with flow rate but flow rates > 1 gpm really don't improve cooling much (1 gpm can transfer ~500 watts of energy w/ only 1 C rise in water temp).  The loop will reach an equilibrium between flow and resistance.  The goal is just to make sure that is >1 gpm.  I am planning for ~1.5 gpm flow through each loop.  6 loops = ~9 gpm across the main line.  The reason we want >1 gpm is to ensure turbulent (not laminar) flow through the block.  Older blocks required much higher flow to be turbulent but modern blocks have improved on that too.  

Sadly watercoolers today have it too easy.  Blocks are less restrictive, they avoid laminar flow even at low flow rates, pumps have higher flow and can handle higher pressure, larger tubing means less loop resistance, etc.  Compared to 10 years ago it is kinda hard to build a bad "normal" loop.  The bad news for us "extreme" builders is that means less stats on flow & pressure because for normal users taking any decent pump, any decent block, and any decent tubing likely will work just fine.

[DeathAndTaxes]

An update:
Had a minor leak at the radiator where the inlet pipe was knocked out of round.     Removed existing fitting, sweated on a 90 deg connector (which I wanted to use anyways, if you got to resweat the pipe at least make it an upgrade).  Was water tight for 12 hours+.

My Primoflex Pro LRT tubing (ID: 7/16 OD: 5/8) arrived.  A perfect fit for 1/2" PEX connectors.  Nice and flexible.  I updated the fitting on the rig to 1/2", attached the tubing put the inline no leak quick disconnect connectors and ran the loop to the manifold.

Had a minor leak when cutting water on.  The 1/2" barbs I bought don't fit on the "metal side" of the waterblock (they fit fine on the other end).    They threads are too long and they bottom out before compressing the o ring completely.  Luckily I had a pair of 1/2" barbs hanging around.   BTW large amounts of shop towels underneath all fittings made initial water turn on less scary.  Used hair dryer to remove any water (I didn't see any but I am paranoid).  Leak tested it for 12 hours.  Looks good.

Powered up the rig this morning.

Code:

 cgminer version 2.3.1 - Started: [2012-03-23 11:43:21]
--------------------------------------------------------------------------------
 (10s):2963.1 (avg):2994.1 Mh/s | Q:4149  A:1364  R:19  HW:0  E:29%  U:20.83/m
 TQ: 10  ST: 11  SS: 8  DW: 1957  NB: 5  LW: 4549  GF: 1  RF: 0
 Connected to http://192.168.0.189:9332 with LP as user user/1000+1
 Block: 000007efc04c94ca1185fdc1e6565e85...  Started: [12:43:03]
--------------------------------------------------------------------------------
 [P]ool management [G]PU management [S]ettings [D]isplay options [Q]uit
 GPU 0:  47.0C  960RPM | 371.2/370.9Mh/s | A:199 R:2 HW:0 U:   2.84/m I: 8
 GPU 1:  48.0C  960RPM | 371.1/370.8Mh/s | A:184 R:5 HW:0 U:   2.63/m I: 8
 GPU 0:  47.0C  960RPM | 371.2/370.9Mh/s | A:200 R:1 HW:0 U:   2.86/m I: 8
 GPU 1:  48.0C  960RPM | 371.1/370.8Mh/s | A:182 R:3 HW:0 U:   2.63/m I: 8
 GPU 4:  45.0C  960RPM | 371.1/370.8Mh/s | A:179 R:1 HW:0 U:   2.55/m I: 8
 GPU 5:  48.5C  960RPM | 371.2/370.8Mh/s | A:184 R:1 HW:0 U:   2.63/m I: 8
 GPU 6:  49.0C  960RPM | 371.2/370.8Mh/s | A:170 R:4 HW:0 U:   2.43/m I: 8
 GPU 7:  48.0C  960RPM | 371.1/370.8Mh/s | A:174 R:2 HW:0 U:   2.48/m I: 8
--------------------------------------------------------------------------------

[2012-03-23 12:53:14] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:19] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:21] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:28] Accepted 00000000.5031a643.261ef557 GPU 0 thread 0 pool 0
[2012-03-23 12:53:33] Accepted 00000000.29cdb6c5.b7edd474 GPU 6 thread 6 pool 0

Clocks are 830/150 at stock voltage.  (Most of cards were stable @ 870 but I decided to start low).
I am planning on eventually overvolting them slightly but that requires a bios flash.  I think 900/150 @ 1.125V should be possible which should give me 3.2 GH/s per rig.

Ambient is about 33C.  I have the garage closed with no AC to simulate hotter ambient temps of this summer.  I don't have my radiator fan yet so this is using a 14" desktop fan pushing minimal air through radiator.

Looking good so far.

Whats next:
Hopefully this weekend I can get the second rig connected to the loop.  Also want to mount the manifold to project plywood and start planning the PEX runs.  I also need to go grab the server rack I bought off a friend.   The next phase involves 20 danger den waterblocks (sound a bulk discount of $99 per block), 5 more server chassis, 50ft of tubing, and some fittings.

[malevolent]

I am planning on eventually overvolting them slightly but that requires a bios flash.  I think 900/150 @ 1.125V should be possible which should give me 3.2 GH/s per rig.

You sure it is worth to overvolt considering the extra power draw?

looks nice btw

[DeathAndTaxes]

You sure it is worth to overvolt considering the extra power draw?

Yeah at least now.  As price/difficulty rises I can move to "normal volt" and eventually undervolting.

Right now the rigs get ~3 MH/W and my electrical rates are $0.10 per kWh. 

That puts my mining cost (electrical only) at:

MH per bitcoin =  (1498294.36282  * 2^32) / (50 * 1000^2) = 128702505.8
kWh per Bitcoin = 128702505.8 / (60 * 60  * 3 * 1000) = 11.9
energy cost per Bitcoin = 11.9 * 0.10 = ~$1.20

Gross margin is = $5.00 - $1.20 = $3.80

So while overvolting will reduce the margin I will produce more coins.  It is only once you start getting close to break even when overvolting becomes unprofitable. 

Overvolting is just something planned.  Once I measure power draw and stable clocks I can get some exact # of what my margins are at various over & undervolts.

TL/DR version:
My thinking is to overvolt and increase gross revenue.  As the rest of network becomes more efficient I can lower voltage to say ahead of the efficiency curve.  Eventually (say 12 to 18 months from now) I may need to be running at much lower hashrate and voltage into order to remain profitable.  I can dynamically balance efficiency vs gross hashing power based on the network's price/difficulty ratio.