A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[BFL-Engineer]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs

Regards,

That's what I figured when I ordered 6 units from you a few days ago. Assuming they get here within the "4 to 6 week" period, I will have a very nice high-hash, low-power and low noise upgrade to my mining rigs within a few weeks.

BTW, it's nice to see you on the forum, BFL-Engineer (Sonny, I presume).

Keep shippin' em' and I'll keep orderin' em as long as I get them within reasonable delays.

Sonny is in sales, I'm in engineering. We'll try our best to satisfy customers. Efforts are
under way to reduce shipping delay.


Regards,

[Brian DeLoach]

I had considered going under water for my 10 Ghash/s, but since I have none of the blocks or rads, it was going to be way too expensive. Instead I went the ghetto route:





That's a 12", 1000 cfm vortex fan (pulling 350w). It is far cheaper and easier to suck the hot air outside. The whole thing (rack, fans, foamboard, tape, etc) took $500 and a weekend to build. I live in S. Florida and this is my solution for the summer. It used to get over 100F in that room with only 4 rigs, now it is maybe 5 degrees warmer than the rest of the house on a hot day with 7 rigs. This may or may not help you, and watercooling is a hell of a lot neater, but from a maximum profit point of view it is cheaper to move a large quantity of air over those cards than it is to water cool them. And yes, that fan is louder than anything any video card . Anyhow, just throwing this out there for ideas. Good luck.

When did you build this? It looks unbelievably similar to mine.

[Isepick]

When did you build this? It looks unbelievably similar to mine.

3 or 4 weeks ago, and you were the direct inspiration for it. I meant to do a build thread about it, but got lazy. It works pretty good for me. I had wanted to relay my positive experience with this design to DAT in case he wanted to try a much cheaper route. As I said, the temperature the room this is in used to be unbearable (hot enough to melt candle wax!), now it is tolerable with even more rigs in it. WAF greatly increased since I pulled 2 rigs off the kitchen table and dropped the ambient temp in the living/dining room area by several degrees.

[DeathAndTaxes]

Isepick that is a nice method to force the waste heat outside  I am looking for quiet and I think that kind of setup exceeds the WAF boundary at the D&T hashing farm.  Plus I would like to see how much waste heat I can capture and use to heat hot water for the house.

If I did the math right using a heat exchanger before the cold water intake of the hot water heater, at 2gpm I could raise the water temp from 60F to 105F before it ever gets to the hot water heater.

[DeathAndTaxes]

An update.  

Rig has been solid for 36 hours now.  I have increased the clocks and get ~ 3.125 GH/s on 1120W 120VAC.  When moving it to 240V PDU wattage dropped to 1090W (2.86 MH/W).   Backing out the system non-GPU idle of 180W puts GPU load at ~230W per card giving GPU efficiency of 3.40 MH/W.

If we assume ~10% PSU inefficiency that's 980W DC putting PSU load at 78%.  Since it is a SeaSonic I am comfortable with it.  The SeaSonic 1250 hasn't disappointed.  Exhaust is warm but not excessively even with the cramped airflow of a rackmounted case.

I have been running cgminer with auto-gpu enabled and a target temp of 62C.   The dynamic is kinda interesting.  The cards are rock solid but as one card goes over 62C it gets the downclocked by cgminer.   The reduced thermal load cools all the GPUs and puts one of them under the target so cgminer raises the clock and and the process repeats.  Kinda a "musical clocks".  The load remains ~1.1KW and hashrate ~3.1GH/s the entire time with different GPUs taking the lead.

The data is showing (which I already knew) that this radiator is simply not designed for a heat load this large.  The fact that I have low RPM fans isn't helping.  I won't be able to push clocks higher without better thermal dispersion but based on the solid performance I think 3.25 GH/s is possible at stock voltage.  I will try using some Scythe Ultra Kaze temporarily to see if I can get the last little bit out of the radiator but we might be reaching the limit of what 4x120mm radiator can dissipate..

Somewhat disapointing is that one GPU won't go above 800 MHz even with low temps.  It just goes instantly sick.  Sadly raising voltage in Linux isn't possible without BIOS flash.  Hopefully cores like that are a minority.

[Isepick]

Isepick that is a good concept to force the waste heat outside  I am looking for quiet and I think that kind of setup exceeds the WAF boundary at the D&T hashing farm.

ROFL. I understand, although when the door to the room is closed you can't really hear it so much, and when it is open its more of a white noise type of deal. And I have to credit Mr. DeLoach for the design, as I probably wouldn't have thought of it on my own . Anyhow, good luck, I am looking forward to seeing where you end up with all this.

[sveetsnelda]

When did you build this? It looks unbelievably similar to mine.

Then both of you need to clean up your rigs.  Sheesh.     Okay, I don't know about Brian's, but what is going on in that picture?  There are fans blowing at the *exhaust* side of quite a few cards!  The fans are just fighting each other..

This is what halfway clean rigs look like.  Pink rope is a requirement:

   

[jamesg]

This is what halfway clean rigs look like.

Looks pretty clean. I think my cases are a tiny bit sturdier than yours though. I have also removed every extra fan out of my setup by undervolting a few rigs. Maybe I'll make a video.

Now back to our regularly scheduled thread topic.

[Brian DeLoach]

3 or 4 weeks ago, and you were the direct inspiration for it.

I'm very happy to see the design replicated!

I kept thinking how almost every detail was spot on, it would have been a huge coincidence if you never saw my rigs. Why is that vortex fan green? As far as I know the company only makes them in gray.

Then both of you need to clean up your rigs.  Sheesh.     Okay, I don't know about Brian's, but what is going on in that picture?  There are fans blowing at the *exhaust* side of quite a few cards!  The fans are just fighting each other..

I actually thought a lot about this. Just through touch I could feel that each end of the card was exhausting heat, even more so for the power cable side (away from the interface). I, like you, always thought of the interface side to be the exhaust side. I even took a smoke machine and introduced some smoke at the fan intake to see where it went. It was about an equal distribution to both sides. Couple that with the fact that I wanted access to the interface side, along with the USB and other components on the motherboard, I decided to suck the heat away from the motherboard/gpu interface.

From the photos, you can see that I sandwiched the GPUs as closely as I could. I did this to maximize the airflow across the GPUs and increase the rate at which fresh air was introduced to the fans.

This is what halfway clean rigs look like.  Pink rope is a requirement:

Your setup looks very clean, but doesn't deal with exhaust heat at all. Isepick and I are down south and heat will become a major problem in a few short months. You also use many fans. I think using one big powerful fan to deal with all the heat is the way to go.

Now back to our regularly scheduled thread topic.

Starting now.

[sveetsnelda]

Looks pretty clean. I think my cases are a tiny bit sturdier than yours though. I have also removed every extra fan out of my setup by undervolting a few rigs. Maybe I'll make a video.

Yeah, you've certainly got a sturdier rack built.  If we both had an earthquake, you'd win.  We don't really get any here though. 
I'm going for cheap & effective, and my time is a factor.  Setting up a new rig takes me about 20 minutes from unboxing to mining.

Now back to our regularly scheduled thread topic.

  It wasn't meant to be a tread hijack.  I just saw someone with blowing fans at the exhaust side of their cards and couldn't help myself.  I had a non-ref card die a while ago and used a reference card in its place for about a week while waiting for a replacement.  Because the card had to face the wrong direction in relation to the others, that GPU fan had to REALLY work hard to remove the air.  Flipping the card around would have *easily* cut the fan speed in half.

[sveetsnelda]

I actually thought a lot about this. Just through touch I could feel that each end of the card was exhausting heat, even more so for the power cable side (away from the interface).

If you've got reference cards exhausting heat through the inlet side, something is horribly wrong.  There shouldn't be any.

I, like you, always thought of the interface side to be the exhaust side.

Er...  that's because it IS the exhaust side (on reference boards...  6990 is a different story). 

I even took a smoke machine and introduced some smoke at the fan intake to see where it went. It was about an equal distribution to both sides.

Sounds like you're using non-reference cards?

I decided to suck the heat away from the motherboard/gpu interface.

That's exactly what you should be doing.  It looks like your fans are blowing at the gpu interface side though?

Your setup looks very clean, but doesn't deal with exhaust heat at all. Isepick and I are down south and heat will become a major problem in a few short months. You also use many fans. I think using one big powerful fan to deal with all the heat is the way to go.

My setup deals with the exhaust heat perfectly.  My hottest card as I type is 74 degrees and the fan is sitting at 40 percent.  This is because it's in the garage.  The non-reference cards in the house are running around 64.0C at only 1166RPM.

The exhaust heat from all of the cards gets pushed out into the hallway and back out the window of an adjacent room.  Since it goes through the hallway, some of the heat gets recirculated throughout the house.  I haven't used the furnace at all this winter (and it gets very cold here).  This type of setup requires that nobody opens a window somewhere else in the house though...  otherwise, the air will go in a different direction.

My shelves are sitting on sliders, so when summer comes around, I just flip the rack around and exhaust the heat out the window.  It gets in the high 90's here and I was cooling the whole rack with a 230 watt evaporative cooler last summer.  The rigs in the garage will be a problem though if I don't move them before summer hits.  I'm building a small shop though to house them all.

Starting now.

Starting...   now?   

[check_status]

@DAT
You can reduce your cooling cost by using the earth as a heat sink and temperatures will always be in the high 50's or low 60's.

The "Earth Coupled Air Tube" technology is a low cost solution to reduce the cost of heating and cooling your home. By using the Earth as a heat sink anyone can heat and cool their house for less. Unlike the ground loop heat pump, air tubes don't require deep wells, compressors, pumps, Storage tanks, coils, heat exchangers, complex plumbing, or all the problems inherent in a complex equipment/technology intensive heating and cooling system. Properly installed air tubes don't have any moving parts. They can't break. The only technology required is a fan to move the air through the tubes and into the house. Earth tubes are relatively inexpensive to install and are inexpensive to operate.

http://earthairtubes.com/

Your cost of operating this is the cost of running a fan. sweet!

[DeathAndTaxes]

You can reduce your cooling cost by using the earth as a heat sink and temperatures will always be in the high 50's or low 60's.

http://earthairtubes.com/

Your cost of operating this is the cost of running a fan. sweet!

That is a little beyond my abilities in terms of earth moving and engineering but a pretty awesome concept.  If I had more land I might look into that for my entire house.

[jjiimm_64]

gota love the 5970's..  this one has been up since the 4th, restarted it then for a cgminer upgrade.

Code:

cgminer version 2.2.1 - Started: [February 4, 2012, 4:21 pm]    Rig: miner10
(5s):2928.27  (avg): 2929.3 Mh/s  |    Q:1586723   A:1491382   R:14116   HW:0   E:?%   U:40.08/m
TQ:?   ST:2979   SS:?   DW:42517   NB:3888   LW:61507   GF:1667   RF:2651
Connected to http://gpumax.com:8332 with LP as user ?

GPU 0:	74.0C	3591RPM 	60% | 	366.2/362.4Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:185253	R:1630	HW:0	U:4.98/m 	I: 7
GPU 1:	70.5C	3591RPM 	60% | 	366.1/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186079	R:1733	HW:0	U:5.00/m 	I: 7
GPU 2:	73.5C	3290RPM 	54% | 	366.1/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186059	R:1759	HW:0	U:5.00/m 	I: 7
GPU 3:	73.0C	3290RPM 	54% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186431	R:1732	HW:0	U:5.01/m 	I: 7
GPU 4:	72.0C	2947RPM 	49% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186842	R:1784	HW:0	U:5.02/m 	I: 7
GPU 5:	72.5C	2947RPM 	49% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:187012	R:1753	HW:0	U:5.03/m 	I: 7
GPU 6:	73.0C	3133RPM 	56% | 	365.9/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186618	R:1829	HW:0	U:5.02/m 	I: 7
GPU 7:	68.5C	3133RPM 	56% | 	365.9/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:187088	R:1896	HW:0	U:5.03/m 	I: 7

[DeathAndTaxes]

Is that a customized version of cgminer?  The frequency & voltage columns.  I like that layout.    I think current layout in cgminer needs some work.  The [g]pu menu takes up too much vertical space especially w/ 8 GPUs.

[jjiimm_64]

Is that a customized version of cgminer?  The frequency & voltage columns.  I like that layout.    I think current layout in cgminer needs some work.  The [g]pu menu takes up too much vertical space especially w/ 8 GPUs.

webmon:

[DeathAndTaxes]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger



Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer. 

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

[rjk]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer. 

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

[DeathAndTaxes]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer.  

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

Will do.  I have a crimping tool from some home remodeling work already.  I will likely make a test loop involving just pump, radiator, and heat exchanger and leave it running.  The reason I would like to use PEX is I could take an off the shelf radiant heating manifold and use that for splitting the run to each rig.  

[ThiagoCMC]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago