(Updated w/ pics) Watercooled Rack of Servers - 50% completed

[DeathAndTaxes]

So the original thread is here (but the plan changed a lot and the thread got sidetracked at points w/ flames & trolls)
https://bitcointalk.org/index.php?topic=66606.0

The Purpose (also know as "are you fraking crazy FPGA are going to take over the world ... ")
I ALREADY have a farm of six 3x5970 rigs plus another 6x 5970s cards looking for a home.  The hardware has long since been paid off and I have some measure of pride knowing the rigs I built are just about as efficient as is possible with GPU tech (although some optimizes 7970 rigs might edge them out).  The goal is to extend that investment as long as possible.  Kinda a GPU "last stand" against the encroaching FPGA hoards  

My goal was to find a way to:
a) survive the summer without the 1/3rd increase in electrical cost
b) avoid paying ~$3K to the upgrade AC system (farm was expanded in the fall so current AC won't be able to handle it)
c) have fun building a big insane system.
d) see if it would be possible to capture some of that heat for preheating hot water heater and/or heating the house.

So I had this crazy idea.  Can you put 24+ HD 5970s in a standard server rack and using a really powerful pump move all that heat (18,000 BTUs) outside?

Well the farm is 50% converted.

The Rack
From top to bottom is APC PDU, management server, gigabit switch, second APC PDU, and the first three water cooled rigs.


Closeup of the rigs
No I am not cooling it with milk.  I guess the flash made it look like that.  Just using plain distilled water.


Connectivity & Power
I need some better cable management but power on left, connectivity on the right is a good start.  I hope to eventually replace all the flash drives with PXE.


Manifold
First three rigs are hooked to the three left most ports.  Isolation valves on top, flow control valves on bottom.
The connected loop on the right ensures even if I absent mindedly disconnect all 3 rigs the pumps will still get about 1gpm flow (important to keep the pump cool).  The photo is kinda hard to tell but this is mounted on the right hand side of the rack when looking from the POV of the first photo.
http://i.minus.com/iBLM12pskhT2g.png

The pump
A real monster but quiet and has a nearly silent fan in the back which reduces the amount of heat which ends up in the loop.  A pair of unions simplify hooking it up to the loop. The pump and fan are the only component which can't run on 240 which necessitates running 120V extension cord.


The radiator
Has about 16x the pipe surface area as your "normal" 3x120mm radiator and about 6x the fin surface area.     I am still using the poly tubing but will be replacing it with PEX when I get the time.  Fan is just a "normal" floor fan.  Probably 1800 cfm.  I may replace it with a car radiator fan or maybe just a outdoor rated workspace fan.



The rack contains:
3x quad 5970s rigs
2x APC 9571 PDU (240V, 30A, 80% derate) for 11.5 KW usable power
1x Cisco 24 port gigabit switch
1x 1U watchdog server (runs bitcoind, p2pool, remote power relay card, and monitoring software)

Current config:
12 5970s (24 GPUs) ~ 9.2 GH/s

When complete:  
24 5970s (48 GPUs) ~ 18-20 GH/s

Theoretical max:
40 5970s (80 GPUs) ~ 30-32 GH/s

[1714805796]

1714805796

[1714805796]

1714805796

[1714805796]

1714805796

[Uncurlhalo]

That thing looks so sexy. I want to see it full of shinny electronics.

[rjk]

Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.

[DeathAndTaxes]

Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.

Yeah the bottom one is done the way I want it. I took the 3.5" plate and drilled 2 holes.  One half of the quick disconnect has a panel mount adapter so it mount directly to the plate.  The plate is then monted in the drive cage.  This makes the internal tubing loop fixed.  Less chance of moving around, stressing the line, or causing a leak.  Right now if you yanked on the tubing for the bottom rig the worst you could do is cause a leak outside the rig.

However the steel is pretty tough. It wore out my cheapo bit after the first plate.   Need to get a stronger bit to drill out the rest.

[check_status]

Looking good so far. Do you have any pics of the other end of your watercooling setup?

[DeathAndTaxes]

Looking good so far. Do you have any pics of the other end of your watercooling setup?

Yeah should be up this evening.

[twoBitBasher]

You know "this evening" is pretty useless without specifying a timezone. For me(gmt+2) "evening" was technically several hours ago 

[Cablez]

How about wait for evening EST? 

[rjk]

How about wait for evening EST? 

That would be now...

[DeathAndTaxes]

Sorry guys.  By midnight EST.  Guess I should have been more specific.

[twoBitBasher]

I wonder why haven't these php forum coders thought about this multi timezone operation. Everyone probably do set the timezone on their profile so it wouldn't be too difficult to show it in posts with the user info on the left. Most regional forums do just fine without it but Bitcointalk is hardly regional.

Edit: okay my bad. It does show if you check users profiles but that's a chore.

[DeathAndTaxes]

Updated first post w/ more pics & details.

[rjk]

Updated first post w/ more pics & details.

Very nice. I just have been re-pasting my cards (because I had to remove the heatsinks in order to take the plastic bits off), and I can only imagine what hell it is to do so many. I've been keeping the same old thermal tape and pads, but even then I can only do 2 at a time before getting frustrated and having to take a break. Have done a dozen so far.

Maybe if/when you switch to 7990s I will be able to buy your 5970s with the blocks already applied to stuff in my rig. Wonder how much they will be worth then.

[twoBitBasher]

Heh that beast of yours do deserve watercooling. It would be the most epic watercooling rig ever. I'd say if you play your cards right you might be able to get somebody sponsor those blocks for you rjk  I bet it could get a lot of coverage on IT-media.
I just wonder what kind of scores would both of yours (dt & rjk) rigs do folding. Top 10?

Car radiator fan is probably not designed for continuous use. I'd say go for the workspace fan.

[coretechs]

Given the can of motor oil on the shelf, I assume this is in your garage?

It's looking great.  Any plans for a backup pump or will you just do software throttling?

[rjk]

I just wonder what kind of scores would both of yours (dt & rjk) rigs do folding. Top 10?

Aha, I doubt it. Folding uses CPUs and Nvidia hardware, which are no good for mining. Mining hardware is AMD, and I doubt that it would fold vary well, although I don't know a whole lot about the folding scene.

[naz86]

When you exchange the heat into you warm water for your house you can make your rigs more efficient. This is what i am planning to do with the heat of my serverroom.... in germany its called luft-wärme-pumpe... in english it could be air-heat-exchanger (or something similar). your efficiency is even higher if you can exchange the heat of water to water....

[DeathAndTaxes]

Given the can of motor oil on the shelf, I assume this is in your garage?

Yeah.

Any plans for a backup pump or will you just do software throttling?

No plans for backup pump at this point.  I have the ability to cut power to the rigs (the reason among others for the watchdog server).  I intend to install both a current transformer and flow meter.  In a pump failure situation the relay board in the watchdog server will trip power on all 6 rigs.

cgminer is already set to do software throttling (reduce clock @ 60C, idle thread @ 70C) but the "power kill" provides a further layer of protection.  It likely isn't necessary but it does buy some piece of mind when you go $4K worth of GPU relying of coolant.

This weekend I intend to waterblock some/all of the last 12 GPUs.  God I am not looking forward to that.

[DeathAndTaxes]

When you exchange the heat into you warm water for your house you can make your rigs more efficient. This is what i am planning to do with the heat of my serverroom.... in germany its called luft-wärme-pumpe... in english it could be air-heat-exchanger (or something similar). your efficiency is even higher if you can exchange the heat of water to water....

Yeah that is the goal once this is done and stable.  A water-water heat exchanger to preheat the cold water line of hot water heater. 

[twoBitBasher]

Let's say you preheat your water. You got 6kw of constant heat coming. Your water boiler doesn't heat the water constantly but just when it's temperature drops below a certain limit(when you use water). What happens to your excess heat? You don't plan on leaving your tap water running do you?

I guess it needs some kind of fancy temperature controlled bypass valve.