(Updated w/ pics) Watercooled Rack of Servers - 50% completed

[cmg5461]

It comes from the river, it goes to the river. I believe is better than to run the air conditioner non-stop.

brilliant idea, really  

According to Nature (2010), about 80% of the world's population (5.6 billion in 2011) live in areas with threats to water security.

How do you deal with the condensation?  Faucet water is much colder than ambient temps

[1713585745]

1713585745

[1713585745]

1713585745

[1713585745]

1713585745

[DeathAndTaxes]

It comes from the river, it goes to the river. I believe is better than to run the air conditioner non-stop.

brilliant idea, really   

According to Nature (2010), about 80% of the world's population (5.6 billion in 2011) live in areas with threats to water security.

How do you deal with the condensation?  Faucet water is much colder than ambient temps

I am not saying he is doing this but with a heat exchanger you can make the GPU loop any temp you want.  Keep GPU loop's flow fixed and adjust the cold side loop (the cold water -> heat exchanger -> drain) flow so that it pulls enough heat from the hot side (GPU loop) to cool it down but keep it above ambient.

[ssateneth]

I'm super jelly that he can just use fresh cold tap water to cool everything instead of us normal people that need to screw around with a radiator, a pump, and a reservoir. Must be soooooo nice having free water. Dont even need air conditioning! All the heat goes down the drain!

[twoBitBasher]

he still needs pump and reservoir, just not radiator as the heat exchanger is the "radiator"

[ciuciu]

I'm doing exactly what you said.
I lost a card a few months ago to condensation. I wasn't home and the Internet went out. The cards got very cold and condensation happened. The lost card was using an XSPC block, all EKs were OK. For this I try to no longer use XSPC, they sure have a condensation problem.

It comes from the river, it goes to the river. I believe is better than to run the air conditioner non-stop.

brilliant idea, really   

According to Nature (2010), about 80% of the world's population (5.6 billion in 2011) live in areas with threats to water security.

How do you deal with the condensation?  Faucet water is much colder than ambient temps

I am not saying he is doing this but with a heat exchanger you can make the GPU loop any temp you want.  Keep GPU loop's flow fixed and adjust the cold side loop (the cold water -> heat exchanger -> drain) flow so that it pulls enough heat from the hot side (GPU loop) to cool it down but keep it above ambient.

[twoBitBasher]

Okay, now that was good to know! This could easily happen to DT as well if you put your rad outside.. Although 24 GPUs would draw about 1060w while idle so there would still be a good amount of heat going in to the loop but if the fan blowing through the the rad keeps running full bore after the net is out temps would still drop rapidly and low.
A kill switch shouldn't be too difficult to do.

Ek blocks are half plastic but condensation would still happen on the steel plate some of the blocks have on top(5870V2 & 7970). Some condensation would also happen on the sides. I think the under side facing the card is getting some heat  and the air doesn't move there as much so it is probably ok.

[cmg5461]

Why don't you wrap the tubes and cover the blocks with cellophane?

[twoBitBasher]

Why don't you wrap the tubes and cover the blocks with cellophane?

I think that would make the rigs a nightmare to maintain. Besides as it isn't the purpose to do extreme cooling here simple temp controlled fan killswitch would do. And a mag valve would do for the tap water cooler dude.

[twoBitBasher]

DT: how many cards did you manage to convert during the weekend ?

[DeathAndTaxes]

DT: how many cards did you manage to convert during the weekend ?

Another 4.  Fourth rig is installed.  16 down, 8 to go.  God I hate installing waterblocks.    If FPGA didn't exist and I planned to rig up an entire datacenter like this I would have to find someone to install blocks for me.

Will provide some updated pics this evening.

[twoBitBasher]

So I guess this was with the Swifttech links. Were they hard to press together vs phobya links?

[DeathAndTaxes]

So I guess this was with the Swifttech links. Were they hard to press together vs phobya links?

Yes this is the first rig to use the Swiftechs. 

The link has some resistance when adjusting the width of the link but not much. 

I found it easiest to install it this way.
1) Unscrew the "collar" (gives you more room to get wrench on the connector).
2) Attach connector to the waterblock (side closest to the collar) *
3) Tighten w/ wrench according to spec.
4) Shorten the link as far as possible
4) Insert all cards into the rig (each is connected on only one side)
5) Attach other side of link **
6) Tighten w/ wrench according to spec

* My water blocks (Danger Den 5970) have a plastic top so I attached the links to that side of each card first (outside of rig)
** This is actually the hardest part.  Kinda hard to get it started (small space, resistance from extending the link and getting threads started).

So far I am really impressed with these links.   They are very very similar to the Koolance links ($15 ea) but are fitted to allow tightening with standard wrenches.  I honestly can't believe they are only $8. 

[twoBitBasher]

In your case when there was nothing connected to the motherboard wouldn't it have been easier to just tighten the links outside of the rig -> attach all the cards together -> place the whole stack on the mobo ?

[DeathAndTaxes]

In your case when there was nothing connected to the motherboard wouldn't it have been easier to just tighten the links outside of the rig -> attach all the cards together -> place the whole stack on the mobo ?

I have never had luck with that.  If you got 2 card you can do that pretty easy.  But with 4 card trying to get all 4 to line up right is tough.  You kinda need 5 hands.

[deepceleron]

Probably the cleverest way to preheat hot water would be to get an old second water heater and put it before your main water heater - you can probably find a craigslist one with a burned out element or such. Instead of a heater element inside it, you would put a "radiator" inside of it for your water cooling. Since you can't open up the tank to put in a normal radiator, perhaps a dozen feet of pex or copper tubing inside the thing circulating your coolant would do the job. The best kind to modify would be an electric with two heater elements, as pictured below, so one element can be your coolant inlet, and one can be your outlet. You remove the elements, fish tubing from the top to the bottom, and shove as much extra pex as will coil around inside the water heater. Then the trick will be finding fittings that let the piping pass into the water tank and still seal the inlets where the tubing passes so they will hold water pressure.

Note that the city water pressure will want to collapse the low-pressure coolant tubing. Although it should resist this crushing pressure, your coolant pump should be right before the water heater pumping coolant into the exchanger. Also note that if the tubing/radiator inside the water pre-heater ruptures, this will allow 60-80psi of water pressure into your coolant lines, so an emergency coolant pressure relief valve should be installed where it can let out water to prevent a hose blowout inside your racks.

With such a setup, you could preheat 15C water up to 40-50C, as warm as the coolant coming out of the systems.

[MrTeal]

Probably the cleverest way to preheat hot water would be to get an old second water heater and put it before your main water heater - you can probably find a craigslist one with a burned out element or such. Instead of a heater element inside it, you would put a "radiator" inside of it for your water cooling. Since you can't open up the tank to put in a normal radiator, perhaps a dozen feet of pex or copper tubing inside the thing circulating your coolant would do the job. The best kind to modify would be an electric with two heater elements, as pictured below, so one element can be your coolant inlet, and one can be your outlet. You remove the elements, fish tubing from the top to the bottom, and shove as much extra pex as will coil around inside the water heater. Then the trick will be finding fittings that let the piping pass into the water tank and still seal the inlets where the tubing passes so they will hold water pressure.

Note that the city water pressure will want to collapse the low-pressure coolant tubing. Although it should resist this crushing pressure, your coolant pump should be right before the water heater pumping coolant into the exchanger. Also note that if the tubing/radiator inside the water pre-heater ruptures, this will allow 60-80psi of water pressure into your coolant lines, so an emergency coolant pressure relief valve should be installed where it can let out water to prevent a hose blowout inside your racks.

With such a setup, you could preheat 15C water up to 40-50C, as warm as the coolant coming out of the systems.

That's actually a really ingenious solution. For a radiator, is there any reason you couldn't take the top off and either insert a real rad or a nice coil of 1/2" copper? You're using it as a tank and insulator, so even if the tank is sealed it's nothing that a Sawzall won't fix. The copper tubing would have no problem surviving 30-40PSI of external pressure.

[twoBitBasher]

If you think about this water boiler business real hard you'll come to the conclusion that this second boiler is just stupid over  the multiple heat exchanger idea. This doesn't save any power over the plate exchanger it just takes a huge amount of space.

In theory this would allow for greater delta T because it has longer time to build up the heat. Adding enough plate exchangers would do roughly the same however.

[MrTeal]

If you think about this water boiler business real hard you'll come to the conclusion that this second boiler is just stupid over  the multiple heat exchanger idea. This doesn't save any power over the plate exchanger it just takes a huge amount of space.

In theory this would allow for greater delta T because it has longer time to build up the heat. Adding enough plate exchangers would do roughly the same however.

Are you sure on that? With the extra water heater, you should be able to get the preheater to the same temperature as your water block exhaust temperature. With plate exchangers, you're still limited by the power of your loop. If you're running the hot water full out and pulling say 10lpm, you would need to dump ~700W of heat through the exchanger to raise the water 1 degree C. You would need a massive amount of plate exchangers and a lot of water in your cooling loop in order to raise the water temperature to close to the exhaust temperature. That sounds a lot more expensive than getting a free or very cheap water heater and some copper tubing, and much more likely to cause pressure issues on your hot water line.

[DeathAndTaxes]

Are you sure on that? With the extra water heater, you should be able to get the preheater to the same temperature as your water block exhaust temperature. With plate exchangers, you're still limited by the power of your loop. If you're running the hot water full out and pulling say 10lpm, you would need to dump ~700W of heat through the exchanger to raise the water 1 degree C. You would need a massive amount of plate exchangers and a lot of water in your cooling loop in order to raise the water temperature to close to the exhaust temperature. That sounds a lot more expensive than getting a free or very cheap water heater and some copper tubing, and much more likely to cause pressure issues on your hot water line.

Why?

Flat plate heat exchanger on the cold water line cost ~$50 and provides a massive Delta T to the cold water.  It recovers most of the energy needed and is both cheap and simple.  The payback period is measure in weeks.  While one can do more remember hot water isn't that expensive maybe $300 per year so complex projects start cutting into your ROI% pretty quickly.

While there are ways to recover more of the heat I just don't see them as viable.  Hacking up an old hot water heater of dubious quality, building your own radiator, running copper loop (not that cheap these days), jury rigging some water tight seals all to get maybe another 10% to 20% of annual energy budget?  It doesn't seem worth it.    I am going to keep it simple.

[MrTeal]

If you think about this water boiler business real hard you'll come to the conclusion that this second boiler is just stupid over  the multiple heat exchanger idea. This doesn't save any power over the plate exchanger it just takes a huge amount of space.

In theory this would allow for greater delta T because it has longer time to build up the heat. Adding enough plate exchangers would do roughly the same however.

Are you sure on that? With the extra water heater, you should be able to get the preheater to the same temperature as your water block exhaust temperature. With plate exchangers, you're still limited by the power of your loop. If you're running the hot water full out and pulling say 10lpm, you would need to dump ~700W of heat through the exchanger to raise the water 1 degree C. You would need a massive amount of plate exchangers and a lot of water in your cooling loop in order to raise the water temperature to close to the exhaust temperature. That sounds a lot more expensive than getting a free or very cheap water heater and some copper tubing, and much more likely to cause pressure issues on your hot water line.

Why?

700W/C is pretty insignificant when you have a 6KW heat source.

The reality is the amount of energy used in domestic water heating is a tiny fraction of the heat produced by mining.  Thus getting ultra max efficiency is silly & expensive for no real tangible benefit. 

I did the math already.  A $50 flat plate heat exchange will dump about 90%+ of the energy I used each month to heat my hot water.  Spending hundreds more to try and get the last 10% (most of which is simply unobtainable).

Your belief that you need a massive heat exchange or more water in the loop (?) is simply incorrect.  The amount of water in the loop is immaterial.  1 gallon in the loop or 20,000 gallons in the loop.  The amount of ENERGY in the loop is the same (it is the amount of energy used by the rigs).

I you're wrong on the last point. Consider a simple closed loop system where you have a pump, a block and a radiator. Your exhaust temperature will be a function of flow rate and power. Whether you have 2 ft of tubing between the GPUs and radiator or 20 ft, as long as you're not dissipating much heat through through the tubing the water between the blocks and radiator would be the same. You have 10x as much water in the tubing in the second case though, and thus 10x as much stored energy.

I don't see why that idea would be expensive though. It seems like something that could be done for $20, and perform as well or better than a large plate exchanger. The biggest downside is the floor space and low SAF.