A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[PulsedMedia]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

[1713449240]

1713449240

[1713449240]

1713449240

[1713449240]

1713449240

[1713449240]

1713449240

[1713449240]

1713449240

[shakaru]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

[cuz0882]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

[PulsedMedia]

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Useless to spent all that time researching, pretty much use any you feel comfortable, or which gives the nicest color in your senses
Forget distilled water, useless waste of money & time for cooling purposes.

Just add at least 1%, but don't exceed 50%.

I'm not a fan of watercooling computers, too much work to maintain and hassle in regular desktops. For servers it makes sense tho.
I recommend the red glycol (i keep forgetting the names lol) because it has the highest boiling point and best protective for modern engines, but for computer watercooling any automotive glycol will work just as fine.

[PulsedMedia]

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

Yet cars have been utilizing both metals mixed in the same system, in the same cooling system for decades upon decades. Those work wonderfully, and the coolant channels in cylinder heads are WAY smaller than your CPU block has, often a diameter of just 5mm in many parts!

So now you are using something which is a strange physical phenomenon known by some sailors to happen in sea?

Coolant, proper glycol, is to be used aswell to minimize the build up caused by bacteria etc.
Some glycols themselves will build up, and still causes dirt to apper in the coolant, hence, the coolant ought to be swapped annually, maybe accompanied by a flushing the system.

Aluminium does oxidize, but it doesn't matter at all in a cooling system, especially if you use red glycol which has properties to protect from that.

Might be a worthwhile read: http://www.gewater.com/handbook/cooling_water_systems/ch_32_closed.jsp
Speaks in length about corrosion in cooling systems.

All you really need is a bottle of cheap glycol and tap water, and annually change the coolant.

[cuz0882]

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

Yet cars have been utilizing both metals mixed in the same system, in the same cooling system for decades upon decades. Those work wonderfully, and the coolant channels in cylinder heads are WAY smaller than your CPU block has, often a diameter of just 5mm in many parts!

So now you are using something which is a strange physical phenomenon known by some sailors to happen in sea?

Coolant, proper glycol, is to be used aswell to minimize the build up caused by bacteria etc.
Some glycols themselves will build up, and still causes dirt to apper in the coolant, hence, the coolant ought to be swapped annually, maybe accompanied by a flushing the system.

Aluminium does oxidize, but it doesn't matter at all in a cooling system, especially if you use red glycol which has properties to protect from that.

Might be a worthwhile read: http://www.gewater.com/handbook/cooling_water_systems/ch_32_closed.jsp
Speaks in length about corrosion in cooling systems.

All you really need is a bottle of cheap glycol and tap water, and annually change the coolant.

Ya, that sounds like a great idea! Maybe I will just put some stop-leak in my pc if a seal goes bad, or maybe even a little quicksilver. What kind of redneck trailor park crap are you talking about? Have you even looked inside of a cpu block? There is a reason why everyone is not using 50/50 antifreeze mix in there pc's. What you said is a clear example of what not to do.. Look it up.

[PulsedMedia]

Dude, obviously you do not have a clue about the underlying physics going on.
FYI, i've even made blocks myself BEFORE watercooling became anywhere near so popular that you could buy anything watercooling related. Back then i were already using compressors to drive -34C coolant to the CPU. Did you know that you can get by just fine with low pressure, low volume pump (cheapest 12V you can find) if your block is properly massive and TDP is below 100W?

Btw, "stop-leak" and other stuff you put in to stop leaks is the stupidest idea ever.

Do you know what is the coolant of choice of extreme high output racing engines which actually partake in sanctioned races?
Now i'm talking about engines like 2L Inline 4 boosted to some 4bar, Cummins 5-8L diesel boosted to 7+bar?
Nope, not distilled water.

TAP WATER, with maybe some water wetter.
And these machines need to be capable of removing heat in the multiple MEGAWATT range.
Are you telling me water is not sufficient to remove your couple hundred watts of heat?

As for the dirt accumulation, i've only seen it once, and it was with off the shelf system with which came the "water additive", very small amount to be used, i was shocked to see that accumulation and i somehow doubt that would happen if proper glycol was being used. I suspect the additive that came with it was just some coloring.

The thing is with water if you add a little bit of heat, but not much, and circulate it constantly, it WILL have bacterial growth, that's why you need to add enough glycol.

Still, WTF is dead water? Other than a sea water phenomenon known to few sailors, which might just be a myth as likely.

[DeathAndTaxes]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.

I didn't mean to imply it was impossible however I intend to run a pure distilled water loop for maximum efficiency.  Watercooling is complicated enough adding mixed metals is adding extra optional complication.  For 99% of users it is simpler/easier to just skip the aluminum parts go all similar metals and bypass the entire issue. 


I have no interest in using antifreeze.
1) It is a pain.
2) It has to be regularly changed
3) Corrosion will occur if it isn't changed or concentration isn't sufficient.
4) It has much lower heat transfer capabilities compared to distilled water which lowers the efficiency of entire system requiring stronger pumps, larger heat exchangers, and faster/louder fans.
5) I will be using a heat exchanger to heat up potable water for human consumption which will require the more expensive non-toxic propylene glycol and corrosion resistance is significantly reduced.

So hopefully to not derail this futher:
a) yes you "can" use anti-freeze.
b) no I don't recommend it because today it is too easy to build a water cooled system with no mixed metals.

[DeathAndTaxes]

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Yuck

Simple answer is to not used mixed metals and then you can simply use pure distilled water (plus silver coil as a natural biocide).  Too easy.

If you must ignore the simple answer then remember that glycol has a lower heat transfer than water.  75% glycol is just stupid.  

More glycol means:
a) lower heat transfer (and higher pump resistance) - undesirable.
a) more corrosion resistance - unless you intend to not change loop for a decade even 5% should be fine.
b) higher boiling point - nope doesn't really apply here.  If you loop is getting to >100C you got problems.
c) lower freezing point - unless the radiator is outside it is a non-issue.  The container will tell you the freeze protection at various strengths.

so use the lowest amount that meets your needs.  

[cuz0882]

Dude, obviously you do not have a clue about the underlying physics going on.
FYI, i've even made blocks myself BEFORE watercooling became anywhere near so popular that you could buy anything watercooling related. Back then i were already using compressors to drive -34C coolant to the CPU. Did you know that you can get by just fine with low pressure, low volume pump (cheapest 12V you can find) if your block is properly massive and TDP is below 100W?

Btw, "stop-leak" and other stuff you put in to stop leaks is the stupidest idea ever.

Do you know what is the coolant of choice of extreme high output racing engines which actually partake in sanctioned races?
Now i'm talking about engines like 2L Inline 4 boosted to some 4bar, Cummins 5-8L diesel boosted to 7+bar?
Nope, not distilled water.

TAP WATER, with maybe some water wetter.
And these machines need to be capable of removing heat in the multiple MEGAWATT range.
Are you telling me water is not sufficient to remove your couple hundred watts of heat?

As for the dirt accumulation, i've only seen it once, and it was with off the shelf system with which came the "water additive", very small amount to be used, i was shocked to see that accumulation and i somehow doubt that would happen if proper glycol was being used. I suspect the additive that came with it was just some coloring.

The thing is with water if you add a little bit of heat, but not much, and circulate it constantly, it WILL have bacterial growth, that's why you need to add enough glycol.

Still, WTF is dead water? Other than a sea water phenomenon known to few sailors, which might just be a myth as likely.

I'm not interested in race car water cooling. You don't need glycol to keep bacteria out of the loop. Dead water is just a biocide additive that keeps bacteria out of the loop similar to silver. You can find it about anywhere that sells water cooling parts. http://www.frozencpu.com/products/11744/ex-liq-154/IandH_Dead-Water_Copper_Sulfate_Biocidal_PC_Coolant_Additive.html#blank

[PulsedMedia]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

[Gomeler]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.

I didn't mean to imply it was impossible however I intend to run a pure distilled water loop for maximum efficiency.  Watercooling is complicated enough adding mixed metals is adding extra optional complication.  For 99% of users it is simpler/easier to just skip the aluminum parts go all similar metals and bypass the entire issue. 


I have no interest in using antifreeze.
1) It is a pain.
2) It has to be regularly changed
3) Corrosion will occur if it isn't changed or concentration isn't sufficient.
4) It has much lower heat transfer capabilities compared to distilled water which lowers the efficiency of entire system requiring stronger pumps, larger heat exchangers, and faster/louder fans.
5) I will be using a heat exchanger to heat up potable water for human consumption which will require the more expensive non-toxic propylene glycol and corrosion resistance is significantly reduced.

So hopefully to not derail this futher:
a) yes you "can" use anti-freeze.
b) no I don't recommend it because today it is too easy to build a water cooled system with no mixed metals.

Whew, there is a lot of FUD in this thread but it appears you at least have a solid plan. Mostly curious if you'll try to harvest the waste heat and how you'll go about doing it. I know you mentioned pre-heating your waterheater, going to stick around to see what you do. I myself built out a small farm of 8 watercooled 5970s. Hopefully you don't buy up all the DangerDen 5970 waterblocks!

[DeathAndTaxes]

Mostly curious if you'll try to harvest the waste heat and how you'll go about doing it. I know you mentioned pre-heating your waterheater, going to stick around to see what you do. I myself built out a small farm of 8 watercooled 5970s. Hopefully you don't buy up all the DangerDen 5970 waterblocks!

My heat exchangers arrived today.    My wife wants a night out so next update will need to wait till tomorrow.

[cuz0882]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

[PulsedMedia]

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

wow, i didn't know distilled water is just boiled! OMFG!! j/k

I've even built my own water blocks.
The commercial products i've seen mostly use 10-15mm hosing. Many of the water blocks has been very simple, hose in, hose out, just a big water space inside the block, and some had ribs to have bigger surface area.


Good quality water pump is a must, i've never said you should skimp on water pump. Not once. Not even hinted.

No matter how you put it, 800-1100watts is still a small heat load relatively, and one definitively shouldn't over complicate things. Keep It Simple, Stupid! or K.I.S.S.

I can see the point in adding the silver coil, or even "dead water" or other chemical to kill bacteria, but i don't see a point in over complicating simple things. None of the commercial single system water cooling package i've had the displeasure to work with had nothing like this. One had so bad additive it kept on accumulating in corners and i had to spend significant amount of time running hot water through it to get that stuff out.

Use copper blocks, use brass fittings if you go with standard hydraulics fittings (ie. NPT, BSP) hell maybe even AN-fittings (aluminium btw!) even tho that's total über overkill, or maybe just use basic hose barbs (1.5-3$ each) and clamps. Go with a high quality pump with a backup pump (2 parallel) so no need to monitor constantly. Use basic standard rubber hose, if you want to be cheap even garden hose will do the trick (yuck).
Add a little bit of glycol for protection of the metals.
and do *NOT* omit periodic maintenance (ie. once a year)

tho adding dead water and silver coil costs next to nothing vs. the other gear. So why not add them.

I must admit tho, i hate water cooling generally, makes maintenance a bitch in regular systems, but for servers (or bitcoin mining cluster) it makes tons of sense.

You should take a look at how industrial water cooling systems work ... They are remarkably simple by design. Even those cooling tens of thousands of servers.

I started fooling around with water cooling when Thunderbirds were a new thing (or maybe even earlier ...), didn't like it then for single systems, don't like it now. Tho a new corsair's set made it remarkably simple, infact so simple i might consider that for myself too, after i've seen how a system like that works later down the road (maintenance wise): http://www.corsair.com/cpu-cooling-kits/hydro-series-water-cooling-cpu-cooler/hydro-series-h100-extreme-performance-liquid-cpu-cooler.html
Just installed H100 for a customer

[ThiagoCMC]

Your 1250W PSU is handling this 4 x 5970 smoothly!?

[DeathAndTaxes]

Your 1250W PSU is handling this 4 x 5970 smoothly!?

Yes.  Remember PSU are rated on DC load not AC.  It is ~1100W AC.  At ~90% efficiency that is <1000W DC.  No problem for a SeaSonic 1250W unit.  Now lesser brands might choke up but any top tier supplier should be able to provide 100% power continually.

[Roadhog2k5]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

Except for that you're missing a step to distill water. Yes you boil it, but what you are collecting is the steam from what you are boiling, which then condenses back into water. You are not just using the water you have boiled directly. And yes, it is a good idea to use distilled water in a car just for that reason. A lot of times non distilled water has minerals in it that will leave buildup/corrode the cooling system components.

[cuz0882]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

Except for that you're missing a step to distill water. Yes you boil it, but what you are collecting is the steam from what you are boiling, which then condenses back into water. You are not just using the water you have boiled directly. And yes, it is a good idea to use distilled water in a car just for that reason. A lot of times non distilled water has minerals in it that will leave buildup/corrode the cooling system components.

I didn't feel like elaborating further, minerals could be a real problem though if you had to keep filling your radiator because of a leak or something.

[ummas]

I`m having strange problem here with system 6GPU+
made 2 systems yesterday, and bouth got it.
GPU`s are shutting down randomly....
I was testing few OS`s: xubuntu, linuxcoin, W7(WTF? - i was desperat)
 

1 rig - 3x5970+5870  - OCZ ZX 1250 - all GPU down after 7h - i was sleeping
2 rig - 3x5870+2x5970 - ENERMAX 1250 + sommething 550W - 3 GPU down after 7H

Both rigs has 890FXA-GD70 and GPUs are watercooled.
Cheked watteg with KILL-O-WAT but only rig no.1 - takes 980W from the wall - 5870 - 950, 150 and 5970 - 750,150 - so not a big deal. temps - max 54. Cheked vrm tmp under W7 - each GPU - higher by 10*C so max ~65...cool and nice, so why down ??
any ideas ??