Submersing a rig

[byronbb]

lol from youtube comments:

put 4 gtx 480, overclock them, and then you would be able to make french fries in it.

[1714776369]

1714776369

[1714776369]

1714776369

[1714776369]

1714776369

[1714776369]

1714776369

[1714776369]

1714776369

[elrodvoss]

I have actually looked into Oil submersion cases for several years and done some tests with them as well.

Ive never had the guts to drop 1200 computer system into a vat of oil, mostly that once you do, it takes ALOT of clean up afterwards and voids about every warranty there is.

A good site that has made custom cases for oil cooled systems is:

http://www.pugetsystems.com/submerged.php

They been doing it for a while and has gone from a modified fish tank to a fully customized case.


You can drop anything into the oil except anything that has a physical motor that spins.  So Fans, Hard Drives, CD/DVD Drives, etc.  You can drop in the motherboard, SSD, CPU, RAM, GPU, even the PSU (the fan is only to cool the heat sinks in the PSU).  Plus you wont get shocked.

So you could get a big plastic tub, put your system into the tub and start filling the tub with oil and it will keep working.  You still need something to circulate the oil or you will get hot spots and the oil will start to heat up.  Now you put your rig inside a big aluminum box with fins and that would keep it "cooler".

The site above will have some nice videos that may impress you.

-elrodvoss

[DeathAndTaxes]

Guys... Im pretty sure he wants to put it in a water proof box. And sink into a pool of Really cold water

That would do nothing.  The rig would cook inside the insulating air.  Air is a horrible heat transfer mechanism.  I mean horrible.  Insulation isn't what keeps your house warm.  Insulation just traps air which is amazingly good insulator.

[SgtSpike]

I have actually looked into Oil submersion cases for several years and done some tests with them as well.

Ive never had the guts to drop 1200 computer system into a vat of oil, mostly that once you do, it takes ALOT of clean up afterwards and voids about every warranty there is.

A good site that has made custom cases for oil cooled systems is:

http://www.pugetsystems.com/submerged.php

They been doing it for a while and has gone from a modified fish tank to a fully customized case.


You can drop anything into the oil except anything that has a physical motor that spins.  So Fans, Hard Drives, CD/DVD Drives, etc.  You can drop in the motherboard, SSD, CPU, RAM, GPU, even the PSU (the fan is only to cool the heat sinks in the PSU).  Plus you wont get shocked.

So you could get a big plastic tub, put your system into the tub and start filling the tub with oil and it will keep working.  You still need something to circulate the oil or you will get hot spots and the oil will start to heat up.  Now you put your rig inside a big aluminum box with fins and that would keep it "cooler".

The site above will have some nice videos that may impress you.

-elrodvoss

Well, I can see why a HDD or CD/DVD drive wouldn't work (HDD needs to be air inside, and there's those holes that say "DO NOT PLUG", and CD/DVD is obvious), but there's no reason you can't have a couple of fans running inside the system, including the one in the PSU.  It won't hurt the fans.  There's plenty enough oil to keep the fan motors cool, plus free lifetime lubrication!

[chromeguy]

i would rig it similar to current watercooled PC systems. using water as a heatsink, complete with pumps & radiator (or pipe it outside to your hottub)

otherwise you need to fork out huge bucks for the fully submersible oil solution like those guys did with that supercomputer.

[RandyFolds]

i would rig it similar to current watercooled PC systems. using water as a heatsink, complete with pumps & radiator (or pipe it outside to your hottub)

otherwise you need to fork out huge bucks for the fully submersible oil solution like those guys did with that supercomputer.

The big bucks aren't for the fully submersible solution, they are for the media that you do the submersing in. Standard computer parts, aside from platter hard drives and optical drives can all take the plunge without issue.

[SgtSpike]

i would rig it similar to current watercooled PC systems. using water as a heatsink, complete with pumps & radiator (or pipe it outside to your hottub)

otherwise you need to fork out huge bucks for the fully submersible oil solution like those guys did with that supercomputer.

If I understand you correctly, you mean pumping oil through a watercooling system?

If so, that's a very bad idea.  Mineral oil has much worse heat conduction than water.  You'd be making a liquid cooling system that is much more expensive than water cooling, and also much less efficient.

[chromeguy]

i would rig it similar to current watercooled PC systems. using water as a heatsink, complete with pumps & radiator (or pipe it outside to your hottub)

otherwise you need to fork out huge bucks for the fully submersible oil solution like those guys did with that supercomputer.

If I understand you correctly, you mean pumping oil through a watercooling system?

If so, that's a very bad idea.  Mineral oil has much worse heat conduction than water.  You'd be making a liquid cooling system that is much more expensive than water cooling, and also much less efficient.

lol no, just watercooling.
which, imo, is an awful lot of mucking around to achieve slightly better cooling than a big heatsink.

I have looked into active cooling (peltiers) but there are condensation issues & they also suck alot of power on their own.

better really to just underclock until peak mh/$/power efficency & just keep buying more and more of them, rather than pressing one unit harder & harder??

[SgtSpike]

i would rig it similar to current watercooled PC systems. using water as a heatsink, complete with pumps & radiator (or pipe it outside to your hottub)

otherwise you need to fork out huge bucks for the fully submersible oil solution like those guys did with that supercomputer.

If I understand you correctly, you mean pumping oil through a watercooling system?

If so, that's a very bad idea.  Mineral oil has much worse heat conduction than water.  You'd be making a liquid cooling system that is much more expensive than water cooling, and also much less efficient.

lol no, just watercooling.
which, imo, is an awful lot of mucking around to achieve slightly better cooling than a big heatsink.

I have looked into active cooling (peltiers) but there are condensation issues & they also suck alot of power on their own.

better really to just underclock until peak mh/$/power efficency & just keep buying more and more of them, rather than pressing one unit harder & harder??

Oh, watercooling, but using a pool as a reservoir then?  That would work, so long as you kept your pumps and lines nice and free from algae and such.

I never really understood the draw of watercooling myself.  If you have a lot of money to burn, I guess. 

[DeathAndTaxes]

I never really understood the draw of watercooling myself.  If you have a lot of money to burn, I guess.  

Or other things are more important than simply pure performance at any noise level.

If you want max performance at cheapest possible performance then nothing beats air.  Crank the fans up so it sounds like an industrial band saw and it can't be beat.

I have 4 rigs in the garage running like that.  In my office I also have a workstation w/ 4x 5970 in a closed case which is quieter than a desk fan.  In the winter in quietly produces 3.1GH and dumps 4000 BTU into the room.  In the summer I move the radiator outside (though window) and it dumps the 4000 BTU outside the house.

[SgtSpike]

I never really understood the draw of watercooling myself.  If you have a lot of money to burn, I guess.  

Or other things are more important than simply pure performance at any noise level.

If you want max performance at cheapest possible performance then nothing beats air.  Crank the fans up so it sounds like an industrial band saw and it can't be beat.

I have 4 rigs in the garage running like that.  In my office I also have a workstation w/ 4x 5970 in a closed case which is quieter than a desk fan.  In the winter in quietly produces 3.1GH and dumps 4000 BTU into the room.  In the summer I move the radiator outside (though window) and it dumps the 4000 BTU outside the house.

Fair enough. 

[rjk]

I never really understood the draw of watercooling myself.  If you have a lot of money to burn, I guess.  

Or other things are more important than simply pure performance at any noise level.

If you want max performance at cheapest possible performance then nothing beats air.  Crank the fans up so it sounds like an industrial band saw and it can't be beat.

I have 4 rigs in the garage running like that.  In my office I also have a workstation w/ 4x 5970 in a closed case which is quieter than a desk fan.  In the winter in quietly produces 3.1GH and dumps 4000 BTU into the room.  In the summer I move the radiator outside (though window) and it dumps the 4000 BTU outside the house.

+1, that's why servers tend to be so loud. They have specially designed airflow paths and extremely high velocity fans.
Water is useful only to keep the noise down.

[P4man]

Im actually reconsidering my position on this and looking into submerging my rigs for when summer comes.

Here is the thing, I have a large Koi pond (50K liter) next to the shed where I would want to put my miners this summer.  And it wouldnt hurt to warm the pond water during fall and winter, even if the effect will be minimal in the best case. Even with a perfect heat exchange, 2000W is going to do next to nothing to the overall water temperature in the pond, but raising the temperature in the filters is beneficial to the filtering process, every degree helps.

Ill have to expand the filtration system for the pond anyway and  I was considering putting a few large (1000L) settling tanks in the shed. By itself, I wouldnt have expected that to do much, if anything, unless perhaps if I created an air duct of sorts around the miners to concentrate the warm air, and blow that warm air through one of the (open) tanks; i already have a 160L/hour air pump, I could add another one for that purpose. It wouldnt be much of a settling tank anymore then, and I guess it wouldnt do too much to reduce ambient heat either. or?

I guess the more efficient thing to do is submerge my rigs in mineral oil, pump the oil through a radiator which is suspended in a settling tank or filter  tank (Im using 300 liter rain barrels for biofiltering, the filter has a pretty high volume, up to 25K liter per hour).

A few questions, not sure if anyone here can help me with

- how toxic is mineral oil? Or would vegetable oil be better here? I dont fancy the idea of a small leak killing all my fish. I remember reading that a cheap way to buy large quantities of oil for submerged cooling is buying them as... horse laxative. I cant imagine a horse laxative being excessively toxic... ?

- what kind of hoses does one need? Do you need a special kind of rubber or plastic for oil? From the shed to the filtering installation is a fair few meters

- any suggestions on oil filtering?

- does anyone have any idea of dimensioning this? I got no clue how big a pump (pressure and flow rate) or radiator I would need. If it helps, by then I plan on having ~2000W worth of mining gear. The pond water rarely gets over 25C and more typical is 20C  and the water in the filter would have no time to heat up significantly because of the high flow rate. I would *guess* keeping the oil below 35-40C would be good enough to keep the cards reasonably cool, although that is more guesswork on my part.

ANy thoughts or ideas on the matter would be appreciated.

[P4man]

Okay I did some research myself. Toxicity is not a problem with pure mineral oil. you can even drink the stuff.

As for capacity planning; Lets assume I only submerge my GPUs (thats what I plan, just place them inverted with extenders so only the gpu's "swim"), lets put it at 1000W worth of GPUs in a 100L tank.lets look at the heating of the oil. I found this:

http://make-biodiesel.org/Biodiesel-Chemistry/time-to-heat-oil.html

From the above link:
Time in hours = 0.5 X Number of Liters X Temperature rise in °C / by element wattage

Solving that for temperature rise:
Temperature rise per hour = 1000W / ( 0.5 * 100L) = 20C per hour or 0.333C per minute.

Thats considerably less than I would have thought.

Lets assume a hot summer day with 35C ambient air temperature and my pond water being at 25C.

assume the submerged radiator manages to cool the oil to 10C above water temp, so to 35C.
I want the oil in mining rig to remain below 50C. Thats a 15C difference
That means I have to turn over the 100L every 5 minutes. So Id need ~1200L/hour throughput.
Actually, the amount of oil used in the rig theoretically wouldnt matter here.

Anyway, some googling found me this:

http://www.ebay.com/itm/350GPH-DC12V-80C-Brushless-Magnetic-Oil-Water-Pump-/160560267427?pt=BI_Pumps&hash=item25622340a3

Pretty much spot on for the capacity I just calculated, although that probably means its not enough when you factor in pump head and the fact the stated throughput is probably for water, not oil.

Still, seems quite doable with all the above assumptions. Looks like all I need is a pump similar to that one, but probably 2x or so as powerful, some (car) oil radiator and some tubing. Most of the stuff seems fairly inexpensive. Oh and Id need a filter somehow, as there is no way to make this setup airtight if Im only going to submerge the GPUs.

Question that remains; with 50C oil temperature, what would my GPU temps be like? Although it is probably different, can anyone using watercooling shed some light on the relationship between water temperature and GPU temps?

[JA37]

How about a little love for "Bubbles"? From 1985.
http://en.wikipedia.org/wiki/Cray-2

[DeathAndTaxes]

Why not just use a traditional water cooling rig w/ a heat exchanger?
http://www.koolance.com/water-cooling/product_info.php?product_id=944

As far as what temps are GPUs.  In watercooling (and I would imagine oil cooling too) water is such a good heat conductor that everything in the loop is within 1 or 2 degrees of everything else.  So if the oil reaches equilibrium around 50C then the GPU would be 50C.

Another way to look at it in reverse.  Say you pond can dissipate more thermal energy than your rigs can produce.  The pond water is 25C.  In the heat exchanger above using 7 LPM pumps (and water) we can transfer ~1600W of thermal energy and keep delta T at <=6C.  This means the hot loop will be in equilibrium at ~ 31C.   

In air cooling the GPU core temp being higher than the exhaust air temp is merely an effect of the poor heat conductivity of air.  If you used enough air (say thousands of cfm of airflow) you could get the input, gpu core, and exhaust air to all be roughly the same.

assume the submerged radiator manages to cool the oil to 10C above water temp, so to 35C.

Assumming you have a properly sized radiator even if you were air cooling the radiator you can keep the delta T <10 C.  With a liquid to liquid transfer you could have a much lower delta T.  <5C or even 2C is possible.

If the radiator will be submerged in water you likely will get better heat transfer with a heat exchanger.  Radiators are designed assuming they will be used to force air through fins to acheive heat transfer.  

[P4man]

Why not just use a traditional water cooling rig w/ a heat exchanger?

Price mostly. And convenience. Equipping 6 or 10 cards with full covers for watercooling is expensive, and it could be hard to find covers for all my cards. Remember I run rat rigs thrown together with leftover parts and then I plug in whatever 58x0 card I can find cheaply. I have like no 2 identical cards, and thats only gonna get worse .

As far as what temps are GPUs.  In watercooling (and I would imagine oil cooling too) water is such a good heat conductor that everything in the loop is within 1 or 2 degrees of everything else.  So if the oil reaches equilibrium around 50C then the GPU would be 50C.

Some more googling suggested that too. Well, up to 5-7C delta. This is getting more and more exciting .

If the radiator will be submerged in water you likely will get better heat transfer with a heat exchanger.  Radiators are designed assuming they will be used to force air through fins to acheive heat transfer. 

Heat exchangers require I pump water through them. Thats probably not a good idea, considering its pond water; its gonna clog sooner or later, even if I were to pump "clean" water from the end of the filter (and Id rather heat the beginning of the cycle to help the aerobic process).  My filters are connected with 130 mm pipes, that doesnt clog so easily . Moreover, it requires 2 pumps, one for the oil, one for the water. Thats 2 points of failure.

 Then there is also... price. A decent sized heat exchanger is not that cheap. I may not need a big one from an efficiency point of view, but Id need one that doesnt clog. Instead I could buy one, or heck, should it prove necessary or useful,  several oil coolers from a car junk yard for next to nothing, or even if you buy them new, something like this:
http://www.ebay.com/itm/UNIVERSAL-15ROW-10AN-TRANSMISSION-RADIATOR-TURBO-ENGINE-OIL-COOLER-ALUMINUM-BLUE-/160721146846?pt=Motors_Car_Truck_Parts_Accessories&vxp=mtr&hash=item256bba13de

isnt gonna break the bank, and I cant see how that would not work when submerged in relatively fast flowing and air bubbling 20-25C water. Like you said, water is a much better heat conductor than air, if it can cool the oil by blowing air over it, its gonna work a hole lot better submerged - I think.

Anyway, Ill probably setup a small scale experiment next month with a single spare card (8800GT) to see what happens. Still have to figure out where to best buy some suitable oil, hoses,  think of a filtration system and come up with a layout thats not completely unworkable. Its not gonna be pretty with all the motherboards suspended inverted above an oil bath, but if it works,  who cares

[RandyFolds]

2000w will more than likely impact 10,000 gallons, though it wholly depends a lot on surface area. I had a 4,500g outdoor reef running 2000 watts of inline heater and it would keep the tank at 76-78 even when it was ~50F ambient and was still kicking off. The reef was lit with LED so there was almost zero infrared contributing to that. Water is an incredibly good insulator.

[DeathAndTaxes]

2000w will more than likely impact 10,000 gallons, though it wholly depends a lot on surface area. I had a 4,500g outdoor reef running 2000 watts of inline heater and it would keep the tank at 76-78 even when it was ~50F ambient and was still kicking off. The reef was lit with LED so there was almost zero infrared contributing to that. Water Air is an incredibly good insulator.

The water rose in temp because it was a good conductor and the air wasn't.  Thus the thermal energy gets "trapped" in the water and the temp rises.

Water is a horrible insulator.  As an example a warm jacket isn't what keeps you warm.  A warm jacket traps air which keeps you warm.  Now take same jacket and soak it (even in hot water), put it on and walk outside.  Brrr.  The water "shotcuts" any insulation allows a highly conductive path between your internal body heat and the very cold outside air.

[RandyFolds]

2000w will more than likely impact 10,000 gallons, though it wholly depends a lot on surface area. I had a 4,500g outdoor reef running 2000 watts of inline heater and it would keep the tank at 76-78 even when it was ~50F ambient and was still kicking off. The reef was lit with LED so there was almost zero infrared contributing to that. Water Air is an incredibly good insulator.

The water rose in temp because it was a good conductor and the air wasn't.  Thus the thermal energy gets "trapped" in the water and the temp rises.

Water is a horrible insulator.

Insulator was the wrong word, but now I need you to explain something. Would the layer of water trapped by a wetsuit be considered an insulator, or is the insulating going on in the neoprene? Wetsuit without that trapped layer doesn't do anything.

Fucking thermodynamics...how does it work?

My point was that dumping a constant 2000w into a body of water will heat it, and I am not sure ambient temperatures and airflow will be enough to 'exhaust' it, so to speak. Throw the sun into the mix and you are going to have some wild daily temperature swings to the detriment of the fishes health.