Two Phase Open Bath Immersion Cooling Thread

[DeathAndTaxes]

I figured I would start this thread as a place to collaborate on immersion cooling.  The type of immersion cooling I am interested in is two phase passive semi-open bath.  Wow that is a mouthful.  So lets start with the basics.

What is immersion cooling?
Immersion cooling is a method of heat transfer where the device to be cooled (namely SHA-256 processor) is immersed in a heat transfer fluid (also called working fluid).  It is helpful to remember that "fluid" doesn't necessarily mean liquid, "Freon" is a form of working fluid although most people think of it as a gas.  Technically even air is a working fluid.  Heat sinks transfer heat to passing air and fans are pumps designed to move low density gas instead of higher density liquids.  Most people don't refer to air cooling as immersion cooling but it may help to understand some of the concepts if you think of Freon, water, and air as working fluids in heat transfer.

Is immersion cooling the same thing as "water cooling"?  
With immersion cooling the heat is transferred directly from the heat source to the working fluid.  In "watercooling" the working fluid is potentially harmful to electronics and thus flows through a sealed loop isolated from the heat source.   A watertight waterblock is used to indirectly transfer the heat from the heat source to the working fluid.  With immersion cooling the working fluid must be non-conductive and that generally limits us to three families of fluids: deionized water, mineral oil, and fluorocarbon-based fluids (namely Fluorinert made by 3M).  Immersion cooling systems have higher fluid cost than watercooling but this is partially offset by the elimination of individual waterblocks.   

What is Two Phase Cooling?
Well lets start with what is one phase cooling.  In a mineral oil or deionized water system the working fluid never boil or freezes and always remains in a single liquid phase.  Cool fluid is pumped past the heat source, where thermal energy is transfered to the fluid by conduction which raises the temperature of the fluid.  The heated fluid is then pumped to a heat exchanger where it is cooled and pumped back to the heat source.  The heat transfer is known as "sensible heat", and the more heat (thermal energy) transferred into the working fluid the more its temperature rises.  The rise in temperature can be controlled by the fluid flow rate.  The faster the fluid flow rate the less energy will be transferred into each unit of fluid and the lower the temperature rise will be.

In two phase cooling the working fluid boils and thus exists in both a liquid and gas phase.  The system takes advantage of a concept known as "latent heat" which is the heat (thermal energy) required to change the phase of a fluid.  The working fluid is only cooled by boiling and thus remains at the boiling point ("saturation temperature").  Energy is transferred from the heat source into the working fluid will cause a portion of it to boil off into a gas.   The gas rises above the fluid pool where it contacts a condenser which is cooler than the saturation temperature.  This causes the fluid to condense back into a liquid and fall (rain) back into the pool. 

What is "open bath" or "semi open bath" means?
Open Bath (sometimes more correctly called semi open bath) means the tank containing the heat source, working fluid and condenser is not a pressure vessel.  The pressure inside the tank/container will be roughly the same pressure as the outside air (<1 PSI difference).  This makes the tank easier and cheaper to construct and the system is never pressurized at dangerous pressures.  In a open bath system equilibrium is achieved by having a condenser capable of sufficient heat transfer to condense the vapor at the same rate it is being produced by boiling.   The volume of vapor remains relatively constant and prevents a rise in pressure.  If the fluid boils at a faster rate than the condenser can condense it back into a liquid then the pressure in the tank will rise and vapor will be lost.  This can be prevented by a safety pressure switch.

Why not used deionized water (DI)?
DI is electrically non conductive but it isn't inert.  It will rapidly pull ions from the surrounding material until it reaches a point where it becomes conductive and damages the cooled components.  Proper immersion cooling with DI requires expensive ion exchange systems and replacement ion exchange resin to continually remove build up of ions in the water.  This process needs to operate continually which increases the cost and the system is never stable.   Without maintenance and continual inspection the system can allow a build up of ions that eventually destroys the equipment being cooled.  While immersion cooling with DI is a viable method it is ill-suited for unmonitored environments.

Why not use mineral oil?
Mineral oil is a single phase cooling system and for any significant heat load will require actively circulating the oil across the processors.   Mineral oil also has a high viscosity and will need powerful pumps capable of handling the higher pressure.   The system needs to be carefully designed to ensure each processor receives sufficient flow.  Mineral oil has a relatively low heat capacity and thermal conductivity which increases the required flowrates and necessitates the use of larger heat exchangers. 

What types of power densities are possible?
3M has conducted experiments cooling 4KW heat loads using 1L of working fluid so in theory heat densities approaching 4,000 W/L are possible.  The constraint on commercialization is that existing servers have relatively low energy densities (well low relative to the limits of immersion cooling).  Even a high end 3U server (4 CPU, multiple GPUs, 4+ 1200W PSU) may only have an energy density of 100W per Liter.    However SHA-256 ASICs have very high energy densities although current systems have server like energy densities due to the limits of air or water cooling.  Take a look inside the case of any 2nd gen ASIC design what takes up the most space?  Air.  The actual ASIC boards are very energy dense however there are surrounded by a significant amount of empty space.   Remember these are using boards designed for air/water cooling.  It may be possible to improve energy density by making custom compact boards.

As an example of what is possible KNCs ASIC boards are 225 cm2 of surface area and use ~120W.  If boards were stacked 1cm apart that would be an energy density of >500 W/L.  Hashfast pcb design (subject to change) is even more energy dense, 280W of power in 240 cm2.  In spaced in a cooling pool 1cm apart that would be an energy density of ~1,200 W/L*.   Another way to look at it is hashing density.   At 1cm spacing KNC would be a hashing density of >500 GH/L and Hashfast would be > 1500 GH/L.    

* There is limited information available on height clearance of HF boards.  If the large FET heatsinks are not removable it may limit the spacing of boards in a cooling pool.

How is the condenser cooled?
The condenser could be cooled using refrigerant directly but a water loop (glycol/water mix) may be a more flexible and economical solution.  The water loop can either be cooled using a commercial water chiller or using a dry tower in ambient air (heat exchanger and fans outside).  The cooling loop will need to be sized large enough to transfer the full heat load out of the water loop.  The advantage of FC-72 over other two-phase working fluids is it has a 56C boiling point which allows reasonable efficiency when cooling using outside air even during summer temps for most parts of the country. The larger difference between the input air (ambient outside air temp) and the input water temp (~50C to 56C) means more efficient heat transfer is (smaller fans, smaller heat exchanger).

What working fluids are available?
3M makes a large number of synthetic working fluids however for two phase cooling we are interested in fluids which boil below the temperature limits of ASICs.   Remember in two phase cooling the fluid will remain at saturation temp (boiling point) excess thermal energy is removed from boiling however the temperature of the fluid will remain in equilibrium at saturation temp.  It is not possible with two phase cooling to have temperatures less than the boiling point of the working fluid.

3M datasheet on all heat transfer fluids: http://multimedia.3m.com/mws/mediawebserver?mwsId=tttttviZIdW5_y7VPZA_qZ0t2XV62EW9iXut2Xut2tttttt--&fn=bro_heattrans.pdf

For this reason the following fluids are most applicable:
Novec 7000 - Boiling Point 34C
FC-3284 - Boiling Point 50C
FC-72  - Boiling Point 56C
Novec 7100 - Boiling Point 61C
FC-770  - Boiling Point 95C (likely too high outside of custom design)

The cost depends on supplier and volume being purchased but generally runs $80 to $100 per Liter.  Fluorinert is often priced by the kilogram and has a high density (1.6 kg per Liter) so take that into consideration if you think you found a "deal".

[DeathAndTaxes]

Major Systems
1) Overview (coming soon)
2) Cooling Tank
3) Connectivity (coming soon)
4) Condenser (coming soon)
5) Chilled Water Supply (coming soon)
6) Safety Systems (coming soon)
7) Power Supply (coming soon)

2) Cooling tank

Material considerations.  Fluorinert is a highly inert fluid, it is also a poor solvent with most materials, and it is hydrophobic.  These characteristics make it an ideal working fluid.  However Fluorinert is an effective solvent for fluorinated compounds (materials containing fluorine),  plasticizers, and some additives to soft "plastics".  

This presents two challenges (and will be covered more in challenges section).  The first it that Fluorinert can replace by volume a portion of the compatible material in gaskets, seals, and orings.   For example when silicon adhesive is immersed in FC-72 up to 7% of the volume is replaced with Fluorinert.  This can lead to failure of seals in sealants, gaskets and o-rings.  Even when they use the same underlying compound "hard" materials are less prone to replacement than "soft" materials (i.e. rigid PVC pipe vs PVC wire insulation).

Material compatibility analysis by CERN
http://detector-cooling.web.cern.ch/detector-cooling/data/3M_FAQ_Fluorinert.pdf
http://detector-cooling.web.cern.ch/detector-cooling/data/Fluoro_Compatibility.htm

It is not possible in one post to do an exhaustive analysis of potential material risks so if you are interesting in working with immersion cooling I strongly recommend reviewing both of the links above and conducting your own research.  DISCLAIMER:  All of this is experimental.  Damage to equipment can occur. If you are unwilling to do self research and testing this type of project may not be realistic.   The challenges of material compatibility and handling dissolved contaminants will be covered in the challenges section.


Material choices for tank
Glass aquarium can be used for a prototype but due to the use of silicon adhesive below the fluid line I would caution against its use long term.  Stainless steel tank is a good option for the ability to design and construct a custom sized tank.  To reduce heat conduction a double tank design could be constructed with an inner stainless steel tank surrounded by insulation, surrounded by an outer tank.  To reduce costs the outer tank could be constructed from a cheaper material than stainless steel.  

For my initial prototype I will be used a polycarbonate NEMA 4 enclosure with the opening facing upward.  It would be best in any inlet be through the sidewall of the tank not the lid however in my prototype I will be cutting through the lid as any mistake can be fixed by replacing the lid instead of the more expensive body.  Some NEMA 4 enclosures come pre tapped with inlets that are gasketed.  That should be avoided for the reasons above instead look for a solid body design.




Design of the cooling tank.  
The tank should be designed to have no penetrations below the fluid line.  The fluid depth should be sufficient to cover all components plus a safety margin of 30mm.  In normal operation the fluid in the tank should be relatively constant (<5mm change in depth) however a failure of one or more cooling components will result in the condenser temperature rising and the condenser will be unable to condense (cool) the working fluid as fast as it is being boiled off and the fluid will fall.  The safety margin ensures that there is a delay between any failure and the components boiling off their fluid and likely being damaged.  To protect the safety margin a depth switch connected to a relay can be used to cut off power to the tank if the fluid level drops.  When planning the tank there should be sufficient space in the tank to contain the boiled Fluorinert gas.  As a starting point you should plan for no less volume than the volume of the fluid.  In addition you should consider the height of the heat exchanger.  

As an example the highest component to be cooled is 100mm from the bottom of the tank.  The normal fluid depth should then be 130mm.  A float switch is installed which will trip and cut all power to the tank if the fluid depth drops below 115mm.  You should plan for a gas height at least equal to the fluid height which would be another 130mm.  The heat exchanger has a height of 50mm.  The tank should have a combined height of at least 310mm.  There should be no inlets in the lower 130mm of the tank.

The components to be cooled are submerged in the working fluid, if possible the tank size should be optimized to minimize the amount of working fluid necessary to submerge the components.   Due to the high cost of Fluorinert (~$80 per Liter) a design which achieves the minimum amount of fluid to transfer the heat will be more economical.  Remember Fluorinert can handle a significant heat load so you can use it sparingly.  3M has shown effective heat transfer with as little as 1L for 4KW of heat load while it is unlikely that you will acheive that level of energy density it shows that excess fluid isn't necessary.

One design consideration is what components will be included in the tank.  An SHA-2 hashing system consists of three major components.  One or more processors boards, a host/controller, and one or more ATX power supplies.  It is possible to use immersion cooling to cool just processor boards or the entire system.  There are advantages and disadvantages to both.

Lets consider an example 4TH/s system using HashFast ASIC boards
Processing Boards - 2780W (87%) [1]
Host - 100W (3%) [2]
Power Supply (10%) [3]
Total: 3200W

[1] 10 boards @ 400 GH/s nominal, 278W per board
[2] Low power PC, using a embedded computer like Beagle Board would reduce wattage further
[3] The wattage is the "lost" power converted to heat.   Multiple 90% efficient ATX style 80-Gold or 80-Platinum PSU.  Power supply could possibly be in n+1 fault tolerant configuration.   In: 3200W AC Out: 2880W DC + 320W heat.

Immersion cooling only processing boards
---------------------------------------
Pros:  
 Highest energy density.  
 Reduced tank size.
 Most efficient use of Fluorinert (highest W/L)

Cons:
 Complicates power delivery.  Either custom high current cables are needed or large number of cables need to pass through the tank bulkhead.  
 Still need to air cool other components.  
 Potentially prevents deployment to areas where temp outside the tank is ill suited for air cooling (i.e. non-air conditioned warehouse).
 Not silent due to PSU fan noise.
 If cooling water line bursts the components outside the tank (as well as operators) are vulnerable to being damaged by short circuit conditions.

Immersion cooling all components
---------------------------------------
Pros:
  Simplified tank connection (can be reduced to: data cable, power cable, water in, and water out lines).
  Power supplies may need to be modified.  Fans should be removed and power supply fan monitoring bypassed (if present).
  Near silent operation (if cooling water heat transfer is in another location).

Cons:
  Larger tank (more significant than may initially appear.  remove heatsinks and fans from a hashing board and the power supply has almost the same volume as the boards it will power).
  Less efficient use of Fluorinert (lower W/L).
  Internal wire management can be more difficult if tank is cramped.
  Replacement of failed host or power supply is more complicated.
 
Depending on the boards used a hybrid between the two options would be to use immersion cooling for the processors and power supplies and place the host/controller outside the tank.  For larger operations this may be desirable as a "hot swap" replacement of defective host could be performed easily.  This may not be possible with all SHA-2 processors but in at least one case the HashFast boards are connected by USB to any host capable of running cgminer and linux.  So data connection to tank could be a single usb port and the host host system be as far from the tank as allowed by USB spec or even one host controlling multiple tanks.  The USB cable length can be extended using cheap Cat5/6 cable with an active converter.

[DeathAndTaxes]

Reserved - Advantages & Challenges

[DeathAndTaxes]

Reserved - Prototype details

[bobsag3]

Im interested. I will read more when I have a few minutes, but I love the idea.

It looks like your need for bare boards, anything bitfurry based would be your best bet, with small sinks on each chip to aid in surface area.

[DyslexicZombei]

Im interested. I will read more when I have a few minutes, but I love the idea.

It looks like your need for bare boards, anything bitfurry based would be your best bet, with small sinks on each chip to aid in surface area.

Oh, we're definitely interested all right.

Our test subject will probably be an old Gen1 Avalon. We're going to shoot for 1GH/s for fun if we test this out. 

[DeathAndTaxes]

Im interested. I will read more when I have a few minutes, but I love the idea.

It looks like your need for bare boards, anything bitfurry based would be your best bet, with small sinks on each chip to aid in surface area.

It is very likely you won't need heatsinks using bitfury chips.  Which is a good thing given the number of chips needed for 1 TH.  FC-72 has a critical heat flux under normal conditions (no flow, no subcooling, flat surface, normal pressure) of ~ 15 W/cm2.  Beyond 15 W/cm2 you get boiling failure and the wall superheat (difference between fluid temp and device temp skyrockets).  Under the critical heat flux the wall super heat is <10 C.

Bitfury (overclocked) ~1 W/GH (chip only).  
Package dimensions 0.7cm x 0.7 cm
1/(0.7*0.7) = 2.04 W/cm2.
Bitfury Heat Flux: ~ 2 W/cm2  < FC-72 Critical Heat Flux: ~ 15W/cm2
No issues having proper boiling without heatsink.

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

[bobsag3]

Im interested. I will read more when I have a few minutes, but I love the idea.

It looks like your need for bare boards, anything bitfurry based would be your best bet, with small sinks on each chip to aid in surface area.

It is very likely you won't need heatsinks using bitfury chips.  Which is a good thing given the number of chips needed for 1 TH.  FC-72 has a critical heat flux under normal conditions (no flow, no subcooling, flat surface, normal pressure) of ~ 15 W/cm2.  Beyond 15 W/cm2 you get boiling failure and the wall superheat (difference between fluid temp and device temp skyrockets).  Under the critical heat flux the wall super heat is <10 C.

Bitfury (overclocked) ~1 W/GH (chip only).  
Package dimensions 0.7cm x 0.7 cm
1/(0.7*0.7) = 2.04 W/cm2.
Bitfury Heat Flux: ~ 2 W/cm2  < FC-72 Critical Heat Flux: ~ 15W/cm2
No issues having proper boiling without heatsink.

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

Possibly BA or CT gear? Or the KnC... except for the lack of chips for a DiY design

[B.T.Coin]

Any news if you will continue this experiment?
I'm very interested in this.

[bitly]

Very interested.

[DeathAndTaxes]

I am busy and going to be even more busy in the coming weeks so while I have the Flourinert I need to put this on hold until after the new year.   I have the tank, the rigs (well KNC and Bitfury, HF hopefully eventually), and the immersion fluid I just don't have the time to make the conversion right now.

[dexX7]

Can you recommend any source for Flourinert? I'm really looking forward to read more, great thread.

[Digigami]

Can you recommend any source for Flourinert? I'm really looking forward to read more, great thread.

I found it available from this retailer, although no prices listed:

http://www.fasttechnologies.com/filtration/3m-fluorinert-%E2%84%A2

and also on fleabay

http://www.ebay.com/itm/3M-FC-40-FLUORINERT-32oz-New-and-Unused-Ready-to-Ship-/110716728982

[bitly]

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

So it's really not the chip but rather the PCB design that is "low GH density". If we could redesign the PCB, it may achieve a much higher density and much smaller volume?

[aerobatic]

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

So it's really not the chip but rather the PCB design that is "low GH density". If we could redesign the PCB, it may achieve a much higher density and much smaller volume?

actually, i'd be worried that the high GH chips from hashfast and cointerra (300-400 watts per chip) would be too hot in terms of spot heat, and you might need some heat spreader thing to increase the surface area.

the low GH chips from bitfury et al will be no problem for immersion cooling.

even the knc chips, medium heat (150 watts per chip).. will need some kindof heat spreader

Also, when choosing the heat spreader, you will want to find one that doesnt stick out too much so that it doesnt affect your board density in the bath.  ie: long thin heat spreaders may be more useful than the high tower heatsinks that knc currently uses on their board with an air fan

[DeathAndTaxes]

The disadvantage of using Bitfury chips is the low "GH density".   PCB size = 12cm x 12cm ?  If boards are spaced 1cm apart = 144cm3.  42 GH per board.  0.29 GH/cm3 or 3.4 CC/GH.   Fluorinert runs about $0.80 per CC ($80 per Liter), thus at 3.4 CC of fluid per GH the fluid cost is $2.72 per GH which is pretty high.  The higher the GH density the less fluid needed per unit of hashpower and the more economical the system.  My goal would be to get fluid costs to <$0.25 per GH and full system cost (excluding SHA-2 boards) <$0.50 per GH.

So it's really not the chip but rather the PCB design that is "low GH density". If we could redesign the PCB, it may achieve a much higher density and much smaller volume?

Well the low GH value of the chip contributes to the low GH density.  Note this isn't to say it wouldn't work.  It certainly would it would just mean using more immersion fluid per GH/s.  With the high cost of Fluorinert that means a higher cost per Gigahash cooled.  Still to be clear it will work fine it just will be less economical.

[DeathAndTaxes]

actually, i'd be worried that the high GH chips from hashfast and cointerra (300-400 watts per chip) would be too hot in terms of spot heat, and you might need some heat spreader thing to increase the surface area.

I need to get final power consumption values for HF or CT but Fluorinert has a critical heat flux of ~20W/cm2.  Originally it looked like HF was going to use an integrated heat spreader like KNC does but they have gone w/ "naked" dies so some sort of heat sink will be needed.   It remains to be seen if CT will use an integrated heat spreader and if the heat flux will be below the critical heat flux of fluorinert.

KNC package size and integrated heat spreader may be large enough to avoid needing a heatsink.  Still given the cost of the immersion fluid I believe HF and CT are the best choices at this point.  Still the heatsinks don't need to be particularly large.  Lets assume HF uses 250W per chip (4 dies).  Even a flat plate heat spreader would only need a surface area of 250/20 = 12.5 cm2 to keep the heat flux below the CHF.  Obviously we would want a margin of safety, so lets say 25 cm2.  That is ~5cm x 5cm flat surface and would be less for a "3D" heatsink as it increases the surface area beyond the linear dimensions.  The nice thing about HF design (the company itself is pretty scammy but they have some nicely designed hardware) is there is a large open space around the die and it uses standard Intel CPU cooler mounting holes so finding a low profile heat sink which lowers the heat flux shouldn't be difficult.  I have looked at a couple low profile server heatsinks which should do the trick but am hoping to find a cheaper solution.

[DeathAndTaxes]

Can you recommend any source for Flourinert? I'm really looking forward to read more, great thread.

I purchased the Flourinert from here:
http://www.parallax-tech.com/fluorine.htm


They don't have the versions suitable for immersion cooling in stock but they can special order.   3M keeps pricing pretty tight so don't expect much price different between vendors.  Fluorinert runs about $80 per Liter depending on the amount purchased and the type of Fluorinert.  Novec is roughly the same but a little cheaper. 

[takagari]

Very Very cool, I was just thinking of this actually.

I'll grab some, find a container and toss a usb miner in. see how the temp sits.

For the high density chips, I was thinking of using a pump to return cooled liquir threw multiple tiny tube's. one pointed at each chip. Or just massive flow, unsure which would be more effective.

[DeathAndTaxes]

and also on fleabay

http://www.ebay.com/itm/3M-FC-40-FLUORINERT-32oz-New-and-Unused-Ready-to-Ship-/110716728982

I would strongly recommend not buying Fluorinert on eBay.  The chance of getting diluted, contaminated, or used fluid is simply too high.  There is no cheap way to test the purity of Fluorinert so most likely they way you find out is "poof" you instantly destroy thousands or tens of thousands of dollars worth of gear.