Novec 7000 Project [immersive evaporating cooling]

[turtle83]

Here is the evidence of the liquid, has improved the performance of a single plate 946 to 963 MH/s, putting three plates have begun to have a pressure problem and I had to stop.


And a pretty cool video with the three boards running.

http://www.youtube.com/watch?v=_pNSUKp8ov4

Looks pretty cool... The pressure problem was because of you not able to condense the vapour fast enough?

[1714175859]

1714175859

[1714175859]

1714175859

[1714175859]

1714175859

[1714175859]

1714175859

[1714175859]

1714175859

[aTg]

Looks pretty cool... The pressure problem was because of you not able to condense the vapour fast enough?

Yes, it should be a very good design for the condenser, not worth anything like heatsinks are used with air.

[aTg]

A theory that was pressurized to connect all 3 plates was that I had too much air inside the tank, I personally do not share but hey I have been tested:





By connecting the second plate pressure starts to rise, the peltier or heatsink not enough.



We can measure the actual temperature of the FPGA by not having a heatsink above, confirms that the Novec 7000 boils at 34 ° as promised.

Video:

http://www.youtube.com/watch?v=QsRQrNSNISg

[aTg]

The problem is already discovered, was that the gas pushing the air up an the capacitor is on top of all the gas tank completely emptied the air and the next thing is to escape over the edge of the tank, the solution is down the condenser and let the lid no hermetic so air tight that moves can leave and re-enter if the FPGAs shuts down and gas down.



Just plug and starts to condense and the peltier does his job:



But when the FPGA begins to reach 230Mhz the condenser cooling capacity declines and reaches 30 º where condensation stops and looks like nothing and dries clear, begin to increase the gas level:



The cooling necessary to have a condenser at 25 ° or 16 ° which would be the ideal is too demanding for some equipment I do not have to necessarily work at 34, can work very well at 50 °.

[turtle83]

The cooling necessary to have a condenser at 25 ° or 16 ° which would be the ideal is too demanding for some equipment I do not have to necessarily work at 34, can work very well at 50 °.

U mean using a different liquid with boiling point 50 °C ?

[DeathAndTaxes]

http://en.wikipedia.org/wiki/FC-72

Boiling Point: 56C

[DeathAndTaxes]

Hi OP, you could try to contact Allied Control (http://www.allied-control.com/index.php). I know they have built and run a 1T+ mining farm using over 6000+ spartan FPGA with immersion cooling. But they have dissolved the farm a few months ago because of the difficulty and the ASIC miner.

Yeah that is another option for working fluid.  IIRC they used Novec 649 (boiling point of 50C).
http://solutions.3mmagyar.hu/3MContentRetrievalAPI/BlobServlet?lmd=1351678101000&locale=hu_HU&assetType=MMM_Image&assetId=1319241050803&blobAttribute=ImageFile

It has a higher boiling point of 50C.   I think a fluid around 45C to 55C is probably the sweet spot for chip cooling. The lower the boiling point relative to ambient temp the more difficult it is to cool the gas to below the boiling point so it condenses back into a liquid.  The issue the OP had with pressure is that the heatsink/peltier were insufficient to extract heat.  The FPGA was injecting x watts into the working fluid and the heatsink was removing y watts.  y<x so the amount of the fluid that was in gas form increased over time.  A larger heatsink, more powerful peltier, or an active compressor would improve stability.  Still one way to avoid it is like aTG said it is easier to just work at a higher temperature.

The again you don't want too high of a boiling point because you start to run up against the thermal limits of ASICs and FPGAs.  If the working fluid reaches a max temp of 50C, then the surface of the chip package will be ~50C.  The internal die temp will higher.  How much higher depends on thing like the chip design, surface area, and chip package.  Using CPU and GPU as a data point they have thermal diodes as part of the die to record the internal temp and they run 10C to 20C higher than working fluid temp.  In a two phase system (liquid->gas->liquid cycle) the package surface is never going to cooler than the boiling point as the boiling is what "removes" the heat.    A different route would be using a much higher boiling point and some circulating pumps with a heat exchanger (radiator) and just keep the entire system in one phase (liquid).  The system would never boil and would use circulation to move the heat from the chip to the radiator, but where is the fun in that.   Anyone know what the upper safe limit is for various FPGA & ASIC designs?  If any have thermal throttling where does it kick in at? On edit: Looks like Avalons default is 60C and BFL is 65C.  Not sure if that can be pushed higher.

Semi-OT but my rant on the excessively high cost limiting what a lot of potential innovation
Novec 649 is still ~$300 a gallon though.  Actually I don't know of any working fluid that has a high Dielectric Strength that isn't $300 per gallon or more.  Pricing is also tightly controlled so no free market at work.  3M is the only major supplier, it holds a lot of patents which it aggresively defends (so chance of startup making a Novec-649 "compatible" fluid is essentially nil) and it doesn't allow discounting.  Distributors or retailers that get caught risk being blacklisted and with only one supplier that means you are out.    It is as shame 3M has never been able (or willing) to bring the cost down.  I know it is a highly profitable niche market and I am sure 3M margins are something like 90%+ but imagine how many more applications would be economical at $20 or even $50 a gallon.  Part of the problem is 3M doesn't really want there to be a cheap solution.  For example one commercial application is for the cooling of lasers.  It kinda is the same scenario as cooling a CPU/GPU/FGPA/ASIC but it is a lot more energy (like 1000W or more) in a much smaller space.  The laser isn't immersed the laser shell is filled with the fluid.  This means much less is used.  A typical laser might only use 250CC (~1/16th of a gallon).  The system is completely sealed so they only replacement fluid is what is lost during maintenance.  Trust me the stuff is treated like liquid gold but you usually need to top the laser back up (maybe 20CC or less).  Maybe someday another company without the vested interest to keep prices high will design and patent a much cheaper working fluid.  If they do the cynic in me says 3M will buy them out and jack the price up 1000%.

[ultrix]

Why not just build a pressure chamber since you have to condense the working fluid and use R134a as its a fraction of the cost?  Would only take 80-100PSI to hit a similar range.

[DeathAndTaxes]

Why not just build a pressure chamber since you have to condense the working fluid and use R134a as its a fraction of the cost?  Would only take 80-100PSI to hit a similar range.

The cost of pressure chamber, the cost of compressor, the ongoing electrical cost. 
Still it is an interesting idea.  Essentially building an air conditioner around the heat source.  Would be a cool experiment.

[turtle83]

The issue the OP had with pressure is that the heatsink/peltier were insufficient to extract heat.  The FPGA was injecting x watts into the working fluid and the heatsink was removing y watts.  y<x so the amount of the fluid that was in gas form increased over time.  A larger heatsink, more powerful peltier, or an active compressor would improve stability.  Still one way to avoid it is like aTG said it is easier to just work at a higher temperature.

Yeah. Its not just about the total watts. At higher temperature delta more watts is going to get transferred. It is very possible that with same setup 50C liquid cools just fine...

[DeathAndTaxes]

The issue the OP had with pressure is that the heatsink/peltier were insufficient to extract heat.  The FPGA was injecting x watts into the working fluid and the heatsink was removing y watts.  y<x so the amount of the fluid that was in gas form increased over time.  A larger heatsink, more powerful peltier, or an active compressor would improve stability.  Still one way to avoid it is like aTG said it is easier to just work at a higher temperature.

Yeah. Its not just about the total watts. At higher temperature delta more watts is going to get transferred. It is very possible that with same setup 50C liquid cools just fine...

I probably wasn't clear but that is what I was saying (or trying to).  Too bad the easiest change (working fluid with higher boiling point) isn't a cheap switch.  Still very cool experiment.

[Gomeler]

Why not just build a pressure chamber since you have to condense the working fluid and use R134a as its a fraction of the cost?  Would only take 80-100PSI to hit a similar range.

I personally would be afraid of building a pressure vessel to contain the R134a. I suppose you could reuse something like a boiler or other already existing pressure vessels that would be sufficiently large. Immersive cooling with fluids that change phase at low temperatures brings all sorts of issues with safety.

It would be very interesting to see if any chips could scale in performance with temperature. Eventually, when chips are cheaper, I'd like to do such testing but sadly now is the time for that to potentially make economic sense. Use 1-10x more power to achieve 1-2x higher hash rate = more profit given the high fixed costs of the chips and general low power requirements. I imagine in the future when I can afford to blow up dozens/hundreds of chips in testing, the fixed costs of the chips will be a fraction of their expected energy consumption over their lifetime and thereby make the process of chilling them a waste of money.

[aTg]

Hi OP, you could try to contact Allied Control (http://www.allied-control.com/index.php). I know they have built and run a 1T+ mining farm using over 6000+ spartan FPGA with immersion cooling. But they have dissolved the farm a few months ago because of the difficulty and the ASIC miner.

I have not contacted them but still closely at their designs and their results, They use Novec 649 in some of their projects.

Yeah that is another option for working fluid.  IIRC they used Novec 649 (boiling point of 50C).

I've been researching this product and is a different family of 7000, this is a acetone that can break down the coating of the wires, we are studying other solution is to use the Novec 7100 has a voiling point to 61º and mix it with the 7000 to reduce it, this tells 3M is possible.

[DeathAndTaxes]

Hi OP, you could try to contact Allied Control (http://www.allied-control.com/index.php). I know they have built and run a 1T+ mining farm using over 6000+ spartan FPGA with immersion cooling. But they have dissolved the farm a few months ago because of the difficulty and the ASIC miner.

I have not contacted them but still closely at their designs and their results, They use Novec 649 in some of their projects.

Yeah that is another option for working fluid.  IIRC they used Novec 649 (boiling point of 50C).

I've been researching this product and is a different family of 7000, this is a acetone that can break down the coating of the wires, we are studying other solution is to use the Novec 7100 has a voiling point to 61º and mix it with the 7000 to reduce it, this tells 3M is possible.

One thing you may want to consider is Novec 1230.  It isn't designed to be a working fluid it is designed to fire supression however it has a high dielectric strength (designed to be sprayed into datacenters).   Getting a hold of it might be tough because it is only sold to OEM who do fire suppression installs.  However if you could acquire some my understanding is that is is only ~$40 per kg and thus substantially lower in cost then Novec 7100 and it has a boiling point of 49C.  If availability was a little better is may be a poor mans immersion cooling fluid.  I don't have a clue where you could acquire it as you likely would need to find a creative OEM.

http://solutions.3m.com/wps/portal/3M/en_US/3MNovec/Home/ProductCatalog/?PC_7_RJH9U5230OOA50IEKHCMDN11H0000000_nid=D7B0VRPJHSbeJ7332M26RCgl

[el_rlee]

Did you watch this http://www.youtube.com/watch?feature=player_detailpage&v=ivVoANqFBuY&t=568 ?

[aTg]

Did you watch this http://www.youtube.com/watch?feature=player_detailpage&v=ivVoANqFBuY&t=568 ?

Yes, I have seen the full video and I'm basing on the design to make the final tank, now I have the problem of deciding which chiller used, i think I'll buy a HAILEA HC-2200BH.
I will install 15 avalon in a space of 80cm and hopefully with this chiller is sufficient but the truth is that i have no idea.

http://www.hailea.com/e-hailea/product1/HC-2200BH.htm

[el_rlee]

Did you watch this http://www.youtube.com/watch?feature=player_detailpage&v=ivVoANqFBuY&t=568 ?

Yes, I have seen the full video and I'm basing on the design to make the final tank, now I have the problem of deciding which chiller used, i think I'll buy a HAILEA HC-2200BH.
I will install 15 avalon in a space of 80cm and hopefully with this chiller is sufficient but the truth is that i have no idea.

http://www.hailea.com/e-hailea/product1/HC-2200BH.htm

I don't find the cooling power specs of this chiller, however 7A electrical means 7x230=1610Watts, which is roughly double of what I have installed for my 3 Avalons.
My chiller is absolutely at the limit with the temperatures we are having at the moment in Shanghai.

I would recommend to use a fluid with higher boiling point and not use a chiller at all, just a dry or wet cooling tower, depending on your location and the fluid.

[18RATTT]

aTg have u decided which chiller you choose? keep us updated will ya

[DeathAndTaxes]

I will install 15 avalon in a space of 80cm and hopefully with this chiller is sufficient but the truth is that i have no idea.

http://www.hailea.com/e-hailea/product1/HC-2200BH.htm

It is weird that the company gives all the specs except the important one. If this reseller is accurate it looks like it can "remove" (transfer from water to air) up to a 3300W heat load.
http://www.orionairsales.co.uk/hailea-water-chiller-hc2200bh-1800-watt--2200-litre-cooling-capacity-1796-p.asp

That would make sense 3300W heat transferred / 1800 W power = COP 1.83   Larger (think whole house) heat pumps have a COP of 3 to 4 and cheaper ones close to 2.5.  So 1.83 at peak capacity for a smaller unit makes sense.

Your experiment has got me designing a immersion cooling system.   To lower operating costs I am considering a two stage cooling loop.  "Hot" water from the condenser is pumped to an aircooled heat exchanger (radiator and fan assembly) which lowers the temp to within 5C to 10C (depending on how large/expensive of an assembly you want.  Preferably this would be outside so the waste heat is simply dumped.  Now in cooler temps this alone likely would be sufficient but on hot summer days it might not be enough so the water then flows to a chiller and back to the condenser.  The only issue is picking the right boiling temp.  Air cooled heat exchange is more efficient the larger the temp difference between water inlet and air inlet.  Water is never going to be hotter than the immersion boiling point.  

Have you found a suitable condenser.  That has actually been harder then I thought it would be.  Really one just needs a bundle or tubes, nothing more complex.  I am sure a radiator would work but that doesn't seem optimal.
 

[Gomeler]

I will install 15 avalon in a space of 80cm and hopefully with this chiller is sufficient but the truth is that i have no idea.

http://www.hailea.com/e-hailea/product1/HC-2200BH.htm

It is weird that the company gives all the specs except the important one. If this reseller is accurate it looks like it can "remove" (transfer from water to air) up to a 3300W heat load.
http://www.orionairsales.co.uk/hailea-water-chiller-hc2200bh-1800-watt--2200-litre-cooling-capacity-1796-p.asp

That would make sense 3300W heat transferred / 1800 W power = COP 1.83   Larger (think whole house) heat pumps have a COP of 3 to 4 and cheaper ones close to 2.5.  So 1.83 at peak capacity for a smaller unit makes sense.

Your experiment has got me designing a immersion cooling system.   To lower operating costs I am considering a two stage cooling loop.  "Hot" water from the condenser is pumped to an aircooled heat exchanger (radiator and fan assembly) which lowers the temp to within 5C to 10C (depending on how large/expensive of an assembly you want.  Preferably this would be outside so the waste heat is simply dumped.  Now in cooler temps this alone likely would be sufficient but on hot summer days it might not be enough so the water then flows to a chiller and back to the condenser.  The only issue is picking the right boiling temp.  Air cooled heat exchange is more efficient the larger the temp difference between water inlet and air inlet.  Water is never going to be hotter than the immersion boiling point.  

Have you found a suitable condenser.  That has actually been harder then I thought it would be.  Really one just needs a bundle or tubes, nothing more complex.  I am sure a radiator would work but that doesn't seem optimal.
 

For your gas to water condenser look into brazed steel plate heat exchangers. I've used the ones sold by dudadiesel as heat exchangers in my cascades/autocascades and other refrigeration systems with no ill effects. Just be careful if you are condensing liquid such that you don't end up with a liquid trap. This is a bigger issue with oil for compressors in refrigeration systems though and luckily I used oil carrying refrigerants to minimize this problem.

All the water loop stuff should be a piece of cake.