Oil Submerged Configurations Discussion Thread

[Bicknellski]

As some of us poor souls are blessed with year round ambient temps of 28C plus and possibly reasonable costs for electricity but also keen on larger farm configurations I was hoping to pool our ideas here into thread about mineral oil submersion cooling. Personally I will be working on a DIY system for this in Indonesia. I will try and document the progress here.

Basing it off the Green Revolution Tank system.

http://www.youtube.com/watch?feature=player_embedded&v=JkuCFA1Vtio

http://www.youtube.com/watch?v=EZmm7P1mPZs

http://www.youtube.com/watch?v=6IX9U2zaI_I

Components:

Tank

Fiberglass Fish Tank $ unknown



Fluid

Crystal Plus 70T Mineral Oil

http://www.steoil.com/wp-content/uploads/2011/04/70T-MSDS.pdf

http://www.whiteoilnusantara.com/



Diagnostics
Pumps
Cabling

Heat Exchanger

Checking Pricing in Jakarta

http://www.metalindoengineering.com/pdf/Air%20Cooled%20HE/Plate%20Fin%20Coil.pdf

$109.00 + Free Shipping (USA ONLY)
12x12 Finned Coil Water to Air Heat Exchanger
Up to 60,000 BTU/hr


http://www.brazetek.com/


Cost Analysis (Savings / ROI)

[1715136772]

1715136772

[1715136772]

1715136772

[Bicknellski]

<RESERVED>

Cooling computer servers using air is inefficient and costly. As a result, some companies are returning to an older technology to cool data centers: submerging the hardware in liquid. Results show that significant energy savings can be achieved using their system in a super-computing environment.

Why not do the same with Mining Farms?

[BenTuras]

You're in good company: http://www.datacenterknowledge.com/archives/2012/09/04/intel-explores-mineral-oil-cooling/

I have also been thinking about this, I have chips for both Bitburner and Klondike boards coming and also have a large fish tank to use for this.

My main concerns are:

• Is it worth it, given the expectation that the boards will probably be costing more to run than generating BTC within a year.
• It can become messy, handling oil dripping electronics
• It will be a mayor disaster when the tank bursts, so I'll probably put the tank in the barn in the back of the garden

[Bicknellski]

Small scale designs is what I personally want as long as it is modular.

Depending on the number of boards at what point does it really make sense to do this?
Depending on overclocking at what point does it really make sense to do this?
Is it suitable for home applications?
Waste heat can it heat your water tank?

Those to me very important questions as well as yours BenTuras before going ahead. Personally I do not want to stop with one tank 10U - 42U capacity. Have to start small obviously with a 5U to 10U capacity as I suspect that the minimum for such a project to has look at the economies of scale. At some point it makes sense to use a system like this. If you are talking aquarium etc I think that is a great starting point for a proof of concept but will it save money? Waste heat might be useful though for heating your pool or house water in general.

Going to source tanks first that has same sort of dimensions as a rack server on its back.

[Bluestreak66]

I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.

[webjoe]

Definitely will be watching the progress on this.  The question remains if the method will be worthwhile for the added cost - I suppose if you had a huge farm this would be meaningful. Perhaps in 6-9 months, you can buy everyone's used Klondikes, overclock it to hell and prolong it's useful life since your electricity costs will be lower when factoring in the electricity for cooling in large scale.

[Bitcoinorama]

I too shall be watching, be careful, but do keep us posted!

[BenTuras]

One of the challenges I got, is select a brand/type mineral oil here in Holland.
I found this one: http://www.sunoco.be/uk/whiteoils_med.html, the MSDS While Oil F.
This firm is located in Belgium, just 40 miles from me.
I have sent them an email with some questions.

[BenTuras]

Another thing I have to decide on is whether or not to put a heatsink on the boards.
And if I do, which cooling paste.
Obviously I can't use the standard conducting cooling paste.

[Bicknellski]

I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.

Any published data on this in terms of QFN chips having been submerged in oil and they have had failures or issues as you describe? Just want to see what has been written up about this. Considering that this sort of cooling in done in a fairly wide application commercially I have yet to see anything specifically about issues with QFN chips immersed in oil. Although the scenario you paint will possibly happen to someone I am sure what would be the likelihood of having to do that? It is a risk that is taking place daily but again I have not found anything yet on this being reported as widespread issue. They are dropping OEM 1U rack mount servers directly in this without any concerns about QFN or other types of chips. Seems strange can't find anything when I search about it.

Also what I am looking to do is scale this up for a data center applications for mining. So water blocks would at some point be more problematic however for a smaller number of boards I'd agree go with air or water blocks.

[Bicknellski]

Another thing I have to decide on is whether or not to put a heatsink on the boards.
And if I do, which cooling paste.
Obviously I can't use the standard conducting cooling paste.

Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php

[crazyates]

Ya I was seriously looking into them a week or two ago. Lots of good stuff. I'd also be interested in this project, but a big question I had: How would you dissipate all that heat once you remove it from the electronics/oil?

[Bicknellski]

Ya I was seriously looking into them a week or two ago. Lots of good stuff. I'd also be interested in this project, but a big question I had: How would you dissipate all that heat once you remove it from the electronics/oil?

Heat Exchanger / Cooling Tower

FOB Price:   US $1 / Piece Get Latest Price
Port:   ningbo.shanghai
Minimum Order Quantity:   50 Piece/Pieces
Supply Ability:   10000 Piece/Pieces per Month
Payment Terms:   T/T




GPC23  <10 GAL    105 LBS   50" X 28"   25 GPM



Evaporation Cooler (For Conception Purposes Only)

http://www.youtube.com/watch?v=EJKH1lqGvbQ

http://www.youtube.com/watch?v=TA847PhQaGE

[BenTuras]

Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php

Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.

[turtle83]

Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php

Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.

Wondering the same thing. Because the board is a fairly large surface area, if you keep the oil flowing using pumps, shouldnt it be sufficient? Need someone with knowledge of thermodynamics to pitch in...

[crazyates]

Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php

Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.

Wondering the same thing. Because the board is a fairly large surface area, if you keep the oil flowing using pumps, shouldnt it be sufficient? Need someone with knowledge of thermodynamics to pitch in...

Of course you still need a heatsink. You just won't need any fans on that heatsink, which helps you save power.

Mineral oil can absorb a lot more heat than air, but it can't absorb that much heat with that little surface area.

[shackleford]

I have had a mineral oil setup for my FPGAs for over a year. Total wattage  is ~65 and the temperature was OK @37-42c ish depending on ambient . Heat syncs are mandatory and so is forced convection (one fan to keep the oil moving) otherwise the hot oil seems to stay around the heat source and does not flow away like you would think because the oil is thick.  I planned on reusing the setup for a BFL single but that won't be happening since they came nowhere close to their power goals. I may try building a new one just because I enjoyed building it. For the much higher wattage it would need a complete redesign and much more oil and more metal/glass/acrylic to transfer heat from the oil to outside the case. Unless you are doing it for fun it is not really worth the expenditure.  Water blocks is probably the better route... but look nowhere as cool (or as silent).

[Bicknellski]

Ates  is right NEEEEEEED the sink.

There are other insoluble (non-grease) heat sink compounds but maybe the pads might be the best route.

[crazyates]

maybe the pads might be the best route.

http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.

[Bicknellski]

maybe the pads might be the best route.

http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.

Sure there are cheaper alternatives directly from China.

http://www.aliexpress.com/item/5Pcs-lot-30CMx3CMx0-3MM-Thermal-Pad-CPU-GPU-DDR-RAM-CHIP-3320-01-05/632016505.html

US $3.22 / lot
5 pieces / lot , $0.64 / piece