Bitcoin Forum

Other => Off-topic => Topic started by: Bicknellski on July 02, 2013, 05:18:40 AM



Title: Oil Submerged Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 05:18:40 AM
As some of us poor souls are blessed with year round ambient temps of 28C plus and possibly reasonable costs for electricity but also keen on larger farm configurations I was hoping to pool our ideas here into thread about mineral oil submersion cooling. Personally I will be working on a DIY system for this in Indonesia. I will try and document the progress here.

Basing it off the Green Revolution Tank system.

http://www.youtube.com/watch?feature=player_embedded&v=JkuCFA1Vtio

http://www.youtube.com/watch?v=EZmm7P1mPZs

http://www.youtube.com/watch?v=6IX9U2zaI_I

Components:

Tank

Fiberglass Fish Tank $ unknown

http://i1219.photobucket.com/albums/dd421/toyodafrp/IMG00483-20101229-1340.jpg

Fluid

Crystal Plus 70T Mineral Oil

http://www.steoil.com/wp-content/uploads/2011/04/70T-MSDS.pdf

http://www.whiteoilnusantara.com/

http://i230.photobucket.com/albums/ee106/PFC4L1FE/mineraloil_zps7f5346ac.png (http://s230.photobucket.com/user/PFC4L1FE/media/mineraloil_zps7f5346ac.png.html)

Diagnostics
Pumps
Cabling

Heat Exchanger

Checking Pricing in Jakarta

http://www.metalindoengineering.com/pdf/Air%20Cooled%20HE/Plate%20Fin%20Coil.pdf

$109.00 + Free Shipping (USA ONLY)
12x12 Finned Coil Water to Air Heat Exchanger
Up to 60,000 BTU/hr

http://www.brazetek.com/files/imagecache/product/wahx_2_11.jpg
http://www.brazetek.com/


Cost Analysis (Savings / ROI)


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 05:22:42 AM
<RESERVED>

Cooling computer servers using air is inefficient and costly. As a result, some companies are returning to an older technology to cool data centers: submerging the hardware in liquid. Results show that significant energy savings can be achieved using their system in a super-computing environment.

Why not do the same with Mining Farms?


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 02, 2013, 06:48:43 AM
You're in good company: http://www.datacenterknowledge.com/archives/2012/09/04/intel-explores-mineral-oil-cooling/

I have also been thinking about this, I have chips for both Bitburner and Klondike boards coming and also have a large fish tank to use for this.

My main concerns are:
  • Is it worth it, given the expectation that the boards will probably be costing more to run than generating BTC within a year.
  • It can become messy, handling oil dripping electronics
  • It will be a mayor disaster when the tank bursts, so I'll probably put the tank in the barn in the back of the garden ;)



Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 07:05:54 AM
Small scale designs is what I personally want as long as it is modular.

Depending on the number of boards at what point does it really make sense to do this?
Depending on overclocking at what point does it really make sense to do this?
Is it suitable for home applications?
Waste heat can it heat your water tank?

Those to me very important questions as well as yours BenTuras before going ahead. Personally I do not want to stop with one tank 10U - 42U capacity. Have to start small obviously with a 5U to 10U capacity as I suspect that the minimum for such a project to has look at the economies of scale. At some point it makes sense to use a system like this. If you are talking aquarium etc I think that is a great starting point for a proof of concept but will it save money? Waste heat might be useful though for heating your pool or house water in general.

Going to source tanks first that has same sort of dimensions as a rack server on its back.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bluestreak66 on July 02, 2013, 07:19:20 AM
I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: webjoe on July 02, 2013, 07:24:51 AM
Definitely will be watching the progress on this.  The question remains if the method will be worthwhile for the added cost - I suppose if you had a huge farm this would be meaningful. Perhaps in 6-9 months, you can buy everyone's used Klondikes, overclock it to hell and prolong it's useful life since your electricity costs will be lower when factoring in the electricity for cooling in large scale.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bitcoinorama on July 02, 2013, 12:28:27 PM
I too shall be watching, be careful, but do keep us posted!


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 02, 2013, 01:49:11 PM
One of the challenges I got, is select a brand/type mineral oil here in Holland.
I found this one: http://www.sunoco.be/uk/whiteoils_med.html, the MSDS While Oil F.
This firm is located in Belgium, just 40 miles from me.
I have sent them an email with some questions.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 02, 2013, 01:51:46 PM
Another thing I have to decide on is whether or not to put a heatsink on the boards.
And if I do, which cooling paste.
Obviously I can't use the standard conducting cooling paste.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 03:53:25 PM
I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.

Any published data on this in terms of QFN chips having been submerged in oil and they have had failures or issues as you describe? Just want to see what has been written up about this. Considering that this sort of cooling in done in a fairly wide application commercially I have yet to see anything specifically about issues with QFN chips immersed in oil. Although the scenario you paint will possibly happen to someone I am sure what would be the likelihood of having to do that? It is a risk that is taking place daily but again I have not found anything yet on this being reported as widespread issue. They are dropping OEM 1U rack mount servers directly in this without any concerns about QFN or other types of chips. Seems strange can't find anything when I search about it.

Also what I am looking to do is scale this up for a data center applications for mining. So water blocks would at some point be more problematic however for a smaller number of boards I'd agree go with air or water blocks.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 04:19:18 PM
Another thing I have to decide on is whether or not to put a heatsink on the boards.
And if I do, which cooling paste.
Obviously I can't use the standard conducting cooling paste.


Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link) (http://www.indium.com/thermal-interface-materials/heat-spring/), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: crazyates on July 02, 2013, 05:05:55 PM
Ya I was seriously looking into them a week or two ago. Lots of good stuff. I'd also be interested in this project, but a big question I had: How would you dissipate all that heat once you remove it from the electronics/oil?


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 05:28:22 PM
Ya I was seriously looking into them a week or two ago. Lots of good stuff. I'd also be interested in this project, but a big question I had: How would you dissipate all that heat once you remove it from the electronics/oil?

Heat Exchanger / Cooling Tower

FOB Price:   US $1 / Piece Get Latest Price
Port:   ningbo.shanghai
Minimum Order Quantity:   50 Piece/Pieces
Supply Ability:   10000 Piece/Pieces per Month
Payment Terms:   T/T

http://t1.gstatic.com/images?q=tbn:ANd9GcR_Q6R-wb0TJKjusH3SOqNMIKVaA8BawxoBy4844MP9ESan6hKr0A


GPC23  <10 GAL    105 LBS   50" X 28"   25 GPM

http://glacierpoolcoolers.com/images/gpc23.png

Evaporation Cooler (For Conception Purposes Only)

http://www.youtube.com/watch?v=EJKH1lqGvbQ

http://www.youtube.com/watch?v=TA847PhQaGE


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 02, 2013, 06:30:08 PM
Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link) (http://www.indium.com/thermal-interface-materials/heat-spring/), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php
Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: turtle83 on July 02, 2013, 06:37:41 PM
Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link) (http://www.indium.com/thermal-interface-materials/heat-spring/), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php
Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.

Wondering the same thing. Because the board is a fairly large surface area, if you keep the oil flowing using pumps, shouldnt it be sufficient? Need someone with knowledge of thermodynamics to pitch in...


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: crazyates on July 02, 2013, 06:43:10 PM
Server Modification & Installation

While the CarnotJet system is being installed in your facility, we will also modify servers at the installation site. Having previously encapsulated server hard drives at GRC’s facility, our technicians now remove server fans (which are stored in case the server is ever reconverted to air use).  Conventional thermal grease will dissolve, and so we replace it with a non-soluble thermal interface material solution, a Heat-Spring® by Indium Corp of America (link) (http://www.indium.com/thermal-interface-materials/heat-spring/), which helps enable this technology.  Solid State Drives do not require any modification.

- See more at: http://www.grcooling.com/product/installation/#sthash.yMkO2kNI.dpuf

Indium foil ... expensive... http://www.aliexpress.com/store/product/Indium-Foil-Size-0-15x100x100mm/334807_917979672.html


Thermal pads might be an option http://www.tglobalthermal.com/h48-2-thermal-conductive-pad.php
Yeah, I saw that too, that's why I am wondering whether or not we need a heatsink.
Wondering the same thing. Because the board is a fairly large surface area, if you keep the oil flowing using pumps, shouldnt it be sufficient? Need someone with knowledge of thermodynamics to pitch in...
Of course you still need a heatsink. You just won't need any fans on that heatsink, which helps you save power.

Mineral oil can absorb a lot more heat than air, but it can't absorb that much heat with that little surface area.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: shackleford on July 02, 2013, 08:43:22 PM
I have had a mineral oil setup for my FPGAs for over a year. Total wattage  is ~65 and the temperature was OK @37-42c ish depending on ambient . Heat syncs are mandatory and so is forced convection (one fan to keep the oil moving) otherwise the hot oil seems to stay around the heat source and does not flow away like you would think because the oil is thick.  I planned on reusing the setup for a BFL single but that won't be happening since they came nowhere close to their power goals. I may try building a new one just because I enjoyed building it. For the much higher wattage it would need a complete redesign and much more oil and more metal/glass/acrylic to transfer heat from the oil to outside the case. Unless you are doing it for fun it is not really worth the expenditure.  Water blocks is probably the better route... but look nowhere as cool (or as silent).


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 08:46:11 PM
Ates  is right NEEEEEEED the sink.

There are other insoluble (non-grease) heat sink compounds but maybe the pads might be the best route.



Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: crazyates on July 02, 2013, 09:12:53 PM
maybe the pads might be the best route.
http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 02, 2013, 09:22:49 PM
maybe the pads might be the best route.
http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.

Sure there are cheaper alternatives directly from China.

http://www.aliexpress.com/item/5Pcs-lot-30CMx3CMx0-3MM-Thermal-Pad-CPU-GPU-DDR-RAM-CHIP-3320-01-05/632016505.html

US $3.22 / lot
5 pieces / lot , $0.64 / piece


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: turtle83 on July 02, 2013, 10:09:28 PM
maybe the pads might be the best route.
http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.

Sure there are cheaper alternatives directly from China.

http://www.aliexpress.com/item/5Pcs-lot-30CMx3CMx0-3MM-Thermal-Pad-CPU-GPU-DDR-RAM-CHIP-3320-01-05/632016505.html

US $3.22 / lot
5 pieces / lot , $0.64 / piece

But... 17.0 W/mK vs 3.2W/mK ...

If i had 400 chips to use up 1 expensive sheet (and all the time in the world), id put the expensive one just under the vias, use the cheap sheet (with holes) for rest of the board.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: PaperClip on July 02, 2013, 11:03:13 PM
I also had idea to submerge miner into oil, but in the end decided to avoid this. A/C will be enough for one device like in my case, and its not so safe to place tank full of hot oil in living room.

Anyway, you may find this advice useful:
Have anyone tried to completely submerge Avalon into mineral oil? That works very well for desktop computers.

Fluid would be too viscous for the fanz...;)

I've got a system that's submerged in mineral oil (quad E5-4650).  Transformer oil to be specific (lower viscosity, higher flash temps, mostly synthetic, no PBAs).   Fans don't really care, although that's not the primary circulation method (pump).


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bluestreak66 on July 02, 2013, 11:12:48 PM
I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.

Any published data on this in terms of QFN chips having been submerged in oil and they have had failures or issues as you describe? Just want to see what has been written up about this. Considering that this sort of cooling in done in a fairly wide application commercially I have yet to see anything specifically about issues with QFN chips immersed in oil. Although the scenario you paint will possibly happen to someone I am sure what would be the likelihood of having to do that? It is a risk that is taking place daily but again I have not found anything yet on this being reported as widespread issue. They are dropping OEM 1U rack mount servers directly in this without any concerns about QFN or other types of chips. Seems strange can't find anything when I search about it.

Also what I am looking to do is scale this up for a data center applications for mining. So water blocks would at some point be more problematic however for a smaller number of boards I'd agree go with air or water blocks.
No one is going to overclock a server in a cooperate environment, with servers up time and longevity is key. When you have that much hardware in such a small space cooling becomes a huge issue thats where the oil cooling comes in. Also with that type of equipment once it breaks, its not repaired at the component level. There is not a specific issue with submerging QFN in oil or any other type of package for that matter. My point was this: Inevitably some chips with die from manufacture defects, power fluctuation, or hundreds of other unforeseen possibilities, cooling is just one of those. Having a 16 chip per board, if one chip dies then chances are you will lose that board because it will be too difficult to repair after being in oil. Ask any repair tech how easy it would be to repair an oil soaked board. I repair arcade equipment at work and can tell you it would not be easy with oil soaked boards.

Convection will draw the heat to the top of the tank pump that through a oil cooler to the bottom to aid in circulation. Straight mineral oil work good and can be purchased cheap by the gallon that's what we used. Transformer oil works well as it is designed to cool transformer windings.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: joeventura on July 03, 2013, 01:11:01 AM
Assuming this is your plan for cooling overclocked (350-400M) Avalons

Oil cooling for stock 4.5-5Gh Klondikes is incredible overkill.
A decent heatsink and a fan is plenty


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 04:02:55 AM
Assuming this is your plan for cooling overclocked (350-400M) Avalons

Oil cooling for stock 4.5-5Gh Klondikes is incredible overkill.
A decent heatsink and a fan is plenty

It isn't overkill when you have 10's or 100's of K16/K64/K256 units.

I put the caveats in the first post and my reply early on. You really do NOT need to do this in terms of a single K16 unless you are doing it to learn more about the process. My intention is to have as I said before a unit that can handle some thing like 5U to 10U as a prototype module that can then scale up by adding more modules. In terms of failures... with a larger array single chip failures are not going to matter much as at this scale you probably can easily remove a K16 board from the array and keep plugging along with no problem. Cleaning up a K16 is not going to be that hard given 100mm x 100mm size and placing a new Avalon chip on the board as will be reasonably simple to do. Infact given I have access to fabrication of boards I will probably have a few dry boards straight from the fabrication unopened. Replacement I don't think is a huge issue with the modular capabilities of the Klondike.

K256 X 10 = 2560 chips roughly 1,024,000 Mh/s if you can attain 400+ Mh/s per chip mineral oil cooled.
K256 X 10 = 2560 chips roughly 896,000 Mh/s if you can attain 350 per chip air cooled with a/c.

There is definitely potential in this.

So continuing on... anyone found problems with the silicone heat sink pads and mineral oil?

I think next I want to mock up is a 5U capacity tank, heat exchanger / cooling tower.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 03, 2013, 04:51:45 AM
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics.  Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil.  Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil.  Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried.  These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in.

As far as heat exchangers, there are two routes I've used with great success:
1) Liquid to air - Simple radiator style setup.  This however should _not_ be cheaped on.  Initially I tried a PC water cooling radiator (leftover from a friend's system).  PC radiators in general are crap.  They don't provide heat curves, operating tolerances, etc.  In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers.  This type of system is the best performance per price for < 35k BTU/hr (about 10kw).  Also, I would avoid using polymer tubing for any of the hoses.  I use silver solder welded copper pipe aka sweated copper.

2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel.  You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer.  I sent the hot oil into one of the chambers, cool water into the other.   For cooling the water, you can use an evaporative system, and ideally one would.  You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature.  This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw.

I wouldn't cheap on the radiators, pipes, and especially pumps.  Avoid pond pumps and low duty cycle pumps.  Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside.

If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block.
I've got a bunch of data on this at home, but am on the road right now.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 04:54:52 AM
http://www.ehow.com/how_7640669_circuit-board-fabrication-cleaning.html (http://www.ehow.com/how_7640669_circuit-board-fabrication-cleaning.html)

Cleaning the Substrate

1 Put on a pair of latex or rubber gloves. One of the worst contaminants in the production of PCBs is oil, which comes from your hands and keeps the surface from accepting resist evenly.

2 Scrub each board with a plastic scrub pad to remove any oxide coating on the copper side of the board. You can acquire a plastic scrub pad at your local grocery store where dish washing products are sold. Do not use water when performing the scrubbing as this will promote a new oxide coating on the board during the process. Handle the board by the edges. The less contact with the copper cladding the less chance of fouling the surface. Many manufacturers use acid to remove the oxide coating, but physical scrubbing works just as well. When you have finished scrubbing it, the surface should appear shiny with no spots.

3 Mix a small amount of metal free soap in deionized water and place in a container large enough to hold the PCBs.

4 Rinse the board in the mild soap solution. This will remove most remaining oil and inorganic contaminants from the surface of the board.

5 Wash the board in an isopropyl alcohol bath to remove any other organic residues remaining.

6 Rinse the cleaned board with acetone. Isopropyl alcohol contains approximately 30% water and the acetone will remove any alcohol or water left adhering to the board. The acetone evaporates quickly and doesn't leave a residue.

Cleaning the Finished Board

7 Put on a pair of latex or rubber gloves to protect the board from the oil present on your skin.

8 Dip a cotton tipped swab into isopropyl alcohol and carefully scrub each solder connection on the PCB. Scrub both sides of the board where solder connections were made.

9 Dip a new cotton tipped swab in deionized water and repeat the process of cleaning each solder connection on both sides of the board. Depending on the type of flux used in the soldering of components on the boards, any residue will be soluble in either isopropyl alcohol or deionized water.

10 Rinse the cleaned board under cold deionized water. This will remove any remaining soap or isopropyl alcohol.

11 Place in an oven at 100 degrees C to dry for approximately five minutes. The heat will dry any remaining water and leave the finished product ready for testing.

Tips & Warnings
Keep your fingers off the surface of the board. Small traces of skin oil can short the board or components on the board.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 04:57:20 AM
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics.  Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil.  Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil.  Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried.  These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in.

As far as heat exchangers, there are two routes I've used with great success:
1) Liquid to air - Simple radiator style setup.  This however should _not_ be cheaped on.  Initially I tried a PC water cooling radiator (leftover from a friend's system).  PC radiators in general are crap.  They don't provide heat curves, operating tolerances, etc.  In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers.  This type of system is the best performance per price for < 35k BTU/hr (about 10kw).  Also, I would avoid using polymer tubing for any of the hoses.  I use silver solder welded copper pipe aka sweated copper.

2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel.  You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer.  I sent the hot oil into one of the chambers, cool water into the other.   For cooling the water, you can use an evaporative system, and ideally one would.  You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature.  This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw.

I wouldn't cheap on the radiators, pipes, and especially pumps.  Avoid pond pumps and low duty cycle pumps.  Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside.

If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block.
I've got a bunch of data on this at home, but am on the road right now.


+1 Sweet information there... I will compare the cost of transformer oil to the straight industrial mineral oil costs here in Jakarta. Great post.

Have you had any issues with repair and degreasing boards then having to reflow etc Ultrix?
Also in terms of thermal compound and pads what do you recommend given the solubility of some of the heat sink compounds as mentioned above.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bluestreak66 on July 03, 2013, 05:11:56 AM

K256 X 10 = 2560 chips roughly 1,024,000 Mh/s if you can attain 400+ Mh/s per chip mineral oil cooled.
K256 X 10 = 2560 chips roughly 896,000 Mh/s if you can attain 350 per chip air cooled with a/c.

There is definitely potential in this.

So continuing on... anyone found problems with the silicone heat sink pads and mineral oil?

I think next I want to mock up is a 5U capacity tank, heat exchanger / cooling tower.

You should have no problem with silicone compatibility with mineral oil. How many boards do you plan on putting in a 5u? K256 x 10 would be 160 boards Is this your plan?


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 05:15:52 AM
No it would be 10 X 1U units of K256.

Have a roughed out design for blades that are 4 x K64s daisy chained.

The prototype will be 2 x 1U (K256) total 512 chips in a blade configuration or 1U server pan then suspended in a tank that can support 5U to 10U rack.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 03, 2013, 05:21:10 AM
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics.  Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil.  Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil.  Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried.  These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in.

As far as heat exchangers, there are two routes I've used with great success:
1) Liquid to air - Simple radiator style setup.  This however should _not_ be cheaped on.  Initially I tried a PC water cooling radiator (leftover from a friend's system).  PC radiators in general are crap.  They don't provide heat curves, operating tolerances, etc.  In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers.  This type of system is the best performance per price for < 35k BTU/hr (about 10kw).  Also, I would avoid using polymer tubing for any of the hoses.  I use silver solder welded copper pipe aka sweated copper.

2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel.  You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer.  I sent the hot oil into one of the chambers, cool water into the other.   For cooling the water, you can use an evaporative system, and ideally one would.  You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature.  This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw.

I wouldn't cheap on the radiators, pipes, and especially pumps.  Avoid pond pumps and low duty cycle pumps.  Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside.

If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block.
I've got a bunch of data on this at home, but am on the road right now.


+1 Sweet information there... I will compare the cost of transformer oil to the straight industrial mineral oil costs here in Jakarta. Great post.

Have you had any issues with repair and degreasing boards then having to reflow etc Ultrix?
Also in terms of thermal compound and pads what do you recommend given the solubility of some of the heat sink compounds as mentioned above.

No way anything will reflow.  You want to use thermocouples to monitor temperature at various locations in your tank and a control system that will kill power to the heat source if a critical temperature is reached.
With regards to system maintenance, I use a process similar to one you described but with a predecessor step of letting the parts hang above a catch pan after heating the parts arbitrarily with a heat gun (or blow dryer).  All and all, its less mess than dealing with dust caked components.

Edited:
My whole process is this:
1) hang component above catchpan and heat as described
2) after 20 minutes or so, dunk in a pan of rubbing alcohol and agitate.
3) let hang for another 10

Again, the first step is to avoid going through tons of paper towels and alcohol.  I've done PCB rework including reflow on parts cleaned by this method.  There is no observable/measurable difference.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 05:25:50 AM
You mean you can't remove parts (de-solder) and then re-flow a PCB after being in oil? In case you want to replace a blown QFN chip. Thanks!

Nice info on the heat gun and drip tray thanks. Be picking your brain more and more.

Need to have a catch tray and rack will think about some sort of caddying on wheels for this.

What about palm or coconut oil? We have bulk amount here in Indonesia wondering if that would be possible? Confirmed and Tested.


http://www.elect.mrt.ac.lk/Coconut_oil_eru_2001.pdf

Code:
Considering all the above facts, we can conclude that coconut oil can be 
used as transformer oil. However further tests need to be done under actual
operating conditions. Coconut oil is freely available in Sri Lanka and it may
become cheaper than other transformer oil. Most importantly it is 100%
environmental friendly, hence it can be easily used as an alternative to
traditional transformer oil.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 03, 2013, 05:33:23 AM
No it would be 10 X 1U units of K256.

Have a roughed out design for blades that are 4 x K64s daisy chained.

The prototype will be 2 x 1U (K256) total 512 chips in a blade configuration or 1U server pan then suspended in a tank that can support 5U to 10U rack.

Minor recommendation, lay the rack on its back.  In fact avoid using off the shelf racks.  Just weld together the structure with some 1" to 1.5", 1/4" thick angle for a rack and frame.  Then weld sheets on for sides of the tank and you're good to go.   The reason I say avoid off the shelf racks is that most are only structurally designed to be run vertically.  Further horizontal affords servicing of individual machines without the need to drain the tank.  You can assemble a pretty nice 41U tank for ~$800 USD.  This will require ~350 gallons of transformer oil, which will run you ~$4k delivered.   I would test the tank with water prior to filling as water is less viscous and namely cheaper.  So for each rack you're looking at about $5k + (cost of tower / number of racks).  When you compare that with the costs of AC, especially in high power density applications, the initial cost is about 1.5x higher, however the fact you can cool 250kw with 10kw of cooling equipment is unbeatable.   In an air environment you're looking at power costs approximately equal to that of the equipment you're running.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 05:56:58 AM
Fabrication costs here are low very very low so I am sure I can have something welded up to any specifications I want.

Ya looking at that Green Revolution Carnojet system the tank actually has cabling and racking solution built into the tank. I want to do a smaller build as I will have this inside on the 3rd floor of my school (empty 3rd floor) and I definitely do not want to be running A/C all day and night to cool even a modest number of units for now but want to be able to scale up from 5U or 10U simply adding the tank and racking that can be modular like a data center racking system but easily handled by kids. I think this project will be added to our curriculum as well for second semester so many interesting aspects to be covered and it lines up perfectly with so many aspect in the science curriculum it is scary!

Thanks again. I will put up a rough design soon as I think given the posts you made I am pretty confident I can acquire pretty much everything I need in very little time and at very reasonable costs.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: turtle83 on July 03, 2013, 07:54:10 AM
You mean you can't remove parts (de-solder) and then re-flow a PCB after being in oil? In case you want to replace a blown QFN chip. Thanks!

Nice info on the heat gun and drip tray thanks. Be picking your brain more and more.

Need to have a catch tray and rack will think about some sort of caddying on wheels for this.

What about palm or coconut oil? We have bulk amount here in Indonesia wondering if that would be possible? Confirmed and Tested.


http://www.elect.mrt.ac.lk/Coconut_oil_eru_2001.pdf

Code:
Considering all the above facts, we can conclude that coconut oil can be 
used as transformer oil. However further tests need to be done under actual
operating conditions. Coconut oil is freely available in Sri Lanka and it may
become cheaper than other transformer oil. Most importantly it is 100%
environmental friendly, hence it can be easily used as an alternative to
traditional transformer oil.

But be prepared for the smell :)


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 08:01:01 AM
Distilled at 200C smell removed... according to the article.


Code:
Coconut oil contains odor and tasting compounds, which are actually 
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.

To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: turtle83 on July 03, 2013, 08:12:23 AM
Distilled at 200C smell removed... according to the article.


Code:
Coconut oil contains odor and tasting compounds, which are actually 
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.

To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay.

Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia...


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 03, 2013, 08:16:52 AM
Distilled at 200C smell removed... according to the article.


Code:
Coconut oil contains odor and tasting compounds, which are actually 
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.

To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay.

Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia...

Reading further can be used in cold climate as well... as they can reduce the pour temp significantly... but I won't be concerned about that here in Jakarta. Although a purified Coconut Oil will be best. Not sure on cost difference between that transformer oil and mineral oil. Will start checking.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 03, 2013, 08:44:50 AM
Distilled at 200C smell removed... according to the article.


Code:
Coconut oil contains odor and tasting compounds, which are actually 
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.

To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay.

Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia...

Reading further can be used in cold climate as well... as they can reduce the pour temp significantly... but I won't be concerned about that here in Jakarta. Although a purified Coconut Oil will be best. Not sure on cost difference between that transformer oil and mineral oil. Will start checking.

I'd stick to synthetic transformer oil, little to no degrading under the heat computers generate.  Although the carnojet system claims to use a "proprietary" coolant, I believe what they use is opticool made by DSI ventures.  I've been meaning to get in touch with them as they're close, however I have a ready supply of the 76 transformer oil here.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 03:26:53 AM
I think the carnojet DOES NOT use proprietary oils... infact all it is unscented mineral oil from what I get off their website.

Looking at tanks have to be careful of the plastic in the tank due to potential temperature spikes.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: itod on July 04, 2013, 07:48:40 AM
There's been some transformer oil leakage in my country from smaller, local, transformers and has been reported in the press as environmental danger, also mentioning cancerous effects, etc. Were these reports FUD or is there really some medical issues from being in contact with it? I plan to keep miner in my home and don't want any medical risk to my family (if there is one at all).


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: m5 on July 04, 2013, 08:40:28 AM
I have had a mineral oil setup for my FPGAs for over a year. Total wattage  is ~65 and the temperature was OK @37-42c ish depending on ambient . Heat syncs are mandatory and so is forced convection (one fan to keep the oil moving) otherwise the hot oil seems to stay around the heat source and does not flow away like you would think because the oil is thick.  I planned on reusing the setup for a BFL single but that won't be happening since they came nowhere close to their power goals. I may try building a new one just because I enjoyed building it. For the much higher wattage it would need a complete redesign and much more oil and more metal/glass/acrylic to transfer heat from the oil to outside the case. Unless you are doing it for fun it is not really worth the expenditure.  Water blocks is probably the better route... but look nowhere as cool (or as silent).
These guys http://www.pugetsystems.com/mineral-oil-pc.php have been running entire PC for few years in a miral oil and their findings are that the oil can easily transfer the heat from CPU even without any pumps. Although the CPU was quite hot (88 degress C), it was running stable which means the heat transfer from it was fast enough (it won't work reliably at 88deg in air). They also noted that bubble of air help to transfer the heat.
But don't forget that oil can only help you to move the heat away from the chips (so you can do more overclocking), but it still heats the room where it is located, so it won't you save anything on A/C for example.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 08:41:02 AM
There's been some transformer oil leakage in my country from smaller, local, transformers and has been reported in the press as environmental danger, also mentioning cancerous effects, etc. Were these reports FUD or is there really some medical issues from being in contact with it? I plan to keep miner in my home and don't want any medical risk to my family (if there is one at all).

No PCBs in the oil we are talking. I personally won't be using anything like that. If any doubt I will go with mineral oil (baby oil) non-toxic. We are also designing the heat exchanger so it will go outside so no worries about that room filling with heat. Like a split air conditioning unit.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 04, 2013, 09:26:53 AM
I just have a brilliant idea...
How about an old washing machine, top loader ?

http://i.marktplaats.com/00/s/NjQwWDQ4MA==/$T2eC16dHJHYFFkLsiUj3BR0Ccjy(hg~~60_85.JPG


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 10:16:07 AM
FOOKING AWESOME IDEA there BEN.  ;D

Capacity? 40 - 60 litres


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 04, 2013, 11:20:17 AM
FOOKING AWESOME IDEA there BEN.  ;D

Capacity? 40 - 60 litres

LOL. Yeah, ideas start brilliant, then you really think and question

  • how to get the old washing powder residu out of the system
  • will the hoses cope with the mineral oil
  • input of the pump is on the bottom where the 'cold' oil is

On the other hand,
  • it should be water tight
  • there is a pump to circulate the oil


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: qiuness on July 04, 2013, 11:32:01 AM
i`m also watching this for my GPU Farm.. although it sounds very expensive for multiple PC rigs


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 11:56:13 AM
I was thinking something like this but as a fiberglass unit.

http://img.directindustry.com/images_di/photo-g/4-wheel-waste-disposal-container-16816-2276585.jpg

Recycling all the rage... what about an old freezer? http://webpages.charter.net/redusc/Freezer.html Don't drill through the walls for piping.

http://bit.ly/YgNpWG


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 04, 2013, 12:07:38 PM
Do you own a refinery ?! That will use a lot of mineral oil, darn!
LOL, -18 degrees C mineral oil, better get one that isn't frozen at that temp! (In reality, the Avalon chips will heat the oil more than the freezer can cool it)


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bitcoinorama on July 04, 2013, 12:09:31 PM
Someone on another thread added a link to an awesome fish tank project with a pc, ideal size, it worked, and they now make a kit all components included for you for around US $500...


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 04, 2013, 12:14:08 PM
Someone on another thread added a link to an awesome fish tank project with a pc, ideal size, it worked, and they now make a kit all components included for you for around US $500...
Do you mean this firm: http://www.pugetsystems.com/ ?


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 12:18:57 PM
No you unplug the freezer use it as the holding tank... see the retrofit for taps in the link...

Fish tanks? Wow... bit too small for a farm. Just get $100 water block from the other cooling thread if you are doing something that small.


Ideal would be these sorts of fiberglass planters I think.

http://www.net-profit-marketing.com/images/fibreglass-planter-box-landscape-planter-box-flower-pot-in-indonesia-21563638.jpg


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: turtle83 on July 04, 2013, 02:02:59 PM
How about using metal tank instead of plastic/fiber glass?

I would think thermal conductivity would help in such setup...


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 03:45:10 PM
How about using metal tank instead of plastic/fiber glass?

I would think thermal conductivity would help in such setup...

Interactivity of metal with oil? Not sure about metal containers. Cost as well might be prohibitive. Heat can get through fiberglass walled tanks... hot tubs are sprayed with insulating foam to save heating costs. So heat will be lost through the tank walls.

Found a perfect tank in baby blue going to check pricing next week.

http://i1219.photobucket.com/albums/dd421/toyodafrp/IMG00483-20101229-1340.jpg


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: 18RATTT on July 04, 2013, 05:30:22 PM
aha! yes baby blue tank... keep on talking boys.. we are getting there, ive been watching since page #1


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 04, 2013, 06:16:47 PM
http://hrale.en.alibaba.com/product/241026077-50345200/Heat_Exchanger_CB52.html

http://i01.i.aliimg.com/photo/v1/241026077/Heat_Exchanger_CB52.jpg

Wort Chillers anyone? Hmmm this might be worth exploring further.

http://www.youtube.com/watch?v=_fhG9QtSSrk


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 10, 2013, 07:26:09 AM
Going to source the following Friday:

OIL (Mineral Oil)
HEAT EXCHANGERS (Plate and Finned Rads)

http://www.metalindoengineering.com/pdf/Air%20Cooled%20HE/Plate%20Fin%20Coil.pdf

I will try and document the shopping trip as well as the build process over the next few weeks. Try to get some pictures up on Friday.

Note me and another member of the Big Picture Mining Cooperative are going to build an Oil Cooling Tank for his older GPUs he has been litecoin mining with and I will be testing my 3 - X6500 FPGA in the same tank as an analog for future miners like Klondikes 16s or Klondike K64 Blades or Bitfury boards. The X6500s run hot in my living room so it should be interesting to see the what happens.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 11, 2013, 07:50:52 AM
Crystal Plus 70T Mineral Oil

http://www.steoil.com/wp-content/uploads/2011/04/70T-MSDS.pdf

Anyone else got some MSDS data sheets on different potential mineral oils or transformer oils?


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: BenTuras on July 11, 2013, 08:43:35 AM
Crystal Plus 70T Mineral Oil

http://www.steoil.com/wp-content/uploads/2011/04/70T-MSDS.pdf

Anyone else got some MSDS data sheets on different potential mineral oils or transformer oils?
I found Shell Diala S2 ZU-I Dried NGT: http://www.shell.co.uk/gbr/products-services/solutions-for-businesses/business-lubricants-tpkg/products/product-range/diala.html

It's a pain to find any mineral oil in Holland. The Dutch translation leads to a pletora of beauty products, pfffft.
I did manage to find a shop for the Shell Diala, only available in 209L barrel at €600. Might be just enough for me, LOL!


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 11, 2013, 09:08:43 AM
Ya not any easier here in Indonesia Ben... I am interested what PRICE will be coming at me.


Pipes... I am thinking: PEX.


PEX (cross-linked polyethylene) has several advantages over copper:

PEX is cheaper than copper. Half-inch PEX tubing costs about a third the price of copper. Some of the savings will be offset by the need for a special tool to install the fittings, but if you’re doing a medium to large plumbing job, you’ll usually save by using PEX instead of copper.

PEX is faster to install than copper. If you use a manifold and “home-run” system (shown below), it’s like running a garden hose to each fixture—super fast and easy. But even if you install PEX in a conventional main line and branch system, the connections are quicker to make than soldering copper.

PEX won’t corrode like copper. If you live in an area with acidic water, copper can corrode over time. PEX is unaffected by acidic water and is therefore a better choice in these areas.

What about PEX vs. CPVC?

PEX and CPVC cost about the same. But there are a few reasons why PEX may be a better choice. First, PEX doesn’t require glue, which means you don’t have to work in well-ventilated spaces or wear a respirator. PEX is less likely than CPVC to burst if it freezes. Also, since PEX is more flexible and is available in long lengths, it can work better for “fishing” through walls in remodeling situations.

http://www.familyhandyman.com/plumbing/plumbing-with-pex-tubing/step-by-step

Different types of pex pipe; the cost, the tools required and applications of and advantages.

http://www.youtube.com/watch?feature=endscreen&v=G7Ya8Nv5g-M&NR=1


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 12, 2013, 10:11:39 AM
https://bitcointalk.org/index.php?topic=179769.msg2712907#msg2712907

Stealing from the water block ideas... muhahaha.

The MO-RA3 is a heat exchanger featuring excellent cooling performance and various possibilities of application. Its reliability allows for usage in workstation and server environments and it is sturdy enough for industrial applications. And, of course, it is powerful enough to cool even the hottest gaming PC. True to the slogan “there is no replacement for surface but more surface”, the MO-RA3 uses 36 m copper tubing and 3.0 square meter of aluminium fins for excellent heat transfer from cooling fluid to ambient air.

The cooling fluid passes the 72 tubes in a four-time parallel layout. This design and the flow-optimised connection block ensure a low flow-resistance, despite the large size of the heat-exchanger. The connection block improves flexibility as well by providing G1/4“ threads in three directions. The fins of the MO-RA3 have been optimised specifically for low-rpm fans. Their enormous surface and effective geometry guarantee excellent cooling performance.

Possible fan configurations

MO-RA3 420 LT

Nine 140 x 140 x 25 mm fans

MO-RA3 420 PRO

Eighteen 140 x 140 x 25 mm fans

http://shop.watercool.de/epages/WatercooleK.sf/en_GB/?ObjectPath=/Shops/WatercooleK/Products/25120

http://shop.watercool.de/WebRoot/Sage/Shops/WatercooleK/512C/DA3C/DF0E/4D35/5205/0A0C/05EA/2944/MO-RA3_420_Pro.jpg


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 14, 2013, 05:43:54 PM
Doing a little layout work...

http://i230.photobucket.com/albums/ee106/PFC4L1FE/tank2_zpsad594deb.png (http://s230.photobucket.com/user/PFC4L1FE/media/tank2_zpsad594deb.png.html)

http://i230.photobucket.com/albums/ee106/PFC4L1FE/tank_zps3f98c86b.png (http://s230.photobucket.com/user/PFC4L1FE/media/tank_zps3f98c86b.png.html)


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 14, 2013, 09:03:19 PM
I was thinking something like this but as a fiberglass unit.

http://img.directindustry.com/images_di/photo-g/4-wheel-waste-disposal-container-16816-2276585.jpg

Recycling all the rage... what about an old freezer? http://webpages.charter.net/redusc/Freezer.html Don't drill through the walls for piping.

http://bit.ly/YgNpWG


Almost all mineral oil distillates degrade HDPE, LDPE, acrylics, etc.   I would avoid plastic and stick to steel as its cheap, easy to work with, and structurally sound.

Also with regards to radiators/pumps.  Again, I'll say avoid PC cooling crap (emphasis on crap).  They all use poor gasket material (EDPM, which swells when it comes in contact with most oils), most have plastic impellers, and provide zero characterization data.   If you use it, you'll be guessing and waiting for failure.  Engineering post component selection rarely provides correct or efficient designs.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 14, 2013, 09:06:09 PM
Also, focus on getting the heat outdoors.  What is the point if you're still blowing it in the room to be dissipated by the air conditioning unit?   You should focus on a model similar to the portable air conditioner units that have window mount exhausts, only in this case, window mount radiators.   Even in extreme environments, like Arizona/New Mexico, you're still able to air exchange the heat and keep the oil at < 55C.


Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 14, 2013, 09:18:58 PM
Crystal Plus 70T Mineral Oil

http://www.steoil.com/wp-content/uploads/2011/04/70T-MSDS.pdf

Anyone else got some MSDS data sheets on different potential mineral oils or transformer oils?

That's only food grade mineral oil and lacks characterization with regards to flash point, PBA content, material interaction.  It would likely attack solder masks and bonders on and in PCB.

76 Transformer Oil - cheap, easy to find
http://www.conncoll.edu/offices/ehs/EnvhealthDocs/Transformer_Oil.pdf
DSI Opticool - hard to find, apparently good stuff
http://dsiventures.com/wp-content/uploads/2013/02/PDS-Opticool.pdf

With regards to volume, 55 gallons is about the smallest you can find.   Usually transformer oil is sold by the tanker (~10k gallons).  It requires 350 gallons to fill a 42U rack sized tank.

I've been meaning to experiment with EDM fluid, as it has similar requirements to transformer oil in its primary function and is available in 5 gallon pails.  However, this being said, its $25USD/gallon in pails.  The only reason I'm suggesting this unknown is it may give opportunity to experiment instead of making conjectures about infeasible things like plastic bathtubs, horse laxatives, and $10 chinese pumps that water cooled PC folks pay $90 for.  Thermodynamics and fluid dynamics are sciences not something you come up with an end all solution without calculation, experimentation, and process metrics and refinement.




Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: ultrix on July 14, 2013, 09:21:20 PM
maybe the pads might be the best route.
http://www.frozencpu.com/products/17467/thr-178/Fujipoly_Ultra_Extreme_System_Builder_Thermal_Pad_-_Full_Sheet_-_200_x_150_x_05_-_Thermal_Conductivity_170_WmK.html

The best you can buy, and should do ~400 chips each sheet, if you cut them small enough. Don't need a whole 1/4 sheet to cover the whole board, esp since these are wicked expensive.

Sure there are cheaper alternatives directly from China.

http://www.aliexpress.com/item/5Pcs-lot-30CMx3CMx0-3MM-Thermal-Pad-CPU-GPU-DDR-RAM-CHIP-3320-01-05/632016505.html

US $3.22 / lot
5 pieces / lot , $0.64 / piece

But... 17.0 W/mK vs 3.2W/mK ...

If i had 400 chips to use up 1 expensive sheet (and all the time in the world), id put the expensive one just under the vias, use the cheap sheet (with holes) for rest of the board.

Those sheets will degrade in oil.  More correctly they will swell and then dissolve.  As will thermal grease.  You want indium foil.  Its expensive, but you don't need much per IC.  Alternatively since the heat per area of the IC in this application is low, why not just lap a copper shim?




Title: Re: DIY Klondikes: Oil Submerged K16 / K64 Blade Configurations Discussion Thread
Post by: Bicknellski on July 15, 2013, 06:00:19 AM
Also, focus on getting the heat outdoors.  What is the point if you're still blowing it in the room to be dissipated by the air conditioning unit?   You should focus on a model similar to the portable air conditioner units that have window mount exhausts, only in this case, window mount radiators.   Even in extreme environments, like Arizona/New Mexico, you're still able to air exchange the heat and keep the oil at < 55C.


That's an opaque wall in the diagram, I guess, is hard to see but I put it between the tank and heat exchanger obviously we would put the heat exchanger outside. Not much room on tinkercad workspace. We will source as you have instructed and update the list. In terms of getting huge orders of oil don't think that will happen not as easy here in Indonesia to source things like this so we will do what we can. Worry about getting bamboozled on the oil. I will try and get a tank that does less than 350 gallons but will need to see on prices. I don't have the resources and need to have something smaller that I can manage.

Also what about fiberglass tanks they should be ok would they not?
What about PEX pipes as they are much easier to fit and manipulate are they prone to degrading with this oil?
Oil or Hydraulic Pumps?

Notes:

Indium under the heatsinks or a copper shim.
Heat exchanger all metal properly engineered.
Transformer oil as specified.
Steel tanks (Fiberglass?).
Fittings and pipes metal (PEX?).

Do proper calculations before parts purchased.

Quote
"Thermodynamics and fluid dynamics are sciences not something you come up with an end all solution without calculation, experimentation, and process metrics and refinement."