|
turtle83
|
 |
July 02, 2013, 10:09:28 PM |
|
But... 17.0 W/mK vs 3.2W/mK ... If i had 400 chips to use up 1 expensive sheet (and all the time in the world), id put the expensive one just under the vias, use the cheap sheet (with holes) for rest of the board. |
|
|
|
PaperClip
Newbie
 Offline
Activity: 56
Merit: 0
|
|
July 02, 2013, 11:03:13 PM |
|
I also had idea to submerge miner into oil, but in the end decided to avoid this. A/C will be enough for one device like in my case, and its not so safe to place tank full of hot oil in living room. Anyway, you may find this advice useful: Have anyone tried to completely submerge Avalon into mineral oil? That works very well for desktop computers.
Fluid would be too viscous for the fanz...  I've got a system that's submerged in mineral oil (quad E5-4650). Transformer oil to be specific (lower viscosity, higher flash temps, mostly synthetic, no PBAs). Fans don't really care, although that's not the primary circulation method (pump). |
|
|
|
|
Bluestreak66
Member
Offline
Activity: 70
Merit: 10
|
|
July 02, 2013, 11:12:48 PM |
|
I've seen this done with pcs and just a heads up If at any point one of those asics dies and the board needs reworked to replace one you might as well toss the board and start over. That oil gets under the chips and will burn when heated with hotair from a rework. Once you do get it off if you get the board "cleaned" off the solder sometime won't stick properly and thats not really something you want to chance on a QFN package. Again I've built computers for a long time and had buddys do this. It seems cool at first but in the end the effort is not worth the end result. What you end up with is a large tank filled with hot oil that you have to work around and inevitably gets all over everything. Not to mention you still have to cool the oil somehow. I think the ideal solution would be to sandwich a waterblock between two klondikes or bitburners. That would make it cost effective because you have one heatsinking device per two boards and you can use off the shelf pc water pumps controllers and radiators.
Any published data on this in terms of QFN chips having been submerged in oil and they have had failures or issues as you describe? Just want to see what has been written up about this. Considering that this sort of cooling in done in a fairly wide application commercially I have yet to see anything specifically about issues with QFN chips immersed in oil. Although the scenario you paint will possibly happen to someone I am sure what would be the likelihood of having to do that? It is a risk that is taking place daily but again I have not found anything yet on this being reported as widespread issue. They are dropping OEM 1U rack mount servers directly in this without any concerns about QFN or other types of chips. Seems strange can't find anything when I search about it. Also what I am looking to do is scale this up for a data center applications for mining. So water blocks would at some point be more problematic however for a smaller number of boards I'd agree go with air or water blocks. No one is going to overclock a server in a cooperate environment, with servers up time and longevity is key. When you have that much hardware in such a small space cooling becomes a huge issue thats where the oil cooling comes in. Also with that type of equipment once it breaks, its not repaired at the component level. There is not a specific issue with submerging QFN in oil or any other type of package for that matter. My point was this: Inevitably some chips with die from manufacture defects, power fluctuation, or hundreds of other unforeseen possibilities, cooling is just one of those. Having a 16 chip per board, if one chip dies then chances are you will lose that board because it will be too difficult to repair after being in oil. Ask any repair tech how easy it would be to repair an oil soaked board. I repair arcade equipment at work and can tell you it would not be easy with oil soaked boards. Convection will draw the heat to the top of the tank pump that through a oil cooler to the bottom to aid in circulation. Straight mineral oil work good and can be purchased cheap by the gallon that's what we used. Transformer oil works well as it is designed to cool transformer windings. |
|
|
|
|
|
joeventura
|
|
July 03, 2013, 01:11:01 AM |
|
Assuming this is your plan for cooling overclocked (350-400M) Avalons
Oil cooling for stock 4.5-5Gh Klondikes is incredible overkill.
A decent heatsink and a fan is plenty
|
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 04:02:55 AM
Last edit: July 03, 2013, 04:15:59 AM by Bicknellski |
|
Assuming this is your plan for cooling overclocked (350-400M) Avalons
Oil cooling for stock 4.5-5Gh Klondikes is incredible overkill.
A decent heatsink and a fan is plenty
It isn't overkill when you have 10's or 100's of K16/K64/K256 units. I put the caveats in the first post and my reply early on. You really do NOT need to do this in terms of a single K16 unless you are doing it to learn more about the process. My intention is to have as I said before a unit that can handle some thing like 5U to 10U as a prototype module that can then scale up by adding more modules. In terms of failures... with a larger array single chip failures are not going to matter much as at this scale you probably can easily remove a K16 board from the array and keep plugging along with no problem. Cleaning up a K16 is not going to be that hard given 100mm x 100mm size and placing a new Avalon chip on the board as will be reasonably simple to do. Infact given I have access to fabrication of boards I will probably have a few dry boards straight from the fabrication unopened. Replacement I don't think is a huge issue with the modular capabilities of the Klondike. K256 X 10 = 2560 chips roughly 1,024,000 Mh/s if you can attain 400+ Mh/s per chip mineral oil cooled. K256 X 10 = 2560 chips roughly 896,000 Mh/s if you can attain 350 per chip air cooled with a/c. There is definitely potential in this. So continuing on... anyone found problems with the silicone heat sink pads and mineral oil? I think next I want to mock up is a 5U capacity tank, heat exchanger / cooling tower. |
|
|
|
|
ultrix
|
|
July 03, 2013, 04:51:45 AM |
|
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics. Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil. Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil. Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried. These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in. As far as heat exchangers, there are two routes I've used with great success: 1) Liquid to air - Simple radiator style setup. This however should _not_ be cheaped on. Initially I tried a PC water cooling radiator (leftover from a friend's system). PC radiators in general are crap. They don't provide heat curves, operating tolerances, etc. In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers. This type of system is the best performance per price for < 35k BTU/hr (about 10kw). Also, I would avoid using polymer tubing for any of the hoses. I use silver solder welded copper pipe aka sweated copper. 2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel. You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer. I sent the hot oil into one of the chambers, cool water into the other. For cooling the water, you can use an evaporative system, and ideally one would. You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature. This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw. I wouldn't cheap on the radiators, pipes, and especially pumps. Avoid pond pumps and low duty cycle pumps. Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside. If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block. I've got a bunch of data on this at home, but am on the road right now. |
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 04:54:52 AM |
|
http://www.ehow.com/how_7640669_circuit-board-fabrication-cleaning.htmlCleaning the Substrate1 Put on a pair of latex or rubber gloves. One of the worst contaminants in the production of PCBs is oil, which comes from your hands and keeps the surface from accepting resist evenly. 2 Scrub each board with a plastic scrub pad to remove any oxide coating on the copper side of the board. You can acquire a plastic scrub pad at your local grocery store where dish washing products are sold. Do not use water when performing the scrubbing as this will promote a new oxide coating on the board during the process. Handle the board by the edges. The less contact with the copper cladding the less chance of fouling the surface. Many manufacturers use acid to remove the oxide coating, but physical scrubbing works just as well. When you have finished scrubbing it, the surface should appear shiny with no spots. 3 Mix a small amount of metal free soap in deionized water and place in a container large enough to hold the PCBs. 4 Rinse the board in the mild soap solution. This will remove most remaining oil and inorganic contaminants from the surface of the board. 5 Wash the board in an isopropyl alcohol bath to remove any other organic residues remaining. 6 Rinse the cleaned board with acetone. Isopropyl alcohol contains approximately 30% water and the acetone will remove any alcohol or water left adhering to the board. The acetone evaporates quickly and doesn't leave a residue. Cleaning the Finished Board7 Put on a pair of latex or rubber gloves to protect the board from the oil present on your skin. 8 Dip a cotton tipped swab into isopropyl alcohol and carefully scrub each solder connection on the PCB. Scrub both sides of the board where solder connections were made. 9 Dip a new cotton tipped swab in deionized water and repeat the process of cleaning each solder connection on both sides of the board. Depending on the type of flux used in the soldering of components on the boards, any residue will be soluble in either isopropyl alcohol or deionized water. 10 Rinse the cleaned board under cold deionized water. This will remove any remaining soap or isopropyl alcohol. 11 Place in an oven at 100 degrees C to dry for approximately five minutes. The heat will dry any remaining water and leave the finished product ready for testing. Tips & Warnings Keep your fingers off the surface of the board. Small traces of skin oil can short the board or components on the board.
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 04:57:20 AM |
|
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics. Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil. Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil. Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried. These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in. As far as heat exchangers, there are two routes I've used with great success: 1) Liquid to air - Simple radiator style setup. This however should _not_ be cheaped on. Initially I tried a PC water cooling radiator (leftover from a friend's system). PC radiators in general are crap. They don't provide heat curves, operating tolerances, etc. In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers. This type of system is the best performance per price for < 35k BTU/hr (about 10kw). Also, I would avoid using polymer tubing for any of the hoses. I use silver solder welded copper pipe aka sweated copper. 2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel. You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer. I sent the hot oil into one of the chambers, cool water into the other. For cooling the water, you can use an evaporative system, and ideally one would. You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature. This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw. I wouldn't cheap on the radiators, pipes, and especially pumps. Avoid pond pumps and low duty cycle pumps. Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside. If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block. I've got a bunch of data on this at home, but am on the road right now. +1 Sweet information there... I will compare the cost of transformer oil to the straight industrial mineral oil costs here in Jakarta. Great post. Have you had any issues with repair and degreasing boards then having to reflow etc Ultrix? Also in terms of thermal compound and pads what do you recommend given the solubility of some of the heat sink compounds as mentioned above. |
|
|
|
Bluestreak66
Member
Offline
Activity: 70
Merit: 10
|
|
July 03, 2013, 05:11:56 AM |
|
K256 X 10 = 2560 chips roughly 1,024,000 Mh/s if you can attain 400+ Mh/s per chip mineral oil cooled.
K256 X 10 = 2560 chips roughly 896,000 Mh/s if you can attain 350 per chip air cooled with a/c.
There is definitely potential in this.
So continuing on... anyone found problems with the silicone heat sink pads and mineral oil?
I think next I want to mock up is a 5U capacity tank, heat exchanger / cooling tower.
You should have no problem with silicone compatibility with mineral oil. How many boards do you plan on putting in a 5u? K256 x 10 would be 160 boards Is this your plan? |
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 05:15:52 AM |
|
No it would be 10 X 1U units of K256.
Have a roughed out design for blades that are 4 x K64s daisy chained.
The prototype will be 2 x 1U (K256) total 512 chips in a blade configuration or 1U server pan then suspended in a tank that can support 5U to 10U rack.
|
|
|
|
|
ultrix
|
|
July 03, 2013, 05:21:10 AM |
|
As previously quoted in this thread, I'm using transformer oil on a quad E5-4650 xeon system and some other electronics. Firstly, I've used various transformer oils, the easiest to acquire with best performance I've found is 76 transformer oil. Also with regards to lexicon, I prefer this term because it refers to a quality and composition of mineral oil. Using this term avoids confusion with the laxative grade (literally) mineral oil that some have tried. These contain sulfides, PBAs, and other chemicals that you would not want to run electronics in. As far as heat exchangers, there are two routes I've used with great success: 1) Liquid to air - Simple radiator style setup. This however should _not_ be cheaped on. Initially I tried a PC water cooling radiator (leftover from a friend's system). PC radiators in general are crap. They don't provide heat curves, operating tolerances, etc. In the end I bought a radiator from Brazetek, http://www.brazetek.com/finned-coil-water-air-heat-exchangers. This type of system is the best performance per price for < 35k BTU/hr (about 10kw). Also, I would avoid using polymer tubing for any of the hoses. I use silver solder welded copper pipe aka sweated copper. 2) Liquid to liquid - For this I milled a simple dual chamber, mutli-plate system from stainless steel. You can buy them from brazetek, but I had a ton of 1/4" 316 stainless stock and a case of beer. I sent the hot oil into one of the chambers, cool water into the other. For cooling the water, you can use an evaporative system, and ideally one would. You can get 5+ ton evaporative towers with pumps and controls locally for $3k+ pretty much anywhere in the US and probably the world. For my system, which was a prototype, I used a compressor to cool the water to various temperatures, some below the wet bulb temperature. This system can basically scale infinitely and reaches a cost/performance advantage over the previous system at around 10-12kw. I wouldn't cheap on the radiators, pipes, and especially pumps. Avoid pond pumps and low duty cycle pumps. Oh, and flush your system before use especially if chinese rads are used as they tend to have finishing bits inside. If you'd like to experiment with this buy a gallon or two of transformer oil at a HAM shop for ridiculously high prices and construct a test load with some kanthal wire cast into a ceramic block. I've got a bunch of data on this at home, but am on the road right now. +1 Sweet information there... I will compare the cost of transformer oil to the straight industrial mineral oil costs here in Jakarta. Great post. Have you had any issues with repair and degreasing boards then having to reflow etc Ultrix? Also in terms of thermal compound and pads what do you recommend given the solubility of some of the heat sink compounds as mentioned above. No way anything will reflow. You want to use thermocouples to monitor temperature at various locations in your tank and a control system that will kill power to the heat source if a critical temperature is reached. With regards to system maintenance, I use a process similar to one you described but with a predecessor step of letting the parts hang above a catch pan after heating the parts arbitrarily with a heat gun (or blow dryer). All and all, its less mess than dealing with dust caked components. Edited: My whole process is this: 1) hang component above catchpan and heat as described 2) after 20 minutes or so, dunk in a pan of rubbing alcohol and agitate. 3) let hang for another 10 Again, the first step is to avoid going through tons of paper towels and alcohol. I've done PCB rework including reflow on parts cleaned by this method. There is no observable/measurable difference. |
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 05:25:50 AM
Last edit: July 03, 2013, 06:23:56 AM by Bicknellski |
|
You mean you can't remove parts (de-solder) and then re-flow a PCB after being in oil? In case you want to replace a blown QFN chip. Thanks! Nice info on the heat gun and drip tray thanks. Be picking your brain more and more. Need to have a catch tray and rack will think about some sort of caddying on wheels for this. What about palm or coconut oil? We have bulk amount here in Indonesia wondering if that would be possible? Confirmed and Tested. http://www.elect.mrt.ac.lk/Coconut_oil_eru_2001.pdfConsidering all the above facts, we can conclude that coconut oil can be
used as transformer oil. However further tests need to be done under actual
operating conditions. Coconut oil is freely available in Sri Lanka and it may
become cheaper than other transformer oil. Most importantly it is 100%
environmental friendly, hence it can be easily used as an alternative to
traditional transformer oil. |
|
|
|
|
ultrix
|
|
July 03, 2013, 05:33:23 AM |
|
No it would be 10 X 1U units of K256.
Have a roughed out design for blades that are 4 x K64s daisy chained.
The prototype will be 2 x 1U (K256) total 512 chips in a blade configuration or 1U server pan then suspended in a tank that can support 5U to 10U rack.
Minor recommendation, lay the rack on its back. In fact avoid using off the shelf racks. Just weld together the structure with some 1" to 1.5", 1/4" thick angle for a rack and frame. Then weld sheets on for sides of the tank and you're good to go. The reason I say avoid off the shelf racks is that most are only structurally designed to be run vertically. Further horizontal affords servicing of individual machines without the need to drain the tank. You can assemble a pretty nice 41U tank for ~$800 USD. This will require ~350 gallons of transformer oil, which will run you ~$4k delivered. I would test the tank with water prior to filling as water is less viscous and namely cheaper. So for each rack you're looking at about $5k + (cost of tower / number of racks). When you compare that with the costs of AC, especially in high power density applications, the initial cost is about 1.5x higher, however the fact you can cool 250kw with 10kw of cooling equipment is unbeatable. In an air environment you're looking at power costs approximately equal to that of the equipment you're running. |
|
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 05:56:58 AM |
|
Fabrication costs here are low very very low so I am sure I can have something welded up to any specifications I want.
Ya looking at that Green Revolution Carnojet system the tank actually has cabling and racking solution built into the tank. I want to do a smaller build as I will have this inside on the 3rd floor of my school (empty 3rd floor) and I definitely do not want to be running A/C all day and night to cool even a modest number of units for now but want to be able to scale up from 5U or 10U simply adding the tank and racking that can be modular like a data center racking system but easily handled by kids. I think this project will be added to our curriculum as well for second semester so many interesting aspects to be covered and it lines up perfectly with so many aspect in the science curriculum it is scary!
Thanks again. I will put up a rough design soon as I think given the posts you made I am pretty confident I can acquire pretty much everything I need in very little time and at very reasonable costs.
|
|
|
|
|
turtle83
|
|
July 03, 2013, 07:54:10 AM |
|
You mean you can't remove parts (de-solder) and then re-flow a PCB after being in oil? In case you want to replace a blown QFN chip. Thanks! Nice info on the heat gun and drip tray thanks. Be picking your brain more and more. Need to have a catch tray and rack will think about some sort of caddying on wheels for this. What about palm or coconut oil? We have bulk amount here in Indonesia wondering if that would be possible? Confirmed and Tested. http://www.elect.mrt.ac.lk/Coconut_oil_eru_2001.pdfConsidering all the above facts, we can conclude that coconut oil can be
used as transformer oil. However further tests need to be done under actual
operating conditions. Coconut oil is freely available in Sri Lanka and it may
become cheaper than other transformer oil. Most importantly it is 100%
environmental friendly, hence it can be easily used as an alternative to
traditional transformer oil. But be prepared for the smell  |
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 08:01:01 AM |
|
Distilled at 200C smell removed... according to the article. Coconut oil contains odor and tasting compounds, which are actually
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.
To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay. |
|
|
|
|
turtle83
|
|
July 03, 2013, 08:12:23 AM |
|
Distilled at 200C smell removed... according to the article. Coconut oil contains odor and tasting compounds, which are actually
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.
To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay. Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia... |
|
|
|
|
Bicknellski (OP)
|
|
July 03, 2013, 08:16:52 AM |
|
Distilled at 200C smell removed... according to the article. Coconut oil contains odor and tasting compounds, which are actually
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.
To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay. Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia... Reading further can be used in cold climate as well... as they can reduce the pour temp significantly... but I won't be concerned about that here in Jakarta. Although a purified Coconut Oil will be best. Not sure on cost difference between that transformer oil and mineral oil. Will start checking. |
|
|
|
|
ultrix
|
|
July 03, 2013, 08:44:50 AM |
|
Distilled at 200C smell removed... according to the article. Coconut oil contains odor and tasting compounds, which are actually
low boiling compounds. Inclusion of such compounds with low boiling point
builds a pressure in the transformer at operation where the temperature rises
normally above 60°C. Deodorization removes odor and taste of coconut oil and
is done by steam distillation at 200°C.
To remove saponificable matter and colorings in coconut oil, a
bleaching process was carried out by using activated clay. Sounds like a good solution then... provided you can distill it properly... and limit usage to tropical climate... if the ambient temp is 15C or so, and your rig is shut down for a day or so.. the coconut oil will start greasing up inside the radiators, tubes, etc... Shouldnt be an issue for indonesia... Reading further can be used in cold climate as well... as they can reduce the pour temp significantly... but I won't be concerned about that here in Jakarta. Although a purified Coconut Oil will be best. Not sure on cost difference between that transformer oil and mineral oil. Will start checking. I'd stick to synthetic transformer oil, little to no degrading under the heat computers generate. Although the carnojet system claims to use a "proprietary" coolant, I believe what they use is opticool made by DSI ventures. I've been meaning to get in touch with them as they're close, however I have a ready supply of the 76 transformer oil here. |
|
|
|
|
|
Bicknellski (OP)
|
|
July 04, 2013, 03:26:53 AM |
|
I think the carnojet DOES NOT use proprietary oils... infact all it is unscented mineral oil from what I get off their website.
Looking at tanks have to be careful of the plastic in the tank due to potential temperature spikes.
|
|
|
|
|
