Power heat and noise?  Guess you have never seen or heard 48 GPU cores running.   The power and noise for 1/1000th of the network is significantly lower today than under the GPU era.  The peak of GPU mining was probably ~20 TH/s.   GPU rigs requires about 500 J/GH, so 20 Th/s is 10,000 KW or 10 MW of power.  Today the network is ~2,500 TH/s but the average ASIC is ~1 J/GH making total power usage ~ 5 MW.   Of course the network is going higher but GPU weren't any less power hungry.  The network will always reach equilibrium where the power consumed is roughly equal to the value of BTC produced with a small margin for capital cost, risk, and profit.


Also there never were millions of miners.   Never.  Not once, not a single day in the history of Bitcoin.  At the peak there was something like 20 TH/s of GPU miners.   The output per card varies but lets take a middle of the road card the 6870 which is about 300 MH/s.  20 TH/s is ~65,000 HD 6870 graphics cards.   That puts the number of miners in the tens of thousands max.

That doesn't look good.  If this is right after restart then likely one of the 4 power modules is defective/damaged or poorly soldered.  With only 3 modules or <100W available it is unlikely you are going to get anything good from this module.  Maybe KNC can just send you a spare module first and you send the defective back second.

Nothing?

As a result of the VRMs being cut from 8 to VRM the modules are running close to max DC power.  Each of the power modules is good for 40A.  At 0.75V that is 30W per module or 120W per board.   Some of the firmware have the chips running at higher 0.9V that is 36W per module or 144W per board.  Short of overvolting there isn't much headroom due to the limits of the power system.  Hooking up 1200W PSU isn't going to do much if the DC to DC modules can't handle more than 600W.

Reserved - Prototype details

Reserved - Advantages & Challenges

Major Systems
1) Overview (coming soon)
2) Cooling Tank
3) Connectivity (coming soon)
4) Condenser (coming soon)
5) Chilled Water Supply (coming soon)
6) Safety Systems (coming soon)
7) Power Supply (coming soon)

2) Cooling tank

Material considerations.  Fluorinert is a highly inert fluid, it is also a poor solvent with most materials, and it is hydrophobic.  These characteristics make it an ideal working fluid.  However Fluorinert is an effective solvent for fluorinated compounds (materials containing fluorine),  plasticizers, and some additives to soft "plastics".  

This presents two challenges (and will be covered more in challenges section).  The first it that Fluorinert can replace by volume a portion of the compatible material in gaskets, seals, and orings.   For example when silicon adhesive is immersed in FC-72 up to 7% of the volume is replaced with Fluorinert.  This can lead to failure of seals in sealants, gaskets and o-rings.  Even when they use the same underlying compound "hard" materials are less prone to replacement than "soft" materials (i.e. rigid PVC pipe vs PVC wire insulation).

Material compatibility analysis by CERN
http://detector-cooling.web.cern.ch/detector-cooling/data/3M_FAQ_Fluorinert.pdf
http://detector-cooling.web.cern.ch/detector-cooling/data/Fluoro_Compatibility.htm

It is not possible in one post to do an exhaustive analysis of potential material risks so if you are interesting in working with immersion cooling I strongly recommend reviewing both of the links above and conducting your own research.  DISCLAIMER:  All of this is experimental.  Damage to equipment can occur. If you are unwilling to do self research and testing this type of project may not be realistic.   The challenges of material compatibility and handling dissolved contaminants will be covered in the challenges section.


Material choices for tank
Glass aquarium can be used for a prototype but due to the use of silicon adhesive below the fluid line I would caution against its use long term.  Stainless steel tank is a good option for the ability to design and construct a custom sized tank.  To reduce heat conduction a double tank design could be constructed with an inner stainless steel tank surrounded by insulation, surrounded by an outer tank.  To reduce costs the outer tank could be constructed from a cheaper material than stainless steel.  

For my initial prototype I will be used a polycarbonate NEMA 4 enclosure with the opening facing upward.  It would be best in any inlet be through the sidewall of the tank not the lid however in my prototype I will be cutting through the lid as any mistake can be fixed by replacing the lid instead of the more expensive body.  Some NEMA 4 enclosures come pre tapped with inlets that are gasketed.  That should be avoided for the reasons above instead look for a solid body design.




Design of the cooling tank.  
The tank should be designed to have no penetrations below the fluid line.  The fluid depth should be sufficient to cover all components plus a safety margin of 30mm.  In normal operation the fluid in the tank should be relatively constant (<5mm change in depth) however a failure of one or more cooling components will result in the condenser temperature rising and the condenser will be unable to condense (cool) the working fluid as fast as it is being boiled off and the fluid will fall.  The safety margin ensures that there is a delay between any failure and the components boiling off their fluid and likely being damaged.  To protect the safety margin a depth switch connected to a relay can be used to cut off power to the tank if the fluid level drops.  When planning the tank there should be sufficient space in the tank to contain the boiled Fluorinert gas.  As a starting point you should plan for no less volume than the volume of the fluid.  In addition you should consider the height of the heat exchanger.  


The components to be cooled are submerged in the working fluid, if possible the tank size should be optimized to minimize the amount of working fluid necessary to submerge the components.   Due to the high cost of Fluorinert (~$80 per Liter) a design which achieves the minimum amount of fluid to transfer the heat will be more economical.  Remember Fluorinert can handle a significant heat load so you can use it sparingly.  3M has shown effective heat transfer with as little as 1L for 4KW of heat load while it is unlikely that you will acheive that level of energy density it shows that excess fluid isn't necessary.

One design consideration is what components will be included in the tank.  An SHA-2 hashing system consists of three major components.  One or more processors boards, a host/controller, and one or more ATX power supplies.  It is possible to use immersion cooling to cool just processor boards or the entire system.  There are advantages and disadvantages to both.

Lets consider an example 4TH/s system using HashFast ASIC boards
Processing Boards - 2780W (87%) [1]
Host - 100W (3%) [2]
Power Supply (10%) [3]
Total: 3200W

[1] 10 boards @ 400 GH/s nominal, 278W per board
[2] Low power PC, using a embedded computer like Beagle Board would reduce wattage further
[3] The wattage is the "lost" power converted to heat.   Multiple 90% efficient ATX style 80-Gold or 80-Platinum PSU.  Power supply could possibly be in n+1 fault tolerant configuration.   In: 3200W AC Out: 2880W DC + 320W heat.

Immersion cooling only processing boards
---------------------------------------
Pros:  
 Highest energy density.  
 Reduced tank size.
 Most efficient use of Fluorinert (highest W/L)

Cons:
 Complicates power delivery.  Either custom high current cables are needed or large number of cables need to pass through the tank bulkhead.  
 Still need to air cool other components.  
 Potentially prevents deployment to areas where temp outside the tank is ill suited for air cooling (i.e. non-air conditioned warehouse).
 Not silent due to PSU fan noise.
 If cooling water line bursts the components outside the tank (as well as operators) are vulnerable to being damaged by short circuit conditions.

Immersion cooling all components
---------------------------------------
Pros:
  Simplified tank connection (can be reduced to: data cable, power cable, water in, and water out lines).
  Power supplies may need to be modified.  Fans should be removed and power supply fan monitoring bypassed (if present).
  Near silent operation (if cooling water heat transfer is in another location).

Cons:
  Larger tank (more significant than may initially appear.  remove heatsinks and fans from a hashing board and the power supply has almost the same volume as the boards it will power).
  Less efficient use of Fluorinert (lower W/L).
  Internal wire management can be more difficult if tank is cramped.
  Replacement of failed host or power supply is more complicated.
 
Depending on the boards used a hybrid between the two options would be to use immersion cooling for the processors and power supplies and place the host/controller outside the tank.  For larger operations this may be desirable as a "hot swap" replacement of defective host could be performed easily.  This may not be possible with all SHA-2 processors but in at least one case the HashFast boards are connected by USB to any host capable of running cgminer and linux.  So data connection to tank could be a single usb port and the host host system be as far from the tank as allowed by USB spec or even one host controlling multiple tanks.  The USB cable length can be extended using cheap Cat5/6 cable with an active converter.

I figured I would start this thread as a place to collaborate on immersion cooling.  The type of immersion cooling I am interested in is two phase passive semi-open bath.  Wow that is a mouthful.  So lets start with the basics.

What is immersion cooling?
Immersion cooling is a method of heat transfer where the device to be cooled (namely SHA-256 processor) is immersed in a heat transfer fluid (also called working fluid).  It is helpful to remember that "fluid" doesn't necessarily mean liquid, "Freon" is a form of working fluid although most people think of it as a gas.  Technically even air is a working fluid.  Heat sinks transfer heat to passing air and fans are pumps designed to move low density gas instead of higher density liquids.  Most people don't refer to air cooling as immersion cooling but it may help to understand some of the concepts if you think of Freon, water, and air as working fluids in heat transfer.

Is immersion cooling the same thing as "water cooling"?  
With immersion cooling the heat is transferred directly from the heat source to the working fluid.  In "watercooling" the working fluid is potentially harmful to electronics and thus flows through a sealed loop isolated from the heat source.   A watertight waterblock is used to indirectly transfer the heat from the heat source to the working fluid.  With immersion cooling the working fluid must be non-conductive and that generally limits us to three families of fluids: deionized water, mineral oil, and fluorocarbon-based fluids (namely Fluorinert made by 3M).  Immersion cooling systems have higher fluid cost than watercooling but this is partially offset by the elimination of individual waterblocks.   

What is Two Phase Cooling?
Well lets start with what is one phase cooling.  In a mineral oil or deionized water system the working fluid never boil or freezes and always remains in a single liquid phase.  Cool fluid is pumped past the heat source, where thermal energy is transfered to the fluid by conduction which raises the temperature of the fluid.  The heated fluid is then pumped to a heat exchanger where it is cooled and pumped back to the heat source.  The heat transfer is known as "sensible heat", and the more heat (thermal energy) transferred into the working fluid the more its temperature rises.  The rise in temperature can be controlled by the fluid flow rate.  The faster the fluid flow rate the less energy will be transferred into each unit of fluid and the lower the temperature rise will be.

In two phase cooling the working fluid boils and thus exists in both a liquid and gas phase.  The system takes advantage of a concept known as "latent heat" which is the heat (thermal energy) required to change the phase of a fluid.  The working fluid is only cooled by boiling and thus remains at the boiling point ("saturation temperature").  Energy is transferred from the heat source into the working fluid will cause a portion of it to boil off into a gas.   The gas rises above the fluid pool where it contacts a condenser which is cooler than the saturation temperature.  This causes the fluid to condense back into a liquid and fall (rain) back into the pool. 

What is "open bath" or "semi open bath" means?
Open Bath (sometimes more correctly called semi open bath) means the tank containing the heat source, working fluid and condenser is not a pressure vessel.  The pressure inside the tank/container will be roughly the same pressure as the outside air (<1 PSI difference).  This makes the tank easier and cheaper to construct and the system is never pressurized at dangerous pressures.  In a open bath system equilibrium is achieved by having a condenser capable of sufficient heat transfer to condense the vapor at the same rate it is being produced by boiling.   The volume of vapor remains relatively constant and prevents a rise in pressure.  If the fluid boils at a faster rate than the condenser can condense it back into a liquid then the pressure in the tank will rise and vapor will be lost.  This can be prevented by a safety pressure switch.

Why not used deionized water (DI)?
DI is electrically non conductive but it isn't inert.  It will rapidly pull ions from the surrounding material until it reaches a point where it becomes conductive and damages the cooled components.  Proper immersion cooling with DI requires expensive ion exchange systems and replacement ion exchange resin to continually remove build up of ions in the water.  This process needs to operate continually which increases the cost and the system is never stable.   Without maintenance and continual inspection the system can allow a build up of ions that eventually destroys the equipment being cooled.  While immersion cooling with DI is a viable method it is ill-suited for unmonitored environments.

Why not use mineral oil?
Mineral oil is a single phase cooling system and for any significant heat load will require actively circulating the oil across the processors.   Mineral oil also has a high viscosity and will need powerful pumps capable of handling the higher pressure.   The system needs to be carefully designed to ensure each processor receives sufficient flow.  Mineral oil has a relatively low heat capacity and thermal conductivity which increases the required flowrates and necessitates the use of larger heat exchangers. 

What types of power densities are possible?
3M has conducted experiments cooling 4KW heat loads using 1L of working fluid so in theory heat densities approaching 4,000 W/L are possible.  The constraint on commercialization is that existing servers have relatively low energy densities (well low relative to the limits of immersion cooling).  Even a high end 3U server (4 CPU, multiple GPUs, 4+ 1200W PSU) may only have an energy density of 100W per Liter.    However SHA-256 ASICs have very high energy densities although current systems have server like energy densities due to the limits of air or water cooling.  Take a look inside the case of any 2nd gen ASIC design what takes up the most space?  Air.  The actual ASIC boards are very energy dense however there are surrounded by a significant amount of empty space.   Remember these are using boards designed for air/water cooling.  It may be possible to improve energy density by making custom compact boards.

As an example of what is possible KNCs ASIC boards are 225 cm2 of surface area and use ~120W.  If boards were stacked 1cm apart that would be an energy density of >500 W/L.  Hashfast pcb design (subject to change) is even more energy dense, 280W of power in 240 cm2.  In spaced in a cooling pool 1cm apart that would be an energy density of ~1,200 W/L*.   Another way to look at it is hashing density.   At 1cm spacing KNC would be a hashing density of >500 GH/L and Hashfast would be > 1500 GH/L.    

* There is limited information available on height clearance of HF boards.  If the large FET heatsinks are not removable it may limit the spacing of boards in a cooling pool.

How is the condenser cooled?
The condenser could be cooled using refrigerant directly but a water loop (glycol/water mix) may be a more flexible and economical solution.  The water loop can either be cooled using a commercial water chiller or using a dry tower in ambient air (heat exchanger and fans outside).  The cooling loop will need to be sized large enough to transfer the full heat load out of the water loop.  The advantage of FC-72 over other two-phase working fluids is it has a 56C boiling point which allows reasonable efficiency when cooling using outside air even during summer temps for most parts of the country. The larger difference between the input air (ambient outside air temp) and the input water temp (~50C to 56C) means more efficient heat transfer is (smaller fans, smaller heat exchanger).

What working fluids are available?
3M makes a large number of synthetic working fluids however for two phase cooling we are interested in fluids which boil below the temperature limits of ASICs.   Remember in two phase cooling the fluid will remain at saturation temp (boiling point) excess thermal energy is removed from boiling however the temperature of the fluid will remain in equilibrium at saturation temp.  It is not possible with two phase cooling to have temperatures less than the boiling point of the working fluid.

3M datasheet on all heat transfer fluids: http://multimedia.3m.com/mws/mediawebserver?mwsId=tttttviZIdW5_y7VPZA_qZ0t2XV62EW9iXut2Xut2tttttt--&fn=bro_heattrans.pdf

For this reason the following fluids are most applicable:
Novec 7000 - Boiling Point 34C
FC-3284 - Boiling Point 50C
FC-72  - Boiling Point 56C
Novec 7100 - Boiling Point 61C
FC-770  - Boiling Point 95C (likely too high outside of custom design)

The cost depends on supplier and volume being purchased but generally runs $80 to $100 per Liter.  Fluorinert is often priced by the kilogram and has a high density (1.6 kg per Liter) so take that into consideration if you think you found a "deal".

Agreed.  I think <10 PH/s one just needs to consider the most likely scenario is the same rapid growth we have been seeing.   If there is a "slow" week it is likely just timing (i.e if HF is delayed a week but then dumps 500 PH/s the next week then difficulty changes will be "lumpy").   10 PH/s is the absolute earliest we will see the curve bend and 20 PH/s is more likely.  If one is pessimistic then change that to 20 PH/s and 30 PH/s.

However at some point it will get "interesting".  The economics at 30 PH/s are a lot different then 3 PH/s.   We are talking 5 billion difficulty and anything less than a bitfury with decent (<$0.10 per kWh) is likely no longer economical. 

They are not.  Given the relatively short lifecycle of miners and the lack of need for upgrade chips designing a socketed package is just additional for no benefit.

Then again with automated fabrication it is more than possible for an assembly house to place/solder/test thousands of boards a day.

So knite,

Was anything interesting asked or said regarding HF at the BTC meetup?

No not forever but it can for some time.  The curve is unlikely to bend before 10 PH/s or so and that is optimistic.   Double that if you are a pessimistic.  The 10 to 20 PH/s range will be interesting because it starts to obsolete (value of electricity > value of BTC) the older gear (Avalon, ASICMiner, BFL).   The newer tech is operational but even if prices get cut in half new sales aren't going to look very attractive.  A simple thought exercise imagine you are looking to buy ASIC gear today, multiply all vendors prices by 10x.   That is how current prices are going to look with 10x the difficulty. 

It isn't a requirement.  How about coins have to be moved within so many years (say 10).  Coins that aren't are deleted by the network and respawned as new miner rewards thus keeping the pool of coins available for stake relatively constant.   That is just one idea.  It isn't an absolute requirement.  The creators even stated so.  It is just one solution to the problem.

It can't but you know difficulty isn't going down so when the network is 80 PH/s BEFORE you place your order and even conservatively it will be 100 PH best case scenario and 125+ PH worst case scenario before you receive your hardware that is going to weigh in on the minds of buyers. 

Of course but
a) nobody profits from the inflation in Bitcoin
b) the inflation is outside the control of any party which could game it to their advantage
c) nobody is forced to used Bitcoin.  If someone doesn't see a value proposistion they can simply choose not to participate

None of those are true with national fiat currencies.

Of course it is.  Just because PPC chose not to go that route doesn't make it impossible.  It was a design CHOICE not a requirement.

Why would you argue over that?  Both of those cases are sunk money already.   What matters is future sales and future profitability.   Someone who already bought from HF or KNC hopefully already did the analysis months ago and their future is already set.

No none of that was correct.   I said hashrate growth will slow and those that have high efficiency (both J/GH and $ per kWh) will be able to remain profitable for an extended period of time.   Will it be 10 PH/s? 20 PH/s? 50 PH/s? 150 PH/s?  I have no idea.  Never claimed to be able to predict the future.   

However unless you think people will continue to dump money into hardware that is DAY 0 UNPROFITABLE the difficulty growth curve will bend.  Day 0 profitability is different then difficulty rising too fast and potentially being unprofitable in the future it is more black and white and even the most delusional miner can do simple math like "at current difficulty I make $20 per day, spend $15 in electricity and thus the break even point for my $5,000 miner is 2.5 years even if difficulty doesn't rise ... so I don't think I will buy today".


BTW 14nm & 11nm are a non issue.  There won't be a 14nm SHA-256 chip until 2018 at the earliest.  There likely won't be a 22/20nm ASIC chip before 2015 and that is optimistic.   It takes roughly 3 years before a new process can even equal the cost of a prior process.   Worrying about 14nm today is just silly.

I think you are misunderstanding (giving you the benefit of the doubt that it isn't just blatant trolling)

OF COURSE difficulty is going to continue to go up for the next 3-6 months.  Nobody NOT A SINGLE PERSON said differently.  The hashrate curve will only bend when electricity and hardware costs results in a day 1 profitability that is poor.  At current prices that is >10-20 PH/s.   If prices are significantly reduced it is more like 20 to 50 PH/s.  Obviously even with rapid growth it is going to take months to get there from here.   

You make that seem like it is a bad thing.   The longer it takes the fatter the profits are.  Still you never ending refrain has been if you don't profit in the first 60 days you never will.  On inefficient hardware that might be true however if you are efficient (both in J/GH and $ per kWh) when the curve DOES bend due to economics it is possible to make a small profit every day for a very long time.   

The gold rush phase will be ending in the next couple months after that ASIC mining is going to look a lot more like GPU mining did.   There won't be front loaded profits and people will be looking at breaking even in 8-12 months.  When it is closer to 8 more hardware will be sold and difficulty will go up faster.  As it gets pushed out towards 12 people will be less interested and hardware sales will slow down.  It will become a "boring" lower margin game.

Fixed that for you.

Let me put it this way.  With difficulty growth a rig with 10x the electrical efficiency (J/GH) and $10 per GH/s might not break even for months if not years.   $33 per GH/s?  Not a chance in hell.