Bitcoin Forum

I am starting to reach the limit of what I can cool with my mining farm and am wondering the best approach to take to sustain further expansion. I am currently 14 "rigs" most with 5 GPUs per box. I will be expanding further next week and was hoping to put off this until after the winter, but that is looking less and less likely.

I am in a 800sqft space with 20ft ceilings and a 3 ton HVAC unit that now seems to run 24/7.

So the big question is: What do you do to cool your 10Gh+ mining farm? Do you have pictures and or best practices?

My 1200 sq/ft shop with 20ft ceilings stays at around 90deg F with the doors closed...I have no AC and I am only running about 3.5Gh/s.  I just have 2 box fans blowing down on the rigs.  It seems there is a 2-1 power usage to cool the rigs with AC from what others have said in this forum. 

I can't wait for winter...I wish I was pushing the hashrates you are but I don't envy your cooling costs.  You are basically running ~14kw worth of heaters constantly...let us know if you find something that works.

i think non HVAC single unit AC is better choice if all ur rigs are running in the same room... probably with some kind of vent sucking air out and the AC only cooling the rigs...

This is the first time in my life I've actually wanted winter to come.

K Supercomputer use water cooling.
Cray-2 was the first supercomputer to be fluorinet cooled.

Air cooling eats more power, makes more noise & does not cool as good.

http://www.powercolor.com/image/techzone/setup_12.jpg
http://www.powercolor.com/image/techzone/teczone_lcs_6990_06.jpg

another option is to make a pool of Mineral Oil..
http://www.youtube.com/watch?v=fJdUhtD1zns (http://www.youtube.com/watch?v=fJdUhtD1zns)

Run an intake and exhaust inline fan, this would involve cutting a 2X6inch holes in the walls leading outside. This will significantly lower your electricity bill and decrease the amount of a/c you would be required to use. PM for more details

LOL...
my stupid PC does ~38°C @ 100% GPU load...

60x HD4650 does 1260MH/s
6x HD6990 does 4800MH/s.

Are you seriously running sixty HD4650s?  Seriously?  Why?

dude...

the fastest supercomputer in the world is watercooled.
has 68544 waterblocks for the CPUs.
& 800 for the MBs.

http://en.wikipedia.org/wiki/K_computer (http://en.wikipedia.org/wiki/K_computer)
http://en.wikipedia.org/wiki/Supercomputing#Energy_consumption_and_heat_management (http://en.wikipedia.org/wiki/Supercomputing#Energy_consumption_and_heat_management)

IBM Blue Waters
"closely packed elements require water cooling."

Please answer my question.  Are you really running sixty HD4650s?  Watercooled too?

Holy shit dude...you could be fighting like, 30,000watts with that unit. You just need an exhaust solution, not a 3 ton compressor.

Hum, no. 1 ton of cooling is 3517 Watt.  So a 3-ton HVAC can cool about 10.6kW of equipment, or a bit more assuming heat loss in the server room.

Some use no HVAC to cool 10.6kW of equipment, but that's a completely different DC cooling strategy :)

Blech.  If you weren't in FL, I'd suggest evap cooling.  You don't really have another option in FL besides AC, I'd imagine.  

I'm cooling:
4  x 5990s
38 x 6970s
8  x 6870s
4  x 5870s
all in one room.  Roughly 12,000 watts (just a little under).  Everything is being cooled buy a single window mounted evap cooler (in an adjacent room) that draws 285 watts at full fan/pump speed (and I shut it off at night).  Because an evaporative cooler is being used, I'm able to keep temperature *and* humidity in check.  If you want to run a large Bitcoin farm, a desert climate is definitely the way to go.

Um...  we're not comparing apples to apples here.  If I wanted to buy waterblocks, pumps, radiators, and tubing for all 54 of my cards, it could cost me well over $12,000 to get it done (not to mention a ton of my time).  That's over half of what I paid for the hardware I'm running.  That'd be a pretty foolish idea.  Also add the fact that I'd *still* need a way to get rid of the heat coming off of the radiators.  I'd rather throw that $12,000 into more hardware.

Water cooling doesn't magically make the cards run cooler -- it just allows you to get rid of the heat quicker.  In other words -- if a card gives off 200 watts in heat, putting on a water block doesn't suddenly make the card only give off 100 watts.  It's still 200w, and I _still_ have to get that heat out of the room.  In other words, I'd have to mount all of my radiators in a window or run water lines outside.  No thanks.  Water cooling only helps if you can't get rid of the heat quick enough.  Air coolers are more than enough in this situation.

If you wanted to stack thousands of GPUs on top of each other with no room to breathe, *then* we'll talk about water cooling.  My mining farm isn't a supercomputer with thousands of 1U boxes crammed against each other.

Look at my sig for details.


I will be PMing shortly.


Why do you suggest that this evap cooling would not work in FL? I am going to look into it now, but any further information would be greatly appreciated.

EDIT: Also, it is very dry in my warehouse at the moment. Much dryer than anywhere else I go. Shouldn't I be keeping this environment at a certain humidity and couldn't an evap cooling system help with this?

Evap cooling wont work in FL because your relative humidity is already very high.  Evap coolers work by introducing cold water into the air through evaporation.  If the air is already saturated with water, more water wont be absorbed easily.  If your humidity is high and you run an evap cooler, you might as well just run a fan instead.

If it's dry in your warehouse, it still wont help much either.  It'll be dry for a little while, but running an evap cooler will saturate the air quickly.  You need a steady supply of dry air.  My $600 cooler will crank through 5500 CFM of air.  Circulate that around the building for 10 minutes and it'll be extremely humid in no time.

So based off of this information, my best bet is getting the hot air out of the building as quickly as possible and pulling in fresh air from outside.

Something like this -> http://www.youtube.com/watch?v=G5f_e4P6gMA

Yeah.  If you have a small room full of cards and no fresh air is circulated, they'll quickly overheat.  If you have a large room full of cards and no fresh air is circulated, they'll still overheat, but it'll take longer.  AC units are just heat pumps though...  if you have enough AC for the amount of heat being introduced to the room, there's no need to pull in any air from outside.  Depending on the placement of the AC, it would be worse in fact.  ...similar to having your AC on in the car but leaving the windows down.

If you don't have the AC capacity to cool the cards, shut the AC off and then just blow cooler air in one side and hot air out the other (exactly how my room is set up).  Even if you have fairly high air temperatures, you should be fine as long as you can make sure that no air gets recirculated across the cards.

Yes, and to fight watt for watt is moronic when simple flowthrough can handle the issue. Why run a huge compressor with your fans if the fans themselves can be sufficient? See smoked_out's post. Another option to increase efficiency would be to cool a small space with AC and draw air from the space, through the room, and out the exhaust. Look up the concept of a 'lung' for cooling indoor grows.

I also live in a desert climate, and wanted to use evap, but was afraid to use an evaporative cooler because of the high humidity it can generate.  But reading about it, I guess if the dew point is very low, the humidity coming out of the cooler stays around 50% anyway, which is not high enough to cause any damage to components. 

With refrigerated air it is hard to get the exhaust air out without polluting the supply air.  To get the enough fresh air flowing into the room so that the hot exhaust can be vented outside while also keeping the room temperature down has been very difficult and it's the main thing holding me back on my capacity right now. 

Giga, your high ceilings are key.  My shop is about 250 sq. ft. with 9 ft ceiling and poor insulation to the hot outside and the most I can pull is 12-13 GH/s with 24,000 btu ~= 3kw of cooling with refrigerated air.  I am also exhausting the hot air through a 6" by 14" duct and a 1500 cfm fan.  Insufficient and will be improved, but the electrical thing takes precedence and I'm still waiting for estimates from my electricians.  If I had a 20 ft ceiling it would make a big difference.  Just cut a hole in the wall at the top and put a 4500 cfm fan blowing all that hot air that collects out.

Correct.  Ideal humidity for a data center is between 45-55ish anyways.  In a desert climate, if the humidity starts to get high outside, it's probably cool enough to just shut off the pump on the evap and just use the fan.

evap cooler + multiple pounds of silica gel ($4/lb)(holds 40% of weight of water)

can be regenerated at 50deg c (capture some gpu exhaust and pipe it to a box)

12.000w of heat cooled with 285w ? Air cooled?
285w? do you have a killawatt meter?
i doubt is 285w.

don`t forget to add video card fans:
HD6970 are 2A = 24watt fans.
12v x 2A = 24watt. each. * 54 cards = 1296watts.

the noisier the hungrier.

assuming your calculations: not including video card fan power draw "free"
not including the price of adding more cooling "also free".

285w/h *12h "half day" = 3.5kw/day.
*7 days *4 weeks =98kw/month.
*12months = 1176kw/year.
if you pay 0.25usd. x kw = $294usd./year.

Cappacitors last 10 Years working under <80°C.
lets see how long yours last.

assuming they last 10years = ~$3000usd. without adding inflation-.
add 5% inflation each year. = ~$5656usd. You Pay, with added cooling alone.

with $3000usd. can buy 30x 6970 waterblocks.
Car/Truck radiators are "free" in junk yards.
only need tubes, pumps & distiled H2O every month.

I think, i stick with WaterCooling.
as they say: "Watercooling system minimizes failure".

The guy on this video:
http://www.youtube.com/watch?v=eLt8Se3vVNg
http://www.youtube.com/watch?v=G5f_e4P6gMA

has 3x4=12x 160watt fans. = 1920watts hour = ~2kw/h running 24h = 48kw/day = x7x4 = 1344kw/month x12 = 16.128kw/year. or *365=17.520KW/ year.
*10 years = 175.200 KWatt.
without adding video card fans @ 0.192usd. = $33.638,4usd. without inflation.
5% inflation over 10 Years = ~$54.793,4usd.

24 pcs x2 5970  = 48 gpu fans at 24 watt each = 1152watts/h x24 hours = 27,648kw/day x7x4x12x10 = ~92.897,28kw (or 100.915,2KW if x365x10)*.192 = $17.836,27usd. (or $19.375,7184) * inflation = $29.053,4usd. (or $31.561usd.)
= $83.846,8usd. or ($86.354,4usd.) without adding cooling investment, failure rate, etc...

Bitcoins today are $4usd. & difficulty 144k
http://www.bitcoinmonitor.com/ (http://www.bitcoinmonitor.com/)
http://bitcoincharts.com/markets/ (http://bitcoincharts.com/markets/)
if makes 50 Bitcoins a month @ $4usd. = $200usd.
cooling eats $2118,144KW a month, fans only not including PSU effficiency, capacitor aging, motherboard, cpu, usb drive or hdd.
*0.192usd. kw/h = $406,69usd.

= needs to make 100 Bitcoins a month to pay cooling. 0 profit.

how much BTC you make a month?
if goes under $4usd. or difficulty over 200k will loose money.

4x 5970 run 3009 MH/s = 752,25 each.
the guy in the video has 48 = 36.108 MH/s or 36.1GH/s

If the fastest computer in the word is watercooled its because its cheaper.
the electric bill of the K computer = 10.000 homes.

kiss (keep it simple stupid)

build a shroud / frame around a 20 in box floor fan
filter the intake air with a 50 cent furnace air filter
mount your rig inside the shroud
point it out a screened window

DONE

Yes.  Measured exactly 285w with a kill-a-watt.  How much power do you think it takes to drive a 5500CFM fan and a small water pump?  Certainly not more than this.  On really dry/hot summer days, my cooler will drop the air temperature over 20 degrees ferenheit (6.6 celsius).


Yes, I need to add in the power from some of my fans.  I was keeping things simple just to compare evap vs AC.


I only pay 0.104377usd per kw during the day.  At night and on weekends, I pay 0.038162usd per kw.  In another month, the air temperature outside will be so cold that I wont need the evap cooler anymore.


Most of my GPUs are running around 65c and 67c.  There are a few that are hitting 75c.  The air coming off of the heatsinks is much cooler.  I doubt the caps are getting that hot.  Besides...  why would I keep these cards for years?  That wouldn't be very smart to not sell them and upgrade eventually.


I have more than 30 cards.  Pumps aren't free.  Car/truck radiators aren't free.  I'd still have to run water lines outside and keep them from freezing if the power ever went out.  Temperatures have hit -10C here before.  Lastly, my TIME isn't free and it will be difficult to sell 50 water blocks when I'm done with these cards.  The 7000 series will be out by then and people wont be looking for 6000 series waterblocks.


For my main machine at home?  Yep.  It's been water cooled for years.  Definitely not for my Bitcoin rigs though.  Too expensive.


But I don't.  :)


But I wont!  :)

At the current difficulty, Bitcoin would have to drop under $2.5 for my rigs to to be unprofitable in the summer.  It would have to drop under $2 for it to be unprofitable in the winter.

He's on the right track.  I'd be finding a way to use that fan for more than one rig though.

20x20 should be good for 3 or more rigs
take out the window glass and use the whole thing .. two fans
use a second window

the key is moving the heat from one place to another
setting the farm on a shelf exposed accomplishes little only prolongs the agony & future problems

fyi don't handle things anymore than necessary  finger grease collects dust

Very enlightening thread, kudos! i was thinking of water cooling a while ago until i factored the costs, i said no way. though i'm just running a 4 card rig but i'm planning to expand soon

so true...

I always want winter to come, this summer weather at +30 or more is ridiculous. I love our bracing winters that break -40 on a regular basis! ... No, that is not sarcastic, I do, mining just makes it even better.

My gas bill for the winter will be peanuts compared to last year!

Should I also be piping fresh air in from the outside at the same rate? I have 20x20x40 space which is 16,000 cubic feet of air. Are you saying that I should remove a quarter of all the air in the building every minute? Also, do you have any links for prices on this equipment?

I don't want to help the water-cooling guy, but he is so clueless I can't help myself.  You do know that all your water-cooled rigs are actually air cooled, right?  I mean, the water goes through a radiator where a fan cools it off before being recirculated.  The increase in room air temperature is the same whether your farm is cooled with air or water.  If you spent more time thinking and less posting, you probably could have figured this out.  Where did you think the generated heat went?  http://en.wikipedia.org/wiki/First_law_of_thermodynamics (http://en.wikipedia.org/wiki/First_law_of_thermodynamics)

Giga: here is what I will be buying if my Lasko 20" box fan in a windows with security/weather shroud doesn't move enough air:
http://www.google.com/search?q=industrial+exhaust+fan&tbm=shop&hl=en&aq=f (http://www.google.com/search?q=industrial+exhaust+fan&tbm=shop&hl=en&aq=f)

what if you store 2 metric tons of water in your back yard, and use the sheer volume of water to keep the temperatures down

over NINE THOUSANNNNNND

Water is a huge sink, and 2 metric tons is not very much. You will still need active cooling on the reservoir.

Fastenal.com has a ton of locations in the us, selling industrial equipment and supplies try them...

 http://www.fastenal.com:80/web/products/detail.ex?sku=7059979&ucst=t

5500 cfm for 711usd, seems good to me

I am in a warehouse that has an office within. The overall dimensions are 20ft x 40ft with 20ft high ceilings. The AC unit currently has two drops into the office and three blowing into the warehouse.

BTW, I figured out the first things to do when it gets too hot, open up the 8x16 roll up door. Dropped the inside temp from 98 to 94 in a couple of minutes and it was free. Still need to figure this thing out though.

the video you linked earlier is a great solution, but it is quite expensive to get a professional HVAC installation like that.  Also it seemed like quite a hassle for that guy every time he needed to tweak something, take down all those fans reach real deep etc.

create some sort of hood/enclosure that can capture hot air from cards and expell it outside through duct piping.  you can also install fans in the outside of the opening to suck all that hot air right out.  if done properly, your AC should barely have to be on.

well enough water to last you through the day, just let it passively cool at night
(i live in an area where night times get pretty cold)

I have been doing more work to rectify my "heat situation" and have consulted multiple hvac / ventilation specialists. The consensus seems to be to either vent the hot air out of the building or condition the air. Don't do both.

Another point they kept bringing up is the very little amount of insulation between the roof and the air inside the building. So I am having extra insulation put on the ceiling early next week to see just how much it helps. I will be going for the highest R value I can get and will follow up once it is installed. Insulating 800sqft should be about a $1 a sqft, a lot cheaper than other solutions.  :)

extra insulation will help retain more heat.

not really true
depends where the heat is coming from
IE it will keep the suns heat out

Problem is, in Florida, you're going to have to do both.  Condition the air AND exhaust the heat.  Like this:

http://www.42u.com/images/Hot-aisle-containment-42u-750.jpg

Except, since its only a 800 sq. ft. data center, instead of recirculating the hot air back into the CRAC you can draw fresh outside air into the CRAC intake and blast the hot air to the sky.

Thanks for this diagram. Actually just had another hvac guy out and he was the most competent of all that have come out so far. This is pretty much the direction he is pushing. Smaller room, high air flow from one side of the room to the other and enough cooling capacity to cool the 17Kw of equipment (5 ton).

Hvac spec that sounds expensive...make sure your spending your money well

If you really want to cool then think like a grower. Mini ductless AC unit and inline ducting for intake and exhaust. Use an environmental control and when temps go higher than ducting can handle they shut off and mini split system kicks in. The inline venting alone can do the job minus those really hot days. I highly recommend the Panasonic whisperline duct fans. Do the math, get the right size, x 2 since you want intake and exhaust unless you have passive intake like an open door to suck air in from.

You can find lots of growers on craigslist selling equipment all the time.

You are talking closet scale equipment. He's got the equivalent of a 20-lamp grow in 800 square feet, not sure of the shape, and 20ft ceilings.

I worked on a 16x1000w op in about 2000ft^2 and we kept temps below 80 with a 5-ton AC and 2 12" CanFans (good brand, but un-throttlable) and two 10" centrifugal fans in and out (with a fair amount of filtration resistance). We upped it to 20 lamps and it started running about 85, just over the capacity of the AC. The power bill was phenomenal, and 2/3rds of the room was on 12/12. Goddamn that place made some money. Puts bitcoin at $30 to shame. Then the sheriffs came and cut down all our children and ended up dismissing the charges after doing about a hundred grand in damage. Good ole' fuckin' amuuurica.

During the day, the sun is producing a pretty serious amount of heat within the warehouse since the sun shines a great deal here in FL. Installing the spray form insulation will allow me to only need to cool the heat that is produced from the equipment during the day and night which *should* help considerably. I have been reading about the spray foam insulations and they have a much greater effect on reducing conductive heat from ceilings even though they may have a smaller R value than traditional insulations.

Installation is happening wednesday so I will soon be able to give some definitive results. I have monitoring equipment for the temps in the building so we will have before and after results with 19.5Gh running in the facility.

Please tell me you sued the city afterward :P

As for the insulation, the point is to keep the environments isolated - yes it keeps the hot air in, but it also keeps ambient hot air out.  I brought my attic insulation from R30 up to R60 last fall and drastically reduced my heating costs in the winter, and my cooling costs in the late spring/summer.

Then I started mining and threw those savings away ;)

Who's got the coin to sue Los Angeles? We wrote it off as a 'well, that sucks, but at least we're not in jail' loss.

BTW, raid was not in response to heat/smell/power consumption...it was dispute with a landlord (who signed a lease that stated we were using the property to cultivate medical cannabis and then narced on us after two years because he got a better offer for the space).

Well whats the results

Ok. Here's what I ended up doing.

1. Added the foam insulation onto the ceiling. This leveled out the temps in the building, cutting down the high temp by about 7 degrees. We had a high inside of 103 a couple of days and how the highest the temps have been is 96. This also cut out the lower temps we were experiencing in the evening. Instead of dropping to 78, we now drop to around 85.

2. I have redirected the entire 3 tons of AC to point directly at the equipment through a single 16" air duct. This has helped lower the temps at the GPU cores without lowering the temp in the building itself. I also added an LG portable AC unit in the office area which cut out 315 sqft of space that the 3 ton AC handled before.

Next I am going to take the 16" air duct and attach smaller ducts to direct cold air directly onto the computers which should help even more. I am also going to build an exhaust manifold like in the video but instead of venting it to the outside, it will send air into the return of the 3 ton AC unit. This way I will not lose all of the conditioned air the AC worked to condition and the hot air will not mix with the cooler air in the building.

The exhaust manifold will also be double sided, basically sandwiched between two shelves. The manifold will basically become the hot isle in the hot/cold isle  diagrams above.

You first.  ;)

Mineral Oil is neat, but the thermal transfer sucks.  Getting rid of the heat from any serious rig using mineral oil would me *much* more trouble than it's worth.  You'd need radiators with 4.3X more surface area than what you'd need to cool them with water (or non-conductive fluid made for this purpose...  which is $20 a liter).

From the same page that you linked:

I was looking into this before I redesigned my case full of five computers for better airflow: The thing is, they care if it looks pretty, where if this is out in an industrial area like it sounds, he doesnt really have to care about four old car radiators outside the window :p Also, they are cooling with ambient room temp: Depending on your geographical location (like here, for example, where delightful winter with -40 degree weather is coming), the heat exchange should be a whole lot more.

Don't get me wrong...  I'd love to do this with my main machine.  It'd work fine for daily use and the occasional game with 1 mid-grade video card.  Start throwing a hundreds of watts in a tank with multiple GPUs or with GPUs that are being stressed for Bitcoin mining and you're going to have a *very* difficult time keeping temperatures under control.

haha

You could, but their rig didn't really produce a lot of heat (relatively anyways).  Their rig worked because it was made of glass.  That had just enough surface area to transfer the heat away.

To put things into perspective -- with water, you'd need a 3x120mm radiator to cool a CPU, power supply, and a couple video cards that were Bitcoin mining.  With mineral oil, you'd need more than 4 of those radiators to get the same cooling capacity.  A 4x GPU mining rig with 200 watt cards *might* be able to be cooled with a large car radiator (depending on the fans used and the outside air temperature).  If I wanted to cool all of my mining rigs, I'd need radiators with more combined surface area than the outside of my house.  :D

You'd need some wicked pumps to move that fluid around too...

No experience, but just thinking out loud; if you'd install a ground water well and put a sizeable container with water (and/or long PVC pipes going back and forth to act as makeshift heat exchanger )  in the datacenter and constantly pump water through the containers, it should remove quite a bit of heat at relatively low cost. Basically cooling the room with ground water.  Where I live ground water has a constant 15C temperature, but I think its the same everywhere. Drilling water wells is surprisingly cheap. Water containers can be found cheap. PVC piping costs next to nothing.  A power efficient pump with sufficient flow will cost you a bit, but the savings compared to AC Im sure would make it worth it.

Oh, and should you have a pool.. use the warm water to heat your pool. You could swim all winter :)

Has anyone that has a water cooled rig tried to use the waterless coolant in it, I use it in my car and it did benefit cooling as specified by the manufacturer.  If anyone feels like experimenting I use the Evan's NPG+ coolant, I have been wondering for a while if it would help in computer applications...it certainly helps in cars.

Here's a link:  http://www.evanscooling.com/

I have been running this in my car for at least 7 years with no issues.  The only problem is you cannot have any water in the system before you add the coolant, there is an installation method that you have to follow.

I have also considered something like this, but I am currently renting, so drilling wells in the backyard might cause the landlord to be a little flustered and stern.

I was meaning to test the conductivity, just never got around to it.

Distilled water is non-conductive, costs <$1 per gallon and has higher thermal conductivity.

Distilled water will short circuit after a while. Its been tested, I think by Toms Hardware. Ill see if I can find the link later.

I do remember reading about some nanostuff you could mix in water or oil to improve thermal conductivity. It was still in development and testing phase a few years ago, but looked promising and was being specifically designed for use in computers, datacenters and some other key niche markets. Not sure if its reached market yet, let alone what it would cost.

Air is a liquid
Air is free
Air is easy to move around

Give the rigs plenty of fresh air and that is that.  you guys are overthinkin it.

get the hot air out of the room, bring fresh air in.  even if that fresh air is 100 f, it is stall alot cooler then the air that the video card just expelled.

It also has a thermal conductivity that is 25x lower than water.

so use 25x as much air...problem solved.

At a certain point, the 25X airflow becomes a noise and possibly cost problem: I mean, sure, high CFM fans are cheap, but 25 is a dam big multiplier.

If the profit margins on mining had stayed at the (clearly ridiculous, but a guy can dream) levels of $30/BTC, then it would almost be worth it to cool it with flourinert. However, 3M wants... a lot for that stuff. A lot a lot. You could make it nearly silent. There is at least one company I read about specializing in data centre cooling with flourinert, and the savings in cooling power is impressive. You have to remember, of course, that it depends on scale: What is easy to aircool in your house might not be easy (or possible) to aircool if you have, 5, 15, or 50 of them.

Anyone ever heard of a datacentre using equipment other than air conditioning, I haven't.

For peace of mind I run my expensive hardware investment with an air conditioner that will always keep the room at 19 celcius - I can go to work, go on holidays, watch tv and be assured that my machines are at a steady temp.

My rigs use 5kW/h and my air con can cool 7kW of heat with just over 2kW of power. Why invest big and then skimp on cooling? Cooling needs to be factored into the cost and is as important as power surges or dirty power - it will destroy your hardware!

With air conditioning you can't introduce separate systems as it messes with the airflow. My GPU's on my 6990's run at around 70 celcius or under continuously. This is with 3x 6990's in each rig running at a standard 830Mhz per GPU, they are also running inside haf x cases with a modified side panel (professionally designed and cut via an engineering company) to fit 4x120mm delta fans for air flow. These fans have dust filters also. Each rig uses about 1.2kW/h and there are four in total. Just to add the fan speed set on the 6990's themselves is 70%, though it can manage no problem at all on 60%.

If your serious then professional is the way to go. Jobs isn't still running his company out of his garage. Business (i.e. investment) requires industry standards. Otherwise it's a joke.

I see people running custom cooling solutions and are just about keeping them under 85 celcius. What if it's a hot day and they pop up to 95? Do you panic? Do you call in sick to work? Do you fly back from your holidays?

I am also considering down clocking my GPU's rather than having them maxed out at 830Mhz all the time - a few less hashes isn't going to kill me. With regard to overclocking, I wouldn't consider it. Cards are certainly not designed to be maxed out all the time, never mind overclocking them.

I have regularly handled ventilation on anywhere up to 25,000 watts worth of lighting, often in cramped spaces. It was easily accomplished without flourinert, just good flow and a properly sized AC. You would still need to run an equitable compressor to chill your cooling liquid.

google for "Data center liquid cooling". 3 million results. It isn't common, but it is not heard of. Keep in mind, bitcoin miners with 4 or even 8 GPUs per system is a lot of heat output per sq foot, well into or past the more exotic data centers like blade servers, etc.

Also, on the note of 25kw of lighting, keep in mind that a high density server cabinet can hit 30 kw all by itself right now, with projections up to 50kw coming in a couple years (Source: http://www.42u.com/liquid-cooling-article.htm). Was all 25kw of lighting confined in a 42u sized space, and, also important, all stacked on top of each other? On that note, remember, that is one cabinet, you sure can get a lot of those in one room.

Also, most data centers these days are looking into a lot of other options to improve PUE, heat wheels, liquid cooling, etc, at a certain point, it isn't efficient to just throw more AC at it.

Water cooling was much more common in early computing history.  As the power density of a rack fell below 5KW it became easy to cool with air.  However power density are rising and for high performance computing (which can easily achieve 8KW/rack +) water cooling is starting to come back.  As someone else said just search google.   Many supercomputers are water cooled because they have high power densities.

Mining blows those densities out of the water (no pun intended).  A 4U case could hold 8 GPU.  1200 to 1500W per system.  11 systems per rack = up to 15KW/rack.  Ouch.


Heat is what matters not load.  I overclock my 5970s 37% to 1000 MHz core.  You know what temp then run at 24/7? 40 deg C.  The fans on the radiator are very low noise too.  Most people air cool @ stock and it sounds like a vacuum cleaner.  My card runs very cool at 37% higher hashrate and the entire system noise is less than single oscillating fan.

As far as overheating.  There is software out there to halt miner, or even shutdown system if temp gets too high.

Now nothing beat the cost of air so if you have a lot of space, don't mind open rigs, and don't care about noise then you should air cool.  However if you want high power densities (i.e. 3x 5970 in a quiet closed case) then water cooling can't be beat.  Two of my rigs sit in the garage and are air cooled but one of my rigs is also my workstation.  It sits indoors in my office.  My wife wouldn't be happy to have fans as loud as a a leaf blower running 24/7 (and it still would do a piss poor job of cooling).  Even if she didn't mine I couldn't get 3x overclocked 5970s cooled to 40C without water cooling.

Ok, I stand corrected but I wonder about heat exchangers....although not as capable of cooling densely packed blade servers, my 3x6990 rigs have enough airflow with 4 deltas that use a total of 80 watts.

I am looking into ducting the exhaust heat directly to a radiator with cold water running through it - the air in the room stays the same but the conditioner does not need to cool as much.

I suppose my real question is...how much does water cooling cost to implement and maintain and what are the risks? My initial cost for air conditioning including install was about 1500 euros. It costs me about 750 euro or so on electricity every month. Yearly running cost is 9000 euros.

I thought that reducing the clock with mean less stress on gpu's, not just using heat as a factor. I'm sure Xeons are designed for durability but why AMD GPU's.

Designed for zero downtime storage? Storage? Storage is easily achieved with hard disks and SAN's, servers are for processing. I have the same level of design as a datacenter regarding cooling. I have airflow and ac, granted I don't have complex issues such as hotspots and lines of racks to take into consideration but I have the two key foundations and they work for my server room.


Not sure what point you are making here but we a moderate climate with average temp of 19 celcius over the course of the year. Sucking out and pulling in is still makeshift.


Just one ac for the datacenter here.

http://www.youtube.com/watch?v=ioCZojN4A0g
http://www.youtube.com/watch?v=BH8X8w8a4f4

Maybe NSA and IBM have no clue though :)

Well I have been saying datacenter but hashing farm is more apt.

In case anyone was wondering what I am cooling, here are some pics.


https://lh4.googleusercontent.com/-DIKVvpcSXZI/ToHFVXENgWI/AAAAAAAAAwc/FaIIdREZQpQ/s720/IMG_0137.JPG

https://lh3.googleusercontent.com/-uDTDLbEEPDQ/ToHFXnN3xgI/AAAAAAAAAwg/RQNePkHKrI4/s720/IMG_0141.JPG

Do I count 90 GPU's there  :o

69 on the rack + another 13 elsewhere.

Cards I use:

MSI 6950s x 15
Sapphire 5830s x 50
AMD 5970s x 12

Everything is stable now but i popped 4 5970s before i started using extenders with the molex. Also popped 3 6950s from when i first started and thought it was cool to run the cards at 100c. Also had 3 5830s that didn't want to be overclocked so i sent them back and got new ones. I have also sent back 6 PSUs and have probably 5 MBs that still need to be RMA'd.

Sweet!  You finally put up some pics.  That is very impressive build.  Clean and standardized-looking.  Good cable management too.   ;D

That NSA computer is oooold, though.

no one said liquid cooling was a recent idea.