A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[DeathAndTaxes]

Finally got it stable and hashing.  It was brutal trying to get 4x5970 working in Linux.  Sadly BAMT doesn't work (no dice for 8 GPU in 32bit kernel).  I tried xubuntu but hosed something up installing SDK 2.1.  Tried to restore from an image I made and it wouldn't boot.  Ended up grabbing LinuxCoin (x64) and dropping in cgminer.  I don't like it but it works for now.

Code:

  cgminer version 2.3.1 - Started: [2012-02-28 12:30:03]
--------------------------------------------------------------------------------
 (5s):2935.9 (avg):2952.6 Mh/s | Q:11590  A:3272  R:57  HW:0  E:28%  U:15.06/m
 TQ: 8  ST: 9  SS: 26  DW: 1950  NB: 17  LW: 15248  GF: 0  RF: 0
 Connected to http://192.168.0.189:9332 with LP as user user
 Block: 000007200ebc4183c7cefd4ed93eea81...  Started: [16:04:46]
--------------------------------------------------------------------------------
 [P]ool management [G]PU management [S]ettings [D]isplay options [Q]uit
 GPU 0:  52.0C  960RPM | 378.0/378.7Mh/s | A:445 R: 7 HW:0 U: 2.05/m I: 8
 GPU 1:  52.5C  960RPM | 378.0/378.7Mh/s | A:399 R:10 HW:0 U: 1.84/m I: 8
 GPU 2:  49.0C  960RPM | 378.1/378.7Mh/s | A:431 R: 4 HW:0 U: 1.98/m I: 8
 GPU 3:  54.0C  960RPM | 378.0/378.7Mh/s | A:403 R: 3 HW:0 U: 1.85/m I: 8
 GPU 4:  59.0C  960RPM | 352.8/353.3Mh/s | A:396 R:10 HW:0 U: 1.82/m I: 8
 GPU 5:  58.0C  960RPM | 378.0/378.6Mh/s | A:407 R: 9 HW:0 U: 1.87/m I: 8
 GPU 6:  55.0C  960RPM | 378.1/378.6Mh/s | A:413 R: 8 HW:0 U: 1.90/m I: 8
 GPU 7:  53.5C  960RPM | 327.8/328.2Mh/s | A:378 R: 6 HW:0 U: 1.74/m I: 8
--------------------------------------------------------------------------------

Clocks are somewhat conservative but it is running @ ~3GH/s and pulling 1112W at the wall (120V).  Once it is stable it will move to the 240V PDU so it should be 20-25W less there.  One of the eight cores wouldn't clock to 835 (went sick 0Mh/s instantly) since I was tired I just left it @ 750 Mhz for right now.  It is pointed at p2pool which makes share count, U, etc "weird" due to dynamic share difficulty.

Total System Load: 1112W
Total System Hashing Rate: :2.95 GH/s
System efficiency: 2.67 MH/W
Measured (not calculated) no-GPU (not even installed in rig) system idle: 190W
GPU AC load:  230W ea
GPU DC load: ~200W ea
Total System thermal load on water loop: 800W
GPU efficiency: 3.21 MH/W.  

On edit: old pic removed (better pics below)

Note:
I don't have 30GH/s the figure comes from the potential.  3GH/s per 4U.  45U in a rack.  1U for switch, 2U for PDUs, 2U for "watchdog" server leaves enough space for 10 4U rigs.  

[1715259589]

1715259589

[1715259589]

1715259589

[Starlightbreaker]

daaaaaaaaamn

[oscer]

Are you keeping that in a DataCenter ?

[DeathAndTaxes]

Are you keeping that in a DataCenter ?

No although my office is looking more and more like a datacenter.  Got to get some sleep but 24 5970s produce a lot of heat and AC cuts into my profits.  If this 1 rig test goes good my goal is to rack of 6 of these in standard server rack with a heat exchanger to a secondary cooling loop which runs outside to a very large radiator.  Dump 6KW of heat directly outside.  We will see how this 1 unit experiment goes. 

[oscer]

Are you keeping that in a DataCenter ?

No although my office is looking more and more like a datacenter.  Got to get some sleep but 24 5970s produce a lot of heat and AC cuts into my profits.  If this 1 rig test goes good my goal is to rack of 6 of these in standard server rack with a heat exchanger to a secondary cooling loop which runs outside to a very large radiator.  Dump 6KW of heat directly outside.  We will see how this 1 unit experiment goes. 

How Much that thing cost you to build ?

[Splirow]

Why are your GPU temps only around 42-44c?

[bravetheheat]

Why are your GPU temps only around 42-44c?

I would guess from the pic that it would be due to water cooling.   

[ZPK]

3 video card do not plug power....why or how ?)
and how motherboard you use ?

[DeathAndTaxes]

3 video card do not plug power....why or how ?)
and how motherboard you use ?

Oops that was an early pic when I was testing each card.  BAMT only worked w/ 3 cards so I unplugged one card, mined, powered down, and plugged in a different one to test all 4.

The MB is the "miner classic"  MSI 890FXA-GD70

[DeathAndTaxes]

How Much that thing cost you to build ?

Well I already had 3 of the waterblocks, tubing, fittings, connectors, and radiator.   Plus my "standard" air cooled rigs are 3x5970, MSI 890FXA-GD70, 2GB of RAM, Sempron, USB Stick, and Corsair/Seasonic 1200W/1250W PSU which all went in here.

So the incremental cost for the "test rig" was just the case, pump and one waterblock ~ $400.

Full conversion of my other rigs would cost ~$900 ea (4x waterblock, fittings, connectors, tubing, heat exchanger, pump, silver kill coil, and case) unless I can get some volume discounts.

To build it from scratch would be ~$1300 plus cost of GPUs.  When I bought them most of them it was more like $300-$350 ea now they are insanely expensive but hopefully that will change.  I would strongly discourage someone from trying this unless they are already a confident miner AND have experience with liquid cooling.  I am not sure I would try this if I didn't already own the aircooled rigs.

The reason for doing it is to:
a) improve the efficiency (getting >3 MH/W now at >3GH/s and with underclocking/undervolting that can rise to 5MH/W if necessary over time to stay profitable)
b) push the cards higher.  I think 3.2 GH/s per rig is possible allowing me to pickup 20% more revenue before the reward cut
c) eliminate roughly $4000 per year in AC costs to be more competitive in face of rising # of FPGAs.
d) keep the temps more stable (50C @ 99% load 24/7 all year long shouldn't be a problem)
e) The wife acceptance factor.  She has been a "trooper" with this mad scientist and his 14GH/s of whirling, buzzing, heat belching fun.  
f) maybe someday provide "free" hot water and heating for the entire house (~$1000 per year).

[DeathAndTaxes]

Some updated pics.


Not your average watercooling job.  Very "industrial". No lights, UV dyes, clear tubing, bling bling leet gamer nonsense.  Maybe I am weird but I think it looks nicer that some flashy setups.

The PSU is a Seasonic 1250W 80-Plus Gold. I only have one of the 80mm fans in the back hooked up.  The sempron puts off almost no heat so might be fine just to use the PSU intake as an exhaust (yes computers did that at one time ). The front left (upper right) is the 3x 5.25" bay.  The front center houses a 3.5" bay w/ 80mm intake fan (removed).  The front right houses a 120mm fan for airflow across the GPUs.   The top cover also has 2x120mm fans but I don't think I will be using them.  This case is sold as a CUDA rack but honestly I don't see it having enough airflow for air cooling 4x Teslas.

Yeah the wiring job is horrible.  I haven't decided on which way the tubing will go.  My first thought it to mount the heat exchanger (should arrive by this weekend) in the front and run "cold loop" lines w/ quick disconnects out the 5.25" bay.


Closeup of the 4x5970s.  Finding a rackmount case w/ 8 expansion slots is tricky. The few that exists are $500+.  Luckily Chenbro makes this case and it wasn't too expensive.  The waterblocks are DangerDen because they are the cheapest full coverage waterblocks.  Watercooling is expensive but "saving" money using non-full coverage block is useless as the VRMs get too hot.  5970s are nice because one block cools two GPUs.  7990 would be even nicer but by the time they are affordable FPGA will likely have killed that idea.  I likely will seal the bridges with silicon sealant and apply plumbers tape to all the threads.  I want it as no maintenance as possible.


Front view w/ filter/door open.  On the left is 120mm intake (0.3A).  I would guestimate it at ~60 CFM.  Provides a slight breeze over the cards which is all that is needed for the non-waterblocked components like the caps.    On the right is a dual bay reservoir which also mounts the MCP655 pump.  


With door closed.  It is currently hooked to this "test radiator".  The end goal (assuming all testing goes good) would be to mount a water to water heat exchanger inside each rig with quick disconnects to attach it to a outer "cold loop" which runs to an outdoor radiator.

I am kinda surprised this radiator is holding up because 4x120mm is way undersized for ~800W thermal load.    Ambient temp is ~23C and cards are running at ~55C after 8 hours of hashing.  Strangely the load "bounces" more than I have seen on other rigs going from 1070W to 1120W.  Need to investigate a little further.

Thoughts, comments, suggestions?

[BFL-Engineer]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs


Regards,

[rjk]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s) with only 4 units, costing total of 2400USD (+ shipping)
and consuming only 330 Watts (less if a high-efficienty power-supply was used to power all 4 units). Virtually silent as compared to GPUs
(26dB each unit) and a faction ( A third so to speak ) on electricity costs


Regards,

This is more fun

And, he can run bitforces off of it!

[Photon939]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s) with only 4 units, costing total of 2400USD (+ shipping)
and consuming only 330 Watts (less if a high-efficienty power-supply was used to power all 4 units). Virtually silent as compared to GPUs
(26dB each unit) and a fraction ( A third so to speak ) on electricity costs


Regards,

And it probably wouldn't arrive until 2013 - Don't get me wrong I'd like to purchase one myself but spamming this guy's build thread is in quite poor taste IMO.


To the OP: I wish I had gone with rackmount cases for my little miners I have here, your setup looks good.

[DeathAndTaxes]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s) with only 4 units, costing total of 2400USD (+ shipping)

And I would get it in 4-6 weeks months ?

If you are going to spam/advertize in my thread it would be nice to read it first.  The MB, PSU, RAM, CPU, and 24x 5970s (the most expensive part) were purchased between 12 months and 6 months ago.  They are already a sunk cost as I have no desire to try an unload all that equipment.  

While I am interested in migrating to FPGAs.  I have a summer temps arriving in <3 months, so I am looking for a solution to expand the longevity of my EXISTING HARDWARE (which has produced if my math is right ~250 quadrillion valid hashes so far).  Hopefully I can get another ~250 quadrillion hashes more as I mine these cards into the ground.

Quick questions:
If I place an order today for 4 BFL Singles today can you guarantee a delivery date?  What date would that be?  Tell you what, if you are willing to guarantee delivery by 1 April publicly on this forum  (w/ $200 penalty paid by BFL for non-delivery) I will buy 4 today.

[BFL-Engineer]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s) with only 4 units, costing total of 2400USD (+ shipping)
and consuming only 330 Watts (less if a high-efficienty power-supply was used to power all 4 units). Virtually silent as compared to GPUs
(26dB each unit) and a fraction ( A third so to speak ) on electricity costs


Regards,

And it probably wouldn't arrive until 2013 - Don't get me wrong I'd like to purchase one myself but spamming this guy's build thread in quite poor taste IMO.


To the OP: I wish I had gone with rackmount cases for my little miners I have here, your setup looks good.

None was intented. My apologies if it appears so...


Regards,

[rjk]

Damnit DeathAndTaxes, you are going to make it hard on me keeping up with the joneses

First things first: what kind of rad are you planning on to dump +6kw of heat? Most oil coolers are rated in horsepower, which is 746 watts per horsepower. You can get a compact (~1 square foot) rad that is good for 10 HP, but only with a loud cooling fan. However, ~no fan or low speed fan gets it a derating to 8 HP (plenty).

Second: Heat exchangers. Some plan to make them pluggable, or some way to make it so you can add/remove them without disturbing the whole cluster? Also, which ones - most I have seen won't handle the heat from 4 5970s.

Third: Power. Is that PSU going to be good enough for continuous duty? Not sure what you run now, but perhaps it will be.

[mtminer]

Pretty impressive.

[jamesg]

e) the wife acceptance factor.  She has been a "trooper" with this mad scientist and 14GH/s of whirling, buzzing, heat belching fun.

I had to get a warehouse. Consider yourself lucky.

[DeathAndTaxes]

Damnit DeathAndTaxes, you are going to make it hard on me keeping up with the joneses

Sorry about that. Have been planning this for a while (Dec was great but June will be brutal w/ 6KW of heat).  Seeing your out of the box thinking got my behind moving.

First things first: what kind of rad are you planning on to dump +6kw of heat? Most oil coolers are rated in horsepower, which is 746 watts per horsepower. You can get a compact (~1 square foot) rad that is good for 10 HP, but only with a loud cooling fan. However, ~no fan or low speed fan gets it a derating to 8 HP (plenty).

I haven't spent too much time on the "outer loop" yet.  I want to stress test this single rig first.  If it fails I didn't waste too much time and money.  I am thinking big (maybe 24" x 24") with a 16" or maybe 2 x 10" fans. 

There are a couple of options car radiator, oil cooler, industrial heat exchanger.  I don't know exactly what yet but it likely will be big. 

I found another company which makes really nice custom units for cooling lasers and other high temp components but their prices are insane (way outside my budget).  They have very detailed charts for c/w which gives me a ballpark idea on where I need to be aiming for (surface area and cfm).

http://www.lytron.com/Heat-Exchangers/Standard/Heat-Exchangers-Tube-Fin

Second: Heat exchangers. Some plan to make them pluggable, or some way to make it so you can add/remove them without disturbing the whole cluster? Also, which ones - most I have seen won't handle the heat from 4 5970s.

Yeah the tubing on the "cold side" of exchanger will have quick disconnects.  This will allow removing one rig from the rack.  For troubleshooting I am planning on keeping my "test radiator" from photo above with a pump and reservoir.  That way I can connect a sick rig to the baby radiator for diagnosis.

I found some brazed flat plate exchangers that with good flow 2gpm can handle 2KW+ with 10C rise over cool side inlet temp.  Remember the AC load on a single 5970s (GPU only) is ~230W so the DC (thermal) load is closer to 200W.  I bought one and will test it out this weekend.  I am thinking of a canister high lift aquarium or pond pump for the "cold" (outer) loop which should keep flow rates high.

Third: Power. Is that PSU going to be good enough for continuous duty? Not sure what you run now, but perhaps it will be.

I think so.  Seasonic is solid and their customer support is great.  I paid for units w/ 5 year warranty so I will be using that.  If they start to fail I will need to think of alternate loading.

[stevegee58]

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

[DeathAndTaxes]

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

If I only convert this one rig the payback is <3 months due to me already having some of the components (sunk cost).  Beyond that it is harder to estimate because I am not sure how far I can push the cards yet and if I will find that changes to my plan need to be made.  Also remember these aren't new rigs but "rig conversions" to need to look at incremental cost and incremental benefit.

Still a rough estimate:
Each rig conversion costs ~$900 and will provide ~600 MH/s extra hashing power ($66 per month) and will also eliminate ~$45 per month in cooling cost so direct payback is ~8 months.  If I can push them beyond 3.2GH/s every 100MH/s provides another ~$9 per month in net revenue. I am hoping I can get a bulk discount at least on the 20 waterblocks so with lower cost and higher hashrate payback might be as good as ~7 months.  Overvolting is also an option.

So I would say absolute best case is ~6 months worst case is 9 months.

[eleuthria]

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power .

[DeathAndTaxes]

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power .

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

I have 60A @ 240V in a sub panel with 2 NEMA L6-30R outlets and a pair of APC AP9571 PDUs (48A derated).  I have loaded ~40A without issue.   Right now I only have gear for 6 complete rigs not a full rack of 10 and I don't have any plans to expand that.

[hmblm1245]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

[DeathAndTaxes]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

[rjk]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

[hmblm1245]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

[DeathAndTaxes]

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy (even heavier filled with water) and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and coolant pouring over $3000 in components.

The RM41300-FS81 is $120 so getting cheaper case would save some money.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.  LOLZ.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the (lots of weight) + (water) + (single tiny screw) = risk issue.

In the end I decided to pay the extra $60 or so and play it safe (probably oversafe).

[rjk]

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and water pouring over $3000 in components.

The RM41300-FS81 is $120 so getting cheaper case would save $70 or so per rig.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the risk of "lots of weight + tiny screw". 

In the end I decided to pay the extra $60 or so and play it safe.  Likely being over cautious but I likely will buy the same expensive case for the next 5 rigs too.

Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case

[DeathAndTaxes]

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.

[Inspector 2211]

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

>directly outside

How do you plan to do that?
Is it a BARN, with a overhanging roof and a huge open window?

Half a year ago I had eight tower cases perched on the windowsill of an office, four where the right windowpane opened and four where the left windowpane opened. I added external ducting and 12cm "exhaust fans" at the mosquito screen of the windowpanes.

Problem was, they were blue LED fans - the building owner drove by at night, went WTF? - and eventually asked me what the f*** I was doing. I gave an evasive answer ("research") and was evicted shortly thereafter with a 3-day notice.  

Now you know why I'm interested in BFL Singles and FLGA boards...  

[hmblm1245]

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.

I think with alot of 133+ cfm fans you can get the temps down. like this guy

http://www.ebay.com/itm/New-nVidia-Tesla-S870-6GB-Memory-GPU-Processing-Computing-Server-System-BitCoin-/220938944468?pt=LH_DefaultDomain_0&hash=item3370fcbfd4#ht_4992wt_1163

But of course those heat sinks are aligned with the air flow, but if you design some simple copper heat sinks and mount them to the 5970s this might work. It would be loud as hell with those 40mm fans but it would only take up 1U

[DeathAndTaxes]

Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

Yeah it would be one per rig but I will need 6 all together.  

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.

That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case

I looked and I couldn't find a store in US that carries it.   I called frozen cpu and they don't stock it because it is explictly not designed for water cooling.  The manufacturer puts in small print not designed for heatsinks over 1 pound.  WTF? What else would you use it for?  Cramming hot aircooled 5970s even closer together?  

How well do those water fittings fit together?

The fittings work well as long as you don't move the cards.
The SLI fitting is adjustable (necessary to get multiple cards installed) but there is a max limit.  If you pull the "end caps" too far apart the tube in the inside can slide past the double o-rings and leak.   You really need to stupidly pull them "far" apart (say 1" more than they are now). The risk is likely small inside a crammed case but I am a coward.

There are other "single piece" methods but they add a lot of cost.

[jake262144]

Some updated pics.
...
Thoughts, comments, suggestions?

She's a beauty, DAT.
Well done - and without a single blue led or cold cathode. Looking at the average watercooled rig I wouldn't deem omitting some sort of l337 light source possible

[rjk]

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.

Stay out, that's my territory

[Red Emerald]

/drool

[Turbor]

Good work. Looks clean and has this professional touch.

[jothan]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs

Regards,

That's what I figured when I ordered 6 units from you a few days ago. Assuming they get here within the "4 to 6 week" period, I will have a very nice high-hash, low-power and low noise upgrade to my mining rigs within a few weeks.

BTW, it's nice to see you on the forum, BFL-Engineer (Sonny, I presume).

Keep shippin' em' and I'll keep orderin' em as long as I get them within reasonable delays.

[Isepick]

I had considered going under water for my 10 Ghash/s, but since I have none of the blocks or rads, it was going to be way too expensive. Instead I went the ghetto route:





That's a 12", 1000 cfm vortex fan (pulling 350w). It is far cheaper and easier to suck the hot air outside. The whole thing (rack, fans, foamboard, tape, etc) took $500 and a weekend to build. I live in S. Florida and this is my solution for the summer. It used to get over 100F in that room with only 4 rigs, now it is maybe 5 degrees warmer than the rest of the house on a hot day with 7 rigs. This may or may not help you, and watercooling is a hell of a lot neater, but from a maximum profit point of view it is cheaper to move a large quantity of air over those cards than it is to water cool them. And yes, that fan is louder than anything any video card . Anyhow, just throwing this out there for ideas. Good luck.

[BFL-Engineer]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs

Regards,

That's what I figured when I ordered 6 units from you a few days ago. Assuming they get here within the "4 to 6 week" period, I will have a very nice high-hash, low-power and low noise upgrade to my mining rigs within a few weeks.

BTW, it's nice to see you on the forum, BFL-Engineer (Sonny, I presume).

Keep shippin' em' and I'll keep orderin' em as long as I get them within reasonable delays.

Sonny is in sales, I'm in engineering. We'll try our best to satisfy customers. Efforts are
under way to reduce shipping delay.


Regards,

[Brian DeLoach]

I had considered going under water for my 10 Ghash/s, but since I have none of the blocks or rads, it was going to be way too expensive. Instead I went the ghetto route:





That's a 12", 1000 cfm vortex fan (pulling 350w). It is far cheaper and easier to suck the hot air outside. The whole thing (rack, fans, foamboard, tape, etc) took $500 and a weekend to build. I live in S. Florida and this is my solution for the summer. It used to get over 100F in that room with only 4 rigs, now it is maybe 5 degrees warmer than the rest of the house on a hot day with 7 rigs. This may or may not help you, and watercooling is a hell of a lot neater, but from a maximum profit point of view it is cheaper to move a large quantity of air over those cards than it is to water cool them. And yes, that fan is louder than anything any video card . Anyhow, just throwing this out there for ideas. Good luck.

When did you build this? It looks unbelievably similar to mine.

[Isepick]

When did you build this? It looks unbelievably similar to mine.

3 or 4 weeks ago, and you were the direct inspiration for it. I meant to do a build thread about it, but got lazy. It works pretty good for me. I had wanted to relay my positive experience with this design to DAT in case he wanted to try a much cheaper route. As I said, the temperature the room this is in used to be unbearable (hot enough to melt candle wax!), now it is tolerable with even more rigs in it. WAF greatly increased since I pulled 2 rigs off the kitchen table and dropped the ambient temp in the living/dining room area by several degrees.

[DeathAndTaxes]

Isepick that is a nice method to force the waste heat outside  I am looking for quiet and I think that kind of setup exceeds the WAF boundary at the D&T hashing farm.  Plus I would like to see how much waste heat I can capture and use to heat hot water for the house.

If I did the math right using a heat exchanger before the cold water intake of the hot water heater, at 2gpm I could raise the water temp from 60F to 105F before it ever gets to the hot water heater.

[DeathAndTaxes]

An update.  

Rig has been solid for 36 hours now.  I have increased the clocks and get ~ 3.125 GH/s on 1120W 120VAC.  When moving it to 240V PDU wattage dropped to 1090W (2.86 MH/W).   Backing out the system non-GPU idle of 180W puts GPU load at ~230W per card giving GPU efficiency of 3.40 MH/W.

If we assume ~10% PSU inefficiency that's 980W DC putting PSU load at 78%.  Since it is a SeaSonic I am comfortable with it.  The SeaSonic 1250 hasn't disappointed.  Exhaust is warm but not excessively even with the cramped airflow of a rackmounted case.

I have been running cgminer with auto-gpu enabled and a target temp of 62C.   The dynamic is kinda interesting.  The cards are rock solid but as one card goes over 62C it gets the downclocked by cgminer.   The reduced thermal load cools all the GPUs and puts one of them under the target so cgminer raises the clock and and the process repeats.  Kinda a "musical clocks".  The load remains ~1.1KW and hashrate ~3.1GH/s the entire time with different GPUs taking the lead.

The data is showing (which I already knew) that this radiator is simply not designed for a heat load this large.  The fact that I have low RPM fans isn't helping.  I won't be able to push clocks higher without better thermal dispersion but based on the solid performance I think 3.25 GH/s is possible at stock voltage.  I will try using some Scythe Ultra Kaze temporarily to see if I can get the last little bit out of the radiator but we might be reaching the limit of what 4x120mm radiator can dissipate..

Somewhat disapointing is that one GPU won't go above 800 MHz even with low temps.  It just goes instantly sick.  Sadly raising voltage in Linux isn't possible without BIOS flash.  Hopefully cores like that are a minority.

[Isepick]

Isepick that is a good concept to force the waste heat outside  I am looking for quiet and I think that kind of setup exceeds the WAF boundary at the D&T hashing farm.

ROFL. I understand, although when the door to the room is closed you can't really hear it so much, and when it is open its more of a white noise type of deal. And I have to credit Mr. DeLoach for the design, as I probably wouldn't have thought of it on my own . Anyhow, good luck, I am looking forward to seeing where you end up with all this.

[sveetsnelda]

When did you build this? It looks unbelievably similar to mine.

Then both of you need to clean up your rigs.  Sheesh.     Okay, I don't know about Brian's, but what is going on in that picture?  There are fans blowing at the *exhaust* side of quite a few cards!  The fans are just fighting each other..

This is what halfway clean rigs look like.  Pink rope is a requirement:

   

[jamesg]

This is what halfway clean rigs look like.

Looks pretty clean. I think my cases are a tiny bit sturdier than yours though. I have also removed every extra fan out of my setup by undervolting a few rigs. Maybe I'll make a video.

Now back to our regularly scheduled thread topic.

[Brian DeLoach]

3 or 4 weeks ago, and you were the direct inspiration for it.

I'm very happy to see the design replicated!

I kept thinking how almost every detail was spot on, it would have been a huge coincidence if you never saw my rigs. Why is that vortex fan green? As far as I know the company only makes them in gray.

Then both of you need to clean up your rigs.  Sheesh.     Okay, I don't know about Brian's, but what is going on in that picture?  There are fans blowing at the *exhaust* side of quite a few cards!  The fans are just fighting each other..

I actually thought a lot about this. Just through touch I could feel that each end of the card was exhausting heat, even more so for the power cable side (away from the interface). I, like you, always thought of the interface side to be the exhaust side. I even took a smoke machine and introduced some smoke at the fan intake to see where it went. It was about an equal distribution to both sides. Couple that with the fact that I wanted access to the interface side, along with the USB and other components on the motherboard, I decided to suck the heat away from the motherboard/gpu interface.

From the photos, you can see that I sandwiched the GPUs as closely as I could. I did this to maximize the airflow across the GPUs and increase the rate at which fresh air was introduced to the fans.

This is what halfway clean rigs look like.  Pink rope is a requirement:

Your setup looks very clean, but doesn't deal with exhaust heat at all. Isepick and I are down south and heat will become a major problem in a few short months. You also use many fans. I think using one big powerful fan to deal with all the heat is the way to go.

Now back to our regularly scheduled thread topic.

Starting now.

[sveetsnelda]

Looks pretty clean. I think my cases are a tiny bit sturdier than yours though. I have also removed every extra fan out of my setup by undervolting a few rigs. Maybe I'll make a video.

Yeah, you've certainly got a sturdier rack built.  If we both had an earthquake, you'd win.  We don't really get any here though. 
I'm going for cheap & effective, and my time is a factor.  Setting up a new rig takes me about 20 minutes from unboxing to mining.

Now back to our regularly scheduled thread topic.

  It wasn't meant to be a tread hijack.  I just saw someone with blowing fans at the exhaust side of their cards and couldn't help myself.  I had a non-ref card die a while ago and used a reference card in its place for about a week while waiting for a replacement.  Because the card had to face the wrong direction in relation to the others, that GPU fan had to REALLY work hard to remove the air.  Flipping the card around would have *easily* cut the fan speed in half.

[sveetsnelda]

I actually thought a lot about this. Just through touch I could feel that each end of the card was exhausting heat, even more so for the power cable side (away from the interface).

If you've got reference cards exhausting heat through the inlet side, something is horribly wrong.  There shouldn't be any.

I, like you, always thought of the interface side to be the exhaust side.

Er...  that's because it IS the exhaust side (on reference boards...  6990 is a different story). 

I even took a smoke machine and introduced some smoke at the fan intake to see where it went. It was about an equal distribution to both sides.

Sounds like you're using non-reference cards?

I decided to suck the heat away from the motherboard/gpu interface.

That's exactly what you should be doing.  It looks like your fans are blowing at the gpu interface side though?

Your setup looks very clean, but doesn't deal with exhaust heat at all. Isepick and I are down south and heat will become a major problem in a few short months. You also use many fans. I think using one big powerful fan to deal with all the heat is the way to go.

My setup deals with the exhaust heat perfectly.  My hottest card as I type is 74 degrees and the fan is sitting at 40 percent.  This is because it's in the garage.  The non-reference cards in the house are running around 64.0C at only 1166RPM.

The exhaust heat from all of the cards gets pushed out into the hallway and back out the window of an adjacent room.  Since it goes through the hallway, some of the heat gets recirculated throughout the house.  I haven't used the furnace at all this winter (and it gets very cold here).  This type of setup requires that nobody opens a window somewhere else in the house though...  otherwise, the air will go in a different direction.

My shelves are sitting on sliders, so when summer comes around, I just flip the rack around and exhaust the heat out the window.  It gets in the high 90's here and I was cooling the whole rack with a 230 watt evaporative cooler last summer.  The rigs in the garage will be a problem though if I don't move them before summer hits.  I'm building a small shop though to house them all.

Starting now.

Starting...   now?   

[check_status]

@DAT
You can reduce your cooling cost by using the earth as a heat sink and temperatures will always be in the high 50's or low 60's.

The "Earth Coupled Air Tube" technology is a low cost solution to reduce the cost of heating and cooling your home. By using the Earth as a heat sink anyone can heat and cool their house for less. Unlike the ground loop heat pump, air tubes don't require deep wells, compressors, pumps, Storage tanks, coils, heat exchangers, complex plumbing, or all the problems inherent in a complex equipment/technology intensive heating and cooling system. Properly installed air tubes don't have any moving parts. They can't break. The only technology required is a fan to move the air through the tubes and into the house. Earth tubes are relatively inexpensive to install and are inexpensive to operate.

http://earthairtubes.com/

Your cost of operating this is the cost of running a fan. sweet!

[DeathAndTaxes]

You can reduce your cooling cost by using the earth as a heat sink and temperatures will always be in the high 50's or low 60's.

http://earthairtubes.com/

Your cost of operating this is the cost of running a fan. sweet!

That is a little beyond my abilities in terms of earth moving and engineering but a pretty awesome concept.  If I had more land I might look into that for my entire house.

[jjiimm_64]

gota love the 5970's..  this one has been up since the 4th, restarted it then for a cgminer upgrade.

Code:

cgminer version 2.2.1 - Started: [February 4, 2012, 4:21 pm]    Rig: miner10
(5s):2928.27  (avg): 2929.3 Mh/s  |    Q:1586723   A:1491382   R:14116   HW:0   E:?%   U:40.08/m
TQ:?   ST:2979   SS:?   DW:42517   NB:3888   LW:61507   GF:1667   RF:2651
Connected to http://gpumax.com:8332 with LP as user ?

GPU 0:	74.0C	3591RPM 	60% | 	366.2/362.4Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:185253	R:1630	HW:0	U:4.98/m 	I: 7
GPU 1:	70.5C	3591RPM 	60% | 	366.1/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186079	R:1733	HW:0	U:5.00/m 	I: 7
GPU 2:	73.5C	3290RPM 	54% | 	366.1/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186059	R:1759	HW:0	U:5.00/m 	I: 7
GPU 3:	73.0C	3290RPM 	54% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186431	R:1732	HW:0	U:5.01/m 	I: 7
GPU 4:	72.0C	2947RPM 	49% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186842	R:1784	HW:0	U:5.02/m 	I: 7
GPU 5:	72.5C	2947RPM 	49% | 	366.0/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:187012	R:1753	HW:0	U:5.03/m 	I: 7
GPU 6:	73.0C	3133RPM 	56% | 	365.9/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:186618	R:1829	HW:0	U:5.02/m 	I: 7
GPU 7:	68.5C	3133RPM 	56% | 	365.9/366.7Mh/s | 	99% |	800Mhz 	300Mhz 	1.05V 	A:187088	R:1896	HW:0	U:5.03/m 	I: 7

[DeathAndTaxes]

Is that a customized version of cgminer?  The frequency & voltage columns.  I like that layout.    I think current layout in cgminer needs some work.  The [g]pu menu takes up too much vertical space especially w/ 8 GPUs.

[jjiimm_64]

Is that a customized version of cgminer?  The frequency & voltage columns.  I like that layout.    I think current layout in cgminer needs some work.  The [g]pu menu takes up too much vertical space especially w/ 8 GPUs.

webmon:

[DeathAndTaxes]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger



Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer. 

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

[rjk]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer. 

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

[DeathAndTaxes]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer.  

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

Will do.  I have a crimping tool from some home remodeling work already.  I will likely make a test loop involving just pump, radiator, and heat exchanger and leave it running.  The reason I would like to use PEX is I could take an off the shelf radiant heating manifold and use that for splitting the run to each rig.  

[ThiagoCMC]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

[DeathAndTaxes]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

Yup that is the one. 

[DeathAndTaxes]

Controlling a farm from single server.

Most farms are run as a set of independent rigs.  With watercooling and a shared outer loop that presents some challenges and risks.

The farm needs to shutdown in the event of a LOCA (to borrow reactor terminology http://en.wikipedia.org/wiki/Loss-of-coolant_accident).   Now including software on the rig which shuts down the rig when its temp rises is possible but not a solid failsafe.  Rising rig temps can cause processes to hang which could keep rig online cooking itself without cooling.  Having each rig monitor sensors creates a lot of duplication.

So my plan is to have a daemon on each rig monitoring GPU temps and if they rise above a failsafe to start a shutdown of the rig.

Backing that up a non-mining server "watchdog1" will provide analog monitoring of the entire farm.  It will also hold copies of all rig config files, collect stats from cgminer API, host p2pool & bitcoind daemons (and possibly act as a PXE server).

I plan to have:
4 outside temperature sensors (water before radiator, water after radiator, air intake, air exhaust)
1 ambient air sensor
1 flow sensor
1 reservoir level sensor (contact only)
1 pump current sensor

Now when watchdog detects something "abnormal" it could send a "emergency power off command" to each rig but we run into the same risk if a rig fails to shutdown.  So instead I am building a power control board which will allow the watchdog to power on, power off or power cycle all the rigs.


Multiple "dumb" quad 5970 water cooled rigs (initially 6).
&
"Watchdog" supervisor to keep them safe.

[shakaru]

D&T,

 You bought this 4U rack case here:

 http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm ?

Thanks!
Thiago

Yup that is the one. 

Not to tread-jack to much. But I have 2 of these (more in storage) for sale for WAAAAAY less than that price.

[DeathAndTaxes]

Not to tread-jack to much. But I have 2 of these (more in storage) for sale for WAAAAAY less than that price.

Not a thread jack at all.  If it is that exact one (RM41300-fs81) I am interested, PM me. 

Chenbro makes other RM41300 models but they only have 7 expansion slots.

[JinTu]

I have been contemplating water cooling for my 4RU 19" rack mounted miners as well. Noise is a big issue for me due to the WAF, so I have been looking at passive radiators like these. They take up a lot more room, but are totally silent. My current thought is to mount an array of these to the side of the 19" rack so it doesn't take up additional space.

[DeathAndTaxes]

I have been contemplating water cooling for my 4RU 19" rack mounted miners as well. Noise is a big issue for me due to the WAF, so I have been looking at passive radiators like these. They take up a lot more room, but are totally silent. My current thought is to mount an array of these to the side of the 19" rack so it doesn't take up additional space.

That is a quiet option. Even a "normal" radiator is very quiet.  My "test radiator" has 4 low speed 120mm fans.  It is really no more noisy than a power supply (a quiet "woosh" instead of any buzz of whine from high RPM fans).  Even a small amount of airflow can significantly improve the cooling power of a radiator.   4x120mm radiator is insufficient for 4x5970s. 

My goal though it to dump the heat outside thus saving all the AC costs.

[check_status]

So I think I found my radiator.

http://www.brazetek.com/products/details/46/15/finned-coil-water-to-air-heat-exchangers/16x16-finned-coil-water-to-air-heat-exchanger

http://www.brazetek.com/files/imagecache/product/wahx_3.jpg

Should have roughly 2x the surface area as the average 12" x 12" oil cooler.   Likely I will use some pex tubing with crimp connectors and a radiant heating circulation pump for the outer loop.  I sent an email to the company asking for some more details but comparing it against other radiators and doing some back of napkin heat flow guestimates I think it should be able to dump 7KW to 8 KW with a 20C rise over ambient.  That would be ~55C water temps (maybe GPU 2 to 3 C higher) in peak of summer.  

For fans I am going to experiment with some cheap electric car radiator fans maybe 2x 7" or 1x 12".  I would prefer 2 fans as it provides some slow fail redundancy.  I only need about 1000 to 1200 cfm of airflow (unless my math is way off) and radiator fans tend to move a lot more.  I am going to see if they can be undervolted to spin slower with less noise.

Let me know how you get along with PEX. It is really stiff, and I would be interested to know how well the press-on fittings hold at low water pressures. Works great for house water at high pressure, since that's what it is meant for.

Will do.  I have a crimping tool from some home remodeling work already.  I will likely make a test loop involving just pump, radiator, and heat exchanger and leave it running.  The reason I would like to use PEX is I could take an off the shelf radiant heating manifold and use that for splitting the run to each rig.  

That radiator is rad. Same price range as units that are sold for PC's but the advantage of dumping more heat. Schweet!

With PEX you can get a 3/8" inside diameter hose on to a 1/2" diameter PEX fitting (not compression). Fit should be very tight.

[The-Real-Link]

Very nice job, Death!  While I don't have a giant rack for my rigs, it's cool to see that those can be used.  I've not done water cooling before though so yeah while it is tempting for even lower temps, that's a lot of extra parts for even a handful of 5970s. 

[DeathAndTaxes]

An update (if the updates seem a little rambling they are.  Just trying to get my thought down on paper as I go along).

Heat Exchanger and coolant lines:
So I bought the radiator water to air heat exchanger.  I talked on the phone with supplier, a nice guy who answered all my questions despite me using the HE in a non conventional manner.  It is designed for transferring heat from a wood burning furnace to air handler.





I also picked up some rolls of 3/8" (branch line) and 1" PEX (mainline).  I am still deciding on the pump.  I need a pretty "beefy" pump.  Something which can push 8-12gpm with 10 feet of head.  My first thought was aquarium pump but they are expensive and seemingly under powered.  Next I thought about radiant heating pumps but I would need a stainless steel model and they aren't cheap.  Someone recommended Iwaki water pumps.



Seems interesting.  The smaller ones are used by extreme watercoolers but I need more extreme than extreme.  Nice thing about these is the build quality is second to none.  I would be confident with it running 24/7.  Any thoughts on a pump?

Changes to the plan:
After some thinking I decided to modify my plan a little.  Rather than use a flat plate heat exchanger (water to water) for each rig I am going to run the coolant in one giant loop.  The main reason is cost.  Having a closed loop system in each rig transferring the heat but not water (via heat exchanger) to an outer loop is nice but expensive.  We are talking $200 per rig for internal pump, reservoir, and heat exchanger.  6 rigs = $1200.  Removing that significantly cuts the cost with a little more work when adding and removing a rig.  


So new design will be:
heat exchanger -> 1" PEX -> pump -> 1" PEX -> 8 cold line port manifold -> 3/8" PEX branch lines in parallel -> each of the 6 rigs -> 3/8" PEX branch lines ->  8 port hot line manifold -> back to heat exchanger.

Each rig will connect to the cold and hot side manifolds with no-spill quick disconnects and flexible tubing (likely Tygon).

Code:

                        3/8" PEX branch lines
from heat-exchanger
                         |--------{rig #1}--------| 
1" PEX cold line         |                        | 1" PEX hot line 
-------------------------|--------{rig #2}--------|-----------------    to pump/reservoir/heat-exchanger   ---> 
                         |                        |
                         |--------{rig #3}--------| 

          cold side manifold                hot side manifold

                -----(flow direction)--------->

Only 3 rigs show for simplicity.  Total of 6 rigs will be connected.

Bleeding the system will be a little more challenging but $1200 in cost savings makes it worthwhile.  Having a large (say 1 gallon+) reservoir higher than the highest rig.  That should allow water pressure to aid in bleeding.  Possibly 2 reservoirs?

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

[rjk]

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

There was a folder on another forum that used a 55 gallon plastic barrel, and a saltwater-rated aquarium pump for longevity. EDIT: Link to forum thread
Do you have your manifolds yet? Here is a good one:


http://www.ebay.com/itm/190550567439

[johnyj]

better places to put the mining rig:

1. In mountains of Tibet
2. water radiator sunk into a lake, or even better a river

[stevegee58]

During the winter I just position my PC next to a slightly open window.  Works great.

[nedbert9]

Impressive job, DandT.  Yeah, get that heat outside.

[tonto]

Jesus.  This idea is just full of win.
 
It's making my idea to water-cool my 7970 through my aquarium seem like weak sauce. 
 
I haven't done it yet, 'cause it's only 55gal aquarium, and I'm still unsure if the heat exchange in my basement would dissipate that heat fast enough before creating fish-soup. 
 
Should I come across a nice sum of money that my wife doesn't find out about, I think I'm going to steal your idea, DeathAndTaxes.

[DeathAndTaxes]

I don't own an RV but this looks like a nice resivour to me

http://www.tank-depot.com/productdetails.aspx?part=85-1859WH



Turn it so the fittings are on top.  
1" in,  1" out, 1/4" vent line, and 1/4" for installing a float sensor.  

I was going to make a cube out of plexi, bond the edges, and tap threads into it but man it is like they made this for extreme watercooling.

[DeathAndTaxes]

An update:

Talking w/ some watercooling guys I am deciding on a different pump.



Likely an Iwaki MD series pump.  These are some monster pumps and have a good rep for requirements of large water cooling projects.  Likely I will need a model larger than what most people consider.  I am thinking of the Iwaki WMD-40RLT.

[rjk]

An update:

Talking w/ some watercooling guys I am deciding on a different pump.

http://i11.photobucket.com/albums/a154/eastcoasthandle/WATER%20COOLING/Iwaki_MD30RT.jpg

Likely an Iwaki MD series pump.  These are some monster pumps and have a good rep for requirements of large water cooling projects.  Likely I will need a model larger than what most people consider.  I am thinking of the Iwaki WMD-40RLT.

You might also consider a bore pump for use in submersible well applications. However that might have a head pressure that is too great.

[DeathAndTaxes]

Nobody makes 1 gallon or larger watercooling reservoirs so anyone have any ideas for constructing one? or re-purposing some other gear?   It should have 1" NPT fitting to allow connection to the PEX mainline.

There was a folder on another forum that used a 55 gallon plastic barrel, and a saltwater-rated aquarium pump for longevity. EDIT: Link to forum thread
Do you have your manifolds yet? Here is a good one:


http://www.ebay.com/itm/190550567439

I haven't found a good cheap manifold yet.  

The problem is it is aluminum.  copper + aluminum in same loop of circulating water =

If they made that exact item in copper it would be perfect.  

I have been looking for something which is copper with 1" supply and 6 to 10 branches outlets w/ 3/8" or 1/2" npt threads that doesn't cost a fortune.  That doesn't seem to exist.  Lots of radiant heating manifolds but they come with high end pex connectors and cost a fortune.  I may have to break out the torch and make my own.  

[P_Shep]

How do you make sure flow though each of the splits is the same? Or rather, that one or more isn't so slow it causes problems?

[DeathAndTaxes]

How do you make sure flow though each of the splits is the same? Or rather, that one or more isn't so slow it causes problems?

Each split has the same resistance because it uses the same components.  That keeps the flow roughly the same though each split.  You couldn't however have 4x5970 in one rig and 3x7970 is another rig.  As long as all the "downstream components" are identical resistance should be roughly the same.  As water velocity increases the back pressure increases and that prevents one loop from flowing faster than another one.

If you did have dissimilar branches you could use a "booster pump" on each branch to boost flows to the one with higher resistance.  As an alternative you could use something called a balancing valve which increases pressure to the lower pressure segment to balance the flows.  I hope to avoid that though (more cost, more complexity).

[PulsedMedia]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

[shakaru]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

[cuz0882]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.
Radiators in standard cars used to be copper! Sometimes copper head gasket is used etc.
Newer cars tend to use aluminium cores for radiators due to their lower price and weight.

Just add high quality coolant or in other words: Glycol for protection. The red stuff is highest quality. You need this in any case, unless you want your loop to gather all kinds of dirt, bacteria etc.
It also prevents corrosion.

In this utilization even 1:20 mixture is probably more than sufficient and it's cheap.

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

[PulsedMedia]

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Useless to spent all that time researching, pretty much use any you feel comfortable, or which gives the nicest color in your senses
Forget distilled water, useless waste of money & time for cooling purposes.

Just add at least 1%, but don't exceed 50%.

I'm not a fan of watercooling computers, too much work to maintain and hassle in regular desktops. For servers it makes sense tho.
I recommend the red glycol (i keep forgetting the names lol) because it has the highest boiling point and best protective for modern engines, but for computer watercooling any automotive glycol will work just as fine.

[PulsedMedia]

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

Yet cars have been utilizing both metals mixed in the same system, in the same cooling system for decades upon decades. Those work wonderfully, and the coolant channels in cylinder heads are WAY smaller than your CPU block has, often a diameter of just 5mm in many parts!

So now you are using something which is a strange physical phenomenon known by some sailors to happen in sea?

Coolant, proper glycol, is to be used aswell to minimize the build up caused by bacteria etc.
Some glycols themselves will build up, and still causes dirt to apper in the coolant, hence, the coolant ought to be swapped annually, maybe accompanied by a flushing the system.

Aluminium does oxidize, but it doesn't matter at all in a cooling system, especially if you use red glycol which has properties to protect from that.

Might be a worthwhile read: http://www.gewater.com/handbook/cooling_water_systems/ch_32_closed.jsp
Speaks in length about corrosion in cooling systems.

All you really need is a bottle of cheap glycol and tap water, and annually change the coolant.

[cuz0882]

Copper should not be used with Aluminum it causes Galvanic corrosion, don't use Aluminum with anything actually.
I tried EK coolant the first time, I was worried about leaks. About 3 months down the road my cpu block got plugged with little fibers, was a major pain figuring out what caused the flow to drop. I'm not sure if it was the fluid or not. It definitely leaves residue on everything that it touches. Now I use distilled water, along with silver coils and a few drops of dead water. Seems to be working great. Here is a simple guide: http://www.overclock.net/t/913181/water-cooling-guide-for-noobs#post11984918

Yet cars have been utilizing both metals mixed in the same system, in the same cooling system for decades upon decades. Those work wonderfully, and the coolant channels in cylinder heads are WAY smaller than your CPU block has, often a diameter of just 5mm in many parts!

So now you are using something which is a strange physical phenomenon known by some sailors to happen in sea?

Coolant, proper glycol, is to be used aswell to minimize the build up caused by bacteria etc.
Some glycols themselves will build up, and still causes dirt to apper in the coolant, hence, the coolant ought to be swapped annually, maybe accompanied by a flushing the system.

Aluminium does oxidize, but it doesn't matter at all in a cooling system, especially if you use red glycol which has properties to protect from that.

Might be a worthwhile read: http://www.gewater.com/handbook/cooling_water_systems/ch_32_closed.jsp
Speaks in length about corrosion in cooling systems.

All you really need is a bottle of cheap glycol and tap water, and annually change the coolant.

Ya, that sounds like a great idea! Maybe I will just put some stop-leak in my pc if a seal goes bad, or maybe even a little quicksilver. What kind of redneck trailor park crap are you talking about? Have you even looked inside of a cpu block? There is a reason why everyone is not using 50/50 antifreeze mix in there pc's. What you said is a clear example of what not to do.. Look it up.

[PulsedMedia]

Dude, obviously you do not have a clue about the underlying physics going on.
FYI, i've even made blocks myself BEFORE watercooling became anywhere near so popular that you could buy anything watercooling related. Back then i were already using compressors to drive -34C coolant to the CPU. Did you know that you can get by just fine with low pressure, low volume pump (cheapest 12V you can find) if your block is properly massive and TDP is below 100W?

Btw, "stop-leak" and other stuff you put in to stop leaks is the stupidest idea ever.

Do you know what is the coolant of choice of extreme high output racing engines which actually partake in sanctioned races?
Now i'm talking about engines like 2L Inline 4 boosted to some 4bar, Cummins 5-8L diesel boosted to 7+bar?
Nope, not distilled water.

TAP WATER, with maybe some water wetter.
And these machines need to be capable of removing heat in the multiple MEGAWATT range.
Are you telling me water is not sufficient to remove your couple hundred watts of heat?

As for the dirt accumulation, i've only seen it once, and it was with off the shelf system with which came the "water additive", very small amount to be used, i was shocked to see that accumulation and i somehow doubt that would happen if proper glycol was being used. I suspect the additive that came with it was just some coloring.

The thing is with water if you add a little bit of heat, but not much, and circulate it constantly, it WILL have bacterial growth, that's why you need to add enough glycol.

Still, WTF is dead water? Other than a sea water phenomenon known to few sailors, which might just be a myth as likely.

[DeathAndTaxes]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.

I didn't mean to imply it was impossible however I intend to run a pure distilled water loop for maximum efficiency.  Watercooling is complicated enough adding mixed metals is adding extra optional complication.  For 99% of users it is simpler/easier to just skip the aluminum parts go all similar metals and bypass the entire issue. 


I have no interest in using antifreeze.
1) It is a pain.
2) It has to be regularly changed
3) Corrosion will occur if it isn't changed or concentration isn't sufficient.
4) It has much lower heat transfer capabilities compared to distilled water which lowers the efficiency of entire system requiring stronger pumps, larger heat exchangers, and faster/louder fans.
5) I will be using a heat exchanger to heat up potable water for human consumption which will require the more expensive non-toxic propylene glycol and corrosion resistance is significantly reduced.

So hopefully to not derail this futher:
a) yes you "can" use anti-freeze.
b) no I don't recommend it because today it is too easy to build a water cooled system with no mixed metals.

[DeathAndTaxes]

You know, I spent to much time researching mixtures. One site started that 3/4 Ethealyne-glycol 1/4 distilled h20. Is there a mix you would recommend for effiency ?

Yuck

Simple answer is to not used mixed metals and then you can simply use pure distilled water (plus silver coil as a natural biocide).  Too easy.

If you must ignore the simple answer then remember that glycol has a lower heat transfer than water.  75% glycol is just stupid.  

More glycol means:
a) lower heat transfer (and higher pump resistance) - undesirable.
a) more corrosion resistance - unless you intend to not change loop for a decade even 5% should be fine.
b) higher boiling point - nope doesn't really apply here.  If you loop is getting to >100C you got problems.
c) lower freezing point - unless the radiator is outside it is a non-issue.  The container will tell you the freeze protection at various strengths.

so use the lowest amount that meets your needs.  

[cuz0882]

Dude, obviously you do not have a clue about the underlying physics going on.
FYI, i've even made blocks myself BEFORE watercooling became anywhere near so popular that you could buy anything watercooling related. Back then i were already using compressors to drive -34C coolant to the CPU. Did you know that you can get by just fine with low pressure, low volume pump (cheapest 12V you can find) if your block is properly massive and TDP is below 100W?

Btw, "stop-leak" and other stuff you put in to stop leaks is the stupidest idea ever.

Do you know what is the coolant of choice of extreme high output racing engines which actually partake in sanctioned races?
Now i'm talking about engines like 2L Inline 4 boosted to some 4bar, Cummins 5-8L diesel boosted to 7+bar?
Nope, not distilled water.

TAP WATER, with maybe some water wetter.
And these machines need to be capable of removing heat in the multiple MEGAWATT range.
Are you telling me water is not sufficient to remove your couple hundred watts of heat?

As for the dirt accumulation, i've only seen it once, and it was with off the shelf system with which came the "water additive", very small amount to be used, i was shocked to see that accumulation and i somehow doubt that would happen if proper glycol was being used. I suspect the additive that came with it was just some coloring.

The thing is with water if you add a little bit of heat, but not much, and circulate it constantly, it WILL have bacterial growth, that's why you need to add enough glycol.

Still, WTF is dead water? Other than a sea water phenomenon known to few sailors, which might just be a myth as likely.

I'm not interested in race car water cooling. You don't need glycol to keep bacteria out of the loop. Dead water is just a biocide additive that keeps bacteria out of the loop similar to silver. You can find it about anywhere that sells water cooling parts. http://www.frozencpu.com/products/11744/ex-liq-154/IandH_Dead-Water_Copper_Sulfate_Biocidal_PC_Coolant_Additive.html#blank

[PulsedMedia]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

[Gomeler]

Copper + Aluminium in the same loop is not a fatal problem. Those are combined in automotive applications for decades.

I didn't mean to imply it was impossible however I intend to run a pure distilled water loop for maximum efficiency.  Watercooling is complicated enough adding mixed metals is adding extra optional complication.  For 99% of users it is simpler/easier to just skip the aluminum parts go all similar metals and bypass the entire issue. 


I have no interest in using antifreeze.
1) It is a pain.
2) It has to be regularly changed
3) Corrosion will occur if it isn't changed or concentration isn't sufficient.
4) It has much lower heat transfer capabilities compared to distilled water which lowers the efficiency of entire system requiring stronger pumps, larger heat exchangers, and faster/louder fans.
5) I will be using a heat exchanger to heat up potable water for human consumption which will require the more expensive non-toxic propylene glycol and corrosion resistance is significantly reduced.

So hopefully to not derail this futher:
a) yes you "can" use anti-freeze.
b) no I don't recommend it because today it is too easy to build a water cooled system with no mixed metals.

Whew, there is a lot of FUD in this thread but it appears you at least have a solid plan. Mostly curious if you'll try to harvest the waste heat and how you'll go about doing it. I know you mentioned pre-heating your waterheater, going to stick around to see what you do. I myself built out a small farm of 8 watercooled 5970s. Hopefully you don't buy up all the DangerDen 5970 waterblocks!

[DeathAndTaxes]

Mostly curious if you'll try to harvest the waste heat and how you'll go about doing it. I know you mentioned pre-heating your waterheater, going to stick around to see what you do. I myself built out a small farm of 8 watercooled 5970s. Hopefully you don't buy up all the DangerDen 5970 waterblocks!

My heat exchangers arrived today.    My wife wants a night out so next update will need to wait till tomorrow.

[cuz0882]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

[PulsedMedia]

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

wow, i didn't know distilled water is just boiled! OMFG!! j/k

I've even built my own water blocks.
The commercial products i've seen mostly use 10-15mm hosing. Many of the water blocks has been very simple, hose in, hose out, just a big water space inside the block, and some had ribs to have bigger surface area.


Good quality water pump is a must, i've never said you should skimp on water pump. Not once. Not even hinted.

No matter how you put it, 800-1100watts is still a small heat load relatively, and one definitively shouldn't over complicate things. Keep It Simple, Stupid! or K.I.S.S.

I can see the point in adding the silver coil, or even "dead water" or other chemical to kill bacteria, but i don't see a point in over complicating simple things. None of the commercial single system water cooling package i've had the displeasure to work with had nothing like this. One had so bad additive it kept on accumulating in corners and i had to spend significant amount of time running hot water through it to get that stuff out.

Use copper blocks, use brass fittings if you go with standard hydraulics fittings (ie. NPT, BSP) hell maybe even AN-fittings (aluminium btw!) even tho that's total über overkill, or maybe just use basic hose barbs (1.5-3$ each) and clamps. Go with a high quality pump with a backup pump (2 parallel) so no need to monitor constantly. Use basic standard rubber hose, if you want to be cheap even garden hose will do the trick (yuck).
Add a little bit of glycol for protection of the metals.
and do *NOT* omit periodic maintenance (ie. once a year)

tho adding dead water and silver coil costs next to nothing vs. the other gear. So why not add them.

I must admit tho, i hate water cooling generally, makes maintenance a bitch in regular systems, but for servers (or bitcoin mining cluster) it makes tons of sense.

You should take a look at how industrial water cooling systems work ... They are remarkably simple by design. Even those cooling tens of thousands of servers.

I started fooling around with water cooling when Thunderbirds were a new thing (or maybe even earlier ...), didn't like it then for single systems, don't like it now. Tho a new corsair's set made it remarkably simple, infact so simple i might consider that for myself too, after i've seen how a system like that works later down the road (maintenance wise): http://www.corsair.com/cpu-cooling-kits/hydro-series-water-cooling-cpu-cooler/hydro-series-h100-extreme-performance-liquid-cpu-cooler.html
Just installed H100 for a customer

[ThiagoCMC]

Your 1250W PSU is handling this 4 x 5970 smoothly!?

[DeathAndTaxes]

Your 1250W PSU is handling this 4 x 5970 smoothly!?

Yes.  Remember PSU are rated on DC load not AC.  It is ~1100W AC.  At ~90% efficiency that is <1000W DC.  No problem for a SeaSonic 1250W unit.  Now lesser brands might choke up but any top tier supplier should be able to provide 100% power continually.

[Roadhog2k5]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

Except for that you're missing a step to distill water. Yes you boil it, but what you are collecting is the steam from what you are boiling, which then condenses back into water. You are not just using the water you have boiled directly. And yes, it is a good idea to use distilled water in a car just for that reason. A lot of times non distilled water has minerals in it that will leave buildup/corrode the cooling system components.

[cuz0882]

Point was, and still is, you don't really need all that fancy stuff, when basics are more than sufficient for relatively small heat loads.

No need to over complicate a very simple thing. All those little stuff fast adds to quite a bit of extra expense, never mind requirement for extra planning.

Hell, a huge european DC uses basicly tap water as a total loss coolant for TENS OF THOUSANDS of servers. Works just fine, most stable servers i've had the pleasure to work with.

There is nothing fancy about distilled water. It's basically just boiled water. It costs about 89 cents a gallon and any store. There is no point of putting it in a car since they heat the water enough to kill bacteria anyway. Clearly you have never even looked inside of a cpu block. Even the EK High flow block is a lot smaller then 5mm.
Water cooled mining rigs last longer, stay quiet and run cooler. My water cooled rigs pretty much never crash and it more then makes up for the extra time putting them together. Everything you have mentioned on this thread is backwards. We are not talking about water cooling 100w cpu blocks or race cars. A decent mining rig draws 800-1100 watts. Anyone with any sense is going to get a nice pump. You obviously don't have any interest in water cooling mining rigs or any experience doing it. I don't know why you are even acting like you know what you talking about, clearly you don't.

Except for that you're missing a step to distill water. Yes you boil it, but what you are collecting is the steam from what you are boiling, which then condenses back into water. You are not just using the water you have boiled directly. And yes, it is a good idea to use distilled water in a car just for that reason. A lot of times non distilled water has minerals in it that will leave buildup/corrode the cooling system components.

I didn't feel like elaborating further, minerals could be a real problem though if you had to keep filling your radiator because of a leak or something.

[ummas]

I`m having strange problem here with system 6GPU+
made 2 systems yesterday, and bouth got it.
GPU`s are shutting down randomly....
I was testing few OS`s: xubuntu, linuxcoin, W7(WTF? - i was desperat)
 

1 rig - 3x5970+5870  - OCZ ZX 1250 - all GPU down after 7h - i was sleeping
2 rig - 3x5870+2x5970 - ENERMAX 1250 + sommething 550W - 3 GPU down after 7H

Both rigs has 890FXA-GD70 and GPUs are watercooled.
Cheked watteg with KILL-O-WAT but only rig no.1 - takes 980W from the wall - 5870 - 950, 150 and 5970 - 750,150 - so not a big deal. temps - max 54. Cheked vrm tmp under W7 - each GPU - higher by 10*C so max ~65...cool and nice, so why down ??
any ideas ??

[phorensic]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

[DeathAndTaxes]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

Not to late.  Haven't pulled the trigger on the pump.  Was doing work on the heat exchanger and sweating npt connections this weekend.

Show me a pump with
a) similar curve as a Iwaki MD-40
b) has 1" NPT intake and discharge
c) has similar heat dump
d) has significantly lower price

and I am interested.  I have looked any haven't found it.

Everything I have found either has horrible head (most aquarium pumps) or is horribly inefficient (most sump dumps) dropping 300W to 500W of heat into the loop or is more expensive than the Iwaki.

If you found a good pump that can get similar flow as the Iwaki MD-40 at 10ft to 15ft of head let me know.

[DeathAndTaxes]

I`m having strange problem here with system 6GPU+
made 2 systems yesterday, and bouth got it.
GPU`s are shutting down randomly....
I was testing few OS`s: xubuntu, linuxcoin, W7(WTF? - i was desperat)
  

1 rig - 3x5970+5870  - OCZ ZX 1250 - all GPU down after 7h - i was sleeping
2 rig - 3x5870+2x5970 - ENERMAX 1250 + sommething 550W - 3 GPU down after 7H

Both rigs has 890FXA-GD70 and GPUs are watercooled.
Cheked watteg with KILL-O-WAT but only rig no.1 - takes 980W from the wall - 5870 - 950, 150 and 5970 - 750,150 - so not a big deal. temps - max 54. Cheked vrm tmp under W7 - each GPU - higher by 10*C so max ~65...cool and nice, so why down ??
any ideas ??

Entire rigs are crashed?  Can't SSH into them and see what happened?  That is weird.  I would turn on logging in cgminer and try to see what happened.  If you need help on logging in cgminer use the cgminer thread.  As a test I would take every card to stock and with a higher memclock of say 280.   Get them running 48 hours+ before you start playing around with overclocking.

My 4x5970 rackmount chassis has been rock solid.  Significantly overclocked it hasn't had any downtime since I started this thread.  No crashes, no failures, no noise just a continual supply of new shares.  The longer that test rigs runs solid, cool, and quiet the more convinced I am of this project despite other setbacks (like radiator arriving damaged, and trying to find a cheap manifold, etc).

[phorensic]

I know it's too late, but you spent way too much on a pump.  Iwaki is premium shit in the aquarium world.  There is a clone available for each Iwaki model made by Pan World.  Pan World was started by a former chief engineer from Iwaki.  They just make the same pumps essentially, at way better prices.  Also, I really doubt you had to go to a full external style pump.  Probably could have gotten away with a beefy submersed pump and put it in a sump, for like half the money.  The goal in the end is ROI, right?

Not to late.  Haven't pulled the trigger on the pump.  Was doing work on the heat exchanger and sweating npt connections this weekend.

Show me a pump with
a) similar curve as a Iwaki MD-40
b) has 1" NPT intake and discharge
c) has similar heat dump
d) has significantly lower price

and I am interested.  I have looked any haven't found it.

Everything I have found either has horrible head (most aquarium pumps) or is horribly inefficient (most sump dumps) dropping 300W to 500W of heat into the loop or is more expensive than the Iwaki.

If you found a good pump that can get similar flow as the Iwaki MD-40 at 10ft to 15ft of head let me know.

Oh cool, you are still deciding on the pump.

I'm going to be fairly blunt about this.  What makes you think you need to pump ~15ft of head at 450+ GPH?  Have you calculated your total head based on real height plus all your fittings and doo-dads?  There are calculators out there, btw.

Based on my experience with even shitty aquarium pumps, they are all underrated on what they will flow at their "supposed" maximum head height.  I think you would be surprised what a ~50-75W submersed pump can do even "choked" down to 1/2" tubing.  Don't get caught up too much in the system designs for water cooling in the overclocking computer world.  They tend to grossly oversize EVERYTHING in search of dropping that last 1 degree.  I was doing computer watercooling before you could just go out and buy a kit in a box.  Some of the setups the enthusiasts started making in the early 2000's up to now are downright hilarious on how oversized everything is.  Some of the more commercial application are more realistic, with way smaller tubing, pumps, and radiators.  I think if you shoot for 50-60C temps at the GPU itself, instead of 2C above ambient like some guys, you would be surprised how cheap your initial investment gets on the equipment, host fast you get your ROI, and how long everything would last.

If you live in a dry environment you could even look into evap cooling to dump the heat instead of a radiator.  You know, how nuclear power plants are cooled?  A while back people designed PVC "water bongs" to blow air up over water that was showering down into a sump.  I made what I called a "bucket bong" that did the same thing in a 5 gallon bucket, just a shower head and an 80mm fan.  The advantage?  Your water coming out of the "bong" was 1-2C *colder* than ambient.  The disadvantage?  You have a bit of humidity to worry about in the room you dump the heat in.  You could either plumb it outside, or just dump some of the air outside with another window fan again, lol.

[DeathAndTaxes]

I do believe I need 10gpm @ 15ft of hea.  For expansion and some margin of safety in flow rates (due to balancing) I would prefer 20gpm or as high as economically possible @ 15 feet of head.   Remember each rig is in parallel so 10gpm across the mainline is only 1.5gpm across the server if the minimum of 6 servers are used.  If that is expanded to 10 servers (~32 GH/s) for a full rack it is only 10 gpm mainline = 1 gpm per server.

I already used a flow/pressure calculator:
The server has 4.2ft of head loss (@ 2gpm) measured by me.
remember 8 GPUs worth of blocks in serial is rather restrictive this isn't your "normal" loop.

As I don't have a manifold yet I am reserving 1.0 ft of head loss for the manifold (and various fittings).
The heat exchanger has 2.2ft of head loss at 20gpm (thankfully it has 1" inlets and 8x 3/8" copper branches).
The mainline as planned is 70ft of 1" PEX.  At 20gpm that is ~8ft of head loss.

4.2 + 1.0 + 2.2 + 8 ~= 15ft.

So options:
Options 1) Could use a pair of radiators in parallel to reduce the flow across each and the corresponding head loss.  At 10 gpm the head loss is only ~ 1 ft. Still I doubt that saves much money when you consider the additional cost of 2nd radiator and fan assembly.  

Option 2) Use a less restrictive mainline.   1" PEX is just so cheap and easy to use.  Going to 1.25" CVPC is an option but a lot more work installing.  

Option 3) I could just accept lower flow across each rig but I worry that if I go that route flow isn't going to be perfectly balanced and one rig ends up getting too low of a flow.  That would require either putting in expensive balancing valves on the manifold or buying a larger pump.

Given those options I would rather not skimp on the pump and end up needing to waste time and money putting a second radiator in parallel, dropping in balancing valves, or re plumbing a larger mainline (1" PEX = $0.80 per foot.  1.5" PEX = $3 per foot ).  Going "cheap" on the pump may save $100 or so but it may end up costing three times that in extra work and modifications.

On evaporative cooler:
I don't think any home grown evaporative cooler is going to handle 7KW of heatload.  Regardless the summers in southern VA approach 90% humidity so I think a forced fan heat exchanger is going to be superior for year round operation.

I don't mind looking at alternative brands I just haven't found anything that gives me what I need for less.  No doubt it likely exists so if you have a pump model in mind I am all ears.  I have found some industrial pumps but they tend to be lower efficiency and dump as much as 500w into the loop which make them useless.

Your thoughts?

[DeathAndTaxes]

I think if you shoot for 50-60C temps at the GPU itself, instead of 2C above ambient like some guys, you would be surprised how cheap your initial investment gets on the equipment, host fast you get your ROI, and how long everything would last.

I am shooting for 55C to 60C temps.  Remember though this is nearly 7KW in heatload and outside temp peaks at 35C.  To keep 7KW of heatload at 25C over ambient requires more cfm, flow, and surface area than you may think.  I did some heat flow calculaitons and to avoid needing 2000 cfm+ requires pretty significant gpm on the liquid side.  Sure cooling in the winter is easier but we need to make it through August in Virginia Beach first.

On edit (found this looking for info on flow rates):
http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp?topic=/iphad/watercool9125-F2A.htm

Looks like IBM recommends 22gpm per server rack and 33 psi (~14 ft of head) of pressure.  Granted no exactly apples to apples (they are also using water chiller to lower input temp of the coolant to <20C) but thought it was interesting comparison.

[Roadhog2k5]

You should be looking at flow rate instead of static head pressure. Static head doesn't matter in a closed loop.

Look at the pressure drop (friction head) of each of your components to pick a proper pump.

EDIT: Also if you want a nice pump, get this. http://www.pexsupply.com/Grundfos-59896773-UPS15-55SFC-Stainless-Steel-3-Speed-Flanged-Circulator-Pump-1-12-HP-115-V

Have three of them for my radiant floor heating, one for each zone. They have no problem pushing through hundreds of feet of pex without skipping a beat.

[phorensic]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

[DeathAndTaxes]

You should be looking at flow rate instead of static head pressure. Static head doesn't matter in a closed loop.

Look at the pressure drop (friction head) of each of your components to pick a proper pump.

I am not sure what you are trying to say and I am not sure you do either? 

The flow rate at given resistance is what we are discussing.  I am not sure what you are advocating differently?

[DeathAndTaxes]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

[Roadhog2k5]

A 7kW heat exchanger doesn't need to be huge.  My go kart produces more heat and the radiator is tiny and doesn't even have a fan =P  jk, bad example.  Just FYI, a trick I learned on radiators in general is that the thicker they are the harder it is to push air through them to make them efficient.  You can have a heat exchanger that dumps 20kW and is only 12 sq. in., but is so thick you need a serious freakin' fan just to push enough air through it (or 100mph of wind hehe).  In your application you have all the space in the world to spread out the surface area, so a heat exchanger that is only 1/2" thick or whatever would be way easier to deal with.

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

Have you considered running two pumps in series? You would gain additional flow, and redundancy in case of pump failure.

[siggy]

To avoid going off topic do you have any recommendations on a pump?  I have a heat exchanger, baring a better recommendation I likely will be getting an Iwaki MD-40 or MD-55 pump.  You said Iwaki's costs to much I am interested in an alternative recommendation.

Do a search for "Panworld pump" .. they're the "inexpensive iwaki's" ..  I've got 2 of them running on my saltwater tank that have been running nonstop for 7 years now.

Sigg.

[phorensic]

Does this satisfy you need to have a massive ass pump? - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html  hehehe, but seriously the price is way better for the same quality.

Even though this doesn't meet your head requirements, I would really like to see what it would do in your actual finalized setup - http://www.marinedepot.com/Aquarium_Systems_Premium_Maxi_Jet_3000_Utility_Pump_775_gph_Up_to_500_Gallons_Per_Hour_Submersible_Water_Pumps-Marineland-AS1311-FIWPSBUF-AS1319-vi.html .  You could sell it to the aquarium nerds if it doesn't work =P  Take into account also that your plumbing on the pump gets reduced to just a pipe that dumps into the sump, and then a hose straight off the outlet of the pump...instead of all the fittings required to get an external hooked up properly, plus union valves to make it easier to service, etc.

[phorensic]

Oops, sorry, the 100PX is considered comparable to the MD40 - http://www.marinedepot.com/Pan_World_50PX_Magnetic_Water_Pump_Up_to_1000_Gallons_Per_Hour_External_Water_Pumps-Pan_World-JW1123-FIWPEPZT-JW1133-vi.html .  I was looking at flow curves instead of the feature briefs.

[DeathAndTaxes]

Even though this doesn't meet your head requirements, I would really like to see what it would do in your actual finalized setup - http://www.marinedepot.com/Aquarium_Systems_Premium_Maxi_Jet_3000_Utility_Pump_775_gph_Up_to_500_Gallons_Per_Hour_Submersible_Water_Pumps-Marineland-AS1311-FIWPSBUF-AS1319-vi.html .  You could sell it to the aquarium nerds if it doesn't work =P  Take into account also that your plumbing on the pump gets reduced to just a pipe that dumps into the sump, and then a hose straight off the outlet of the pump...instead of all the fittings required to get an external hooked up properly, plus union valves to make it easier to service, etc.

Even if we take the specs at face value it is 8 gpm at 4 ft of head.  No curve or max head is provided but I imagine that is the sweet spot.  Likely sub 4gpm at 8ft of head and 0gpm at <15 ft.  I don't even see that being in the same league of the kind of pump necessary to cool 8 servers in parallel w/ 8 GPU each in series, through 70ft of tubing to a massive radiator with 64 ft of 3/8" copper tubing. 

[DeathAndTaxes]

The 100PX is interesting.  In searches for Pan World the brand "BlueLine" keeps coming up.  Looks very similar to Iwaki.  

HD 40
http://www.aquacave.com/Blueline-40-HD-ExternalAquarium-Water-Pump-by-Pan-World-P488.aspx

HD 55
http://www.aquacave.com/Blueline-55-HD-External-Aquarium-Water-Pump-by-Pan-World-P490C762.aspx

Lots of merchant indicate Blueline is produced by Pan World but Pan world website has no reference no Blueline.



If these specs are to be believed then BlueLine might be the pump I need.

[DeathAndTaxes]

Finally pulled the trigger on a Pan World 150 PS.  Found one at a good price (cheaper than other places are selling 100 PS)



Looks like 12 gpm @ 14 feet of head.  Should handle the resistence of 1" PEX mainline just fine (saving me the cost and tedious work of assembling 70 ft of CPVC).  I am hoping it gets here my Friday but it might not arrive till next week.   Come on Fedex lottery.

[rjk]

Man it would be cool to have a 3 phase pump on a PID loop with a VFD. Then you could add and remove rigs with no need to worry about balancing anything out (for the most part). The pump would just slow down as more resistance is added (closing off valves to swap a rig, or whatever) or speed up if you add more rigs to the loop.

[phorensic]

OK, you found the other brand that is a clone of the Iwaki.  I knew from my saltwater aquarium days there were 2 Iwaki clones, one was Pan World and the other I couldn't think of for the life of me so I didn't list it.  I would consider a BlueLine a contender also.  They all share excellent the same engineering specs, with different people actually making them.  Fake edit: I see you bought a used PanWorld 150.  Badass!  As long as the price isn't outrageous it is still cool that you got an overkill pump.  And yes, rjk, VFD would be the ultimate luxury on a setup like this.  Totally unnecessary, but we are nerds, so every improvement is cool!

Should put one of these in the system: http://www.marinedepot.com/Digital_Aquatics_ReefKeeper_Lite_Controller_Multi_Item_Monitors_Controllers_for_Saltwater_Aquariums-Digital_Aquatics-DA1131-FITEMOMI-vi.html . I mean, wouldn't it be cool to monitor the pH, temp, and *salinity* of your *freshwater* cooling loop? Im just kidding man, seriously, just kidding.  I want to see this project grow.

[ummas]

Since my last post i discovered, that my sempron was unlocked to Atholon II and it maight be problem of the all GPU crush. seccond rig is stable after changing intensity to d (jumps 3-6) - aprox 20mhash loss :/ but stable for last 24h.

rig was simple to get on with ssh, but all gpus was down and i had to reeboot it. xorg 99.9% of cpu bug
thats why i was digging in the bios and i figured the unloced core.

thats haw i become 2x2.5Ghs 100% Watercooled rigs owner

I`m using 12V laing`s with modifed electrics - pulls 20-30W (depands on Voltage) and makes low noise and preaty nice flow.
My temps are aprox 51-55 with just 3x120Fans on old 3x120x2 (2 radiators puted toghether).
My system was build from scraches. I paid really low prize for it, for example: 1 block cost me 60$, pump 60$, 2 new blocks costs me 180$ each, block for 5870 costed 30$. 2 Radiators and other accesiores i had for a long time working in my old systems.

[Red Emerald]

So this thread is mostly about the hardware, but I'm curious what you use for the software.

I know you've mentioned that part of your farm is mining at p2pool.  Do you use BAMT? Linuxcoin? Your own installs?

[DeathAndTaxes]

I was/am using BAMT for the 3x5970 air cooled rigs  but it is x86 only which means it won't work w/ 8 GPU for the watercooled conversion.  For the first watercooled rig I am using a modified version of LinuxCoin.   I may consider a custom distro at some point.

The farm is on p2pool (well part of it is right now), but p2pool is not installed on each rig.  The "watchdog server" runs bitcoind and p2ool.  My workstation runs a backup copy of bitcoind & p2pool.  Each rig has first p2pool instance as primary, the second p2pool instance as secondary and bitminter as tertiary mining pools using failover.  Something that is a low priority but I would like to look into, is using a static distro which is reconfigured on each boot.  There would be no USB drives.  The rigs would simply boot from LAN and pull the distro from a PXE server (watchdog again).   Once booted the rig would pull a cgminer config file from the server.

My goal is to make the actual rigs as "dumb" as possible.  Turning them into 3GH/s hashing appliances with control/config/monitoring done by the watchdog.  It looks like my pump might arrive today.

[Spotswood]

I'm curious as to how much heat that pump is going to add to the loop. 

[siggy]

I'm curious as to how much heat that pump is going to add to the loop. 

less than 1/2 of 1 GPU equivelant.   These things are used a lot for salt-water fish tanks.. they are very efficient pumps. 

Sigg

[DeathAndTaxes]

I'm curious as to how much heat that pump is going to add to the loop.  

About 90W.  In a "normal" water cooling system that would be excessive.  Given I have a heat load of ~6KW, 90 more isn't much.  One way to look at it is I am replacing 24 blower fans @ 8W ea = 192W with a 90W pump and 40W fan.

The pump did arrive today.  UPS ground and it got here in 2 days.     It is impressive.  Everything about it screams build quality and performance.  Did some testing and ended up soaking myself.  Even at 10ft of head it moves a ton of water.   Hopefully I can get the first rig hooked up to the main loop this weekend and then order parts for the rest of the rigs.    No time to waste we already have 80F weather.   

[rjk]

The pump did arrive today.  UPS ground and it got here in 2 days.     It is impressive.  Everything about it screams build quality and performance.  Did some testing and ended up soaking myself.  Even at 10ft of head it moves a ton of water.

Lol, I bet the sequence of events was something similar to the following:

Hell yes, the pump is here! *rips open the box like a kid at christmas*
Lets see if it works - *plugs in for only 3 seconds with no primer water, cause 3 seconds won't kill it, right?*
Sweet, it works. *runs off to the garage to attach some hose to the intake, and maybe a little bit to the output*
*fills up tub of water, primes pump and plugs in again*
Oh shit son, water everywhere hopefully none got on my rigs shitshitshit
*stupid grin while cleaning up the mess*

[DeathAndTaxes]

Something like that except I never run pumps without priming and I did have the common sense to not test it around the rigs.

Actually everything went fine in first two tests.  Then I figured I could test @ 10ft of head by putting the pump downstairs with a 5 gallon bucket, run discharge hose up the stairs to bathtub on the second floor.  Great plan but how to start and stop the pump.  So I give my wife instructions and warning about not letting intake run dry.  Everything is going good but I start thinking man this is a lot of water is the bucket almost empty.  I turn my head to yell down to her to make sure the bucket doesn't run dry (it didn't) and when I did I dropped the discharge hose.   Of course then it starts snaking around the bathroom and me trying to direct it towards the bathtub catch 20gpm of water to the face.

Short story is the pump "should" be more than powerful enough. 

[johnyj]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

[DeathAndTaxes]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

Yeah I thought about that.  I even tried looking for smaller commercial ones as inspiration.  Seems they don't really exist for anything <20KW.  Even the 20KW units look pretty compact about the size of say 3 or 4 outdoor AC units side by side. 

The other option would be to bury 100m or so of tubing below the front line and have it act as a giant geothermal heatsink.

[cuz0882]

DAT I guess you might consider building a cooling tower when your operation further scaled up. I read about those 50-meter-high cooling tower can cool tons of hot water with only natural airflow, no fans

Yeah I thought about that.  I even tried looking for smaller commercial ones as inspiration.  Seems they don't really exist for anything <20KW.  Even the 20KW units look pretty compact about the size of say 3 or 4 outdoor AC units side by side. 

The other option would be to bury 100m or so of tubing below the front line and have it act as a giant geothermal heatsink.

That's a interesting idea, wonder how effective it would be. I'm a little skeptical about running everything in one loop. Repairs and downtime could be a major problem. Maybe one large loop that cools everything with each pc having its own smaller loop. Using heat exchangers to connect them.

[DeathAndTaxes]

I'm a little skeptical about running everything in one loop. Repairs and downtime could be a major problem. Maybe one large loop that cools everything with each pc having its own smaller loop. Using heat exchangers to connect them.

That was my original idea the major problem is that you now need n+1 pumps, radiators, and n heat exchangers.  Add in the additional cost of more fittings and the cost is simply not viable.

Each rig can be seperated from the main loop without affecting other rigs.  If the main loop is down then heat exchanger or not all the rigs will need to be idle anyways. 

[DeathAndTaxes]

So an update.

The manifold:
I found a manifold for my rigs.  It is a radiant heating manifold.  I was at local plumbing supply house and after explaining what I needed  one of the employees found a returned open box manifold.  This is a $250+ part but was marked down to $120 due to some damaged.  Got them to drop it to $70 with no returns unless not water tight.    I will post some images tonight but here is a stock photos.  The blue caps at the top are balancing valves and the red tubes at the bottom are flow meters. 



One of the balancing valves is jammed, and two of the crimp rings are warped, one of the temp guages doesn't go above 80C.  It also has some cosmetic damages and tooling marks.  No idea what idiot tried to use this and why the store accepted the return but I got a lot of nice features for very little cost. 

Now it is designed for connecting PEX piping and the adapters for each zone are proprietary which makes it a challenge to repurpose.  Through trial and error I found that 1/2" ID, 5/8" OD tubing is very close to the nominal dimensions of 1/2" PEX and remains water tight even under pressure using the compression fittings which work with this manifold. I ordered some 7/16" ID tubing which I think will provide an even tighter seal.  Another option is changing the fittings on each zone to ones designed for 5/8" PEX which is a good fit for 1/2" ID, 3/4" OD tubing.  Since the tubing will be outside the server (connects quick disconnects on front of server to manifold) having a thicker sidewall may be better.

In testing I got 5 loops (no rigs connected) running @ > 1.6 gpm (the max on flow meters).    The balancing valves will make is very easy to ensure each rig gets at least 1 gpm to ensure turbulent flow.  Honestly despite being damaged this is a total score.

The pump and loop:
Hooked the pump up to using some PVC Unions.  At this point I am using 3/4" ID tubing (fits tightly over 1" PEX barbs) for the mainline because PEX isn't exactly flexible or easy to work with when testing.  Purging air from the system was a challenge.  I made a PVC "T" to allow a line for adding coolant and purging air. It only worked marginally.   It took hours to purge air from the loops and I don't think I got it all.  Went back to Lowes and picked up a 2" PVC T with some 1" threaded adapters.  The larger diameter means water will slow down as it enters the T allows the air to separate easier.  

The heat exchanger:


The good news.  It is massive and well built with a ton of surface area both the fins and the internal tubing.  The surface area of copper tubing is important because it is how heat is transferred from the water to the fins.  You can have a giant heat sink but if it only has a small amount of tubing it isn't going to be very efficient.

To put it into perspective your average 3x120mm water cooling radiator has 1/8" copper tubing usually 4 parallel lines which make one pass.  
Surface area of the copper tubing on 2x120mm radiator  = 2 * Pi * (1/8) * 14 * 2 * 4 = 88 in^2

This heat exchanger using 8 parallel lines of 3/8" copper tubing and makes 3 round trip passes.  
Surface area of copper tubing on this water to air heat exchanger = 2 * Pi * (3/8) * 20 *6 * 8 = 2261 in^2  

The bad news.  I bought it from an ebay seller and some fins were damaged.  I can fix them but I need a "rake".  Worse for some reason rather than capping the 1" copper tubes with plastic or rubber plugs they sweated bronze caps (using what looks like two gallons of solder).  In the process it looks like they knocked the tubes out of round.  I had a nightmare of a time sweating new fittings on.   I ended up cutting the caps off and having to bend the tubes back round.  It look a lot longer than I had planned.  Should have spent a little more because I certainly paid for any discount in manhours.

Disclaimer:
I hope these posts have shown that there is nothing "standard" about cooling a rack of servers.  Everything requires adjustment and modification as you go.  Luckily I own a lot of tools.  If I didn't I likely would have had to spent $1000 in tools and supplies (wrenches, channel locks, pipe cutters, soldering torch, mapp gas, tube cutters, pipe dope, thread sealant, etc).

What's next:
Will get some photos tonight.  Hopefully (work dependent) in the next couple days I can get the time to hook the test rig to the new cooling loop.  I have enough waterblocks to test a second 3x5970 rig maybe this weekend.  If everything goes good I need to pull the trigger on 20x 5970s waterblocks.

[cuz0882]

I was thinking of adding a fourth card on one of my water cooled rigs. There would be no slot spacing between 2 of the cards though. I was thinking of using male to male adapter. You know if that would work? http://www.amazon.com/Enzotech-Rotary-Adapter-BRMM-G14-Black/dp/B007AIXE1M/ref=sr_1_36?s=electronics&ie=UTF8&qid=1332286492&sr=1-36

[DeathAndTaxes]

it probably will work but I am not positive.  That is why I used MSI 890FXA-GD70 motherboard.  It has 4 16x slots 2 apart. 

Likely you may need to do some trial and error to find something which works with a single slot.

[DeathAndTaxes]

Some photos

The manifold.  5 test loops.  The balancing valves make it easy to keep flow even through all loops even if I create artificial restriction in one of the loops.  No rigs attached yet but hopefully I can get the first rig hooked up tomorrow and maybe second partial rig by this weekend.  If all goes well then I need to order 20 more waterblocks and for on the next 4-6 rigs.


The pump assembly.  Used two unions and the assembly to the left is the bubble eliminator MK II.  The larger pipe size slows the water velocity down which makes it easier to separate air.  Air flows up the T line which is where additional coolant is added. 


The radiator.  Also 4 of my existing aircooled 3x5970s rigs.  Will be nice to have some peace and quiet in the garage again.


The entire loop.  I am currently using 3/4" tubing but hopefully this weekend I can mount manifold properly to a sheet of plywood (which will mount to side of server rack).  Once everything has been tested in their final position I intend to replace the tubing with 1" PVC.

[rjk]

Beautiful. Those test loops look a little kinked, hope you will be using unkinkable hose for the final run. BTW, how much flow restriction should I expect from a waterblock? I'm sure normal pressure shouldn't be more than 1-2 bar, but when sizing everything up, the pump expert wants to know how much actual restriction there is, and I can't find anything relevant in the tech specs.

[DeathAndTaxes]

Yeah the tubing is the Lowes chepo vinyl stuff.  I got better tubing but no sense in cutting it up for test loops.  It was hard to keep it from getting kinked but I intentionally kinked some of them to increase resistance to simulate a server loop.  Not very scientific but luckily this pump seems have more than enough juice.

Most watercoolers measure resistance in terms of ft of head loss.  It varies by waterblock and finding exact specs can be tough but resistance is usually on the order of 1 to 2 ft of head loss per block at 1 gpm.  If the blocks are in series then the head loss if cumulative.  If the blocks are in parallel the head loss for the loop is the most restrictive.

There isn't a lot of information because really modern blocks aren't that restrictive and modern pumps have more than enough head.  This can make it tough to find good stats.  I have found reviews on pumps tend to have better stats on loop resistance than review on the blocks themselves.

Restriction (head loss) increases with flow rate but flow rates > 1 gpm really don't improve cooling much (1 gpm can transfer ~500 watts of energy w/ only 1 C rise in water temp).  The loop will reach an equilibrium between flow and resistance.  The goal is just to make sure that is >1 gpm.  I am planning for ~1.5 gpm flow through each loop.  6 loops = ~9 gpm across the main line.  The reason we want >1 gpm is to ensure turbulent (not laminar) flow through the block.  Older blocks required much higher flow to be turbulent but modern blocks have improved on that too.  

Sadly watercoolers today have it too easy.  Blocks are less restrictive, they avoid laminar flow even at low flow rates, pumps have higher flow and can handle higher pressure, larger tubing means less loop resistance, etc.  Compared to 10 years ago it is kinda hard to build a bad "normal" loop.  The bad news for us "extreme" builders is that means less stats on flow & pressure because for normal users taking any decent pump, any decent block, and any decent tubing likely will work just fine.

[DeathAndTaxes]

An update:
Had a minor leak at the radiator where the inlet pipe was knocked out of round.     Removed existing fitting, sweated on a 90 deg connector (which I wanted to use anyways, if you got to resweat the pipe at least make it an upgrade).  Was water tight for 12 hours+.

My Primoflex Pro LRT tubing (ID: 7/16 OD: 5/8) arrived.  A perfect fit for 1/2" PEX connectors.  Nice and flexible.  I updated the fitting on the rig to 1/2", attached the tubing put the inline no leak quick disconnect connectors and ran the loop to the manifold.

Had a minor leak when cutting water on.  The 1/2" barbs I bought don't fit on the "metal side" of the waterblock (they fit fine on the other end).    They threads are too long and they bottom out before compressing the o ring completely.  Luckily I had a pair of 1/2" barbs hanging around.   BTW large amounts of shop towels underneath all fittings made initial water turn on less scary.  Used hair dryer to remove any water (I didn't see any but I am paranoid).  Leak tested it for 12 hours.  Looks good.

Powered up the rig this morning.

Code:

 cgminer version 2.3.1 - Started: [2012-03-23 11:43:21]
--------------------------------------------------------------------------------
 (10s):2963.1 (avg):2994.1 Mh/s | Q:4149  A:1364  R:19  HW:0  E:29%  U:20.83/m
 TQ: 10  ST: 11  SS: 8  DW: 1957  NB: 5  LW: 4549  GF: 1  RF: 0
 Connected to http://192.168.0.189:9332 with LP as user user/1000+1
 Block: 000007efc04c94ca1185fdc1e6565e85...  Started: [12:43:03]
--------------------------------------------------------------------------------
 [P]ool management [G]PU management [S]ettings [D]isplay options [Q]uit
 GPU 0:  47.0C  960RPM | 371.2/370.9Mh/s | A:199 R:2 HW:0 U:   2.84/m I: 8
 GPU 1:  48.0C  960RPM | 371.1/370.8Mh/s | A:184 R:5 HW:0 U:   2.63/m I: 8
 GPU 0:  47.0C  960RPM | 371.2/370.9Mh/s | A:200 R:1 HW:0 U:   2.86/m I: 8
 GPU 1:  48.0C  960RPM | 371.1/370.8Mh/s | A:182 R:3 HW:0 U:   2.63/m I: 8
 GPU 4:  45.0C  960RPM | 371.1/370.8Mh/s | A:179 R:1 HW:0 U:   2.55/m I: 8
 GPU 5:  48.5C  960RPM | 371.2/370.8Mh/s | A:184 R:1 HW:0 U:   2.63/m I: 8
 GPU 6:  49.0C  960RPM | 371.2/370.8Mh/s | A:170 R:4 HW:0 U:   2.43/m I: 8
 GPU 7:  48.0C  960RPM | 371.1/370.8Mh/s | A:174 R:2 HW:0 U:   2.48/m I: 8
--------------------------------------------------------------------------------

[2012-03-23 12:53:14] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:19] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:21] LONGPOLL requested work restart, waiting on fresh work
[2012-03-23 12:53:28] Accepted 00000000.5031a643.261ef557 GPU 0 thread 0 pool 0
[2012-03-23 12:53:33] Accepted 00000000.29cdb6c5.b7edd474 GPU 6 thread 6 pool 0

Clocks are 830/150 at stock voltage.  (Most of cards were stable @ 870 but I decided to start low).
I am planning on eventually overvolting them slightly but that requires a bios flash.  I think 900/150 @ 1.125V should be possible which should give me 3.2 GH/s per rig.

Ambient is about 33C.  I have the garage closed with no AC to simulate hotter ambient temps of this summer.  I don't have my radiator fan yet so this is using a 14" desktop fan pushing minimal air through radiator.

Looking good so far.

Whats next:
Hopefully this weekend I can get the second rig connected to the loop.  Also want to mount the manifold to project plywood and start planning the PEX runs.  I also need to go grab the server rack I bought off a friend.   The next phase involves 20 danger den waterblocks (sound a bulk discount of $99 per block), 5 more server chassis, 50ft of tubing, and some fittings.

[malevolent]

I am planning on eventually overvolting them slightly but that requires a bios flash.  I think 900/150 @ 1.125V should be possible which should give me 3.2 GH/s per rig.

You sure it is worth to overvolt considering the extra power draw?

looks nice btw

[DeathAndTaxes]

You sure it is worth to overvolt considering the extra power draw?

Yeah at least now.  As price/difficulty rises I can move to "normal volt" and eventually undervolting.

Right now the rigs get ~3 MH/W and my electrical rates are $0.10 per kWh. 

That puts my mining cost (electrical only) at:

MH per bitcoin =  (1498294.36282  * 2^32) / (50 * 1000^2) = 128702505.8
kWh per Bitcoin = 128702505.8 / (60 * 60  * 3 * 1000) = 11.9
energy cost per Bitcoin = 11.9 * 0.10 = ~$1.20

Gross margin is = $5.00 - $1.20 = $3.80

So while overvolting will reduce the margin I will produce more coins.  It is only once you start getting close to break even when overvolting becomes unprofitable. 

Overvolting is just something planned.  Once I measure power draw and stable clocks I can get some exact # of what my margins are at various over & undervolts.

TL/DR version:
My thinking is to overvolt and increase gross revenue.  As the rest of network becomes more efficient I can lower voltage to say ahead of the efficiency curve.  Eventually (say 12 to 18 months from now) I may need to be running at much lower hashrate and voltage into order to remain profitable.  I can dynamically balance efficiency vs gross hashing power based on the network's price/difficulty ratio.

[DeathAndTaxes]

An update.  Real job has been brutal last two weeks leaving me less time to work on my project.

I did assemble the rack though (got it off craigslist for $100 like 9 months ago).  The front inside rail is bent in a roughly 4" section which is why the PDU and switch is put mid rack.  



Pretty empty so far but hopefully more by next week.

I got the manifold mounted on plywood (which is going to mount to the side of the rack).

[Cablez]

Just wicked,  a watercoolers wet dream.   

[Omni]

Sweet looking setup bro.

[twoBitBasher]

Did you buy the waterblocks yet for the rest of your GPUs? How is your project progressing?  This is awesome build!