A journey of extreme watercooling: Cooling a rack of GPU servers without AC.

[stevegee58]

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

[1715344703]

1715344703

[1715344703]

1715344703

[1715344703]

1715344703

[DeathAndTaxes]

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

If I only convert this one rig the payback is <3 months due to me already having some of the components (sunk cost).  Beyond that it is harder to estimate because I am not sure how far I can push the cards yet and if I will find that changes to my plan need to be made.  Also remember these aren't new rigs but "rig conversions" to need to look at incremental cost and incremental benefit.

Still a rough estimate:
Each rig conversion costs ~$900 and will provide ~600 MH/s extra hashing power ($66 per month) and will also eliminate ~$45 per month in cooling cost so direct payback is ~8 months.  If I can push them beyond 3.2GH/s every 100MH/s provides another ~$9 per month in net revenue. I am hoping I can get a bulk discount at least on the 20 waterblocks so with lower cost and higher hashrate payback might be as good as ~7 months.  Overvolting is also an option.

So I would say absolute best case is ~6 months worst case is 9 months.

[eleuthria]

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power .

[DeathAndTaxes]

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power .

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

I have 60A @ 240V in a sub panel with 2 NEMA L6-30R outlets and a pair of APC AP9571 PDUs (48A derated).  I have loaded ~40A without issue.   Right now I only have gear for 6 complete rigs not a full rack of 10 and I don't have any plans to expand that.

[hmblm1245]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

[DeathAndTaxes]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

[rjk]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

[hmblm1245]

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots".

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

[DeathAndTaxes]

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy (even heavier filled with water) and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and coolant pouring over $3000 in components.

The RM41300-FS81 is $120 so getting cheaper case would save some money.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.  LOLZ.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the (lots of weight) + (water) + (single tiny screw) = risk issue.

In the end I decided to pay the extra $60 or so and play it safe (probably oversafe).

[rjk]

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and water pouring over $3000 in components.

The RM41300-FS81 is $120 so getting cheaper case would save $70 or so per rig.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the risk of "lots of weight + tiny screw". 

In the end I decided to pay the extra $60 or so and play it safe.  Likely being over cautious but I likely will buy the same expensive case for the next 5 rigs too.

Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case

[DeathAndTaxes]

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.

[Inspector 2211]

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

>directly outside

How do you plan to do that?
Is it a BARN, with a overhanging roof and a huge open window?

Half a year ago I had eight tower cases perched on the windowsill of an office, four where the right windowpane opened and four where the left windowpane opened. I added external ducting and 12cm "exhaust fans" at the mosquito screen of the windowpanes.

Problem was, they were blue LED fans - the building owner drove by at night, went WTF? - and eventually asked me what the f*** I was doing. I gave an evasive answer ("research") and was evicted shortly thereafter with a 3-day notice.  

Now you know why I'm interested in BFL Singles and FLGA boards...  

[hmblm1245]

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81?

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.

I think with alot of 133+ cfm fans you can get the temps down. like this guy

http://www.ebay.com/itm/New-nVidia-Tesla-S870-6GB-Memory-GPU-Processing-Computing-Server-System-BitCoin-/220938944468?pt=LH_DefaultDomain_0&hash=item3370fcbfd4#ht_4992wt_1163

But of course those heat sinks are aligned with the air flow, but if you design some simple copper heat sinks and mount them to the 5970s this might work. It would be loud as hell with those 40mm fans but it would only take up 1U

[DeathAndTaxes]

Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

Yeah it would be one per rig but I will need 6 all together.  

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.

That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case

I looked and I couldn't find a store in US that carries it.   I called frozen cpu and they don't stock it because it is explictly not designed for water cooling.  The manufacturer puts in small print not designed for heatsinks over 1 pound.  WTF? What else would you use it for?  Cramming hot aircooled 5970s even closer together?  

How well do those water fittings fit together?

The fittings work well as long as you don't move the cards.
The SLI fitting is adjustable (necessary to get multiple cards installed) but there is a max limit.  If you pull the "end caps" too far apart the tube in the inside can slide past the double o-rings and leak.   You really need to stupidly pull them "far" apart (say 1" more than they are now). The risk is likely small inside a crammed case but I am a coward.

There are other "single piece" methods but they add a lot of cost.

[jake262144]

Some updated pics.
...
Thoughts, comments, suggestions?

She's a beauty, DAT.
Well done - and without a single blue led or cold cathode. Looking at the average watercooled rig I wouldn't deem omitting some sort of l337 light source possible

[rjk]

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.

Stay out, that's my territory

[Red Emerald]

/drool

[Turbor]

Good work. Looks clean and has this professional touch.

[jothan]

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs

Regards,

That's what I figured when I ordered 6 units from you a few days ago. Assuming they get here within the "4 to 6 week" period, I will have a very nice high-hash, low-power and low noise upgrade to my mining rigs within a few weeks.

BTW, it's nice to see you on the forum, BFL-Engineer (Sonny, I presume).

Keep shippin' em' and I'll keep orderin' em as long as I get them within reasonable delays.

[Isepick]

I had considered going under water for my 10 Ghash/s, but since I have none of the blocks or rads, it was going to be way too expensive. Instead I went the ghetto route:





That's a 12", 1000 cfm vortex fan (pulling 350w). It is far cheaper and easier to suck the hot air outside. The whole thing (rack, fans, foamboard, tape, etc) took $500 and a weekend to build. I live in S. Florida and this is my solution for the summer. It used to get over 100F in that room with only 4 rigs, now it is maybe 5 degrees warmer than the rest of the house on a hot day with 7 rigs. This may or may not help you, and watercooling is a hell of a lot neater, but from a maximum profit point of view it is cheaper to move a large quantity of air over those cards than it is to water cool them. And yes, that fan is louder than anything any video card . Anyhow, just throwing this out there for ideas. Good luck.