Bitcoin Forum
December 08, 2016, 02:30:25 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 [2] 3 4 5 6 7 8 »  All
  Print  
Author Topic: A journey of extreme watercooling: Cooling a rack of GPU servers without AC.  (Read 25800 times)
stevegee58
Hero Member
*****
Offline Offline

Activity: 783



View Profile
February 28, 2012, 04:06:47 PM
 #21

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

You are in a maze of twisty little passages, all alike.
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
1481207425
Hero Member
*
Offline Offline

Posts: 1481207425

View Profile Personal Message (Offline)

Ignore
1481207425
Reply with quote  #2

1481207425
Report to moderator
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 04:36:37 PM
 #22

Based on current USD/BTC and your kwh rate, what's your estimated investment payback period?

If I only convert this one rig the payback is <3 months due to me already having some of the components (sunk cost).  Beyond that it is harder to estimate because I am not sure how far I can push the cards yet and if I will find that changes to my plan need to be made.  Also remember these aren't new rigs but "rig conversions" to need to look at incremental cost and incremental benefit.

Still a rough estimate:
Each rig conversion costs ~$900 and will provide ~600 MH/s extra hashing power ($66 per month) and will also eliminate ~$45 per month in cooling cost so direct payback is ~8 months.  If I can push them beyond 3.2GH/s every 100MH/s provides another ~$9 per month in net revenue. I am hoping I can get a bulk discount at least on the 20 waterblocks so with lower cost and higher hashrate payback might be as good as ~7 months.  Overvolting is also an option.

So I would say absolute best case is ~6 months worst case is 9 months.
eleuthria
Legendary
*
Offline Offline

Activity: 1750


BTC Guild Owner


View Profile WWW
February 28, 2012, 04:54:49 PM
 #23

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power Smiley.

R.I.P. BTC Guild, 2011 - 2015.
BTC Guild Forum Thread
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 05:05:10 PM
 #24

Just remember that the real cost in a datacenter isn't space, but amperage.  If those are running on 120v, you'll have a hard time finding a datacenter that can give you access to nearly 100 amps @ 120v with just 1 rack.

If you're just going to put them in your own 45U rack, I hope you've got a separate feed going into your house/office that can give you that kind of power Smiley.

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

I have 60A @ 240V in a sub panel with 2 NEMA L6-30R outlets and a pair of APC AP9571 PDUs (48A derated).  I have loaded ~40A without issue.   Right now I only have gear for 6 complete rigs not a full rack of 10 and I don't have any plans to expand that.
hmblm1245
Hero Member
*****
Offline Offline

Activity: 635


View Profile
February 28, 2012, 05:19:14 PM
 #25

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 05:23:40 PM
 #26

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots". Smiley

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm

rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 28, 2012, 05:26:17 PM
 #27

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots". Smiley

http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm


Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
hmblm1245
Hero Member
*****
Offline Offline

Activity: 635


View Profile
February 28, 2012, 05:34:58 PM
 #28

What chassis is that? i have been looking for a 8 exp. slot rack mount for months can can only find pricey tesla supermicros and the RM41300-FS81.

That is it.  As far as I know it is literally the only sub $400 8 expansion slot 4U chassis on the planet.  Kinda bizzare given punching an eighth slot has essentially no real cost.  Its not like someone wanting 7 slots would say "forget this case it has 1 too many slots". Smiley

Provantage is out of stock but looks like they have the cheapest price: $128
http://www.provantage.com/chenbro-micom-rm41300-fs81~7CHEN0RM.htm


Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81? Smiley
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 05:35:27 PM
 #29

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy (even heavier filled with water) and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and coolant pouring over $3000 in components. Sad

The RM41300-FS81 is $120 so getting cheaper case would save some money.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.  LOLZ.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the (lots of weight) + (water) + (single tiny screw) = risk issue.

In the end I decided to pay the extra $60 or so and play it safe (probably oversafe).
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 28, 2012, 05:38:05 PM
 #30

Couldn't you use a 7-slot case that has a little more room where the 8th slot would go, and replace the bracket with a single slot bracket on the 5970?
http://www.plinkusa.net/4u5u.htm <-- lots of cheap 4U 7 slot cases

EDIT: Bracket: http://shop.aquacomputer.de/product_info.php?products_id=2460

I thought about it but the waterblocks are heavy and I worry about all that weight supported by a single slot screw.  Then you combine the catastrophic damage of slot bending and water pouring over $3000 in components.

The RM41300-FS81 is $120 so getting cheaper case would save $70 or so per rig.  The bracket you linked to is very hard to find and Aqua wants $35 to ship to US so it ends up being almost $15 ea.    I could dremmel the slot bracket to make it a single slot but that makes it kinda hard to resell and still got the risk of "lots of weight + tiny screw". 

In the end I decided to pay the extra $60 or so and play it safe.  Likely being over cautious but I likely will buy the same expensive case for the next 5 rigs too.
Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case Wink

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 05:39:25 PM
 #31

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81? Smiley

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.
Inspector 2211
Sr. Member
****
Offline Offline

Activity: 383



View Profile
February 28, 2012, 05:41:50 PM
 #32

It will be in my own rack no datacenter.  The largest advantage comes from dumping heat directly outside so datacenter doesn't really make sense.

>directly outside

How do you plan to do that?
Is it a BARN, with a overhanging roof and a huge open window?

Half a year ago I had eight tower cases perched on the windowsill of an office, four where the right windowpane opened and four where the left windowpane opened. I added external ducting and 12cm "exhaust fans" at the mosquito screen of the windowpanes.

Problem was, they were blue LED fans - the building owner drove by at night, went WTF? - and eventually asked me what the f*** I was doing. I gave an evasive answer ("research") and was evicted shortly thereafter with a 3-day notice.  Cry

Now you know why I'm interested in BFL Singles and FLGA boards...  Roll Eyes
hmblm1245
Hero Member
*****
Offline Offline

Activity: 635


View Profile
February 28, 2012, 05:47:19 PM
 #33

Thanks! Chenbro's site shows that as a tower. i ended up just going with a 3 card setup for my cluster b/c i could not find any decently priced 4 gpu rackmount cases. Maybe i should replace my setup with these RM41300-FS81? Smiley

It is designed to be both.  It has rack dimensions and rack handles on the front but it also comes with stand you can attach to use it as a tower (likely adding to the cost).

Other than the fact that it is stupid that noboody else has a 8 slot case under $400 and I felt kinda forced into it getting this chassis it has been solid so far.  Fits my needs exactly.   I wouldn't use it for air-cooling though which is supposedly what it is designed for.  I would worry about not getting enough air in.

I think with alot of 133+ cfm fans you can get the temps down. like this guy

http://www.ebay.com/itm/New-nVidia-Tesla-S870-6GB-Memory-GPU-Processing-Computing-Server-System-BitCoin-/220938944468?pt=LH_DefaultDomain_0&hash=item3370fcbfd4#ht_4992wt_1163

But of course those heat sinks are aligned with the air flow, but if you design some simple copper heat sinks and mount them to the 5970s this might work. It would be loud as hell with those 40mm fans but it would only take up 1U Smiley
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
February 28, 2012, 05:50:56 PM
 #34

Yes, it isn't cheap. I assume you would only need one per case (last card using it) and you could prop it against the side of the case. How well do those water fittings fit together?

Yeah it would be one per rig but I will need 6 all together.  

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.

Quote
That bracket was just from a quick search, I am sure DD and others make cheaper ones. You save because you don't have to spend as much on the rack case Wink

I looked and I couldn't find a store in US that carries it.   I called frozen cpu and they don't stock it because it is explictly not designed for water cooling.  The manufacturer puts in small print not designed for heatsinks over 1 pound.  WTF? What else would you use it for?  Cramming hot aircooled 5970s even closer together?  Smiley

Quote
How well do those water fittings fit together?

The fittings work well as long as you don't move the cards. Smiley
The SLI fitting is adjustable (necessary to get multiple cards installed) but there is a max limit.  If you pull the "end caps" too far apart the tube in the inside can slide past the double o-rings and leak.   You really need to stupidly pull them "far" apart (say 1" more than they are now). The risk is likely small inside a crammed case but I am a coward.

There are other "single piece" methods but they add a lot of cost.
jake262144
Full Member
***
Offline Offline

Activity: 210


View Profile
February 28, 2012, 09:55:46 PM
 #35

Some updated pics.
...
Thoughts, comments, suggestions?
She's a beauty, DAT.
Well done - and without a single blue led or cold cathode. Looking at the average watercooled rig I wouldn't deem omitting some sort of l337 light source possible Cheesy
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
February 28, 2012, 10:54:44 PM
 #36

Although with right MB and if AMD ever fixed their fracked drivers you could use more than one to get 10 or 12 GPU in one case.  6x7990 in one 4U chassis.  Hmm... See now you distracted me.
Stay out, that's my territory Tongue Grin

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
Red Emerald
Hero Member
*****
Offline Offline

Activity: 742



View Profile WWW
February 28, 2012, 11:00:08 PM
 #37

/drool

Turbor
Legendary
*
Offline Offline

Activity: 1008


BitMinter


View Profile WWW
February 28, 2012, 11:38:50 PM
 #38

Good work. Looks clean and has this professional touch.

jothan
Full Member
***
Offline Offline

Activity: 184


Feel the coffee, be the coffee.


View Profile
February 29, 2012, 02:14:26 AM
 #39

If you had chosen BitFORCE, you could've built the same system (3328 MH/s, about 328 MH/s higher than the actual solution) with only 4 units,
costing total of 2400USD (+ shipping) and consuming only 330 Watts (less if a high-efficiency power-supply was used to power all 4 units).
Virtually silent as compared to GPUs (26dB each unit) and a fraction ( A third so to speak ) on electricity costs Smiley

Regards,

That's what I figured when I ordered 6 units from you a few days ago. Assuming they get here within the "4 to 6 week" period, I will have a very nice high-hash, low-power and low noise upgrade to my mining rigs within a few weeks.

BTW, it's nice to see you on the forum, BFL-Engineer (Sonny, I presume).

Keep shippin' em' and I'll keep orderin' em as long as I get them within reasonable delays.

Bitcoin: the only currency you can store directly into your brain.

What this planet needs is a good 0.0005 BTC US nickel.
Isepick
Full Member
***
Offline Offline

Activity: 181


View Profile
February 29, 2012, 02:41:16 AM
 #40

I had considered going under water for my 10 Ghash/s, but since I have none of the blocks or rads, it was going to be way too expensive. Instead I went the ghetto route:





That's a 12", 1000 cfm vortex fan (pulling 350w). It is far cheaper and easier to suck the hot air outside. The whole thing (rack, fans, foamboard, tape, etc) took $500 and a weekend to build. I live in S. Florida and this is my solution for the summer. It used to get over 100F in that room with only 4 rigs, now it is maybe 5 degrees warmer than the rest of the house on a hot day with 7 rigs. This may or may not help you, and watercooling is a hell of a lot neater, but from a maximum profit point of view it is cheaper to move a large quantity of air over those cards than it is to water cool them. And yes, that fan is louder than anything any video card Wink. Anyhow, just throwing this out there for ideas. Good luck.

Pages: « 1 [2] 3 4 5 6 7 8 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!