Bitcoin Forum
December 10, 2016, 12:46:56 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 3 4 5 »  All
  Print  
Author Topic: (Updated w/ pics) Watercooled Rack of Servers - 50% completed  (Read 9866 times)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 24, 2012, 01:22:04 AM
 #1

So the original thread is here (but the plan changed a lot and the thread got sidetracked at points w/ flames & trolls)
https://bitcointalk.org/index.php?topic=66606.0

The Purpose (also know as "are you fraking crazy FPGA are going to take over the world ... ")
I ALREADY have a farm of six 3x5970 rigs plus another 6x 5970s cards looking for a home.  The hardware has long since been paid off and I have some measure of pride knowing the rigs I built are just about as efficient as is possible with GPU tech (although some optimizes 7970 rigs might edge them out).  The goal is to extend that investment as long as possible.  Kinda a GPU "last stand" against the encroaching FPGA hoards  

My goal was to find a way to:
a) survive the summer without the 1/3rd increase in electrical cost
b) avoid paying ~$3K to the upgrade AC system (farm was expanded in the fall so current AC won't be able to handle it)
c) have fun building a big insane system.
d) see if it would be possible to capture some of that heat for preheating hot water heater and/or heating the house.

So I had this crazy idea.  Can you put 24+ HD 5970s in a standard server rack and using a really powerful pump move all that heat (18,000 BTUs) outside?

Well the farm is 50% converted.

The Rack
From top to bottom is APC PDU, management server, gigabit switch, second APC PDU, and the first three water cooled rigs.


Closeup of the rigs
No I am not cooling it with milk.  I guess the flash made it look like that.  Just using plain distilled water.


Connectivity & Power
I need some better cable management but power on left, connectivity on the right is a good start.  I hope to eventually replace all the flash drives with PXE.


Manifold
First three rigs are hooked to the three left most ports.  Isolation valves on top, flow control valves on bottom.
The connected loop on the right ensures even if I absent mindedly disconnect all 3 rigs the pumps will still get about 1gpm flow (important to keep the pump cool).  The photo is kinda hard to tell but this is mounted on the right hand side of the rack when looking from the POV of the first photo.
http://i.minus.com/iBLM12pskhT2g.png

The pump
A real monster but quiet and has a nearly silent fan in the back which reduces the amount of heat which ends up in the loop.  A pair of unions simplify hooking it up to the loop. The pump and fan are the only component which can't run on 240 which necessitates running 120V extension cord.


The radiator
Has about 16x the pipe surface area as your "normal" 3x120mm radiator and about 6x the fin surface area.  Smiley   I am still using the poly tubing but will be replacing it with PEX when I get the time.  Fan is just a "normal" floor fan.  Probably 1800 cfm.  I may replace it with a car radiator fan or maybe just a outdoor rated workspace fan.



The rack contains:
3x quad 5970s rigs
2x APC 9571 PDU (240V, 30A, 80% derate) for 11.5 KW usable power
1x Cisco 24 port gigabit switch
1x 1U watchdog server (runs bitcoind, p2pool, remote power relay card, and monitoring software)

Current config:
12 5970s (24 GPUs) ~ 9.2 GH/s

When complete:  
24 5970s (48 GPUs) ~ 18-20 GH/s

Theoretical max:
40 5970s (80 GPUs) ~ 30-32 GH/s
1481374016
Hero Member
*
Offline Offline

Posts: 1481374016

View Profile Personal Message (Offline)

Ignore
1481374016
Reply with quote  #2

1481374016
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481374016
Hero Member
*
Offline Offline

Posts: 1481374016

View Profile Personal Message (Offline)

Ignore
1481374016
Reply with quote  #2

1481374016
Report to moderator
1481374016
Hero Member
*
Offline Offline

Posts: 1481374016

View Profile Personal Message (Offline)

Ignore
1481374016
Reply with quote  #2

1481374016
Report to moderator
Uncurlhalo
Full Member
***
Offline Offline

Activity: 140


Look it up.


View Profile
April 24, 2012, 01:31:47 AM
 #2

That thing looks so sexy. I want to see it full of shinny electronics.

1JacobTM32mCqTD6NWRsBwtxbT7z7fUGWE
I love building things.
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
April 24, 2012, 02:51:09 PM
 #3

Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 24, 2012, 03:01:37 PM
 #4

Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.

Yeah the bottom one is done the way I want it. I took the 3.5" plate and drilled 2 holes.  One half of the quick disconnect has a panel mount adapter so it mount directly to the plate.  The plate is then monted in the drive cage.  This makes the internal tubing loop fixed.  Less chance of moving around, stressing the line, or causing a leak.  Right now if you yanked on the tubing for the bottom rig the worst you could do is cause a leak outside the rig.

However the steel is pretty tough. It wore out my cheapo bit after the first plate.   Need to get a stronger bit to drill out the rest.
check_status
Full Member
***
Offline Offline

Activity: 196


Web Dev, Db Admin, Computer Technician


View Profile
April 24, 2012, 07:33:04 PM
 #5

Looking good so far. Do you have any pics of the other end of your watercooling setup?

For Bitcoin to be a true global currency the value of BTC needs always to rise.
If BTC became the global currency & money supply = 100 Trillion then ⊅1.00 BTC = $4,761,904.76.
P2Pool Server List | How To's and Guides Mega List |  1EndfedSryGUZK9sPrdvxHntYzv2EBexGA
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 24, 2012, 07:34:00 PM
 #6

Looking good so far. Do you have any pics of the other end of your watercooling setup?

Yeah should be up this evening.
twoBitBasher
Member
**
Offline Offline

Activity: 85



View Profile
April 25, 2012, 12:09:49 AM
 #7

You know "this evening" is pretty useless without specifying a timezone. For me(gmt+2) "evening" was technically several hours ago  Tongue

If you think my comments have benefitted you it would be nice to hear thanks Smiley

Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz
Cablez
Legendary
*
Offline Offline

Activity: 1400


I owe my soul to the Bitcoin code...


View Profile
April 25, 2012, 01:13:13 AM
 #8

How about wait for evening EST?  Tongue

Tired of substandard power distribution in your ASIC setup???   Chris' Custom Cablez will get you sorted out right!  No job too hard so PM me for a quote
Check my products or ask a question here: https://bitcointalk.org/index.php?topic=74397.0
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
April 25, 2012, 01:13:44 AM
 #9

How about wait for evening EST?  Tongue
That would be now... Wink

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 25, 2012, 01:17:26 AM
 #10

Sorry guys.  By midnight EST.  Guess I should have been more specific.
twoBitBasher
Member
**
Offline Offline

Activity: 85



View Profile
April 25, 2012, 01:35:41 AM
 #11

I wonder why haven't these php forum coders thought about this multi timezone operation. Everyone probably do set the timezone on their profile so it wouldn't be too difficult to show it in posts with the user info on the left. Most regional forums do just fine without it but Bitcointalk is hardly regional.

Edit: okay my bad. It does show if you check users profiles but that's a chore.

If you think my comments have benefitted you it would be nice to hear thanks Smiley

Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 25, 2012, 02:58:12 AM
 #12

Updated first post w/ more pics & details.
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
April 25, 2012, 03:08:06 AM
 #13

Updated first post w/ more pics & details.
Very nice. I just have been re-pasting my cards (because I had to remove the heatsinks in order to take the plastic bits off), and I can only imagine what hell it is to do so many. I've been keeping the same old thermal tape and pads, but even then I can only do 2 at a time before getting frustrated and having to take a break. Have done a dozen so far.

Maybe if/when you switch to 7990s I will be able to buy your 5970s with the blocks already applied to stuff in my rig. Grin Wonder how much they will be worth then.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
twoBitBasher
Member
**
Offline Offline

Activity: 85



View Profile
April 25, 2012, 04:44:20 AM
 #14

Heh that beast of yours do deserve watercooling. It would be the most epic watercooling rig ever. I'd say if you play your cards right you might be able to get somebody sponsor those blocks for you rjk  Cool I bet it could get a lot of coverage on IT-media.
I just wonder what kind of scores would both of yours (dt & rjk) rigs do folding. Top 10?

Car radiator fan is probably not designed for continuous use. I'd say go for the workspace fan.

If you think my comments have benefitted you it would be nice to hear thanks Smiley

Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz
coretechs
Donator
Sr. Member
*
Offline Offline

Activity: 362



View Profile
April 25, 2012, 01:54:13 PM
 #15

Given the can of motor oil on the shelf, I assume this is in your garage?

It's looking great.  Any plans for a backup pump or will you just do software throttling?

http://bitcoindoc.com - The Rise and Rise of Bitcoin | http://nxtportal.org - Nxt blockchain explorer
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
April 25, 2012, 01:58:16 PM
 #16

I just wonder what kind of scores would both of yours (dt & rjk) rigs do folding. Top 10?
Aha, I doubt it. Folding uses CPUs and Nvidia hardware, which are no good for mining. Mining hardware is AMD, and I doubt that it would fold vary well, although I don't know a whole lot about the folding scene.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
naz86
Member
**
Offline Offline

Activity: 111


View Profile
April 25, 2012, 06:06:32 PM
 #17

When you exchange the heat into you warm water for your house you can make your rigs more efficient. This is what i am planning to do with the heat of my serverroom.... in germany its called luft-wärme-pumpe... in english it could be air-heat-exchanger (or something similar). your efficiency is even higher if you can exchange the heat of water to water....

TOP TRADER:
pazor
rampone

Multipremiumaccount mit Bitcoin bezahlen:

https://secure.premiumize.me/inv/3977664725/

http://www.pyramining.com/referral/7qrnh4s6g            2012-07-10 14:37:47 UTC    still active
http://www.pyramining.com/referral/cp9bqzs6e    2012-06-06 14:09:08 UTC    still active
http://www.pyramining.com/referral/senh7t3b9            2012-06-06 14:09:08 UTC    still active
http://www.pyramining.com/referral/8pbfmhync    2012-06-06 14:09:08 UTC    still active
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:45:43 AM
 #18

Given the can of motor oil on the shelf, I assume this is in your garage?

Yeah.

Quote
Any plans for a backup pump or will you just do software throttling?

No plans for backup pump at this point.  I have the ability to cut power to the rigs (the reason among others for the watchdog server).  I intend to install both a current transformer and flow meter.  In a pump failure situation the relay board in the watchdog server will trip power on all 6 rigs.

cgminer is already set to do software throttling (reduce clock @ 60C, idle thread @ 70C) but the "power kill" provides a further layer of protection.  It likely isn't necessary but it does buy some piece of mind when you go $4K worth of GPU relying of coolant.

This weekend I intend to waterblock some/all of the last 12 GPUs.  God I am not looking forward to that. Sad
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
April 26, 2012, 01:48:31 AM
 #19

When you exchange the heat into you warm water for your house you can make your rigs more efficient. This is what i am planning to do with the heat of my serverroom.... in germany its called luft-wärme-pumpe... in english it could be air-heat-exchanger (or something similar). your efficiency is even higher if you can exchange the heat of water to water....

Yeah that is the goal once this is done and stable.  A water-water heat exchanger to preheat the cold water line of hot water heater. 
twoBitBasher
Member
**
Offline Offline

Activity: 85



View Profile
April 26, 2012, 03:45:18 AM
 #20

Let's say you preheat your water. You got 6kw of constant heat coming. Your water boiler doesn't heat the water constantly but just when it's temperature drops below a certain limit(when you use water). What happens to your excess heat? You don't plan on leaving your tap water running do you? Tongue

I guess it needs some kind of fancy temperature controlled bypass valve.

If you think my comments have benefitted you it would be nice to hear thanks Smiley

Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz
Pages: [1] 2 3 4 5 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!