Bitcoin Forum

This is a (mostly) theoretical question, as doing the things I'm about to suggest certainly void warranty, and probably aren't worth it.  Basically, do not do this ;)  This isn't exactly specific to mining, but since miners are generally the only people crazy enough to try such things, I'm asking here.

I've recently decided to steal a couple 6950's from my mining rigs to build myself a sweet, sweet crossfire gaming rig.  It is truly awesome and all that jazz, however it's loud as all get out and hotter than hell, even with an open slot between my cards.  To alleviate those issues, I'm turning to water cooling.

Now, the motherboard I'm using (Gigabyte 990FX UD3) has 4 16x length slots, two at 16x and two at 8x.  It occurred to me while researching waterblocks that my two-slot cards would effectively become 1 slot cards, and I thought 'hey, I could squeeze 4 cards onto this board!'

Reality and disappointment struck soon after when I realized my cards have two DVI headers stacked on top of eachother, as such:

http://dl.dropbox.com/u/15821147/mining/xfxbackpanel.jpg

The header is a solid-block rather than a ribbon, so the card still occupies two PCI slots even without the monster HSF.

Now, the crazy question - is it possible to desoldier the DVI connectors entirely, removing this restriction?

Leave all the 'this is stupid, use extenders' or 'not worthwhile for mining', 'run it open air', or 'zomg wtf over' type responses at the door, I'm merely asking about the possibility and ramifications, not whether it's sane or worth my time or monies.

The little eeng experience I have tells me the DVI pins are each just connected to individual traces on the board, and 'nothing is plugged in' is basically the same as 'no header connected at all', is this accurate?

As for mining applicability, this would allow squeezing 4 2-slot cards into a single motherboard, well-cooled, inside a case, which obviously has it's appeal ;)

If you are not planning on water cooling I can't see how this would work in that little space. But considering you are practically removing any resale value from the card I don't think this is a good idea.

There are some water cooled 6990s around which I think could be single slot.
But again: Don't do it.

As for your question, yeah it could work.

I didn't think it was that TLDR;, but I did say I'm turning to water cooling ;)

My warranty is already technically voided since I've semi-permanently shorted the write-protect pins on the BIOS, so that's not of much concern.  Neither is resale value - this is now a 'gaming' rig, so I don't plan on getting rid of them anytime soon.  A virtually silent 1.4gh/s enclosed rig that also heats my office would just be a nice bonus.

You didn't write how you are planning to cool it.

There are some 5770s and 6770s with passive heatsinks, I use one in my desktop and it is pretty quiet.
And those have huge heatsinks, are you planning to adapt those cards to something like that?

Not sure why you need them to be single slots how many do you plan on installing?

If you want to watercool your best bet is to sell the 6950s for 6990s or 5970s.  

1) They have a different slot backplane and some watercooling shops sell single slot versions.  You simply remove the DVI nuts (where DVI plug screws into), and two small screws at the end of the PCB (you can see them at the top in the photo).  Then you do the reverse to install a single slot version.
http://www.guru3d.com/imageview.php?image=21084

2) Waterblocks are expensive and you will want full cover waterblocks (which also cool the VRMs).  Given the cost getting 2x GPU for a single waterblock is more economical.

3) With 5970s or 6990s you likely don't need to make them single slot.  The ability to put back to back in 2 slots means you can fit up to 4 on one motherboard (8 GPU total).  Remember 8 GPU is going to be >1KW and it takes a significant radiator to cool more than that.

4) Windows drivers limit you to 8 GPU.  Cross fire is limited to 8 GPU.

So the hardcore solution is simply 4x 5970s/6990s watercooled each taking up 2 slots.  If you want to do it w/ 4x 6950 you can however it is a lot of expense for roughly half the hashing power. 

In either case you just need
1) case w/ 8 slots.  They do make them.  The 8th slot is sometimes called 7+1 because it allows the 7th MB slot to fit double width card.

2) motherboard with 7 expansion slots

3) a motherboard layout like this:
Slot 1 - PCIe 16X
Slot 1 - anything
Slot 1 - PCIe 16X
Slot 1 - anything
Slot 1 - PCIe 16X
Slot 1 - anything
Slot 1 - PCIe 16X

If you only want the power of 4x 6950 I would sell them on ebay and buy a pair of 5970s which you can water cool and fit into just about any motherboard.

Really single slot watercooling only makes sense if you want to use >8 GPU.  I have considered putting 7 5970 into a single workstation all watercooled but the radiator would need to be massive, likely 9x 120mm in size and it would only work in Linux.

But.. I did.  Twice...


I am planning to water cool the cards, there's no other way to fit them in a single PCI slot space otherwise.

Specifically, I'm either looking at full-coverage 6870 waterblocks (these XFX cards share the exact same PCB layout), or going the slightly cheaper route of universal waterblocks and a full-coverage heatsink from swiftech:

Full block:
http://www.performance-pcs.com/catalog/images/koolance/vid-ar687.jpg
 
or
 
Universal + heatsink:
http://www.performance-pcs.com/catalog/images/swiftech/HD6950HS.jpg
http://www.performance-pcs.com/catalog/images/swiftech/mcw-80.jpg

The important factors here are

1) the motherboard 16x slot spacing prevents more than two 2-slot cards from being fitted directly to the board (and therefore fit reasonably inside a case)
2) I already own these cards, and selling them all used I'd maybe afford a single 6990 or possibly two 5970s
3) I'm looking at ~$550 for a budget watercooling setup already, I'd prefer not to sink even more money in if I can avoid it

This is all about squeezing what I've already got into the most compact, crazy setup I can create - not about efficiency or overall performance.  I don't want to buy even more components, the wifey would have my nuts.

Remember the cost of full coverage waterblocks they aren't cheap.  2x5970 give you pretty close to the performance of 4x 6950s.  It will only require 2 waterblocks not 4.  If you can get a pair of used 5970s for within $200 of what you can sell your 4x6950s for you likely are coming out ahead and won't need to do any desoldering.  The universal waterblocks might be fine for gaming but mining 24/7 @ 100% load puts insane pressure on the VRMs.  Without active cooling they are going to be 110C+ and that will put a limit on your hashrates.

You probably would be fine desoldering the top DVI port as long as you don't get the PCB too hot but I don't know of anyone who has tried that.

Alternatively if this if your board.  You can go with tri-fire (3x 6950) using 2 slots per card.
http://www.gigabyte.com/products/product-page.aspx?pid=3894#ov

You just need a case with 7+1 slots to fit a double card in the 7th slot of the MB.

The swiftech heatsink looks pretty beefy anyway, and I'll likely be mounting a couple 120mm fans to blow across the sinks.  If I can keep the VRMs under ~90C that's about what I'm getting now, which doesn't seem unreasonable considering the tiny little heatsink affixed to them now.  I'd prefer the 'universal' because a) it's about half the price of a full block, and b) I can reuse it in the future, assuming the standard mounts don't change significantly.
 

I'd be using a directed hot-air gun and solder wick to do this, so the heat would be pretty localized.  The trick here is there's no 'top' or 'bottom' header - it's a single block, so to remove one header I have to remove both.


It's exactly that board, actually.  In  all honesty I'm starting with water cooling just the two currently installed cards and working up from there.  I think I can go tri-fire with my current case with a quick dremel mod, I've got enough clearance between the bottom slot and the PSU I believe, even though I'm lacking the 8th slot.  Especially if the bottom card is also water-cooled.

Well it sounds like you have a plan.  I would start with 2 cards keeping dual slots, then try three cards modding the case.  If you want to go more extreme you could always mod the cards at that point to get four cards.  If you buy a larger radiator than you need now you can always expand with little more cost than getting an extra waterblock and doing the slot modification.

By "top" I mean the DVI port which is in the slot that the actual PCIE connector is not (i.e. next to the XFX air cutout).  

Here is a custom single slot bracket for HD 6990.  Just eyeballing it, it looks like it would work for your 6950 too
http://www.frozencpu.com/products/13220/pci-12/PCI_Single_Slot_Bracket_-_HD6990_PCI-BCKT-6990.html

http://www.frozencpu.com/images/products/main/pci-12.jpg

Here's an image I took when I did the circuit-write BIOS mod:

http://dl.dropbox.com/u/15821147/mining/radeon_6950_circuit-writer-mod-04.jpg

You can sort of see what I mean when I say the header is one 'block' - both DVI ports are a single 'thing', inseparable.  Any cards I mod this way would need a custom bracket, as both DVI ports will be gone.  I'd either just cut the current bracket and block the empty DVI port, or somehow craft a new one with just the HDMI/display port cutouts.

For the curious, you'll also notice the VRM heatsink on the left - this heatsink is the only difference between the 1GB 6950 and a 6870 layout (aside the GPU obviously).   The little green circle is around the (sloppily) modded BIOS chip.

Ouch. I gotcha now.  Yeah that is a rough mod.  You would need to desolder than dual DVI block and then find a compatible single DVI block and solder it on.  Not something I would want to consider. 

So it looks like triple dual slot w/ case mod is the easier route. 
If you want to squeeze in 4+ GPU I would go with dual GPU card before trying to desolder and resolder the DVI block unless you don't mind losing both DVI and just connect it via the DP plugs.  They make DP -> DVI cables.  Of course the resale value of card is essentialy $0 at that point.   ;D

Heh, yeah that's a given.  If I were really good I could potentially put it back on later, but the scorch marks might become a bit obvious ;)

I wouldn't miss the DVI, I'm using just the single HDMI header on the top card, and can extend to two more monitors with the DP connections.  I'd say any more monitors than that is overkill, but here I am trying to stuff 4 GPUs in a closed case :D

Actually I checked when I ran home for lunch and the PCI-e slot layout lets me squeeze them all in with only modding two of the 4 cards, so the 'main' card in the crossfire setup will still have all it's display headers intact.

Well sounds like you got the game plan.  Watercooling is spouse friendly. 
My wife asked my why I didn't do it years ago.   ;D

Now that the weather is getting cool she commented it is like a really quiet heater.  This summer I am going to try and rig some "shelf" to put my radiator outside the office window and dump the heat directly outside. 

One thing I would recommend if "Tygon Silver" anti-microbial tubing.  It is soft and flexible but still retains its shape.  I run pure distilled water w/ Tygon tubing and silver coil in the resivour.  No need for toxic goop and the water is still clean 9 months later. 

I was going to go with the DangerDen tubing packs, seem to be highly regarded, nearly or just-as flexible as the Tygon, and way cheaper. 

I'm planning on getting one of those 9x120mm monster radiators so I'll have plenty of headroom for nutzo overclocking and such, and the CPU will be part of my loop as well.  Good call on hanging it outside during the summer!

Yeah I am thinking of getting one of those someday.  Expensive but certainly has a lot of cooling capacity.

Nice watercooling tips here but too bad it is not really that cheap.

Full coverage waterblocks cover entire card including all VRM, GPU, PCIe bridge, and other components.

Immersion cooling works but isn't economical.  There are very few low conductive fluids that have higher thermal conductivity than plan ole water.  The few that are have incredibly high price tags.  Mineral oil is a good medium cost heat transfer fluid but is roughly 10x the cost of distilled water and has 1/4 the heat capacity.

Not sure where you get the idea that waterblocks can't keep video cards cool.  My GPU are <50C and VRM are <70C with a 18% overvolt.  

What limits further overclocking isn't temp so some ultra complicated and expensive immersion rig isn't going to get higher throughput.

What limits further overclocking is:
1) I don't have any desire to push higher voltage through card which is necessary to push clocks over 1Ghz.
2) As voltage increases power consumption increases by the square so I don't max overclock my cards anymore due to simple economics.


Thermodynamics tells us that would be bad.  Any immersion fluid while it may have higher conductivity than air has much less conductivity than copper.  You would need an astronomical flow rate (think giant sump pump) to avoid the chip cooking before it can transfer that thermal energy into the fluid. 

Air is a fluid.  You already have immersion cooling with air.  We use copper heatsinks because they can transfer that thermal energy from the die very quickly.  Even waterblocks which use have thermal conductivity have a decent amount of copper to spread out that point energy source.  Remember a GPU not only emits 200W+ of thermal energy it does so from an area tinier than your finger nail.

Sometimes I think you're just making up half these colloquialisms on the spot :P

Anyway - submersion is a viable solution, the trick to keep the VRM/RAM/GPU from overheating are big fat heatsinks on the chips - they work the same submerged as they do in air - and pointing an (also submerged) fan at them/on them to push fluid around.  

In the same respect the stock HSF boxes would work, but you'd burn out the fans pretty quickly moving around anything more viscous than air.  Of course, this is still combined with pumping all the fluid through an external radiator.

I think running a few tubes around and spot-cooling is a little less messy though ;)

Mineral oil cooling for bitcoin. Anyone try it ?

I'm not sure anyone's actually taken the leap, but it's been talked about a few times.

The general consensus is that it's not worth destroying your cards for the ePeen factor.  Once you dunk them in mineral oil you're certainly not going to be able to resell them, your warranties are null and void, and they become rather difficult to clean up and use outside the mineral oil bath.

There's no reason it wouldn't work, but the downsides tend to steer people away.

Well would they corrode or something if wash them with water and let them dry :P ?

Probably not if you used distilled water, but mineral oil is hydrophobic.  You'd need some kind of detergent to bind the oil, like dish soap. 

You wouldn't be able to just dunk it either as the water would seep between the PCB layers and make quite a mess.  Come to think of it, the mineral oil probably will too. 

Any way you look at it, submerging your card in some kind of liquid isn't something you can just do on a whim and expect to turn back.

That is a valid point. Did anyone so far do this and test out of the PCB remains solid etc. ?

I wish I had the link.  There is an outfit that is selling computer aquariums that they have soaked in mineral oil.  They were on their second or third iteration, which each being something like 2-3 years and going strong.  I remember them saying even the fans were ok.  The only thing they didn't submerge were the power supplies and the hard drives. They mentioned trying a bubbler to help with the diffusion, but I can't remember if they said it helped or hindered. 

http://www.pugetsystems.com/submerged.php (http://www.pugetsystems.com/submerged.php)

These guys?

[edit] Guess that link is a bit old, but still relevant - they're up to 'V4' now: http://www.pugetsystems.com/aquarium-computer.php (http://www.pugetsystems.com/aquarium-computer.php)

Yes!

and I was wrong.  They did submerge the power supply as well.  Obviously this keeps things running cooler but the power use is gonna be much higher.  I don't know how you would make that work for larger scale mining operations without killing your bottom line.

Back to the OT if I may - it turns out those Swiftech universal GPU blocks end up taking up 2 slots as well.

I think I'm going to end up going with the full-coverage blocks just to keep things more open in the future.

DeathandTaxes, what fitting are you using to connect your 5970s blocks?  I'm having trouble finding what I need..

Someone needs to try this with mining ! EPIC !

Um I can double check when I get home but I believe they are these.

http://www.frozencpu.com/products/10742/ex-tub-667/Bitspower_SLI_Crossfire_Multi-Link_Adapter_Pair_-_Black_Sparkle_BP-BSWP-C47.html?tl=g30c101s873

and these:

http://www.frozencpu.com/products/10744/ex-tub-669/Bitspower_SLI_Crossfire_Crystal_Link_Tube_Set_-_2_Slot_Spacing_BP-CLTAC-S2.html?tl=g30c101s873

They work great but bad news is they don't make a 1 slot connector (i.e. no space between blocks).  


It looks like something like this would work for cards in adjacent slots.
http://www.frozencpu.com/products/8566/ex-tub-448/Feser_Crossfire_SLI_Multi_Spacing_Fittings.html?tl=g30c101s873#blank



Here is the whole category of toys:
http://www.frozencpu.com/cat/l3/g30/c101/s873/list/p1/Liquid_Cooling-Fittings-Accessories-SLI_Connectors-Page1.html

Thanks D&T, those are about what I was coming up with too, I wasn't sure if the '1-slot' they were referring to actually meant 'blocks in adjacent slots', guess it doesn't.  It looks like the Feser ones could work.  I was looking at some similar fittings from DangerDen but they only came in a pack with one set of fittings and three tubes of differing length - for $12... and I'd need 3.. 

All the fittings and connectors and odd and ends are adding up to be almost 1/4 of the total cost, it's insane!

I'm kind of thinking that I'll just go with tubing and compression fittings for now - I've read that it's a total pain in the ass to get the tubing length just right, but it should work just fine and won't impact flow rates much.  The XSPC compression fittings I'm looking at are pretty low-profile, so I think I can squeeze two of them and a small chunk of tube between the blocks. 

Of course I've thought lots of things lately and been totally wrong ;)

Compression fittings aren't exactly cheap either.  You won't save much.  For 4 cards you will need 12 compression fittings (excluding the "outside" fittings to connect to rest of network.  The small amount you will save will be paid in pure frustration trying to get it to fit together.  Also mounting the cards will be a lot of "fun". The SLI bridge connectors provide some strength to the multi-GPU package making it easier to install and uninstall (looks like one giant 6 or 8 slot GPU) :).

I think now you might be realizing why I said if you can get good price of 5970s then selling the GPU for 5970s makes sense.  You only need 1 waterblock for each set of GPU.  4 GPU = only 2 cards = 1 bridge not 3.  All those things start adding up.

Basically watercooling has such a high unit cost per element being cooled you want that element to be as expensive and powerful as possible.  Dual GPU cards are a perfect fit.

It's true, and if this rig were strictly for mining I'd be going with the 5970 hands-down.  From the benchmarks I've seen however, a pair of 6950s in CF handily outperforms it in most games - albeit at a higher total TDP and complexity, and of course I have the 1GB models instead of the 2GB used in most of the benches..  but since I've got them already, that's what I'm working with ;)

I use these Feser Crossfire / SLI Multi Spacing Fittings
http://www.frozencpu.com/products/8566/ex-tub-448/Feser_Crossfire_SLI_Multi_Spacing_Fittings.html?tl=g30c101s873#blank (http://www.frozencpu.com/products/8566/ex-tub-448/Feser_Crossfire_SLI_Multi_Spacing_Fittings.html?tl=g30c101s873#blank)
as the poster above sugested.

i have 6 6970's watercooled side by side and works great.

Dear god man.. do you have pics of that?

not yet.
i should take some pictures.

i have the rads mounted in my cold air ducts.
so it directly heats teh house as well :)

How do you get around the 8 GPU limit in AMD drivers?

He doesn't - 6970 is a single-GPU card.

DOH.  I could have sworn he said 5...970.

MadHacker grab some photos.  6 watercooled GPU is worth sharing.

6970's are only 1 GPU, your thinking of 6990's or 5970's
so i'm only using 6 GPUs total.
having a real problem getting 7 GPUs working in a single machine

Mad, are you doing serial (in one card, out to the next) or parallel (in all, out all)?

parallel.
SideWinder had a deal on the Crosfire/SLI fittings so i bought 12 of them for $4 each.
they work great.

You don't want to do serial if each waterblock is the same.  The water temp rises across each GPU load.  The last card gets the hottest coolant and yet you will still expect the same performance from it.  Parallel is much better.  The inlet temp for each waterblock remains the same.

IF you have dissimilar blocks (like GPU block and CPU block) you need to do serial (or separate loops) because otherwise water will "cheat".  If one block has higher resistance the water will flow around it through the other block which is not good.  If you decide to water cool your CPU also make sure it is before the GPU.  The GPU have higher thermal load and thus will warm the coolant more.  CPU are also less accommodating of high core temps than GPU are.

Am I correct that I only need 6 fittings for 4 blocks then?



IN - 
     ----------- (gpu1)
       |      |
     ----------- (gpu2)
       |      |
     ----------- (gpu3)
       |      |
     ----------- (gpu4)
                 - OUT     
             

Where each | is an SLI fitting?

that is correct.

I found this is easier to route due to power supply and case location.  It is also easier to drain and disconnect as both of the inlets to the GPU cluster are on the same waterblock.



I added an arrow > on each water block to indicate that the water flows across each water block.

You just don't want to do this

That will cook the GPU in seconds after booting.  :o

This is not entirely equal, your gpu 4 will get less flow. Although if your interconnects don't pose a lot of friction, it shouldn't be much of a problem.

True however flow usually isn't the limit of cooling capacity.   It takes a surprising little amount of flow to remove even 1 KW thermal load.  As long as your flow is high enough that it doesn't result in laminar flow you will be fine.  To anyone who doesn't know laminar flow is when the water becomes "smooth" and "efficient".  We don't want that because then only heat is transfered to the edge of the water stream reducing cooling capacity.  Modern waterblocks tend to create enough "chaos" that laminar flow won't happen unless flow rate is very very low. 

Most pumps have more than enough head to keep flow rate > 1 gpm which is more than enough to avoid laminar flow.

actual i have the same set up as you have listed above.
checking temps they are 45-53 degC
ambiant temp is about 28Deg.
6970 OC to 965, getting 432MH/s

Before you fill your reservoirs, be sure to read this: https://bitcointalk.org/index.php?topic=40325.0;topicseen  ;D

MadHacker, pics pics pics!

There are a lot of aquarium plumbing solutions that may help for these crazy custom configurations. They are also probably a fair bit cheaper for Guest fittings and valves and the like. You can also find some crazy gang-valves that will allow you to tailor the flow to each exchanger. www.marinedepot.com is a pretty good source to at least find pics of it...local is always faster and cheaper.

OH MAN.  Thanks for the laughs.  I love how you got "Bonkers" pissed off because he thought you were serious about the gravy.

Honestly people love to over think it.  There are no low cost fluids that have higher thermal conductivity than distilled water.  Distilled water is cheap, has low conductivity, high thermal conductivity, is easy to pump, and is non-reactive.  Kinda hard to beat that ... well except maybe with Gravy.

unfortunatlty distiled water startes out not very conductive... but over time it picls up ions from the metal in the blocks...
then 1 leak and all can be toast...

I have RMA'd quite a few motherboards in the past because they couldn't swim (of course i didn't tell Asus that)

as for Picts... I'll try to get some taken over the weekend.

I am pretty sure Bonkers was a troll from SomethingAwful, but he/she sure succeeded in getting some people worked up in very short order.

Just ordered all the parts to start this up, thanks for all the tips guys :)

Why not like this?


      IN   
       |       
       -->------- (gpu1)
              |
       --<------- (gpu2)
       |     
       -->------- (gpu3)
              |
       --<------- (gpu4)
       |     
      OUT

I did a *lot* of reading and research last night, the consensus seems to be that it doesn't matter too much, but parallel flow through the gpu array improves overall flow rates through any other blocks in your system,  usually the CPU.

Ḃecause then gpu 4 will only get warm fluid, and run much hotter than the rest.

If each GPU caused a 1 deg rise in water temp then the inlet temp on GPU4 would be 3 deg higher.  It likely doesn't matter because water is such an effective heat transfer fluid that all the water in entire loop is within 1 to 4 degrees of water anywhere else in the loop.

Still parallel is considered a better approach.  Also with your layout you have 6 right turns.  Ever 90 turn is roughly equal to another 2 feet of tubing in terms of the resistance it creates.  Once again pumps today are so powerful and min effective flow rates so low it is likely academic but generally a parallel approach w/ minimum number of 90deg turns is the best.

I'd be more worried of the loss of turbulence on the blocks themselves because when you split your pressure with 4 cards in parallel, you're losing all that pressure. Fluid dynamics brah. The closer a moving fluid is to a solid object, the slower it moves, and slower moving fluid = less cooler fluid over the hot block pins over a period of time = hotter gpu

I honestly don't expect it to make much of a difference - if MadHacker's got 6 blocks in parallel and he's getting good temps, my eventual 4 cards should be more than fine.  As for the semantics..

Consider the specific heat of water -  about 4 joules/gram °C - or how much energy is required to raise the temp of 1 gram of water by 1C.  We'll take an example of 4 cards where each card is able to transfer exactly that 4 joules to a gram of water flowing through at 1gpm:

- In series, flow at 1gpm through all 4 cards, 30C water exits the loop at 34C, as each card has provided 4 joules or 1C.

- In parallel, with flow at 1/4gpm through each card, each raises the temp 4C (by providing 16 joules) - still the 30C water exits the loop at 34C.

So with the same overall flow rate through the loop, the water is raised by the same total temperature regardless of whether it's series or parallel.  

However, in the series example, the the last card gets warmer water - consistently 3C higher than the first card.  In parallel, each card gets the same incoming 30C water, so temperatures should be the same across all cards.

Additionally, in series each water block drops the overall pressure, meaning the pump must work harder to maintain 1gpm.  In parallel, the pressure drop should be roughly the same as just one block, perhaps even less if the total channel width through the blocks is more than the width of the tubing.  The pump doesn't need to work as hard.

So all together, parallel seems like the better plan - more even temps on each card, and a smaller overall pressure drop.

[edit] And I realize now I've just reiterated DeathAndTaxes' last post.. lmao.  Anyway as long as the flow decrease through the cards in parallel isn't so great that the water simply can't absorb any more heat, the cards should not 'run hotter' in parallel.

In theory but water is such an efficient cooler than even ~4 gpm is sufficient to transfer 1000W thermal load.  Remember temp is the difference between thermal energy in - thermal energy out.  If the water can remove all the thermal energy then temp will never rise above ambient.  However no radiator is perfectly efficient and that becomes the bottleneck not water flow.

Even entry level pumps is capable of 100+ gallons per minute.  It only takes ~4 gpm to remove 1000W of heat so water flow beyond a critical threshold is pretty much meaningless.  Lots of reviews that confirm this.  

There is ONE caveat.  If water flow drops low enough the stream will become laminar rathe than chaotic.  This was more a problem 10 years ago when watercooling wasn't as well designed.  Today waterblocks are designed to keep flow chaotic even at low flow rates (down to even 1gpm) thus cooling performance doesn't suffer.  If you ever get enough restriction that waterflow drops below the laminar threshold you will know.  It will be hard to keep cards cool at idle.  Cooling performance will fall off a cliff (I mean like a 90% reduction in c/w).

One last thing 100W thermal load doesn't mean 1000W at the wall.  Lets look at my system.  3x5970 & i5-2500K.  Pulls about 960W AC (at the wall) at peak.  However backing out PSU inefficiency that is about 844W DC load.  Now fans, HDD, memory, motherboard, and CPU pull about 150W so the GPU are "only" pulling ~700W.  1 gpm can transfer ~262W with <1 deg C rise in temp.  So assuming radiator could dump all that heat it would only require a flow of 2.6gpm to keep temp rise <1 deg C.  Of course the radiator isn't that good (4x120mm w/ 1200RPM fans to keep noise down) so the Delta-T is closer to 5C.

Desolder fail.

I attempted to pull the DVI header off an old nVidia 7800 - while I had success wicking up all the DVI pins, the larger through-hole pins that hold the header in place would not budge.

Doesn't look like I'll be squeezing these into a single slot unless I can figure out a better way to remove the DVI block.

/sigh

[edit] After doing a bit of research, it seems my iron was just turned down too low for ROHS/lead-free solder, so I'm going to give this another shot on the nVidia card.  Also going to pick up one of those desoldering vac pumps to try to simplify things, solder wick just wasn't cutting it for me. 

I'll try to post some pics if I manage to get it detached without completely botching things :)

you could just cut the pins. not like you will be sending the card back for RMA or reselling the card in teh near future

Might be a little tough - the pins holding the header in place are pretty beefy, and the header is flush with the board.  I suppose if I were an expert with a dremel/rotary tool I could hack it off.  Desoldering seems like the best bet still assuming i can melt the joint without melting the PCB :D

i'm prety sure that a small set of wire cutters will easily cut through the header.
however my experience with soldering is that great so that is why i thought of it.

Hooray for the interwebs, everything I need to water cool delivered to my door:

http://dl.dropbox.com/u/15821147/snaps/2011-11-17%2021.44.11.jpg


Here's a shot to show just how ridiculously huge the radiator is:

http://dl.dropbox.com/u/15821147/snaps/2011-11-17%2019.49.58.jpg


After much fighting with the DVI header on my scrap card, I've decided it's not worthwhile to risk the 6950s.  I managed to get it detached, but it was not pretty at all, and I'm fairly sure I damaged a couple of the traces in the process.

Instead I think I'll be going tri-fire instead of quad, trading my mobo for another with the 16x slots in a better arrangement, and selling off my spare 6950 to pick up the new mobo + 3rd water block.

I noticed you got Walmart water. :)

Purple cap = solid.

Same stuff I always use.  Works great and is cheap. 

PS I am jealous of that radiator although now w/ winter temps my 4x140 is doing fine.  By next summer I am going to need a bigger or second radiator though.

Hah, yeah, 'Great Value' indeed.  I was a bit worried that it'd be crap honestly, but I figured if I started to see a lot of mineral deposit I'd flush it all and double-distill my own.  Good to know it's not just a fancy label.

I modified the side panel of my case to mount the radiator last night, and it was total fail.  Whole rig toppled over on it's side as soon as I let go :D  

I'm guessing it's going to be somewhere around 10lbs or so once it's full of water and all the fans are attached, so off to the hardware store I go to rig a radiator stand.

Yeah the other colored caps are for "drinking water" and "filtered water" but they are junk.  Purple cap (and it is usually a whole $0.10 more per gallon) is the distilled.

Surprisingly good stuff.  Ultra low total dissolved solids.  Lots of people use it for aquariums because it is good quality and low price.   If you have any left over it makes great coffee because it has absolutely no taste.


Have fun and be sure to give us some photos once you get i running.

nice rad...
but for monster rad here is a nice one
http://www.frozencpu.com/products/14193/ex-rad-141/MagiCool_Limited_Edition_Monsta_420360_Triple_120mm_140mm_Xtreme_Performance_Radiator.html (http://www.frozencpu.com/products/14193/ex-rad-141/MagiCool_Limited_Edition_Monsta_420360_Triple_120mm_140mm_Xtreme_Performance_Radiator.html)
just to bad its only a tripple and not a quad. LOL

The radiator OP has gets better cooling capacity at lower fan speeds.  That radiator you linked to is nice and "dense" but high fin density means you need much higher air pressure to properly cool it.  That means faster, louder fans. 

My current rig is almost quiet but wasn't designed to handle 3x5970.  I want to expand that to 4x5970.  I am considering putting the radiator outside and "your" radiator would be nice for that (who cares about the increased noise) however if that doesn't work and I need to keep it inside I would rather like a the larger higher c/w radiator the OP got as it can be cooled with very low speed fans.

All about the right tool for the job. 

mysel i just use Koolance HX-CU1320VS Quads
$70 so its cheep.
and i throw on Scythe Ultra Kaze 120MM 38MM Fans
its in another room so don't care about noise.

Yeah I don't think the wife will go for that.  She already is a sport for accepting 10GH in the garage and another 2.5GH in the office.  :)

well i can't hear the fans. the 9 5830's i have running drown out the sound.
have 2 more rads on order and 3 5970 waterblocks.,
now they are what i can hear and i'm 2 rooms away.
once they are watercooled it should be much more quite.

Yate Loons are for the win.

http://dl.dropbox.com/u/15821147/snaps/2011-11-19%2001.17.04.jpg

Even with all 9 fans spinning at full it's still 90% quieter than it was before.  Turned down low it's just barely audible, the wifey couldn't tell they were on until she saw them spinning :D

I managed to save the work I did mounting the rad to the side panel with a little ghetto wooden stand, it's rock solid now and keeps the case off the ground an extra 3/4" to boot.

Also got all the water blocks mounted, tomorrow is modding the case to mount the pump in a reasonable spot, and getting it all plumbed up.  Hoping to leak-test tomorrow night :)

Looks awesome.  You have convinced me on getting that radiator.

BTW what fan controller are you using.  Some have a problem w/ the current of 9 fans.

It's an el cheapo Sunbeam: http://www.newegg.com/Product/Product.aspx?Item=N82E16811998808

It's 20W per channel, 4 channels, and I've got 3 fans spliced together on a channel.  It does what it needs to do for now but I'll be looking for a better one soon, and/or modifying this one with better components.

The starting voltage for the yate loons is a little high, so if the controller is turned down too low on power-up they don't all spin up.  I can just crank the knobs up a bit to get them spinning, then adjust them down where they need to be, but it'd be nice knowing all my fans are spinning without having to check.

I'm also testing with a small (12V @ 1500mA) external power supply, which could be the culprit since it technically doesn't push enough current to power all 9 fans.  It may be a totally different story connected to the real PSU.

[edit] External power supply wasn't the problem, same thing happens hooked up to my internal.  Appears the fan controller just doesn't supply a high enough voltage at low - I have about 75% of the dial range to play with still, so no huge deal really.

Lots of progress yesterday!

Mounted all the blocks and cut out some of the harddrive bay wall to fit the pump:

http://dl.dropbox.com/u/15821147/snaps/2011-11-19%2016.57.52-2.jpg


Then I drilled a hole in the case wall where the radiator is mounted for the fan wires.  Here's a protip:

Keep all power tools 10' away from your radiator.

I feel so foolish looking back on it now, I had a bit of a mishap with the drill:

http://dl.dropbox.com/u/15821147/snaps/2011-11-19%2021.42.47.jpg

I was being "ever so careful", but I destroyed a few fins and nicked the water channel with the slightest misstep. 

Thankfully I was able to repair the hole with a little bit of solder, and it's now watertight again.  It's a good thing the radiator is so large, I don't think the missing fins will effect performance much if at all, though restricted flow through the patched channel may just a little.

After that little fiasco, I got all the tubing connected, and it's now leak testing and bleeding!

http://dl.dropbox.com/u/15821147/snaps/2011-11-19%2023.16.35.jpg

I can't wait to fire it up :)

I don't have any experience with watercooling, but isn't leaktesting possible before hooking it up? Well afterwards you have to check again if all your nuts and bolts are tightened correctly, but you could test missing rubbers or leaking cables etc beforehand, no?

Yes it is possible, but you'd need to leak test again after hooking it all up anyway, so why not go in running.

The only component that's powered on right now is the pump, so if it did happen to leak it wouldn't be the end of the world, I'd simply have to correct the problem and wait for it to dry out.

But after about 14 hours it's still leak free, so I think I'm good to go :)

Most leaks are going to be at the connectors so leak testing outside the rig is kinda useless and provides a false sense of security.   When you leak test the system is powered off so if there is a leak as long as you dry it completely before powering on it isn't a big issue.  Distilled water isn't very conductive.

So best thing is to connect everything, disconnect power to everything except the pump and then jumper the PSU to power on PSU.  Run everything for 24 hours, refilling resivour as you bleed out air.  Check for leaks and if good after 24 hours power on system for real.

Doing it that way I have never lost a component.

I have had a few times where the leak doesn't become evident untill the water gets warm.
but i to let the pump run for a few hours before powering up the machine. catches most little leeks.

Things did not go so well when I booted up - my first GPU was not making contact apparently and soared to 110C idle, shutting off completely within about 15 seconds.  I may have busted this card while troubleshooting.

The second wasn't making good contact, but was holding around 60C idle.

By comparison, my Phenom II 1090T was 28C idle for the short time I got to monitor it.


I highly recommend against the Koolance full-coverage blocks, at least for my combination.

While there is no full-coverage block for non-reference 6950s, the PCB design for reference 6870s is exactly the same as the XFX 6950s I've got, short one mounting hole.  All the VRM, ram, etc are in exactly the right spots.

The Koolance 6870 blocks I got are crap, however, as far as I can tell.  The mounting spacers don't line up properly with all the holes; 5 out of 9 mounts line up perfectly, the others are off by only a millimeter or so, but just enough prevent proper mounting.

On the stock heatsink the 4 mounting spacers around the GPU are flush with the top of the PCB (chip side), and XFX uses screws with springs that 'push through' to keep pressure on the GPU plate.  The Koolance block, on the other hand, has spacers that are flush with the *bottom* of the PCB, so the springed screws I have apply no pressure at all, hence the temperature fail.  Koolance does not supply any screws for these 4 mounting holes, even though they're the most important.

I had to drain my loop and re-do both GPU blocks, adding some washers to get better pressure between the GPU and the cooling plate, and I'm now leak testing again.  Hopefully it goes better this time around.

Unfortunately it was pretty difficult to find reviews for any of the 6870 waterblocks so comparison was hard when I picked them up; I went with the slightly more expensive Koolance blocks versus the 'cheap' WaterCooling blocks, figuring a more well-known brand and higher price would indicate better quality.  Lesson learned.


[edit]  While I was at it, I also replaced that awful tube between the GPU blocks and the reservoir - I had made it longer with the intention of being able to move the reservoir in and out of the case for filling, since it's mounted at the top.  Turns out this was applying uneven pressure at the GPU fitting, and it was starting to slip pretty badly.  I shortened it up almost 2" (!!) and it's much better now, and I can still move the reservoir easily.  I'm learning lots of lessons from this endeavor :D

I bought a koolance block for my 6990,
everythign lined up prety good however VReg temp was still way to high.
i ended up filing down one of the standoffs by a tiny amount to get better contact.
now the VReg temp stays below 60Deg (highest Vreg temp by GPUZ)
have it overclocked to 960Mhz and is running nice and cool. (45 & 51 Deg.)

I just bought 3 5970 EK blocks.
once i get them mounted up i'll posts some temperatures.

Looking much better now after redoing the GPU blocks.

I've been running prime95 torture, and two instances of guiminer for about 2 hours now:

http://dl.dropbox.com/u/15821147/screengrabs/wc-stresstest-20111121.jpg

This is with all the fans on high, so a grain of salt there, I'll mostly be running this with the fans turned down to a comfortable level, though it's really not too loud on full.

Water cooling is amazing, I should've done this ages ago! :)

Now the question is.. blue or red Mayhem dye? :D

http://dl.dropbox.com/u/15821147/snaps/2011-11-21%2020.38.59.jpg

Nice...
nice and cool under water...
and to get it more silent you just need to add more rad and then slow down your fans.
in theory with a big enough rad you can go with no fans... but that would be a pretty big rad.
myself i use a radbox and and have the tubing go through the wall so i don't have any fans in my machine buzzing away.

question what gadget do you have running on your desktop?
I like it.

It's Open Hardware Monitor [openhardwaremonitor.org (http://openhardwaremonitor.org)] - I tried tons of different apps and gadgets and found that to be the best.  It's very customizable, and free to boot.

I'm getting about 48-49C under full load with the fans turned down as low as they go; the pump hums a little louder than all the fans, so I'm definitely happy with that.  It's not silent, but it's pretty damn quiet :)

[edit] Actually the loudest thing in the rig: the kill-coil bouncing around the reservoir now and again :D

Is that little acrylic box (at least that's what I think it is) with hose leading to the intake of the pump the reservoir?

Yes it is.  It's a Swiftech micro-res.

I was going to go without and just use a t-line, but the reservoir setup was just easier.  I compromised with the micro-res instead of one of the big bubbling monsters ;)

How well does it seal and how often do you need to top off? I am just curious because distilled and DI uptake contaminants so easy...I have run into some problems with aquaria because of it.

It seems to seal pretty well.  It's got four G1/4 fittings so the unused ports seal up nicely with o-ringed caps.

As for topping it off.. can't say for long term but the level has held steady since bleeding out all the air.  It's only been about 24 hours now though, so time will tell.

ok third time trying to post this message... each time it timed out... got my Internet sux

I have used a T-Line for years, it worked and no extra cost for res.
i have since used the micro-res on 2 of my mining rigs, and like it, blead time is down to minutes instead of an hour.
have been running them for over a month and havn't had to top them up since day 2.
water levels don't apear to be dropping either.

the Open Hardware Monitor software looks nice...
not truly a widows gadget but it works as well as one.
think I will install it on my mining rigs.

I noticed you have your system running with the crossfire connectors...
do you get a performance hit doing this?
i supose you also you this system for gaming... duno

Honestly I think it not being a 'real' windows gadget helps - it uses very few resources and doesn't succumb to the shitty UI mechanisms forced on the native gadgets, way less buggy.


The miner instances I was running last night were bothered more by the CPU running at full tilt ;)

Without prime95 running the crossfire didn't seem to slow it down much, but guiminer is kind of crappy for watching that sort of thing.  This is definitely my 'gaming' rig, mining will be secondary unless the bitcoin prices jump up a bit more..  which is really a shame, it's a pretty damn quiet 720mh/s :D

Crossfire can be disabled on the fly in ATI controls anyway, so it's simple enough to just turn it on when I want to game.  That damn 100% CPU bug sucks however..  I may have it netboot linux if i really want to mine with it, in which case crossfire won't hinder it at all.

I have the same res. I top-off about once every 6 months.

Do you measure the TDS of the water? Distilled H2O uptakes contaminants from the air very aggressively. I am just wondering how often a 100% change-out is necessary.

Watercooling is a closed loop system.  There is negligible exposure to the air.  Filling the resivour to the max and refilling it infrequently will reduce the exposure less.  That being said I usually change the water once a year just for piece of mind.

myself i had my server running 2 years before changing the water.
a couple drops of Hydrex. and no wories of anything growing that i don't want.

Definitely don't have the tools to measure particulates, though a good friend is studying water management and may have access to some pretty sophisticated tools soon.  It'd be fun to know anyway.

TDS in distilled water is really only going to affect it's electrical conductivity, with only very minor affect on thermal conductivity. Short of things growing in it, which the kill-coil should prevent, it technically shouldn't ever be necessary to change it out. 

These systems, while well sealed, aren't perfect however, so I'm erring on the side of caution and will probably give it a rinse and refill once a year as well.

That is, once (if ever) I'm done toying around with it - I've got a replacement mobo and a waterblock for a third card coming later this week, so I'll be draining and refilling once again this weekend :)

That is a good point.  Most people who watercool do it as a semi-hobby and thus I don't think I have ever gone more than a year without making some change to the system.  Easy to simply drain, flush, and refill at that point.

Well this rig is finally 'done' :D

It isn't exactly the frugal miner of lore but it sure can play some games.

http://dl.dropbox.com/u/15821147/snaps/2011-12-02%2014.26.05.jpg

After squeezing 3 cards together, it turns out the waterblocks and mounting screws would've gotten in the way of putting them closer than 2-slot spacing.

http://dl.dropbox.com/u/15821147/snaps/2011-12-04%2008.59.21.jpg

I'm not sure why the tubes look brown in that pic.. bad reflections or something, I'm am not running sewage through them :P

And the coup de grâce:

http://dl.dropbox.com/u/15821147/snaps/2011-12-02%2020.02.38.jpg

The CPU idles ~24-26C and I haven't seen it go above 33C under load.  The 6950s idle ~31C and spike to about 46-47C while mining.  Ambient is ~22C (72F).



'Final' specs:

Corsair HX750
Gigabyte 990FXA-UD5
8GB Gskill Sniper @ DDR3-1600
AMD Phenom II 1090T @ 3.9ghz
2x OCZ Vertex 3 120gb SSD, raid-0 ( >1000mb/s! )
And of course 3x XFX Radeon HD 6950


Now to go find a game I can't run at ultra / 1080p :D

[edit] That freakin apple sticker keeps jealously sneaking into my pics..  funny thing is I own exactly zero apple products..

So the rig has been running for a little while after adding the third card, temps are excellent and performance is stellar, but..

That brown tint in the tubing is actually there - at the time it only showed up like that in the pictures so I assumed it was just poor lighting, but now it actually looks like that - all my tubing is coated with that nasty brown color.

Has anyone run into this kind of thing before?  I did get the usual white clouding that I was more or less expecting prior to this last upgrade, but brown?

I'm using only distilled water and a kill-coil in the reservoir.

I think it's either biological, or some kind of oxidizing contaminant, since it wasn't apparent straight away.. the pics above were all taken at around the same time, and you can see in the top pic there's no apparent brown in the tubing.  The second pic was after bleeding for about 18hr. Scratch that, the second pic was 2 days later.

The only new addition was the third GPU block, it's possible it might have been 'dirty' I guess, and thinking back I'm not sure I flushed it before adding it into the loop, but I didn't flush the first two either and the system ran 'clean' for quite a while.

I don't think (I hope not anyway) that it's biological - the kill-coil should've prevented a colony from forming, and if it were algae I'd expect to see a green or reddish tint, not brown..

Another possibility - at the same time I added that block and refilled, I took everything apart and modded the windows into the case.  There was quite a bit of steel chaff floating around (case was definitely steel, not aluminum), it's possible some of this dust got into the loop.  Would steel oxidize and make this color?

[edit] You might also notice the reservoir is crystal clear in that pic - it still is.  Whatever this is it's not sticking to the acrylic, but it seems to love the tubing.

Hard to tell from pics, but ferric oxides (aka rust) def. can cause such a brownish tint.
Steel dust from grinding would be a good source. Did you have the loop open while you were dremeling the case?

Aye, the rig was all taken apart with the components lying on a shelf ~15ft away. I'd plugged all the free tubes with cotton since I knew I'd be spitting metal dust everywhere, but it'd seem I wasn't careful enough.

I'm ordering some new tubing this afternoon, I'll be flushing it out and replacing all the tubing when it arrives.  In the meantime, is a little rust going to be a problem for my copper/nickel blocks and rad?

I'm thinking I'll need to take the pump apart and clean it out thoroughly - if it is steel dust it's likely been collecting around the magnetic fields in the motor..

i think if it was rust it would show up as discolored in the resevoir as well...
however why its discolored is a mystery to me..

My guess is that it's just not concentrated enough to be obvious in solution, and has been slowly depositing on the tube walls.  In that second pic it's obvious there's a tint, but it wasn't actually visible to me at that point.  I pondered then why my pretty tubing was showing up so badly in the pics. Over the next few days, more material oxidized and/or built up on the tube walls, making it more apparent visually.

Right now, visually the appearance is close to the second pic in my post above, but to the camera it looks like someone shat in the tank:

http://dl.dropbox.com/u/15821147/snaps/2011-12-09%2013.50.22.jpg

all my tubing looks terible.
always get a white build up, wel green actualy since i use a few drops of HydrX (swiftech stuff) in my loop
i never used silver so i can't say if that has any affect.
why it is brown is a bit of a mystery...
but since all your water in the resevoir is clear... or apears to be from the pictures you have taken...
myself i wouldn't wory about it...
then again i havn't drained/flused my loop in a year and a half in my main machine
water still looks clear in my resevoir, however my tubing is green so i can't realy tell what color the build up is,

Havent read the entire thread, but isnt 750W a bit low? That CPU alone at that clock could be pulling 150-160W at stock voltage (and possibly considerably more if you overvolted it). Add the motherboard, RAM, water pump, fans, and you are probably >200W, leaving 550W for the three cards. YOu dont mention overclocking the GPUs, but even at stock speeds thats cutting it close IMO.

It's right at the high water mark running everything at full tilt, so yeah it's a little undersized.  With all three cards mining and prime95 running my killawatt reads 760-780W at the wall.

But, I don't really mine with this rig, and have yet to actually stress the system running any of the games I've tried.  Generally it tops out about 430W on my killawatt playing Skyrim, 550-600W playing Witcher 2 which actually uses all 3 cards a little more.

If I did mine, it'd be in linux without the 100% CPU bug, I'd expect to see ~600-630W at the wall with just the GPUs spinning.

just a thought...
put a white piece of paper behind the tubes...
see if its just showing the background color of your case /MB (although unlikely).

color of tubing is probably from the flash. Turn it off, turn more indirect lights on.

Nope, still brown.


Nope, still brown ;)  Less brown, but still brown.

I can see that it's clearly brown, even in the tubing outside of the case where it connects to the radiator.

Those things definitely accentuate the color tint, however.   With more ambient light, a white card behind the tubes, and flash off it looks fairly clear.  It still looks like there's a heavy smoker living in my radiator, and it's still enough to make me want to flush out the system and replacing the tubing.

Prior to this go-round the tubing was a little cloudy/milky white, but it didn't look like literal crap.

I gave up a long time ago on "cool" colored tubes, dyes, UV lights, and general leetness.

I use Tygon Silver tubing.  Medical grade stuff that last and lasts.  Never gets brittle, or loses flexibility.  The combination of opaque tubing (no light for plant life) and silver coating keeps the water clear however it does look downright boring.