Watercooled rig - from scratch to (hopefully) reliable operation. Build saga.

[Commie]

Disclaimer: This isn't a commercial project so please don't ask me about ROI, build budget etc - I wasn't taking that into consideration at all, just doing it for fun as I like to build stuff. I doubt the money I spent were spent wise LOL. However, I know there are folks around who run watercooled rigs so advice, tips and healthy criticism are always welcome.

It's hot where I live. Hot and humid all year round. So when I saw memory temperatures on my small 3080 rig started climbing up it was time to do something about it. Thermal pads replacement and more fans didn't really help much, cards worked stable but constantly seeing 98-104C temperatures bothered me a lot. Then I bumped into a youtube video where some guy ran 8 or 10 cards rig, watercooled, and was quite happy with results, so that was it, I decided to give it a go.



Major objectives apart from lowering the temps:

- Reliable
- Serviceable and as much modular as I can do, without the need to drain the whole system every time I add or change something.
- Relatively compact
- Ability to move radiators to the outside of the house if I ever decide to do so.

Then I started looking at what's available, I never built custom watercooling loops before. Here's what I ended up buying:

- EK Pro distribution block, https://www.ekwb.com/shop/ek-pro-manifold-2cpu-4gpu-acetal, allows to connect up to 7 cards. I plan to run 6 or 7 3080s in this rig.
- Koolance quick disconnect couplings like this one, https://koolance.com/qd3-ms10x16-bk-quick-disconnect-no-spill-coupling-male-for-10mm-x-16mm-3-8in-x-5-8in-black
- 420mm x 45mm radiator, a friend also gave me another 280mm x 300mm radiator
- Plenty of 140mm fans
- 10x16mm tubing
- XSPC D5 pump with reservoir (also from a friend) plus an XC600 pump as a backup to be connected in series
- Flow indicator, temperature sensor, fittings etc etc.
Sourcing waterblocks was challenging - 3 cards I had in the beginning of this project were all different brands, so I ended up buying 2 EKWB Quantum Vector blocks and a Bitspower one.

Having all that ordered it was time to think of a frame for all this. My usual 20x20 aluminum extrusion rig frames weren't of much use, it would take a lot of effort to mount all the stuff on them, so I ended up looking at wire racks as they give more mounting options. Bought one of those and then designed and 3D printed mounting hardware for it. This is what I came up with -



I wanted all cards and PC components to be above water loop to minimize potential damage in case of leaks etc, so PSUs, switches etc will be on the top, motherboard is side mounted and all the watercooling equipment will take middle and bottom levels. Here is what middle shelf looks like with motherboard on the side.



Designed and printed radiator mounts and fitted them between middle and bottom shelves. Undo 3 screws and easily remove/relocate both of them, or add another one. They are connected to the rest of the system via Alphacool extention kit https://skinflint.co.uk/alphacool-eisbaer-gpx-extention-kit-12563-a1602263.html
so Ia I want to disconnect and move them to the outside it'll be relatively easy.



That's it for now, I started servicing and mounting waterblocks on my cards today, will continue this thread once that is done. As I said before, thoughts, advice etc are welcome.

 

[senyorito123]

Please make sure to it that you have a strong bond of water proofing on your rig so it won't cause a trouble leaks towards sensitive components.
Setting up using water coolant is very complicated, it needs proper handling. When it runs successfully I believed your operation will become smoothly running without any problems on overheating.
My advice for you, choose the best rig to run so it's worth for your cooling set-up.

[alucard20724]

Disclaimer: This isn't a commercial project so please don't ask me about ROI, build budget etc - I wasn't taking that into consideration at all, just doing it for fun as I like to build stuff. I doubt the money I spent were spent wise LOL. However, I know there are folks around who run watercooled rigs so advice, tips and healthy criticism are always welcome.

It's hot where I live. Hot and humid all year round. So when I saw memory temperatures on my small 3080 rig started climbing up it was time to do something about it. Thermal pads replacement and more fans didn't really help much, cards worked stable but constantly seeing 98-104C temperatures bothered me a lot. Then I bumped into a youtube video where some guy ran 8 or 10 cards rig, watercooled, and was quite happy with results, so that was it, I decided to give it a go.


Major objectives apart from lowering the temps:

- Reliable
- Serviceable and as much modular as I can do, without the need to drain the whole system every time I add or change something.
- Relatively compact
- Ability to move radiators to the outside of the house if I ever decide to do so.

Then I started looking at what's available, I never built custom watercooling loops before. Here's what I ended up buying:

- EK Pro distribution block, https://www.ekwb.com/shop/ek-pro-manifold-2cpu-4gpu-acetal, allows to connect up to 7 cards. I plan to run 6 or 7 3080s in this rig.
- Koolance quick disconnect couplings like this one, https://koolance.com/qd3-ms10x16-bk-quick-disconnect-no-spill-coupling-male-for-10mm-x-16mm-3-8in-x-5-8in-black
- 420mm x 45mm radiator, a friend also gave me another 280mm x 300mm radiator
- Plenty of 140mm fans
- 10x16mm tubing
- XSPC D5 pump with reservoir (also from a friend) plus an XC600 pump as a backup to be connected in series
- Flow indicator, temperature sensor, fittings etc etc.
Sourcing waterblocks was challenging - 3 cards I had in the beginning of this project were all different brands, so I ended up buying 2 EKWB Quantum Vector blocks and a Bitspower one.

Having all that ordered it was time to think of a frame for all this. My usual 20x20 aluminum extrusion rig frames weren't of much use, it would take a lot of effort to mount all the stuff on them, so I ended up looking at wire racks as they give more mounting options. Bought one of those and then designed and 3D printed mounting hardware for it. This is what I came up with -


That's it for now, I started servicing and mounting waterblocks on my cards today, will continue this thread once that is done. As I said before, thoughts, advice etc are welcome.

definately like that distribution block.  I'll look into doing this for my four VII that are watercooled.  currently working on replacing the current thermal pads with carbon thermal pads... they have higher thermal conductivity.

I don't think this will fix the problem of the memory temps rising.  The reason i say this is because your gpu temps are fine.  You'll still run into the thermal temps on the memory rising over time as the thermal pads on the memory degrade over time.  just make sure you have good thermal pads when you replace them next.

try the thermal pads that came with your blocks... if values are bad for memory, try and find better thermal pads.
you can always source thermal pads from digikey and such.

[Commie]

I don't think this will fix the problem of the memory temps rising.  The reason i say this is because your gpu temps are fine.  You'll still run into the thermal temps on the memory rising over time as the thermal pads on the memory degrade over time.  just make sure you have good thermal pads when you replace them next.

try the thermal pads that came with your blocks... if values are bad for memory, try and find better thermal pads.
you can always source thermal pads from digikey and such.

I use Thermalright Odyssey 12.8 w/mk pads in all of my cards. You're right about them degrading, it's been half a year since I installed them and having looked at them today they'd be needed to change again in a month or two. Not crumble yet but start to dry up. Right after installation I was enjoying 86-88C mem junction temperatures though.

[alucard20724]

I don't think this will fix the problem of the memory temps rising.  The reason i say this is because your gpu temps are fine.  You'll still run into the thermal temps on the memory rising over time as the thermal pads on the memory degrade over time.  just make sure you have good thermal pads when you replace them next.

try the thermal pads that came with your blocks... if values are bad for memory, try and find better thermal pads.
you can always source thermal pads from digikey and such.

I use Thermalright Odyssey 12.8 w/mk pads in all of my cards. You're right about them degrading, it's been half a year since I installed them and having looked at them today they'd be needed to change again in a month or two. Not crumble yet but start to dry up. Right after installation I was enjoying 86-88C mem junction temperatures though.

half a year is pretty good... definitely better than stock.   i use the thermalright pads also.. they're better than the "blue" ones all over amazon.  those didn't last long at all.  the blue ones only lasted about a month. 

[fmz89]

that was epic and big budget if you trying whole gpu to water cooled in this method, better using 3M fluid with sub emmersion cooling, easier and lower cost, especially for small room and for large farm

[Wotan Wipeout]

This is what you can expect with ETH (3080 watercooled):

+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
| ID  GPU  Temp Fan    Speed      Shares   Core  Mem  Power  Efficiency  |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
|  0  3080   44 0 %  99.38 MH/s   5273/0/0 1470 10401 232 W  428.35 KH/W |
|  1  3080   44 0 %  99.38 MH/s   5201/0/0 1500 10401 225 W  441.68 KH/W |
|  2  3080   43 0 % 100.93 MH/s   5433/0/0 1455 10551 232 W  435.04 KH/W |
|  3  3080   35 0 % 101.93 MH/s   5296/0/0 1545 10652 226 W  451.02 KH/W |
|                            401.62 MH/s  21203/0/0            915 W  438.92 KH/W |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+

https://geizhals.de/alphacool-nexxxos-monsta-360-white-special-edition-35435-14193-a901781.html?v=k&hloc=at&hloc=de

Watercooled 3070s:
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
| ID  GPU  Temp Fan    Speed      Shares   Core  Mem  Power  Efficiency  |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
|  0  3070   48 0 %  62.20 MH/s   2779/0/6 1530  8080 120 W  518.37 KH/W |
|  1  3070   49 0 %  62.10 MH/s   2778/0/0 1080  8080 120 W  517.50 KH/W |
|  2  3070   46 0 %  62.21 MH/s   2751/0/0 1500  8080 120 W  518.38 KH/W |
|  3  3070   46 0 %  62.19 MH/s   2723/0/0 1425  8080 120 W  518.25 KH/W |
|  4  3070   44 0 %  62.20 MH/s   2759/0/0 1470  8080 120 W  518.35 KH/W |
|  5  3070   43 0 %  62.16 MH/s   2748/0/0 1350  8080 120 W  518.01 KH/W |
|                           373.06 MH/s  16538/0/6            720 W  518.14 KH/W |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+

Watercooled 2080s:
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
| ID  GPU  Temp Fan    Speed      Shares   Core  Mem  Power  Efficiency  |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
|  0  2080   43 0 %  44.34 MH/s   7454/0/5 1635  7949 130 W  341.10 KH/W |
|  1  2080   41 0 %  44.24 MH/s   7423/0/0 1515  7949 130 W  340.31 KH/W |
|  2  2080   46 0 %  44.66 MH/s   7566/0/0 1605  8000 130 W  343.56 KH/W |
|  3  2080   41 0 %  44.36 MH/s   7652/0/0 1575  7949 130 W  341.23 KH/W |
|  4  2080   41 0 %  43.33 MH/s   7401/0/0 1665  7799 130 W  333.33 KH/W |
|                           220.94 MH/s  37496/0/5            650 W  339.91 KH/W |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+

Most of my rigs are watercooled..... so if you ran into trouble, tell me.

[Commie]

that was epic and big budget if you trying whole gpu to water cooled in this method, better using 3M fluid with sub emmersion cooling, easier and lower cost, especially for small room and for large farm

Immersion cooling is a concept I like too, maybe will try it one day as well. What stops me now is a relatively high cost of commercially available equipment and liquid itself. DIY won't make it much cheaper.

[Commie]

This is what you can expect with ETH (3080 watercooled):

+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
| ID  GPU  Temp Fan    Speed      Shares   Core  Mem  Power  Efficiency  |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+
|  0  3080   44 0 %  99.38 MH/s   5273/0/0 1470 10401 232 W  428.35 KH/W |
|  1  3080   44 0 %  99.38 MH/s   5201/0/0 1500 10401 225 W  441.68 KH/W |
|  2  3080   43 0 % 100.93 MH/s   5433/0/0 1455 10551 232 W  435.04 KH/W |
|  3  3080   35 0 % 101.93 MH/s   5296/0/0 1545 10652 226 W  451.02 KH/W |
|                            401.62 MH/s  21203/0/0            915 W  438.92 KH/W |
+---+-----+----+---+-----------+----------+----+-----+-----+-------------+

https://geizhals.de/alphacool-nexxxos-monsta-360-white-special-edition-35435-14193-a901781.html?v=k&hloc=at&hloc=de

Thanks Wotan, I believe we briefly chatted about this few months ago when I was planning the project.

What are your ambient and mem junction temperatures? You're using 1 radiator for 4 cards, right? How do you control radiator fan speeds or are they 100% all the time?

[Wotan Wipeout]

Yeah i think you are right. My 3 Fans run 100% always.
Taking fans with more pressure allows more thinkness at the radiator.
Always used Kaze 3000, but they are no longer buyable.

I dont remember the ambient temperature.
Talking about the ram temperature takes us to the PCB design.
Some are just not plain. My last one was perfect and even without
highend pads its the coldest of all.

[FP91G]

You can do any cooling, but the most important thing is that this investment will pay off.
How much does such a water cooling kit cost for your mining farm?
Of course, if you design and assemble everything correctly, then such a cooling system will last a very long time, but you have to pay dearly for it.

[Wotan Wipeout]

My 1080 run since 5 Years and they have a higher core because lower temps.
So more hash, and it pays off.
On the power side you have a bit less power usage. Think about 6 or 8 gpus and each 3 fans,
adding a few fans for the case.
Watercooled i need only 3 Fans overall.
Besides, when you sell the GPUs the fans are brand new and thats a benefit too.

[FP91G]

Besides, when you sell the GPUs the fans are brand new and thats a benefit too.

This is an advantage if the seller deceives the buyer and says that the video card has not been used much and looks like new
And before selling, I change all the fans, because the old ones either work poorly or have been dismantled a long time ago.
A new fan for aliexpress costs 7-10 dollars. The video card also looks good, almost as good as new.

[Commie]

Time to update.



It's been 3.5 months since I turned this rig on with 3 GPUs at the start. My first impressions weren't that great, at the ambient temperature of 32C I only saw one card's memory junction becoming much cooler, another one dropped just 4-6C and the last one didn't improve at all constantly hitting 104-106C. My first thought was radiators weren't large enough for 3080s, so I found a local company that builds custom copper radiators, had one made at my specs and installed it outside the house. Funny that, this one was significantly cheaper than branded PC one LOL



So my temperatures improved... a little bit. I'd probably keep scratching my head over this, but after adding 4th and 5th card everything became clear. I have 2 Asus TUFs (they're the coolest ones), 2 cards with reference design (RE) PCBs and a FE one, which is the hottest. The difference is in waterblocks quality. I tried to same some cash and bought a Bitspower waterblock for one of my RE cards and a Bykski for the FE. Bad, bad mistake. Other cards had EKWB. As soon as I added another RE card with EKWB waterblock I could clearly see the difference, notice GPU3 and GPU4 on the screenshot. Besides, Bykski has terrible screws supplied with it, very tiny,  soft and easily broken.



Lesson learned, 2 more EKWB blocks are ordered, problem solved.

Another thing I figured out there's no need to order backplates to mount with waterblocks, there's no improvement for 3080s. GPU2 is running with EKWB backplate, GPU1 is without it. They are same cards.

My plan is to grow this rig to 7 cards (limited by distribution block connectors). Will I build my next rig watercooled? Not sure yet, I am looking towards A5000 cards now and they're designed for air only. But if I decide to go for non-LHR 3080/3090s again I'd definitely consider water cooling for them.

Watercooled i need only 3 Fans overall.

Wotan are you saying one 3 fan radiator is enough to cool your 4x3080 or 5x1070 rig? From what I see in my setup it's not the case. Also, what brand are your waterblocks?

I don't think this will fix the problem of the memory temps rising.  The reason i say this is because your gpu temps are fine.  You'll still run into the thermal temps on the memory rising over time as the thermal pads on the memory degrade over time.  just make sure you have good thermal pads when you replace them next.

try the thermal pads that came with your blocks... if values are bad for memory, try and find better thermal pads.
you can always source thermal pads from digikey and such.

Actually it did fix the problem and at current temperatures I expect pads to run 1-2 years before they need replacement.  Yes I used pars than came with EKWB blocks and they're quite good, I also service/repair my and other's cards sometimes and have a good stock of Thermalright and Laird 90000 pads of various thicknesses.

You can do any cooling, but the most important thing is that this investment will pay off.
How much does such a water cooling kit cost for your mining farm?
Of course, if you design and assemble everything correctly, then such a cooling system will last a very long time, but you have to pay dearly for it.

Agreed, it's expensive. Decent waterblock with proper fittings/tubes  will set you back around $300. But on the long run with added reliability it pays back. The rig runs super stable and I don't have to worry about broken fans, dusting radiators, changing pads every half year (and this one IS expensive) and most importantly - lifespan of my cards. Take good care of them and they will last for long, long time.

[Wotan Wipeout]

Yes, depending on the Fans and the temperature at the radiator 3 fans are enough.
I use server cases so always 1 good 3 fan radiator.
On my 3080s i have got ek waterblocks.

0: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.06 MH/s 228 W 54℃ / 0%
1: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.07 MH/s 217 W 54℃ / 0%
2: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.07 MH/s 222 W 52℃ / 0%
3: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10475 100.59 MH/s 210 W 45℃ / 0%
400.78 MH/s   917 W   

Nice setup!

My latest watercooled build, similiar to your...

0: Sapphire RX 6700/6700 XT 12GB 113-D5121200-L05 1300/1074 643 mV 49.67 MH/s 1300 mV/650 mV 99 W 48℃ / 80%
1: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 650 mV 47.04 MH/s 1400 mV/700 mV 94 W 51℃ / 80%
2: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 650 mV 47.13 MH/s 1350 mV/700 mV 91 W 50℃ / 80%
3: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 650 mV 47.13 MH/s 1300 mV/650 mV 88 W 51℃ / 80%
4: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 650 mV 47.13 MH/s 1300 mV/650 mV 88 W 53℃ / 80%
5: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 700 mV 47.10 MH/s 1300 mV/650 mV 93 W 48℃ / 80%
6: Sapphire RX 6700/6700 XT 12GB 113-D5121100-L02 1300/1074 650 mV 47.13 MH/s 1300 mV/650 mV 89 W 52℃ / 80%
332.33 MH/s   752 W

[Commie]

Yes, depending on the Fans and the temperature at the radiator 3 fans are enough.
I use server cases so always 1 good 3 fan radiator.
On my 3080s i have got ek waterblocks.

0: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.06 MH/s 228 W 54℃ / 0%
1: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.07 MH/s 217 W 54℃ / 0%
2: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10425 100.07 MH/s 222 W 52℃ / 0%
3: Msi NVIDIA RTX 3080 10GB 94.02.42.00.F7 1155/10475 100.59 MH/s 210 W 45℃ / 0%
400.78 MH/s   917 W   

Nice setup!

I guess you're using Hive, so no memory temps on Nvidia?  Lock your core at 1080-1100, it should improve a bit. I also guess you're in much cooler climate than I am.

Thanks for the compliment. I'm also planning to but that EK double pump you mentioned while ago, just to have it as a spare. But right now 2 pumps in series do the job just fine, backing each other up.

How often do you change the coolant?

[Wotan Wipeout]

I am located in germany and use simplemining. Tested the ram temps in windows so i know all is safe.
Could push the ram higher, up to 104 mhs eth.

Never changed the coolant. Over time (months) it will become less a bit and i fill it up. Checking the water
all 4 to 8 weeks. I would go for DP Ultra coolant with a color. (i have always green)
This will help in case you have a leakage.

My 1080s run watercooled since they came out, years ago.

Check this out:

https://www.amazon.de/gp/product/B07PDZ9H5X/ref=ppx_yo_dt_b_asin_title_o08_s00?ie=UTF8&psc=1

Under construction:

[adaseb]

I also would love to have the option of doing this. However the bulk space, high cost it just wouldn’t make sense. I had a Antminer long time ago it was called the Antminer C1 and it was actually like an Antminer S3 but water cooled. It came with the heatsink and everything. And that miner was so quiet you didn’t even know it was there.

You could have that miner next to your bed and you wouldn’t have issues with sleep or anything. With GPUs it would probably be even more quiet since their surface is much larger so it would require less fan speed. It would be a neat quiet operation.

[Commie]

Never changed the coolant. Over time (months) it will become less a bit and i fill it up. Checking the water
all 4 to 8 weeks. I would go for DP Ultra coolant with a color. (i have always green)
This will help in case you have a leakage.

I use distilled water with ~5% of antifreeze in it to prevent growth. So far so good.

Check this out:

https://www.amazon.de/gp/product/B07PDZ9H5X/ref=ppx_yo_dt_b_asin_title_o08_s00?ie=UTF8&psc=1

Yeah, I'll get a similar one soon. Does it require external power or works only from HDMI/USB?

Under construction:

LOL this setup reminds me of something 7 cards are already in, are you planning to add 2nd distribution block? What quick disconnects/angle fittings are used, are these Alphacool (since you're in Germany that'd be my first thought)

[Wotan Wipeout]

You only need USB to power it. And these cheap things have a very good display.

This rig heats my home - silent - until the winter ends in about 1 or 2 months.
The GPUs will be placed in a server case than and mine somewhere else.
In a server case i can put up to 8 GPUs, but i think i stick to seven gpus.
The rig itself will stay for new watercooled gpus until i have enough for the next rig.
This is the reason for the big block.