GPU & CPU Mining Rig (With added virtualization lab functionalities)

[afrik]

All,

I need to buy a new virtualization lab server as my current dell blades are dying - this for the purpose of studying, home lab, etc.)

Because I can put most of that purchase against company expense, my plan is to combine GPU Mining Rig & Virtualization Host. When not in use for Lab/Virtualization I'd like to utilize the CPU for CPU mining (CryptoNight (?), etc)

I'm planning to buy a modern motherboard with a good CPU for the purpose of Type 2 virtualization (I'm fully aware that T1 hypervisors provide higher performance/availability/security etc. than T2) and CPU mining as well as up to 4 or 6 good graphics cards for mining (probably ETHash).

Any ideas on recommended hardware would be most welcome. Sadly, I have lost touch with hardware quite a while ago and not sure where to start.

My limit is around $3000 / £2500 for the hardware but if recommended/needed can probably exceed that.


Many thanks for any advice.

Tom

[MATHReX]

If you are looking for Ethash algorithm particularly, I'll suggest:
RX570 for USD 230-270
RX580 for USD 240-290
GTX 1070ti for USD 500
VEGA 56 for USD 459

Pick any GPU from above, they all perform extremely well on Ethash algorithm.
As you have price restrictions, I'll suggest you go with 7x RX570 or RX580s.
That'll give 210 MH/s on Ethash algorithm.

Now pick ASUS Z370-F which is a really good motherboard with easy 7 GPU support.
For CPU, I'll suggest going with 8th Gen Coffee Lake Processor (i5 or i7)
16 GB DDR4 2133MHz RAM
250 GB SSD and 2 TB 7200RPM HDD should be good for storage
Dual Corsair RM850x for PSU.

[afrik]

Thanks MATHReX!

If you are looking for Ethash algorithm particularly, I'll suggest:
RX570 for USD 230-270
RX580 for USD 240-290
GTX 1070ti for USD 500
VEGA 56 for USD 459

Pick any GPU from above, they all perform extremely well on Ethash algorithm.
As you have price restrictions, I'll suggest you go with 7x RX570 or RX580s.
That'll give 210 MH/s on Ethash algorithm.

I will see what's left in the pocket after purchasing mobo/cpu as well as what is available on the market

Now pick ASUS Z370-F which is a really good motherboard with easy 7 GPU support.
For CPU, I'll suggest going with 8th Gen Coffee Lake Processor (i5 or i7)
16 GB DDR4 2133MHz RAM
250 GB SSD and 2 TB 7200RPM HDD should be good for storage
Dual Corsair RM850x for PSU.

I didn't know that Intel has gen 8 CPU. So far I was looking at Ryzen 7 1700 or 1800x but I could not find a mobo that would provide both: working 6/7 GPU ports and would be good for virtualization purposes (e.g. additional RAM banks, 2 LAN cards, although I could live without that)

Any idea how will new gen 8 i7 compare to Ryzen in mining scenarios?


many thanks

[NameTaken]

I use my mining rig also for running VMs, containers and compiling code. Host running Ubuntu 16.04 with AMDGPU-PRO driver for RX 470s but does not support R9 280X so I use libvirt and VFIO to pass that into a guest VM. IOMMU groupings:

Code:

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:190f] (rev 07)
IOMMU Group 10 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:a296] (rev f0)
IOMMU Group 11 00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:a297] (rev f0)
IOMMU Group 12 00:1d.0 PCI bridge [0604]: Intel Corporation Device [8086:a298] (rev f0)
IOMMU Group 13 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a2c5]
IOMMU Group 13 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a2a1]
IOMMU Group 13 00:1f.3 Audio device [0403]: Intel Corporation Device [8086:a2f0]
IOMMU Group 13 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a2a3]
IOMMU Group 14 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]
IOMMU Group 15 03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804]
IOMMU Group 16 04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 16 04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series] [1002:aaa0]
IOMMU Group 17 05:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 17 05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 18 06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 18 06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 19 07:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:2142]
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 00:01.1 PCI bridge [0604]: Intel Corporation Sky Lake PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 1 01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 1 02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 1 02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 20 08:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 20 08:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 21 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:67df] (rev cf)
IOMMU Group 21 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aaf0]
IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a2af]
IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a2ba]
IOMMU Group 4 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a282]
IOMMU Group 5 00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a2e7] (rev f0)
IOMMU Group 6 00:1b.4 PCI bridge [0604]: Intel Corporation Device [8086:a2eb] (rev f0)
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a290] (rev f0)
IOMMU Group 8 00:1c.1 PCI bridge [0604]: Intel Corporation Device [8086:a291] (rev f0)
IOMMU Group 9 00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a294] (rev f0)

[wacko]

I didn't know that Intel has gen 8 CPU. So far I was looking at Ryzen 7 1700 or 1800x but I could not find a mobo that would provide both: working 6/7 GPU ports and would be good for virtualization purposes (e.g. additional RAM banks, 2 LAN cards, although I could live without that)

Any idea how will new gen 8 i7 compare to Ryzen in mining scenarios?

What kind of virtualization lab are you planning to build? How many guests and what kind of systems? In most scenarios, especially for home labs (learning etc) the main part to worry about is the RAM. There's just never enough of it as soon as you start adding those guests. Ryzen is alright, I guess, but intel i7 is a weird choice. I mean, they're both not very popular. Most people (except maybe those that are power limited = expensive electricity/noise requirements etc) build home labs around used server hardware. If you're on very limited budget then it's dual lga 1366, if your budget can stretch a bit then it's dual lga 2011. These platforms are popular for a reason: the boards and CPUs are cheap, and, most importantly, there's plenty of memory slots and plenty of cheap registered DDR3 on ebay. Tyan S7012 (s. 1366) is ~ $100 new on ebay, and that thing has 18 ddr3 slots. Intel S2600 (s. 2011) can be had for $150-200 used, and it has 16 ddr3 slots. For 1366 you can have a couple of 6-core CPUs, for 2011 they go up to 12 cores each. If you're planning to attach 6+ GPUs to that system, then s. 2011 is the way to go. Compared to that any desktop system like intel's s. 11xx or amd's ryzen is just.. well, not very attractive, to put it mildly. Only 4 memory slots, and you're forced to use expensive ddr4. If you want ECC you'll have to stick to expensive unbuffered modules. Only one onboard network adapter, and often not even officially supported by hypervisors like esxi. Not to mention you can run in all kinds of VT-D related problems with desktop boards. Proper server hardware is more straightforward in this regard.

If you also want to mine Monero then definitely go with s. 2011. A couple of decent CPUs can hash at 1000+ H/s.

[afrik]

Thanks all for responses.

I currently have an ageing esxi platform (Dell PowerEdge C6100 blades server) - extremely loud, big, power hungry. And I just bricked one of the blades trying to upgrade to unsupported bios (hoping for esxi 6/6.5 support)

For my new platform, I was hoping I could combine both virtualisation host and GPU mining (+ potential CPU mining when VMs are down).
In terms of OS I'd like to use WS2016 with HyperV &  Desktop Experience providing the GPU drivers will work.

I've had a quick look at Ryzen CPUs - it seems they don't support nested virtualization - so providing WS16HV can be used for mining, that leaves me either Desktop Intel CPU ( Gen 8, thanks MATHReX) or server type MoBos.

Wacko, can you please suggest a mobo that will support 6+ 'mining' GPUs? This is where I'm most lacking in terms of requirements/knowledge, there are different PCIe slots
Would this one be OK for my purposes?
https://www.ebay.co.uk/itm/ASUS-Z9PA-D8-S2600CO4-C602-LGA2011-Motherboard/172718149859?hash=item2836cdd0e3:g:vmAAAOSwDiBZN86t

Also, what 2 CPUs would be recommended that provide good balance between cost / virtualisation & mining power / electricity drawn?


Once again thanks for all the answers/

Edit:
The Asus MB I found above seems only support 5 PCIe.
Slot Location 2 : PCI-E x8 (Gen2 X4 Link)
Slot Location 3 : PCI-E x16 (Gen3 X16 Link), Auto switch to x8 link if slot 4 is occupied
Slot Location 4 : PCI-E x8 (Gen3 X8 Link)
Slot Location 5 : PCI-E x16 (Gen3 X16 Link), Auto switch to x8 link if slot 6 is occupied
Slot Location 6 : PCI-E x8 (Gen3 X8 Link), MIO supported

[afrik]

if your budget can stretch a bit then it's dual lga 2011. These platforms are popular for a reason: the boards and CPUs are cheap, and, most importantly, there's plenty of memory slots and plenty of cheap registered DDR3 on ebay. Tyan S7012 (s. 1366) is ~ $100 new on ebay, and that thing has 18 ddr3 slots. Intel S2600 (s. 2011) can be had for $150-200 used, and it has 16 ddr3 slots. For 1366 you can have a couple of 6-core CPUs, for 2011 they go up to 12 cores each. If you're planning to attach 6+ GPUs to that system, then s. 2011 is the way to go.

I found another board:  SuperMicro  X9DRD-7LN4F
https://www.supermicro.com/products/motherboard/xeon/c600/x9drd-7ln4f.cfm
6 (x8) PCI-E 3.0 slots - I assume that is OK for mining GPUs mentioned above?

[wacko]

Wacko, can you please suggest a mobo that will support 6+ 'mining' GPUs? This is where I'm most lacking in terms of requirements/knowledge, there are different PCIe slots

I guess intel s2600 should support 6, probably even more with splitters. I know for a fact that this one support 13 gpus: http://www.supermicro.com/products/motherboard/Xeon/C600/X9SRL-F.cfm
For a home lab I'd obviously go with a dual socket board though.

I currently have an ageing esxi platform (Dell PowerEdge C6100 blades server) - extremely loud, big, power hungry. And I just bricked one of the blades trying to upgrade to unsupported bios (hoping for esxi 6/6.5 support)

That's why people buy eatx server boards for home labs. You just put it inside a full tower case, install a couple of large coolers on the CPUs, maybe attach a couple of small fans on some of the board's rads... and then you've got a very quiet system. My s2600 board was consuming ~ 150W idle. That's of course more than modern desktop platforms, but still reasonable.

All of this, of course, is different if you're actually going to attach 6 GPUs to that system. You probably don't even need any case anymore since all you cards will be on risers and an open frame is just more suitable for this kind of setup.

I found another board:  SuperMicro  X9DRD-7LN4F
https://www.supermicro.com/products/motherboard/xeon/c600/x9drd-7ln4f.cfm
6 (x8) PCI-E 3.0 slots - I assume that is OK for mining GPUs mentioned above?

There are no guarantees that a board will just work with 6+ GPUs, especially when you add an extra level of complexity (virtualization — I assume you're going to be "passing through" the GPUs to a guest system?). For the most part dual socket 2011 boards are a safe choice for 6+ GPUs, but it's not always 100% working. That's one of the reasons people stick to desktop hardware for mining rigs, a lot of people used it, tested it and everyone just knows what works and what doesn't. With server boards you gotta do a lot more research.

[afrik]

Wacko, thanks!

All of this, of course, is different if you're actually going to attach 6 GPUs to that system. You probably don't even need any case anymore since all you cards will be on risers and an open frame is just more suitable for this kind of setup.

yes, no plans for a chassis.
just MB + 2 PSU (It says E-ATX, will ATX work?). Is it possible to connect 2 PSUs at the same time?

+ 6 Sapphire Nitro+ 580 8GB (following a discussions in a different discussion thread on this forum.

I found another board:  SuperMicro  X9DRD-7LN4F
https://www.supermicro.com/products/motherboard/xeon/c600/x9drd-7ln4f.cfm
6 (x8) PCI-E 3.0 slots - I assume that is OK for mining GPUs mentioned above?

There are no guarantees that a board will just work with 6+ GPUs, especially when you add an extra level of complexity (virtualization — I assume you're going to be "passing through" the GPUs to a guest system?). For the most part dual socket 2011 boards are a safe choice for 6+ GPUs, but it's not always 100% working. That's one of the reasons people stick to desktop hardware for mining rigs, a lot of people used it, tested it and everyone just knows what works and what doesn't. With server boards you gotta do a lot more research.

I'm planning to use Microsoft Windows Server 16 with Hyper-V enabled, so the Parent OS will be doing the mining job.

The board specification says 6 (x8) PCI-E 3.0 slots - in theory, is that good enough for  6 Sapphire Nitro+ 580 8GB ??

[wacko]

just MB + 2 PSU (It says E-ATX, will ATX work?). Is it possible to connect 2 PSUs at the same time?

+ 6 Sapphire Nitro+ 580 8GB (following a discussions in a different discussion thread on this forum.

E-ATX is a standard for size, not PSU connectors (it's "extended ATX", simply implying that the board is larger than a regular ATX). The power connectors are the same, just make sure the board doesn't require any proprietary connectors and only has the standard 24-pin ATX + two 8-pin EPS. It is possible to connect 2 PSUs, and it's probably even the way to go considering your GPU choice. The main problem in powering the rigs (at least for me) is not to get enough power but to get enough cables and connectors.

First of all, you'll want at least one PSU with two EPS 8-pin connectors. You could try to power the MB with two separate PSUs, but I've previously tried that and had some problems with stability of the system. I think it's not recommended, even read some comments out there that it could damage the board. So, the board should be powered by its own PSU. Meaning you'll have to get a PSU that has two 8-pin EPS connectors.

Technically you could get away with just one PSU running the whole rig, but you might run into some problems with cables. Your Nitro cards each require 8-pin pci-e + 6-pin pci-e. So, for 6 cards you need 12 pci-e connectors in total. Each riser also needs one power connector, and ideally you either want molex or pci-e 6-pin risers. If it's 6-pin risers and you're building it all with one PSU then you're in trouble cause even the higher end 1200W PSUs usually have up to 12 pci-e connectors in total, and all 12 will be spent on your Nitro cards. So if you go with one PSU then you'll only be able power your risers with molex connectors. And again, ideally you want one cable per riser. But none of the 1200W PSUs that I know of come with 6 cables with molex connectors. They usually have 2-3 cables with molexes and 2-4 cables with sata connectors. And that's the main reason to go with two PSUs: even though it's easy to get a single PSU powerful enough to run 6 cards, most of them don't have enough cables to properly attach those GPUs.

If you go with 2 PSUs it becomes easier. One 750-1000W PSU to power the board and some cards, the other 750-1000W PSU to power the rest of the cards. That is, of course, if you're 100% set on your GPU choice. Because there are plenty of cards out there with only 1 pci-e connector. None of my Polaris cards (rx470/rx480), for example, have more than 1 pci-e connector. Most of the nvidia gtx 1070 cards also have one 8-pin pci-e connector. If you go with such cards then you'll only need 6 pci-e connectors from your PSU, and it becomes possible to properly power the whole rig with just one PSU.

I'm planning to use Microsoft Windows Server 16 with Hyper-V enabled, so the Parent OS will be doing the mining job.

The board specification says 6 (x8) PCI-E 3.0 slots - in theory, is that good enough for  6 Sapphire Nitro+ 580 8GB ??

Theoretically yes. In practice, most likely.