Bitcoin Forum

Hi, I have some experience building 6 gpu mining rigs, the mobo of choice use to be the Asrock h80 btc pro or the h97 Anniversary. As most of miners know, those boards are disappearing from the market, I don't even know  if it is a low stock issue which would be temporary, or if Asrock stopped their production.

Right now Im looking for a replacement, maybe a socket 1151  mobo, that is able to handle 6 gpu. I have an MSI Z170-A Pro. The board has enough PCIe to connect the 6 gpus, but with only 5 the system becomes highly unstable, and with 6 its a constant reboot festival.

So my question is, which modern Mother board that is still in production, and is proven to be able to handle 6 gpus  do you recommend???? or  maybe any of you were able to run a 6 gpu rig using a 1151 Motherboard (z170 chipset) and you could share how you manage to do it?

Thank you in advance for taking the time to read and answer to this.

what windows version are you using with your MSI Z170-A Pro setup ?

I tried an ASUS H170 Gaming board and only 3/6 slots worked.
4/6 was possible depending on the right slot assignment and UEFI Setup configuration but it wasn't really stable under mining load for longer periods.
5/6 was a disaster: the misbehaving SATA controller caused data corruption on the root filesystem up to the point where Windows became unbootable.
6/6 was better, it didn't even get through the firmware initialization, so it had no chance of corrupting data on the SATA drive.

I took it back to the store and left with a 6-slot Gigabyte Z97 Gaming 3 (v1.1) because they had one of those at stock. It works great so far (and it looks like a decent PC mainboard unlike those 80/90-series 6-slot Asrock boards, so it might be easier to sell when the time comes...).

https://www.aliexpress.com/item/90-for-new-Original-for-Biostar-H61B-H61-LGA-1155-DDR3-16G/32723336716.html

https://www.aliexpress.com/item/PC-computer-motherboards-BIOSTAR-TB85-ETH-LGA-1150-Intel-Core-i7-i5-i3-Xeon-Pentium-DDR3/32677397969.html

The boards mentioned above below are not LGA1151, of which the OP is ask.

The consensus that I've read in these threads is that the Z170 chipset does not handle more than 4 cards very reliably/at all.

Thats why I posted links to other chipsets, the Z170 chipset will only handle 3 or 4 cards depending on the board. OP stated "maybe a socket 1151  mobo."

I must agree too Z170 chipset is having problem with 3+ cards on most boards, its prolly bios related, but who ever is doing testing first disable all non needed IO and internal devices as:
- sound
- ports
- any add in hw as firewire, wifi, etc
- set PCI-e to x1 mode if possible

And only then test it, also better use win10 because it should handle 8 GPU with no moded drivers

With this board you need powered risers. The asrock btc boards have 2x 4pin molexes in the board to give extra power.

Asrock has made the decision to stop manufacturing both the Pro BTC and the H97 Anniversary.  The reason I was given was slow sales.  They will do a special run of the mobos if you have a large enough order.  I would expect the order to have to be a few $100k.
 

I wonder if anybody tried the workstation counterparts of the B150 and H170, namely and respectively the C232 and C236 chipset based boards (these are probably the exact same silicon but with microcode support enabled for some entry level E3V5 Xeon CPUs and ECC memory [when the Xeon or Pentium/Core CPU also supports ECC] on top of the "lesser" consumer CPU and memory parts).

These boards aren't exactly cheap in general (but neither too expensive compared to some of the "fancier" consumer parts, especially if they live up to their promise of being more stable and reliable than their consumer counterparts) and I am yet to see any SKU with >5 PCI-E slots (they might restrict the numbers on these boards for an inherent reason rather than wiring up countless slots regardless if they will work or not :D) but they might worth a closer look and try (search for a relatively cheap one with >4 slots, =6 if any and give it a try...).

I like the Asrock motherboard with the  2x 4pin molexes in the board to give extra power. that is much safer.

You actually need zero molex on the mainboard when you have all the cards on molex powered risers (the BIOS will probably demand it anyway and there is no harm in obeying).
It shouldn't be a problem to run 2 cards sitting directly in 16x slots either (if the PSU has decent quality ATX 24-pin wiring/termination and the motherboard was designed for 2+ graphics cards).

On the other hand, I managed to burn two molex/sata cables and a modular socket on a Be Quiet! PSU while running six RX 480 cards on molex powered risers.
Since the PSU had 5 sockets for sata/molex cables (it was actually 6 but I failed to realize that there is an extra, different looking socket, intended for a short molex cable which is there for motherboards with an extra molex but could be used as any standard molex...) and I had 6 cards, I decided to connect 2 cards per cables (6 risers to 3 cables in total).
At first, one of the modular sockets melted a bit with the cable connector getting badly burned in it. (It was still running but I noticed the smell, so turned it off and started investigating.)
Since the other 2 cables looked perfectly fine I thought it was a random fault (either I failed to push it into the socket properly, or the connector was faulty to begin with, etc). I was down to 4 functional sockets in total anyway (well, actually 5, though I believed it's only 4 but it was less than 6 now anyhow), I didn't really have a better choice but to pick another cable and continue to run with 2 risers per cables.
The next time I noticed some smell again, turned it off, and saw as another sata/molex cable (with 2 risers on it) started to burn out. This time, the socket on the PSU remained intact but the cable started to loose it's coating and the SATA plug itself was really hot.

So, these modular sata/molex sockets/cables are clearly not ready to handle this kind of load (2x75W or whatever these cards might eat from the PCI-E slot).
The PCI-E, CPU and most of the ATX 24-pin wires seem to be significantly thicker than these sata/molex wires. The former group of wires go from point-to-point, directly terminated at both ends while the latter ones are "chained" in a way which is (in hindsight) clearly not ready to handle this kind of load (it's basically a single long cable and all the SATA/molex plugs are simply pressed on them, cutting through the insulation and keeping the metallic contact thorough "pressured razor blade" kind of fashion).
It's also notable that the extra short molex, intended for direct motherboard connection, is also point-to-point, directly terminated at both ends (not a razor blade pressured and chain-able kind like all the rest).

The TL;DR lesson is that even though the ATX 24-pin connector/cable clearly has it's own limits, the sata/molex cables intended for hard/optical drives can be significantly weaker.

I recently replaced that PSU and wired the new one with 1 card per molex/sata cable from the beginning (it has 6 identical sockets for these kind of cables). I will probably use the slightly damaged PSU with 3 or 4 cards at max later on (when I will have some fresh mining profit to reinvest).

Ill give it a try, however i just cannot wrap my head around the fact that the z170 chipset has more pciE lanes compared to the h80 or h97, in every single connectivity feature the z170 is superior to the h97, and somehow the former performs much worse than the later.
I guess that it is a hardware issue, but mostly a uefi issue, so maybe a custom rom or manufaturer update will solve this, however i wasnt able to fin any information of anyone trying to  develop a custom rom for that purpose. And the bios updates do nothing regarding the 6 gpu compatibility issue.

Hey thank you so much for answering, I'm having almost the same issues with my  MSI z170-A pro. But what I find really interesting is the fact that the last batch of asrocks h97 anniversary that I bough (six of them from some amazon vendor called outlet pc or something like that), were presenting the same faulty sata behavior, but they did it  when I put the 6th gpu on the system. windows sometimes didn't want to boot and  the solid state disk wasn`t able to pass the seatools tests as long as you had the 6 gpu connected and the system managed to boot. If  i did unplug one gpu or put the solid state disk on another mother board of the same model but a previous batch  the disk did pass all the tests.

So i really like to know what causes those faulty sata behaviors, or how to stop them and at the same time keep using the 6 gpu

I wonder if genesis mining use those regulars MB or if they use some custom made Motherboards and uefis.

I'm using windows 10, as I have told before I have successfuly build several rigs before, like 30 of them with no problems whatsoever ( using the asrock h97 anniversary, until my last batch and the recent out of stock situation), it seems that there is some bios issue preventing the z170 chipset from successfuly handling more than 4 gpu , but i havent found a workaround yet ( to be honest iwasnt able to even find someone developing a solution for that)

This article alone won't help you fully uncover the whole problem (it's not even directly aimed at this exact issue of miners, just a general analysis):
http://www.tomshardware.com/news/intel-100-series-hsio-chipset,30210.html (http://www.tomshardware.com/news/intel-100-series-hsio-chipset,30210.html)
I think the author was on the right track in understanding the SkyLake platform's possible connectivity issues after piecing some important things together but he left the matter alone before a real final conclusion was formed.

I think the real problem is some undisclosed further limitation about how these HSIO lanes can be allocated to PCI-E or other devices:

http://media.bestofmicro.com/B/4/530464/original/Intel-Chipset-HSIO.PNG

As you can see, some HSIO lanes are fully or loosely pre-allocated by Intel. Some lanes can be configured as:
- USB only
- PCI-E only
- either PCI-E or USB
- either PCI-E or SATA
- Now look at that Gbe !!!

The weirdest thing on this graph for me is the limitation regarding the (built-in?) Gbe network controller's HSIO lane allocation. I would think it should either act like any other PCI-E device (freely allocated at any open PCI-E capable lane) or have it's unique fix preset like the USB or SATA controllers do (like USB#1 can only go to Lane#1 and SATA#7 can only go to Lane#20, etc). What kind of logic might be hiding behind that?

---> Take note about how you must pick 4 neighboring lanes starting at N*4+1 (like 5,6,7,8 but not 6,7,8,9) for PCI-E 4x devices.
Another strange thing here is how you can theoretically have as much as 5 separate 4x devices (if you are willing to sacrifice all SATA and Gbe but otherwise it's still 4 with 1 Gbe and 1+ SATA), yet only 3 of these kind of 4x devices are allowed to be used simultaneously.

At some point I though I figured this out: I can operate 3 PCI-E cards because I had 5 PCI-E 4x blocks, 1 allocated for SATA and 1 for Gbe (and/or some SATA at the same time), thus leaving me with 3 free PCI-E 4x blocks, all of which are capable of operating a single PCI-E device regardless if that is 1x, 2x or 4x
But... I also had an integrated USB3.1 and a sound controller which are both PCI-E devices! So, this 4x block = 1 device can't be true. Well, unless I was actually operating 2 cards on the PCH and the third one was in the 16x slot and it's wired directly to the CPU (not micro-managed by the PCH - but I think it is because the Z170 motherboards can be configured in 8+8 modes whereas H170 boards are limited to a single 16x slot but this could be an artificial limitation imposed at firmware leve but this would also explain why Z170 seems to handle 4 cards and H170 only 3 cards: it's 2 vs. 1 cards on the "direct" CPU 16 lines and always only 2 on the PCH).

And now, if you thought you almost figured this out as much as it's possible..., look at how SATA#0 and #1 can walk around the table! What does that supposed mean? At this point I am beginning to question if this table exists to shows the fixed limitations (more than the USB ones, that is) or it's merely an example which illustrates several possible (and probably recommended) alternatives at a single graph. So, my weak speculation is destroyed here.


------------------


If you are into some experimentation (either by virtue or out of necessity), it would be interesting to hear about any of these C232 or C236 workstation boards (yon don't need Xeon CPU and ECC memory, just plug in your cheap Pentium and regular non-ECC sticks if you wish):
http://asrock.com/mb/Intel/E3V5%20WS/index.us.asp (http://asrock.com/mb/Intel/E3V5%20WS/index.us.asp) (5 slots for moderate price)
https://www.msi.com/Motherboard/C236A-WORKSTATION.html#hero-overview (https://www.msi.com/Motherboard/C236A-WORKSTATION.html#hero-overview) (6 slots, expensive)
http://www.gigabyte.com/products/product-page.aspx?pid=5806#ov (http://www.gigabyte.com/products/product-page.aspx?pid=5806#ov) (6 slots, expensive!)

hi thanks for this clarificarion thread ... i have z170 asrock extreme 4 .. and win10 recognize 6 gpu but 3 of them are in conflict (code 12) ... i can run and work only whit 4 gpu :( i need change my mobo ... for 6 or 8 gpu how mobo need buy for one happy mining and no problem? thanks for help

but the problem is for z170 chipset or CPU Skylake at this point? thanks

Set Max TOLUD = 3,5 Gb instead Auto in BIOS.

Thats not even an option on most of those boards , never saw it on  the MSI and Gigabyte 2011-3 and 1151 z170  boards i have

on my asrock extreme4 i have TLOUD option ... but in Auto all work fine but i can install only 4 GPU ... when i set TLOUD on 2,5 or 3,5 Gb on my debud display of Mobo error 5d or 55 (Memoy Error) ... i have installed 16 GB DDR4 2600Mhz (2x8) Gb.... what is the problem? thanks to all

I have MSI Z170A krait gaming 3x with 7 PCie , but I can't run more than 4 GPU  :'(

yeah i had the same one , POS chipset i had six new boards of those sent them all back to new egg fuck that.  What sucks is the 6100 i3 is a freaking awesome proccessor for $99 and we cant use it, it has hyperthreading four proccessors show up and its fast easily clocks for 4.0GHZ on the stock cooler.

Had to spend all six boards and six proccessors back due to this mess up by intel and ordered the MSI gaming 5 boards with we dual core pentiums what a shame.

This board has created many problems for me >:( I hate the creators of this miracle .

Reset bios to default settings, when set only TOLUD option. Try to boot.
If it does not help? then try to do the same with only one GPU.

GIGABYTE 990FXA-UD3 Rev.4 and rev.5 have 6 PCIE slots and work perfect!!! Used by me with 4 X 7990 and two 970 GTX! All 10 GPU work perfect.

Is that in Windows with 10 GPU?

Most would not mix AMD dual gpu cards which each show up as 2 gpus and Nvidia cards which is why you reached 10 in windows 10.  The driver for a single type of gpu in windows 10 is 8 gpus.

GIGABYTE Z170-Gaming K3 with latest BIOS

For anyone else out there with this issue, update your bios using one of the 3 methods outlined in this great video: https://www.youtube.com/watch?v=ze4Dg11W3hU

I used @BIOS and it was easy :)

I am trying to get a board to run 6 GPU, and here i can only find this models: ASROCK Z97X KILLER; ASROCK Z97 EXTREME4.
This two motherboard can work easily with 6 GPU?
Thanks

Yes. They can.