[Guide] How to run up to 13 GPUs on most motherboards

[nitrobg]

After lots of researching, trial and error, I have managed to run 10 GPUs on my ASRock X99 Extreme6 using two PCI-E multipliers for $12 each off eBay or AliExpress. It's likely that I'll be able to run up to 13 if I had more cards.

You need:

• Linux or Windows 10 with 1709 update
• Above 4G Decoding enabled
• If using Windows - install it on UEFI mode with GPT partition table
• PCI-E set to Gen1 or Gen2
• Intel motherboard on H61 chipset or newer; AMD motherboards with UEFI.

If you are missing either of these prerequisites, it is quite possible that it won't work. Motherboards without Above 4G decoding can run up to 6 GPUs, more are unlikely. Older motherboards with a legacy BIOS can run up to 4-5 cards max, assuming that you don't encounter IRQ issues. The hard limit seems to be 13 regular cards, unknown more mining cards.

Potential issues:

• If your install is on Legacy MBR mode - you can convert it without data loss using the built-in MBR2GPT.
• If some of the cards are missing - make sure that they're not in Device Manager - Other devices and waiting to have their driver installed.
• If the cards are recognized but output error codes in Device Manager - try nuking your driver using DDU and then reinstall it. Make sure that you patch the AMD driver if you are BIOS modded.
• If the doesn't boot or doesn't recognize the cards altogether - try switching the PCI-E slot. My motherboard refuses to boot if I plug the multiplier on the first PCI-E x16.

More troubleshooting tips - try updating the BIOS to a newer version, disable unneeded peripherals (audio, additional SATA controllers, whatever), disable iGPU, set PCI-E to a different Gen, even reset the CMOS. HackFlags registry trick might work but I've never tried it.

YMMV, of course - please let me know if it worked for you. So far I've tried it on ASRock X99 Extreme6 and Gigabyte GA-H110-D3A (>6 cards), MSI H110M PRO-VD and several other mATX motherboards for 6 cards.


Donations:
BTC: 1P4oRbB76QGw3dzmhcxSFo1B9pxSswipfs
BCH: 1M6TaEKzKYQEQdoBXRbPhzm3m23iYxFhyU
ETH: 0x2211d972934ec7a66225883355e9d23f848f634a
ETC: 0x64b30f6f1acd4ab9f1e5d6da7e7d9d22d506c8c4

[uskovartem]

What the reason to install GPT, but not MBR ?

[nitrobg]

What the reason to install GPT, but not MBR ?

AFAIK it is needed for a 64-bit address space on Windows, it is used to allocate memory for >6 GPUs, otherwise you'll get error code 12 or not boot at all.
Can't find the right source but eGPU users encounter the same problem and have a similar workaround.

[amit_talkin]

Hi,

I recently bought PCIE 1 to 4 expansion cards from below.
https://www.ebay.com/itm/USB-PCI-E-1X-Expansion-Kit-1-to-4-Ports-Switch-Multiplier-Hub-Riser-Card/122820612734?hash=item1c98add27e:g:TD4AAOSw9gRaFAFL

However, on my old system with chipset X58 and also on new H97 chipset, only maximum 6 GPUs working. I tried every possible settings in BIOS but no luck getting more than 6 GPUs working on any of these chipset.
I am now thinking to move towards newer chipset like B250. If I purchase lower end mobo. with 1x PCIE 16 and 1x PCIE1x slot , will it work? Do newer chipset support 8 GPUs? I am not talking about special mining motherboard, just a regular one.

[nitrobg]

X58 is too old and is likely limited to 5-6 GPUs.
H97 should work though. Does it even boot with more than 6 GPUs? If it does, your OS is likely the issue.

You might be able to get 5-6 cards with the small boards, but YMMV. The chipset doesn't really matter I think, the multiplier combines the signal of all 4 cards into 1 lane.

[grosminer]

Great information. Thank you for sharing.
Didn't know about the partition mode

[amit_talkin]

X58 is too old and is likely limited to 5-6 GPUs.
H97 should work though. Does it even boot with more than 6 GPUs? If it does, your OS is likely the issue.

You might be able to get 5-6 cards with the small boards, but YMMV. The chipset doesn't really matter I think, the multiplier combines the signal of all 4 cards into 1 lane.

Yea I figured it lately that x58 can only handle up to 4 GPUs.
I have a Gigabyte B85M-Gaming 3 motherboard, it works fine with 6 GPUs, but with 8 GPUs it just fails to even load into bios. That even with 4 GPUs plugged into PCIE1x slot via extender! so this means chipset is limiting its usage to 6 GPUs.

[hack4love]

What amd driver you using  . What new in 1709 update I think the and driver not allowed more than 8 card not the windows

[CarlOrff]

Nice search : thank you for sharing it.
Somone try this PCIe multiplier with the ASRock H81 Pro BTC ? With how many GPUs ?
I'm curious about this solution :-).

[amit_talkin]

What amd driver you using  . What new in 1709 update I think the and driver not allowed more than 8 card not the windows

I have 6x1070 1x1080 and 1x1080ti. OS is not related at all as it doesnt even pass bios screen.

[dominuspro]

Thank you for sharing... But the main problem is where to get 13 gpus for normal price in EU

[amit_talkin]

Okey I got it working on my MSI Z170A Gaming M5 motherboard. Got all 8 GPUs detected and working on Windows 10 1709. Did bios update and I got "Above 4g Decoding" option. Set it to enabled, and set PCIE speed to gen2.
Popped 1to4 risers to each PCIEx16 slot, plugged 4 GPUS in each. You don't even need UEFI , my ssd had generic MBR partition and it worked fine.
So, still confused if it will work on other chipset. Currently Z170 motherboard is costlier than current gen. Is there anyone who can confirm this working on entry level B250M motherboards?

[amit_talkin]

Update : I just purchased ASUS Prime B250M-K motherboard. It has 1xPCIE16x and 2xPCIE1x slots. I disabled CSM, enabled 4G decoding, disabled every possible extra items in bios like serial port, IOAPI entries etc, it only allows 5 GPUs to work properly, rest 3 GPU gets detected but it ends up with Code12 error in device description ( Out of PCIE lanes ).
However here is a weird thing, when I uninstall a GPU in device manage, then it uninstalls all GPU from device manage as being in same class, now after uninstallation, all 8 GPUs shows up as "VGA Video Controller". Now I right click on one of these items, and select "update drivers" then it again installs all 8 GPUS but this time with 7 of them working!!!
Now, if I reboot the PC goes back to OS again, it again shows only 5 of them working and 3 goes with CODE12. I am trying like for 18hours with diff. things. Any clue guys?

p.s. I tried ethOS, it detects all 8 GPUS and all works without any issues.

[gsrcrxsi314]

X58 is too old and is likely limited to 5-6 GPUs.
H97 should work though. Does it even boot with more than 6 GPUs? If it does, your OS is likely the issue.

You might be able to get 5-6 cards with the small boards, but YMMV. The chipset doesn't really matter I think, the multiplier combines the signal of all 4 cards into 1 lane.

H97 or any other lga1150 socket board won’t work. For more than like 7 GPUs.

I have 2 boards of this era, one is an Asrock H97 Anniversary, the other is a Gigabyte B85M board. Both of them will only work up to a max of 7 GPUs. If 8 GPUs are plugged in in any configuration, the board won’t even POST.

You need a newer board that supports the Above 4G decoding setting. These old boards won’t do it.

I have 8 running fine on a Z270 board, and I’m sure I can do more if I wanted.

[gfigg42]

X58 is too old and is likely limited to 5-6 GPUs.
H97 should work though. Does it even boot with more than 6 GPUs? If it does, your OS is likely the issue.

You might be able to get 5-6 cards with the small boards, but YMMV. The chipset doesn't really matter I think, the multiplier combines the signal of all 4 cards into 1 lane.

I've ran 7 on an old X58 board using a PCIe port multiplier card under Windows 7. Ran perfectly fine for weeks.

[remauto1187ma]

Anyone ever tried the pcie multiplier on a Asus B250 Mining Expert Motherboard to get more than the 13 out of it without using the 6 P106 cards it requires to do just that?

[enhu]

How do you check whether all GPUs are recognized in a linux system, which command?   I can see the image only shows windows where Nvidia series and one Radeon. I'm new to linux but prefer to use it because I don't want to keep reinstalling when a simply virus bothers the system and seem updating windows is unstoppable.

[Bimmber]

type: nvidia-smi

[nitrobg]

Update:
ASRock H81 Pro BTC R2.0 won't post with more than 7 cards.

[bluecoinX]

Why does a motherboard use 13 CARDS? Is it just for fun or space? Why not buy a better graphics card?