How to expand the number of PCIe devices on a Main Board?

[jmigdlc99]

Thanks, nitrobg.  By disabling as much PCIe stuff (USB Root controllers, NICs, etc) the ASUS board will post with 16 GPUs but only 12 are recognized by W10.  The MB does not have "Above 4GB" setting in UEFI.  Perhaps all 16 could be used if available.  Thanks all for the help.  I learned a lot.  Enjoy, John.

actually it has "above 4G decoding" setting.

Unlikely, because he's using an AMD board and Above 4G is an Intel feature - I'll need to dig into my AMD rig's bios to see what the option was. I'm not sure what are AMD's limits because it could be utilizing the PCI-E lanes differently.
OP, please try running Linux.

umm, what?
ASUS B250 Mining Expert is an Intel board, I have one and I'm pretty sure that it has the "above 4g decoding" option in UEFI.
12 GPUs are recognized because Adrenaline Driver for Windows can run up to 12 AMD GPUs max.

The ASUS B250 mining expert motherboard can be used in conjuction with PCIE slot expanders, you just have to disable unused components. Some guy on youtube (if i recall correctly, Bits Be Trypin is the name of the channel) was able to mine with 21-GPUs on this motherboard.

Note that you need to use mining edition GPUs in addition to your normal GPUs to effectively get beyond the 13-GPU limit.

[darkneorus]

yup. I have 11x RX580 + 8x P104x100s on that mobo. working just fine

[nitrobg]

Thanks, nitrobg.  By disabling as much PCIe stuff (USB Root controllers, NICs, etc) the ASUS board will post with 16 GPUs but only 12 are recognized by W10.  The MB does not have "Above 4GB" setting in UEFI.  Perhaps all 16 could be used if available.  Thanks all for the help.  I learned a lot.  Enjoy, John.

actually it has "above 4G decoding" setting.

Unlikely, because he's using an AMD board and Above 4G is an Intel feature - I'll need to dig into my AMD rig's bios to see what the option was. I'm not sure what are AMD's limits because it could be utilizing the PCI-E lanes differently.
OP, please try running Linux.

umm, what?
ASUS B250 Mining Expert is an Intel board, I have one and I'm pretty sure that it has the "above 4g decoding" option in UEFI.
12 GPUs are recognized because Adrenaline Driver for Windows can run up to 12 AMD GPUs max.

OP was talking about a X399 Threadripper board. But yeah - ASUS B250 boards, including most Intel ones, have Above 4G.

[remauto1187ma]

Thanks, nitrobg.  By disabling as much PCIe stuff (USB Root controllers, NICs, etc) the ASUS board will post with 16 GPUs but only 12 are recognized by W10.  The MB does not have "Above 4GB" setting in UEFI.  Perhaps all 16 could be used if available.  Thanks all for the help.  I learned a lot.  Enjoy, John.

actually it has "above 4G decoding" setting.

Unlikely, because he's using an AMD board and Above 4G is an Intel feature - I'll need to dig into my AMD rig's bios to see what the option was. I'm not sure what are AMD's limits because it could be utilizing the PCI-E lanes differently.
OP, please try running Linux.

umm, what?
ASUS B250 Mining Expert is an Intel board, I have one and I'm pretty sure that it has the "above 4g decoding" option in UEFI.
12 GPUs are recognized because Adrenaline Driver for Windows can run up to 12 AMD GPUs max.

Yes it does indeed. See post#8 for a link to youtube video showing the bios. (not my video)

[MisterJJ]

Thanks much for all responses!  Here is a screenshot from a different user using an ASRock AMD MB.
https://s33.postimg.cc/bq0bc11f3/Above_4_GB.jpg
This user reported that the setting did not work.  ASUS does not seem interested.  Thanks and enjoy, John.

EDIT: The video above is from an Intel MB, not AMD. 

[remauto1187ma]

Thanks much for all responses!  Here is a screenshot from a different user using an ASRock AMD MB.

This user reported that the setting did not work.  ASUS does not seem interested.  Thanks and enjoy, John.

EDIT: The video above is from an Intel MB, not AMD. 

Of course its an Inetl MB...Its an ASUS B250 Mining Expert and thats what that conversation was about. Read ALL of the context.

[MisterJJ]

Thanks, remauto1187ma.  I will be more careful.  Please tell us how to increase the number of PCIe devices allowed by the UEFI.  To date, the only method known is to Disable all the devices not needed.  One real problem is that some MBs do not allow most devices to be disabled.  Perhaps you can tell us how to do this or recommend what to ask the MB Vendor to implement.  Thanks and enjoy, John.

[remauto1187ma]

Thanks, remauto1187ma.  I will be more careful.  Please tell us how to increase the number of PCIe devices allowed by the UEFI.  To date, the only method known is to Disable all the devices not needed.  One real problem is that some MBs do not allow most devices to be disabled.  Perhaps you can tell us how to do this or recommend what to ask the MB Vendor to implement.  Thanks and enjoy, John.

Go into device manager and Disable them there or even uninstall the Drivers for them then Disable.  Dont forget to show hidden device under Device Manager then View Tab then Show Hidden Devices.  Remove all failed previous attempts of an install for GPU including the Audio Drivers that failed.   *HINT* Look for failed codes or exclamation points in device manager.

[MisterJJ]

Thanks, remauto1187ma.  Good advice.  I will keep it in mind.  Enjoy, John.

[ProperS]

Hello, I am the original poster that explored this issue.

I want to share my findings as it may be useful to others.

Big thanks to MisterJ for starting a thread here to help!


My initial issue with X399 has been that motherboard would take up to 7 GPUs and then stop posting, all hardware was tested and worked outside of the build.
Although it posted with 7 and worked system tossed errors in the os when one of the devices was used, it will work but just had errors that according to OS logs were resolved(OS used is UBUNTU) and errors posted in the CLI
When i say 7 GPUs i mean number fo GPU devices detected, some video cards have 2 processing units and those are detected as two separate units, this means you are getting 2 GPUs as you connect each card.
The conclusion is that threadripper with default config will work in a stable manner with 6 pcie devices in the default configuration.

I wanted to connect allot more GPUs and for my application, I need allot of ram and a lot of CPU power, threadripper platform was a perfect fit on paper.
But 6 GPU limit kill all of it, I did continue to dig through and got much further

I got information that address allocation may be an issue and tried disabling sound controller, and to my surprise, it booted fine with 8 gpus, then I disabled network controller and it booted with 9. I was on the right track
It became clear from my research and testing that biggest consumer of address space is usb controller,  in fact on asus X399 22 addresses are allocated to USB controller and there is another usb 3.1 controller that also eats space
When usb controller was disabled, system boots even with 20 gpus but OS does not see more than 12.  All 12 GPUs work as expected, i have enough capacity on this CPU to go to 20 but for now, i am exploring what that limit could be.

Maybe it is driver or chipset related if you know of anything let me know

[MisterJJ]

ProperS,  thanks for the reply.  Here is the link to a thread with a similar experience as your's: https://community.amd.com/message/2864370#comment-2864370.  The user believes and I agree that the Above 4 GB option would allow use of all GPUs that are boot-able.  I assume your MB does not support this option.  What board are you using?  Hopefully a UEFI update will correct this soon.  Threadripper II is rumored to be released next month (August 2018).  Perhaps that will spark some new UEFI updates.  Thanks and enjoy, John.

[ProperS]

You are referring to my original post on AMD forums, =)

I recently did some tests and got 13 GPUs to post and work, the system performs in a stable manner but it does weird things during boot. It would start then shut down and start again but once it gets past that it boots up fine

 ASUS did not respond with anything useful and abandoned my supports requests without closure.

[MisterJJ]

Sorry Proper(S)!  How cool is it to refer a poster to their own thread?  One point I will make is in a very few weeks Threadripper II will be released and all the MB vendors will release new UEFIs.  Some have already.  GIGABYTE has released UEFI F10 with AGESA 1.1.0.0.  I think ASRock did also but then removed it.  That start/stop stuff may be memory training.  If you will explain the "weird" boot stuff detail, I will think on it.  I would prefer you use AMD forum, but this is OK if you prefer.  Enjoy, John.