7xGPU + Windows // modding AMD video driver is the answer

[ioglnx]

I would be interested on how you guys install new driver or initial setup with more than 3 cards.
I tried the method 1 card install and then install next one after each other..

But i often came across the point of loop flickering, BSOD, or random..how to say looks like MEM freeze artifacted picture.
It also takes a long time ..very long e.g with 4 cards until the setup is done and inbetween the system often freeze mouse and keyboard not working and then it suddenly after 10min works a few seconds and freezes again.

It took me always more than 2h to get 6 cards to work after countless bsods, crashes, freezes on Windows 10..So I'm curious how you guys do it and what problems you experienced.

I'd say you have some hardware issue.

AMD driver install can take a lot of time(max I've seen ~1h) because it keeps re-installing driver for all GPUs on each GPU setup, hence 6 ! installs on 6xGPU rig.
However, I've never seen lockups on driver setup. On properly working PC I mean.

Also, imo W10/2016 is very HDD IO demanding. So slower than 7/2k8.

And BSODs... well.. what do they say exactly?

AMD driver cased the bsod as well on intall or deinstall.
Hardware is new and stock settings and run 1 week stable on Prime95 and other tools.

It just happens when I install AMD drivers and no the card not even modified even at stock the install often hungs or freeze.
I also noticed sometimes when i add the 6th card or 5th..after bsod while install it works after reboot.

sofar I tested also 6-7 nvidia gpus and no issues..install takes long too but it always finish and works.
HDD ..no seriously I use a SSD one of the fastest available there shouldn't be a bottleneck.

[lbr]

Well... AMD driver.. sux ; ) That's for sure.
But, maybe there is also other stuff in play.
I've no experience at all with multi nVidia GPUs.. except with a lot of AMD + some nVidia.

Some stuff that can interfere(what I've seen) - not enough RAM, dual/single/whatever RAM issues(some motherboards don't work with a lot of GPUs and 4x2GB), PCIe bandwidth and revision(should be the lowest possible, x1@1.1), integrated GPU(disabled), incorrectly "hacked" Win(only KMS should be used). Especially Intel(z77) motherboards are completely fucked up when it concerns PCIe lanes distribution.
Weird things - GA-990fxa rev.1 boots headless only with "Full Screen Logo Show" enabled Oo, some GPUs just don't work with risers at all.

So..... well.... >

And deffo you are not using "one of the fastest available there" SSDs ; ) If by "there" you mean "in the world".

[lbr]

+ for a lot of mobos display must be connected to the primary GPU selected in BIOS

[ioglnx]

Well... AMD driver.. sux ; ) That's for sure.
But, maybe there is also other stuff in play.
I've no experience at all with multi nVidia GPUs.. except with a lot of AMD + some nVidia.

Some stuff that can interfere(what I've seen) - not enough RAM, dual/single/whatever RAM issues(some motherboards don't work with a lot of GPUs and 4x2GB), PCIe bandwidth and revision(should be the lowest possible, x1@1.1), integrated GPU(disabled), incorrectly "hacked" Win(only KMS should be used). Especially Intel(z77) motherboards are completely fucked up when it concerns PCIe lanes distribution.
Weird things - GA-990fxa rev.1 boots headless only with "Full Screen Logo Show" enabled Oo, some GPUs just don't work with risers at all.

So..... well.... >

And deffo you are not using "one of the fastest available there" SSDs ; ) If by "there" you mean "in the world".

This SSD still rocks - and still compete with newer SSD for SATA..I am not talking about NVMe.
Well the RAM and Lane config is the same as for the NV..I just switched the gpus..on a fresh install of windows.

And it doesn't matter on which PCIe port i attach the 3++ card it happens equal which card, which port I take when installing the AMD crapson bleepshit drivers.
I wasted already so many hours 12h in total to get it working.. but i tend to throw it out of the window and let a car smash it.

[lbr]

..
I wasted already so many hours 12h in total to get it working.. but i tend to throw it out of the window and let a car smash it.

Time to disclose your hardware ; )
Motherboard?

[cc4fun]

Hello,
I'm trying to run 7xXFX RX 480 GTR Black (8GB) on MSI Z97 Gaming 5.

Every time I connect 7th GPU my rig won't even start, it just shows this right after power on:
https://postimg.org/image/45s15sapz/
(80x25 text mode with totally corrupted characters, blinking)

I've tried to:
- use various risers (ribbon powered, ribbon non-powered, two types of USB including version 005S)
- play around with BIOS seetings (GEN1-GEN3)
- add extra PSU (1200W+750W, both 80PLUS Gold)
- turn off all unnecessary devices in BIOS (LAN Adapter, Audio, legacy IO)

None of this helps when all 7 PCI slots are in use. Regardless of what I do, end result is always the same.
With 6 GPUs system starts OK.

Any ideas?

[lbr]

https://postimg.org/image/45s15sapz/
Any ideas?

Cool ; )

What's 80x25? Seems like it's not present in the manual.
Ok, that's text mode res ; ) So what does POST indicator say?
@70~77,D2 Late South Bridge Initialization
92~96,B5,D4 PCI bus Initialization@

Display should be attached to GPU in PCI_E2.
CPU PCI-E Lane Configuration -> dunno, but lowest possible, Gen1
PEGX - Gen X-> Gen1
PCI Latency Timer -> play with it
Initiate Graphic Adapter -> PEG
IGD Multi-Monitor -> Disabled
Integrated Graphics Share Memory -> dunno.. 32M
Integrated GPU should be disabled anyway, tho I can't see BIOS option to do it.
Intel(R) Rapid Start Technology -> Disabled
Internal GOP Configuration -> What's that?
Full Screen Logo Display -> Enabled
Boot Mode Select -> try both

Also try shorting PCI-e presence on all risers.
And maybe different BIOS versions.
RAM - try more/less/different.

https://bitcointalk.org/index.php?topic=1661923.0
https://bitcointalk.org/index.php?topic=1640050.0

[cc4fun]

https://postimg.org/image/45s15sapz/
Any ideas?

Cool ; )

What's 80x25? Seems like it's not present in the manual.
Ok, that's text mode res ; ) So what does POST indicator say?
@70~77,D2 Late South Bridge Initialization
92~96,B5,D4 PCI bus Initialization@

Display should be attached to GPU in PCI_E2.
CPU PCI-E Lane Configuration -> dunno, but lowest possible, Gen1
PEGX - Gen X-> Gen1
PCI Latency Timer -> play with it
Initiate Graphic Adapter -> PEG
[...]

I am so grateful for this hint. Thanks!!
Disabling on-board graphic adapter did the trick.

So now I have another issue. Both 6th and 7th GPU report Error Code 43 in Windows 8 PRO. Previously it was 6th GPU but I hoped that rootcausing 7th GPU start-up failure would help both for issues. Anyway what I already did after launching all 7 GPUs:
1 ) Remove all drivers in safe mode using DDU
2 ) Observe that actually some standard VGA Adapters report issues in Device Manager (code 10, 38, 38, 38 respectively):
https://s27.postimg.org/tx7itnez7/20161211_165713.jpg
3 ) Install 16.11.5 drivers
4 ) Observe that  GPUs on PCI 7, 8 and 6 show Error Code 43
5 ) Run 6xGPU MOD in admin mode
6 ) 1 GPU fixed, GPUs on PCI 7 and 8 show Error Code 43. Physically these are PCI slots: PCI_E1 and PCI_E6.
7 ) Try to revert to original BIOS (using micro-switch on the GPU) - doesn't help
8 ) Re-run 6xGPU MOD - doesn't help
9 ) Reduced RAM from 8GB to 4GB - doesn't change anything (2 GPUs with code 43 remain, so presumably if RAM was the issue more GPUs would cease to work)
10 ) Make sure that by swapping risers, cables, riser cards, GPUs - the issue doesn't change, these are always the same PCI slots that are affected.
11 ) Additionaly I noticed that graphic driver crashes randomly when there is at least one GPU with Error 43 (sometimes even in idle or right after start up).

I will keep on investigating and reading two threads you linked. But it seems in my case issue is related to some basics in Windows since even witout AMD driver there are Standard VGA adapter issues.

[batko]

Instal win 10, my 7xRX 480 works fine.

[ferol]

I would really love if someone could help me with another issue you might encounter in the future.

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

[cc4fun]

I would really love if someone could help me with another issue you might encounter in the future.

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

While I'm still working on my issue, maybe this could help:
- Does this always happen to the same GPU? If so, swap it with another one and see if the issue follows the GPU or the PCI/riser. If the issue follows the GPU try flashing it with the BIOS copied from a GPU that doesn't fail (backup first).
- Maybe this is miner's feature to slow the card down if it get's overheated - what's the temp it achieves prior to the failure? And the time you spend to reinstall drivers allows the GPU to cool down?

[lbr]

I would really love if someone could help me with another issue you might encounter in the future.

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

Raise vcore.

[ferol]

I would really love if someone could help me with another issue you might encounter in the future.

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

While I'm still working on my issue, maybe this could help:
- Does this always happen to the same GPU? If so, swap it with another one and see if the issue follows the GPU or the PCI/riser. If the issue follows the GPU try flashing it with the BIOS copied from a GPU that doesn't fail (backup first).
- Maybe this is miner's feature to slow the card down if it get's overheated - what's the temp it achieves prior to the failure? And the time you spend to reinstall drivers allows the GPU to cool down?

Hi! thanks for the quick response.

Nope, I realized it is always the 2nd GPU showed on Afterburner, I swapped every GPU between every PCIE slot, I also tested every card on the 16x port to see if every one of them could send the signal to the screen with no problems.

The low mining speed can also be seen on the ethereum miner.

Every GPU are working between 65 and 71 ºC.

I have taken between a day to 20 minutes, from the crash to the reinstall, I dont think this is a temp issue as the speed wont go back up even if I let it rest for a day.

BTW, I just updated MOBOs bios but I am still having the same issue, and I still cant even get the NiceHashs benchmark done. Might it be MOBOS fault?

I would really love if someone could help me with another issue you might encounter in the future.

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

Raise vcore.

Hi!, I just tried rising vcore by 6% and power limit by 10% and it might just have lasted a bit longer, but in the end, it ended up crashing again.

[lbr]

Hi!, I just tried rising vcore by 6% and power limit by 10% and it might just have lasted a bit longer, but in the end, it ended up crashing again.

Raise vcore until it stops crashing. Or decrease core clock.

[lbr]

Also, any interferance with the GPU except the miner can cause GPU to go idle/low power state.
F.e. - Screen saver/GPU-Z/MSI Afterburner/any soft which uses GPU.
Some software on launch/some rendering/profile apply/etc makes GPU to enter 3D mode, after it is finished GPU goes idle/low state.

Or it simply hangs in OpenCL due to unstable clocks and then resets itself.
Visually it can be seen on display as very quick flicker.

[ferol]

Also, any interferance with the GPU except the miner can cause GPU to go idle/low power state.
F.e. - Screen saver/GPU-Z/MSI Afterburner/any soft which uses GPU.
Some software on launch/some rendering/profile apply/etc makes GPU to enter 3D mode, after it is finished GPU goes idle/low state.

Or it simply hangs in OpenCL due to unstable clocks and then resets itself.
Visually it can be seen on display as very quick flicker.

I just installed windows 10 to see if that could solve the problem. Same Bios settings (8x 4x 4x, gen 2 PCIE ports, inboard audio disabled), amd drivers ver 16.9, afterburner, CDM ver 7.4, aand thats it.

Afterburner mods: +6% vcore, +10% power limit.

Results: 1 gpu mining at 4mh/s, 5 gpus mining at 25 mh/s. And a new problem: I dont see any speed improvement by modifying the clocks, its like afterburner didnt work. I havent overlocked my gpus since the release of ZCash but I remember getting speeds of up to 29 mh/s per gpu with my other rigs.

[lbr]

And a new problem: I dont see any speed improvement by modifying the clocks, its like afterburner didnt work.

Maybe it didn't - check core clock in GPU-Z and AF.

[cc4fun]

Instal win 10, my 7xRX 480 works fine.

This helped!
Fresh Windows 10 installation and all 7 GPUs able to run even without 6xGPU MOD! I can't believe how much time I lost trying to get 5+ GPUs to run on Windows 7...

Thank you batko and lbr for help!

Now I "just" need to resolve stability issue of 6th and 7th GPU, which crash always after 5-6 minutes of mining.

[ferol]

And a new problem: I dont see any speed improvement by modifying the clocks, its like afterburner didnt work.

Maybe it didn't - check core clock in GPU-Z and AF.

Hi, well, this did the trick:

I would really love if someone could help me with an annoying issue.

I really dont know if it is the miner causing it, windows, or maybe the GPU...

For some reason, after a couple of hours mining perfectly, one gpu drops its mem clock from whatever stock setting it is to 300, which causes that gpu to mine a 1/5 % of the speed, usually around 47 sols/s (amd 470 4 gb modded). so I am constantly having to reinstall drivers, patch, etc, to get it work normally again, since simple restarts do not solve the problem.

I am using 6x 470 4gb cards, z97 gaming 5 mobo, 8gb ram, windows 8.1

Do you have a screen plugged in (or dummy plug)? If it's the 'out card' it could be a restful state issue that your power management is forcing the card into.

HI! yes, i keep a screen plugged in all the time. I always use the same port to connect the pc to the scree, i think its the first 16x port, I think it is also the only one that allows me to enter the BIOS settings.

Double check, I had the same 300MHz problem when I first set up my rigs, it turns out I had to have a monitor plugged in to the primary slot - for me that was the 2nd PCI e slot from the top (not the first 1x slot).

I didnt remember I had to connect the screen to a different gpu during the installation. Everything seems to be working Almost fine.
My current configuration is:
Core volt +6
Power limit +10
Core clock 1300
Memory clock 1950

Windows 10 Professional

Hashing speed for CDM ver 7.4 are between 27.7 and 28.0
Temps go from 68 to 73.

For some reason it is mining a bit slower than the rest of my rigs, may be because of the Windows version.
I will let know if it remains stable.

THANKS A LOT FOR YOUR HELP!!!!

[ferol]

Instal win 10, my 7xRX 480 works fine.

This helped!
Fresh Windows 10 installation and all 7 GPUs able to run even without 6xGPU MOD! I can't believe how much time I lost trying to get 5+ GPUs to run on Windows 7...

Thank you batko and lbr for help!

Now I "just" need to resolve stability issue of 6th and 7th GPU, which crash always after 5-6 minutes of mining.

Did you try modifying the bios? set PCIE ports to gen 2, choose the 8x 4x 4x option, deactivate integrated audio port, update bios, and connect the screen to the first 16x port.