[OS] nvOC easy-to-use Linux Nvidia Mining

[weareken]

First, thank you so much fullzero  for this.  I've rolled my own in the past, so this was a nice find and saved me a lot of trouble to do my own.

Have a couple of questions
1) I'm using 4 x PNY GTX 1060 3GB cards on a Gigabyte Z97 SOC Force motherboard.  Under Win10 with power 70, cclock 150, and mclock 850 I consistently get ~23 Mh/s.  With nvOC with the same settings I get 21.4 Mh/s.  Increasing the mclock doesn't seem to increase the hashrate.  cclock at 100 doesn't seem to make a difference either.

My oneBash looks like this:

Code:

COIN="ETH"

POWERLIMIT="70"                 # YES NO

INDIVIDUAL_POWERLIMIT="NO"      # YES NO

__CORE_OVERCLOCK=150
MEMORY_OVERCLOCK=850

INDIVIDUAL_CLOCKS="NO"          # YES NO

When the bash file starts it shows that the attributes are set, but it appears that the clock settings aren't taking effect. I don't see any errors thrown when Claymore starts up except that the cuda library doesn't have a version number.

2) I added a couple of parameters to the Eth command line, but it doesn't seem to pick them up.  For instance, when try to interactively change the dcri it tells me that i need to set -asm 1, but that is already set in the bash file.

Code:

if [ $COIN == "ETH" ]
then
HCD='/home/m1/9.0/ethdcrminer64'
ETHADDR="$ETH_ADDRESS/$ETH_WORKER"

if [ $ETHERMINEdotORG == "YES" ]
then
ETHADDR="$ETH_ADDRESS.$ETH_WORKER"
fi

until $HCD -epool $ETH_POOL -ewal $ETHADDR -epsw x -mode 1 -esm 0 -estale 0 -asm 1 -dcri 12
   do
   echo "FAILURE; reinit in 5" >&2
   sleep 5
done
fi

Any recommendations would be much appreciated.  Windows 10 sucks for mining - so much babysitting, so hoping to get this tweaked and running well.

Powerlimits in windows are in percent TDP; in linux they are in watts.  Thus a powerlimit of 70 in windows is .7 * 120 watts or whatever the TDP is (84 watts if the card TDP is 120); while 70 in linux is 70 watts.  I would recommend comparing Windows and linux powerlimits with a kill-a-watt to get them exactly equal; but most likely a 3gb 1060 has a 120 or 125 watt TDP.

Also linux OC offsets are scaled differently than windows; you will need to use higher offsets to get the same results in linux.

In regards to Claymore ETH settings; the Claymore readme says -asm is for AMD only.  I haven't tested this but -dcri 30 (or any valid number) is only supposed to work with mode 1 (ETH only) if -asm mode is used.

fullzero, thank you for the quick reply.  This was hugely helpful.

So strange that Windows is percentage of TDP and linux is watts.  The PNY 1060-3's are 120W TDP, so I've set it at 84 in linux.  I'll check it out with my power meter.

I increased the mclock to 1500, which is getting me about 22.7 Mh/s now.  I'll keep playing with it, but I see that the nvidia control panel has a max of 2000 for memory, whereas it's 1000 in afterburner on Windows.  It seems like the offset might be 2:1 on linux, so my 850 would be 1700 on linux.

Finally, with the dcri, you're right.  I have a mix of AMD and nvidia cards, so just assumed it worked on nvidia.  Tried it on a Windows nvidia only rig, and it gives the same error, so pretty sure dcri tuning doesn't work with nvidia cards.

Thanks again, this rocks!

[1713433838]

1713433838

[1713433838]

1713433838

[1713433838]

1713433838

[WarwickNZ]

Hi,

I've just tried switching some of my rigs to solely mine SIA to start building a position, but it doesn't seem to work? Switched back to dual for now, any ideas?

Mining to nanopool, no settings changed other than the coin selection, works fine on the claymore dual.

Thanks.

[weareken]

A tip and a question:

Tip:  If you're tuning your GPUs to reduce power, nvidia-smi can be quite helpful.  I open it in a second ssh session so I can watch the miner output in the main window and the GPU in the second.

You can run it with this (refreshes every 0.1 seconds):

Code:

watch -n0.1 nvidia-smi

Question:

My rig is using about 50W more than it should as compared to same Eth mh/s under windows.  I've got cclock set to -35 and mclock set to 820.  pl is set to 75.  I get ~23mh/s, which is what I'm shooting for.  from the nvidia-smi output it looks like the problem is that three of the cards (this rig is 3x PNY and 1xEVGA) are not honoring the powerlimit.  Still new to tuning nvidia cards until linux.  Any ideas?

Code:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.13                 Driver Version: 378.13                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 0000:01:00.0      On |                  N/A |
| 35%   63C    P2    73W /  75W |   2742MiB /  3012MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 0000:02:00.0     Off |                  N/A |
| 46%   72C    P2    95W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 106...  Off  | 0000:03:00.0     Off |                  N/A |
| 44%   70C    P2    94W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 106...  Off  | 0000:07:00.0     Off |                  N/A |
| 43%   69C    P2    94W /  75W |   2219MiB /  3013MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+

Update:
Apparently quite a few people are experiencing this issue with GTX 1060's https://devtalk.nvidia.com/default/topic/1011804/linux/nvidia-smi-not-fully-supported-on-gtx-1060/.  In this case, mine is a PNY 3GB version which someone else is having issues with as well.  Unclear as to whether this is a driver bug or somehow related to particular motherboards (a Gigabyte z97 SOC Force in my case).

[lbrasi]

Hi All,

I have decided to give nvOC a go, I am using an older Gigabyte GA-EX58-UD4P (Socket LGA1366) board and an i7 920 CPU.  I have two EVGA GTX 1080ti on risers in both PCI-E x16 slots.
I am able to boot fine but things just are not stable, I can run for about 15 hours but later GPU0 drops to about 150-170w and SOLs are suffering, I can reboot and get it working but it seems to happen again.

I have no idea if any BIOS changes need to be made to get this running stable or not, does anyone have any idea what I can try?

Also if I kill the mining process and make changes to the onebash config it doesn't seem to accept the changes, more specially the fan speed.  I attempt to launch the miner again and get the below error but the process does start;


Failed to connect to Mir: Failed to connect to server socket: No such file or directory
Unable to init server: Could not connect: Connection refused

ERROR: The control display is undefined; please run `nvidia-settings --help`
       for usage information.

[Maxximus007]

A tip and a question:

Tip:  If you're tuning your GPUs to reduce power, nvidia-smi can be quite helpful.  I open it in a second ssh session so I can watch the miner output in the main window and the GPU in the second.

You can run it with this (refreshes every 0.1 seconds):

Code:

watch -n0.1 nvidia-smi

Question:

My rig is using about 50W more than it should as compared to same Eth mh/s under windows.  I've got cclock set to -35 and mclock set to 820.  pl is set to 75.  I get ~23mh/s, which is what I'm shooting for.  from the nvidia-smi output it looks like the problem is that three of the cards (this rig is 3x PNY and 1xEVGA) are not honoring the powerlimit.  Still new to tuning nvidia cards until linux.  Any ideas?

Code:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.13                 Driver Version: 378.13                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 0000:01:00.0      On |                  N/A |
| 35%   63C    P2    73W /  75W |   2742MiB /  3012MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 0000:02:00.0     Off |                  N/A |
| 46%   72C    P2    95W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 106...  Off  | 0000:03:00.0     Off |                  N/A |
| 44%   70C    P2    94W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 106...  Off  | 0000:07:00.0     Off |                  N/A |
| 43%   69C    P2    94W /  75W |   2219MiB /  3013MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+

Update:
Apparently quite a few people are experiencing this issue with GTX 1060's https://devtalk.nvidia.com/default/topic/1011804/linux/nvidia-smi-not-fully-supported-on-gtx-1060/.  In this case, mine is a PNY 3GB version which someone else is having issues with as well.  Unclear as to whether this is a driver bug or somehow related to particular motherboards (a Gigabyte z97 SOC Force in my case).

Thanks for the tip!
Looking at the table posted: The cap is in place (75W), but the card is not honouring it. There might be a firmware upgrade for the card?
Out of curiosity: What's the avg Sol/s you get with these cards, or do you just mine Ethash with it? Do you mine single or dual mode?

[Brucelats]

Guys! I use Asus Z270-P motherboard. And i am having problems when i add 5th gpu. I am using ubuntu desktop 16.04. I use 5x gtx 1070.

Please, can anyone give me some info how to install my fifth card and then eventualy add 6th or 7th. How to setup Bios, what version etc. I guess nvOC wont work properly also before i deal.with this.

Please.help me spent last day trying to figure it all out.

When i add 5th gpu its either stuck. It doesnt let me enter bios even. I just see black screen. Or if i change bios options, i get into ubuntu but system sees only 4 gpus. 5th is invisible. Gpu 0 1 2 3 are shown 4 is not there.

[Brucelats]

Anyone have any suggestions to this problem, I am all ears.

I have one of my rigs which uses a Asus Prim z270-a when I put the 6th GPU in it will not POST to the bios screen therefore not start up.

From troubleshooting it only happens if I have 2 GPUs plugged into the X16 slots, if I only have one it works fine.

I have changed the BIOS to the recommend from the thread here for the z270-a and set it all to GEN2.

Any other suggestions?

There were 4 Gen2 settings I needed to change, did you change them all? Could be an issue running 2 on the board with 7 or more, i've only tested with 1 on the board +6 risers or all risers. Check for faulty riser cables too, that slowed up a friend of mine getting 7 going on this board.

This was my thought as well; WarwickNZ.

I'm not sure if you can use 2x GPUs direct on the mobo + 5x with risers with this mobo.

When using a GPU directly more pcie lanes / pcie bandwidth is used.  This may result in insufficient bandwidth on one of the cards to enable normal operation.

While using a card directly I would change the pcie bandwidth to gen2 for all cards.  Then when you get your risers, I would change it back to auto. 

As WarwickNZ indicated; there are multiple pcie bandwidth related settings in the bios.  There may also be an SLI setting that needs to be disabled as well.

I should probably rephrase. I have all 6 GPUs on risers, the only time it does not post is when I have 2 of the x16 slots with the risers in them. I will double check and make sure they are all the same gen type. I had it on auto first but when I did, it wouldn't even boot past one card. I'm to the point I may just reset the MB bios via cmos and start fresh.

Have u managed to get it work?

[Maxximus007]

I like rufus; I have used it for years, it can also be used to image the USB.  

I think the hddRawCopy tool, is easier to use for members who have less experience imaging; there are also a lot of members who are already familiar with it from smOS.

I use an img format for the image; specifically to enable the use of most imaging programs.

This one is also easy to use and is available for mac / linux and windows:

https://etcher.io/

Thanks, being on a Mac, etcher looks great. Saves me starting up Virtualbox

Just leaves me one thing: the Windows partition is unreachable with a Mac. Seemingly this has to do with some flags missing for the FAT partition. Anyone has a solution for this?

[Nexillus]

Anyone have any suggestions to this problem, I am all ears.

I have one of my rigs which uses a Asus Prim z270-a when I put the 6th GPU in it will not POST to the bios screen therefore not start up.

From troubleshooting it only happens if I have 2 GPUs plugged into the X16 slots, if I only have one it works fine.

I have changed the BIOS to the recommend from the thread here for the z270-a and set it all to GEN2.

Any other suggestions?

There were 4 Gen2 settings I needed to change, did you change them all? Could be an issue running 2 on the board with 7 or more, i've only tested with 1 on the board +6 risers or all risers. Check for faulty riser cables too, that slowed up a friend of mine getting 7 going on this board.

This was my thought as well; WarwickNZ.

I'm not sure if you can use 2x GPUs direct on the mobo + 5x with risers with this mobo.

When using a GPU directly more pcie lanes / pcie bandwidth is used.  This may result in insufficient bandwidth on one of the cards to enable normal operation.

While using a card directly I would change the pcie bandwidth to gen2 for all cards.  Then when you get your risers, I would change it back to auto. 

As WarwickNZ indicated; there are multiple pcie bandwidth related settings in the bios.  There may also be an SLI setting that needs to be disabled as well.

I should probably rephrase. I have all 6 GPUs on risers, the only time it does not post is when I have 2 of the x16 slots with the risers in them. I will double check and make sure they are all the same gen type. I had it on auto first but when I did, it wouldn't even boot past one card. I'm to the point I may just reset the MB bios via cmos and start fresh.

Have u managed to get it work?

Going to be trying it again today. I will get back with you.

[cryptosharks]

Can someone share the OC settings for GTX 1080 OC 8GB (non TI), please?

These are my OC settings for nvOC:

POWERLIMIT="NO"
__CORE_OVERCLOCK=275
MEMORY_OVERCLOCK=600
FAN_SPEED=100

The card gets heated: 82 C (stays there 24/7 consistently)

With the above OC settings, I get 315-330 MH/s on LBC and 495-505 Sol/s on ZEC.

With the same OC settings as on nvOC, Windows 10 is able to get 345-365 MH/s on LBC and 545-565 Sol/s on ZEC with temperature not going over 79 C. I used MSI Afterburner on Windows.

Any ideas?

[dt8666]

Hi. I am first time using this. Follow instruction and use the Atom to edit the bash file.

When boot up, after all start i saw the screen is up.

But it keep saying:
Xxxxxxxxxx.....1263-A96E/onebash
No such file or directory.

What should do?

Appreciate help.

[Maxximus007]

Can someone share the OC settings for GTX 1080 OC 8GB (non TI), please?

These are my OC settings for nvOC:

POWERLIMIT="NO"
__CORE_OVERCLOCK=275
MEMORY_OVERCLOCK=600
FAN_SPEED=100

The card gets heated: 82 C (stays there 24/7 consistently)

With the above OC settings, I get 315-330 MH/s on LBC and 495-505 Sol/s on ZEC.

With the same OC settings as on nvOC, Windows 10 is able to get 345-365 MH/s on LBC and 545-565 Sol/s on ZEC with temperature not going over 79 C. I used MSI Afterburner on Windows.

Any ideas?

Well, in nvOC (Linux) the OC is different. So you might want to up your mem OC quite a bit.

As written before by fullzero:
Powerlimits in windows are in percent TDP; in linux they are in watts.  Thus a powerlimit of 70 in windows is .7 * 120 watts or whatever the TDP is (84 watts if the card TDP is 120); while 70 in linux is 70 watts.  I would recommend comparing Windows and linux powerlimits with a kill-a-watt to get them exactly equal; but most likely a 3gb 1060 has a 120 or 125 watt TDP.

Also linux OC offsets are scaled differently than windows; you will need to use higher offsets to get the same results in linux.

[xleejohnx]

I like rufus; I have used it for years, it can also be used to image the USB.  

I think the hddRawCopy tool, is easier to use for members who have less experience imaging; there are also a lot of members who are already familiar with it from smOS.

I use an img format for the image; specifically to enable the use of most imaging programs.

This one is also easy to use and is available for mac / linux and windows:

https://etcher.io/

Thanks, being on a Mac, etcher looks great. Saves me starting up Virtualbox

Just leaves me one thing: the Windows partition is unreachable with a Mac. Seemingly this has to do with some flags missing for the FAT partition. Anyone has a solution for this?

What are you trying to do?
Explain into details what you trying to do and I might be able to help out
I deal with windows and Mac daily

[CarlOrff]

A tip and a question:

Tip:  If you're tuning your GPUs to reduce power, nvidia-smi can be quite helpful.  I open it in a second ssh session so I can watch the miner output in the main window and the GPU in the second.

You can run it with this (refreshes every 0.1 seconds):

Code:

watch -n0.1 nvidia-smi

Question:

My rig is using about 50W more than it should as compared to same Eth mh/s under windows.  I've got cclock set to -35 and mclock set to 820.  pl is set to 75.  I get ~23mh/s, which is what I'm shooting for.  from the nvidia-smi output it looks like the problem is that three of the cards (this rig is 3x PNY and 1xEVGA) are not honoring the powerlimit.  Still new to tuning nvidia cards until linux.  Any ideas?

Code:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 378.13                 Driver Version: 378.13                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 106...  Off  | 0000:01:00.0      On |                  N/A |
| 35%   63C    P2    73W /  75W |   2742MiB /  3012MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 106...  Off  | 0000:02:00.0     Off |                  N/A |
| 46%   72C    P2    95W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 106...  Off  | 0000:03:00.0     Off |                  N/A |
| 44%   70C    P2    94W /  75W |   2219MiB /  3013MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 106...  Off  | 0000:07:00.0     Off |                  N/A |
| 43%   69C    P2    94W /  75W |   2219MiB /  3013MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+

Update:
Apparently quite a few people are experiencing this issue with GTX 1060's https://devtalk.nvidia.com/default/topic/1011804/linux/nvidia-smi-not-fully-supported-on-gtx-1060/.  In this case, mine is a PNY 3GB version which someone else is having issues with as well.  Unclear as to whether this is a driver bug or somehow related to particular motherboards (a Gigabyte z97 SOC Force in my case).

Tried :
sudo nvidia-smi -i 1,2,3 -pl 75

[cryptosharks]

Can someone share the OC settings for GTX 1080 OC 8GB (non TI), please?

These are my OC settings for nvOC:

POWERLIMIT="NO"
__CORE_OVERCLOCK=275
MEMORY_OVERCLOCK=600
FAN_SPEED=100

The card gets heated: 82 C (stays there 24/7 consistently)

With the above OC settings, I get 315-330 MH/s on LBC and 495-505 Sol/s on ZEC.

With the same OC settings as on nvOC, Windows 10 is able to get 345-365 MH/s on LBC and 545-565 Sol/s on ZEC with temperature not going over 79 C. I used MSI Afterburner on Windows.

Any ideas?

Well, in nvOC (Linux) the OC is different. So you might want to up your mem OC quite a bit.

As written before by fullzero:
Powerlimits in windows are in percent TDP; in linux they are in watts.  Thus a powerlimit of 70 in windows is .7 * 120 watts or whatever the TDP is (84 watts if the card TDP is 120); while 70 in linux is 70 watts.  I would recommend comparing Windows and linux powerlimits with a kill-a-watt to get them exactly equal; but most likely a 3gb 1060 has a 120 or 125 watt TDP.

Also linux OC offsets are scaled differently than windows; you will need to use higher offsets to get the same results in linux.

Thanks.

How about setting the temperature limit? In MSI Afterburner, I was able to set the temperature limit as well. I've been using Windows so far, so please don't mind my questions.

@fullzero, cheers for creating nvOC and sharing. You gotta start charging  

[Maxximus007]

I like rufus; I have used it for years, it can also be used to image the USB. 

I think the hddRawCopy tool, is easier to use for members who have less experience imaging; there are also a lot of members who are already familiar with it from smOS.

I use an img format for the image; specifically to enable the use of most imaging programs.

This one is also easy to use and is available for mac / linux and windows:

https://etcher.io/

Thanks, being on a Mac, etcher looks great. Saves me starting up Virtualbox

Just leaves me one thing: the Windows partition is unreachable with a Mac. Seemingly this has to do with some flags missing for the FAT partition. Anyone has a solution for this?

What are you trying to do?
Explain into details what you trying to do and I might be able to help out
I deal with windows and Mac daily

After writing the image to an USB my Mac won't recognise the USB nor the Win partition on the USB. Tried several utilities but it seems to be an issue with the way the partition table is made. Unfortunately I couldn't find a working solution..
So for me it's easier atm to boot up a rig with the USB, ssh to it and edit the oneBash remotely. But if you have a way to mount, that would be great!

[Maxximus007]

Can someone share the OC settings for GTX 1080 OC 8GB (non TI), please?

These are my OC settings for nvOC:

POWERLIMIT="NO"
__CORE_OVERCLOCK=275
MEMORY_OVERCLOCK=600
FAN_SPEED=100

The card gets heated: 82 C (stays there 24/7 consistently)

With the above OC settings, I get 315-330 MH/s on LBC and 495-505 Sol/s on ZEC.

With the same OC settings as on nvOC, Windows 10 is able to get 345-365 MH/s on LBC and 545-565 Sol/s on ZEC with temperature not going over 79 C. I used MSI Afterburner on Windows.

Any ideas?

Well, in nvOC (Linux) the OC is different. So you might want to up your mem OC quite a bit.

As written before by fullzero:
Powerlimits in windows are in percent TDP; in linux they are in watts.  Thus a powerlimit of 70 in windows is .7 * 120 watts or whatever the TDP is (84 watts if the card TDP is 120); while 70 in linux is 70 watts.  I would recommend comparing Windows and linux powerlimits with a kill-a-watt to get them exactly equal; but most likely a 3gb 1060 has a 120 or 125 watt TDP.

Also linux OC offsets are scaled differently than windows; you will need to use higher offsets to get the same results in linux.

Thanks.

How about setting the temperature limit? In MSI Afterburner, I was able to set the temperature limit as well. I've been using Windows so far, so please don't mind my questions.

@fullzero, cheers for creating nvOC and sharing. You gotta start charging  

As far as I know, there is no option in Linux / Nvidia drivers to set a max temp. It can be done in EWBF miner, but that would mean it will idle the card until below threshold. You could limit your power to reach the desired temp, but that's not quite the same.

[UberDaemon]

I should probably rephrase. I have all 6 GPUs on risers, the only time it does not post is when I have 2 of the x16 slots with the risers in them. I will double check and make sure they are all the same gen type. I had it on auto first but when I did, it wouldn't even boot past one card. I'm to the point I may just reset the MB bios via cmos and start fresh.

Use all 3 of the x16 slots, then begin populating the x1 slots beginning with the one closest to the CPU and moving away from it.


https://i.imgur.com/lytgMw4.jpg

[UberDaemon]

Just wanted to say thanks for making this! Happy to donate if you have an address to send it to.

Refer to the untouched OneBash file for OP's wallet addresses to donate, or you can donate by running your rig with his address(es) still listed in the OneBash file.

[Nexillus]

I should probably rephrase. I have all 6 GPUs on risers, the only time it does not post is when I have 2 of the x16 slots with the risers in them. I will double check and make sure they are all the same gen type. I had it on auto first but when I did, it wouldn't even boot past one card. I'm to the point I may just reset the MB bios via cmos and start fresh.

Use all 3 of the x16 slots, then begin populating the x1 slots beginning with the one closest to the CPU and moving away from it.

I did not try this but I did get it working.

Motherboard changes to get it to post:
DMI max gen 2
pciex16_1 link gen 2
pciex16_2 link gen 2
pcie spread spectrum clock disable
pcei speed gen 2
restore ac power on
hd audio controler disable
post delay time 0 sec
above 4g decoding enable

Current slots in use (check MB manual)
PCIEX1_1
PCIEX16_1
PCIEX1_2
PCIEX1_3
PCIEX1_4
PCIEX16_3

If I use the PCIEX16_2 slot claymore miner will not detect the GPU in that slot, skipping it posts and all 6 are registered.
 
Link to the stats
CC: 100
Mem: 1275
dcri: 20
Watt: 115 per card
802 watts from the wall