HOW TO SET UP OVERCLOCKING AND FAN CONTROL ON UBUNTU 16.04 FOR NVIDIA CARDS

[thevictimofuktyranny]

This is quick Guide for setting up multiple Nvidia GPUs on Ubuntu 16.04LTS and 17.10LTS with Full Desktop.

Enabling all GPUs with overclocking and fan control.

CURRENTLY, THIS GUIDE REQUIRES (UBUNTU 16.04.04LTS - LATEST VERSION) ONE NVIDIA GPU NEEDS TO BE CONNECTED TO A MONITOR.

IT IS NOW SUPER-SIMPLE.

These are the steps are now:

1) install Ubuntu 16.04LTS or 17.04LTS

2) Update the Operating System via Software Centre. REBOOT

3) Go to Additional Drivers and switch the the CPU drivers, if not automatically loaded. (If you have problems with CPU drivers switch back to the Ubuntu default) REBOOT

4) Go to Additional Drivers and switch too Nvidia Drivers - I recommend you use default 378 driver optimised for Ubuntu OS. REBOOT

5) Open a Terminal and enter each line:

sudo update-grub

sudo nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration

(You can run with offer coolbits settings, 31 is frequently used as well).
 
REBOOT

Fans control and overclocking is now enabled.

To finish, you will create a startup sh file for each GPU so the overclocks and fan speed are loaded when you log into Ubuntu 16.04LTS.

Create some empty documents on Ubuntu Desktop and call them whatever you like. Make sure the filename has .sh at the end.

Paste in:

!/bin/bash

nvidia-settings -a '[gpu:0]/GPUGraphicsMemoryOffset[3]=100'

nvidia-settings -a '[gpu:0]/GPUMemoryTransferRateOffset[3]=100'

nvidia-settings -a '[gpu:0]/GPUFanControlState=1'

nvidia-settings -a '[fan:0]/GPUTargetFanSpeed=80'

Amend the clocks (GPU and Memory) and fan speeds to whatever you're comfortable with. Make separate documents for each GPU by changing the numbers for each card.

Save and open properties and make each file "executable".

Go to Startup Applications and ADD each .sh to the programs you run when you log in.

--------------------------------------------------------------------------Problems setting nvidia-xconfig for multi-GPU rigs try this work-around---------------------------

Firstly, log in and enable overclocking on one GPU:

sudo nvidia-xconfig -a --cool-bits=28

Log out and then log back in, then enable overclocking on all the other identical GPUs:

sudo nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration

Then log out and log back in and all the GPUs will have overclocking and fan control enabled.


-------------------------------------------------------------------------Reducing Watts Used By the GPUs---------------------------------------------------------------------------------------------------

Set Nvidia Drivers to persistent state (you must be in root - open terminal and enter "sudo -i"):

nvidia-smi -pm 1

First ask nvidia-smi what's the max power and min power limits are:

nvidia-smi -i 0 -q -d POWER

This will show MAX Power and MIN POWER allowed.

GTX 750TI as an example:
MIN POWER 30 W
MAX POWER 38.5 W

Then, you can reduce the watts to the MIN POWER allowed:

sudo nvidia-smi -pl 30

This gives you a net reduction of 22%.

Tested on Ubuntu, with max GPU load via running Unigine Heaven 4 Benchmark at MIN POWER.

For rigs with identical GPUs, you can set all power watts for all the cards at the same time with:
nvidia-smi -pm 1
sudo nvidia-smi -pl 30

---------------------------------------------------------------------Losing Share Efficiency after Updating the OS Security-----------------------------------------------------------------------

Switch to Ubuntu stock non-Nvidia drivers - Roboot.
On next boot up switch back to 378 drivers - Roboot
Re-enable overclocking and fan control.
Performance on share efficiency will be restored expected rates.
 

-------------------------------------------------------------------------PSU Capacitor Ageing--------------------------------------------------------------------------------------------------------------

The principle effects of this, will be a loss of efficiency. A PSU running at 88% efficiency after a 5 years will run at lower efficiency, closer to 78%.

Naturally, this will lead to higher wasted watts, depending on your locations electricity pricing buying a new PSU could be a worthwhile undertaking.

An extra 80watts wasted on a 800watt load does work out to be $84 (at $0.12 1 kilowatt-hour (kWh)) in a year.


----------------------------------------------------------Old Method as Reference Material and No Longer Needed-----------------------------------------------------------------------------

Install Ubuntu 16.04 with enabling software update options for Ubuntu Development team and third parties.

On reboot, after installation open up Ubuntu Software - update the the OS via Ubuntu Software (important to use the OS tool and not a terminal) and reboot.

Next, Software & Updates , and then Additional Drivers and install the Nvidia 367.57 drivers - these include extra tweaks from the Ubuntu development team for max GPU performance. Unfortunately, they do not allow you overclocking, but you will fix this latter on.

Next, go to Search Your Computer and bring up Nvidia Contol Center. And X Server Configuration and save the configuration file.

Next, open a terminal and enter the following:

sudo update-grub
sudo nvidia-xconfig -a --cool-bits=28 --allow-empty-initial-configuration

Reboot

This will enable all GPUs with screens and fan control on all GPUs.

Now, lets go get the latest drivers from Nvidia and Cuda 8 from their website or whatever Cuda version you need for your mining software. Save these downloads too the default Download folder.

Next, you will disable the Nvidia 367.57 driver to install the latest Nvidia drivers by going back into Software & Updates , and then Additional Drivers and select Nouveau Display Drivers. Apply changes.

Reboot


--------------------------------INSTALL DRIVERS VIA ADDITIONAL DRIVERS UBUNTU 16.04LTS----------------------------
Intstall the latest drivers with following instructions:

Press Control Alt F2 to get into non-desktop display.

Log in.

Switch off x-server with:

sudo service lightdm stop.

Go to downloads folder with:

cd ~/Downloads

ls

This will display driver name and run with:

sudo sh ./AND NVIDIA NAME LISTED

There will be two error messages, but select continue installation and say yes at the prompts.

Then, switch the x-server back on with:

sudo service lightdm start.

Reboot
---------------------------------------------------------------------------------------------------------------------------------------------------
Now, when you go back to the Nvidia Control Panel, it will show overclocking is enabled on all GPU's.

Next, install Cuda 8 with by opening a terminal in the Downloads folder:

sudo sh cuda_8.0.44_linux.run

Press "Control C" too fast foward to the end of EULA and "accept".

Say "No" to install drivers (trying to install these drivers when you have active Nvidia drivers will wreck the OS) and yes to toolkit, link and samples.

[pallas]

Thanks for this guide!

[thevictimofuktyranny]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

[Nikolaj]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

Thank you, nice guide

ps: Merry Christmas

[xPwnK]

Thank you for this guide! I'm sure this will help people who are new to ubuntu.

[thevictimofuktyranny]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

Thank you, nice guide

ps: Merry Christmas

Merry Christmas as well

[crysx]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

Thank you, nice guide

ps: Merry Christmas

Merry Christmas as well

have a happy Christmas mate ...

when i get back into it all in the next couple of days - ill see what ( if any ) of this guide works with redhat based systems ...

once again - no one has any guide or intention on releasing anything for rhel based systems ( like feora ) even though it is onf of the highest rated and used system on the planet ...

rhel itself is used in the majority of the corporate systems backend - and no one has tapped into that market to supply to those people and systems as well ... except cwi of course ...

tanx and njoi the time of this joyous occasion ...

#crysx

[thevictimofuktyranny]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

Thank you, nice guide

ps: Merry Christmas

Merry Christmas as well

have a happy Christmas mate ...

when i get back into it all in the next couple of days - ill see what ( if any ) of this guide works with redhat based systems ...

once again - no one has any guide or intention on releasing anything for rhel based systems ( like feora ) even though it is onf of the highest rated and used system on the planet ...

rhel itself is used in the majority of the corporate systems backend - and no one has tapped into that market to supply to those people and systems as well ... except cwi of course ...

tanx and njoi the time of this joyous occasion ...

#crysx

The latest Mint version is based on Ubuntu 16.04LTS - so I'd reckon that will be similar process

[crysx]

Thanks for this guide!

No problems, mate

It sorts out a lot of problems people have been having getting the max GPU performance on Linux installations and enabling all the GPUs overclocking and fan controls.

Thank you, nice guide

ps: Merry Christmas

Merry Christmas as well

have a happy Christmas mate ...

when i get back into it all in the next couple of days - ill see what ( if any ) of this guide works with redhat based systems ...

once again - no one has any guide or intention on releasing anything for rhel based systems ( like feora ) even though it is onf of the highest rated and used system on the planet ...

rhel itself is used in the majority of the corporate systems backend - and no one has tapped into that market to supply to those people and systems as well ... except cwi of course ...

tanx and njoi the time of this joyous occasion ...

#crysx

The latest Mint version is based on Ubuntu 16.04LTS - so I'd reckon that will be similar process

yup ...

debian based systems like that are fundamentally different to the rhel based systems ...

from installs to support in repos - some things are similar - but very few ... most things between the different distributions are completely different - but usually not impossible to do ...

the type of mining farm that is currently being devised is unique to cwi and thefarm ... it is being done no where else in the world and the cooling design i am designing for the miners themselves - is just as unique ... so when an oc is stable for a particular type of card or chipset - then the cooling design can be implemented to better keep the gpu cards themselves much cooler - and the mining much more simplified but highly optimized with our miner ...

so this is one of the things i would like to see happen within the coming 12months of 2017 come to fruition ... alongside all the other things we have - this will make a handy addition to the knowledgebase we are also putting together ... this will also simplify the installation procedures which almost NO developmental projects out there support off the bat - redhat systems ...

community members and devs alike seem to think the word linux is synonymous with ubuntu / debian ... its not ...

will let you know how we go in the coming months ...

#crysx

[QuintLeo]

Ubuntu varients like XUbuntu should work the same.

 XUbuntu has that nice "startup" stuff built into the XFCE desktop though, and doesn't make you put ".sh" at the end of shell scripts to run them, though they DO need to be CHMODed to an executeable status.


 Most mining software programmers write to Ubuntu (I'm not sure why that started but it's definitely the default by now).
 Getting most mining sofware to work on non-Ubuntu (or at least non-Debian) varients therefore becomes a major chore at best.

 I just wish Grub didn't make "cloning" an existing working installation a PITA (I can clone a working Slackware setup with a single "DD" command, trivial easy and built into ANY distro, because of it's use of LILO).

 I suspect it wouldn't be such a PITA if grub didn't force the use of that stupid "UUID" .... GARBAGE ....

[crysx]

Ubuntu varients like XUbuntu should work the same.

 XUbuntu has that nice "startup" stuff built into the XFCE desktop though, and doesn't make you put ".sh" at the end of shell scripts to run them, though they DO need to be CHMODed to an executeable status.


 Most mining software programmers write to Ubuntu (I'm not sure why that started but it's definitely the default by now).
 Getting most mining sofware to work on non-Ubuntu (or at least non-Debian) varients therefore becomes a major chore at best.

 I just wish Grub didn't make "cloning" an existing working installation a PITA (I can clone a working Slackware setup with a single "DD" command, trivial easy and built into ANY distro, because of it's use of LILO).

 I suspect it wouldn't be such a PITA if grub didn't force the use of that stupid "UUID" .... GARBAGE ....

hehehe ...

i actually couldnt agree with you more ...

debian based distros are now the 'norm' - but that doesnt mean that a multitude of rhel based systems are not in use today - because they are ... infact in much more businesses and corporates than any other distribution ...

but come crypto mining - and debian based systems become something that is used throughout - and it blows my mind why ... not that i am against these systems - just that im agaians the use of the term 'linux' to describe ubuntu - rather than ALL distributions ...

anyway - there are so many that have made it a warcry now - that newbies that come into the miner scene seem to get led down into a spiral rabbit hole and never come out the same when it comes to distros of any sort ... ive been a redaht guy ( yup - ive haerd it all before and heard all the condolences speeches ) but redhat systems are the number ONE and primary business systems in the world to date ... so in a crypto sense - it may not be too much of accolade - but in the business world ( which makes up more then 88% of the linux systems globally ) it matters that redhat is the stable and common system ... which means if crypto mining actually took anything really seriously with regards to implementation in this field - they may just have some corporate grunt on cryptos side ...

but alas ...

the details of the instructions for oc and linux of this thread however - are actually quite good ...

tanx for that victim ...

#crysx

[thevictimofuktyranny]

Ubuntu varients like XUbuntu should work the same.

 XUbuntu has that nice "startup" stuff built into the XFCE desktop though, and doesn't make you put ".sh" at the end of shell scripts to run them, though they DO need to be CHMODed to an executeable status.


 Most mining software programmers write to Ubuntu (I'm not sure why that started but it's definitely the default by now).
 Getting most mining sofware to work on non-Ubuntu (or at least non-Debian) varients therefore becomes a major chore at best.

 I just wish Grub didn't make "cloning" an existing working installation a PITA (I can clone a working Slackware setup with a single "DD" command, trivial easy and built into ANY distro, because of it's use of LILO).

 I suspect it wouldn't be such a PITA if grub didn't force the use of that stupid "UUID" .... GARBAGE ....

Well, the Ubuntu development team have extra optimisation to ensure the max performance happens with GPU drivers. There are quite a few decent games playable on Ubuntu now.

For example: I tested a straight install of the latest Nvidia drivers, without installing via Software & Updates , and then Additional Drivers the Ubuntu officially tested Nvidia 367.57 drivers.

I found that the straight install of latest Nvidia drivers became unstable and would hash at 40% of the GPUs actual max hashrate speed.

Therefore, the Ubuntu development teams have extra installs, which make Nvidia drivers stable and achieve the max GPU performance, which carry over when updating too the latest Nvidia drivers.

[crysx]

Ubuntu varients like XUbuntu should work the same.

 XUbuntu has that nice "startup" stuff built into the XFCE desktop though, and doesn't make you put ".sh" at the end of shell scripts to run them, though they DO need to be CHMODed to an executeable status.


 Most mining software programmers write to Ubuntu (I'm not sure why that started but it's definitely the default by now).
 Getting most mining sofware to work on non-Ubuntu (or at least non-Debian) varients therefore becomes a major chore at best.

 I just wish Grub didn't make "cloning" an existing working installation a PITA (I can clone a working Slackware setup with a single "DD" command, trivial easy and built into ANY distro, because of it's use of LILO).

 I suspect it wouldn't be such a PITA if grub didn't force the use of that stupid "UUID" .... GARBAGE ....

Well, the Ubuntu development team have extra optimisation to ensure the max performance happens with GPU drivers. There are quite a few decent games playable on Ubuntu now.

For example: I tested a straight install of the latest Nvidia drivers, without installing via Software & Updates , and then Additional Drivers the Ubuntu officially tested Nvidia 367.57 drivers.

I found that the straight install of latest Nvidia drivers became unstable and would hash at 40% of the GPUs actual max hashrate speed.

Therefore, the Ubuntu development teams have extra installs, which make Nvidia drivers stable and achieve the max GPU performance, which carry when updating too the latest Nvidia drivers.

hence ...

backing what i was saying ...

if dev teams would actually support redhat systems as much as debian based systems - we would have a multitude of systems to choose from - and not a finite set of systems ...

though i totally agree with ubuntus support structure and setup - i am saddened by the lack of support for almost any other distro other that ubuntu / debian based ...

which we will hopefully change this coming year ...

#crysx

[painmaker]

hi there and thanks for this writeup! pretty much covers what i've tried/observed so far.
besides setting coolbits to 28 (which i don't think is necessary as a value of 16 should suffice unless one wants to fiddle with GPU-voltages) i've also come across the xorg.conf setting of

  Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefault=0x1; PowerMizerDefaultAC=0x1

which ought to help set a performance-level where GPUMemoryTransferRateOffset can bet set which is in my case only the cards highest performance-level (can be checked with nvidia-settings -q GPUPerfModes -t)!

however, after booting the card seems to be in the highest perf-level but as soon as i start my miner the perf-leves goes back to the second highest level where the GPUMemoryTransferRateOffset cannot be set.
any idea what i might do wrong on this? 

setting fanspeed and GPUGraphicsClockOffset seems to work fine as both are settable not only in the highest perf-level...
driver used on my debian-system is 375.20.

cheers!

[thevictimofuktyranny]

hi there and thanks for this writeup! pretty much covers what i've tried/observed so far.
besides setting coolbits to 28 (which i don't think is necessary as a value of 16 should suffice unless one wants to fiddle with GPU-voltages) i've also come across the xorg.conf setting of

 Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefault=0x1; PowerMizerDefaultAC=0x1

which ought to help set a performance-level where GPUMemoryTransferRateOffset can bet set which is in my case only the cards highest performance-level (can be checked with nvidia-settings -q GPUPerfModes -t)!

however, after booting the card seems to be in the highest perf-level but as soon as i start my miner the perf-leves goes back to the second highest level where the GPUMemoryTransferRateOffset cannot be set.
any idea what i might do wrong on this?  

setting fanspeed and GPUGraphicsClockOffset seems to work fine as both are settable not only in the highest perf-level...
driver used on my debian-system is 375.20.

cheers!

Firstly, this does not work:

Option "RegistryDwords" "PowerMizerEnable=0x1; PerfLevelSrc=0x2222; PowerMizerDefault=0x1; PowerMizerDefaultAC=0x1

I would delete it off, I tried it and all it did was mess up settings.

Nvidia Compute always drops memory to 2nd memory profile setting, which is legacy problem from the launch of 10 series cards.

You need to set up .sh overclocking profile that will add 404mhz (or whatever you GPU is short at) overclock to the memory to get the gaming memory performance of 8008mhz applied to the Compute tasks.  

[vv181]

thanks for the guide!
btw please make overvolting guide on linux.

[thevictimofuktyranny]

thanks for the guide!
btw please make overvolting guide on linux.

The professional overclockers on youtube have said that Nvidia 10 series card don't get a worthwhile boost to higher mhz stability by adding more voltage.

[painmaker]

[SNIP]

[SNIP]
Nvidia Compute always drops memory to 2nd memory profile setting, which is legacy problem from the launch of 10 series cards.

You need to set up .sh overclocking profile that will add 404mhz (or whatever you GPU is short at) overclock to the memory to get the gaming memory performance of 8008mhz applied to the Compute tasks.  

thanks a lot for your reply! i'm not sure i understand you correctly what you mean by 'set up .sh overclocking profile'.
i assume you mean i need to create a bash-script (.sh) which gets executed during startup (just as you posted in you guide)?
this is exactly what i have done and this leads to the described behaviour in my case (level 2 (without clock-offsets) of a total of 3 levels).

without the RegistryDwords-option, the card is stuck in level 2 (of 3) right from the beginning.

[thevictimofuktyranny]

[SNIP]

[SNIP]
Nvidia Compute always drops memory to 2nd memory profile setting, which is legacy problem from the launch of 10 series cards.

You need to set up .sh overclocking profile that will add 404mhz (or whatever you GPU is short at) overclock to the memory to get the gaming memory performance of 8008mhz applied to the Compute tasks.  

thanks a lot for your reply! i'm not sure i understand you correctly what you mean by 'set up .sh overclocking profile'.
i assume you mean i need to create a bash-script (.sh) which gets executed during startup (just as you posted in you guide)?
this is exactly what i have done and this leads to the described behaviour in my case (level 2 (without clock-offsets) of a total of 3 levels).

without the RegistryDwords-option, the card is stuck in level 2 (of 3) right from the beginning.

Yeah, bash-script.

Then, it is a problem with your driver installation.

Or, did you accidental say "Yes" to install Nividia drivers when installing Cuda 8 - that will wreck everything.

Or, you've got a corrupted Ubuntu OS installation.

Anyway, I would zero wipe your SSD and start again.

[ken-ray]

Install Ubuntu 16.10

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-370 nvidia-cuda-toolkit
sudo nvidia-xconfig -a --cool-bits=31 --allow-empty-initial-configuration