gpu-watch: dynamic GPU temperature monitoring and fan control

[xf2_org]

URL: http://yyz.us/bitcoin/gpu-watch.py

This script will monitor all GPUs in a machine (by default, two), and control fan speed accordingly.

gpu-watch sleeps for N seconds, then samples the temperatures of all GPUs.  If the temperature is too low, fanspeed decreases by 5%.  If the temp is too high, fanspeed increases by 10%.

Settings:
poll_time: number of seconds to sleep between runs
card_first: id of first GPU
card_count: number of GPUs
temp_low: low temperature threshold, at which fan speed decreases
temp_high: high temperature threshold, at which fan speed increases

I run with temp_low==70 and temp_high==76.  If the range is too low, fan speed will constantly change.

[1714925004]

1714925004

[1714925004]

1714925004

[1714925004]

1714925004

[1714925004]

1714925004

[1714925004]

1714925004

[w128]

URL: http://yyz.us/bitcoin/gpu-watch.py

This script will monitor all GPUs in a machine (by default, two), and control fan speed accordingly.

gpu-watch sleeps for N seconds, then samples the temperatures of all GPUs.  If the temperature is too low, fanspeed decreases by 5%.  If the temp is too high, fanspeed increases by 10%.

Settings:
poll.time: number of seconds to sleep between runs
card.first: id of first GPU
card.count: number of GPUs
temp.low: low temperature threshold, at which fan speed decreases
temp.high: high temperature threshold, at which fan speed increases

I run with temp.low==70 and temp.high==76.  If the range is too low, fan speed will constantly change.

How hard would it be to make this plot a curve based on a handful of user-provided points or a table? The configurable curve in MSI Afterburner is very nice for this purpose.

I'd certainly pay some kind of bounty to see that implemented.

[ryepdx]

How hard would it be to make this plot a curve based on a handful of user-provided points or a table? The configurable curve in MSI Afterburner is very nice for this purpose.

I'd certainly pay some kind of bounty to see that implemented.

Can't you do that with Overdrive in Windows and AmdOverdriveCtrl in Linux?

[w128]

How hard would it be to make this plot a curve based on a handful of user-provided points or a table? The configurable curve in MSI Afterburner is very nice for this purpose.

I'd certainly pay some kind of bounty to see that implemented.

Can't you do that with Overdrive in Windows and AmdOverdriveCtrl in Linux?

I'm not sure but, I'm a fan of community built tools and given a choice I'd rather have it in something like this that I could potentially modifying and extend to my hearts content than vendor's potentially limited tools.

[Mononofu]

How hard would it be to make this plot a curve based on a handful of user-provided points or a table? The configurable curve in MSI Afterburner is very nice for this purpose.

I'd certainly pay some kind of bounty to see that implemented.

Can't you do that with Overdrive in Windows and AmdOverdriveCtrl in Linux?

With AmdOverdriveCtrl you can do that. You can choose between steps, linear changes or bezier-curves. Works like a charm for me

[elmom]

How hard would it be to make this plot a curve based on a handful of user-provided points or a table? The configurable curve in MSI Afterburner is very nice for this purpose.

I'd certainly pay some kind of bounty to see that implemented.

Can't you do that with Overdrive in Windows and AmdOverdriveCtrl in Linux?

With AmdOverdriveCtrl you can do that. You can choose between steps, linear changes or bezier-curves. Works like a charm for me

The fan control worked great, but the over/underclocking part messes my GPU. This script seems like a nice and lean tool. Now I just need a simple tool that can set the clocks beyond what aticonfig allows.

[barbagianni]

I run with temp.low==70 and temp.high==76.  If the range is too low, fan speed will constantly change.

hi xf2_org,
I tried your script but as I'm not confident with python language, can you pls write me an example on how to write the settings?

[disq]

Code:

Traceback (most recent call last):
  File "./gpu-watch.py", line 127, in <module>
    w.watch()
  File "./gpu-watch.py", line 75, in watch
    self.watch_card(card)
  File "./gpu-watch.py", line 79, in watch_card
    temp = card.temp()
  File "./gpu-watch.py", line 21, in temp
    line = subprocess.check_output(
AttributeError: 'module' object has no attribute 'check_output'

Code:

$ python -V
Python 2.6.6

what do I need? any way to make it compatible with lower versions of python like 2.6.x?

[xf2_org]

Script updated with two fixes:
- make configuration file work (./gpu-watch.py CONFIG-FILE)
- do not proceed past low end (zero percent), as temps fall

[xf2_org]

I tried your script but as I'm not confident with python language, can you pls write me an example on how to write the settings?

You create a configuration file, a text file with one setting per line in a SETTING=VALUE format, such as

poll_time=30
card_first=0
card_count=1

Code:

$ python -V
Python 2.6.6

what do I need? any way to make it compatible with lower versions of python like 2.6.x?

Tested on python 2.7.  Patches welcome for older versions...

[barbagianni]

poll_time=30
card_first=0
card_count=1

I still have this error with python 2.7:

registering card 0 (id 0)
registering card 1 (id 1)
Sat May 28 19:26:39 2011 Polling GPU data...
Traceback (most recent call last):
  File "gpu-watch.py", line 126, in <module>
    w.watch()
  File "gpu-watch.py", line 74, in watch
    self.watch_card(card)
  File "gpu-watch.py", line 80, in watch_card
    print "Temperature %.2fC, fan speed %d%%" % (temp, fanspeed)
TypeError: float argument required, not NoneType

[xf2_org]

registering card 0 (id 0)
registering card 1 (id 1)
Sat May 28 19:26:39 2011 Polling GPU data...
Traceback (most recent call last):
  File "gpu-watch.py", line 126, in <module>
    w.watch()
  File "gpu-watch.py", line 74, in watch
    self.watch_card(card)
  File "gpu-watch.py", line 80, in watch_card
    print "Temperature %.2fC, fan speed %d%%" % (temp, fanspeed)
TypeError: float argument required, not NoneType

You passed an incorrect card setup to gpu-watch.

[barbagianni]

You passed an incorrect card setup to gpu-watch.

it works with this parameters:

poll_time=30
card_first=0
card_count=0

thanks

[xf2_org]

You passed an incorrect card setup to gpu-watch.

it works with this parameters:

poll_time=30
card_first=0
card_count=0

card_count=0 means zero cards...

[barbagianni]

card_count=0 means zero cards...

If I write card_count=1, it gives this error:

C:\Python27>gpu-watch.py CONFIG-FILE
Sat May 28 21:11:22 2011 Server Starts
registering card 0 (id 0)
Sat May 28 21:11:22 2011 Polling GPU data...
Traceback (most recent call last):
  File "C:\Python27\gpu-watch.py", line 130, in <module>
    w.watch()
  File "C:\Python27\gpu-watch.py", line 75, in watch
    self.watch_card(card)
  File "C:\Python27\gpu-watch.py", line 81, in watch_card
    print "Temperature %.2fC, fan speed %d%%" % (temp, fanspeed)
TypeError: float argument required, not NoneType

[inh]

works great for me, thanks!

[marcus_of_augustus]

Nice idea.

Won't work on multi-gpu cards like 5970 or 6990 since they have two adapter per fan. Unless it was modified to address the fans at 0.0, 0.2 and 0.4, etc (for a bank of 3 5970s) and somehow average or take highest temp. from adapter pairs, [0.0 0.1] [0.2.0.3] [0.4 0.5] etc as inputs ....

[mikegogulski]

Modified version below. REQUIRES dual-GPU cards. Autodetects the cards, and manages all of them. Panic switch at 90C immediately sets that card's fan to 100%.

Code:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sys
import time
import re
import os
import subprocess
from string import split

settings = {
	'poll_time': 60,
	'temp_low': 70.0,
        'temp_high': 76.0,
	'aticonfig_path': '/usr/bin/aticonfig'
}

class GPU():
	def __init__(self, id, cardid, description):
		self.id = id
		self.cardid = cardid
		self.description = description

	def temp(self):
		os.environ['DISPLAY'] = ":0"
		try:
			line = subprocess.check_output(
				[settings['aticonfig_path'],
				 "--adapter=" + str(self.id),
				 "--odgt"])
			m = re.search('Temperature - (\d+\.\d+)\s*C', line)
			if m is None:
				return None

			return float(m.group(1))
		except OSError:
			return None

class Card():
	def __init__(self, id, gpus):
		self.id = id
		self.x11_id = ':0.' + str(id)
		self.gpus = gpus

	def fanspeed(self, val=None):
		os.environ['DISPLAY'] = self.x11_id
		if val is not None:
			subprocess.call( [settings['aticonfig_path'],
				 "--pplib-cmd", "set fanspeed 0 " + str(val)])
		else:
			line = subprocess.check_output(
				[settings['aticonfig_path'],
				 "--pplib-cmd", "get fanspeed 0"])
			m = re.search('Fan Speed: (\d+)%', line)
			if m is None:
				return False
			return int(m.group(1))

class Watcher():
	def __init__(self):
		self.cards = []
		os.environ['DISPLAY'] = ":0"
		out = subprocess.check_output([settings['aticonfig_path'],
			"--list-adapters"])
		card = 0
		gpu = 0
		lines = split(out, "\n")
		for line in lines:
			r = re.search('^[ \*]+(\d+)\. [\d:\.]+ (.+)$', line)
			if r is None:
				continue
			gpuid = int(r.group(1))
			desc = r.group(2)
			if gpu % 2 == 0:
				self.cards.append(Card(card, []))
			self.cards[-1].gpus.append(GPU(gpuid, card, desc))
			print "gpu %d card %d desc %s" % (gpuid, card, desc)
			if gpu % 2 == 1:
				card = card + 1
			gpu = gpu + 1

	def watch(self):
		while True:
			for card in self.cards:
				if card is None:
					continue
				self.watch_card(card)
			time.sleep(settings['poll_time'])
			print "----------------------------------"

	def watch_card(self, card):
		fanspeed = card.fanspeed()
		fandelta = 0
		print "Card %d: fan speed %d%%" % (card.id, fanspeed)
		for gpu in card.gpus:
			temp = gpu.temp()
			print "Card %d GPU %d: Temperature %.2f °C" % (
				card.id, gpu.id, temp)
			if temp >= 90.0:
				fandelta = 100 - fanspeed
				break
			if temp > settings['temp_high']:
				if fanspeed <= 90:
					fandelta = +10
				else:
					fandelta = 100 - fanspeed
				break
			elif temp < settings['temp_low']:
				if fanspeed <= 5:
					continue
				fandelta = -5
		if fandelta != 0 and fanspeed + fandelta >= 10:
			print "Card %d: Adjusting fan %d%% to %d%%" % (
				card.id, fandelta, fanspeed + fandelta)
			card.fanspeed(fanspeed + fandelta)

if __name__ == '__main__':
	if len(sys.argv) > 1:
		try:
			f = open(sys.argv[1])
			for line in f:
				m = re.search('^(\w+)\s*=\s*(\S.*)$', line)
				if m is None:
					continue
				settings[m.group(1)] = m.group(2)
			f.close()
		except (OSError, IOError), e:
			pass

	settings['poll_time'] = int(settings['poll_time'])
	settings['temp_low'] = float(settings['temp_low'])
	settings['temp_high'] = float(settings['temp_high'])
	print repr(settings)
	w = Watcher()
	w.watch()

[ryepdx]

With AmdOverdriveCtrl you can do that. ...

The fan control worked great, but the over/underclocking part messes my GPU. ...

Wait, what has it been doing with your GPU? I've been using it for over/underclocking my GPUs and I can't seem to get anything near what I should be getting.

[MiningBuddy]

This is an excellent script and I have been testing it, but I have found a problem.
If I stop the load on my gpu the script continues to lower my temps until the fan is 0% and it keeps trying to lower it further.
When the load is applied back onto my cards the temp rises very fast, like 30C straight up to 80C before the script has changed the fan speed above 20%.

Is there anyway to code this to use a fan profile system similar to other popular fan control software?