How to reboot a rig if it stops mining?

[paulearley]

I have modified not.you's code, cleaning it up a bit and adding the ability to just text you when your miner has stopped or text you and automatically reboot. It has been working for about 2 weeks without false alarms and 1 needed reboot.

Name the code below with file name MinitorMining.ps1

Code:

$LogFile     = "MiningMonitor.log"
$ProcessName = "openhardwaremonitor"
$Date        = Get-Date
$TooLow      = 40		  # This is the Celsius temperature you wish to "flag" as an offline GPU
$TooHigh     = 70         # This temperature triggers a warning text
$lowestReportedTemp = 1000  # Remember to check that HardwareMonitor is reporting in Celsius
$numIterations = 9        # Set the number of iterations to test before considering a "failed" miner process. 
$rebootIfLow = $true     # Set to true to send text and reboot (production) when low temps are found. Set to false to only send text.

# Textual elements
$textBodyReboot = "At $Date, Miner rebooted - GPU temperature dropped to:"
$textBodyWarning = "At $Date, GPU temperature dropped to:"

# Constants for email message
$senderEmail = 'youremail@yourprovider.com'
$senderPassword = 'emailpassword'
# you must set this for your phone number and carrier. The list is here: http://www.emailtextmessages.com/
$textNumberAndCarrier = '1234567899@txt.yourcarrier.net' 
$textSubject =  'Miner Message';
$smtpServer = 'smtp.mail.yourprovider.com'
$port = '587'
$useSsl = $true


if ($rebootIfLow) {
    Write-Host "Monitor will send Message and Reboot if Needed"
    $textBody = $textBodyReboot
    } 
    else {
    Write-Host "Monitor is in Message Only Mode"
    $textBody = $textBodyWarning
    }

# [console]::TreatControlCAsInput = $true
try 
{
#Test if openhardwaremonitor is running and if not, start it
if((get-process $ProcessName -ErrorAction SilentlyContinue) -eq $Null)
    { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }
else
    { echo "OpenHardwareMonitor is running" }

$countOfIterationsLow = 0
#if the computer just started or OpenHardwareMonitor is slow starting we will have problems, so insert wait here
Start-Sleep 120

For ($i=1; $i -le $numIterations; $i++) {
	# Query GPU temperature from OpenHardwareMonitor
	$GPUTempObj = Get-WmiObject -namespace root\openhardwaremonitor -class sensor | Where-Object {$_.SensorType -Match "temperature" -and $_.Identifier -like "*gpu*"} 
    $GPU_Num = 0;
    ForEach($GPU In $GPUTempObj) 
		 {
		 $GPUTemp = $GPU.value
		 if($GPUTemp -lt $TooLow)
			{
				Write-Host "GPU$GPU_Num is $GPUTemp, below cutoff of $TooLow"
				"$Date - GPU$GPU_Num is $GPUTemp, below cutoff of $TooLow" | Out-File $LogFile -Append
				$countOfIterationsLow = $countOfIterationsLow + 1 
                if ($GPUTemp -lt $lowestReportedTemp) { $lowestReportedTemp = $GPUTemp }		
			}
		  else 
			{
				Write-Host "GPU$GPU_Num is OK at $GPUTemp degrees (above $TooLow)"   
			}
            $GPU_Num++
		 }
         if ($Host.UI.RawUI.KeyAvailable -and (3 -eq [int]$Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyUp,NoEcho").Character))
            {
                Write-Host "You pressed CTRL-C. Do you want to continue Mining Monitor (Y/N)?" 
                $key = $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown")
                if ($key.Character -eq "N") { break; }
            }   
            
	#if we have bad timing on a driver crash (and recovery) or work restarts we may get low results, so we wait between tests
	if ($i -lt $numIterations) {
        Write-Host "- - - - - - - - - - -"
		Start-Sleep 20
		}
    }

# All tests (determined by &numIterations) have to get a low result restart
if($countOfIterationsLow -ge $numIterations)
     {"$Date - $textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $TooLow - restarting" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = $textSubject
			Body       = "$textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $TooLow"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		if ($rebootIfLow) {
			Restart-Computer -force
		}
     }
elseif ($lowestReportedTemp -lt $TooLow)
     {"$Date - $countOfIterationsLow results below cutoff, lowest was $lowestReportedTemp with report cutoff of $TooLow" | Out-File $LogFile -Append}
else 
    {"$Date - No concerning results, cutoff temperature currently: $TooLow" | Out-File $LogFile -Append}
}

finally
{
    "$Date - Mining Monitor Stopped" | Out-File $LogFile -Append
}   

Below are the contents of batch file to start it. I named mine "StartMiningMonitor.bat", but it can be named anything you want

Code:

@echo off
set counter=0
:start
cls
set /a counter=counter+1
echo Mining Monitor loop %counter%
powershell.exe .\MonitorMining.ps1
TIMEOUT /T 30
goto start 

NOTE: This script has to be placed in the same folder as OpenHardwareMonitor. If you want to place it somewhere else, the following line has to be modified:

Code:

   { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }

You are free to use and adapt to your needs. I would appreciate improvement suggestions and feedback.

othgeek

[bigjee]

I'm using modded RX480 8GB with Samsung memory and out of 24 cards, 2 on different rigs make problems. Yes, one option is to revert to stock BIOS, but I'm looking for another solution since that happens maybe once per day. Win 10 Pro.

Claymore's Dual Eth+Dcr miner shows 28.4MH/s for all 5 cards in the rig, and randomly, as I said, once per day, one card in the setup starts to drop to 21MH/s and eventually after few minutes 0MH/s. Then miner's watchdog tries to restart mining which goes well, but is stuck on the list of GPU's and doesn't start mining. The solution is to manually close the miner and just start it again.

EDIT: To make it clear, the rig doesn't freeze. Only the miner stops mining. So I can manually restart the miner WITHOUT rebooting the rig, but that was one option to restart the miner as well.

Now I'm searching for a solution which would reboot the whole rig (or just kill the miner process and start it again) if the GPU temperature falls below say 60°C ... something opposite to the overheating protection

Any ideas?

Seems like youve narrowed down the problem to 2 rigs.
How did you initially tune the cards.
did you use HWOinfo to set the right clock, core, power settings?
How many memory errors are you getting over an hour/ a day?
Might be better if you stabilise these two rigs before adding watchdogs etc.

[samsoccer7]

This seems pretty awesome.  Couple questions.  While testing I changed the TooHigh to 69 but it still keeps saying "OK" and is not triggering a text.  Secondly, instead of reboot, is it possible to change the command to shutdown?  When I switch some algos I lose 3 GPUs pretty regularly and the only fix so far is to shutdown then turn on again (which I can do with bios power on)?  The bigger issue is why the high point isn't triggering a text.  Thanks!

I have modified not.you's code, cleaning it up a bit and adding the ability to just text you when your miner has stopped or text you and automatically reboot. It has been working for about 2 weeks without false alarms and 1 needed reboot.

Name the code below with file name MinitorMining.ps1

Code:

$LogFile     = "MiningMonitor.log"
$ProcessName = "openhardwaremonitor"
$Date        = Get-Date
$TooLow      = 40		  # This is the Celsius temperature you wish to "flag" as an offline GPU
$TooHigh     = 70         # This temperature triggers a warning text
$lowestReportedTemp = 1000  # Remember to check that HardwareMonitor is reporting in Celsius
$numIterations = 9        # Set the number of iterations to test before considering a "failed" miner process. 
$rebootIfLow = $true     # Set to true to send text and reboot (production) when low temps are found. Set to false to only send text.

# Textual elements
$textBodyReboot = "At $Date, Miner rebooted - GPU temperature dropped to:"
$textBodyWarning = "At $Date, GPU temperature dropped to:"

# Constants for email message
$senderEmail = 'youremail@yourprovider.com'
$senderPassword = 'emailpassword'
# you must set this for your phone number and carrier. The list is here: http://www.emailtextmessages.com/
$textNumberAndCarrier = '1234567899@txt.yourcarrier.net' 
$textSubject =  'Miner Message';
$smtpServer = 'smtp.mail.yourprovider.com'
$port = '587'
$useSsl = $true


if ($rebootIfLow) {
    Write-Host "Monitor will send Message and Reboot if Needed"
    $textBody = $textBodyReboot
    } 
    else {
    Write-Host "Monitor is in Message Only Mode"
    $textBody = $textBodyWarning
    }

# [console]::TreatControlCAsInput = $true
try 
{
#Test if openhardwaremonitor is running and if not, start it
if((get-process $ProcessName -ErrorAction SilentlyContinue) -eq $Null)
    { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }
else
    { echo "OpenHardwareMonitor is running" }

$countOfIterationsLow = 0
#if the computer just started or OpenHardwareMonitor is slow starting we will have problems, so insert wait here
Start-Sleep 120

For ($i=1; $i -le $numIterations; $i++) {
	# Query GPU temperature from OpenHardwareMonitor
	$GPUTempObj = Get-WmiObject -namespace root\openhardwaremonitor -class sensor | Where-Object {$_.SensorType -Match "temperature" -and $_.Identifier -like "*gpu*"} 
    $GPU_Num = 0;
    ForEach($GPU In $GPUTempObj) 
		 {
		 $GPUTemp = $GPU.value
		 if($GPUTemp -lt $TooLow)
			{
				Write-Host "GPU$GPU_Num is $GPUTemp, below cutoff of $TooLow"
				"$Date - GPU$GPU_Num is $GPUTemp, below cutoff of $TooLow" | Out-File $LogFile -Append
				$countOfIterationsLow = $countOfIterationsLow + 1 
                if ($GPUTemp -lt $lowestReportedTemp) { $lowestReportedTemp = $GPUTemp }		
			}
		  else 
			{
				Write-Host "GPU$GPU_Num is OK at $GPUTemp degrees (above $TooLow)"   
			}
            $GPU_Num++
		 }
         if ($Host.UI.RawUI.KeyAvailable -and (3 -eq [int]$Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyUp,NoEcho").Character))
            {
                Write-Host "You pressed CTRL-C. Do you want to continue Mining Monitor (Y/N)?" 
                $key = $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown")
                if ($key.Character -eq "N") { break; }
            }   
            
	#if we have bad timing on a driver crash (and recovery) or work restarts we may get low results, so we wait between tests
	if ($i -lt $numIterations) {
        Write-Host "- - - - - - - - - - -"
		Start-Sleep 20
		}
    }

# All tests (determined by &numIterations) have to get a low result restart
if($countOfIterationsLow -ge $numIterations)
     {"$Date - $textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $TooLow - restarting" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = $textSubject
			Body       = "$textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $TooLow"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		if ($rebootIfLow) {
			Restart-Computer -force
		}
     }
elseif ($lowestReportedTemp -lt $TooLow)
     {"$Date - $countOfIterationsLow results below cutoff, lowest was $lowestReportedTemp with report cutoff of $TooLow" | Out-File $LogFile -Append}
else 
    {"$Date - No concerning results, cutoff temperature currently: $TooLow" | Out-File $LogFile -Append}
}

finally
{
    "$Date - Mining Monitor Stopped" | Out-File $LogFile -Append
}   

Below are the contents of batch file to start it. I named mine "StartMiningMonitor.bat", but it can be named anything you want

Code:

@echo off
set counter=0
:start
cls
set /a counter=counter+1
echo Mining Monitor loop %counter%
powershell.exe .\MonitorMining.ps1
TIMEOUT /T 30
goto start 

NOTE: This script has to be placed in the same folder as OpenHardwareMonitor. If you want to place it somewhere else, the following line has to be modified:

Code:

   { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }

You are free to use and adapt to your needs. I would appreciate improvement suggestions and feedback.

othgeek

[paulearley]

samsoccer7: The tooHigh trigger is not yet implemented -- that code will be posted in the next few days. Sorry for the teaser.

The ShutDown is a simple process -- look in the code for "Restart-Computer" and replace it with "Stop-Computer".

othgeek

[samsoccer7]

Awesome I’ll try it out. When I tested the too low it restarted but I didn’t get an email. Not sure what I’m doing wrong I used gmail Ssl info and put it in there but it didn’t work. Is there some other email setup I need to do?

samsoccer7: The tooHigh trigger is not yet implemented -- that code will be posted in the next few days. Sorry for the teaser.

The ShutDown is a simple process -- look in the code for "Restart-Computer" and replace it with "Stop-Computer".

othgeek

[paulearley]

Email setup is tricky. If you are using 2 factor authorization on Gmail you cannot send an email without getting a special code (search for this in google account). See this link for more information: https://www.digitalocean.com/community/tutorials/how-to-use-google-s-smtp-server. I avoided this with Gmail for this very reason and use yahoo. You could easily set up a new yahoo account just to send you these texts.

othgeek

[paulearley]

Below is the updated PowerShell script to monitor for GPUs being to cool (indicating monitor hang) or too hot.
It uses OpenHardwareMonitor, and is based on not.you's original script.

The file name, as noted above. should be called MonitorMining.ps1 and resides in the same directory as OpenHardwareMonitor.

Code:

$LogFile     = "MiningMonitor.log"
$verboseLogInfo = $false
$ProcessName = "openhardwaremonitor"
$Date        = Get-Date
$tooLow      = 40		    # This is the Celsius temperature you wish to "flag" as an offline GPU
$reportIftooHigh = $true
$tooHigh     = 60           # This temperature triggers a warning text
$wayTooHigh  = 82           # This temperature and above shuts down the miner if reached on ONE occasion
$lowestReportedTemp  = 1000 # Remember to check that HardwareMonitor is reporting in Celsius
$highestReportedTemp = 0
$numIterations = 9          # Set the number of iterations to test before considering a "failed" miner process (too low). 
$rebootIfLow = $true        # Set to true to send text and reboot (production) when low temps are found. Set to false to only send text.

# Textual elements
$textBodyReboot = "At $Date, Miner rebooted - GPU temperature dropped to:"
$textBodyWarning = "At $Date, GPU temperature dropped to:"
$textBodyHotWarning = "At $Date, a GPU reached warning temp of:"

# Constants for email message
$senderEmail = 'youremailaddress@yourprovider.com'
$senderPassword = 'yourpassword'
# you must set this for your phone number and carrier. The list is here: http://www.emailtextmessages.com/
$textNumberAndCarrier = 'yourphonenumber@txt.yourcarrier.net'
$textSubject =  'Miner Message';
$smtpServer = 'smtp.mail.yahoo.com'
$port = '587'
$useSsl = $true


if ($rebootIfLow) {
    Write-Host "Monitor will send Message and Reboot if any GPU is consistently below $tooLow"
    $textBody = $textBodyReboot
    } 
    else {
    Write-Host "Monitor will send Text Message if any GPU is consistently below $tooLow"
    $textBody = $textBodyWarning
    }
if ($reportIftooHigh) { Write-Host "Monitoring for hot GPUs. Message sent at $tooHigh, Reboot if any GPU reaches $wayTooHigh" }

# [console]::TreatControlCAsInput = $true
try 
{
#Test if openhardwaremonitor is running and if not, start it
if((get-process $ProcessName -ErrorAction SilentlyContinue) -eq $Null)
    { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }
elseif ($verboseLogInfo )
    { echo "OpenHardwareMonitor is running" }

$countOfIterationsLow = 0
$countIterationsHigh  = 0
$hotGPU = 99
#if the computer just started or OpenHardwareMonitor is slow starting we will have problems, so insert wait here
Start-Sleep 90

For ($i=1; $i -le $numIterations; $i++) {
	# Query GPU temperature from OpenHardwareMonitor
	$GPUTempObj = Get-WmiObject -namespace root\openhardwaremonitor -class sensor | Where-Object {$_.SensorType -Match "temperature" -and $_.Identifier -like "*gpu*"} 
    $GPU_Num = 0;
    ForEach($GPU In $GPUTempObj) 
		 {
		 $GPUTemp = $GPU.value
		 if($GPUTemp -lt $tooLow)
			{
				Write-Host "GPU$GPU_Num is $GPUTemp, below cutoff of $tooLow"
				"$Date - GPU$GPU_Num is $GPUTemp, below cutoff of $tooLow" | Out-File $LogFile -Append
				$countOfIterationsLow = $countOfIterationsLow + 1 
                if ($GPUTemp -lt $lowestReportedTemp) { $lowestReportedTemp = $GPUTemp }		
			}
		  else 
			{
				Write-Host "GPU$GPU_Num is OK at $GPUTemp degrees (above $tooLow)"   
			}
            $GPU_Num++
		 }
         if ($reportIftooHigh) {
             if($GPUTemp -ge $tooHigh)
			    {
				    Write-Host "GPU$GPU_Num is $GPUTemp, hotter than the cutoff of $tooHigh degrees"
				    "$Date - GPU$GPU_Num is $GPUTemp, hotter than cutoff of $tooHigh" | Out-File $LogFile -Append
				     $countOfIterationsHigh++ 
                    if ($GPUTemp -gt $highestReportedTemp) { $highestReportedTemp = $GPUTemp; $hotGPU = $GPU_Num }		
			    }
              if ($GPUTemp -ge $wayTooHigh) { Stop-Computer -force }  
         }
         if ($Host.UI.RawUI.KeyAvailable -and (3 -eq [int]$Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyUp,NoEcho").Character))
            {
                Write-Host "You pressed CTRL-C. Do you want to continue Mining Monitor (Y/N)?" 
                $key = $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown")
                if ($key.Character -eq "N") { break; }
            }   
            
	#if we have bad timing on a driver crash (and recovery) or work restarts we may get low results, so we wait between tests
	if ($i -lt $numIterations) {
        Write-Host "- - - - - - - - - - -"
		Start-Sleep 20
		}
    }

# All tests (determined by &numIterations) have to get a low result restart
if($countOfIterationsLow -ge $numIterations)
     {"$Date - $textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $tooLow - restarting" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = $textSubject
			Body       = "$textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $tooLow"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		if ($rebootIfLow) {
			Restart-Computer -force
		}
     }
elseif ($lowestReportedTemp -lt $tooLow)
     {"$Date - $countOfIterationsLow results below cutoff, lowest was $lowestReportedTemp with report cutoff of $tooLow" | Out-File $LogFile -Append}
elseif ($verboseLogInfo) 
    {"$Date - No concerning results, cutoff temperature currently: $tooLow" | Out-File $LogFile -Append}


# Some tests (determined by &numIterations) reported GPUs too high
if($countOfIterationsHigh -ge $numIterations) {
     "$Date - GPU$hotGPU reported $highestReportedTemp degrees $countOfIterationsHigh times with warning set to $tooHigh" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = "Hot GPU - $textSubject"
			Body       = "$Date - GPU$hotGPU reached $highestReportedTemp degrees $countOfIterationsHigh times. Current warning set to $tooHigh"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		}
}
finally
{
    if ($verboseLogInfo) { "$Date - Mining Monitor Stopped" | Out-File $LogFile -Append }
} 

Here is the batch file, which should be in the same directory and can be named anything (well anything.bat).

Code:

@echo off
set counter=0
:start
cls
set /a counter=counter+1
echo Mining Monitor loop %counter%
powershell.exe .\MonitorMining.ps1
TIMEOUT /T 30
goto start 

[samsoccer7]

I don’t use 2FA for gmail so I’m not sure why it’s not working. I’ll try yahoo and see if that works. Thanks for posting the new script. Sick today but I’ll try it out soon. This could be a game changer for monitoring my rigs.

Email setup is tricky. If you are using 2 factor authorization on Gmail you cannot send an email without getting a special code (search for this in google account). See this link for more information: https://www.digitalocean.com/community/tutorials/how-to-use-google-s-smtp-server. I avoided this with Gmail for this very reason and use yahoo. You could easily set up a new yahoo account just to send you these texts.

othgeek

[samsoccer7]

I got this message when trying it out today (I used 70 as the toohigh, left 82 as the waytoohigh)

Send-MailMessage : The SMTP server requires a secure connection or the client was not authenticated. The server
response was: 5.7.1 Authentication required
At C:\Users\Mining2\Downloads\openhardwaremonitor-v0.8.0-beta\OpenHardwareMonitor\monitormining.ps1:136 char:3
+         Send-MailMessage @MailArgs
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.Mail.SmtpClient:SmtpClient) [Send-MailMessage], SmtpExcept
   ion
    + FullyQualifiedErrorId : SmtpException,Microsoft.PowerShell.Commands.SendMailMessage

Below is the updated PowerShell script to monitor for GPUs being to cool (indicating monitor hang) or too hot.
It uses OpenHardwareMonitor, and is based on not.you's original script.

The file name, as noted above. should be called MonitorMining.ps1 and resides in the same directory as OpenHardwareMonitor.

Code:

$LogFile     = "MiningMonitor.log"
$verboseLogInfo = $false
$ProcessName = "openhardwaremonitor"
$Date        = Get-Date
$tooLow      = 40		    # This is the Celsius temperature you wish to "flag" as an offline GPU
$reportIftooHigh = $true
$tooHigh     = 60           # This temperature triggers a warning text
$wayTooHigh  = 82           # This temperature and above shuts down the miner if reached on ONE occasion
$lowestReportedTemp  = 1000 # Remember to check that HardwareMonitor is reporting in Celsius
$highestReportedTemp = 0
$numIterations = 9          # Set the number of iterations to test before considering a "failed" miner process (too low). 
$rebootIfLow = $true        # Set to true to send text and reboot (production) when low temps are found. Set to false to only send text.

# Textual elements
$textBodyReboot = "At $Date, Miner rebooted - GPU temperature dropped to:"
$textBodyWarning = "At $Date, GPU temperature dropped to:"
$textBodyHotWarning = "At $Date, a GPU reached warning temp of:"

# Constants for email message
$senderEmail = 'youremailaddress@yourprovider.com'
$senderPassword = 'yourpassword'
# you must set this for your phone number and carrier. The list is here: http://www.emailtextmessages.com/
$textNumberAndCarrier = 'yourphonenumber@txt.yourcarrier.net'
$textSubject =  'Miner Message';
$smtpServer = 'smtp.mail.yahoo.com'
$port = '587'
$useSsl = $true


if ($rebootIfLow) {
    Write-Host "Monitor will send Message and Reboot if any GPU is consistently below $tooLow"
    $textBody = $textBodyReboot
    } 
    else {
    Write-Host "Monitor will send Text Message if any GPU is consistently below $tooLow"
    $textBody = $textBodyWarning
    }
if ($reportIftooHigh) { Write-Host "Monitoring for hot GPUs. Message sent at $tooHigh, Reboot if any GPU reaches $wayTooHigh" }

# [console]::TreatControlCAsInput = $true
try 
{
#Test if openhardwaremonitor is running and if not, start it
if((get-process $ProcessName -ErrorAction SilentlyContinue) -eq $Null)
    { Start-Process -FilePath ".\OpenHardwareMonitor.exe" -WorkingDirectory ".\"} # -WindowStyle Minimized; echo "Starting OpenHardwareMonitor..." }
elseif ($verboseLogInfo )
    { echo "OpenHardwareMonitor is running" }

$countOfIterationsLow = 0
$countIterationsHigh  = 0
$hotGPU = 99
#if the computer just started or OpenHardwareMonitor is slow starting we will have problems, so insert wait here
Start-Sleep 90

For ($i=1; $i -le $numIterations; $i++) {
	# Query GPU temperature from OpenHardwareMonitor
	$GPUTempObj = Get-WmiObject -namespace root\openhardwaremonitor -class sensor | Where-Object {$_.SensorType -Match "temperature" -and $_.Identifier -like "*gpu*"} 
    $GPU_Num = 0;
    ForEach($GPU In $GPUTempObj) 
		 {
		 $GPUTemp = $GPU.value
		 if($GPUTemp -lt $tooLow)
			{
				Write-Host "GPU$GPU_Num is $GPUTemp, below cutoff of $tooLow"
				"$Date - GPU$GPU_Num is $GPUTemp, below cutoff of $tooLow" | Out-File $LogFile -Append
				$countOfIterationsLow = $countOfIterationsLow + 1 
                if ($GPUTemp -lt $lowestReportedTemp) { $lowestReportedTemp = $GPUTemp }		
			}
		  else 
			{
				Write-Host "GPU$GPU_Num is OK at $GPUTemp degrees (above $tooLow)"   
			}
            $GPU_Num++
		 }
         if ($reportIftooHigh) {
             if($GPUTemp -ge $tooHigh)
			    {
				    Write-Host "GPU$GPU_Num is $GPUTemp, hotter than the cutoff of $tooHigh degrees"
				    "$Date - GPU$GPU_Num is $GPUTemp, hotter than cutoff of $tooHigh" | Out-File $LogFile -Append
				     $countOfIterationsHigh++ 
                    if ($GPUTemp -gt $highestReportedTemp) { $highestReportedTemp = $GPUTemp; $hotGPU = $GPU_Num }		
			    }
              if ($GPUTemp -ge $wayTooHigh) { Stop-Computer -force }  
         }
         if ($Host.UI.RawUI.KeyAvailable -and (3 -eq [int]$Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyUp,NoEcho").Character))
            {
                Write-Host "You pressed CTRL-C. Do you want to continue Mining Monitor (Y/N)?" 
                $key = $Host.UI.RawUI.ReadKey("NoEcho, IncludeKeyDown")
                if ($key.Character -eq "N") { break; }
            }   
            
	#if we have bad timing on a driver crash (and recovery) or work restarts we may get low results, so we wait between tests
	if ($i -lt $numIterations) {
        Write-Host "- - - - - - - - - - -"
		Start-Sleep 20
		}
    }

# All tests (determined by &numIterations) have to get a low result restart
if($countOfIterationsLow -ge $numIterations)
     {"$Date - $textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $tooLow - restarting" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = $textSubject
			Body       = "$textBody $lowestReportedTemp on $countOfIterationsLow occasions with cutoff of $tooLow"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		if ($rebootIfLow) {
			Restart-Computer -force
		}
     }
elseif ($lowestReportedTemp -lt $tooLow)
     {"$Date - $countOfIterationsLow results below cutoff, lowest was $lowestReportedTemp with report cutoff of $tooLow" | Out-File $LogFile -Append}
elseif ($verboseLogInfo) 
    {"$Date - No concerning results, cutoff temperature currently: $tooLow" | Out-File $LogFile -Append}


# Some tests (determined by &numIterations) reported GPUs too high
if($countOfIterationsHigh -ge $numIterations) {
     "$Date - GPU$hotGPU reported $highestReportedTemp degrees $countOfIterationsHigh times with warning set to $tooHigh" | Out-File $LogFile -Append
		$MailArgs = @{
			From       = $senderEmail
			To         = $textNumberAndCarrier
			Subject    = "Hot GPU - $textSubject"
			Body       = "$Date - GPU$hotGPU reached $highestReportedTemp degrees $countOfIterationsHigh times. Current warning set to $tooHigh"
			SmtpServer = $smtpServer 
			Port       = $port
			UseSsl     = $useSsl
			Credential = New-Object pscredential $senderEmail,$($senderPassword |ConvertTo-SecureString -AsPlainText -Force)
			}
		Send-MailMessage @MailArgs
		}
}
finally
{
    if ($verboseLogInfo) { "$Date - Mining Monitor Stopped" | Out-File $LogFile -Append }
} 

Here is the batch file, which should be in the same directory and can be named anything (well anything.bat).

Code:

@echo off
set counter=0
:start
cls
set /a counter=counter+1
echo Mining Monitor loop %counter%
powershell.exe .\MonitorMining.ps1
TIMEOUT /T 30
goto start 

[paulearley]

Hmmm... I am not having problems there. Have you reviewed this information: https://help.yahoo.com/kb/SLN4724.html.
Have you tried port 465? (non-SSL).
othgeek

[majika]

Just a thought
regarding managing fully processes on the OS (if win) Leverage the power of SysIntenals Suite of tools.
Especially, for remote tasks which could be used as an alternative to the "OpenHardwareMonitor" && Powershell script/method contained in this thread. (Works great btw).

https://docs.microsoft.com/en-us/sysinternals/downloads/pskill

To this day they are very useful utils as were they from back in the day.!

[Kasmetski]

If you need a tool to reset the whole rig - like pushing the power button. You can use my tool - https://bitcointalk.org/index.php?topic=1933467.0 .It's free and you need only a Raspberry Pi and relay + some wires.

[dongle]

If your miner has not been mining for more than an hour, the pool will usually email you. You can also apply for SMS.

[agricointech]

If mining rig crash, you can use wifi plug to force power cycle.

like these products

Wifi Plug

[Bimmber]

I'm using modded RX480 8GB with Samsung memory and out of 24 cards, 2 on different rigs make problems. Yes, one option is to revert to stock BIOS, but I'm looking for another solution since that happens maybe once per day. Win 10 Pro.

Claymore's Dual Eth+Dcr miner shows 28.4MH/s for all 5 cards in the rig, and randomly, as I said, once per day, one card in the setup starts to drop to 21MH/s and eventually after few minutes 0MH/s. Then miner's watchdog tries to restart mining which goes well, but is stuck on the list of GPU's and doesn't start mining. The solution is to manually close the miner and just start it again.

EDIT: To make it clear, the rig doesn't freeze. Only the miner stops mining. So I can manually restart the miner WITHOUT rebooting the rig, but that was one option to restart the miner as well.

Now I'm searching for a solution which would reboot the whole rig (or just kill the miner process and start it again) if the GPU temperature falls below say 60°C ... something opposite to the overheating protection

Any ideas?

Might not be ideal solution for you, but my rigs are plugged into wall via wifi socket. If I`m not at home, I check once an hour power consumption and if its below of what its normally consuming, I know its not working as it should. I then press OFF on my phone`s app, wait 20 seconds, then ON. I`m using SMOS, which mines automatically once you apply power to your rig, so there is nothing else I need to do.

Example:
https://www.tp-link.com/us/products/details/cat-5516_HS110.html

[Thamon]

Wouldnt it just be easier if the nicehash miner executed a command on GPU lost. It know's it lost it when it shutsdown and restarts the excavator. Just look and see that you have settings for X GPU's and only X-? are currently available. HMM. execute the users shell command.  which could be a attempt to recover GPU then reboot or just reboot the system