SRR = Simple Rig Resetter
it's a little board that connects to your power/reset pins on the motherboard, it then listens for heartbeats from specific rigs, when the heartbeat stops, it means the rig froze, it hard re-sets the specific rig.

https://simplemining.net/download/SRR/PDF/SRR-manual-2017-02-10.pdf

I've been using one for a few months, saved me lots of time at the beginning.

Cool, thanks! I'll look again in the previous pages to catch up with this discussion.

I noticed that xmr-stack takes a while to load up all the gpu's with work - on 13 card systems this can take up to 3-4 minutes (it also depends on the thread/block count in xmr-stak config.txt - the settings from fullzero load under 2G of data, but you can tweak those numbers higher getting, resulting higher hash rates, but also increasing the initialization time).
Once stak loads all the cards, and assuming your OC settings are stable - it will behave itself for days. I meant to modify watchdog to recognize stak is running and increase the initialization time appropriately, but I still need to migrate to v0019. for now if you want to run watchdog, change the initial timeout to 360 seconds it should do the trick.
Once all the cards are loaded stak will spit out a bunch of submits to the pool in one shot.

thanks a lot Fullzero

I'm now able to use my 7x GPU with your modification

unfortunately watchdog is not working anymore as it saying that GPU utilisation is to low

not a big deal

thanks

Hi All, quick question, anyone try connecting over eth-proxy (eth-proxy.py) to nicehash?
If so, have you had any issues passing the rig id over the proxy to nicehash (eth-proxy.py uses "/" while nicehash likes ".")

PS
I am aware of the nicehash integration in nvOC and the available stratum proxies in the included miners, however, I am investigating this particular path at the moment.

If you have another rig, I would try testing each of the GPUs an verifying they work individually; if they all do, then I would get a pico for your server PSU and try using only the server PSU.  If you are using an atx PSU for the mobo and a server psu for everything else without joining them; this can cause problems.

Actually xmr-stak-cpu is a great idea! It's super stable - I actually added it to one image of nvOS where I'm also cpu mining a while ago and it plays really nice with everything else on the distro.
While at it I would suggest adding xmr-stak-nvidia I meant to test it when I had  some cycles to spare, but since xmr-stak-cpu is on the radar, maybe we can also add it's nvidia cousin?

It's great to see everyone get involved and speaking for myself, to feel like it's OK to contribute once in a while as well.
Hats off to fullzero, Maxximus007 and _Parallax_!

Here you go:
# IAmNotAJeep BTC address:  <13PnEKpfVzNseWkrm6LoueKcCMPj74zPv7>

OK thanks for the clarification, it's really neat and rewarding to see different approaches to this problem
Here is why I coded to test that all the cards meet the threshold as one with "if &&": as an example I'll use an event from from my test rig overnight: one card dropped, the "if &&" script waited for claymore to recover for one minute, then booted the system and that was that.
Total down time, 2 mins, if you add the 1 minute of reduced capacity waiting for the miner to right itself, 3 minutes impact.

The "if &&" code does tests for a graceful miner recovery -  by continuing to test the cards for above threshold utilization for 60 seconds after it detects a fault.
If the miner recovers, but just sits there (saw both Claymore/Genoil do exactly that a number of times) that's not good enough and the system gets a boot.
My other miner restart script did not handle this exact case and once every few days I would find the miner sitting pretty and blowing bubbles mining on one or two cards until I noticed because it did not "see" all the cards anymore but it did see some so it thought it "recovered".

If the miner recovers properly, all cards need to hit above threshold  and we can flush the counter and life goes on.
On my test rig, graceful miner recovery occurred 5-6 times in the past 24 hours without prompting a restart - which is desirable above either running at reduced capacity or 5-6 reboots (IMHO).

In contrast - if we test each card independently and increment the error counter one by one until it reaches the number of GPU's, then - depending on the number of cards in the system it could take a long time for all of them to fail - the more cards, the more time to fail (right? am I misunderstanding anything?) So the same event, would unfold differently: the test rig would continue at reduced capacity until COUNT reaches # of GPU's - but since it resets at next check, we can hobble on 5,4,3,2, 1 card until they all die or and the script kicks in or we freeze and require a manual intervention. This could be hours of impact (again if I'm reading this wrong, my apologies, but this is what I'm getting out of looking at it.)

So IMHO, by testing that all the cards meet the 90% utilization threshold (as one, all or nothing = if &&), we avoid hours of impact/decreased capacity. My other concern is that as soon as cards start dropping off one at a time the system gets unstable, increasing the risk of a hang or corrupted file system due to a hard crash.
My view is that it should be cycled at maximum stability for a graceful restart.

Maybe there is a third approach not considered yet, Thoughts?

... edit:
Actually one more thought - I did not test for this yet so I don't know the answer - but in the case where the miner does not see all the cards anymore, does this mean that nvidia-smi ALSO does not see all the cards anymore? If so, and if we get the number of cards from nvidia-smi, wouldn't the script assume that the rig has the right number of cards every time that nvidia-smi stop seeing one? I do recall cards disappearing even from nvidia-smi but I never kept track of this so I don't know how often this condition actually occurs.
  

@ Maxximus007
Thanks for putting these together, great collab!
I'm not a bash expert, so maybe I'm reading this wrong, but here are some thoughts.
The combined code seems to be evaluating each gpu individually for the fault condition to be met, which means if one fails and you have say 5 other cards working then it keeps going until all the cards give reduced output since all of them have to fail individually to increment the counter?So if 5/6 fail we keep going? (Again just looking at it and tracing it in my head so maybe I'm reading wrong).
The way I was thinking about it, is that I wanted all the cards to work at above 90% efficiency and reboot as soon as any card strays beyond the threshold - this is why I did the "if and" statement and didn't iterate though "if" statements alone (I didn't know how to iterate "if and" based on an unknown number of cards lol). I had a version giving 6xOK and such but I think it's more efficient to just get 1xOK if ALL meet the 90% criteria and start the countdown as soon as anything is out of norm - and if the miner recovers, flush the counter. I observed a number of these conditions with Claymore where it recovers half the time, but then eventually craps out and the script kicks in. I haven't seen it on my Genoil rig yet since my other script has kept it in check without any softcrash for day 3 now.

A thought about the power draw as threshold measure - it is power limit/card specific and I guess people would need to tune their power threshold to their power limit so I agree it's best to use gpu util. (My cards are at 82W limit for example).
Thoughts?


  

Pretty cool!  I'll try it tonight, lets hope this put the softcrash issues behind us.

Hey thats funny I just made a script doing something similar, although it checks the powerdraw.
Here it is:

You can combine the above with your code, and find the utilization like this:
You have to iterate the GPU, starting at 0 to get them all

#!/bin/bash

# Miner restart script V002
# By Maxximus007 && IAmNotAJeep
# for nvOC by fullzero
#

#########################
### BELOW CODE, NO NEED FOR EDITING
#########################
echo "$(date) - Starting miner restart script." | tee -a ${LOG_FILE}
# Creating a log file to record restarts
LOG_FILE="/home/m1/restartlog.txt"
if [ ! -e "$LOG_FILE" ] ; then
    touch "$LOG_FILE"
fi

MIN_UTIL=90
RESTART=0

while true
do
sleep 60

GPUS=$(nvidia-smi --query-gpu=count --format=csv,noheader,nounits | tail -1)

gpu=0
COUNT=0

while [ $gpu -lt $GPUS ]
do
  { IFS=', ' read UTIL; } < <( nvidia-smi -i $gpu --query-gpu=utilization.gpu --format=csv,noheader,nounits)

  let UTILIZATION=$( printf "%.0f" $UTIL )

  # If current utilizations lower than the limit count them:
  if [ $UTILIZATION -lt $MIN_UTIL ]
  then
    let COUNT=COUNT+1
  fi

  let gpu=gpu+1
done

if [ $COUNT -eq $GPUS ]
then
  if [ $RESTART -gt 1 ]
  then
    echo "$(date) - Utilization is too low: reviving did not work so restarting system" | tee -a ${LOG_FILE}
    sudo shutdown now -r
  fi
  echo "$(date) - Utilization is too low: kill miner and oneBash" | tee -a ${LOG_FILE}
  # If miner runs in screen 'miner' kill the screen
  screen -X -S miner kill
  # Best to restart oneBash - settings might be adjusted already
  kill ps -ef | awk '$NF~"oneBash" {print $2}'
  let RESTART=RESTART+1
else
  echo "$(date) - All good! Will check again in 60 seconds"
fi

done