How do you deal with overnight rig hangs? I ended up writing my own bot

[ocbot]

Hey miners. I run a few HiveOS rigs (NVIDIA, mixed cards - 3070s, a 3090, a 4070). For years the same stuff drove me nuts: hand-tuning overclocks per card, overnight hangs you only notice in the morning from the pool graph, and driving to the farm just to hit reset.

I tried the stock watchdogs - they either reboot the rig on any hiccup (and wreck your tuning) or stay silent when a card is actually dead. So I sat down and wrote my own bot. What it does:

- tunes overclocks per card, and only accepts a result after a long stability run (2 hours in my setup), not "worked for five minutes";
- tells a one-off glitch from real instability: no pointless reboots, but real lockups aren't ignored either;
- recovers a fully wedged driver by itself within minutes, through an escalation chain;
- keeps a total PSU power budget, so tuning one card doesn't starve the others;
- reports everything to Telegram: what it tuned, what crashed, what it fixed.

Bottom line on my rigs: zero overnight downtime, no manual tuning at all. One weird thing I noticed: on my 3090 one miner kept giving ~70 instead of the usual ~130, until the bot cross-checked the preset against a different miner and rejected it. Anyone seen that with onezerominer?

How do you all handle hangs and tuning? Maybe I reinvented the wheel and there's something decent out of the box?

[BitMaxz]

You are in the Bitcoin mining section, where most of the units here are ASIC for mining BTC. You might get less response here, so I suggest moving this to the altcoin mining section instead, or I'll ask mods to move this thread.

You might be pushing your cards too high, which causes hangs if you are using the HiveOS; it is supposed to be automatically restarted if you enable the watchdog.

If the watchdog didn't work well, you better try another setup than pushing your cards too much.
Have you reviewed the miner logs instead check the reason why it hangs?

Using a bot won't solve anything; you're just pushing your card that is the way to shrink the lifespan of your card without understanding what the cause of your issue is.

If it's just overheating, then try increasing the fan speed and reducing the frequency at least to drop down the maximum temp, or you can set the watchdog; if the temp hits that temp, it should restart. That's my solution before and during summer.