How do you deal with overnight rig hangs? I ended up writing my own bot

[ocbot]

Hey miners. I run a few HiveOS rigs (NVIDIA, mixed cards - 3070s, a 3090, a 4070). For years the same stuff drove me nuts: hand-tuning overclocks per card, overnight hangs you only notice in the morning from the pool graph, and driving to the farm just to hit reset.

I tried the stock watchdogs - they either reboot the rig on any hiccup (and wreck your tuning) or stay silent when a card is actually dead. So I sat down and wrote my own bot. What it does:

- tunes overclocks per card, and only accepts a result after a long stability run (2 hours in my setup), not "worked for five minutes";
- tells a one-off glitch from real instability: no pointless reboots, but real lockups aren't ignored either;
- recovers a fully wedged driver by itself within minutes, through an escalation chain;
- keeps a total PSU power budget, so tuning one card doesn't starve the others;
- reports everything to Telegram: what it tuned, what crashed, what it fixed.

Bottom line on my rigs: zero overnight downtime, no manual tuning at all. One weird thing I noticed: on my 3090 one miner kept giving ~70 instead of the usual ~130, until the bot cross-checked the preset against a different miner and rejected it. Anyone seen that with onezerominer?

How do you all handle hangs and tuning? Maybe I reinvented the wheel and there's something decent out of the box?

[BitMaxz]

You are in the Bitcoin mining section, where most of the units here are ASIC for mining BTC. You might get less response here, so I suggest moving this to the altcoin mining section instead, or I'll ask mods to move this thread.

You might be pushing your cards too high, which causes hangs if you are using the HiveOS; it is supposed to be automatically restarted if you enable the watchdog.

If the watchdog didn't work well, you better try another setup than pushing your cards too much.
Have you reviewed the miner logs instead check the reason why it hangs?

Using a bot won't solve anything; you're just pushing your card that is the way to shrink the lifespan of your card without understanding what the cause of your issue is.

If it's just overheating, then try increasing the fan speed and reducing the frequency at least to drop down the maximum temp, or you can set the watchdog; if the temp hits that temp, it should restart. That's my solution before and during summer.

[ocbot]

Thanks for the tips! Looks like the thread landed in the altcoin mining section already - makes sense, my cards are GPU rigs, not ASICs.

On pushing cards too high - that's actually where the bot helps rather than hurts. It doesn't chase max hashrate: when a card crashes during a tuning step, it backs off and locks a safer point, and a preset only counts after a 2-hour stable run. End result is lower clocks than I'd dare to set by hand.

On watchdogs: the stock HiveOS watchdog reboots the whole rig on a single stuck card and loses all per-card state, or doesn't fire at all on a dead-but-spinning card (miner reports 0, fans spin happily). Mine escalates instead: restart the miner, then reset the one card, reboot only as a last resort.

Logs - yes, first thing I did. The bot parses miner logs and driver errors itself; that ~70 vs ~130 on my 3090 came straight from comparing runs across two miners.

Overheating isn't my issue luckily - cards sit around 60-65C. My hangs were driver-level wedges at night.