How often do you check your farm, steps to gain some peace of mind.

[mikeywith]

Sometimes I think I check my gears and hashrate way too often, I think I log in to the farm PC at least once or twice a day, but I look at the pool status no less than 10 times a day to make sure everything is working fine despite having notifications all over the place, I can't help but look to make sure everything is working fine.

How often do you check on those boys? what do you think is normal and at what stage would you recommend someone to go see a therapist?

Also what measures do you take to reduce the need for you to visit your pool app or webpage?

Things that I do (which obviously aren't helping pretty much with the constant paranoia)

1- a local monitoring software that monitors hashrate and temp with Email notification to an email that's open on my mobile phone, I get emails for over-temp, gear off-line, and lower-than-expected hashrate.

2- a temperature and humidity sensor with an app notification for temp/humidity.

3- phase monitoring relay which shuts down the whole farm when there is under/over voltage or phase failure.

4- Motion detection on the CCTVs to spot any intrusion.

5- Some Fireball fire extinguishers placed at the farm (want to get more of them soon since summer is approaching).

6- Oversized all the wirings and electricity infrastructure and not going above 80% of anything.

7- A heat gun to monitor all wires, sockets, MCBs, Busbars, etc when I go to the farm, just to make sure nothing is getting hot.

8- weekly check-ups on the transformer gauges to make sure the oil level is in place.

9-I shut down everything during rainy storms and abnormal heat waves.

10- periodical dusting and cleaning (depends on the weather).

These are pretty much all the things I do to have some peace of mind and to ensure that everything runs smoothly for as good as possible, looking forward to hearing yours.

[dansus021]

Right now I don't have rig but back a year ago I have the same feel as you do, although is only a small miner and I run it in my other room not a "Farm" and Im using Miningrigrental to see whether my miner connect and running well because when sometimes goes wrong they will email me like drop of hahsrate too.

I think Bitcoin pool nowadays can give an alert when something goes wrong, Just make sure the cable is nice and tidy

[kano]

6 minutes

Firstly, as I explain with statistics, on Help->Luck on my web site, if you check every minute you'll get false positives.
If you check every 2 or more minutes, you'll get false positives about once every 450 years if you have 1000000 miners
If you have fewer than 1000000 miners and checks longer than every 2 minutes you'll have even less false positives.

However, the 6 minutes is a script I have permanently running that emails me and the datacentre, when a miner hasn't been mining for 5 minutes.
If you have miners that fail a lot, that can be problematic, but mine rarely fail so it doesn't matter.
The 6 minute check is pretty good also to allow miners to reconnect if there's a temporary outage/glitch/reboot and not complain about them.

The thing to most take note of is that noone in a datacentre is ever going to run to your miner and coddle it back to life as soon as it fails.
How long they take to do that would help deciding how often to check them.
Obviously it doesn't hurt to have something check every 6 minutes, so that it lets you and them know pretty quickly.

[KiaKia]

You will be spending too much to keep your mining farm on watch, I don't need a CCTV,  why would you? Unless you are living in a criminal world where people can burst into your place and steal your mining rigs, yes fire extinguisher always comes in handy, I almost burn down my home when I use a cheap socket extension for my mining rig, find a cooler spot or garage and use a monitoring app, Nicehash, and HiveOS have this, almost all other mining OS have monitoring features so it's nothing new. This is all you need, and do not travel to the next city when you plan to keep mining around the clock, Fire outbreak can happen at any time and even if you are monitoring and you are not very close, you will lose a lot before you get back to your farm, don't travel and mine Bitcoin.

[mikeywith]

I think Bitcoin pool nowadays can give an alert when something goes wrong, Just make sure the cable is nice and tidy

Most do, but I don't think it's a great idea to count on them 100%, better have other measures that go along side the pool notification.

Indeed, in fact, most pools only have 10 mins hashrate report and above, the ones that report "current" hashrate are just way in accurate and will give you many false positives.

You will be spending too much to keep your mining farm on watch, I don't need a CCTV,  why would you? Unless you are living in a criminal world where people can burst into your place and steal your mining rigs

Seems we are talking about two different things here, you are talking about home mining where you have a few rigs in the garage vs a mining farm which in 99.9% of the cases won't be anywhere near your garage, I have yet to see any mining farm or a datacenter /host that are not equipped with CCTVs, even the cheapest mining contains that ship from China come with CCTVs by default, it's not even an adds-on or anything like that, it comes with the walls, I am willing to as far as betting that at least 95% of mining farms have CCTVs because 95% of the mining farms are located in places where the people who own them don't sleep next door to them.

and do not travel to the next city when you plan to keep mining around the clock

Again confirms why we aren't talking about the same thing, but I agree if the setup is in your home and you are not following the code, it is probably very risky to leave your gears unwatched, in fact, it's risky to run them in the first place, but still, there is a huge difference between running a few gears at home vs a large farm in another city/country.

[philipma1957]

I think Bitcoin pool nowadays can give an alert when something goes wrong, Just make sure the cable is nice and tidy

Most do, but I don't think it's a great idea to count on them 100%, better have other measures that go along side the pool notification.

Indeed, in fact, most pools only have 10 mins hashrate report and above, the ones that report "current" hashrate are just way in accurate and will give you many false positives.

You will be spending too much to keep your mining farm on watch, I don't need a CCTV,  why would you? Unless you are living in a criminal world where people can burst into your place and steal your mining rigs

Seems we are talking about two different things here, you are talking about home mining where you have a few rigs in the garage vs a mining farm which in 99.9% of the cases won't be anywhere near your garage, I have yet to see any mining farm or a datacenter /host that are not equipped with CCTVs, even the cheapest mining contains that ship from China come with CCTVs by default, it's not even an adds-on or anything like that, it comes with the walls, I am willing to as far as betting that at least 95% of mining farms have CCTVs because 95% of the mining farms are located in places where the people who own them don't sleep next door to them.

and do not travel to the next city when you plan to keep mining around the clock

Again confirms why we aren't talking about the same thing, but I agree if the setup is in your home and you are not following the code, it is probably very risky to leave your gears unwatched, in fact, it's risky to run them in the first place, but still, there is a huge difference between running a few gears at home vs a large farm in another city/country.

We have a small commercial setup.  But we pull 160kwatts and have 110 pieces of gear.

I have pc access to every unit. The warehouse has cameras . I can sometimes go 30 days without being at the warehouse. But I look every day at my pc's to see status of gear.

[kano]

...

Indeed, in fact, most pools only have 10 mins hashrate report and above, the ones that report "current" hashrate are just way in accurate and will give you many false positives.
...

Yeah I don't use hash rate to decide - I use that as a warning. not as an alert.
and aside, hash rate is just a calculation from shares.

Time of/since last share is accurate to milliseconds on my pool, so if it says 300 seconds (5mins) since your last share to the pool, it has indeed been 300 seconds +/- a lot less than a second.

For alerts, I'm looking for miners that stop mining and needing a restart,
whereas if a board is failing, it'll start showing regular warnings of a lower hash rate (I think there's only been one that's done that so far)

[mikeywith]

I have pc access to every unit. The warehouse has cameras . I can sometimes go 30 days without being at the warehouse. But I look every day at my pc's to see status of gear.

Ya having access to a PC that runs on the same LAN where miners are located is a must (unless you have your gears at a hosting company where they don't give you that access).

whereas if a board is failing, it'll start showing regular warnings of a lower hash rate (I think there's only been one that's done that so far)

Pretty much depends on what type of miner you use, with Whatsminer it's pretty rare to have permanent hashboard damage for new gears, the software isn't perfect in the tuning aspect tho, so it could tune with low hashrate and thus needs a reboot to fix it, so using a local-management software, in that case, is usually a must, with Antminer it's a bit different, it's common for a hashboard to lose hashrate and requires reboots, but it tunes a lot faster and gets to full speed in no time compared to Whatsminers.

By the way, since you already have a script in place, is it also available for your pool users or just a personal thing? I think getting email notifications for lower than average hashrate for all pool users (if they want to set up) would be a good addition to any mining pool.

[paid2]

7- A heat gun to monitor all wires, sockets, MCBs, Busbars, etc when I go to the farm, just to make sure nothing is getting hot.
These are pretty much all the things I do to have some peace of mind and to ensure that everything runs smoothly for as good as possible, looking forward to hearing yours.

This is a tip that I agree with 100%, it has given me so much peace and tranquility. It's well worth it!

In our installation we also have CCTV and alarms, I can't imagine having so much investment in hardware and so much money at stake without a minimum of protection.
For one of our places, we also have private security patrols but we pay the costs in a shared way with the neighbors, it costs a lot . But this one is in an area where it is justified and necessary.

Otherwise I only look at the hashrate a little, I have alerts when a worker is offline for more than 10 minutes and that is enough for me. I can spend sometimes 2 weeks without going on site because we can do checks remotely. I monitor once a day if everything is fine, and then I rely on the alerts.
I used Antminer-Monitor for a long time (https://github.com/anselal/antminer-monitor) but we have since switched to something cleaner.

Where I feel like I look too much is my bestshares on the rigs that are mining solo, even though I know it's useless, I look at my bestshares at least 5 times a day 

[mikeywith]

This is a tip that I agree with 100%, it has given me so much peace and tranquility. It's well worth it!

I learned it the hard way to be honest, sometimes regardless of how you oversize your cables and tighten them, one or some of them might start to become a bit loose and leads to overheating and eventually even fire, so checking that nothing is warmer than usual gives me a truckload of piece of mind.

For one of our places, we also have private security patrols but we pay the costs in a shared way with the neighbors, it costs a lot . But this one is in an area where it is justified and necessary.

Ya, I would imagine something like that should cost a lot regardless of which country that setup might be.

I used Antminer-Monitor for a long time (https://github.com/anselal/antminer-monitor) but we have since switched to something cleaner.

I use AwesomeMiner, it's owned by a fellow bitcointalk member.

[kano]

...
By the way, since you already have a script in place, is it also available for your pool users or just a personal thing? I think getting email notifications for lower than average hashrate for all pool users (if they want to set up) would be a good addition to any mining pool.

Nah it's not a simple issue to be sending emails to lots of people regularly - your mail servers get flagged as spammers very fast.

It is 'almost' at the point where if someone with a sizeable hash rate wanted me to trial it with them I'd consider it.
But haven't done much coding for a few months for known reasons.

[mikeywith]

Nah it's not a simple issue to be sending emails to lots of people regularly - your mail servers get flagged as spammers very fast.

You can let them use their own mail servers, anyone with a Gmail account can get mail server info, this is how some software work.

However, almost all pools i used still send out notfications via emai from their own mail server, not sure how they avoid getting flagged, I assume they use a dozen different mail servers.

[Sledge0001]

Nah it's not a simple issue to be sending emails to lots of people regularly - your mail servers get flagged as spammers very fast.

You can let them use their own mail servers, anyone with a Gmail account can get mail server info, this is how some software work.

However, almost all pools i used still send out notfications via emai from their own mail server, not sure how they avoid getting flagged, I assume they use a dozen different mail servers.

They aren't getting flagged because they are fully "Can Spam Act of 2003" compliant https://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003 and most enterprise level organizations have configured DKIM, DMARC and SPF records to match and confirm the identity of their email server(s) and ip addresses.  You can test your email server(s) configuration and conformity here: https://easydmarc.com/tools/ .
 


I check in at least a few times a day to see how the "kids" are doing.

Email alerts have also been configured so that if a unit goes offline or isn't hashing up to a specific rate then the miner is sent an automated reboot.

[mikeywith]

They aren't getting flagged because they are fully "Can Spam Act of 2003" compliant https://en.wikipedia.org/wiki/CAN-SPAM_Act_of_2003 and most enterprise level organizations have configured DKIM, DMARC and SPF records to match and confirm the identity of their email server(s) and ip addresses.  You can test your email server(s) configuration and conformity here: https://easydmarc.com/tools/ .
 

I still doubt it's something easy to do, given that some multi-billion dollar companies like Hikvision with 12B in revenue last year alone still don't use their own mail server for all types of alerts coming from all the different DVRs, NVRs, and Cameras they sell, you still need to set up your own mail server to get notifications.

I check in at least a few times a day to see how the "kids" are doing.

Lol, that's what my wife calls them whenever I check and she sees me busy looking at my phone/laptop.

[DaveF]

Since my miners are spread far and wide I just rely on the pools to email me when one drops off.
Probably not the best way, by all my miners are old and paid for and the power is a fixed cost to a certain point so it's all profit.
Back in the day it was different but now, it's more of a good enough so long as most are OK theory. If one of my S9 is down for a full day it's only going to be $1 in lost profit. The L3+ units only generate about $1.25 a day.

Not worth stressing about.

-Dave

[philipma1957]

Since my miners are spread far and wide I just rely on the pools to email me when one drops off.
Probably not the best way, by all my miners are old and paid for and the power is a fixed cost to a certain point so it's all profit.
Back in the day it was different but now, it's more of a good enough so long as most are OK theory. If one of my S9 is down for a full day it's only going to be $1 in lost profit. The L3+ units only generate about $1.25 a day.

Not worth stressing about.

-Dave

Yeah I would prefer to have five or ten miners a dirt cheap power and left them run til they die.

I have more gear than I want to work with.

A long ride to check on the gear when it goes down.

It was manageable until Covid hit..

Wife has some permanent lung damage we still do not know if it will slowly get worse or become stable.
My diabetes is just a bit out of control.

And my bro-in-law is suffering from dementia. Lives  💯 maybe 125 kilometers 75 miles.

So those three real world things are very likely to force me out of active mining ⛏️


It would be nice to be 33 and just have a single s19.

[DaveF]

Since my miners are spread far and wide I just rely on the pools to email me when one drops off.
Probably not the best way, by all my miners are old and paid for and the power is a fixed cost to a certain point so it's all profit.
Back in the day it was different but now, it's more of a good enough so long as most are OK theory. If one of my S9 is down for a full day it's only going to be $1 in lost profit. The L3+ units only generate about $1.25 a day.

Not worth stressing about.

-Dave

Yeah I would prefer to have five or ten miners a dirt cheap power and left them run til they die.

I have more gear than I want to work with.

A long ride to check on the gear when it goes down.

It was manageable until Covid hit..

Wife has some permanent lung damage we still do not know if it will slowly get worse or become stable.
My diabetes is just a bit out of control.

And my bro-in-law is suffering from dementia. Lives  💯 maybe 125 kilometers 75 miles.

So those three real world things are very likely to force me out of active mining ⛏️


It would be nice to be 33 and just have a single s19.

Ouch, that is a lot of stuff to have going on. I would pass on the s19 and just take being 33 again. Hurt my knee a couple of weeks ago and I'm still limping....

I know this might be blasphemy but can you cut down on the hardware and shrink the farm(s) a bit so it's less stress / work? No point in killing yourself for it, if you can't enjoy it.

If you don't need the money so long as you are not loosing money less stress is good.

-Dave

[mikeywith]

I know this might be blasphemy but can you cut down on the hardware and shrink the farm(s) a bit so it's less stress / work? No point in killing yourself for it, if you can't enjoy it.

I know Phill loves mining, so why not do the opposite of that? bring in more gear, and make it worth it to hire someone to do the periodical check-ups, it doesn't have to be a salary, it could be paid per visit, or a percentage of income, I don't think it's impossible to find someone who is young, willing to learn a thing or two about maintaining a mining farm, lives nearby, and can be trusted.

It takes on average no more than half an hour a day, I'd say 3 hours a week is the average, a farm with 5ph makes roughly $350 a day, so even if that person asked for 100$/hr, it's just a matter of 3 something days worth of mining, I don't know about wages in the state where Phil's farm is located, but I highly doubt 100$, it will be on average 1 drive per week, any time of the day, 3 hours, reboot a device or two, change 2 fans, dust off a PSU and go home.

Ya, the same guy needs to ship a miner to another state for warranty repair and call the electric guy to come to fix something. Still, the work they will do doesn't require any special skills, so if I had to look for a guy like that,  trust and desire for work would be the only things I would look for.

[kano]

Nah it's not a simple issue to be sending emails to lots of people regularly - your mail servers get flagged as spammers very fast.

You can let them use their own mail servers, anyone with a Gmail account can get mail server info, this is how some software work.

However, almost all pools i used still send out notfications via emai from their own mail server, not sure how they avoid getting flagged, I assume they use a dozen different mail servers.

I don't send spam - it's simple.

However, as soon as someone has the option to get a lot of email ... 'notifications' ... there will be some who will purposely flag those messages ... coz the internet has plenty of assholes who like doing this