Tools for monitoring mining Farms

[Red Emerald]

Ever since cgminer got an rpc interface, I've wanted a reliable way to monitor my miners.  I started off building a quick and dirty python script, but quickly realized that I was reinventing the wheel.  There are already lots of different (open source) tools for monitoring networks.  Most all of them already include email notifications and nifty graphs.

If we build tools that provide an SMNP interface for cgminer, we can use any of these tools pretty easily.  Its also simple enough to just use netcat, but I think a more structured interface would be better.

One of the problems is there are lots of tools.  I've experimented with a few, but haven't really enjoyed any of them. I guess this is why people have been writing custom tools.

Nagios is a pain to configure (although NConf makes it easier) and doesn't have graphs.  I was also getting lots of false positives.

I've just started playing with Zabbix, but its configuration is very complex.  I think I just need to learn it a little better.  Once its configured, it looks really nice.

Does anyone have any suggestions as to what monitoring software is be good to use?  I want to watch CPU, Network, GPUs, etc.  Email notifications for numbers being out of bounds or services going offline are a must.  Prowl/SMS/carrier pigeons aren't necessary, but would be nifty.

EDIT: If you are using a custom script, could you please share your code?

EDIT: Zabbix configuration was frustrating, so I've switched back to Nagios (configured with NConf) and Cacti with NPC (so that theres one web interface to look at).

[Red Emerald]

Here's a script that will work for monitoring an ATI GPU with SNMP and it doesn't care what mining program you are using.
https://bitcointalk.org/index.php?topic=48771.0

Anubis is built specifically for cgminer
https://bitcointalk.org/index.php?topic=57342.0

[jjiimm_64]

I created this one.  it is working well, will be updating it soon.

https://bitcointalk.org/index.php?topic=58834.0

[Internet151]

I'm using anubis to monitor my 24 rigs/96 gpu's.

[The LT]

And I must say that I like your idea of using industry-standard protocols for GPU monitoring. But I believe that it actually has to be a totally separate utility/script, and it should not be integrated with a web-UI or a mining application.

I have a few years of experience with Zabbix, albeit not so in-depth experience...  A I've learned it can be a royal pain to configure, but it really does its job well once you get your templates done. But that said, I think that Zabbix is a bit out-of-scope in this discussion. SNMP on the other hand - is.

Given it a little thought, I'd say that we need to think along the lines of a program which would provide everything a bridge between SNMP and ATI drivers, all described via MIB tables. If you remember, zabbix has an AGENT utility for grabbing system parameters not provided in net-snmp via shell-scripts, that's quite close to what is needed. It can even be made to work in Zabbix but somehow it doesn't look that elegant to me.

Nagios and Cacti are much easier to configure, I'd give them that...

I still believe not only stats are necessary, but basic GPU control as well...

Anyways, I'd rather use Zabbix or Nagios with dirty hacks rather than current alternatives provided here. Those are professional monitoring applications, and while looking a bit overkill for the task, they excel at providing concise statistics over any time-periods, provide reports and analytics. I could calculate the temperature fluctuations between summer and winter seasons, make small voltage tweaks and gather statistics on hashrates and share counts. The only thing we need is a concise SNMP interface to mining software and GPU statistics rolled into one program.

Thoughts? Comments?

[Red Emerald]

I still believe not only stats are necessary, but basic GPU control as well...

I hadn't though about this too much mostly because once I setup my miner, I didn't need to touch it besides to change what pool I used.

The more I look at it, the more I like SNMP for this.  It's built for exactly what we are doing.

SNMP exposes management data in the form of variables on the managed systems, which describe the system configuration. These variables can then be queried (and sometimes set) by managing applications.

Once we have an easy way to get cgminer using SNMP, then anyone can easily use enterprise-level or custom tools to monitor and configure their miners remotely.

I really didn't want to have to install anything extra on the miner to get this working.

Maybe we should start another bounty for a cgminer SNMP interface instead of the current RPC

[The LT]

I still believe not only stats are necessary, but basic GPU control as well...

I hadn't though about this too much mostly because once I setup my miner, I didn't need to touch it besides to change what pool I used.

The more I look at it, the more I like SNMP for this.  It's built for exactly what we are doing.

SNMP exposes management data in the form of variables on the managed systems, which describe the system configuration. These variables can then be queried (and sometimes set) by managing applications.

Once we have an easy way to get cgminer using SNMP, then anyone can easily use enterprise-level or custom tools to monitor and configure their miners remotely.

I really didn't want to have to install anything extra on the miner to get this working.

Maybe we should start another bounty for a cgminer SNMP interface instead of the current RPC

RPC is fine, as long as it supprts enough parameters and statistics. It can then be wrapped to SNMP in a pretty straightforward way.

[kjj]

I marked "Other" in the poll.

I still use cdhowie's mining proxy, with a bunch of modifications, for central management, and linuxcoin with custom scripts for on-box management.

My local scripts are smart enough to restart stuck miners, and my central tool is capable of doing remote reboots on boxes that are totally hung.