Daemons monitoring

[mr.tall]

Hello

I have few different daemons who like to die time to time, and i think about some monitoring for all daemons.
What is the best method to montor? Check some port, or command "getinfo" etc?
Thank you in advice!

[bob123]

What kind of monitoring are you thinking about?

You can either connect via an RPC port and code a small script which polls the information you want to check.
This includes network information (e.g. connected peers, new blocks, transactions, ...).

Or you can monitor the output of the log file.
Either check the debug.log from bitcoin core (standard directory: ~/.bitcoin) or the system log (/var/log/syslog).

This mainly depends on what exactly you want to monitor.

[mr.tall]

What kind of monitoring are you thinking about?
 ...
This mainly depends on what exactly you want to monitor.

Thank you, Bob!
My idea is checking that daemon actually alive, and working stable: synchronise with last block, respond for RPC commands and etc.

[Coin-1]

What kind of monitoring are you thinking about?
 ...
This mainly depends on what exactly you want to monitor.

Thank you, Bob!
My idea is checking that daemon actually alive, and working stable: synchronise with last block, respond for RPC commands and etc.

For example, you can use the Nagios system. It supports the plugin for bitcoind monitoring:
https://github.com/mattdy/nagios-bitcoin

It is declared that this plugin requires Nagios, jq for parsing JSON, bitcoind with RPC enabled.

[pebwindkraft]

...
What is the best method to monitor?

Not sure what might be "the best" method, as no criteria is given...
I have a running unixoide script using "bitcoin-cli getblockchaininfo", and redirect it e.g. into /tmp/getblockchaininfo.tmp
It is running a 0.14 and 0.15 daemon, not yet on the latest 0.16 bitcoin daemon.
Then I do:
 - grep "main", to be sure we are on live blockchain
 - grep "blocks", the current block on my node
 - grep "headers", the number of blocks known by the bitcoin world
 - "verificationprogress", a number >= 0.99 when in sync,
    otherwise this number indicates "trying to load old blocks to get in sync"

I have some more scripts to put this all into a monitoring solution. It is run by a crontab daemon every 5min. There is one thing to pay attention to: if running on a slow (weak CPU) system, the RPC call can take more than 5min to come back, while the chain is not sync'd. So if you crontab the process, need to check first, if the process is still running, before calling again.

Otherwise the NAGIOS solution proposed by Coin-1 is a good approach as well.

[bob123]

My idea is checking that daemon actually alive, and working stable: synchronise with last block, respond for RPC commands and etc.

To check whether your daemon is alive, does receive blocks, verifies them, etc... just look at the debug.log (standard directory on linux: ~/.bitcoin).
You will see relevant information regarding block count, transaction, .. , pebwindkraft's solution should be working for you.

If you want to check if RPC commands are working correctly (which they always should if your daemon is not corrupted), the only possibility i might think of is to code
a small script which does call the RPC commands regularly.

[psycodad]

To check for coinds being alive it is most of the time sufficient to query rpc with almost any command like getbalance or getinfo.

If on a unix-like system like linux, you can run a cronjob that queries getinfo and if timed out, restarts the daemon i.e.:

Code:

#!/bin/bash

timeout 10 somecoin-cli getinfo
if [[ $? > 0 ]]; then
  echo "somecoind is hanging - restarting somecoind"
  kill -9 $(ps auxww|grep somecoind|grep -v grep|awk '{print $2}')
  sleep 5
  /usr/local/bin/somecoind >/dev/null 2>&1
fi

I am using a way more elaborate script that reads coinds to check from a text file.

HTH

[MrCrank]

To check for coinds being alive it is most of the time sufficient to query rpc with almost any command like getbalance or getinfo.

If on a unix-like system like linux, you can run a cronjob that queries getinfo and if timed out, restarts the daemon i.e.:

Code:

#!/bin/bash

timeout 10 somecoin-cli getinfo
if [[ $? > 0 ]]; then
  echo "somecoind is hanging - restarting somecoind"
  kill -9 $(ps auxww|grep somecoind|grep -v grep|awk '{print $2}')
  sleep 5
  /usr/local/bin/somecoind >/dev/null 2>&1
fi

I am using a way more elaborate script that reads coinds to check from a text file.

HTH

So, nice idea.
I have same problem too and want to try this script.
I think this resolve this issue.
Thanks.