cgmon - mining monitor for Linux - auto restart, reboot, sick gpu, ASIC, &more

[dr00g]

Right, here is a little "bugfix":

The variable is not initialized:
# uncomment for sgminer
#set use_sgminer "yes"

Which leads to:
can't read "use_sgminer": no such variable
    while executing
"if {$mine_for == "litecoin" && $use_sgminer == "no"} {set cgminer_option1 "--scrypt" } else { set cgminer_option1 ""}"
    (file "./cgmon.tcl" line 287)

It is fixed by:
The variable is not initialized:
# uncomment for sgminer
set use_sgminer "no"
#set use_sgminer "yes"

If you uncomment the one below, the variable will be changed. In this, you can control sgminer with just commenting/uncomment the variable below.

Apart from that, the script runs for me. Did you change anything else?


Also, if you will in:
set cgminer_path ""
Then you only have to set the directory, like /home/user/sgminer/, where the sgminer file is located inside. You don't have to fill out the directory path, if you have installed cgminer, like I have by compiling and installing with "sudo make install".

[amptivate]

My cron log is not generated. I changed the directory for the log to /home/user/cgmon.log yet it is never generated, even when I execute the cron command manually.

I am trying to get this working with BAMT 1.3... I am getting random device being hung, and my rig is unable to reboot itself apprently.

Here is my cron.log from /var/log/ when I run the cron command directly through console

Jan 30 15:25:01 (none) /USR/SBIN/CRON[5494]: (munin) CMD (if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi)
Jan 30 15:25:01 (none) /USR/SBIN/CRON[5495]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugi$
Jan 30 15:25:01 (none) /USR/SBIN/CRON[5496]: (root) CMD (/opt/bamt/mother)
Jan 30 15:25:04 (none) /USR/SBIN/CRON[5492]: (CRON) error (grandchild #5494 failed with exit status 1)

[dr00g]

Try changing values to:
set cgminer_exec "cgminer"
set cgminer_path "/opt/miners/cgminer/"

Did you remember to change username in:
set mining_user "user"

[amptivate]

Try changing values to:
set cgminer_exec "cgminer"
set cgminer_path "/opt/miners/cgminer/"

Did you remember to change username in:
set mining_user "user"

Yes, the username is the default "user" - as is the one for BAMT...unless I need to change it to root?


The values are set to as you suggested above. I chmod it again, still nothing


Edit: this also seems to showup in syslog - Jan 30 15:35:05 (none) /USR/SBIN/CRON[6088]: (CRON) error (grandchild #6090 failed with exit status 1)

when i try to run the cron manually

[dr00g]

Does it come with any output when you run it?
Is it running in screen? Type: screen -r
To deattach, press Ctrl + A + D

Did you remember to install the required packages?
# 1) Install PHP, TCL and screen. 
# CentOS: yum install php53 tcl screen
# Ubuntu: apt-get install php5 tcl screen

[amptivate]

Does it come with any output when you run it?
Is it running in screen? Type: screen -r
To deattach, press Ctrl + A + D

Did you remember to install the required packages?
# 1) Install PHP, TCL and screen. 
# CentOS: yum install php53 tcl screen
# Ubuntu: apt-get install php5 tcl screen

I type "/home/user/cgmon.tcl >/dev/null 2>&1" and i get no output, just a new line...

php5, tcl, and screen are all installed and updated.

[dr00g]

Oh, ">/dev/null 2>&1" is exactly for not making any output, I guess it should be removed, both from cronjob line to actually get logoutput...
When you run it manually, type:
sudo ./cgmon.tcl
That should trigger it with some output.

[amptivate]

Oh, ">/dev/null 2>&1" is exactly for not making any output, I guess it should be removed, both from cronjob line to actually get logoutput...
When you run it manually, type:
sudo ./cgmon.tcl
That should trigger it with some output.

Thanks dr00g! Got it working! PHP wasnt in the correct directory... once the log started showing I was able to troubleshoot.

I have a question though. The reason I was looking for a monitoring solution in the first place is because my cgminer randomly freezes and stops responding. I am able to SSH through my phone or computer and coldreboot the rig. Will this script do the equivalent, or does it ONLY restart/reboot when cgminer quits completely?

[dr00g]

I had that problem aswell, in my case it turned out my GPU apparantly didn't like my thread-concurrency...

Anyway, to solve it I found a script which I found here a while ago: http://forum.feathercoin.com/index.php?topic=5989.0

Add this to a file, like /home/username/amd_crash_monitor.sh

#!/bin/bash
if ( dmesg | grep "ASIC hang happened" ); then
  echo "Catalyst has crashed! Rebooting..."
  /sbin/shutdown -r now
fi

Add it to crontab the same way as cgmon, works same way with the extra flag if you want to see output or not.

After the driver has crashed it could take a few minutes before it appears in the log, and the rig reboots.

Remember that you likely should change the reboot line to work with BAMT.

[funkycold]

Try changing values to:
set cgminer_exec "cgminer"
set cgminer_path "/opt/miners/cgminer/"

Did you remember to change username in:
set mining_user "user"

Thx dr00g!

I already had figured out the set_sgminer = "no" part, the issue was with the path.
I got everything working now.

For those who may struggle, here is what to put since the new 0.1b6 has two fields (path and exec) which kind of threw me off a little.

set cgminer_exec "cgminer"
set cgminer_path "/your/path/to/cgminer/"


About the GPU disabled. Yes i waited 3-4minutes and it did detect my DEAD/SICK GPU...one little problem, it rebooted my computer instead of cgminer...is this normal?

Here is the output message i got...

"GPU 0 no accepted shares in 458 seconds. GPU probably hung.
cgminer not running, starting....."

and it reboot my computer...Is this normal?

A small suggestion, maybe you should only reboot that particular GPU, I was looking at API commands for cgminer, The command for restarting a GPU is gpurestart|N where N is the GPU number. So gpurestart|0 would restart GPU0.

Just a suggestion, since GPU restarts are much quicker than cgminer restarts.

Thx and the only thing missing now would be:

1) Set Hash rate, if falls below reboot cgminer

Thx Guys!!

[MrBobC]

How can I get the e-mail notifications to work?  I tried using smtp.gmail.com:25 but that doesn't send anything.  Does anyone know of an smtp setting I could use so the e-mail gets sent?  Thanks!

[dr00g]

You are welcome, promise you will teach other people mining too!

It depends on what is the problem, sometimes it can require a reboot as it is the actual driver that has crashed.

What do you need that for, experiencing that hashrate drops and a cgminer restart to fix it?

[dr00g]

Unfortunately you can't use gmail with this script: https://bitcointalk.org/index.php?topic=353436.msg4759718#msg4759718

[jdape]

2) I;m also having another problem where even if i disable all my GPU's manually from CGminer i still get all "GPU's running healthy" message from cgmon.

Am I doing something wrong? shouldnt it detect DEAD GPU's and restart cgminer?

Thx
 

If your GPUs are disabled, cgmon won't 'see' them and so it will assume everything is running smoothly.   If one of the GPU's is marked as sick, dead, or stops outputting accepted shares, cgmon will log it, notify you, and then reboot the server.

-j

[jdape]

Right, here is a little "bugfix":

The variable is not initialized:
# uncomment for sgminer
#set use_sgminer "yes"

Which leads to:
can't read "use_sgminer": no such variable
    while executing
"if {$mine_for == "litecoin" && $use_sgminer == "no"} {set cgminer_option1 "--scrypt" } else { set cgminer_option1 ""}"
    (file "./cgmon.tcl" line 287)

It is fixed by:
The variable is not initialized:
# uncomment for sgminer
set use_sgminer "no"
#set use_sgminer "yes"

If you uncomment the one below, the variable will be changed. In this, you can control sgminer with just commenting/uncomment the variable below.

Apart from that, the script runs for me. Did you change anything else?


Also, if you will in:
set cgminer_path ""
Then you only have to set the directory, like /home/user/sgminer/, where the sgminer file is located inside. You don't have to fill out the directory path, if you have installed cgminer, like I have by compiling and installing with "sudo make install".

Thanks for this.  I've fixed the uninitialized variable error in the latest version on the website.

[jdape]

Thx dr00g!

I already had figured out the set_sgminer = "no" part, the issue was with the path.
I got everything working now.

For those who may struggle, here is what to put since the new 0.1b6 has two fields (path and exec) which kind of threw me off a little.

set cgminer_exec "cgminer"
set cgminer_path "/your/path/to/cgminer/"

Thank you.  I've now added this explanation for cgminer_path:

Code:

# this is required only if your cgminer_exec binary is not installed in your mining_user PATH (like /usr/bin/ or whatever)
# example:  /home/user/cgminer-3.1.1/       (must end with a '/')
set cgminer_path ""

About the GPU disabled. Yes i waited 3-4minutes and it did detect my DEAD/SICK GPU...one little problem, it rebooted my computer instead of cgminer...is this normal?

Here is the output message i got...

"GPU 0 no accepted shares in 458 seconds. GPU probably hung.
cgminer not running, starting....."

and it reboot my computer...Is this normal?

A small suggestion, maybe you should only reboot that particular GPU, I was looking at API commands for cgminer, The command for restarting a GPU is gpurestart|N where N is the GPU number. So gpurestart|0 would restart GPU0.

Just a suggestion, since GPU restarts are much quicker than cgminer restarts.

For me at least, when a GPU goes SICK or DEAD a full reboot of the computer is always required to get the GPU working again.   This has been the case with all my rigs and GPUs.  It could still be just me though since my rigs are running mostly identical software...  Anyhow, that's why it restart the entire computer.

[jdape]

I have a question though. The reason I was looking for a monitoring solution in the first place is because my cgminer randomly freezes and stops responding. I am able to SSH through my phone or computer and coldreboot the rig. Will this script do the equivalent, or does it ONLY restart/reboot when cgminer quits completely?

Typically cgminer will still be running when a situation requiring a reboot occurs...  So no, if a GPU crashes, it will reboot whether cgminer is running or not.

If cgminer itself crashes and stops responding -- cgmon is not setup to deal with that situation.   Let me know if that's actually what's going on and if so I can probably add a feature to reboot in that case.

-j

[jdape]

# 0.1b7
#   Added detection of cgminer/AMD crash aka 'asic hang' (thanks dr00g!)
#   Fixed rebooting on BAMT.
#   Default share timer changed from 5 to 10 minutes.
#   Accepted share counts and rate added to logfile.  Example below.

Code:

Jan 30 17:55:02 miner8 GPU 0 Shares accepted since last run:  16  (1.07 shares/min)
Jan 30 17:55:02 miner8 GPU 1 Shares accepted since last run:  29  (1.93 shares/min)
Jan 30 17:55:02 miner8 GPU 2 Shares accepted since last run:  1  (0.07 shares/min)
Jan 30 17:55:02 miner8 GPU 3 Shares accepted since last run:  17  (1.13 shares/min)

[funkycold]

For me at least, when a GPU goes SICK or DEAD a full reboot of the computer is always required to get the GPU working again.   This has been the case with all my rigs and GPUs.  It could still be just me though since my rigs are running mostly identical software...  Anyhow, that's why it restart the entire computer.

its fine, rebooting the computer isnt a bad idea for dead/sick GPU's considering ubuntu is really quick to reboot unlike Windows....Linux is the only way to go for continuous mining IMO....thx for everything jdape! It is getting closer to CGwatcher....

[funkycold]

Hey, I found this post a while ago, and as I am using sgminer and also wanted some easier customization, I have reworked the script quite a lot.

Still needs a little more work though, then I will be sharing it here.

I highly appreciate your work!

Sgminer is the only way to go for newer GPU's (R9 290, R9 290X, R9 280X, etc...)
it is very stable and has lots of bugs fixed....just my opinion