cgmon - mining monitor for Linux - auto restart, reboot, sick gpu, ASIC, &more

[SlipperySlope]

When cgminer starts, it takes a few seconds to open the API port provided that you have enabled it as the cgmon instructions specify. You will see a detail line in the cgminer terminal window that says that the API port is open. Until that happens cgmon is not able to see cgminer and says so in its own log.

You can trace cgminer using the -T command line option, which logs all messages to the terminal window without overwriting any previous information.

[1713563690]

1713563690

[1713563690]

1713563690

[1713563690]

1713563690

[1713563690]

1713563690

[Coinstron]

Hi! tnx for great script!
but it is very sad to see this on my rig at mornings

Mar 03 20:00:02 bamt-miner - cgmon 1.0.8 - sgminer running and all GPUs healthy.
Mar 03 20:02:01 bamt-miner - cgmon 1.0.8 - sgminer running and all GPUs healthy.
Mar 03 20:04:03 bamt-miner - cgmon 1.0.8 - sgminer running and all GPUs healthy.
Mar 03 20:06:11 bamt-miner - sgminer is not responding.  Rebooting.
Mar 03 20:06:21 bamt-miner -
Mar 03 20:08:44 bamt-miner - sgminer not running, starting via this command:
Mar 03 20:08:44 bamt-miner - /opt/miners/sph-sgminer/sgminer --api-listen   -c /opt/miners/sph-sgminer/sgminer.conf
Mar 03 20:08:45 bamt-miner - sgminer started successfully.  Use 'screen -r' to attach to sgminer and Control-a-d to detach.
Mar 03 20:10:01 bamt-miner - sgminer not running, starting via this command:
Mar 03 20:10:01 bamt-miner - /opt/miners/sph-sgminer/sgminer --api-listen   -c /opt/miners/sph-sgminer/sgminer.conf
Mar 03 20:10:02 bamt-miner - sgminer failed to start.  Try running the mining command above to find the error.  Also, double check your GPU options.
Mar 03 20:12:01 bamt-miner - sgminer not running, starting via this command:
Mar 03 20:12:01 bamt-miner - /opt/miners/sph-sgminer/sgminer --api-listen   -c /opt/miners/sph-sgminer/sgminer.conf
Mar 03 20:12:02 bamt-miner - sgminer failed to start.  Try running the mining command above to find the error.  Also, double check your GPU options.
Mar 03 20:14:01 bamt-miner - sgminer not running, starting via this command:
Mar 03 20:14:01 bamt-miner - /opt/miners/sph-sgminer/sgminer --api-listen   -c /opt/miners/sph-sgminer/sgminer.conf
Mar 03 20:14:02 bamt-miner - sgminer failed to start.  Try running the mining command above to find the error.  Also, double check your GPU options.

sometimes X is not starting after reboot and cgmon restart sgminer in endless cycle. maybe rebooting rig after N failed starts will be good solution?

[kinsik]

When cgminer starts, it takes a few seconds to open the API port provided that you have enabled it as the cgmon instructions specify. You will see a detail line in the cgminer terminal window that says that the API port is open. Until that happens cgmon is not able to see cgminer and says so in its own log.

You can trace cgminer using the -T command line option, which logs all messages to the terminal window without overwriting any previous information.

The -T parameter shows me nothing usefull so far...
What port should be opened and forwarded on router/firewall and should I use in GPU parameters smth like "--api-port XXXX --api-allow W:MY_internal _IP" ?

[virtualreality69]

Jdape,
First off Thanks for donating your time into this project and giving it to the community! I'm sure this will be a great tool for me in the future (once all the parts are in for my 2nd and 3rd rig).
 
Just installed it and its up and running, although I haven't had time to test/play with it yet. At some point I'll be sure to throw a few coins your way.

To Jdape and the community here,
Is there any additional official/unofficial documentation on cgmon? I'm specifically looking to see if I can pass new values to cgminer while its running and how to check cgmon's (or cgminers) current status via ssh. Apologies if this was covered at some point and I missed it.

Cheers,
- VR

Tired of finding miners sick, dead, or crashed for long periods of time?  I was!  Here's cgmon, a lightweight, free, and open source script which performs these functions in Linux:

• Monitors health of all GPU's/ASIC's.  If a GPU crash or problem is detected, reboots the computer and starts mining automatically.
• Auto restart cgminer when it crashes.
• Auto restart of computer if cgminer becomes unresponsive.
• Monitors Accepted Share counts per GPU - restart & notify if a GPU is not outputting accepted shares.
• Monitors network connection and mining pools for outages.
• Email notification of all actions.
• Now has SMTP AUTH support for gmail and other ISPs*
• Automated version checking and upgrading

• Runs via command line in 'screen' so you can access cgminer at anytime.

[/b]

• Runs cgminer in the background

• Can be started/viewed via SSH

[/b]

• Supports unlimited GPU's/ASIC's.
• Supports up to five mining pools.
• Supports sgminer 4.0.0

[haggis]

I use BAMT 1.5 and always use "sync;coldreboot" to reboot the machine. What do I need to change in the script to use it with cgmon, too?

[Gazylion]

Script always instantly wants to shut down my rig after execution...

Code:

root@cryptominer-pl:/home/user/Downloads# ./cgmon.tcl
Mar 07 17:33:45 cryptominer-pl - GPU 0 is sick or dead, rebooting...
 Status:             [Last Not Well] => 1394209759
                                                                               
Broadcast Message from root@cryptominer-pl                                     
        (somewhere) at 17:33 ...                                               
                                                                               
GPU 0 DEAD - Rebooting in 10s...                                               
                                  

Code:

Time         Total    gpu0    gpu1
17:39:35    970.30  485.20  485.10
17:39:42    967.60  484.00  483.60
17:39:42    967.90  484.00  483.90

Not SMOS 1.3 by beebee with coinvert script, mining scrypt with cgminer-karloth.

[jormugandr]

Hey so I'm a total linux noob. I have BAMT 1.3 and I downloaded cgmon.tcl but I can't figure out how to open the file to make the appropriate edits. I use sgminer so I can't run it until I make the edits others previously mentioned in this thread.

How do I open the tcl file lol?!

[virtualreality69]

I think you win the biggest linux noob award.  This shouldnt be posted here as it pertains to linux and not so much cgmon. Look for a good linux turorial out on the interwebs if you have other linux (and not gcmon) related questions


Google how to open a file with nano (or any Linux text editor). Basically just change directory to the location its in and run:
>nano cgmon.tcl

http://askubuntu.com/questions/54221/how-to-edit-files-in-a-terminal-with-nano


 

Hey so I'm a total linux noob. I have BAMT 1.3 and I downloaded cgmon.tcl but I can't figure out how to open the file to make the appropriate edits. I use sgminer so I can't run it until I make the edits others previously mentioned in this thread.

How do I open the tcl file lol?!

[Kael]

Has anyone developed a more detailed instruction or install-script for this? Trying to get it to work on BAMT 1.3 w/kalroth, having a helluva a time. I've searched this forum for answers.

I have the cgmon_daemon stuff working. It launches, waits 30 seconds, and then says "cgminer is not responding, rebooting". I have it set to start after the "mine" starts (which it does), but does not communicate.

I'm using:

--api-allow W:127.0.0.1

On BAMT, what's the best course of action? Disable bamt's autostart and let cgmon do it all?


Problem 2:
If cgmon launches cgminer it says "use screen -r to see output" (or whatever). I see my power usage go up, like the cards are active. But no screen shows for cgminer. Screen -ls shows nothing, sudo screen -ls shows nothing.

[haggis]

Has anyone developed a more detailed instruction or install-script for this? Trying to get it to work on BAMT 1.3 w/kalroth, having a helluva a time. I've searched this forum for answers.

I have the cgmon_daemon stuff working. It launches, waits 30 seconds, and then says "cgminer is not responding, rebooting". I have it set to start after the "mine" starts (which it does), but does not communicate.

I'm using:

--api-allow W:127.0.0.1

On BAMT, what's the best course of action? Disable bamt's autostart and let cgmon do it all?


Problem 2:
If cgmon launches cgminer it says "use screen -r to see output" (or whatever). I see my power usage go up, like the cards are active. But no screen shows for cgminer. Screen -ls shows nothing, sudo screen -ls shows nothing.

I had/have similar problems with BAMT 1.5, too. Sometimes the whole machine hangs up while rebooting, requirering a hard reset. Sometimes it reboots but starts 2 sgminer instances (1 by BAMT and 1 by cgmon). This leads to a blocked API Port because it's already used by the other instance - making the cgmon script believe sgminer is not working. Rebooting again. However, the last reboot worked without problem. But I don't know what causes the game between cgmon and bamt to crash and sometimes not.

Bamts screen session runs as root while cgmon starts its sessions as "user".

[Kael]

I had/have similar problems with BAMT 1.5, too. Sometimes the whole machine hangs up while rebooting, requirering a hard reset. Sometimes it reboots but starts 2 sgminer instances (1 by BAMT and 1 by cgmon). This leads to a blocked API Port because it's already used by the other instance - making the cgmon script believe sgminer is not working. Rebooting again. However, the last reboot worked without problem. But I don't know what causes the game between cgmon and bamt to crash and sometimes not.

Bamts screen session runs as root while cgmon starts its sessions as "user".

I think I may have worked it out. I started completely fresh and went from there - documented all steps from a fresh BAMT install. I'm going to compile it into a how-to and post it here. I also created a installation shell script for bamt users that'll walk you through each step. I'm also working on modifying the code (maybe jdarpe will accept the changes) so that if it detects BAMT=YES, it'll use the mine commands that bamt has preinstalled over trying to execute its own scripts. Should solve the double miner issues.

I'll probably post it tomorrow after I test it on a few more new installs. It's late and I want to go relax.

[haggis]

I had/have similar problems with BAMT 1.5, too. Sometimes the whole machine hangs up while rebooting, requirering a hard reset. Sometimes it reboots but starts 2 sgminer instances (1 by BAMT and 1 by cgmon). This leads to a blocked API Port because it's already used by the other instance - making the cgmon script believe sgminer is not working. Rebooting again. However, the last reboot worked without problem. But I don't know what causes the game between cgmon and bamt to crash and sometimes not.

Bamts screen session runs as root while cgmon starts its sessions as "user".

I think I may have worked it out. I started completely fresh and went from there - documented all steps from a fresh BAMT install. I'm going to compile it into a how-to and post it here. I also created a installation shell script for bamt users that'll walk you through each step. I'm also working on modifying the code (maybe jdarpe will accept the changes) so that if it detects BAMT=YES, it'll use the mine commands that bamt has preinstalled over trying to execute its own scripts. Should solve the double miner issues.

I'll probably post it tomorrow after I test it on a few more new installs. It's late and I want to go relax.

That would be great! Does it also fix the "no shares detected" after doing a "mine restart" manually?

[Kael]

That would be great! Does it also fix the "no shares detected" after doing a "mine restart" manually?

I'll check that. I want to do one last completely fresh run through before uploading it. I'll mark out all changes here.

-------

Going to break this into two posts for ease. This is the setup script.

** THIS IS FOR BAMT USERS .. SMOS/Others use at your own risk, I do not know if it will work for you **

INSTALL CGMON on BAMT
** Tested to work on BAMT 1.3 fresh install, your mileage may vary.
** You can do most of the commands manually, but a simple .sh script has been provided.

- Download the .sh file at bottom of post and place it in /home/user/ directory
or
- Follow step by step manual way below.

SCRIPT WAY:

- After placing it in /home/user/ directory, give execute rights to file.

Code:

chmod +x /home/user/inst_cgmon.sh 

- Then execute it via Terminal

Code:

 bash /home/user/inst_cgmon.sh 

- Follow along the prompts, brief explanation:
-- Get required files, including cgmon.tcl, applies proper permissions
-- Prompts for option to launch NANO edit of cgmon.tcl
-- Offers to create crontab for every 2 minutes (even)

For what edits should be used in cgmon.tcl for BAMT, see next post.

MANUAL WAY:
1. Open a ROOT Terminal and run the following commands (wait for each to finish).

Code:

apt-get update -y
apt-get install tcl -y
apt-get install php5-cli -y
apt-get install php5 -y
cd /home/user
wget http://www.forked.net/~apex/cgmon/cgmon.tcl
chmod +x cgmon.tcl
echo -e "" > /home/user/cgmon.log
chmod 755 /home/user/cgmon.log

2. Create the CRON job
Open a ROOT Terminal and type "crontab -e"
Add the following:

Code:

*/2 * * * *	root	/home/user/cgmon.tcl >/dev/null 2>&1 

Save (write) it.

3. See next post for recommended BAMT edits (how I got it to work) of cgmon.tcl

Can't attach, so download here: http://kaelmedia.com/files/bamt-scripts/inst_cgmon.sh
Open that file in your browser to see the code.

[Kael]

Code Edits to CGMON.TCL for BAMT

* What it does: It creates another config line to bypass the internal launch commands of CGMON.TCL and use the BAMT "Mine Restart" commands. This prevents the dual miner issue.
* I'm sure there was a more elegant way of doing this, but I'm more of an automation scripter and not a programmer.
* May need to reapply some of (or all) these edits if you run an update.
* Not fully tested, I'm running it on a rig that has usual "SICK/DEAD" issues. As soon as that flies, I'll edit this to update. I know it'll fire and reboot OK if it the pools become unreachable (disconnected cable).

Edit CONF Section with this (some may not be needed, but I did them anyways):

Code:

set conf(cgminer_gpu_options)
-- set to: "-c /etc/bamt/cgminer.conf"
set conf(cgminer_extra_options)
-- set to: "--api-listen"
set conf(mining_user)
-- set to: "root"
** It wouldn't run as user for me. Bamt does the mining as root.
set conf(cgminer_path)
-- set to: "/opt/miners/cgminer/"
set conf(miner_requires_scrypt_flag)
-- set to: "no"
** If you're running BAMT 1.3, 1.4, 1.5 as stock with SGMiner.
set conf(cgmon_logfile)
-- set to: "home/user/cgmon.log"
** we're running ROOT, but want the log file in the user directory. If you didn't use the setup inst_cgmon.sh script I provided in the last post, you may need to create this file in /home/user and assign it user: root, group: root.
# set conf(BAMT)
-- set to: remove the "#" and set it to "yes"

Now the actual additions:

Underneath "set conf(BAMT)" add..

Code:

# Use BAMT's internal mine commands?
set conf(use_bamt_mine_cmds) "yes"

With that added, go to line 259, it should say "append mining_command". Change from this:

Code:

append mining_command " $conf(cgminer_gpu_options)"
 		
		notice $mining_command

		# delete count of accepted shares when starting miner - otherwise a reboot could occur from reading irrelavent share counts
		exec rm -f /tmp/accepted_count

		# create bash script in /tmp/ to max use of environment variables for mining
		if {![catch { set outfd [open "/tmp/cgmon-mine.sh" w] }]} {
			puts $outfd "#!/bin/bash"
			puts $outfd "export DISPLAY=:0"
			puts $outfd "export GPU_MAX_ALLOC_PERCENT=100"
			puts $outfd "export GPU_USE_SYNC_OBJECTS=1"
			puts $outfd "$conf(screen_cmd) $mining_command"
			close $outfd
		}
		exec chmod 755 /tmp/cgmon-mine.sh
		exec chown $conf(mining_user) /tmp/cgmon-mine.sh
		set exec_cmd "/tmp/cgmon-mine.sh"

		# run the bash script as the correct user (the one running X hopefully)
		catch {exec whoami} script_user
		if {$script_user == $conf(mining_user)} {
			catch {exec /bin/bash -c $exec_cmd} out
		} elseif {$script_user == "root"} {
			catch {exec su $conf(mining_user) -c /bin/bash -c $exec_cmd} out
		} else {
			notice "attempted to start $conf(cgminer_exec) as the wrong user.  exiting."
			exit
		}
		
		# Check if miner started or not.

to this:

Code:

 		append mining_command " $conf(cgminer_gpu_options)"
 		
		# BAMT Alternate Message
		if {$conf(use_bamt_mine_cmds) == "yes"} {
		notice "Mining started via mine restart command, if errors exist, check bamt.conf or cgminer.conf in /etc/bamt/"
		} else {
		notice $mining_command
		}

		# delete count of accepted shares when starting miner - otherwise a reboot could occur from reading irrelavent share counts
		exec rm -f /tmp/accepted_count

		# create bash script in /tmp/ to max use of environment variables for mining
		if {![catch { set outfd [open "/tmp/cgmon-mine.sh" w] }]} {
			puts $outfd "#!/bin/bash"
			puts $outfd "export DISPLAY=:0"
			puts $outfd "export GPU_MAX_ALLOC_PERCENT=100"
			puts $outfd "export GPU_USE_SYNC_OBJECTS=1"
			puts $outfd "$conf(screen_cmd) $mining_command"
			close $outfd
		}
		exec chmod 755 /tmp/cgmon-mine.sh
		exec chown $conf(mining_user) /tmp/cgmon-mine.sh
		set exec_cmd "/tmp/cgmon-mine.sh"
		
		# run the bash script as the correct user (the one running X hopefully)
		# BAMT Command Mine Start
		if {$conf(use_bamt_mine_cmds) == "yes"} {
			catch {exec su root -c /bin/bash -c "/etc/init.d/mine restart"} out
		} else {
		catch {exec whoami} script_user
		if {$script_user == $conf(mining_user)} {
			catch {exec /bin/bash -c $exec_cmd} out
		} elseif {$script_user == "root"} {
			catch {exec su $conf(mining_user) -c /bin/bash -c $exec_cmd} out
		} else {
			notice "attempted to start $conf(cgminer_exec) as the wrong user.  exiting."
			exit
		}}
		
		# Check if miner started or not.
		

The edits are between line 256 and 287 in an unedited CGMON.TCL 1.0.8. Or between "append mining_command" and "#check if miner started or not".

[haggis]

Thank you a million, Kael! I've followed your instructions and will see if it works

[Kael]

Thank you a million, Kael! I've followed your instructions and will see if it works

Yeah, post back if you have issues. I did notice it reboots fine for no-network, but doesn't reboot if sick/dead. Working on figuring that one out.

[haggis]

Yes, I had only one DEAD case and I don't know what exactly happened. It did not restart correctly. Then I turned the power off completely. Finally after the 5th cold reboot it worked again. Maybe this isn't something that can be fixed by a script...it looked like a hardware issue :/

[Kael]

Yes, I had only one DEAD case and I don't know what exactly happened. It did not restart correctly. Then I turned the power off completely. Finally after the 5th cold reboot it worked again. Maybe this isn't something that can be fixed by a script...it looked like a hardware issue :/

As others have mentioned, the CRON seems to be finicky. If I ran the script through the ROOT's CRONTAB, it'd run but never reboot. If I ran it manually as a sudo, it'd reboot fine. The Daemon posted up above seems to fix this issue.

I updated the install script .. here: http://www.kaelmedia.com/files/bamt-scripts/inst_cgmon.sh

It's now:
1) Install Required Files
2) Enable SMTP Authentication (addition files)
3) Nano edit cgmon.tcl?
4) Choose to install CRON, Daemon, or skip it.

It'll install the Daemon files by ckopobapka - they're already configured for the script. Since you already did it, you can just choose "no" until you get to option 4. Choose the Daemon option. Let it finish, then do a "crontab -e" and erase the cronjob for cgmon, save that file. Do a sync;coldreboot .


Additionally, I did notice that the cgmon.log file doesn't get updated when it reboots because of a SICK/DEAD card (not good for troubleshooting) because the Daemon seems to reboot the system before it can. I'm working on a quick fix for that. Just need one of my cards to go sick real quick (they never do it when I want them to).

[virtualreality69]

Kael,
Thanks for the updates for BAMT! I really appreciate your contributions to the community.  

I'm interested to know if you are aware of how to display cgminer's current status/info when its run with cgmon?  I have verified cgmon is running and I've added it to my crontab.  Cgmon starts cgminer with system boot which is great. The only thing I haven't sorted out yet is how to view the current status/info of cgminer via ssh or local terminal.  I'd really like to be able to check on the cards every day or two just to make sure they're not getting too warm with the upcoming change in seasons (winter to spring).

I dont know if you also had this issue (see below) or if you were quoting someone from earlier. 

Cheers,
- VR


UPDATE: "screen -r" apparently works now. I'm not sure if I was trying to run it before the system fully booted or what the problem was. 

Problem 2:
If cgmon launches cgminer it says "use screen -r to see output" (or whatever). I see my power usage go up, like the cards are active. But no screen shows for cgminer. Screen -ls shows nothing, sudo screen -ls shows nothing.

[rorbit]

Hi all,

I got cgmon working with LeaseRig: http://leaserig.net/index.jsp?rfid=1681

It's easy, just replace the references to the default port 4028 in the cgmon script to the port you registered with LeaseRig.

E.g. port 7778 in the example below.

Use these cgminer parameters to restrict the access to your rig to certain functions and only from the LeaseRig IP:

--api-groups L:switchpool:addpool:removepool:restart:save:*  --api-allow W:127.0.0.1,L:89.212.242.33 --api-port 7778

If you'd like to hire out your rig please use my referral link above.

It's a great service.