Linux: running headless and automation

[mrenouf]

Hello.

I'm trying to determine the best way to run Phoenix from the console. It's unfortunate that X needs to be running in order to talk to the drivers and run OpenCL kernels. I figured just starting X on it's own ("sudo X -ac -noreset -:0" will do it!) and "export DISPLAY=:0" ... but phoenix doesn't seem to want to work properly in that situation...

What I want is a bulletproof system which starts up fast and predictably with the least amout of stuff running. In case of freezes or power loss I want to be able to remotely power cycle the machine and know it will come back online and start mining automatically.

Also... has anyone developed monitoring scripts to watch hashrate and automatically kill/restart or reboot in case of issues? I had a problem with phoenix the other day where one process just kept getting: "[10/06/2011 16:24:46] Warning: work queue empty, miner is idle" with a few rejected shares now and then. A simple restart of the client fixed it. I know this code is python so I'm considering adding some stuff to it to provide stats to monitoring systems and provide a mechanism for automation.

[supa]

I use an init.d script with -

xinit -e metacity
export DISPLAY=0:0
sleep 3
xterm &
xterm &
AMDOverDriveCtrl &
aticonfig --odsc 1005,225
aticonfig --odcc
xterm & -e bash --rcfile startpoclbm.sh

.... I like metacity.  Feel free to replace with whatever you want.

I could also probably remove the AMDOverDriveCtrl + aticonfig x 2 line and just use an AMDOverDriveCtrl profile...

But I didn't.


As far as a "watchdog" feature built into poclbm - it's in Python.  Build your own.  Something even as simple as echoing out the current time to a file, then have a second script kill/reset poclbm/pheonix if the time is > 5 minutes or something.

My miners don't lock up.  Unless I'm effing with aticonfig.

[dudel42]

for running headless follow the instructions at

http://developer.amd.com/support/KnowledgeBase/Lists/KnowledgeBase/DispForm.aspx?ID=19

that works just fine for me.

I have another problem with the automatic miner startup though. my /etc/init/btcminer.conf file:

Code:

description     "Start BTC Mining"
start on runlevel [2345]
stop on runlevel [!2345]
kill timeout 30
script
  LOGIN=xxx
  #Wait 30 seconds to make sure X is started.
  sleep 30
  exec /home/tom/ati899.sh
  exec /home/tom/atifan80.sh
  exec /usr/bin/screen -dmS gpu0 su -c '/home/tom/startminerloop.sh 0' ${LOGIN}
  exec /usr/bin/screen -dmS gpu1 su -c '/home/tom/startminerloop.sh 1' ${LOGIN}
end script

when I run the 4 commands after exec manually in ssh they work just fine, but somehow not from the startup script. i guess there are some peculiarities to ubuntu upstart that I'm not familiar with

[supa]

If you're using aticonfig commands... You have to make sure that script starts after X has initted.

Otherwise it will fail.  Maybe that's the problem?

I forget the path for ubuntu, but it's something like.... /etc/init.d/rc.3 (or 5 if you're using GUI mode) and there should be a series of scripts with S01ThisService, S35ThatService, etc.

S - Start
35 - Order (there will probably be close to 34 things that start before this)
ServiceName

So S99bitcoin would be pretty safe (it should be the last thing to start).

Hope that makes sense, going kinda fast (at work )

[dudel42]

I got fed up with the upstart script/conf, so I managed another way to automatically start the miners. here's the solution, in case it helps someone else.

in /etc/rc.local

Code:

#!/bin/sh -e
#
# ... comments removed ...
#
# Wait 30 seconds to make sure X is started.
sleep 30
su LOGINNAME -c "/home/LOGINNAME/runminers.sh" >/dev/null 2>&1
exit 0

and in runminers

Code:

/home/LOGINNAME /ati899.sh
/home/LOGINNAME /atifan80.sh
/usr/bin/screen -dmS gpu0 /home/LOGINNAME /startminerloop.sh 0
/usr/bin/screen -dmS gpu1 /home/LOGINNAME /startminerloop.sh 1

to set core clock, fan speed and then start an instance of each miner with screen.

screen -r gpu0/1 to view each session.

would be nice to be able to view both (or more) in screen split screen view... but I haven't figured out how to display the second session in the other window half yet.

for now i'm happy that the miners finally start up automatically

[supa]

Good job!

You could redirect your miner script output to something like /tmp/miner if you want to recall it later.

And Ctrl-A S will do horizontal split screen.

[dudel42]

Good job!

You could redirect your miner script output to something like /tmp/miner if you want to recall it later.

And Ctrl-A S will do horizontal split screen.

yeah, i figured out how to split the screen, but not how to view the second session in the split view...

[supa]

Good job!

You could redirect your miner script output to something like /tmp/miner if you want to recall it later.

And Ctrl-A S will do horizontal split screen.

yeah, i figured out how to split the screen, but not how to view the second session in the split view...

There are a couple of ways.

Here's one that I use -

Ctrl-A S (split screen)
Ctrl-A : (command)
Type "screen /bin/bash"
Then "screen -ls" to see your sessions.
And finally screen -r <pid> to attach your new window to an existing session.

You can Ctrl-A Ctrl-A or Ctrl-A tab to flip between them.

[spiccioli]

I got fed up with the upstart script/conf, so I managed another way to automatically start the miners. here's the solution, in case it helps someone else.

in /etc/rc.local

snipped text....

dudel42,

I'm using linuxcoin and I've made something very similar to your solution, but, instead of having several detached sessions each with a miner inside (for a multi gpu setup, of course), I have shell script inside my home called 'mining.sh' which is called by rc.local and has inside

Code:

su user -c 'cd $HOME && screen -dmS miners -c /home/user/screenrc.miners'

This starts a detached screen session called 'miners' and makes screen call screenrc.miners which has something like this:

Code:

source $HOME/.screenrc
screen -t bitcoin.cz0 ./bitcoin.sh 0
screen -t bitcoin.cz1 ./bitcoin.sh 1
screen -t temp.0 watch -n 10 "DISPLAY=:0 aticonfig --adapter=all --odgt"
screen -t temp.1 watch -n 10 "DISPLAY=:0.1 aticonfig --adapter=all --odgt"

and what this does is to calls .screenrc, if it exists, and then creates several sessions (like C-A c) named with the word following -t each of which runs a script which calls the real miner passing a '0' or '1' to differentiate between gpus.

It also calls watch -n 10 "...." which shows every 10 seconds the temperature of the GPUs

The advantage of this solution is that you have a single screen session to attach to (screen -r) when you log in from remote and then you switch between the various 'windows' with C-a n (where n is a number from 0 to your last session -1) or C-A " (which shows you the session list).

Hope this helps.

spiccioli.

[jondecker76]

I'm working on an automated mining management and load balancing system as we speak.  Here is the official thread on it:
http://forum.bitcoin.org/index.php?topic=16548.0

[martok]

Just to put another solution out there. I don't like the idea of waiting 30 seconds for x to start. It's hackish and assumes that X will start in 30 seconds. I use gdm3 and have daemon.conf set to automatically log in the bitcoin user. Then in /home/bitcoin/.gnomerc, you can have all your miner start code, not having to care about $DISPLAY or whether X has launched properly. Further, this doesn't require creating a custom init.d script.

I think this could be adjusted further by taking out gnome entirely and having a line like:
@reboot startx
in a user crontab then using a .xinitrc to launch things.

[supa]

Ya, 30 seconds is an awfully long time.

You could always use something like xwininfo to verify it's running.

For example, if this fails, you have no server running -

xwininfo -display :0 -root

Return on fail should be - "xwininfo: error: unable to open display ":0""
I think xset q could be used, too.

xset -display :0 q

I wouldn't fiddle with gdm - I only use a basic wm and want the minimal work requests to the video card.

[um0rion]

Im just poking my head in to offer the following link... its a complete linux guide to running headless. I stopped before that part but the rest of the scripts work fine, headless part probably does too.

http://forum.bitcoin.org/index.php?topic=9239.0

[dukejer]

Just to put another solution out there. I don't like the idea of waiting 30 seconds for x to start. It's hackish and assumes that X will start in 30 seconds. I use gdm3 and have daemon.conf set to automatically log in the bitcoin user. Then in /home/bitcoin/.gnomerc, you can have all your miner start code, not having to care about $DISPLAY or whether X has launched properly. Further, this doesn't require creating a custom init.d script.

I think this could be adjusted further by taking out gnome entirely and having a line like:
@reboot startx
in a user crontab then using a .xinitrc to launch things.

Hi martok,

When I originally wrote the upstart script I was in a hurry and wanted to get my rig mining up as soon as possible.  I still plan on rewriting the btcminer scripts to take out the sleep 30 and search for X running.   I new it was hackish at the time but in bitcoin time is money.  That would also speed up the process but I have not had the time to do that yet.  One advantage of running X straight without runing gdm is that all the extra process that are associated with Gnome and GDM are not running.  On the latest systems I set up I build them with the mini.iso ubuntu image without even loading a desktop.  The only thing I installed was xorg and the utilities to build the ati driver and other software like python.

Here is the original post I wrote for headless mining.

http://forum.bitcoin.org/index.php?topic=9239.0

- Dukejer

Extra Bitcoins burning a hole in your pocket?  Send them to 19oQLN2v1auiGqBDFfxSDUfu8Ax6ooT6xs if you feel my posts help you.

[dukejer]

Ya, 30 seconds is an awfully long time.

You could always use something like xwininfo to verify it's running.

For example, if this fails, you have no server running -

xwininfo -display :0 -root

Return on fail should be - "xwininfo: error: unable to open display ":0""
I think xset q could be used, too.

xset -display :0 q

I wouldn't fiddle with gdm - I only use a basic wm and want the minimal work requests to the video card.

Hi supa,

I made some changes to my btcminer_X.conf upstart scripts.  They will now check for X and start up automatically.

Original scripts      http://forum.bitcoin.org/index.php?topic=9239.0

cat /etc/init/btcminer_0.conf

description     "Start BTC Mining"
start on runlevel [2345]
stop on runlevel [!2345]
kill timeout 30
script
  LOGINUSER=ChangeToMyLoginUser
  export DISPLAY=:0
  TIMEOUT=30
  XPID=""
  while true
  do
    sleep 1
    TIMEOUT=`/usr/bin/expr ${TIMEOUT} - 1`
    test ${TIMEOUT} -lt  0 && {
      echo "X not running."
      exit 255
    }
   
    XPID=`/usr/bin/pgrep -f -x /usr/bin/X`
    test -z ${XPID} && continue
    exec /usr/bin/screen -dmS gpu0 su -c '/usr/local/bin/startminer.sh 0' ${LOGINUSER}
    exit 0
  done
end script


- Dukejer

Extra Bitcoins burning a hole in your pocket?  Send them to 19oQLN2v1auiGqBDFfxSDUfu8Ax6ooT6xs if you feel my posts help you.

         
 

[Bitonetta]

Is there a way to script screen so I can start it up in a single 'screen' session?

I don't want 4 instances of screen running.  I want a single one with 4 /bin/bash.  I can split screen them all then remotely.

Basically, a script way of:
$ screen -dmS Miner
$ ./startminer1.sh (inside screen)
CTRL-A c (Create new screen)
$ ./startminer2.sh (inside same initial screen)
....

Should be able to start detached but if not, no biggie.  I would rather be able to monitor all 4 gpus from a single SSH connection.  Right now I need to start up each gpu miner manually in each screen window.  It would be great to be able to do it via a script.

Thanks.

[supa]

...

Pretty clever!

[Grinder]

Is there a way to script screen so I can start it up in a single 'screen' session?

Make a script (screenminers.sh) that contains

Code:

#!/bin/sh
screen ./startminer1.sh
screen ./startminer2.sh

and run
screen -dmS Miner ./screenminers.sh

[Bitonetta]

Is there a way to script screen so I can start it up in a single 'screen' session?

Make a script (screenminers.sh) that contains

Code:

#!/bin/sh
screen ./startminer1.sh
screen ./startminer2.sh

and run
screen -dmS Miner ./screenminers.sh

Thanks!  I just tried it.  That's what I was looking for.  I didn't know that calling screen from within will attach to the current process.  Assumed it would spawn another instance of itself.

I don't assume screen (or sh/bash) has a way to not close itself when the ./startminer1.sh closes.  Perferably after the .sh closes, it'll just sit at $

Regardless, thanks for help.