BAMT - Easy persistent USB key based linux for dedicated miners/mining farms

[lodcrappo]

I am having an issue where mining on a rig will hault for no apparent reason. I can still ssh into the box and if I reboot, everything starts up fine again. Also, when I try to access gpumon after ssh-ing into the box, the process seem to be hung.

Is there a way to monitor the phoenix processes and if they become hung, restart them or the box itself?

If phoenix itself locks up, it will be killed automatically by the timeout and restarted by the wrapper.

However, if you lock up your GPU, there is nothing BAMT can do for you.  Restarting phoenix isn't going to make any difference, the GPU will still be hung.   Even if I did add some sort of auto reboot, there is a decent chance a reboot will not solve the problem, or could even make it worse.

The mining time you are losing due to lockups is massive.  Any auto recovery I did come up with, even if there was something that would work reliably (and I don't think there is) would only reduce those losses, not eliminate them.  

So, the right solution here is to *stop locking up your GPU*.  Remove all overclocking.  Does the problem go away?
If not, you probably have heating issues, power supply issues, or just a bad card.  

[lodcrappo]

I've messed up a couple pools files in /etc/bamt/ and I want to delete them. However, from File Manager I get a permissions error that won't allow me to delete, rename, or overwrite the old poolsX files. Someone help this linux noob, please. I have changed the default P/W.

I didn't know we even had a "File Manager" to be honest.

BAMT is not designed to be used from the GUI, it only has X on it because you need that to mine.

The simplest way to manage your BAMT config is from gpumon.  Press 'p' to edit the pools file, or 'c' to edit the configuration.  Make your changes and press Ctrl-X,y,enter.  Once back in gpumon, shift-R will restart mining if needed.

If you want to do anything more than that, you'll need to become a little familiar with the linux command line.  It's not hard.  Google is your friend.

[mikeo]

I've messed up a couple pools files in /etc/bamt/ and I want to delete them. However, from File Manager I get a permissions error that won't allow me to delete, rename, or overwrite the old poolsX files. Someone help this linux noob, please. I have changed the default P/W.

As long as you are comfortable with a little command line, that is the easiest way to do it: Go to your "start menu that we cannot call a start menu because microsoft trademarked it", whatever it is called these days: Under the top option, I think it is system? There is a root terminal option. You'll have to enter your root password, default is "changeme", but you did specify you changed it. Once in there, the following commands, without quotes of course:
"cd /etc/bamt" This brings you to the directory /etc/bamt, similar to cd on dos systems
"ls" this is similar to the dos "dir" command, in that it will show you all of the files in the directory
To remove a file, like if the file is your pool32 , just go "rm ./pool32", and it should delete it, so you can put whatever else you want in its place. The "./" just tells it to look in the current directory and no where else for the file... Generally totally not needed, but safer :p

I usually remote in to my BAMT boxes from my desktop, so I do not remember how to do it in the gui, although you might want to look for somethign like "root file manager" in the menu if you want to do it that way.

@kirax,
Thanks! That worked nicely. Now if you'd like to share, in the same simplistic step-by-step way, how you remote into your miners from a desktop machine I'll see if I can figure out a way to send a little coinage your way.

[gnar1ta$]

So, the right solution here is to *stop locking up your GPU*.  Remove all overclocking.  Does the problem go away?

If it's only 1 GPU out of 6 you don't want to lose Mhash by dropping all the clocks, especially on the price paid for the MB and PS.  What's the best way to find that 1?

[blackhat]

What's the best way to find that 1?

Set up your monitoring. The GPU that bail out first is the culprit.

Maybe you will have to clock down neighbor cards as well, because the GPU chip heats up the card next of it too. I noticed that a certain card between two others got literally fried because on the soldering side of the PCB the card is even hotter, and the fans of the middle card suck the hot air up. So, I had to downclock the middle card and the right because the right's bailing out and the middle's melting down. Bad.

[lodcrappo]

What's the best way to find that 1?

Set up your monitoring. The GPU that bail out first is the culprit.

Maybe you will have to clock down neighbor cards as well, because the GPU chip heats up the card next of it too. I noticed that a certain card between two others got literally fried because on the soldering side of the PCB the card is even hotter, and the fans of the middle card suck the hot air up. So, I had to downclock the middle card and the right because the right's bailing out and the middle's melting down. Bad.

these are good points.  a card that seems to be running OK but is generating way too much heat to its neighbor might be making an otherwise good card lock up.  

there is no easy answer or single process that will reveal why your GPUs are locking up.  I suggest disabling all overclocking because then at least you know if, in general, overclocking is causing the lockup or not.  It is not a solution but merely the first diagnostic step.  If you turn off o/c and problems go away, now you know its something you can fix by playing with o/c.  If the problems persist after turning off o/c, you know its something with the hardware, power heat or just bad components.

Anyway, none of this has anything to do with BAMT.  We have taken lots of time in this thread with hardware issues, but we've yet to discover anything that can be changed in BAMT that will make any difference with these sorts of problem.  They are just the result of pushing hardware to the limit of what it can do, or packing it into cases in ways it wasn't meant to be packed into, stressing power supplies beyond their tolerances, etc.  These questions should really move to a more general:  "Why does my mining rig lock up" thread.

[tysat]

I had a similar issue with the system locking up on me.  The fix ended up being taking the OC on the culprit card down 20 mhz, and my system has been stable for 2+ weeks now.

[jamesg]

Thank you for all of the feedback. I really appreciate you guys willing to be so helpful. I am only having these problems on 2 machines out of 13 so the % that is dropping out is quite low. I will definitely take the clocks down and make sure the cards are spaced as far apart as possible.

I now have a couple weeks full time on this so I will be making the appropriate adjustments from your recommendations.

[kirax]

I've messed up a couple pools files in /etc/bamt/ and I want to delete them. However, from File Manager I get a permissions error that won't allow me to delete, rename, or overwrite the old poolsX files. Someone help this linux noob, please. I have changed the default P/W.

As long as you are comfortable with a little command line, that is the easiest way to do it: Go to your "start menu that we cannot call a start menu because microsoft trademarked it", whatever it is called these days: Under the top option, I think it is system? There is a root terminal option. You'll have to enter your root password, default is "changeme", but you did specify you changed it. Once in there, the following commands, without quotes of course:
"cd /etc/bamt" This brings you to the directory /etc/bamt, similar to cd on dos systems
"ls" this is similar to the dos "dir" command, in that it will show you all of the files in the directory
To remove a file, like if the file is your pool32 , just go "rm ./pool32", and it should delete it, so you can put whatever else you want in its place. The "./" just tells it to look in the current directory and no where else for the file... Generally totally not needed, but safer :p

I usually remote in to my BAMT boxes from my desktop, so I do not remember how to do it in the gui, although you might want to look for somethign like "root file manager" in the menu if you want to do it that way.

@kirax,
Thanks! That worked nicely. Now if you'd like to share, in the same simplistic step-by-step way, how you remote into your miners from a desktop machine I'll see if I can figure out a way to send a little coinage your way.

Well, if you are running Mac, I don't recall how to do it, but I imagine OSX has SSH support built in. For windows, you want "putty", from http://www.chiark.greenend.org.uk/~sgtatham/putty/. Go to download putty, and just download putty, you don't need puttytel or anything. Open putty after you download it (No install needed, even), and put in the IP of whichever BAMT rig you want to connect to (if you need the IP, then on your BAMT rig, in a command line window, go "ifconfig" to find your ip address.) Hit connect, and you will need your username/password for the BAMT rig (The default user is username: user, password: live), and from there, it will be almost just like you are sitting in front of the computer at a command window. You can run gpumon (or, in my case, mgpumon) from there and then watch stuff from the comfort of your desktop.

[mikeo]

Thanks, kirax. my desktop machine is dual purpose: Win7 for mundane work & gaming, mining with BAMT when it isn't being used for those tasks. I understand that Putty is needed for Win7, but how to I access/control the two other miners when my desktop is also mining with BAMT? Is there a version of Putty for Debian linux? I've read about people using "ssh" from a linux box to control other linux boxes, but have no idea how to make that work.

[jamesg]

Thanks, kirax. my desktop machine is dual purpose: Win7 for mundane work & gaming, mining with BAMT when it isn't being used for those tasks. I understand that Putty is needed for Win7, but how to I access/control the two other miners when my desktop is also mining with BAMT? Is there a version of Putty for Debian linux? I've read about people using "ssh" from a linux box to control other linux boxes, but have no idea how to make that work.

From any linux box run the command

ssh root@192.168.1.111

This command will allow you to ssh to any other linux box. If you have created another user on the system, replace root with that user's name.

[mikeo]

@lodcrappo

Here is script that reads GPU temps once a second and varies fan speeds to try and maintain stable temps throughout the day. It came from this post about a very cool case design https://bitcointalk.org/index.php?topic=40709.0 and I thought it may trigger an idea for you.

Code:

#!/usr/bin/python

from re import findall, search
from shlex import split
from subprocess import Popen, PIPE
from sys import stdout, argv
from os import environ

# define target range for temps
TEMP_COOL = 60
TEMP_LOW = 68
TEMP_HIGH = 72
TEMP_CRIT = 80

# define fan speeds limits
FAN_MIN = 20
FAN_MAX = 100

# strings for all hardware polling commands
GET_ADAPTERS = 'aticonfig --list-adapters'
GET_TEMPS = 'aticonfig --odgt --adapter=all'
GET_CLOCKS = 'aticonfig --odgc --adapter=all'
GET_FANS = 'aticonfig --pplib-cmd "get fanspeed 0"'
SET_FANS = 'aticonfig --pplib-cmd "set fanspeed 0 $"'

# make it easier to display terminal text in color
TEXT_COLORS = ["black", "red", "green", "yellow", "blue", "magenta", "cyan", "white"]
def textcolor(text, color):
	code = str(TEXT_COLORS.index(color) + 30)
	return "\033[0;" + code + "m" + text + "\033[m"

# get all current stats from the hardware (except fans)
raw_adapters = Popen(split(GET_ADAPTERS), stdout=PIPE).communicate()[0]
raw_temps = Popen(split(GET_TEMPS), stdout=PIPE).communicate()[0]
raw_clocks = Popen(split(GET_CLOCKS), stdout=PIPE).communicate()[0]

# make a list of the adapter ids
adapter_ids = findall(r"\*?\s+(\d)\.", raw_adapters)

# getting fan data is a bit specialized and requires some iteration
raw_fans = {}
for i in adapter_ids:
	n = int(i)

	# the DISPLAY env var needs to be set before each fan is polled
	environ["DISPLAY"] = ":0." + i
	raw_fans[n] = Popen(split(GET_FANS), stdout=PIPE).communicate()[0]

# all of the parsed stats will be stored here
adapters = []

# iterate over each adapter to parse stats and store them
for i in adapter_ids:
	n = int(i)

	# parse the stats with regex keeping things as adaptable as possible
	temp = search("(?s)" + "Adapter " + i + ".*?" + r"(\d+\.\d+)", raw_temps)
	clocks = search("(?s)" + "Adapter " + i + ".*?" + r"(\d+)\s+(\d+)" + ".*?" + r"(\d+)\s+(\d+)" + ".*?" + r"(\d+)%", raw_clocks)
	fan = search(r"(\d+)%", raw_fans[n])

	# store the parsed data into the adapter list and convert types
	adapters.insert(n, {})
	cur = adapters[n]
	cur["dev"] = n
	cur["temp"] = float(temp.group(1))
	cur["core"] = int(clocks.group(1))
	cur["mem"] = int(clocks.group(2))
	cur["pcore"] = int(clocks.group(3))
	cur["pmem"] = int(clocks.group(4))
	cur["load"] = int(clocks.group(5))
	cur["fan"] = int(fan.group(1))
	cur["dfan"] = 0
	
# if temp is outside range then adjust the fan speed up or down
def adjust_fan(cur):
	if cur["temp"] < TEMP_LOW: cur["dfan"] = -1
	elif cur["temp"] > TEMP_HIGH: cur["dfan"] = 1
	while cur["fan"] + cur["dfan"] < FAN_MIN: cur["dfan"] += 1
	while cur["fan"] + cur["dfan"] > FAN_MAX: cur["dfan"] -= 1
	if cur["dfan"] != 0:
		cur["fan"] += cur["dfan"]
		environ["DISPLAY"] = ":0." + str(cur["dev"])
		command = SET_FANS.replace("$", str(cur["fan"]))
		Popen(split(command), stdout=PIPE)

# color the temp display based on its current value
def color_temps(cur):
	color = "green"
	if cur["temp"] < TEMP_COOL: color = "blue"
	elif cur["temp"] < TEMP_LOW: color = "cyan"
	elif cur["temp"] > TEMP_CRIT: color = "red"
	elif cur["temp"] > TEMP_HIGH: color = "yellow"
	cur["temp"] = textcolor("%.1fC" % cur["temp"], color)

# stores the order, label, and format string of each output
outputs = (
	("dev", "GPU%d"),
	("temp", "%s"),
	("load", "%d%%"),
	("core", "%d"),
	("pcore", "%d"),
	("mem", "%d"),
	("pmem", "%d"),
	("fan", "%d%%"),
	("dfan", "%+d%%"))

# print column headers and a row of data for each adapter
# also, adjust fan speeds for each adapter if necessary
for output in outputs: stdout.write(output[0] + "\t")
stdout.write("\n")
for adapter in adapters:
	adjust_fan(adapter)
	color_temps(adapter)
	for output in outputs:
		name = output[0]
		format = output[1]
		value = adapter[name]
		stdout.write(format % value + "\t")
	stdout.write("\n")

[lodcrappo]

@lodcrappo

Here is script that reads GPU temps once a second and varies fan speeds to try and maintain stable temps throughout the day. It came from this post about a very cool case design https://bitcointalk.org/index.php?topic=40709.0 and I thought it may trigger an idea for you.

It's a nice idea, but it has a fatal flaw.  When a GPU locks up, it usually hangs any software that tries to read it's status (temp, etc). 

This is why BAMT does not provide any automatic fan control:  They don't work, and they fail at the very worst times...typically when your GPU is overheating.   The last thing you want is your GPU melting down/locking when it could have been fine, if only the fan had been running faster, yet that is exactly what you're asking for by using any software based fan control.

If you must have variable fan speed, strongly recommend you edit your GPU bios to set the built in, "hardware" based fan control to a curve you like.   (yes it is still done in software, but on software in the GPU not on your PC, and its much more reliable).  Even better is to simply set the fans to 100% and forget about it.

Anyway, although BAMT will never support software fan control for the reasons explained above, that script should work fine if you'd like to install it and use it yourself.

[mikeo]

@lodcrappo,

OK, thanks for the explanation about controlling fans.

Now, before I jump into centralized config control I would like to clarify a few things about the following example code from the wiki before I do a linux noob move:

Code:

  # central config management
  do_manage_config: 0
  # this command is executed every 60 seconds, if it changes contents of /etc/bamt/ mining is restarted
  config_update_cmd: /usr/bin/rsync -aL rsync://192.168.1.1:873/config/miner012/* /etc/bamt/

1. First, this code would run from which machines, miners or central control machine?
2. The IP address is that of the central control machine?
3. "config" is the file the settings are pulled from?
4. "miner012" refers to the GPU cards on the miner runing the code?

Sorry for my lack of linux savvy.

[lodcrappo]

@lodcrappo,

OK, thanks for the explanation about controlling fans.

Now, before I jump into centralized config control I would like to clarify a few things about the following example code from the wiki before I do a linux noob move:

Code:

  # central config management
  do_manage_config: 0
  # this command is executed every 60 seconds, if it changes contents of /etc/bamt/ mining is restarted
  config_update_cmd: /usr/bin/rsync -aL rsync://192.168.1.1:873/config/miner012/* /etc/bamt/

1. First, this code would run from which machines, miners or central control machine?
2. The IP address is that of the central control machine?
3. "config" is the file the settings are pulled from?
4. "miner012" refers to the GPU cards on the miner runing the code?

Sorry for my lack of linux savvy.

Don't worry about the specific syntax of that example command in the default config.  It is there to speed things up for people who understand what it's doing  You will probably want to just delete that.

You will need to create a command that pulls the file(s) you want to sync from somewhere and puts them into /etc/bamt.  There are many ways to do that.   The right way for you will depend on your own network.  Copying/syncing files around on a linux box is a very common thing to do, and you will have no trouble finding examples of ways this can be done using google.  

When do_manage_config is set to 1 on a BAMT machine, then every 60 seconds that machine will:

1. snapshot /etc/bamt
2. run the command in config_update_cmd
3. compare /etc/bamt to the snapshot taken earlier.
4. if anything in /etc/bamt is different, restart mining

So it's really very simple and basic.  Any command that gets the files from "somewhere" onto the local machine will do.
BAMT is just the framework here, not the mechanism.

Possibilities are pulling your config from a web server, using rsync, using scp, mounting a network drive and just copying files, or probably 1000s of other techniques.   One guy is using a dropbox account and wget to pull the files for instance.  You also need to consider whether you want to sync both the bamt.conf and the pools file, or just one or the other, and if you want all rigs to share a common config/pools, or all have their own, or something in between.
It's really all up to you.

[lodcrappo]

The new proxy is working nicely.  I'm at the point where I need to finalize how work is routed and distributed to pools so I can come up with some way to describe that in a configuration file.

Right now, it works on the concept of priority.  You can define an unlimited number of pools, and each is given a priority.  Work is always requested from the highest priority pool that works at the moment the getwork request is made.  In other words, only the highest priority pool ever gets a request, unless that pool isn't responding.

I added one simple extension to this rule, which is that work is done round-robin among equally ranked pools.  Traditionally it seems this has been done to increase availability by simpler mining systems.  There is really no need for that in the BAMT proxy, since a working pool is always found for every individual request at the time it is made.  If high availability is your goal, you would simply define the pools in the order of your preference, they will "load balance" as needed automagically to ensure your miners are always working.  As many shares as possible will always be mined at your most desired pool.

So.. my question for everyone is:  Is there any other reason you'd want to split work up into different pools?  Do some people just want to mine in more than one place?  If so, how do you want to be able to split work up?  Is round robin/equal shares to all pools good enough? If not, what strategy would you like to see?

[kirax]

The new proxy is working nicely.  I'm at the point where I need to finalize how work is routed and distributed to pools so I can come up with some way to describe that in a configuration file.

Right now, it works on the concept of priority.  You can define an unlimited number of pools, and each is given a priority.  Work is always requested from the highest priority pool that works at the moment the getwork request is made.  In other words, only the highest priority pool ever gets a request, unless that pool isn't responding.

I added one simple extension to this rule, which is that work is done round-robin among equally ranked pools.  Traditionally it seems this has been done to increase availability by simpler mining systems.  There is really no need for that in the BAMT proxy, since a working pool is always found for every individual request at the time it is made.  If high availability is your goal, you would simply define the pools in the order of your preference, they will "load balance" as needed automagically to ensure your miners are always working.  As many shares as possible will always be mined at your most desired pool.

So.. my question for everyone is:  Is there any other reason you'd want to split work up into different pools?  Do some people just want to mine in more than one place?  If so, how do you want to be able to split work up?  Is round robin/equal shares to all pools good enough? If not, what strategy would you like to see?

I think that some people might want a % split off into separate pools.. and with that and failover, I think people could create whatever config they wanted.

[chunglam]

Thanks for this great tool for mining. I put it into my three rigs, so far no issue found . Just one suggestion - add pool statistic to mgpumon. Thanks again.

[gnar1ta$]

I think that some people might want a % split off into separate pools.. and with that and failover, I think people could create whatever config they wanted.

+1

[lodcrappo]

I think that some people might want a % split off into separate pools.. and with that and failover, I think people could create whatever config they wanted.

+1

Ok.. for this type of thing, help me to understand the reason someone would do this.  It adds quite a bit of complexity to the configuration file, so I want to make sure we can do what you want to do but also want to make it straightforward.  It would help to know why this is desired.