Smartcoin Linux mining administration. [MULTI-MACHINE SUPPORT NOW IN!]

[jondecker76]

Has anyone experienced this?
When adding a new profile, sometimes there are 0 miners to select from. Running r496e

If this happens again, can you check your log for a message: "ERROR: SQL Query failed".

My guess its from a query being blocked from retrying until its considered "dead".  Today I am going to be working on optimizing the database calls some, which will help.  I'm also going to raise the amount of times database queries retry - right now they will retry for about 1/4 of a second.  I'm going to raise that up to about 10 seconds to ensure that queries go through, but it is still possible to fail on a bad query or bad sqlite3 error.

[jondecker76]

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!

There was a pretty major rewrite of the code that generates the profile stack.  I have personally seen it working under "Failover" profile. Please let me know if anyone sees any problems with any of the other profiles.

Code:

martcoin r501e 12:24:25
----------------------------------------
Host: localhost
GPU[0]: Temp: 79.00 load: 99%
GPU[1]: Temp: 76.50 load: 99%
GPU[2]: Temp: 65.00 load: 99%
CPU Load : 13.38%

Profile: Failover
--------X8S--------
GPU[0]: <<<DOWN>>>
GPU[0]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
Total : [0 MHash/sec] [0 Accepted] [0 Rejected] [0.00% Rejected]

Failover to: BTCGuild Me!
--------BTCGuild--------
GPU[0]: [367.15 MHash/sec] [35 Accepted] [0 Rejected] [0% Rejected]
GPU[1]: [368.16 MHash/sec] [28 Accepted] [0 Rejected] [0% Rejected]
GPU[2]: [367.96 MHash/sec] [20 Accepted] [0 Rejected] [0% Rejected]
Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

Grand Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

This is why I love the Failover profile!  Does anyone have a current list of pools using PPLNS?  It kind of sucks for those of us doing honest failover opposed to pool hopping...

[krzynek1]

Thanks for the screenshot of the error -I'm looking into it - though I'm not finding anything yet.  Are all 3 miners the same? (I.e. running the same distro and version). Also, can you post the result of this:

Code:

sqlite3 ~/.smartcoin/smartcoin.db "SELECT * FROM device;"

Yes, CPU usage going up in recent versions is normal.  In older versions, there was a delay put into the loop on purpose to make it run slower. I removed the delay to make the loop run much faster (you should see that the display updates more frequently now). It shouldn't hurt anything though

all 3 miners are the same, same operating system, ati sdk etc. screen refreshes are faster, that is correct, but cpu usage goes too much, ~80 % compared to maybe 20 % with previous releases is not good from economic point of view, mayby you could add some variable how fast user want the refreshes ?

result of that command :

1|1|G1|1|gpu|0|0
2|1|G2|2|gpu|0|0
3|1|G3|3|gpu|0|0
4|1|G4|4|gpu|0|0

[plantucha]

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!

There was a pretty major rewrite of the code that generates the profile stack.  I have personally seen it working under "Failover" profile. Please let me know if anyone sees any problems with any of the other profiles.

Code:

martcoin r501e 12:24:25
----------------------------------------
Host: localhost
GPU[0]: Temp: 79.00 load: 99%
GPU[1]: Temp: 76.50 load: 99%
GPU[2]: Temp: 65.00 load: 99%
CPU Load : 13.38%

Profile: Failover
--------X8S--------
GPU[0]: <<<DOWN>>>
GPU[0]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
Total : [0 MHash/sec] [0 Accepted] [0 Rejected] [0.00% Rejected]

Failover to: BTCGuild Me!
--------BTCGuild--------
GPU[0]: [367.15 MHash/sec] [35 Accepted] [0 Rejected] [0% Rejected]
GPU[1]: [368.16 MHash/sec] [28 Accepted] [0 Rejected] [0% Rejected]
GPU[2]: [367.96 MHash/sec] [20 Accepted] [0 Rejected] [0% Rejected]
Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

Grand Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

This is why I love the Failover profile!  Does anyone have a current list of pools using PPLNS?  It kind of sucks for those of us doing honest failover opposed to pool hopping...

same on my rig - failover no go
single profile is running ok
automatic no go

[jondecker76]

Update r502e now available

- Fixed a bug in the installer where one of the new settings was not being created in the database on new installs (affected failover on high rejection percentage)

- There is a new "Loop Delay" setting in the settings menu.  Here you are able to add a manual delay to the status loop (default=0, no delay). Older versions of smartcoin used a delay of 5 each loop to keep CPU usage down.  Of course, there is a tradeoff between speed and CPU usage, so now the user can decide where they want that tradeoff to be with this new setting. Delay value is in seconds.
You will need to restart smartcoin after updating before the new setting takes effect.

[jondecker76]

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!

There was a pretty major rewrite of the code that generates the profile stack.  I have personally seen it working under "Failover" profile. Please let me know if anyone sees any problems with any of the other profiles.

Code:

martcoin r501e 12:24:25
----------------------------------------
Host: localhost
GPU[0]: Temp: 79.00 load: 99%
GPU[1]: Temp: 76.50 load: 99%
GPU[2]: Temp: 65.00 load: 99%
CPU Load : 13.38%

Profile: Failover
--------X8S--------
GPU[0]: <<<DOWN>>>
GPU[0]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[1]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
GPU[2]: <<<DOWN>>>
Total : [0 MHash/sec] [0 Accepted] [0 Rejected] [0.00% Rejected]

Failover to: BTCGuild Me!
--------BTCGuild--------
GPU[0]: [367.15 MHash/sec] [35 Accepted] [0 Rejected] [0% Rejected]
GPU[1]: [368.16 MHash/sec] [28 Accepted] [0 Rejected] [0% Rejected]
GPU[2]: [367.96 MHash/sec] [20 Accepted] [0 Rejected] [0% Rejected]
Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

Grand Total : [1103.27 MHash/sec] [83 Accepted] [0 Rejected] [0% Rejected]

This is why I love the Failover profile!  Does anyone have a current list of pools using PPLNS?  It kind of sucks for those of us doing honest failover opposed to pool hopping...

same on my rig - failover no go
single profile is running ok
automatic no go

What do you mean by "no go"?

[krzynek1]

JD, man you're fast 

updated to r502e , used 8 as loop delay

cpu have much less stress, load from 0.76 to 0.10 after 10 minutes

abut that error flashing on one of rigs, still apears

it must be related somehow to database, because when i was changing delay loop value, def value was 11 (?) instead of 0 on other rigs

 

[jondecker76]

JD, man you're fast 

updated to r502e , used 8 as loop delay

cpu have much less stress, load from 0.76 to 0.10 after 10 minutes

abut that error flashing on one of rigs, still apears

it must be related somehow to database, because when i was changing delay loop value, def value was 11 (?) instead of 0 on other rigs

 

Ok, must have been an update error.  Does it clear up if you change the value then restart smartcoin?

[jondecker76]

Update r506e now available

Thanks to plantucha, i fould and fixed a few bugs, and made a few changes.
- The auto profile now adheres to the auto_allow field for devices.
- The auto profile now only considers 'gpu' type devices. If you want to mix cpu and gpu types, do this via a manual profile!
- There was a bug in the new failover generation that is fixed. It only affected those using multiple miner types
- Lockup detection will now not happen on an instance showing Down (0 hashes per second). This way, in a profile with multiple instances, a single downed instance won't trip the failover.
- The miner parsers now differentiate between "Down" (0 hashes) and "Idle" (no miner output at all)

[Fletch]

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!

Very nice to hear you managed to optimize the SQL traffic. However, even when running r501e, I'm still seeing about 26% CPU utilization (underclocked Sempron 140). In fact, it didn't decrease noticeable from the previous version. Any idea what might be causing the high CPU utilization? My miners aren't in my home, so I haven't been able to check whether the 26% CPU utilization actually translates into a higher power drain as compared to idle, but in theory, it should.

It should be noted that starting my miners manually in separate screens using the exact same startup string as in Smartcoin results in a CPU utilization around 1 or 2%.

Edit: Just noticed the update adding the loop delay. Will play with that and get back to you.

[krzynek1]

JD, man you're fast 

updated to r502e , used 8 as loop delay

cpu have much less stress, load from 0.76 to 0.10 after 10 minutes

abut that error flashing on one of rigs, still apears

it must be related somehow to database, because when i was changing delay loop value, def value was 11 (?) instead of 0 on other rigs

 

Ok, must have been an update error.  Does it clear up if you change the value then restart smartcoin?

do you mean delay loop value ?

if yes, changing delay doesn't fix anything

this error occurred after r452s to r496s update, in which delay loop wasn't existing  

[Fletch]

I changed the loop delay to 10 and that dropped the CPU utilization drastically (from 26% to around 4%). It would be interesting to disable certain features in smartcoin_status to figure out what is actually causing the high load.

One thing to note though. The CPU utilization reported by iostat is the average since the last reboot so it's not super useful. I would prefer good old /proc/loadavg.

[mackminer]

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through. 

Thanks.

[jondecker76]

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!

Very nice to hear you managed to optimize the SQL traffic. However, even when running r501e, I'm still seeing about 26% CPU utilization (underclocked Sempron 140). In fact, it didn't decrease noticeable from the previous version. Any idea what might be causing the high CPU utilization? My miners aren't in my home, so I haven't been able to check whether the 26% CPU utilization actually translates into a higher power drain as compared to idle, but in theory, it should.

It should be noted that starting my miners manually in separate screens using the exact same startup string as in Smartcoin results in a CPU utilization around 1 or 2%.

Edit: Just noticed the update adding the loop delay. Will play with that and get back to you.

Yes, it will be up to you with the new loop delay setting to balance speed vs. CPU usage.  Smartcoin does a lot more than just open/close miner instances. It has to get a dump each miner screen window for each loop iteration and parse it for output, calculate hash rates, do the lockup detection, failover detection, etc - so of course there is a CPU penalty for all of this (I would guess that the dumping/parsing of the miner screen session windows is the most computationally expensive).

Regarding usage - the difference between 10% load and 30% load is under 20 watts on a 95W processor, so for me the difference extra load is a moot point.  Even so, my old p4 processor handles 3GPUs with 9 instances at under 13% load, which I don't think is bad at all.  For underclocked and low wattage processors, its even a smaller difference in wattage for extra load.  But again, just inject some delay with the loop delay setting and you can bring your CPU usage much lower if you want. Note, that if you run an unusually high loop delay, you may want to reduce the Failover Threshold and Lockup Threshold settings as well, as they are measured in loop iterations.

I will look into other methods of determining CPU load, but an average works really well here, as the script its self has some areas of really high usage, and possibly even no usage at all (during the loop delay for example)  Can you post some comparisons of /proc/loadavg against iostat?

[jondecker76]

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through.  

Thanks.

As soon as I make sure recent changes look good, and do some sqlite throughput testing, error logging and some other minor things, I will start digging into the multi machine support. Since most of it is already in place in the lower level of the code, it shouldn't be too awfully hard or time consuming to implement. I'm hoping for this next coming weekend to have things at least roughed out and working.

[Reuef]

Jon,

I love the system and it works great for me.  I just thought I would share the scripts I wrote to help switch profiles via Web request incase anyone finds it useful.


poollookup.php

Code:

<?php

if (isset($_GET['SetPool']))
	{
	$pool = (int)$_GET['SetPool'];
	// I use /tempdir on my server as a temp file directory, you can use whatever you choose
	$res=fopen('/tempdir/pool.txt',"w");
	fwrite($res,$pool);
	fclose($res);
	echo 'Pool is set to Profile ID '.$pool.'<br><a href="poollookup.php">Click here to return to main page.</a>';
		
	}
elseif (isset($_GET['GetPool']))
	{
	$pool = readfile('/tempdir/pool.txt');
	}
elseif (isset($_GET['KillMonitoring']))
	{
	$pool = 'Stop';
	$res=fopen('/tempdir/pool.txt',"w");
	fwrite($res,$pool);
	fclose($res);
	echo 'Pool monitoring will be stopped';
	}
else
{

//I have my actual list generated at runtime from my db file, below is just an example of what you can do.
 	?>
	<html>
	<body>
	Current Pool Profile is <?php readfile('/tempdir/pool.txt'); ?>:<br>
	Choose Which Profile you want to Run:<br>
	5)<a href="poollookup.php?SetPool=5">Profile A</a><br>
	2)<a href="poollookup.php?SetPool=2">Profile B</a><br>
	3)<a href="poollookup.php?SetPool=3">Profile C</a><br>
	
	
	</body>	
	</html>
	<?php
}
?>

smartcoinmonitor.php

Code:

<?php
UpdateSmartcoin();
function UpdateSmartcoin(){
	
$ch = curl_init();

curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($ch, CURLOPT_URL,"http://{Webserver}/poollookup.php?GetPool");
curl_setopt($ch, CURLOPT_HEADER, 0);
ob_start();
curl_exec($ch);
curl_close($ch);
$res = ob_get_clean();

if ($res != 'Stop')
{
	
if ($db = new SQLite3("/home/{USERNAME}/.smartcoin/smartcoin.db",SQLITE3_OPEN_READWRITE)) { 

	$FindCurProfile = "SELECT `fk_profile` FROM `current_profile` WHERE `fk_machine` = 1";
	$currProfile = $db->querySingle($FindCurProfile);

		
	if (($currProfile != (int)$res) && ((int)$res != NULL))
	{
	$db->exec('UPDATE `current_profile` SET `fk_profile` = '.$res.' WHERE `fk_machine` = 1;');
	echo 'Switching to Profile:'.$res."\n";		
	}
	else
	{
		
	echo 'Sticking with Profile:'.$currProfile."\n";	
	}
	$db->close();
} else {
    die("Sqlite failed to open\n");
}

//Have the system wait 30 seconds and check again.
sleep(30);
UpdateSmartcoin();
}
else
{
	exit;
}
}

?>

smartcoinmonitor.php is run on your miner system(s), and poollookup.php is on your web server (which can also be on one of your miners).  Please remember to put the proper webser address in as well as the proper location of your smartcoin.db file.

Purpose:
I have 3 mining rigs running now using Smartcoin, but having 3 SSH's to change a profile on is a bit cumbersome sometimes, so I have all my miners (all with the same profile IDs) ask my webserver which profile they should run.

The code isn't pretty but it is useful to me and it may be useful to you.

Requirements:
a webserver with PHP5
PHP 5.3 or higher on your miner
LibCurl on your miner

Sample usage:
php smartcoinmonitor.php

or use Screen to be able to disconnect from it while it runs.

screen -S monitor php smartcoinmonitor.php

If this is helpful to you please consider a donation to 12MjprGQKxhSHUKaqT7d4QuZ54LCPqr1pd

[DBordello]

Did poclbm change recently confusing smartcoin?  I just installed poclbm (git://github.com/m0mchil/poclbm) and smartcoin for the first time. The syntax smartcoin was using (--host, etc) was invalid.  It also does not appear to be able to scrape the output (I can see the miners running in screen, but smartcoin does not read it).   Additionally, it was (kind of) using invalid device ids.  Poclbm uses device 0 as the CPU.  Therefore the first miner ended up on the CPU. 

I was able to work around all of these except the scraping issue.  I was just wondering.

Dan

[plantucha]

Did poclbm change recently confusing smartcoin?  I just installed poclbm (git://github.com/m0mchil/poclbm) and smartcoin for the first time. The syntax smartcoin was using (--host, etc) was invalid.  It also does not appear to be able to scrape the output (I can see the miners running in screen, but smartcoin does not read it).   Additionally, it was (kind of) using invalid device ids.  Poclbm uses device 0 as the CPU.  Therefore the first miner ended up on the CPU. 

I was able to work around all of these except the scraping issue.  I was just wondering.

Dan

I think yet only supported is Phoenix (GPU) and Cgminer (CPU)

[mackminer]

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through.  

Thanks.

As soon as I make sure recent changes look good, and do some sqlite throughput testing, error logging and some other minor things, I will start digging into the multi machine support. Since most of it is already in place in the lower level of the code, it shouldn't be too awfully hard or time consuming to implement. I'm hoping for this next coming weekend to have things at least roughed out and working.

Excellent - great job.

[jondecker76]

Jon,

I love the system and it works great for me.  I just thought I would share the scripts I wrote to help switch profiles via Web request incase anyone finds it useful.


poollookup.php

Code:

<?php

if (isset($_GET['SetPool']))
	{
	$pool = (int)$_GET['SetPool'];
	// I use /tempdir on my server as a temp file directory, you can use whatever you choose
	$res=fopen('/tempdir/pool.txt',"w");
	fwrite($res,$pool);
	fclose($res);
	echo 'Pool is set to Profile ID '.$pool.'<br><a href="poollookup.php">Click here to return to main page.</a>';
		
	}
elseif (isset($_GET['GetPool']))
	{
	$pool = readfile('/tempdir/pool.txt');
	}
elseif (isset($_GET['KillMonitoring']))
	{
	$pool = 'Stop';
	$res=fopen('/tempdir/pool.txt',"w");
	fwrite($res,$pool);
	fclose($res);
	echo 'Pool monitoring will be stopped';
	}
else
{

//I have my actual list generated at runtime from my db file, below is just an example of what you can do.
 	?>
	<html>
	<body>
	Current Pool Profile is <?php readfile('/tempdir/pool.txt'); ?>:<br>
	Choose Which Profile you want to Run:<br>
	5)<a href="poollookup.php?SetPool=5">Profile A</a><br>
	2)<a href="poollookup.php?SetPool=2">Profile B</a><br>
	3)<a href="poollookup.php?SetPool=3">Profile C</a><br>
	
	
	</body>	
	</html>
	<?php
}
?>

smartcoinmonitor.php

Code:

<?php
UpdateSmartcoin();
function UpdateSmartcoin(){
	
$ch = curl_init();

curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, 0); 
curl_setopt($ch, CURLOPT_URL,"http://{Webserver}/poollookup.php?GetPool");
curl_setopt($ch, CURLOPT_HEADER, 0);
ob_start();
curl_exec($ch);
curl_close($ch);
$res = ob_get_clean();

if ($res != 'Stop')
{
	
if ($db = new SQLite3("/home/{USERNAME}/.smartcoin/smartcoin.db",SQLITE3_OPEN_READWRITE)) { 

	$FindCurProfile = "SELECT `fk_profile` FROM `current_profile` WHERE `fk_machine` = 1";
	$currProfile = $db->querySingle($FindCurProfile);

		
	if (($currProfile != (int)$res) && ((int)$res != NULL))
	{
	$db->exec('UPDATE `current_profile` SET `fk_profile` = '.$res.' WHERE `fk_machine` = 1;');
	echo 'Switching to Profile:'.$res."\n";		
	}
	else
	{
		
	echo 'Sticking with Profile:'.$currProfile."\n";	
	}
	$db->close();
} else {
    die("Sqlite failed to open\n");
}

//Have the system wait 30 seconds and check again.
sleep(30);
UpdateSmartcoin();
}
else
{
	exit;
}
}

?>

smartcoinmonitor.php is run on your miner system(s), and poollookup.php is on your web server (which can also be on one of your miners).  Please remember to put the proper webser address in as well as the proper location of your smartcoin.db file.

Purpose:
I have 3 mining rigs running now using Smartcoin, but having 3 SSH's to change a profile on is a bit cumbersome sometimes, so I have all my miners (all with the same profile IDs) ask my webserver which profile they should run.

The code isn't pretty but it is useful to me and it may be useful to you.

Requirements:
a webserver with PHP5
PHP 5.3 or higher on your miner
LibCurl on your miner

Sample usage:
php smartcoinmonitor.php

or use Screen to be able to disconnect from it while it runs.

screen -S monitor php smartcoinmonitor.php

If this is helpful to you please consider a donation to 12MjprGQKxhSHUKaqT7d4QuZ54LCPqr1pd

Thanks for posting your php scripts, surely there should be plenty of users that find it useful!
I do plan on making an access API in the future so external scripts can more easily interact with smartcoin. Be sure to post your suggestions for this feature, I'll be sure to add them to my list