Bitcoin Forum
December 09, 2016, 02:12:46 AM *
News: To be able to use the next phase of the beta forum software, please ensure that your email address is correct/functional.
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 [35] 36 37 38 39 40 41 42 43 »
  Print  
Author Topic: Smartcoin Linux mining administration. [MULTI-MACHINE SUPPORT NOW IN!]  (Read 99668 times)
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 24, 2011, 09:10:37 PM
 #681

Update r506e now available

Thanks to plantucha, i fould and fixed a few bugs, and made a few changes.
- The auto profile now adheres to the auto_allow field for devices.
- The auto profile now only considers 'gpu' type devices. If you want to mix cpu and gpu types, do this via a manual profile!
- There was a bug in the new failover generation that is fixed. It only affected those using multiple miner types
- Lockup detection will now not happen on an instance showing Down (0 hashes per second). This way, in a profile with multiple instances, a single downed instance won't trip the failover.
- The miner parsers now differentiate between "Down" (0 hashes) and "Idle" (no miner output at all)


RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
1481249566
Hero Member
*
Offline Offline

Posts: 1481249566

View Profile Personal Message (Offline)

Ignore
1481249566
Reply with quote  #2

1481249566
Report to moderator
Fletch
Full Member
***
Offline Offline

Activity: 168


I'll have a steak sandwich and a... steak sandwich


View Profile
July 24, 2011, 09:24:30 PM
 #682

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!
Very nice to hear you managed to optimize the SQL traffic. However, even when running r501e, I'm still seeing about 26% CPU utilization (underclocked Sempron 140). In fact, it didn't decrease noticeable from the previous version. Any idea what might be causing the high CPU utilization? My miners aren't in my home, so I haven't been able to check whether the 26% CPU utilization actually translates into a higher power drain as compared to idle, but in theory, it should.

It should be noted that starting my miners manually in separate screens using the exact same startup string as in Smartcoin results in a CPU utilization around 1 or 2%.

Edit: Just noticed the update adding the loop delay. Will play with that and get back to you.

HashPeak - GPU mining hashrate peak detector
BTC: 1FLETCHvcUKosefrcZCLUQTtvx4WvgnYMC
krzynek1
Jr. Member
*
Offline Offline

Activity: 41


View Profile
July 24, 2011, 09:26:15 PM
 #683

JD, man you're fast  Shocked

updated to r502e , used 8 as loop delay

cpu have much less stress, load from 0.76 to 0.10 after 10 minutes

abut that error flashing on one of rigs, still apears

it must be related somehow to database, because when i was changing delay loop value, def value was 11 (?) instead of 0 on other rigs

 
Ok, must have been an update error.  Does it clear up if you change the value then restart smartcoin?


do you mean delay loop value ?

if yes, changing delay doesn't fix anything

this error occurred after r452s to r496s update, in which delay loop wasn't existing  
Fletch
Full Member
***
Offline Offline

Activity: 168


I'll have a steak sandwich and a... steak sandwich


View Profile
July 24, 2011, 09:57:37 PM
 #684

I changed the loop delay to 10 and that dropped the CPU utilization drastically (from 26% to around 4%). It would be interesting to disable certain features in smartcoin_status to figure out what is actually causing the high load.

One thing to note though. The CPU utilization reported by iostat is the average since the last reboot so it's not super useful. I would prefer good old /proc/loadavg.

HashPeak - GPU mining hashrate peak detector
BTC: 1FLETCHvcUKosefrcZCLUQTtvx4WvgnYMC
mackminer
Sr. Member
****
Offline Offline

Activity: 335



View Profile
July 24, 2011, 10:17:55 PM
 #685

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through.  Undecided

Thanks.

1tpXGRKnzaM42B1A1NZhmsX3tngwkqVEf
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 24, 2011, 11:31:40 PM
 #686

Update r501e now available
- SQL queries will retry for a lot longer if the database is locked. This greatly reduces the chances of a lost query!
- Major optimizations made in SQL queries.  There is still a little room for improvement, but SQL traffic is reduced by at least 70%!
Very nice to hear you managed to optimize the SQL traffic. However, even when running r501e, I'm still seeing about 26% CPU utilization (underclocked Sempron 140). In fact, it didn't decrease noticeable from the previous version. Any idea what might be causing the high CPU utilization? My miners aren't in my home, so I haven't been able to check whether the 26% CPU utilization actually translates into a higher power drain as compared to idle, but in theory, it should.

It should be noted that starting my miners manually in separate screens using the exact same startup string as in Smartcoin results in a CPU utilization around 1 or 2%.

Edit: Just noticed the update adding the loop delay. Will play with that and get back to you.
Yes, it will be up to you with the new loop delay setting to balance speed vs. CPU usage.  Smartcoin does a lot more than just open/close miner instances. It has to get a dump each miner screen window for each loop iteration and parse it for output, calculate hash rates, do the lockup detection, failover detection, etc - so of course there is a CPU penalty for all of this (I would guess that the dumping/parsing of the miner screen session windows is the most computationally expensive).

Regarding usage - the difference between 10% load and 30% load is under 20 watts on a 95W processor, so for me the difference extra load is a moot point.  Even so, my old p4 processor handles 3GPUs with 9 instances at under 13% load, which I don't think is bad at all.  For underclocked and low wattage processors, its even a smaller difference in wattage for extra load.  But again, just inject some delay with the loop delay setting and you can bring your CPU usage much lower if you want. Note, that if you run an unusually high loop delay, you may want to reduce the Failover Threshold and Lockup Threshold settings as well, as they are measured in loop iterations.

I will look into other methods of determining CPU load, but an average works really well here, as the script its self has some areas of really high usage, and possibly even no usage at all (during the loop delay for example)  Can you post some comparisons of /proc/loadavg against iostat?

RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 24, 2011, 11:36:48 PM
 #687

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through.  Undecided

Thanks.
As soon as I make sure recent changes look good, and do some sqlite throughput testing, error logging and some other minor things, I will start digging into the multi machine support. Since most of it is already in place in the lower level of the code, it shouldn't be too awfully hard or time consuming to implement. I'm hoping for this next coming weekend to have things at least roughed out and working.

RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
Reuef
Jr. Member
*
Offline Offline

Activity: 41


View Profile
July 25, 2011, 12:33:33 AM
 #688

Jon,

I love the system and it works great for me.  I just thought I would share the scripts I wrote to help switch profiles via Web request incase anyone finds it useful.


poollookup.php
Code:
<?php

if (isset($_GET['SetPool']))
{
$pool = (int)$_GET['SetPool'];
// I use /tempdir on my server as a temp file directory, you can use whatever you choose
$res=fopen('/tempdir/pool.txt',"w");
fwrite($res,$pool);
fclose($res);
echo 'Pool is set to Profile ID '.$pool.'<br><a href="poollookup.php">Click here to return to main page.</a>';

}
elseif (isset(
$_GET['GetPool']))
{
$pool readfile('/tempdir/pool.txt');
}
elseif (isset(
$_GET['KillMonitoring']))
{
$pool 'Stop';
$res=fopen('/tempdir/pool.txt',"w");
fwrite($res,$pool);
fclose($res);
echo 'Pool monitoring will be stopped';
}
else
{

//I have my actual list generated at runtime from my db file, below is just an example of what you can do.
 
?>

<html>
<body>
Current Pool Profile is <?php readfile('/tempdir/pool.txt'); ?>:<br>
Choose Which Profile you want to Run:<br>
5)<a href="poollookup.php?SetPool=5">Profile A</a><br>
2)<a href="poollookup.php?SetPool=2">Profile B</a><br>
3)<a href="poollookup.php?SetPool=3">Profile C</a><br>


</body>
</html>
<?php
}
?>

smartcoinmonitor.php
Code:
<?php
UpdateSmartcoin
();
function 
UpdateSmartcoin(){

$ch curl_init();

curl_setopt ($chCURLOPT_SSL_VERIFYHOST0);
curl_setopt ($chCURLOPT_SSL_VERIFYPEER0); 
curl_setopt($chCURLOPT_URL,"http://{Webserver}/poollookup.php?GetPool");
curl_setopt($chCURLOPT_HEADER0);
ob_start();
curl_exec($ch);
curl_close($ch);
$res ob_get_clean();

if (
$res != 'Stop')
{

if (
$db = new SQLite3("/home/{USERNAME}/.smartcoin/smartcoin.db",SQLITE3_OPEN_READWRITE)) { 

$FindCurProfile "SELECT `fk_profile` FROM `current_profile` WHERE `fk_machine` = 1";
$currProfile $db->querySingle($FindCurProfile);


if (($currProfile != (int)$res) && ((int)$res != NULL))
{
$db->exec('UPDATE `current_profile` SET `fk_profile` = '.$res.' WHERE `fk_machine` = 1;');
echo 'Switching to Profile:'.$res."\n";
}
else
{

echo 'Sticking with Profile:'.$currProfile."\n";
}
$db->close();
} else {
    die(
"Sqlite failed to open\n");
}

//Have the system wait 30 seconds and check again.
sleep(30);
UpdateSmartcoin();
}
else
{
exit;
}
}

?>


smartcoinmonitor.php is run on your miner system(s), and poollookup.php is on your web server (which can also be on one of your miners).  Please remember to put the proper webser address in as well as the proper location of your smartcoin.db file.

Purpose:
I have 3 mining rigs running now using Smartcoin, but having 3 SSH's to change a profile on is a bit cumbersome sometimes, so I have all my miners (all with the same profile IDs) ask my webserver which profile they should run.

The code isn't pretty but it is useful to me and it may be useful to you.

Requirements:
a webserver with PHP5
PHP 5.3 or higher on your miner
LibCurl on your miner

Sample usage:
php smartcoinmonitor.php

or use Screen to be able to disconnect from it while it runs.

screen -S monitor php smartcoinmonitor.php

If this is helpful to you please consider a donation to 12MjprGQKxhSHUKaqT7d4QuZ54LCPqr1pd



Reuef
12MjprGQKxhSHUKaqT7d4QuZ54LCPqr1pd
DBordello
Sr. Member
****
Offline Offline

Activity: 350


BTCPak.com - Exchange your Bitcoins for MP!


View Profile WWW
July 25, 2011, 03:09:42 AM
 #689

Did poclbm change recently confusing smartcoin?  I just installed poclbm (git://github.com/m0mchil/poclbm) and smartcoin for the first time. The syntax smartcoin was using (--host, etc) was invalid.  It also does not appear to be able to scrape the output (I can see the miners running in screen, but smartcoin does not read it).   Additionally, it was (kind of) using invalid device ids.  Poclbm uses device 0 as the CPU.  Therefore the first miner ended up on the CPU. 

I was able to work around all of these except the scraping issue.  I was just wondering.

Dan

www.BTCPak.com - Exchange your bitcoins for MP: Secure, Anonymous and Easy!
plantucha
Jr. Member
*
Offline Offline

Activity: 56


View Profile WWW
July 25, 2011, 04:02:43 AM
 #690

Did poclbm change recently confusing smartcoin?  I just installed poclbm (git://github.com/m0mchil/poclbm) and smartcoin for the first time. The syntax smartcoin was using (--host, etc) was invalid.  It also does not appear to be able to scrape the output (I can see the miners running in screen, but smartcoin does not read it).   Additionally, it was (kind of) using invalid device ids.  Poclbm uses device 0 as the CPU.  Therefore the first miner ended up on the CPU. 

I was able to work around all of these except the scraping issue.  I was just wondering.

Dan

I think yet only supported is Phoenix (GPU) and Cgminer (CPU)

mackminer
Sr. Member
****
Offline Offline

Activity: 335



View Profile
July 25, 2011, 08:28:02 AM
 #691

Hi,
Just wondering if there is any update on the multi mining machine setup?

The thread is very long to read through.  Undecided

Thanks.
As soon as I make sure recent changes look good, and do some sqlite throughput testing, error logging and some other minor things, I will start digging into the multi machine support. Since most of it is already in place in the lower level of the code, it shouldn't be too awfully hard or time consuming to implement. I'm hoping for this next coming weekend to have things at least roughed out and working.

Excellent - great job.

1tpXGRKnzaM42B1A1NZhmsX3tngwkqVEf
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 25, 2011, 11:49:03 AM
 #692

Jon,

I love the system and it works great for me.  I just thought I would share the scripts I wrote to help switch profiles via Web request incase anyone finds it useful.


poollookup.php
Code:
<?php

if (isset($_GET['SetPool']))
{
$pool = (int)$_GET['SetPool'];
// I use /tempdir on my server as a temp file directory, you can use whatever you choose
$res=fopen('/tempdir/pool.txt',"w");
fwrite($res,$pool);
fclose($res);
echo 'Pool is set to Profile ID '.$pool.'<br><a href="poollookup.php">Click here to return to main page.</a>';

}
elseif (isset(
$_GET['GetPool']))
{
$pool readfile('/tempdir/pool.txt');
}
elseif (isset(
$_GET['KillMonitoring']))
{
$pool 'Stop';
$res=fopen('/tempdir/pool.txt',"w");
fwrite($res,$pool);
fclose($res);
echo 'Pool monitoring will be stopped';
}
else
{

//I have my actual list generated at runtime from my db file, below is just an example of what you can do.
 
?>

<html>
<body>
Current Pool Profile is <?php readfile('/tempdir/pool.txt'); ?>:<br>
Choose Which Profile you want to Run:<br>
5)<a href="poollookup.php?SetPool=5">Profile A</a><br>
2)<a href="poollookup.php?SetPool=2">Profile B</a><br>
3)<a href="poollookup.php?SetPool=3">Profile C</a><br>


</body>
</html>
<?php
}
?>

smartcoinmonitor.php
Code:
<?php
UpdateSmartcoin
();
function 
UpdateSmartcoin(){

$ch curl_init();

curl_setopt ($chCURLOPT_SSL_VERIFYHOST0);
curl_setopt ($chCURLOPT_SSL_VERIFYPEER0); 
curl_setopt($chCURLOPT_URL,"http://{Webserver}/poollookup.php?GetPool");
curl_setopt($chCURLOPT_HEADER0);
ob_start();
curl_exec($ch);
curl_close($ch);
$res ob_get_clean();

if (
$res != 'Stop')
{

if (
$db = new SQLite3("/home/{USERNAME}/.smartcoin/smartcoin.db",SQLITE3_OPEN_READWRITE)) { 

$FindCurProfile "SELECT `fk_profile` FROM `current_profile` WHERE `fk_machine` = 1";
$currProfile $db->querySingle($FindCurProfile);


if (($currProfile != (int)$res) && ((int)$res != NULL))
{
$db->exec('UPDATE `current_profile` SET `fk_profile` = '.$res.' WHERE `fk_machine` = 1;');
echo 'Switching to Profile:'.$res."\n";
}
else
{

echo 'Sticking with Profile:'.$currProfile."\n";
}
$db->close();
} else {
    die(
"Sqlite failed to open\n");
}

//Have the system wait 30 seconds and check again.
sleep(30);
UpdateSmartcoin();
}
else
{
exit;
}
}

?>


smartcoinmonitor.php is run on your miner system(s), and poollookup.php is on your web server (which can also be on one of your miners).  Please remember to put the proper webser address in as well as the proper location of your smartcoin.db file.

Purpose:
I have 3 mining rigs running now using Smartcoin, but having 3 SSH's to change a profile on is a bit cumbersome sometimes, so I have all my miners (all with the same profile IDs) ask my webserver which profile they should run.

The code isn't pretty but it is useful to me and it may be useful to you.

Requirements:
a webserver with PHP5
PHP 5.3 or higher on your miner
LibCurl on your miner

Sample usage:
php smartcoinmonitor.php

or use Screen to be able to disconnect from it while it runs.

screen -S monitor php smartcoinmonitor.php

If this is helpful to you please consider a donation to 12MjprGQKxhSHUKaqT7d4QuZ54LCPqr1pd




Thanks for posting your php scripts, surely there should be plenty of users that find it useful!
I do plan on making an access API in the future so external scripts can more easily interact with smartcoin. Be sure to post your suggestions for this feature, I'll be sure to add them to my list

RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 25, 2011, 11:50:38 AM
 #693

Did poclbm change recently confusing smartcoin?  I just installed poclbm (git://github.com/m0mchil/poclbm) and smartcoin for the first time. The syntax smartcoin was using (--host, etc) was invalid.  It also does not appear to be able to scrape the output (I can see the miners running in screen, but smartcoin does not read it).   Additionally, it was (kind of) using invalid device ids.  Poclbm uses device 0 as the CPU.  Therefore the first miner ended up on the CPU.  

I was able to work around all of these except the scraping issue.  I was just wondering.

Dan

Can you post the smartcoin status screen output, along with the output of poclbm from the miner screen? Perhaps something has changed, I did test it and it worked just a couple of weeks ago

Also, to my knowlege poclbm doesn't always assume device 0 is cpu (unless something has changed in newer poclbm revisions). Here is the output of me running poclbm with no parameters:
Code:
jondecker76@miner1:~/poclbm$ ./poclbm.py                                       
No device specified or device not found, use -d to specify one of the following
                                                                               
[0]     Cypress                                                                 
[1]     Cypress                                                                 
[2]     Cypress                                                                 
[3]     Intel(R) Pentium(R) 4 CPU 2.80GHz                                       


RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
elrock
Jr. Member
*
Offline Offline

Activity: 41


View Profile
July 25, 2011, 12:27:56 PM
 #694

What is the default miner output format string?  I tried editing the format string and ended up erasing it entirely.  Weird characters appeared every time I pressed one of the arrow keys or the delete key to move the cursor.  Now it's totally blank and I can't see a thing.
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 25, 2011, 12:32:40 PM
 #695

What is the default miner output format string?  I tried editing the format string and ended up erasing it entirely.  Weird characters appeared every time I pressed one of the arrow keys or the delete key to move the cursor.  Now it's totally blank and I can't see a thing.

The default is:
Code:
[<#hashrate#> MHash/sec] [<#accepted#> Accepted] [<#rejected#> Rejected] [<#rejected_percent#>% Rejected]

RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
elrock
Jr. Member
*
Offline Offline

Activity: 41


View Profile
July 25, 2011, 12:44:07 PM
 #696

Thanks -- I found it by searching through the thread.  I know you've already got tons on your plate, but it might be helpful to new users to include that info in the first post.
Fletch
Full Member
***
Offline Offline

Activity: 168


I'll have a steak sandwich and a... steak sandwich


View Profile
July 25, 2011, 02:01:15 PM
 #697

Note, that if you run an unusually high loop delay, you may want to reduce the Failover Threshold and Lockup Threshold settings as well, as they are measured in loop iterations.
I'll do that. My loop delay is 10 and I've set the failover threshold to 3 and the lockup threshold to 10. Would that make sense?

Perhaps you can add a sentence under "Edit Settings" describing what these settings actually mean?

I will look into other methods of determining CPU load, but an average works really well here, as the script its self has some areas of really high usage, and possibly even no usage at all (during the loop delay for example)  Can you post some comparisons of /proc/loadavg against iostat?
It's difficult to compare load averages with straight up CPU utilization. They're not measuring the same thing. I would say load averages are more interesting though as they reflect how the CPU (or CPUs) is coping with the work load. It's quite possible to run at 100% CPU without overloading the machine. It's also possible to overload the machine without running the CPU at 100%. More info here: http://en.wikipedia.org/wiki/Load_(computing)

HashPeak - GPU mining hashrate peak detector
BTC: 1FLETCHvcUKosefrcZCLUQTtvx4WvgnYMC
jondecker76
Full Member
***
Offline Offline

Activity: 238


View Profile
July 25, 2011, 02:56:09 PM
 #698

kennel -
Your settings look good to me.  If you get any false triggers of failover and/or lockup, you can always increase them 1 or 2 at a time until you find a good balance.

I'll experiment  with the /proc load average tonight (just got to decide whether to go with 1 minute or 5 minute average, though I'm leaning towards 1 minute) Additionally,  it would get rid of another dependency.

Regarding better descriptions of the settings, I'm probably going to leave it documentation in the future.  The settings system is very dynamic in that I just add an entry into the database and it automagically appears and just works - this makes development a little easier on me.  I may rethink it a bit once I can get out of beta, as I won't be adding entries so often then and it may be worth extending the system to be more verbose.

RollerBot Advanced Trading Platform
https://bitcointalk.org/index.php?topic=447727.0
BTC Donations for development: 1H36oTJsi3adFh68wwzz95tPP2xoAoTmhC
EnzoMatrix
Jr. Member
*
Offline Offline

Activity: 42


View Profile
July 25, 2011, 05:48:24 PM
 #699

kennel -
Your settings look good to me.  If you get any false triggers of failover and/or lockup, you can always increase them 1 or 2 at a time until you find a good balance.

I'll experiment  with the /proc load average tonight (just got to decide whether to go with 1 minute or 5 minute average, though I'm leaning towards 1 minute) Additionally,  it would get rid of another dependency.

Regarding better descriptions of the settings, I'm probably going to leave it documentation in the future.  The settings system is very dynamic in that I just add an entry into the database and it automagically appears and just works - this makes development a little easier on me.  I may rethink it a bit once I can get out of beta, as I won't be adding entries so often then and it may be worth extending the system to be more verbose.

If you are using the database for that then you could add a description field utilize that for basic info with out too much modification to your existing code ( just a thought ) keep up the great work love the code so far

--Enzo

Donate Address := 1Ng89ohn4mjWg7SsQizYUPRdVjtWm4d9p3
EnzoMatrix
Jr. Member
*
Offline Offline

Activity: 42


View Profile
July 25, 2011, 05:50:13 PM
 #700

If you are going to utilize the load / cpu averages make sure that the thresholds are adjustable as my mining box also runs a minecraft server so my cpu runs at around 30% - 35% pretty constantly.

Donate Address := 1Ng89ohn4mjWg7SsQizYUPRdVjtWm4d9p3
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 [35] 36 37 38 39 40 41 42 43 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!