Flexible mining proxy

[cdhowie]

What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account?

Then the proxy will use the same credentials for all three workers.  If pointing all three miners at the same pool using the same account works fine, then pointing them at the same pool using the same account via the proxy will also work.

@cdhowie, this app is amazing, but need some work of sql optimization and hash-rate calculator

How so?  The hash-rate calculation queries perform very quickly on my install, and EXPLAIN shows that the indexes are used appropriately.  Which commit are you running from?

[1713575022]

1713575022

[cdhowie]

diablo doesnt work at all, after a while it the proxy stops accepting shares. Rejected doesnt even show up.

The proxy's lack of support for X-Roll-NTime may have been to blame for this.  Pull down the latest code; I just added X-Roll-NTime support about 45 minutes ago.

[Grinder]

EDIT: I have a workable solution. I was able to successfully submit work with the proxy to MultiPool.
I will discuss the fix further with cdhowie in the morning and see how he wants to go about implementing it.

Did you get anywhere with this?

[kripz]

What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account?

Then the proxy will use the same credentials for all three workers.  If pointing all three miners at the same pool using the same account works fine, then pointing them at the same pool using the same account via the proxy will also work.

So no problems with getworks and the like? One card solves something and the others some how knows to getwork again?

I'll try the ntime fix later when i get home.

[cdhowie]

try to update to latest version now

Code:

Current database version is 2.  Attempting migration to version 3.

Migrating 2 -> 3 ...

The work_data.data column cannot be shrunk because that would result in duplicate primary key values.  Please truncate the work_data table and try migrating again.

Final database version: 2.

Database migration did not fully complete.  Correct the errors displayed above and try again.

EDIT: I fixed it by deleting the whole database and import a fresh database and load backup to my worker/worker_pool/pool

You could have just done what it suggested: TRUNCATE TABLE work_data;

EDIT: I have a workable solution. I was able to successfully submit work with the proxy to MultiPool.
I will discuss the fix further with cdhowie in the morning and see how he wants to go about implementing it.

Did you get anywhere with this?

Multipool is currently down, so I haven't been able to review the problem and proposed fix.

So no problems with getworks and the like? One card solves something and the others some how knows to getwork again?

I'm not sure what you mean.  The mining software is responsible for determining how to allocate work to the various devices it supports.  My proxy doesn't know or care how many GPUs you have.

[kripz]

So if i have 2 or more cards, an instance of poclbm or phoenix (which requires one instance per device) hasher running for each card, each pointed to the one proxy account which points to the one miner account on a pool, it doesn't cause issues?

What about multiple accounts for the proxy (so a proxy account for each device) which all point to the one miner account on a pool?

I was thinking the proxy may hand out the same work to multiple cards and if one of the cards solve it, then the others wont know about it and keep hasing.

[Grinder]

Multipool is currently down, so I haven't been able to review the problem and proposed fix.

He's put the code on Github, so someone else set up a server here: http://multiclone.us.to:18080/ (mining port 18337).
It has the same problem.

http://forum.bitcoin.org/index.php?topic=17970.msg302834#msg302834

[eusor]

I also just added the new s3.mining.eligius.st pool. For some weird reason, I'm getting a lot of invalid shares :

Submitted duplicate shares   17   4
Submitted stale shares   35   0

Which I don't get if I access directly. Anyone else noticed this behavior? The only thing I can see is that worker use the same login on the pool.

[cdhowie]

I also just added the new s3.mining.eligius.st pool. For some weird reason, I'm getting a lot of invalid shares :

Submitted duplicate shares   17   4
Submitted stale shares   35   0

Which I don't get if I access directly. Anyone else noticed this behavior? The only thing I can see is that worker use the same login on the pool.

If you ignore the duplicate/stale stats and focus only on the accepted shares, are the numbers very different over time from what you see mining without the proxy?  I am wondering if the proxy code that retries share submissions on errors is successfully submitting it the first time, but thinks that there was some connection issue and tries again, thereby giving you a duplicate or stale share but also correctly giving you the one accepted share.

[SteveFL]

Been lurking in this thread for awhile now.  Most of the issues I've encountered have already been reported by other users and patched, but I have two things I can't figure out:

1) I have a very disparate number of getworks to shares in the pool summary (251 to 26).

2) I noticed on one of my any of my clients running poclbm that every 60 seconds after starting I get an RPC error before it continues.  I haven't seen this on phoenix or Ufasoft's miners...

Code:

30/06/2011 09:58:43, Setting pool test @ proxypool:8332
30/06/2011 09:58:44, Using new LP URL /index.php/1/aHR0cDovL21pbmVjby5pbjozMDAwL0xQ
30/06/2011 09:58:44, LP connected to proxypool:8332
30/06/2011 09:58:44, 3829 khash/s
...
30/06/2011 09:59:38, 2290 khash/s
30/06/2011 09:59:39, Problems communicating with bitcoin RPC 0 2
30/06/2011 09:59:40, 1017 khash/s
...

I don't know if these are all related or its just a poclbm issue.  I was thinking a timeout in apache/php could be the cause but I don't know the best way to diagnose.

I had noticed for the last two weeks I had an abnormally high number of apache2 processes running (150+) for about 20 active miners.  I'd eventually have to restart after apache stopped responding.  This seems to have cleared with the changes committed in the last 3 days.  Again, also not sure if related.

Excellent project by the way.  I just wish I hadn't been tainted by coding microsoft.net all these years so I could contribute some code.

[cdhowie]

Been lurking in this thread for awhile now.  Most of the issues I've encountered have already been reported by other users and patched, but I have two things I can't figure out:

1) I have a very disparate number of getworks to shares in the pool summary (251 to 26).

The getwork code performs failover after 3 seconds if a pool doesn't respond.  This means that a little lag in, say, the DNS lookup can trigger a failover.  This will not be a problem with the C# backend since it will keep a queue of a few pieces of work per pool, populated on a background thread.  Getworks from a worker will pull off of the queue and trigger the background thread to refill it.

2) I noticed on one of my any of my clients running poclbm that every 60 seconds after starting I get an RPC error before it continues.  I haven't seen this on phoenix or Ufasoft's miners...

That's odd.  poclbm is all I use and I haven't noticed too many problems with it.  I do note that it tends to abort a getwork request after about 5 seconds and try again immediately.  This means that poclbm and the PHP failover code are often at odds, since by the time that the proxy fails over to the second pool, poclbm has aborted the request and is trying again.

I had noticed for the last two weeks I had an abnormally high number of apache2 processes running (150+) for about 20 active miners.  I'd eventually have to restart after apache stopped responding.  This seems to have cleared with the changes committed in the last 3 days.  Again, also not sure if related.

Hmm, I'm not sure what I could have changed that would fix this.  The frequency of requests might be causing Apache to try to keep more HTTP worker processes running to serve them.  And if your miners recognize the X-Roll-NTime header (which I just added), that means they will usually only do a handful of getworks per round, and so the number of requests to Apache will drop dramatically.

Excellent project by the way.  I just wish I hadn't been tainted by coding microsoft.net all these years so I could contribute some code.

I don't think you'd be prevented from contributing to the PHP side of the project.    And since I'm not building a compatible framework (like Mono is), I don't think I really need to exclude MS.NET developers from the C# side of the project either.

[cdhowie]

The C# getwork/LP backend is now feature-complete and is being tested.  If there are no major issues, expect a release of it before Tuesday.

[kripz]

Please consider porting to Java, now i'll have to install mono?

Everybody will probably have Java already.

[cdhowie]

Please consider porting to Java, now i'll have to install mono?

I thought about writing it in Java, until I realized that means I would have to write it in Java.  Then I took some Pepto for the resulting nausea.

Everybody will probably have Java already.

[citation needed]

If on a LAMP server, Java probably won't be installed by default.  If on a desktop, Java still probably won't be installed by default (unless it's GNU classpath) and in some cases Mono will be installed too, for Banshee/Tomboy/F-Spot.

Honestly, I'd rather focus my energy on adding support for PostgreSQL, which I should have done in the first place...

[btc_man]

So when you say "At this point you should be able to point your workers at the proxy and they will start working." in the installation instructions/



What address should I be using?

[kripz]

Your server IP or hostname.

[btc_man]

this is what is showing up in the apache logs

Code:

192.168.2.2 - default [03/Jul/2011:10:03:50 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:00 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:10 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:20 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:30 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"

And this is the command that i am using to launch the miners

Code:

python phoenix.py -u http://192.168.2.7:8337 -k poclbm DEVICE=2 BFI_INT WORKSIZE=128 AGGRESSION=8 FASTLOOP VECTORS

Can you maybe try to tell me what the problem is or what other info i need to figure the problem out?

[cdhowie]

this is what is showing up in the apache logs

Code:

192.168.2.2 - default [03/Jul/2011:10:03:50 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:00 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:10 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:20 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"
192.168.2.2 - default [03/Jul/2011:10:04:30 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4"

And this is the command that i am using to launch the miners

Code:

python phoenix.py -u http://192.168.2.7:8337 -k poclbm DEVICE=2 BFI_INT WORKSIZE=128 AGGRESSION=8 FASTLOOP VECTORS

Can you maybe try to tell me what the problem is or what other info i need to figure the problem out?

Response 401 is "unauthorized."  Your command does not appear to include any authentication information.  You need to add the username and password of one of the proxy's worker accounts.

[error]

Thanks for the proxy. Seems to work pretty well so far.

I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST:

Code:

--- a/htdocs/common.inc.php    2011-05-29 22:20:55.000000000 -0400
+++ b/htdocs/common.inc.php      2011-05-29 23:29:49.000000000 -0400
@@ -154,7 +154,7 @@
     $port = ($default_port == $_SERVER['SERVER_PORT']) ? "" :
         ":" . $_SERVER['SERVER_PORT'];
 
-    $base = "$scheme://{$_SERVER['SERVER_NAME']}$port" . get_site_uri();
+    $base = "$scheme://{$_SERVER['HTTP_HOST']}$port" . get_site_uri();
 
     return $base . $uri;
 }

Hey, you still haven't included this fix, as far as I can tell. I just pulled master and it isn't there.

[cdhowie]

Thanks for the proxy. Seems to work pretty well so far.

I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST:

Code:

--- a/htdocs/common.inc.php    2011-05-29 22:20:55.000000000 -0400
+++ b/htdocs/common.inc.php      2011-05-29 23:29:49.000000000 -0400
@@ -154,7 +154,7 @@
     $port = ($default_port == $_SERVER['SERVER_PORT']) ? "" :
         ":" . $_SERVER['SERVER_PORT'];
 
-    $base = "$scheme://{$_SERVER['SERVER_NAME']}$port" . get_site_uri();
+    $base = "$scheme://{$_SERVER['HTTP_HOST']}$port" . get_site_uri();
 
     return $base . $uri;
 }

Hey, you still haven't included this fix, as far as I can tell. I just pulled master and it isn't there.

Sorry, I must have missed or forgotten about the original bug report.  The Github issue tracker is where I track bug reports and patches, so if it's not there then there's a good chance I won't remember it.

Patch applied and pushed, thanks!