Flexible mining proxy

[leepfrog]

Great peace of software. Managed to get it running in under 10 minutes.
I've always had some cases in the past where there were connection problems and the miner was in a stale state after connectivity has been restored. Looking forward to see if this is solved when using the proxy.

It also would be an interesting idea to take such a service and offer it for hosting, so that the average miner does not have to worry about pool failover any more as he can simply point to some public proxy which handles the requests.

[1714967427]

1714967427

[1714967427]

1714967427

[1714967427]

1714967427

[1714967427]

1714967427

[1714967427]

1714967427

[leepfrog]

Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again:

Traceback (most recent call last):
  File "phoenix.py", line 125, in <module>
  File "twisted\internet\base.pyc", line 1162, in run
  File "twisted\internet\base.pyc", line 1171, in mainLoop
--- <exception caught here> ---
  File "twisted\internet\base.pyc", line 793, in runUntilCurrent
  File "minerutil\RPCProtocol.pyc", line 127, in callback
  File "minerutil\RPCProtocol.pyc", line 319, in handleWork
exceptions.TypeError: argument of type 'NoneType' is not iterable
[23/05/2011 01:04:42] Result: 4b18edf7 rejected

My connection has some packet loss (~2-4%) but the latency in general is good (under 150ms). Could this be a problem?

[oVPN]

hi

there are problems with php-authentication, if php is running as fast-cgi, but after switching to mod_php, login to admin-panel is working.

i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts

[twmz]

I really like the idea of this tool, but have failed to get it working well.

I use phoenix as my miner on ubuntu 11.04.  I have tried a couple different web hosts including my own ubuntu 11.04 box, and I have the following symptoms:

1. lots more stale shares than when connected directly.  long polling does not seem to work with phoenix through the proxy but works fine when connected directly.
2. phoenix miner *hammers* the web server with requests to index.php.  Something like 10-20 requests per second as seen in the apache access.log.  When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so).
3. lower than normal hash rate when using the proxy.  I suspect this may be because the miner is so busy doing #2 as fast as it can.

I also tried poclbm on my Mac and it had similar problems with stale shares.  I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network.  I also regularly got errors about "NoneType" not being indexable.

From this thread, though, it sounds like some people are having great success with it, so I wish I knew what to do differently.

[datathe1st]

I'd like to set this up to manage a mining for computers I have in different locations, but I am hesitant because of reports by some users of stale shares.

Thank you so very much cdhowie for even releasing what looks like an amazing piece of software.


Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that?

[Dusty]

Watching (very nice project btw)

[cdhowie]

Just pushed some updates.

• Applied patch from cdecker implementing hash speed estimation for workers (issue 5).
• Implemented timezone conversion (issue 7).

These changes require modifications to your config.inc.php.  If you pull without updating your config.inc.php, the software will break.

To address some posts:

Actually this just didn't work very well for me   Set it up with BTC Guild as primary and Slush as secondary.  BTC Guild is having intermittent connection issues, so I wanted a backup.  However, overnight all of my GUIMiner clients got stuck with a "connection problem".  My one phoenix client (which is also on same machine as my proxy) continued working fine however.

Some miners will bail on a connection if it doesn't return work within 3-5 seconds.  This is not always enough time for the proxy to detect a fail condition, and so the miner gives up before the proxy has a chance to try another pool.

Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again:

I have started seeing similar problems on my install.  This is being actively researched.

I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network.  I also regularly got errors about "NoneType" not being indexable.

I have been seeing the same with poclbm lately.  I'm not sure what changed to make this happen, but I suspect it is related to the stale shares problem.

i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts

I haven't used that particular miner and so I'm not sure if the proxy supports it.  If the miner isn't doing anything too wacky it should work...  If I get some free time I might investigate this.

long polling does not seem to work with phoenix through the proxy but works fine when connected directly.

There are some issues with long polling that are not easily addressed by the current model.  Long polling should in theory work just fine if your preferred pool doesn't go down, but during a failover scenario it can get a little bit wonky.  Some miners behave differently with respect to how they handle changing long-polling URLs, and so this is something that will take a lot of engineering to fix.

In the meantime I may supply a config option to disable long polling entirely; broken long polling is worse than none at all.

phoenix miner *hammers* the web server with requests to index.php.  Something like 10-20 requests per second as seen in the apache access.log.  When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so).

Sounds like a bug in Phoenix.  Can you take some sniffs of this behavior with tcpdump -w bmp.pcap port 80 and email me the pcap file(s)?  This will help me figure out what's going on.  (Please compress them, and you can optionally encrypt them to my GPG key.  This should not expose your mining account passwords, only your local proxy worker and possibly admin passwords.)

Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that?

As long as the computers can make outbound connections on whatever port you are running your web server on, yes.  Transparent (or non-transparent) HTTP proxies should be able to process these requests too, though they might strip long-polling info.

[twmz]

Hey, cdhowie.  I'm still not clear on what was happening with the proxy, but I have a couple clues.

I did determine that the latest phoenix behavior is to re-request the long polling URL immediatlly if the LP does not return a valid getwork response.  So, I think the issue with phoenix hammering the web server was because the proxied long-poll address must have been immediately returning, which caused phoenix to request it again right away, which caused...  well, a very fast loop of LP requests to the proxy.

In the end, I only know enough about PHP to be dangerous, and less about how to run an apache web server.  In the end, I was so inspired by the idea of this proxy that I ended up recoding it in ASP.NET, not because it is better than ASP.NET but just because I have 10 years of experience with it and know how to debug.  For the actual proxying of the JSON-RPC requests, I followed your algorithm almost exactly, just in C#, ASP.NET MVC, IIS and SQL Server.  I now have it working and long polling is working as well although the whole design of long polling tie-ing up a web service thread for 10-20 minutes doesn't play all nicely with IIS, but it doesn't matter too much for my 4 measly miners.  A side bonus is that ASP.NET has a very capable charting component and so I was able to quickly whip up some graphs for the dashboard as well.

To be clear, I doubt anyone will care about my implementation.  An IIS web server and SQL Server are much harder to come by than a Linux, Apache, MySQL, and PHP server.  I just happen to have a suitable windows server in my house already, so it worked well for me.  That said, as soon as I polish it a little more, I will post the source code somewhere given that I did borrow the idea and some of the assets from your project.

[anisoptera]

To be clear, I doubt anyone will care about my implementation. 

I do, I definitely want to see the code.

[Cdecker]

Just seeing now that you have a payout monitor in there, that would be a cool feature, though it adds a lot of problems since some pools don't provide that info and even worse, they don't adhere to a consistent standard for publishing them.

[twmz]

Just seeing now that you have a payout monitor in there, that would be a cool feature, though it adds a lot of problems since some pools don't provide that info and even worse, they don't adhere to a consistent standard for publishing them.

Yes, there is no standard.  But it only turned out to be a couple lines of code specific to each type of pool to fish the balance out of each pools JSON API.

It actually doesn't work that well and is the main thing I am working on right now before releasing the code in the next day or two.  Making all of the JSON API calls to retrieve current balances is making the dashboard load slowly, so I'm moving balance updating to a background thread.

[0xlemming]

Is the stuff in work_data needed for something importent? I added to my runnning script a function which remove all entries older than 10h.

[twmz]

10h should be ok.  It is needed when a miner eventually submits a share.  The proxy uses the information in the work_data table to look up which pool the miner got that work from so that it gets submitted back to the right place.  So don't delete everything in the table, but it is not needed as soon as your miners have finished working on their current work task.

[error]

Thanks for the proxy. Seems to work pretty well so far.

I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST:

Code:

--- a/htdocs/common.inc.php    2011-05-29 22:20:55.000000000 -0400
+++ b/htdocs/common.inc.php      2011-05-29 23:29:49.000000000 -0400
@@ -154,7 +154,7 @@
     $port = ($default_port == $_SERVER['SERVER_PORT']) ? "" :
         ":" . $_SERVER['SERVER_PORT'];
 
-    $base = "$scheme://{$_SERVER['SERVER_NAME']}$port" . get_site_uri();
+    $base = "$scheme://{$_SERVER['HTTP_HOST']}$port" . get_site_uri();
 
     return $base . $uri;
 }

[pwnyboy]

For those of you affected by deepbit/slush outages today, how well did this code hold up for you?

[error]

For those of you affected by deepbit/slush outages today, how well did this code hold up for you?

It's pretty transparently switching between pools. The miner never notices a thing.

[pwnyboy]

Sweet.  I have what looks like a working implementation, minus the aforementioned difficulties with long-polling that me and several others have experienced.  But it wasn't in use today for that reason.  I'm hoping to find a simple http proxying script and just change the LP URL handed out by cdhowie's script, to make the miners hit a different script to handle long polling in a non-problematic way.  Or so I hope.

[d3c0n808]

I'm sorry but I need to ask a question.  Currently for pools I have deepbit and solo, deepbit has a higher priority but doesnt connect for some reason.  I rebooted, the computer can resolve the dns but none of the miners connect to deepbit just the solo.  Perhaps this has been asked before.  Im wondering if there is some type of cache that needs to be reset or what.  Im using poclbm gui and poclbm on my dedicated miners(slackware)  The admin pages seems to be working fine but its just not kicking over to deepbit.  Thank you in advance and the address i ahve in the pool config is http://pit.deepbit.net:8332

[CubedRoot]

Going to look into this one for myself.  I am having a problem mining from work   I have my own co-located server, but dont want to setup a pool on it.  I might could use this to connect my miners to my co-located server on a port that isnt blocked by our firewall, and then use my co-located server to connect to the pools.

[Yngwie]

I can't configure this proxy((
Admin panel works fine, i add pools (deepbit: http://deepbit.net:8332 and btcguild: http://btcguild.com:8332) then created worker (login q password 1) add pool workers credentials for it (email_5 password_of_worker_in_pool for deepbit login_2 password_of_worker_in_pool for btc guild). But when i started miner (i tried poclbm, bitcoin-miner, diablo miner windows with my servers ip as url port 80 login and password from proxy worker) proxy returned
HTTP/1.1 200 OK

Date: Mon, 30 May 2011 07:46:57 GMT

Server: Apache

X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy

Set-Cookie: PHPSESSID=kph1h3dib6nnad4okc3k29kp85; path=/

Expires: Thu, 19 Nov 1981 08:52:00 GMT

Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0

Pragma: no-cache

Content-Length: 85

Content-Type: application/json-rpc



{"error":"No enabled pools responded to the work request.","result":null,"id":"json"}

How can i solve this problem?