Flexible mining proxy

[cdhowie]

Ok further to my issue, I captured one of those sessions per below:

If I'm reading this right, the miner is actually closing the long-polling connection of its own accord, mid-stream.  What miner are you using?  Perhaps I can reproduce the issue locally.

[pwnyboy]

Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing.  I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.

Two additional feature requests:

1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone?  I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have.

2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far).  Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script?

Thanks again for all of your hard work!

[cdhowie]

Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing.  I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.

Cool, keep me posted on that.

1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone?  I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have.

Sure!  I planned to do this at one point but forgot about it later.  Thanks for reminding me.

2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far).  Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script?

Yeah, this is probably something that could be done.  The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible.  I assume this would be used to, for example, email you if one of the miners stops requesting work?

I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run.

Thanks again for all of your hard work!

No problem!  I'm glad you find the software useful. 

[pwnyboy]

Yeah, this is probably something that could be done.  The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible.  I assume this would be used to, for example, email you if one of the miners stops requesting work?

Correct.  One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like:

`$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2`

And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds.

I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run.

Or the script could be a field in the database for workers, i.e. if the miner is a Windows box you might want a simple email, but if it's Linux, you might want to execute a different script that runs 'expect' and attempts to restart the miner by logging in via ssh and giving it a kick.  For that matter, the other two could be database arguments, but if it's too difficult to be practical from a programmatical standpoint, no worries.

Thanks again for all of your hard work!

No problem!  I'm glad you find the software useful. 
[/quote]

Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours.  So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages.  Phoenix 1.45 seems to sporadically connect to the long-polling URL.  Phoenix 1.2 will connect, but it disconnects after a period of time.  I think I see why.  I hit my local mining proxy and fetched the long polling URL.  The first time it worked, and actually sent me info on a new block.  But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix:

GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1
Connection: close
Host: bitproxy.xxxxx.com
Authorization: Basic xxx
User-Agent: phoenix/v1.2


(((2 minutes elapse, and then...)))


HTTP/1.1 200 OK
Date: Tue, 10 May 2011 06:00:12 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.1.6
X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy
Set-Cookie: PHPSESSID=4j2r53dsse5afoa5u9k3oratn0; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 78
Connection: close
Content-Type: application/json-rpc

{"error":"Invalid response from long-poll request.","result":null,"id":"json"}Connection closed by foreign host.


Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block.  This should be easily duplicable for you.  If not, I can PM you the info on my proxy and let you see for yourself.  Hope that helps.

[Cdecker]

Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing.  I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.

Cool, keep me posted on that.

1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone?  I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have.

Sure!  I planned to do this at one point but forgot about it later.  Thanks for reminding me.

2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far).  Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script?

Yeah, this is probably something that could be done.  The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible.  I assume this would be used to, for example, email you if one of the miners stops requesting work?

I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run.

Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^

BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^

Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d

Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^

[cdhowie]

Correct.  One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like:

`$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2`

And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds.

I'm not sure I could include the IP since that's not logged anywhere.  But I could include the worker's database ID as well as the worker name.

Or the script could be a field in the database for workers, i.e. if the miner is a Windows box you might want a simple email, but if it's Linux, you might want to execute a different script that runs 'expect' and attempts to restart the miner by logging in via ssh and giving it a kick.  For that matter, the other two could be database arguments, but if it's too difficult to be practical from a programmatical standpoint, no worries.

To keep database schema changes to a minimum, I'm not sure if I would want to do this.  But if I pass the worker's ID to the script, you could have your script look at the ID and figure out what to do.

Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours.  So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages.  Phoenix 1.45 seems to sporadically connect to the long-polling URL.  Phoenix 1.2 will connect, but it disconnects after a period of time.  I think I see why.  I hit my local mining proxy and fetched the long polling URL.  The first time it worked, and actually sent me info on a new block.  But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix:

...

Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block.  This should be easily duplicable for you.  If not, I can PM you the info on my proxy and let you see for yourself.  Hope that helps.

Hmm.  This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this.  It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker.

[cdhowie]

Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^

There is support in the MVC framework for JSON output.  The dashboard view page already implements IJsonView so try visiting /admin/?format=json.  This works for me.

BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^

You're right, I'm not sure why that's there.  That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems.  

Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d

Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^

I haven't looked at it in-depth yet.  Let me know when you are done with it and have tested it and I'll review the pull request.

[Cdecker]

Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^

There is support in the MVC framework for JSON output.  The dashboard view page already implements IJsonView so try visiting /admin/?format=json.  This works for me.

Great, it works for me too, just had to change the URL in my scripts. I think using such an API to run cronjobs against is more useful than having it crammed into the web interface.

BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^

You're right, I'm not sure why that's there.  That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems. 

Agreed

Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d

Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^

I haven't looked at it in-depth yet.  Let me know when you are done with it and have tested it and I'll review the pull request.

I guess I can drop the stats branch then, since it already works out of the box from your dashboard. On the worker-hashing branch I made the average interval configurable because it strongly depends on the hashing speed (high hashing speeds can get a more accurate reading with shorter intervals, while slow miners will have the average jump wildly around for small values). I think it is ready to be pulled. Once again sorry for all the noise I created in here ^^

[cdhowie]

Great, it works for me too, just had to change the URL in my scripts. I think using such an API to run cronjobs against is more useful than having it crammed into the web interface.

Yup, that's why it's there.    Not all of the pages have proper JSON support implemented, simply because for most of them it wouldn't be terribly useful.  But at some point, the proxy should allow full manipulation of the database objects via JSON, allowing you to code an alternative interface (like a desktop app) against the JSON API.  That probably wouldn't be a good thing to do, but it would at least be possible.

I guess I can drop the stats branch then, since it already works out of the box from your dashboard. On the worker-hashing branch I made the average interval configurable because it strongly depends on the hashing speed (high hashing speeds can get a more accurate reading with shorter intervals, while slow miners will have the average jump wildly around for small values). I think it is ready to be pulled. Once again sorry for all the noise I created in here ^^

Alright, I'll have a look at it at some point later.  Thanks for the patch!

[molecular]

hey cdhowie,

great proxy, just checking it out.

Had problems with the php notices spamming the pages to the point where chrome refused to render them (they get inserted before DOCTYPE decl):

Example:

Code:

<br /> 
<b>Notice</b>:  Undefined variable: viewdata in <b>/var/www/localhost/htdocs/bmproxy/admin/pool.php</b> on line <b>55</b><br /> 
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> 
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> 
    <head> 
...

I simply put

Code:

error_reporting(E_ERROR);

in my config.inc.php, that fixed the issue nicely.

Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices.

[cdhowie]

Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices.

Hmm, I thought I did.  Maybe I just did that in my own php.ini or something.  Anyway, thanks for the comment.  I'll add this in a future push.

[pwnyboy]

Hmm.  This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this.  It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker.

Ahhh I see the problem.  From your proxy to deepbit:

GET /listenChannel HTTP/1.0
Authorization: Basic youcanthavemypassword;]
Host: deepbit.net:8332

I watched it send this to deepbit once and it worked, but the next time it didn't.  The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0.  Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1.  So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not.  You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1).  If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1?

Thanks!

[cdhowie]

I've just done a push.  I recommend that everyone update from master.

This update fixes some PHP warnings that would get triggered on a failover, effectively breaking failover entirely on configurations where PHP warnings are emitted.  The pool timeout has also been lowered from 5 seconds to 2 since the default configuration of some miners is to request work every 5 seconds.  These miners will break if the proxy takes too long to respond, terminating the active getwork request and issuing another.  (This timeout will be made configurable in a later push so that you can customize it to your configuration.)

An error_reporting() call has also been added to the config.inc.php.sample file.  It's strongly recommended that you copy this into your config.inc.php to mitigate any possible future issues caused by PHP warnings getting mixed into the JSON response for workers.

[cdhowie]

Ahhh I see the problem.  From your proxy to deepbit:

GET /listenChannel HTTP/1.0
Authorization: Basic youcanthavemypassword;]
Host: deepbit.net:8332

I watched it send this to deepbit once and it worked, but the next time it didn't.  The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0.  Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1.  So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not.  You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1).  If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1?

Thanks!

Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems.  My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine.

[pwnyboy]

Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems.  My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine.

I just cranked up a new kind of sniffer, one that logs each unique tuple into a separate file.  What I found is that upon receiving my input text (fed to your proxy via telnet) as follows:

GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1
Connection: close
Host: bitproxy.xxxxx.com
Authorization: Basic xxx
User-Agent: phoenix/v1.2
(and one more trailing CR)

Your proxy actually connected to deepbit twice.  The first time was apparently to setup the actual long-polling session for my telnet to the proxy.  The second communication was completely blank - neither side said anything.  After a couple of minutes, my telnet session spat back the following:

HTTP/1.1 200 OK
Date: Wed, 11 May 2011 09:59:01 GMT
Server: Apache/2.2.3 (CentOS)
X-Powered-By: PHP/5.1.6
X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy
Set-Cookie: PHPSESSID=bd8s2r4gnb80pgl6n3hd3ne606; path=/
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Content-Length: 78
Connection: close
Content-Type: application/json-rpc

{"error":"Invalid response from long-poll request.","result":null,"id":"json"}

This was not in response to anything being sent from deepbit to the proxy - deepbit never at any time uttered another word.  I ran the same testing iteration again and got a successful long poll when the pool jumped to the next block, but on the next iteration it was exactly the same behavior as above.  (note that all of this is still based on a 2-day old pull, I did not pull the code again per your mention above).

Does that help?

[pharaon]

can you give me a name of web hoster where your proxy can work ?

All i test, had 'php_auth_user' disable ......

Sorry for my poor english

[BurningToad]

Took me a while, but I got it set up! Thanks!

1 BTC heading your way

[BurningToad]

Actually this just didn't work very well for me   Set it up with BTC Guild as primary and Slush as secondary.  BTC Guild is having intermittent connection issues, so I wanted a backup.  However, overnight all of my GUIMiner clients got stuck with a "connection problem".  My one phoenix client (which is also on same machine as my proxy) continued working fine however.

[Cdecker]

Actually this just didn't work very well for me   Set it up with BTC Guild as primary and Slush as secondary.  BTC Guild is having intermittent connection issues, so I wanted a backup.  However, overnight all of my GUIMiner clients got stuck with a "connection problem".  My one phoenix client (which is also on same machine as my proxy) continued working fine however.

Sounds strange. The switch should occur either upon work request or work submission. The latter isn't very nice because a share might be lost. We are working on exponential backoff to temporarily blacklist flapping pools (intermittent connection issues). Should it occur again please provide a tcpdump (just let tcpdump run along capturing traffic from/to the pool proxy port 80, and use temporary passwords between pool proxy and the workers).

[cdhowie]

The switch should occur either upon work request or work submission.

Switches will never occur on work submission.  Submissions always route back to the pool that the work came from.