|
Title: Flexible mining proxy Post by: cdhowie on April 06, 2011, 10:26:05 PM Edit 2011-05-04: The software has been released! (View post (http://bitcointalk.org/index.php?topic=5506.msg106640#msg106640))
---------- Hey all, I'm trying to gauge interest for this. I've been hacking for about a week and a half on a mining proxy written in PHP using MySQL for the data store. I've been running my own miners against it with no problems for the last week. The basic idea is that you can run multiple miners against multiple pools (same or different, it doesn't matter) and miners can fail-over to other pools if something happens to its preferred pool. Additionally, using the web interface, you can manage pool assignments from any physical location; your miners won't notice that anything has changed when you switch them between pools. The information on the dashboard can also be used to help determine when a miner goes AWOL. Here's the more detailed list of how it all works:
All this is manageable from a web-based control panel. Right now the project is not terribly polished -- not well enough for a release anyway -- but the core seems to be working great. If there is any interest in such a project, I will probably release it under the AGPL. I'm interested in the views and perspectives of my fellow miners as to whether this project would have any value to the wider community. Mandatory screenshot: http://www.chrishowie.com/files/mining-proxy-dashboard.png Title: Re: Flexible mining proxy Post by: xenon481 on April 06, 2011, 10:32:39 PM I really like the idea of being able to do failovers, but the best part is that it is a great starting place for doing a really sophisticated implementation of Raulo's Pool Hopping Exploit! :'(
Title: Re: Flexible mining proxy Post by: cdhowie on April 06, 2011, 10:40:59 PM I really like the idea of being able to do failovers, but the best part is that it is a great starting place for doing a really sophisticated implementation of Raulo's Pool Hopping Exploit! :'( Just to be clear, I take no liability for any forks of the project. ;)Title: Re: Flexible mining proxy Post by: slush on April 06, 2011, 10:53:18 PM You did first metapool. It was just matter of the time, but still - congratz ;D
Edit: Does this solve long polling somehow? Title: Re: Flexible mining proxy Post by: jgarzik on April 06, 2011, 10:55:33 PM IMO it makes much more sense to add multi-pool support to each mining client.
That way it doesn't break long-polling, and you can more easily utilize pool-specific features as they appear (such as using BDP (http://bitcointalk.org/index.php?topic=3493.0)). A meta-pool is an additional point of failure. Title: Re: Flexible mining proxy Post by: cdhowie on April 06, 2011, 11:49:13 PM You did first metapool. It was just matter of the time, but still - congratz ;D Thanks. :)Edit: Does this solve long polling somehow? The current revision does not, but this is going to be implemented pretty soon. Obviously it will only work on pools that support LP themselves; it will just proxy the LP request.IMO it makes much more sense to add multi-pool support to each mining client. I agree. But I don't have the desire to hack on every client out there to implement something like this. Further, this approach also gives me other benefits like the ability to manage miners remotely, retarget their pool assignment without restarting them, etc.If someone implements a consistent multi-pool interface in all the mining clients, I would probably deprecate this project. But in the meantime it fills the gap and also gives you some other nifty features that client-side multi-pool support alone wouldn't. That way it doesn't break long-polling, and you can more easily utilize pool-specific features as they appear (such as using BDP (http://bitcointalk.org/index.php?topic=3493.0)). LP is only "broken" in that I have not yet implemented it. There's no technical reason it can't be done, I've just been focused on other aspects of the proxy.A similar BDP proxy could be implemented alongside the existing HTTP-based proxy, with minimal database schema changes. Once that is done, you could theoretically run HTTP-only miners against a BDP pool efficiently, by having the HTTP proxy get work from the local BDP hub process. A meta-pool is an additional point of failure. To a degree, yes. I run my proxy on my LAN, so network failure is extremely unlikely. A software bug is the only thing I can think of that would cause a problem (other than the whole box going down, at which point I lose DNS too), and with (... checking DB ...) 82,000 getwork requests processed in the last 2.5 days, my confidence in the proxy code is very high at this point.Title: Re: Flexible mining proxy Post by: cdhowie on April 12, 2011, 06:42:43 PM Long-polling proxying is now implemented. The only remaining feature on my list is connection pooling to take advantage of HTTP 1.1 keep-alive connections, but I'm not sure how feasible this is in PHP without using some external connection-pooling daemon. I might make a release before this feature is implemented.
Title: Re: Flexible mining proxy Post by: mlg.odk on April 18, 2011, 03:35:58 PM This project appears to be very interesting and is in fact exactly what I've been looking for to connect all the machines I have here to one single worker account.
Is there any release date set already? Title: Re: Flexible mining proxy Post by: cdhowie on April 19, 2011, 05:09:16 PM This project appears to be very interesting and is in fact exactly what I've been looking for to connect all the machines I have here to one single worker account. If you are using e.g. slush's pool, you should still have a separate account for each worker. My proxy allows multiple miners to authenticate to it with separate credentials, and the proxy will then authenticate to pools using credentials stored for that worker. In other words, each worker-pool assignment has its own pool credentials. Is there any release date set already? It's "as soon as I clean up the Git repo." :) I hope to get to that this week. Title: Re: Flexible mining proxy Post by: mlg.odk on April 19, 2011, 08:27:53 PM If you are using e.g. slush's pool, you should still have a separate account for each worker. My proxy allows multiple miners to authenticate to it with separate credentials, and the proxy will then authenticate to pools using credentials stored for that worker. In other words, each worker-pool assignment has its own pool credentials. Generally this is how I do it. However, let's take slush's pool and a GPU cluster which is not dedicated to generating Bitcoins and does this only when being idle as example: The amount of GPUs and machines available constantly changes. While there is a max. limit of machines, I'd still require a software to dynamically associate a worker account to each machine. This is because (here's where slush's pool comes in) slush's reward calculating formula includes the time at which the last share was submitted. Of course this is a great way to prevent cheating, however it also means that if one machine goes offline or switches to a different task, the reward it would usually have made quickly shrinks to zero. Therefore using a proxy tool like yours for such a cluster is more effective and way easier to manage and automate. It's "as soon as I clean up the Git repo." :) I hope to get to that this week. Great! I'll see to drop you some coins when it is out :) Title: Re: Flexible mining proxy Post by: cdhowie on April 20, 2011, 12:12:34 PM Generally this is how I do it. However, let's take slush's pool and a GPU cluster which is not dedicated to generating Bitcoins and does this only when being idle as example: The amount of GPUs and machines available constantly changes. While there is a max. limit of machines, I'd still require a software to dynamically associate a worker account to each machine. This is because (here's where slush's pool comes in) slush's reward calculating formula includes the time at which the last share was submitted. Of course this is a great way to prevent cheating, however it also means that if one machine goes offline or switches to a different task, the reward it would usually have made quickly shrinks to zero. Therefore using a proxy tool like yours for such a cluster is more effective and way easier to manage and automate. The way I understand the math, all of your recently-submitted shares are considered, no matter which worker they were submitted through -- so running several workers on one worker account or several should make no difference to the amount of reward you get when the block is solved. Even if one machine drops out... since the shares are considered collectively, it doesn't matter which account they were submitted through. They will age just like every other share you submit. (Correct me if I'm wrong, slush.) In other words, the older shares you submitted from the still-active machines have aged and become worthless too, you just can't tell because they are submitting enough new shares to keep the reward up. Now, having said that, my proxy doesn't "dynamically associate a worker account to each machine." You still need to set up worker accounts in my proxy script. The difference is that you can assign those worker accounts to more than one pool. (Although you could probably hack it to do what you want. ;)) Title: Re: Flexible mining proxy Post by: mlg.odk on April 20, 2011, 01:40:25 PM The way I understand the math, all of your recently-submitted shares are considered, no matter which worker they were submitted through -- so running several workers on one worker account or several should make no difference to the amount of reward you get when the block is solved. Even if one machine drops out... since the shares are considered collectively, it doesn't matter which account they were submitted through. They will age just like every other share you submit. (Correct me if I'm wrong, slush.) In other words, the older shares you submitted from the still-active machines have aged and become worthless too, you just can't tell because they are submitting enough new shares to keep the reward up. This is interesting, in that case there still is the clear advantage of being able to quickly switch to a different pool in case that the current one goes offline for whatever reason. Now, having said that, my proxy doesn't "dynamically associate a worker account to each machine." You still need to set up worker accounts in my proxy script. The difference is that you can assign those worker accounts to more than one pool. (Although you could probably hack it to do what you want. ;)) Ah I think you misunderstood me there, sorry if I was not clear enough. I ment to say that the alternative to what I said was your software; I understood the purpose of your software correctly. Title: Re: Flexible mining proxy Post by: pwnyboy on April 23, 2011, 10:36:39 PM What is the status of this? I'd like to start testing and maybe contributing code if you're ready to release it into the wild.
Title: Re: Flexible mining proxy Post by: BitLex on April 23, 2011, 11:30:33 PM i would also like to test this, so please keep us up to date. ;)
Title: Re: Flexible mining proxy Post by: anisoptera on May 04, 2011, 06:39:21 PM +1 for wanting to test this please :)
Title: Re: Flexible mining proxy Post by: pwnyboy on May 04, 2011, 08:10:12 PM After recent strangeness with Deepbit, I'm putting a 10 BTC bounty on this, contingent upon ultimate release of the working product and sourcecode before I'm forced to write my own. Looking forward to getting up and running.
Title: Re: Flexible mining proxy Post by: cdhowie on May 04, 2011, 11:04:44 PM Alright, I've cleaned up the repo and finished adding the final touches (for now...). Grab the code: https://github.com/cdhowie/Bitcoin-mining-proxy
Setup directions are in the INSTALL file. Please read the whole thing before asking questions; I took a long time writing it and proofread it several times to make sure it's complete and correct. Let me know if anything is unclear or broken. And please feel free to use GitHub's issue tracking to report issues. :) If you like the software and find it useful, there is a GPG-signed Bitcoin address on the about page where you can send donations. Any amount would be appreciated. For those interested in reliability information, I've been running one miner against it consistently for a full month, and a few other miners against it sporadically. The getwork proxy has always worked 100% for me in my deployed copy, except during power failures. ;) @pwnyboy: Thanks for offering the bounty! I was already getting it ready for release when you posted, so you don't have to send the bounty if you don't want (since it really wasn't responsible for motivating me; I just finally got off my lazy butt since everyone's been waiting for it). But I won't turn it down either. ;) If you still want to send, feel free to use the donation address. Title: Re: Flexible mining proxy Post by: pwnyboy on May 06, 2011, 04:32:47 AM Thanks for the release! I'll dedicate some time to pulling and installing it tomorrow, and pass that money along once it's up. After all, a promise is a promise :) Thanks again.
Title: Re: Flexible mining proxy Post by: cdhowie on May 06, 2011, 04:36:29 AM Thanks for the release! I'll dedicate some time to pulling and installing it tomorrow, and pass that money along once it's up. After all, a promise is a promise :) Thanks again. Cool, thanks! Hope the software works for you. If you run into any trouble, let me know. I intend to support the software I publish. (That goes for everyone.) Title: Re: Flexible mining proxy Post by: aahzmundus on May 06, 2011, 04:40:52 AM If you run into any trouble, let me know. I intend to support the software I publish. (That goes for everyone.) He is serious about this, worked with me on and off for a good hour helping me figure stuff out step by step. Already made a small donation, well worth it. Title: Re: Flexible mining proxy Post by: BitLex on May 06, 2011, 04:52:28 AM i only get an xml-parsing error when trying to access /admin/
Code: XML Parsing Error: XML or text declaration not at start of entity guess it's just a blank line hiding somewhere, but i can't find it. Title: Re: Flexible mining proxy Post by: pwnyboy on May 06, 2011, 05:38:03 AM If you run into any trouble, let me know. I intend to support the software I publish. (That goes for everyone.) He is serious about this, worked with me on and off for a good hour helping me figure stuff out step by step. Already made a small donation, well worth it. Sweet! Maybe you can advise on the following. I broke down and decided to stay up past my bedtime to pull your code and put it online. I took your advice and put this on a separate vhost in apache, because my mining client (phoenix) would not hit http://domain.com/some/big/url as you pointed out. I threw this on a CentOS box that already has working php/mysql sites on it, so any dependencies other than those two are hopefully met already. When I connect Phoenix, it tells me 'failed to connect, retrying'. When I examine my apache error log, I see this: [Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Notice: Undefined index: lpurl in /home/rrb/bitproxy/index.php on line 70 [Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Notice: Undefined variable: http_response_header in /home/rrb/bitproxy/common.inc.php on line 117 [Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Warning: Invalid argument supplied for foreach() in /home/rrb/bitproxy/index.php on line 215 Additionally, you might find this of interest. When I hit the admin url, the admin interface seems to work fine but merely hitting /admin/ throws the following into the apache error log: [Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: format in /home/rrb/bitproxy/mvc.inc.php on line 38 [Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: tempdata in /home/rrb/bitproxy/common.inc.php on line 177 [Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: tempdata in /home/rrb/bitproxy/common.inc.php on line 177 (IPs redacted). I'm pretty sure I followed your instructions to the letter, but have not yet taken a look at the code to try and debug this (maybe tomorrow). Any ideas? Title: Re: Flexible mining proxy Post by: pwnyboy on May 06, 2011, 07:32:12 AM So insomnia set in, and I decided to dig deeper. I found a couple more things:
1) In your database schema, in the worker_pool table, the type for the 'enabled' field is a tinyint and should be boolean it looks like, unless you have some special state info to store that's not 0 or 1? 2) I discovered #1 while attempting to go back and edit the workers and pool info that I'd setup the first time, in an effort to debug what was happening with the behavior I reported above. While using the various "edit" dialogs, I'm hitting blank pages being returned, i.e. in the "edit pool" dialog at http://xxx/admin/pool.php?id=1&action=edit - it returns nothing in View Source as well. Having said that, and after editing the databases by hand (because it looks like I had the port number to the pool AND my login credentials entered incorrectly), I got further with the miner now (I'm guessing before it would not connect because the credentials at the pool were incorrect. Here's what happens now: [06/05/2011 03:18:17] Connected to server [06/05/2011 03:18:23] Disconnected from server [06/05/2011 03:18:23] Result: 5b433c62 rejected [06/05/2011 03:18:24] Connected to server [06/05/2011 03:18:26] Disconnected from server [06/05/2011 03:18:26] Result: 9e17d0db rejected [06/05/2011 03:18:27] Connected to server [06/05/2011 03:18:31] Disconnected from server My first instinct was to turn on keepalives in the webserver, I did so and set them to allow an unlimited number with a 30 second timeout, but it only prolonged the generation of the same errors by 10 or 20 seconds. In the end it still disconnected my client from the server. I tried m0mchil's miner with similar results: 06/05/2011 03:25:22, Unexpected error: Traceback (most recent call last): File "BitcoinMiner.pyo", line 141, in mine File "BitcoinMiner.pyo", line 168, in sendResult File "BitcoinMiner.pyo", line 177, in getwork File "BitcoinMiner.pyo", line 204, in request TypeError: string indices must be integers 248779 khash/s This after running 10 to 20 seconds also. Not seeing anything in the logs that would give me any indication of the problem. Without cranking up a sniffer, I'm out of ideas; any suggestions? Title: Re: Flexible mining proxy Post by: cdhowie on May 06, 2011, 11:25:36 AM When I connect Phoenix, it tells me 'failed to connect, retrying'. When I examine my apache error log, I see this: This notice is expected. PHP likes to whine if you access array keys that aren't defined, but it's impractical to always check if they're set first. :)[Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Notice: Undefined index: lpurl in /home/rrb/bitproxy/index.php on line 70 [Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Notice: Undefined variable: http_response_header in /home/rrb/bitproxy/common.inc.php on line 117 This is a bit troubling, as $http_response_header is a reserved variable (http://php.net/manual/en/reserved.variables.httpresponseheader.php). Which version of PHP are you using? Did you make sure to enter "http://" at the beginning of each pool URL? (Something like "deepbit.net:8332" will not work; you need "http://deepbit.net:8332".) You may also get this error if the HTTP host can't be reached at all, for example due to a DNS lookup failure.[Fri May 06 01:30:05 2011] [error] [client xxx.113] PHP Warning: Invalid argument supplied for foreach() in /home/rrb/bitproxy/index.php on line 215 This is related to the previous issue; fixing it should make this one go away.Additionally, you might find this of interest. When I hit the admin url, the admin interface seems to work fine but merely hitting /admin/ throws the following into the apache error log: Yes, this is PHP being picky again. You can ignore these. :)[Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: format in /home/rrb/bitproxy/mvc.inc.php on line 38 [Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: tempdata in /home/rrb/bitproxy/common.inc.php on line 177 [Fri May 06 01:31:15 2011] [error] [client xxx.252] PHP Notice: Undefined index: tempdata in /home/rrb/bitproxy/common.inc.php on line 177 1) In your database schema, in the worker_pool table, the type for the 'enabled' field is a tinyint and should be boolean it looks like, unless you have some special state info to store that's not 0 or 1? MySQL does not have a real boolean type (http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html#id853095); BOOLEAN is an alias for TINYINT(1).2) I discovered #1 while attempting to go back and edit the workers and pool info that I'd setup the first time, in an effort to debug what was happening with the behavior I reported above. While using the various "edit" dialogs, I'm hitting blank pages being returned, i.e. in the "edit pool" dialog at http://xxx/admin/pool.php?id=1&action=edit - it returns nothing in View Source as well. Hmm, this is almost certainly a bug. Can you edit other pools, or is it just this one? Does anything show up in the Apache logs when you hit this page?Having said that, and after editing the databases by hand (because it looks like I had the port number to the pool AND my login credentials entered incorrectly), I got further with the miner now (I'm guessing before it would not connect because the credentials at the pool were incorrect. Here's what happens now: This is definitely weird. I use poclbm without any issues, so you should be able to use it too. I'm mostly worried about the rejected shares in the first example. Were all shares rejected or just those two?[06/05/2011 03:18:17] Connected to server [06/05/2011 03:18:23] Disconnected from server [06/05/2011 03:18:23] Result: 5b433c62 rejected [06/05/2011 03:18:24] Connected to server [06/05/2011 03:18:26] Disconnected from server [06/05/2011 03:18:26] Result: 9e17d0db rejected [06/05/2011 03:18:27] Connected to server [06/05/2011 03:18:31] Disconnected from server My first instinct was to turn on keepalives in the webserver, I did so and set them to allow an unlimited number with a 30 second timeout, but it only prolonged the generation of the same errors by 10 or 20 seconds. In the end it still disconnected my client from the server. I tried m0mchil's miner with similar results: 06/05/2011 03:25:22, Unexpected error: Traceback (most recent call last): File "BitcoinMiner.pyo", line 141, in mine File "BitcoinMiner.pyo", line 168, in sendResult File "BitcoinMiner.pyo", line 177, in getwork File "BitcoinMiner.pyo", line 204, in request TypeError: string indices must be integers 248779 khash/s This after running 10 to 20 seconds also. Not seeing anything in the logs that would give me any indication of the problem. Without cranking up a sniffer, I'm out of ideas; any suggestions? Using a sniffer might be a good idea, since it's hard to tell just from these logs exactly what's going wrong. Title: Re: Flexible mining proxy Post by: cdhowie on May 06, 2011, 11:30:30 AM i only get an xml-parsing error when trying to access /admin/ Yeah, there's definitely a blank line being output prior to the XML header. I don't see the same behavior on my install, in fact the dashboard validates as XHTML strict (as should every other page, but I just checked the dashboard right now and it validates). I can't find anywhere in the code where the blank line would be getting written.Code: XML Parsing Error: XML or text declaration not at start of entity guess it's just a blank line hiding somewhere, but i can't find it. Title: Re: Flexible mining proxy Post by: pwnyboy on May 06, 2011, 11:33:50 PM Thanks for the replies. I should start by saying that I think those errors I was seeing in the errorlog were related to my incorrect entering of the parameters for deepbit (username/password, and the port number). Although I did have it entered in the correct format as you mentioned. In a future version you might want to do some more extensive error handling to let the user know in the admin panel that the "last result" from the pool is that it can't connect because of x or y, and the actual error sent back from the pool was z.
I sniffed out the communication going from Phoenix to the proxy only, and this is what it looks like (including timestamps above): 1) Initial connection: 19:14:20.136363 POST / HTTP/1.1 Connection: close Content-Length: 44 Host: bitproxy.xxx.com Content-Type: application/json Authorization: Basic xxx User-Agent: phoenix/v1.2 {"method": "getwork", "params": [], "id": 1} 2) One more (not sure why this was necessary): 19:14:20.693905 POST / HTTP/1.1 Connection: close Content-Length: 44 Host: bitproxy.xxx.com Content-Type: application/json Authorization: Basic xxx User-Agent: phoenix/v1.2 {"method": "getwork", "params": [], "id": 1} 3) Another for long polling?: 19:14:20.717374 GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1 Connection: close Host: bitproxy.xxx.com Authorization: Basic xxx User-Agent: phoenix/v1.2 4) And one more.. not sure why this was necessary, the client didn't say it had seen a new block: 19:14:21.213894 POST / HTTP/1.1 Connection: close Content-Length: 302 Host: bitproxy.xxx.com Content-Type: application/json Authorization: Basic xxx User-Agent: phoenix/v1.2 {"method": "getwork", "params": [], "id": 1} 5) Ahh now we're getting somewhere....14 seconds later when we have some work done: 19:14:35.187166 POST / HTTP/1.1 Connection: close Content-Length: 302 Host: bitproxy.xxx.com Content-Type: application/json Authorization: Basic xxx User-Agent: phoenix/v1.2 {"method": "getwork", "params": ["00000001856410222014682e1d4361839ddae0adf1f7415584a4e6d8000006d20000000086ed4af 00f553972b297b1cdbd24d6b99e4720df26ff41229dbd2384c1d0cab44dc480c81b0098fa35cafe bd00000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000"], "id": 1} At this point, the client disconnects from the proxy and throws the following in the client: [06/05/2011 19:15:02] Disconnected from server [06/05/2011 19:15:02] Result: a8b4a195 rejected Then immediately reconnects to the server: [06/05/2011 19:15:03] Connected to server And sends the same message as in 1) above. I should also note that the apache logs are consistent with the above and I do not see any signs of an error related to these http transactions in the error log. Hope that helps. Title: Re: Flexible mining proxy Post by: cdhowie on May 07, 2011, 01:08:36 AM In a future version you might want to do some more extensive error handling to let the user know in the admin panel that the "last result" from the pool is that it can't connect because of x or y, and the actual error sent back from the pool was z. Yes, better validation and logging are on the cards. I was more concerned with getting the code ready for a proper release than adding features, so this should be coming pretty soon.I sniffed out the communication going from Phoenix to the proxy only, and this is what it looks like (including timestamps above): Can you post some more logs, but include the replies from the proxy? It's hard to diagnose problems when I can only see one half of the conversation.Authorization: Basic xxx If I were you I would sanitize these headers too; your username and password are exposed to anyone who understands HTTP basic authentication.3) Another for long polling?: Yes, this is a long polling request.4) And one more.. not sure why this was necessary, the client didn't say it had seen a new block: It's possible that your client is pre-fetching the next chunk of work so that when it's done with the current chunk it can move on to the next right away.5) Ahh now we're getting somewhere....14 seconds later when we have some work done: This is where response logs from the proxy would be most helpful, since I can't really tell why it's failing without knowing what the proxy is saying in response.... At this point, the client disconnects from the proxy and throws the following in the client: [06/05/2011 19:15:02] Disconnected from server [06/05/2011 19:15:02] Result: a8b4a195 rejected Title: Re: Flexible mining proxy Post by: pwnyboy on May 07, 2011, 02:05:27 AM Thanks for the reply, I've sanitized things further in the last post, even though the worker password is of no consequence for me. Here is the work being sent and the corresponding error:
POST / HTTP/1.1 Connection: close Content-Length: 302 Host: bitproxy.xxx.com Content-Type: application/json Authorization: Basic xxx User-Agent: phoenix/v1.2 {"method": "getwork", "params": ["0000000129d09b9af4d37ae0641aeaf3618fb62692477035e4f4a17300009697000000004a84ffa 3bb5f69b95dd4656b1267f99f0675fca59a2899b31a07f3eb31992d8e4dc4a7561b0098fa7958ce a500000000000000000000000000000000000000000000000000000000000000000000000000000 0000000000000000000"], "id": 1} Date: Sat, 07 May 2011 01:59:14 GMT Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.1.6 X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy Set-Cookie: PHPSESSID=6r9lc3ejja26ok9fv5ve84dc44; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 66 Connection: close Content-Type: application/json-rpc {"error":"Work not found in proxy database.","result":null,"id":1} Title: Re: Flexible mining proxy Post by: cdhowie on May 07, 2011, 02:53:36 AM Here is the work being sent and the corresponding error: Hmm. Can you post the replies to the previous getworks too? Somehow the work either isn't getting saved in the database correctly, or the worker isn't sending the work back correctly, or something. I'd have to see the getwork responses and the work submission requests to diagnose this further.Title: Re: Flexible mining proxy Post by: pwnyboy on May 07, 2011, 06:33:15 AM Here is a complete exchange including both sides from tcpdump. The IPs were masked with "xxx" on the first 3 octets as you'll see:
Code: # tcpdump -n -s 0 -A dst port 80 or src port 80 > /tmp/dump.txt Title: Re: Flexible mining proxy Post by: cdhowie on May 07, 2011, 05:44:41 PM Here is a complete exchange including both sides from tcpdump. The IPs were masked with "xxx" on the first 3 octets as you'll see: Perfect, that's exactly the data I needed.The error you are getting back from the work submission should indeed not be happening; the submitted work does match one of the getwork requests. Can you execute this query against your database and see if it gets any results? If it doesn't, can you verify that your work_data table has any records in it at all? Code: SELECT * FROM `work_data` WHERE `data` = '000000015bb9a0f983558324beead35e719c4e1e4f3e3d0a8f6550bb00003a5e0000000035cfac9a5865f1e5500714aa46e327a3727fd5a481f0b708820f794f6e0dada84dc4e4f11b0098fa'; Edit: I just checked the code and if the INSERT into the work_data table fails, it will do so silently, still returning the work to the client. I need to patch this, since the client shouldn't be given work that can't be registered, or the proxy won't know which pool to route it too (I think this is the problem you are seeing). We still need to figure out why the INSERT is failing though. Edit 2: I think I found the problem with work submissions. Please update to the latest code from Git. Title: Re: Flexible mining proxy Post by: pwnyboy on May 07, 2011, 11:44:26 PM My work_data table had 192 entries. I cloned the tree again and replaced the contents of htdocs, copying over the same config file, and the error is still the same on the client side. Let me know if there is anything else I can do to help troubleshoot.
Title: Re: Flexible mining proxy Post by: cdhowie on May 07, 2011, 11:47:56 PM My work_data table had 192 entries. I cloned the tree again and replaced the contents of htdocs, copying over the same config file, and the error is still the same on the client side. Let me know if there is anything else I can do to help troubleshoot. Hmm. Can you run the query I gave and see what it returns? Also, you shouldn't need to clone anything again -- just git pull from inside the git root.Title: Re: Flexible mining proxy Post by: pwnyboy on May 08, 2011, 12:21:12 AM Yes, it's in there.
mysql> SELECT * FROM `work_data` WHERE `data` = '000000015bb9a0f983558324beead35e719c4e1e4f3e3d0a8f6550bb00003a5e0000000035cfac9 a5865f1e5500714aa46e327a3727fd5a481f0b708820f794f6e0dada84dc4e4f11b0098fa'; +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | worker_id | pool_id | data | time_requested | +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | 1 | 1 | 000000015bb9a0f983558324beead35e719c4e1e4f3e3d0a8f6550bb00003a5e0000000035cfac9 a5865f1e5500714aa46e327a3727fd5a481f0b708820f794f6e0dada84dc4e4f11b0098fa | 2011-05-07 06:21:42 | +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ 1 row in set (0.00 sec) mysql> Title: Re: Flexible mining proxy Post by: cdhowie on May 08, 2011, 04:25:54 AM Yes, it's in there. Hmm, weird. Can you put the contents of this script in a PHP file alongside index.php (in other words, directly in the web root) and paste me its output? Of course, sanitize the username and password if they get displayed.Code: <?phpTitle: Re: Flexible mining proxy Post by: pwnyboy on May 08, 2011, 07:10:59 AM That script returns "No rows found." Preemptively I'm going to send you the contents of tables "worker", "worker_pool" and "pool" with passwords removed, since it looks like that's the information you're looking for with the aforementioned php script:
mysql> select * from worker; +----+-------+-----------+ | id | name | password | +----+-------+-----------+ | 1 | w0113 | xxxxx | +----+-------+-----------+ 1 row in set (0.00 sec) mysql> select * from worker_pool -> ; +---------+-----------+------------------------+---------------+----------+---------+ | pool_id | worker_id | pool_username | pool_password | priority | enabled | +---------+-----------+------------------------+---------------+----------+---------+ | 1 | 1 | xxxxxxxxx@xxxxx.com_10 | xxxxxxxxxxx | 1 | 1 | +---------+-----------+------------------------+---------------+----------+---------+ 1 row in set (0.00 sec) mysql> select * from pool; +----+---------+-------------------------+---------+ | id | name | url | enabled | +----+---------+-------------------------+---------+ | 1 | deepbit | http://deepbit.net:8332 | 1 | +----+---------+-------------------------+---------+ 1 row in set (0.00 sec) Based on my lay knowledge of php, I can't imagine why it doesn't return anything from the script you sent. Is it possible that my installation (default php 5.1 from CentOS 5.x with php-* installed, mysql-* installed, and the add-in json libraries) is different, misconfigured, or missing a dependency? For reference, it already works fine with other popular MySQL/PHP off the shelf applications. Also sent you 5 BTC to the address in your signature for all the work you're putting in; this is in addition to my previous commitment to send you 10 when it appears that this works and is viable. Thanks again for your help. Title: Re: Flexible mining proxy Post by: pwnyboy on May 08, 2011, 07:20:13 AM One more that I meant to send:
mysql> select * from work_data where data = '000000015bb9a0f983558324beead35e719c4e1e4f3e3d0a8f6550bb00003a5e0000000035cfac9 a5865f1e5500714aa46e327a3727fd5a481f0b708820f794f6e0dada84dc4e4f11b0098fa'; +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | worker_id | pool_id | data | time_requested | +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ | 1 | 1 | 000000015bb9a0f983558324beead35e719c4e1e4f3e3d0a8f6550bb00003a5e0000000035cfac9 a5865f1e5500714aa46e327a3727fd5a481f0b708820f794f6e0dada84dc4e4f11b0098fa | 2011-05-07 06:21:42 | +-----------+---------+----------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------+ 1 row in set (0.00 sec) mysql> Title: Re: Flexible mining proxy Post by: cdhowie on May 08, 2011, 01:50:32 PM That script returns "No rows found." #$%&. Me either... though I do have a few ideas....... Based on my lay knowledge of php, I can't imagine why it doesn't return anything from the script you sent. Is it possible that my installation (default php 5.1 from CentOS 5.x with php-* installed, mysql-* installed, and the add-in json libraries) is different, misconfigured, or missing a dependency? For reference, it already works fine with other popular MySQL/PHP off the shelf applications. It should work fine. I'm on PHP 5.3.3, but all my code should be compatible with most PHP5 versions (and probably some PHP4 versions too).To further diagnose this, can you try this script now? Code: <?phpAlso sent you 5 BTC to the address in your signature for all the work you're putting in; this is in addition to my previous commitment to send you 10 when it appears that this works and is viable. Thanks again for your help. Cool, thanks! Hopefully we can get this working soon. Obscure bugs are annoying. :(Title: Re: Flexible mining proxy Post by: Cdecker on May 08, 2011, 05:18:39 PM I love this one. It make switching really painless, especially after todays outage of slush's pool this proved a timesaver. What I noticed however that the failing pool will still be asked for work, and only he doesn't return any, the next one will be asked. Are there plans for exponential backoff in order to avoid the extra request and the extra idle time? Just a mechanism to automatically blacklist a pool should it fail, and increase the blacklist time on each subsequent failure would be cool.
The problem today is that slush sometimes gives me work, but does not accept the result (timeout/502 errors/...). Also I get a lot of these: Quote 192.168.3.5 - pilum [08/May/2011:19:21:43 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" Which makes me think that the diablo miner does not really take advantage of the long polling feature, and I start feeling bad for bombing pools with all those long poll requests. Any ideas?192.168.3.5 - pilum [08/May/2011:19:21:45 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:47 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:49 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:49 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" Title: Re: Flexible mining proxy Post by: pwnyboy on May 08, 2011, 07:17:20 PM To further diagnose this, can you try this script now? This is the result, sanitized to remove portions of my email/password at deepbit: original: No rows found hardcoded: Success (array ( 'pool_id' => '1', 0 => '1', 'username' => 'xxx@xxx.com_10', 1 => 'xxx@xxx.com_10', 'password' => 'xxx', 2 => 'xxx', 'url' => 'http://deepbit.net:8332', 3 => 'http://deepbit.net:8332', )) nodup: Success (array ( 'pool_id' => '1', 0 => '1', 'username' => 'xxx@xxx.com_10', 1 => 'xxx@xxx.com_10', 'password' => 'xxx', 2 => 'xxx', 'url' => 'http://deepbit.net:8332', 3 => 'http://deepbit.net:8332', )) no underscore: No rows found Title: Re: Flexible mining proxy Post by: cdhowie on May 08, 2011, 07:54:12 PM This is the result, sanitized to remove portions of my email/password at deepbit: Thanks, that's all I need to know to fix the problem. Apparently the older PDO library doesn't support using the same query argument twice in a query. A fix is forthcoming.Title: Re: Flexible mining proxy Post by: cdhowie on May 08, 2011, 08:04:18 PM This is the result, sanitized to remove portions of my email/password at deepbit: Thanks, that's all I need to know to fix the problem. Apparently the older PDO library doesn't support using the same query argument twice in a query. A fix is forthcoming.Title: Re: Flexible mining proxy Post by: cdhowie on May 08, 2011, 08:15:49 PM I love this one. It make switching really painless, especially after todays outage of slush's pool this proved a timesaver. I'm glad you like it. :)What I noticed however that the failing pool will still be asked for work, and only he doesn't return any, the next one will be asked. Are there plans for exponential backoff in order to avoid the extra request and the extra idle time? Just a mechanism to automatically blacklist a pool should it fail, and increase the blacklist time on each subsequent failure would be cool. This is a good idea. I've added it (https://github.com/cdhowie/Bitcoin-mining-proxy/issues/3) to the issue tracker.The problem today is that slush sometimes gives me work, but does not accept the result (timeout/502 errors/...). Hmm, I see. I'll try to implement something so that if a pool doesn't respond normally to a share submission (e.g. it doesn't accept or reject the share, but gives some other HTTP or JSON-level error) the software will retry a few times, and disabled the pool for some time if it still can't reach the server. This will depend on the other feature being implemented first. (This one has also been added (https://github.com/cdhowie/Bitcoin-mining-proxy/issues/4) to the issue tracker.)Also I get a lot of these: Hmm, that is definitely weird. Can you sniff those requests and paste the HTTP conversation (sanitizing authentication details) so that I can see what's going on?Quote 192.168.3.5 - pilum [08/May/2011:19:21:43 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" Which makes me think that the diablo miner does not really take advantage of the long polling feature, and I start feeling bad for bombing pools with all those long poll requests. Any ideas?192.168.3.5 - pilum [08/May/2011:19:21:45 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:47 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:49 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" 192.168.3.5 - pilum [08/May/2011:19:21:49 +0200] "POST /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=2 HTTP/1.1" 200 1181 "-" "Java/1.6.0_22" Title: Re: Flexible mining proxy Post by: pwnyboy on May 09, 2011, 12:32:27 AM Quote from: cdhowie Fix just got pushed to the Git repository; please pull and see if the problem goes away. Thanks, that gets connected and accepting work now, but after maybe 3 minutes of mining, I see a random disconnect: [08/05/2011 20:27:46] Connected to server [08/05/2011 20:28:30] Result: 9f0a0a6e accepted [08/05/2011 20:28:31] Result: 6731f6a4 accepted [08/05/2011 20:28:55] Result: a0527e09 accepted [08/05/2011 20:28:59] Result: 196f2c04 accepted [08/05/2011 20:29:17] Result: 65173661 accepted [08/05/2011 20:29:21] Result: 564ba46a accepted [08/05/2011 20:29:47] Disconnected from server [08/05/2011 20:29:48] Connected to server [08/05/2011 20:30:05] Result: 40071f20 accepted [08/05/2011 20:30:24] Result: a6206ab4 accepted [08/05/2011 20:30:33] Result: 73d3021d accepted [08/05/2011 20:30:34] Disconnected from server [307.16 Mhash/sec] [9 Accepted] [0 Rejected] [RPC (+LP)] Let me know how I can help diagnose this. Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 03:03:54 AM Thanks, that gets connected and accepting work now, but after maybe 3 minutes of mining, I see a random disconnect: The long-polling request is probably getting terminated. This should only happen if you have PHP's safe mode enabled -- set_time_limit() has no effect when running under safe mode, so the most likely cause here is that PHP is simply killing the script doing the long-polling request because it is running for too long. (Note that the "disconnection" message comes almost exactly two minutes after the "connection" message.)Try disabling safe mode. Safe mode can only be disabled in php.ini or your Apache configuration; .htaccess files can not disable safe mode. Title: Re: Flexible mining proxy Post by: pwnyboy on May 09, 2011, 03:27:18 AM Yes it's already disabled in php.ini:
; ; Safe Mode ; safe_mode = Off Though I think it would be a good idea to go back and mention as dependencies (in the INSTALL file) that safe_mode should be disabled, and that this (quite obviously) requires php json support, which is an add-on for PHP versions prior to 5.3. Anyway this could very well have something to do with long polling. I tried setting my max execution time (php.ini) to 1500 seconds, up from 30 seconds. It did the following: [08/05/2011 23:21:44] Connected to server [08/05/2011 23:21:49] Result: be0189d3 accepted [08/05/2011 23:21:54] Result: 62cffe98 accepted [08/05/2011 23:21:58] Result: 65b66b0e accepted [08/05/2011 23:22:31] Result: 2ca96088 accepted [08/05/2011 23:22:38] Result: 8bd7c27e accepted [08/05/2011 23:22:39] LP: New work pushed [08/05/2011 23:22:47] Result: c662da7b accepted [08/05/2011 23:23:20] Result: b9b3a17f accepted [08/05/2011 23:23:30] Result: 889e14f2 accepted [08/05/2011 23:23:31] Result: 1b237f90 accepted [08/05/2011 23:23:38] Result: 9da72831 accepted [08/05/2011 23:24:33] Result: ca84878c accepted [08/05/2011 23:24:40] Disconnected from server [08/05/2011 23:24:41] Connected to server [08/05/2011 23:24:43] Result: 1b1964ce accepted [08/05/2011 23:24:57] Result: 6b5df998 accepted [08/05/2011 23:25:41] Result: 86510be4 accepted Any other ideas? Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 04:04:30 AM Anyway this could very well have something to do with long polling. I tried setting my max execution time (php.ini) to 1500 seconds, up from 30 seconds. It did the following: Hmm... run a packet sniffer and see what's going on with the LP request?.... Any other ideas? Title: Re: Flexible mining proxy Post by: Cdecker on May 09, 2011, 11:54:33 AM Hmm, that is definitely weird. Can you sniff those requests and paste the HTTP conversation (sanitizing authentication details) so that I can see what's going on? Will do ^^Title: Re: Flexible mining proxy Post by: Cdecker on May 09, 2011, 12:32:44 PM Will do ^^ I should have seen it immediately: Diablo uses POST request while well behaving clients (phoenix, poclbm, ...) use GET requests. Should be a simple matter of routing POST to the same handler :)In the meantime I'm working on a hashrate estimate for the workers, which I'd find quite usefull. Do you accept patches/pull requests? Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 02:02:22 PM I should have seen it immediately: Diablo uses POST request while well behaving clients (phoenix, poclbm, ...) use GET requests. Should be a simple matter of routing POST to the same handler :) Ah, good catch. I should have followed the "be lenient in what you accept" rule when coding...In the meantime I'm working on a hashrate estimate for the workers, which I'd find quite usefull. Do you accept patches/pull requests? Absolutely, as long as you release your patches under the AGPLv3. Have you patched it to accept POST for long-polling requests, or should I fix that?Title: Re: Flexible mining proxy Post by: Cdecker on May 09, 2011, 02:16:08 PM I'm not really familiar with the framework you're using, so I started by implementing the hash speed for the workers. As for the routing of the post request, I guess you'd be faster than me :-)
Title: Re: Flexible mining proxy Post by: Cdecker on May 09, 2011, 02:35:10 PM Just issued a pull request for the hashing speed and shares in the last hour on the dashboard for each worker. Hope you like it ^^
Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 02:39:21 PM I'm not really familiar with the framework you're using, so I started by implementing the hash speed for the workers. As for the routing of the post request, I guess you'd be faster than me :-) It's a custom lightweight MVC framework that doesn't do terribly fancy routing, but still allows me to separate my code better. :)I've pushed a fix to the diablo-lp-fix branch. Can you try that and see if it fixes the issue for you? Just issued a pull request for the hashing speed and shares in the last hour on the dashboard for each worker. Hope you like it ^^ Thanks! I'll have a look later today.Title: Re: Flexible mining proxy Post by: Cdecker on May 09, 2011, 02:50:48 PM Works like a charm, and it fixed my 100% problem with Diablo too ^^
Title: Re: Flexible mining proxy Post by: pharaon on May 09, 2011, 03:14:44 PM How to set up a "no admin ask password" to test it
Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 03:22:20 PM Works like a charm, and it fixed my 100% problem with Diablo too ^^ Neat, I'll merge the fix into master.Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 03:22:32 PM How to set up a "no admin ask password" to test it I'm not sure what you mean. Can you elaborate?Title: Re: Flexible mining proxy Post by: pharaon on May 09, 2011, 03:39:24 PM How to set up a "no admin ask password" to test it I'm not sure what you mean. Can you elaborate?yes sorry.... In your Install file you say: 2. Create a MySQL user for the proxy, giving it the privileges SELECT, INSERT, UPDATE, DELETE, and LOCK TABLES on the proxy's database only. (Optional, but recommended.) Is it the same user than: 3. Set the admin user and password. This will be used when logging in to the web management console. and an other question is, when i connect on the "/" of my proxy, it ask me a login and password. Is it the same than in the admin section ? sorry for my bad english Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 03:59:08 PM In your Install file you say: No. The first is a user in the MySQL database; all of the workers as well as the admin account will cause the proxy to authenticate to the database server with these credentials. The admin user is a separate set of credentials that you use to log in to the proxy management console. You can pick whatever you want for the admin credentials, but the MySQL credentials must match a MySQL user.2. Create a MySQL user for the proxy, giving it the privileges SELECT, INSERT, UPDATE, DELETE, and LOCK TABLES on the proxy's database only. (Optional, but recommended.) Is it the same user than: 3. Set the admin user and password. This will be used when logging in to the web management console. and an other question is, when i connect on the "/" of my proxy, it ask me a login and password. No. When you use the "/admin" URI, it will ask for admin credentials. When you use the "/" URI it will ask for worker credentials.Is it the same than in the admin section ? Title: Re: Flexible mining proxy Post by: pwnyboy on May 09, 2011, 05:50:16 PM Ok further to my issue, I captured one of those sessions per below:
Code: 13:43:46.455793 IP xxx.113.52046 > xxx.170.http: S 744407596:744407596(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> This is mid-stream, i.e. I let the miner run for maybe 2 minutes, then turned on the sniffer when I thought it would throw the error, so it sent in maybe 2 or 3 successful shares, then the miner threw a disconnection error per below: [09/05/2011 13:43:22] Result: f780500d accepted [09/05/2011 13:43:53] Result: 98a41184 accepted [09/05/2011 13:44:06] Disconnected from server [09/05/2011 13:44:07] Connected to server The capture looks completely normal to me, based on my (very lay) knowledge of the protocol. Title: Re: Flexible mining proxy Post by: cdhowie on May 09, 2011, 05:56:57 PM Ok further to my issue, I captured one of those sessions per below: If I'm reading this right, the miner is actually closing the long-polling connection of its own accord, mid-stream. What miner are you using? Perhaps I can reproduce the issue locally.Title: Re: Flexible mining proxy Post by: pwnyboy on May 09, 2011, 11:34:38 PM Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing. I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.
Two additional feature requests: 1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone? I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have. 2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far). Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script? Thanks again for all of your hard work! Title: Re: Flexible mining proxy Post by: cdhowie on May 10, 2011, 03:19:09 AM Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing. I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes. Cool, keep me posted on that.1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone? I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have. Sure (https://github.com/cdhowie/Bitcoin-mining-proxy/issues/7)! I planned to do this at one point but forgot about it later. Thanks for reminding me.2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far). Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script? Yeah, this is probably something that could be done. The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible. I assume this would be used to, for example, email you if one of the miners stops requesting work?I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run. Thanks again for all of your hard work! No problem! I'm glad you find the software useful. :)Title: Re: Flexible mining proxy Post by: pwnyboy on May 10, 2011, 06:19:10 AM Yeah, this is probably something that could be done. The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible. I assume this would be used to, for example, email you if one of the miners stops requesting work? Correct. One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like: `$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2` And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds. Quote I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run. Or the script could be a field in the database for workers, i.e. if the miner is a Windows box you might want a simple email, but if it's Linux, you might want to execute a different script that runs 'expect' and attempts to restart the miner by logging in via ssh and giving it a kick. For that matter, the other two could be database arguments, but if it's too difficult to be practical from a programmatical standpoint, no worries. Thanks again for all of your hard work! No problem! I'm glad you find the software useful. :)[/quote] Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours. So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages. Phoenix 1.45 seems to sporadically connect to the long-polling URL. Phoenix 1.2 will connect, but it disconnects after a period of time. I think I see why. I hit my local mining proxy and fetched the long polling URL. The first time it worked, and actually sent me info on a new block. But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix: GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1 Connection: close Host: bitproxy.xxxxx.com Authorization: Basic xxx User-Agent: phoenix/v1.2 (((2 minutes elapse, and then...))) HTTP/1.1 200 OK Date: Tue, 10 May 2011 06:00:12 GMT Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.1.6 X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy Set-Cookie: PHPSESSID=4j2r53dsse5afoa5u9k3oratn0; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 78 Connection: close Content-Type: application/json-rpc {"error":"Invalid response from long-poll request.","result":null,"id":"json"}Connection closed by foreign host. Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block. This should be easily duplicable for you. If not, I can PM you the info on my proxy and let you see for yourself. Hope that helps. Title: Re: Flexible mining proxy Post by: Cdecker on May 10, 2011, 12:02:58 PM Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing. I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes. Cool, keep me posted on that.1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone? I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have. Sure (https://github.com/cdhowie/Bitcoin-mining-proxy/issues/7)! I planned to do this at one point but forgot about it later. Thanks for reminding me.2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far). Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script? Yeah, this is probably something that could be done. The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible. I assume this would be used to, for example, email you if one of the miners stops requesting work?I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run. BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^ Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^ Title: Re: Flexible mining proxy Post by: cdhowie on May 10, 2011, 02:25:02 PM Correct. One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like: I'm not sure I could include the IP since that's not logged anywhere. But I could include the worker's database ID as well as the worker name.`$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2` And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds. Or the script could be a field in the database for workers, i.e. if the miner is a Windows box you might want a simple email, but if it's Linux, you might want to execute a different script that runs 'expect' and attempts to restart the miner by logging in via ssh and giving it a kick. For that matter, the other two could be database arguments, but if it's too difficult to be practical from a programmatical standpoint, no worries. To keep database schema changes to a minimum, I'm not sure if I would want to do this. But if I pass the worker's ID to the script, you could have your script look at the ID and figure out what to do.Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours. So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages. Phoenix 1.45 seems to sporadically connect to the long-polling URL. Phoenix 1.2 will connect, but it disconnects after a period of time. I think I see why. I hit my local mining proxy and fetched the long polling URL. The first time it worked, and actually sent me info on a new block. But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix: Hmm. This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this. It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker.... Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block. This should be easily duplicable for you. If not, I can PM you the info on my proxy and let you see for yourself. Hope that helps. Title: Re: Flexible mining proxy Post by: cdhowie on May 10, 2011, 02:29:05 PM Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^ There is support in the MVC framework for JSON output. The dashboard view page already implements IJsonView so try visiting /admin/?format=json. This works for me.BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^ You're right, I'm not sure why that's there. That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems. ;)Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d I haven't looked at it in-depth yet. Let me know when you are done with it and have tested it and I'll review the pull request.Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^ Title: Re: Flexible mining proxy Post by: Cdecker on May 10, 2011, 04:21:56 PM Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^ There is support in the MVC framework for JSON output. The dashboard view page already implements IJsonView so try visiting /admin/?format=json. This works for me.BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^ You're right, I'm not sure why that's there. That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems. ;)Edit: My first attempt on a JSON stats dump is here: https://github.com/cdecker/Bitcoin-mining-proxy/commit/82f8ad9e352352cdbe81d84a8939b817204cb59d I haven't looked at it in-depth yet. Let me know when you are done with it and have tested it and I'll review the pull request.Edit2: hope you didn't yet look at the stats and the worker-hashing branch because I was missing two GROUP BYs in the shares subquery which resulted in the hashing rate of all miners to be reported on a single miner instead of distinguishing. I fixed it in the branches ^^ Title: Re: Flexible mining proxy Post by: cdhowie on May 10, 2011, 04:30:38 PM Great, it works for me too, just had to change the URL in my scripts. I think using such an API to run cronjobs against is more useful than having it crammed into the web interface. Yup, that's why it's there. :) Not all of the pages have proper JSON support implemented, simply because for most of them it wouldn't be terribly useful. But at some point, the proxy should allow full manipulation of the database objects via JSON, allowing you to code an alternative interface (like a desktop app) against the JSON API. That probably wouldn't be a good thing to do, but it would at least be possible.I guess I can drop the stats branch then, since it already works out of the box from your dashboard. On the worker-hashing branch I made the average interval configurable because it strongly depends on the hashing speed (high hashing speeds can get a more accurate reading with shorter intervals, while slow miners will have the average jump wildly around for small values). I think it is ready to be pulled. Once again sorry for all the noise I created in here ^^ Alright, I'll have a look at it at some point later. Thanks for the patch!Title: Re: Flexible mining proxy Post by: molecular on May 10, 2011, 10:34:13 PM hey cdhowie,
great proxy, just checking it out. Had problems with the php notices spamming the pages to the point where chrome refused to render them (they get inserted before DOCTYPE decl): Example: Code: <br /> I simply put Code: error_reporting(E_ERROR); Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices. Title: Re: Flexible mining proxy Post by: cdhowie on May 10, 2011, 11:45:40 PM Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices. Hmm, I thought I did. Maybe I just did that in my own php.ini or something. Anyway, thanks for the comment. I'll add this in a future push.Title: Re: Flexible mining proxy Post by: pwnyboy on May 11, 2011, 04:15:14 AM Quote from: cdhowie Hmm. This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this. It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker. Ahhh I see the problem. From your proxy to deepbit: GET /listenChannel HTTP/1.0 Authorization: Basic youcanthavemypassword;] Host: deepbit.net:8332 I watched it send this to deepbit once and it worked, but the next time it didn't. The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0. Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1. So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not. You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1). If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1? Thanks! Title: Re: Flexible mining proxy Post by: cdhowie on May 11, 2011, 06:49:17 AM I've just done a push. I recommend that everyone update from master.
This update fixes some PHP warnings that would get triggered on a failover, effectively breaking failover entirely on configurations where PHP warnings are emitted. The pool timeout has also been lowered from 5 seconds to 2 since the default configuration of some miners is to request work every 5 seconds. These miners will break if the proxy takes too long to respond, terminating the active getwork request and issuing another. (This timeout will be made configurable in a later push so that you can customize it to your configuration.) An error_reporting() call has also been added to the config.inc.php.sample file. It's strongly recommended that you copy this into your config.inc.php to mitigate any possible future issues caused by PHP warnings getting mixed into the JSON response for workers. Title: Re: Flexible mining proxy Post by: cdhowie on May 11, 2011, 06:54:27 AM Ahhh I see the problem. From your proxy to deepbit: Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems. My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine.GET /listenChannel HTTP/1.0 Authorization: Basic youcanthavemypassword;] Host: deepbit.net:8332 I watched it send this to deepbit once and it worked, but the next time it didn't. The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0. Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1. So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not. You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1). If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1? Thanks! Title: Re: Flexible mining proxy Post by: pwnyboy on May 11, 2011, 10:12:39 AM Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems. My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine. I just cranked up a new kind of sniffer, one that logs each unique tuple into a separate file. What I found is that upon receiving my input text (fed to your proxy via telnet) as follows: Quote GET /index.php?lpurl=http%3A%2F%2Fdeepbit.net%3A8332%2FlistenChannel&pool=1 HTTP/1.1 Connection: close Host: bitproxy.xxxxx.com Authorization: Basic xxx User-Agent: phoenix/v1.2 (and one more trailing CR) Your proxy actually connected to deepbit twice. The first time was apparently to setup the actual long-polling session for my telnet to the proxy. The second communication was completely blank - neither side said anything. After a couple of minutes, my telnet session spat back the following: Quote HTTP/1.1 200 OK Date: Wed, 11 May 2011 09:59:01 GMT Server: Apache/2.2.3 (CentOS) X-Powered-By: PHP/5.1.6 X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy Set-Cookie: PHPSESSID=bd8s2r4gnb80pgl6n3hd3ne606; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 78 Connection: close Content-Type: application/json-rpc {"error":"Invalid response from long-poll request.","result":null,"id":"json"} This was not in response to anything being sent from deepbit to the proxy - deepbit never at any time uttered another word. I ran the same testing iteration again and got a successful long poll when the pool jumped to the next block, but on the next iteration it was exactly the same behavior as above. (note that all of this is still based on a 2-day old pull, I did not pull the code again per your mention above). Does that help? Title: Re: Flexible mining proxy Post by: pharaon on May 16, 2011, 02:58:57 PM can you give me a name of web hoster where your proxy can work ?
All i test, had 'php_auth_user' disable ...... Sorry for my poor english Title: Re: Flexible mining proxy Post by: BurningToad on May 17, 2011, 04:27:25 AM Took me a while, but I got it set up! Thanks!
1 BTC heading your way Title: Re: Flexible mining proxy Post by: BurningToad on May 17, 2011, 11:27:00 AM Actually this just didn't work very well for me :( Set it up with BTC Guild as primary and Slush as secondary. BTC Guild is having intermittent connection issues, so I wanted a backup. However, overnight all of my GUIMiner clients got stuck with a "connection problem". My one phoenix client (which is also on same machine as my proxy) continued working fine however.
Title: Re: Flexible mining proxy Post by: Cdecker on May 17, 2011, 12:18:29 PM Actually this just didn't work very well for me :( Set it up with BTC Guild as primary and Slush as secondary. BTC Guild is having intermittent connection issues, so I wanted a backup. However, overnight all of my GUIMiner clients got stuck with a "connection problem". My one phoenix client (which is also on same machine as my proxy) continued working fine however. Sounds strange. The switch should occur either upon work request or work submission. The latter isn't very nice because a share might be lost. We are working on exponential backoff to temporarily blacklist flapping pools (intermittent connection issues). Should it occur again please provide a tcpdump (just let tcpdump run along capturing traffic from/to the pool proxy port 80, and use temporary passwords between pool proxy and the workers).Title: Re: Flexible mining proxy Post by: cdhowie on May 17, 2011, 02:11:18 PM The switch should occur either upon work request or work submission. Switches will never occur on work submission. Submissions always route back to the pool that the work came from.Title: Re: Flexible mining proxy Post by: leepfrog on May 22, 2011, 09:07:50 PM Great peace of software. Managed to get it running in under 10 minutes.
I've always had some cases in the past where there were connection problems and the miner was in a stale state after connectivity has been restored. Looking forward to see if this is solved when using the proxy. It also would be an interesting idea to take such a service and offer it for hosting, so that the average miner does not have to worry about pool failover any more as he can simply point to some public proxy which handles the requests. Title: Re: Flexible mining proxy Post by: leepfrog on May 23, 2011, 04:22:25 AM Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again:
Traceback (most recent call last): File "phoenix.py", line 125, in <module> File "twisted\internet\base.pyc", line 1162, in run File "twisted\internet\base.pyc", line 1171, in mainLoop --- <exception caught here> --- File "twisted\internet\base.pyc", line 793, in runUntilCurrent File "minerutil\RPCProtocol.pyc", line 127, in callback File "minerutil\RPCProtocol.pyc", line 319, in handleWork exceptions.TypeError: argument of type 'NoneType' is not iterable [23/05/2011 01:04:42] Result: 4b18edf7 rejected My connection has some packet loss (~2-4%) but the latency in general is good (under 150ms). Could this be a problem? Title: Re: Flexible mining proxy Post by: oVPN on May 23, 2011, 10:43:54 AM hi
there are problems with php-authentication, if php is running as fast-cgi, but after switching to mod_php, login to admin-panel is working. i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts :( Title: Re: Flexible mining proxy Post by: twmz on May 26, 2011, 06:38:51 AM I really like the idea of this tool, but have failed to get it working well.
I use phoenix as my miner on ubuntu 11.04. I have tried a couple different web hosts including my own ubuntu 11.04 box, and I have the following symptoms: 1. lots more stale shares than when connected directly. long polling does not seem to work with phoenix through the proxy but works fine when connected directly. 2. phoenix miner *hammers* the web server with requests to index.php. Something like 10-20 requests per second as seen in the apache access.log. When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so). 3. lower than normal hash rate when using the proxy. I suspect this may be because the miner is so busy doing #2 as fast as it can. I also tried poclbm on my Mac and it had similar problems with stale shares. I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network. I also regularly got errors about "NoneType" not being indexable. From this thread, though, it sounds like some people are having great success with it, so I wish I knew what to do differently. Title: Re: Flexible mining proxy Post by: datathe1st on May 26, 2011, 08:34:16 AM I'd like to set this up to manage a mining for computers I have in different locations, but I am hesitant because of reports by some users of stale shares.
Thank you so very much cdhowie for even releasing what looks like an amazing piece of software. Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that? Title: Re: Flexible mining proxy Post by: Dusty on May 26, 2011, 10:15:36 AM Watching (very nice project btw)
Title: Re: Flexible mining proxy Post by: cdhowie on May 26, 2011, 04:26:46 PM Just pushed some updates.
These changes require modifications to your config.inc.php. If you pull without updating your config.inc.php, the software will break. To address some posts: Actually this just didn't work very well for me :( Set it up with BTC Guild as primary and Slush as secondary. BTC Guild is having intermittent connection issues, so I wanted a backup. However, overnight all of my GUIMiner clients got stuck with a "connection problem". My one phoenix client (which is also on same machine as my proxy) continued working fine however. Some miners will bail on a connection if it doesn't return work within 3-5 seconds. This is not always enough time for the proxy to detect a fail condition, and so the miner gives up before the proxy has a chance to try another pool.Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again: I have started seeing similar problems on my install. This is being actively researched.I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network. I also regularly got errors about "NoneType" not being indexable. I have been seeing the same with poclbm lately. I'm not sure what changed to make this happen, but I suspect it is related to the stale shares problem.i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts :( I haven't used that particular miner and so I'm not sure if the proxy supports it. If the miner isn't doing anything too wacky it should work... If I get some free time I might investigate this.long polling does not seem to work with phoenix through the proxy but works fine when connected directly. There are some issues with long polling that are not easily addressed by the current model. Long polling should in theory work just fine if your preferred pool doesn't go down, but during a failover scenario it can get a little bit wonky. Some miners behave differently with respect to how they handle changing long-polling URLs, and so this is something that will take a lot of engineering to fix.In the meantime I may supply a config option to disable long polling entirely; broken long polling is worse than none at all. phoenix miner *hammers* the web server with requests to index.php. Something like 10-20 requests per second as seen in the apache access.log. When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so). Sounds like a bug in Phoenix. Can you take some sniffs of this behavior with tcpdump -w bmp.pcap port 80 and email me the pcap file(s)? This will help me figure out what's going on. (Please compress them, and you can optionally encrypt them to my GPG key. This should not expose your mining account passwords, only your local proxy worker and possibly admin passwords.)Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that? As long as the computers can make outbound connections on whatever port you are running your web server on, yes. Transparent (or non-transparent) HTTP proxies should be able to process these requests too, though they might strip long-polling info.Title: Re: Flexible mining proxy Post by: twmz on May 28, 2011, 11:18:24 PM Hey, cdhowie. I'm still not clear on what was happening with the proxy, but I have a couple clues.
I did determine that the latest phoenix behavior is to re-request the long polling URL immediatlly if the LP does not return a valid getwork response. So, I think the issue with phoenix hammering the web server was because the proxied long-poll address must have been immediately returning, which caused phoenix to request it again right away, which caused... well, a very fast loop of LP requests to the proxy. In the end, I only know enough about PHP to be dangerous, and less about how to run an apache web server. In the end, I was so inspired by the idea of this proxy that I ended up recoding it in ASP.NET, not because it is better than ASP.NET but just because I have 10 years of experience with it and know how to debug. For the actual proxying of the JSON-RPC requests, I followed your algorithm almost exactly, just in C#, ASP.NET MVC, IIS and SQL Server. I now have it working and long polling is working as well although the whole design of long polling tie-ing up a web service thread for 10-20 minutes doesn't play all nicely with IIS, but it doesn't matter too much for my 4 measly miners. A side bonus is that ASP.NET has a very capable charting component and so I was able to quickly whip up some graphs for the dashboard as well. To be clear, I doubt anyone will care about my implementation. An IIS web server and SQL Server are much harder to come by than a Linux, Apache, MySQL, and PHP server. I just happen to have a suitable windows server in my house already, so it worked well for me. That said, as soon as I polish it a little more, I will post the source code somewhere given that I did borrow the idea and some of the assets from your project. https://i.imgur.com/N24ZB.png Title: Re: Flexible mining proxy Post by: anisoptera on May 29, 2011, 06:21:39 AM To be clear, I doubt anyone will care about my implementation. I do, I definitely want to see the code. :) Title: Re: Flexible mining proxy Post by: Cdecker on May 29, 2011, 09:53:33 AM Just seeing now that you have a payout monitor in there, that would be a cool feature, though it adds a lot of problems since some pools don't provide that info and even worse, they don't adhere to a consistent standard for publishing them.
Title: Re: Flexible mining proxy Post by: twmz on May 29, 2011, 12:38:36 PM Just seeing now that you have a payout monitor in there, that would be a cool feature, though it adds a lot of problems since some pools don't provide that info and even worse, they don't adhere to a consistent standard for publishing them. Yes, there is no standard. But it only turned out to be a couple lines of code specific to each type of pool to fish the balance out of each pools JSON API. It actually doesn't work that well and is the main thing I am working on right now before releasing the code in the next day or two. Making all of the JSON API calls to retrieve current balances is making the dashboard load slowly, so I'm moving balance updating to a background thread. Title: Re: Flexible mining proxy Post by: 0xlemming on May 29, 2011, 06:34:33 PM Is the stuff in work_data needed for something importent? I added to my runnning script a function which remove all entries older than 10h.
Title: Re: Flexible mining proxy Post by: twmz on May 29, 2011, 06:51:47 PM 10h should be ok. It is needed when a miner eventually submits a share. The proxy uses the information in the work_data table to look up which pool the miner got that work from so that it gets submitted back to the right place. So don't delete everything in the table, but it is not needed as soon as your miners have finished working on their current work task.
Title: Re: Flexible mining proxy Post by: error on May 30, 2011, 03:40:03 AM Thanks for the proxy. Seems to work pretty well so far.
I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST: Code: --- a/htdocs/common.inc.php 2011-05-29 22:20:55.000000000 -0400 Title: Re: Flexible mining proxy Post by: pwnyboy on May 30, 2011, 03:58:35 AM For those of you affected by deepbit/slush outages today, how well did this code hold up for you?
Title: Re: Flexible mining proxy Post by: error on May 30, 2011, 04:10:30 AM For those of you affected by deepbit/slush outages today, how well did this code hold up for you? It's pretty transparently switching between pools. The miner never notices a thing. Title: Re: Flexible mining proxy Post by: pwnyboy on May 30, 2011, 04:29:08 AM Sweet. I have what looks like a working implementation, minus the aforementioned difficulties with long-polling that me and several others have experienced. But it wasn't in use today for that reason. I'm hoping to find a simple http proxying script and just change the LP URL handed out by cdhowie's script, to make the miners hit a different script to handle long polling in a non-problematic way. Or so I hope.
Title: Re: Flexible mining proxy Post by: d3c0n808 on May 30, 2011, 05:09:35 AM I'm sorry but I need to ask a question. Currently for pools I have deepbit and solo, deepbit has a higher priority but doesnt connect for some reason. I rebooted, the computer can resolve the dns but none of the miners connect to deepbit just the solo. Perhaps this has been asked before. Im wondering if there is some type of cache that needs to be reset or what. Im using poclbm gui and poclbm on my dedicated miners(slackware) The admin pages seems to be working fine but its just not kicking over to deepbit. Thank you in advance and the address i ahve in the pool config is http://pit.deepbit.net:8332
Title: Re: Flexible mining proxy Post by: CubedRoot on May 30, 2011, 07:20:16 AM Going to look into this one for myself. I am having a problem mining from work :) I have my own co-located server, but dont want to setup a pool on it. I might could use this to connect my miners to my co-located server on a port that isnt blocked by our firewall, and then use my co-located server to connect to the pools.
Title: Re: Flexible mining proxy Post by: Yngwie on May 30, 2011, 09:04:41 AM I can't configure this proxy((
Admin panel works fine, i add pools (deepbit: http://deepbit.net:8332 and btcguild: http://btcguild.com:8332) then created worker (login q password 1) add pool workers credentials for it (email_5 password_of_worker_in_pool for deepbit login_2 password_of_worker_in_pool for btc guild). But when i started miner (i tried poclbm, bitcoin-miner, diablo miner windows with my servers ip as url port 80 login and password from proxy worker) proxy returned HTTP/1.1 200 OK Date: Mon, 30 May 2011 07:46:57 GMT Server: Apache X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy Set-Cookie: PHPSESSID=kph1h3dib6nnad4okc3k29kp85; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 85 Content-Type: application/json-rpc {"error":"No enabled pools responded to the work request.","result":null,"id":"json"} How can i solve this problem? Title: Re: Flexible mining proxy Post by: error on May 30, 2011, 09:28:11 AM Add more pools?
Title: Re: Flexible mining proxy Post by: d3c0n808 on May 30, 2011, 10:04:55 AM I can't configure this proxy(( Admin panel works fine, i add pools (deepbit: http://deepbit.net:8332 and btcguild: http://btcguild.com:8332) then created worker (login q password 1) add pool workers credentials for it (email_5 password_of_worker_in_pool for deepbit login_2 password_of_worker_in_pool for btc guild). But when i started miner (i tried poclbm, bitcoin-miner, diablo miner windows with my servers ip as url port 80 login and password from proxy worker) proxy returned HTTP/1.1 200 OK Date: Mon, 30 May 2011 07:46:57 GMT Server: Apache X-Source-Code: https://github.com/cdhowie/Bitcoin-mining-proxy Set-Cookie: PHPSESSID=kph1h3dib6nnad4okc3k29kp85; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Length: 85 Content-Type: application/json-rpc {"error":"No enabled pools responded to the work request.","result":null,"id":"json"} How can i solve this problem? I'm getting this exact same error. Ive added slush's pool, deepbit, btcmine, and solo and solo is the only one that works.... Title: Re: Flexible mining proxy Post by: error on May 30, 2011, 11:27:16 AM Sorry, I meant to say: ENABLE your pools!
Title: Re: Flexible mining proxy Post by: d3c0n808 on May 30, 2011, 11:35:33 AM I followed the instructions to the T, and yes all 4 pools are enabled.
Title: Re: Flexible mining proxy Post by: 0xlemming on May 30, 2011, 05:34:34 PM I'm getting this exact same error. Ive added slush's pool, deepbit, btcmine, and solo and solo is the only one that works.... Title: Re: Flexible mining proxy Post by: d3c0n808 on May 30, 2011, 10:46:59 PM I'm getting this exact same error. Ive added slush's pool, deepbit, btcmine, and solo and solo is the only one that works.... This seems like its some kind of dependancy. I reinstalled the database using phpmysql to see if perhaps mysql admin didnt install the scheme properly. That did not resolve the issue. So I loaded it on my suse box at another location and it works without a hitch. It would seem to me there is some dependy that slackware and your distro don't have that are required. PHP magic quotes is is disabled so I know its not that I also tried to install php 5.2 on my slackware box and that did not resolve the issue either. Title: Re: Flexible mining proxy Post by: pwnyboy on May 31, 2011, 01:14:32 AM Slackware? Heh I'd hang it up. I don't even bother trying to run anything meaningful on it (i.e. above basic apache, php and mail). I love slack, been an avid user since version 3.3 (circa 1996). But it has very little utility.
I can also confirm that this package, as it sits, is very distribution-specific. It runs out of the box on Cent 5.6 but is broken on FC14x64, have not had the time to troubleshoot further. Out of curiosity, have you installed json for php? Title: Re: Flexible mining proxy Post by: d3c0n808 on May 31, 2011, 01:23:37 AM Slackware? Heh I'd hang it up. I don't even bother trying to run anything meaningful on it (i.e. above basic apache, php and mail). I love slack, been an avid user since version 3.3 (circa 1996). But it has very little utility. I can also confirm that this package, as it sits, is very distribution-specific. It runs out of the box on Cent 5.6 but is broken on FC14x64, have not had the time to troubleshoot further. Out of curiosity, have you installed json for php? I prefer slackware to just about any other distro. Its fast, simple, efficient, sbopkg has come along way and i build most of my own modules, kernels etc etc. I wasn't sure if slackware's php package had json rpc built in, so in installed it, no go. Rebuilt php with json argument, installed pkg(after removing old pkg ofcourse) still no go. I don't think its a json issue. I could be wrong and maybe its a version issue. Im just going to keep using it on my suse box as the slackware box seems to have enough to do already Title: Re: Flexible mining proxy Post by: pwnyboy on May 31, 2011, 01:32:40 AM I agree, I'd use Slack for everything if I could. I mentioned JSON because in PHP versions prior to 5.3 you have to install it separately. I asked cdhowie to mention this somewhere as a dependency, as to cut down on potential support issues, but I guess it's still not in the docs.
Title: Re: Flexible mining proxy Post by: d3c0n808 on May 31, 2011, 01:45:51 AM I agree, I'd use Slack for everything if I could. I mentioned JSON because in PHP versions prior to 5.3 you have to install it separately. I asked cdhowie to mention this somewhere as a dependency, as to cut down on potential support issues, but I guess it's still not in the docs. All my miners run slack, sdk 2.1, cat 11.4 no problem. Its just this darn proxy hahaha. Slack 13.37 ships with 5.3.6 so json should be installed by default and I believe it is as installing it manually created a json error stating that the module was already loaded. Im wondering if 5.3.6 just doesnt play well with the proxy oh well eventually i imagine ill figure it out. Suse in runlevel 3 isnt too bad, just not as slick as slack hahaha. Title: Re: Flexible mining proxy Post by: kjj on May 31, 2011, 02:29:55 AM I'm running the proxy on Slackware. Slackware 10.2, no less.
Works great with BTC Guild, Deepbit, Eligius, a local bitcoind (last resort), and a local pushpool (for testing, that was a much bigger hassle to build on Slack). Works great with poclbm and phoenix, but tends to cause DiabloMiner to get hung. Just finished setting up a cron job to SNMP-diddle an APC networked PDU to reboot my boxes if they ever go more than 15 minutes without a work request. Title: Re: Flexible mining proxy Post by: d3c0n808 on May 31, 2011, 02:32:41 AM I'm running the proxy on Slackware. Slackware 10.2, no less. Works great with BTC Guild, Deepbit, Eligius, a local bitcoind (last resort), and a local pushpool (for testing, that was a much bigger hassle to build on Slack). Works great with poclbm and phoenix, but tends to cause DiabloMiner to get hung. Just finished setting up a cron job to SNMP-diddle an APC networked PDU to reboot my boxes if they ever go more than 15 minutes without a work request. Full install or did you have to build your own php package? Also is there any special configurations you used? Im still trying to get my slack to work hahaah Title: Re: Flexible mining proxy Post by: kjj on May 31, 2011, 02:37:13 AM I custom build PHP and Apache (and openssl, and mod_ssl, and mm, and mysql, and...). Wrote a script for it many, many years ago.
I did have to add --with-pdo-mysql to the file that stores my PHP config options to get this working. Title: Re: Flexible mining proxy Post by: pwnyboy on May 31, 2011, 02:57:07 AM Just finished setting up a cron job to SNMP-diddle an APC networked PDU to reboot my boxes if they ever go more than 15 minutes without a work request. Nice! You should open-source that.. I was just pondering making such a thing for my AP9211s :) Title: Re: Flexible mining proxy Post by: kjj on May 31, 2011, 03:15:21 AM It'll take some work to adapt to your location. Oh, and this is not pretty.
First file is a PHP script that runs out of cron every 10 minutes. Second file is a BASH CGI that lives on another box. If your flex proxy can see the PDU directly, you can cut out the middle man and either use snmpget() and snmpsend() PHP functions, if you have them, or exec the binaries directly. My worker IDs are 3 and 4, so I plugged them into ports 3 and 4 on the PDU. That isn't necessary, unless you are OCD like me. The arg to the CGI is pasted directly onto the end of the OID passed to SNMP, which in this case is the outlet number on the strip. Oh, and you should totally add some input checking to the CGI, because as it is now, you are asking to get pwn3d by executing attacker-supplied variables. Search for __ (double underlines), those are things that you need to change. Tested and working on a APC 885-1890 (1U, 8 outlet, 15Amp, 120v). Yours may differ. Thank god I was able to find the OIDs for SNMP, I wasn't looking forward to writing an expect script to navigate the telnet menu. Code: (cron_job) #!/bin/php Code: (bash_cgi) #!/bin/bash Title: Re: Flexible mining proxy Post by: pwnyboy on May 31, 2011, 03:44:55 AM Exxxcellent. This will come in handy when I get all of the long-polling implementation bugs worked out. Thanks!
Title: Re: Flexible mining proxy Post by: cdhowie on May 31, 2011, 03:23:35 PM I've pushed a series of changes that should clear up several issues. Long-polling should be much more stable now; I've been running my miner all night and have seen plenty of new block notifications in the miner log, and a significant reduction in rejected shares.
To address Deepbit specifically, it has started returning an absolute URL in the X-Long-Polling header, and the proxy did not support this. Now it does. Note that Deepbit is currently infinitely redirecting long-polling requests for some reason; I am going to contact Tycho about this and see what's up. In the meantime, long-polling against Deepbit will not work -- but it will appear to! The proxy will now sleep for 30 minutes after a failed long-polling request and then pretend that it was a getwork request. This is because some miners will give up on long-polling altogether if any long-polling request ever fails. By pretending that the request succeeded, the miner will continue to send long-polling requests, so when long-polling starts working again (or the proxy fails-over to a different pool) the miner will begin benefiting from long-polling again without needing to be restarted. I believe that some of these problems were the cause of Phoenix hammering Apache with long-polling requests. Those of you who were having trouble with Phoenix, consider giving the proxy another go with the latest code. Please thoroughly stress-test the code in the repository and let me know what other issues you find. :) UPDATE: Tycho fixed long-polling. The proxy should now work correctly with Deepbit. Title: Re: Flexible mining proxy Post by: kjj on May 31, 2011, 03:31:20 PM Any idea why my one Diablo miner occasionally goes idle when using the proxy? Seems to happen at random, sometimes a few minutes, sometimes many hours.
Title: Re: Flexible mining proxy Post by: twmz on May 31, 2011, 05:19:36 PM I believe that some of these problems were the cause of Phoenix hammering Apache with long-polling requests. Those of you who were having trouble with Phoenix, consider giving the proxy another go with the latest code. There was also a bug in Phoenix with LP urls that used query strings. Not sure if that may also make a difference. http://forum.bitcoin.org/index.php?topic=6458.msg152263#msg152263 Title: Re: Flexible mining proxy Post by: cdhowie on May 31, 2011, 06:11:15 PM There was also a bug in Phoenix with LP urls that used query strings. Not sure if that may also make a difference. Good catch. Yeah, that would cause problems. I might work around this by using PATH_INFO to convey the required data.http://forum.bitcoin.org/index.php?topic=6458.msg152263#msg152263 Title: Re: Flexible mining proxy Post by: kjj on May 31, 2011, 06:28:26 PM So, I got sick of MySQL's buggy time zone handling and made an improvement to my script. For this to work, you need to add apc_port as an INTEGER DEFAULT 0 column to the worker table. Set the port to toggle when reboot is requested, or leave at zero to ignore that worker.
Code: (improved reboot script) #!/bin/php Title: Re: Flexible mining proxy Post by: error on June 01, 2011, 01:38:14 AM There was also a bug in Phoenix with LP urls that used query strings. Not sure if that may also make a difference. Good catch. Yeah, that would cause problems. I might work around this by using PATH_INFO to convey the required data.http://forum.bitcoin.org/index.php?topic=6458.msg152263#msg152263 Thanks. It was because of this proxy that I caught it. :) Title: Re: Flexible mining proxy Post by: twmz on June 01, 2011, 05:25:03 AM I do, I definitely want to see the code. :) My alternate implementation in ASP.NET, as it exists today is now published here: http://code.google.com/p/btcproxy/. It has been functioning well for me over the past 4-5 days, but I have only tested it with phoenix miner, poclbm, and the pools shown in the screenshot. Right now, I have not written any documentation on how to install it, so you'll need to be comfortable with ASP.NET development and SQL to get it deployed. I'll work on a brief readme.txt over the next couple days that spells out the prerequisites and general installation steps. As I make additional improvements or refinements, I'll update the code on that website and not hijack this thread any further. Title: Re: Flexible mining proxy Post by: burtyb on June 01, 2011, 11:56:19 AM I'm not using PHP > 5.3 yet (CentOS 5.x) so I had to make a little change to get it to run without the DateTime class.
Basically I've replaced (format_date function in common.inc.php) Code: $obj = new DateTime($date, new DateTimeZone('UTC'));with Code: return date($BTC_PROXY['date_format'],strtotime($date)); I didn't bother with the timezone as it looks like it's only used to display but that might be fixed with something like Code: date_default_timezone_set($BTC_PROXY['timezone']); ChrisB. Title: Re: Flexible mining proxy Post by: ciuciu on June 01, 2011, 06:45:00 PM Hi,
I try to setup the proxy but I can't connect to it using the phoenix miner. I did setup the pools and workers. I'm using start /D C:\phoenix phoenix -u http://user:pass@mywebspace.com:80/ -k poclbm device=1 VECTORS BFI_INT FASTLOOP AGGRESSION=7 without success. Can somebody show me an example of the phoenix command line for connecting it correctly to the web proxy? Thanks. Title: Re: Flexible mining proxy Post by: ciuciu on June 01, 2011, 08:08:20 PM Tried now and still not working.
Thanks. Title: Re: Flexible mining proxy Post by: btcLeger on June 01, 2011, 10:15:12 PM nice idea but im getting lots of blank pages when handling the backend. This happens when
a) editing a pool b) saving worker-pool managemant goes to worker-pool.php seems missing parameters (would be nice to jump back to actual woker overview page e.g. ?id=1&action=index xampp with PHP 5.2.4 Title: Re: Flexible mining proxy Post by: cdhowie on June 02, 2011, 12:39:56 AM nice idea but im getting lots of blank pages when handling the backend. I have not tested with many versions, but others have reported that at least PHP 5.3 is required to use the proxy. I will add this to the readme at some point.xampp with PHP 5.2.4 Title: Re: Flexible mining proxy Post by: pwnyboy on June 02, 2011, 11:12:02 AM nice idea but im getting lots of blank pages when handling the backend. I have not tested with many versions, but others have reported that at least PHP 5.3 is required to use the proxy. I will add this to the readme at some point.xampp with PHP 5.2.4 I can also confirm many blank pages and strangeness under Cent 5.x with the default PHP version 5.1.x. A stated dependency on PHP 5.3 would go a long way I'm sure. Title: Re: Flexible mining proxy Post by: carlo on June 02, 2011, 01:50:46 PM Hello everybody,
i love your proxy its nice to have a central control unit. There are some minor problems: It seems a bit slow for me, often my miners (poclbm) report this: warning: job finished, miner is idle Ideas: Maybe i need to pimp the apache.conf a bit. Not sure right now, my System: Debian 5 / apache - Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_python/3.3.1 Python/2.5.2 mod_ssl/2.2.9 OpenSSL/0.9.8g mod_wsgi/2.5 Theres a new problem with they nice proxy, it happens with different pools, this results in a lot of failed work submissions: Code: 02/06/2011 08:56:10, Unexpected error: Maybe its all apaches fault :) I hope i will figure it out soon, thanks for your time spending in this project. Title: Re: Flexible mining proxy Post by: burtyb on June 02, 2011, 11:21:32 PM I have not tested with many versions, but others have reported that at least PHP 5.3 is required to use the proxy. I will add this to the readme at some point. Using my changes above I've been using it with PHP 5.1.6 with no apparent problems, do you know of any other reasons why it requires PHP 5.3? ChrisB. Title: Re: Flexible mining proxy Post by: kjj on June 03, 2011, 03:18:12 PM This join makes me sad.
Code: ("sad join") SELECT Solution? Code: ("delta.sql") ALTER TABLE work_data DROP PRIMARY KEY; Code: ("new cron job, hourly") <?php Title: Re: Flexible mining proxy Post by: pwnyboy on June 04, 2011, 05:50:28 AM Any news on the high rejection rate? That's the only thing keeping me from using this software, the idea behind it is fantastic! You're saying the long polling apparatus is still problematic, even with the latest code? Title: Re: Flexible mining proxy Post by: Disposition on June 05, 2011, 01:28:50 AM everything is running fine except after awhile it just times out for all my clients, feels like there's some apache config problem.
I noticed earlier post you mentioned something about safe mode, and that's off by default so I'm a bit confused to why this is happening, appreciated if you can look into it edit: it seems like this only applies to btcguild for some reason. Title: Re: Flexible mining proxy Post by: gigabytecoin on June 05, 2011, 06:51:45 AM Keep up the great work OP!
Title: Re: Flexible mining proxy Post by: pwnyboy on June 05, 2011, 07:12:40 PM everything is running fine except after awhile it just times out for all my clients, feels like there's some apache config problem. I noticed earlier post you mentioned something about safe mode, and that's off by default so I'm a bit confused to why this is happening, appreciated if you can look into it edit: it seems like this only applies to btcguild for some reason. For long polling to work correctly through the proxy, you'd need (at minimum) to have a long execution time in php.ini. Something like this: Code: max_execution_time=7500 That assumes it would take 2 hours (probably a maximum of maximums) to find a block, so I added an extra 5 minutes for additional padding. In actuality I've never seen it take more than 40 or 50 minutes to find a block though, so 7500 should be perfectly fine. Note that modifying php.ini is a global setting, so you might be able to/might want to modify this only for the virtualhost that runs your proxy, depending on your specific OS and apache/php implementation. Don't forget to restart httpd to make the setting take effect. Also there might be other settings that need to be modified, as it's really not normal to have an http connection open for 2 hours. But this one is the most obvious. Title: Re: Flexible mining proxy Post by: Disposition on June 06, 2011, 12:25:55 AM @pwnyboy
Thanks, will stress test n run this for a bit. Title: Re: Flexible mining proxy Post by: Disposition on June 07, 2011, 04:57:11 AM much better now, except I'm still getting a weird unable to auth error some times (I assume due to the time out between proxy and server)
the proxy(NJ based), btcmine(NY) based, Me(NYC) is really close, the ping is great so I think it's something with this script. pwnyboy any other php.ini configuration I should know about it? Title: Re: Flexible mining proxy Post by: btcLeger on June 07, 2011, 06:04:35 PM Ok with php 5.3.5 things work better but there is still a blank page when editing a worker. Changed name or password is not saved.
I've let it run for 2 hours with 4 workers and 1 Ghash/s total and got 3% rejected shares with guiminer using poclbm no matter which pool I join. The rate of stale shares is 10 times higher on deepbit when using the proxy. Besides of that I like the idea and the concept. I will use the proxy and donate if the stale shares problem gets solved. Features I'd love to have: - autorefresh for dashboard - workerstatus with configureable timeout display (green/red row background) Title: Re: Flexible mining proxy Post by: Disposition on June 08, 2011, 04:53:04 AM bump, seriously if this technical problem is solved and timing-out auth issue is fixed, I put a 5btc bounty on it.
edit: nvm I fixed it. Title: Re: Flexible mining proxy Post by: exfuga on June 08, 2011, 05:41:00 AM Love, love, love the idea!
One thing, on the main admin page I get this error, and it only displays half of the table. "This page contains the following errors: error on line 37 at column 13: Extra content at the end of the document Below is a rendering of the page up to the first error." I am on PHP 5.3.2 Title: Re: Flexible mining proxy Post by: Papipapito on June 08, 2011, 04:38:28 PM Hello
Have a problem with a miner Ufasoft, with diablo and poclbm work fine The miner connecting to proxy and start a work, but in 5 minutes generate a timeout and miner stop the log of apache: - - [08/Jun/2011:18:06:51 +0200] "POST / HTTP/1.1" 401 484 "-" "Ufasoft bitcoin-miner/0.10 CPU/Intel(R) Pentium(R) 4 CPU 3.00GHz/f43 PC=2 CC=1" - 2 [08/Jun/2011:18:06:51 +0200] "POST / HTTP/1.1" 400 392 "-" "Ufasoft bitcoin-miner/0.10 CPU/Intel(R) Pentium(R) 4 CPU 3.00GHz/f43 PC=2 CC=1" - 2 [08/Jun/2011:18:07:11 +0200] "POST / HTTP/1.1" 200 1015 "-" "Ufasoft bitcoin-miner/0.10 CPU/Intel(R) Pentium(R) 4 CPU 3.00GHz/f43 PC=2 CC=1" And now work and in 10 minutes, timeout and not appears nothing in logs Any ideas? Sorry for my bad english Title: Re: Flexible mining proxy Post by: tito13kfm on June 09, 2011, 06:06:48 AM So awesome! Donation sent!
I look forward to future updates. Would love graphs! I don't even know what kind, but I love graphs! Title: Re: Flexible mining proxy Post by: pwnyboy on June 09, 2011, 06:30:39 AM bump, seriously if this technical problem is solved and timing-out auth issue is fixed, I put a 5btc bounty on it. edit: nvm I fixed it. For the benefit of others, and because it's unclear whether a long polling problem still exists with the proxy code, would you care to share the nature of your problem and the solution? Title: Re: Flexible mining proxy Post by: Azelphur on June 10, 2011, 07:49:25 PM Anyone else care to check the contents of their /var/lib/php5? after running this for a while I started getting no disk space errors, upon further investigation I found out that PHP had created 15.5 million session files in /var/lib/php5. Fun.
Should anyone else bump into this problem, you'll probably notice that nothing can delete the files (rm, ls, find etc all hang) after much fiddling, the solution is to use ls -U1, which will print the file list unsorted, and then pipe the output to rm, like so. ls -U1 /var/lib/php5 | xargs rm -f Gonna take me 5 days of rm to clean up all the session files, fun fun :D Besides this bug (which may just be some issue on my end) it seems great, I love how I can have pool redundancy in case one goes down. Title: Re: Flexible mining proxy Post by: btcLeger on June 11, 2011, 06:12:25 PM Anyone else care to check the contents of their /var/lib/php5? after running this for a while I started getting no disk space errors, upon further investigation I found out that PHP had created 15.5 million session files in /var/lib/php5. Fun. cannot confirm this. have the proxy running under xampp. Yes, there are loads of sessions generated in the tmp-folder (about 10 per minute per miner) but expired session are getting deleted by the system. Must be a misconfigured serverside variable. Its not uncommon, Ive seen this often on some webservers when millions of sessionfiles slow down webservices. But dont ask me for a solution. Thats work for those who configure servers... ;DShould anyone else bump into this problem, you'll probably notice that nothing can delete the files (rm, ls, find etc all hang) after much fiddling, the solution is to use ls -U1, which will print the file list unsorted, and then pipe the output to rm, like so. ls -U1 /var/lib/php5 | xargs rm -f Gonna take me 5 days of rm to clean up all the session files, fun fun :D Besides this bug (which may just be some issue on my end) it seems great, I love how I can have pool redundancy in case one goes down. Title: Re: Flexible mining proxy Post by: btcLeger on June 11, 2011, 06:23:36 PM Another suggestion for improved usability: I find it a bit unhandy to add new workers and pools when already having lots of workers & pools in the database. Would be nice to have a worker-edit page to configure all existing pools at once without the need to edit them all individually. Same with the pools. when adding a new pool, would be nice to update all existing workers for the pool on one page instead of picking them individually. Thats 30+ mouseclicks, where 3 would be sufficient...
Title: Re: Flexible mining proxy Post by: jondecker76 on June 12, 2011, 10:43:43 AM I just finished setting up the bitcoin-mining-proxy. I have the mysql database working, and have the pools and workers set up. Now I have a couple of questions!
1) When I go to my <ip_of_proxy> it asks for username and password, but they don't work there like they do at <ip_of_proxy>/admin. Is this normal? 2) How do I point my miner to the proxy? I couldn't find anything in the documentation?? Since its on the same host, I'm sure its going to be 127.0.0.1 - but what port?? If I use 8332, it will interfere with the locally running bitcoind, correct? So far very impressive! Help me to point my miners (running phoenix r100) to the proxy and I'll post back results! thanks again Title: Re: Flexible mining proxy Post by: Cdecker on June 12, 2011, 10:46:49 AM I just finished setting up the bitcoin-mining-proxy. I have the mysql database working, and have the pools and workers set up. Now I have a couple of questions! 1) the /admin you use the user and pass from your config, while / is for the workers to contact, so there you'd use the miners username and password1) When I go to my <ip_of_proxy> it asks for username and password, but they don't work there like they do at <ip_of_proxy>/admin. Is this normal? 2) How do I point my miner to the proxy? I couldn't find anything in the documentation?? Since its on the same host, I'm sure its going to be 127.0.0.1 - but what port?? If I use 8332, it will interfere with the locally running bitcoind, correct? So far very impressive! Help me to point my miners (running phoenix r100) to the proxy and I'll post back results! thanks again 2) Since you run the proxy on a webserver you'll probably have to point the workers to http://<ip>:80/ :D HTH, cdecker Title: Re: Flexible mining proxy Post by: jondecker76 on June 12, 2011, 10:54:44 AM I thank you, one Decker to another :P
Worked great! Seamless failover for me as deepbit is priority 1, but it is down so its working great with the next pool in line - excellent! Hash rate is normal - so far so good Title: Re: Flexible mining proxy Post by: jondecker76 on June 12, 2011, 11:54:52 AM is priority 1 higher than priority 2, or is priority 2 higher than priority 1?
Title: Re: Flexible mining proxy Post by: jondecker76 on June 12, 2011, 12:10:47 PM didn't work well for me at all...... Had 3 pools set up, and 3 workers set up (I have 3 5850's, so a worker for each)..
priorities... Worker1: 1 bitcoin.cz 2 btcguild.com 3 deepbit.net worker 2: 1 btcguild.com 2 deepbit.net 3 bitcoin.cz worker 3 1 deepbit.net 2 bitcoin.cz 3 btcguild.com All 3 of my cards were hashing for bitcoin.cz for some reason, even though all 3 pools were up... So each card should have been going to a separate pool........ The other problem is, after a few minutes, phoenix would display that the work queue was empty and hashes would drop to 0 (on all 3 cards/workers) and hang there.... So, so far, not so good unfortunately :( I now have each card manually hashing each of the 3 pools listed above, and they all work fine..... Just not through the proxy. Wil experiment some more later Title: Re: Flexible mining proxy Post by: kjj on June 12, 2011, 03:02:21 PM is priority 1 higher than priority 2, or is priority 2 higher than priority 1? 2 will be used before 1. Better idea is to space them out by 10s, or even 20s, so that you can squeeze pools in between without having to renumber them all. Title: Re: Flexible mining proxy Post by: jgarzik on June 12, 2011, 04:14:15 PM See also https://github.com/kylegibson/poclbm for a poclbm fork that offers fallback support. A bit easier than a proxy.
Title: Re: Flexible mining proxy Post by: kjj on June 12, 2011, 04:20:57 PM didn't work well for me at all...... Had 3 pools set up, and 3 workers set up (I have 3 5850's, so a worker for each).. priorities... Worker1: 1 bitcoin.cz 2 btcguild.com 3 deepbit.net worker 2: 1 btcguild.com 2 deepbit.net 3 bitcoin.cz worker 3 1 deepbit.net 2 bitcoin.cz 3 btcguild.com All 3 of my cards were hashing for bitcoin.cz for some reason, even though all 3 pools were up... So each card should have been going to a separate pool........ The other problem is, after a few minutes, phoenix would display that the work queue was empty and hashes would drop to 0 (on all 3 cards/workers) and hang there.... So, so far, not so good unfortunately :( I now have each card manually hashing each of the 3 pools listed above, and they all work fine..... Just not through the proxy. Wil experiment some more later Check your URLs, usernames, and passwords. Every time that I've run into a problem like this, it was because one or more pools was down, or because I had botched my configuration. Title: Re: Flexible mining proxy Post by: jondecker76 on June 12, 2011, 05:37:50 PM very strange.. it works fine if there is only one pool, but when I add multiples, it simply doesn't work. I have also noticed that my stale/rejected shares went up extremely high... In deepbit, for the last month I have been under 1% stales. Using the proxy, I'm at almost 20% rejected shares :(
I was really hoping that this would work Title: Re: Flexible mining proxy Post by: pwnyboy on June 12, 2011, 10:13:39 PM very strange.. it works fine if there is only one pool, but when I add multiples, it simply doesn't work. I have also noticed that my stale/rejected shares went up extremely high... In deepbit, for the last month I have been under 1% stales. Using the proxy, I'm at almost 20% rejected shares :( I was really hoping that this would work The stale shares you're referring to are because of the aforementioned long-polling issues. Nobody is sure what makes it work, or not work, through the current codebase. The best suggestion I can give you is to run something like PHProxy at http://url.of.your.minining.proxy/phproxy/ and edit the code to send out this URL to your miners, rather than trying to use the in-built broken proxying code. At least I think that'll work, from what I recall of the code and the way that LP works. If you go this route, report back and let us know how it works out. Title: Re: Flexible mining proxy Post by: cdhowie on June 13, 2011, 04:58:20 AM Anyone else care to check the contents of their /var/lib/php5? after running this for a while I started getting no disk space errors, upon further investigation I found out that PHP had created 15.5 million session files in /var/lib/php5. Fun. Thanks for the report. This was indeed a bug, and has been fixed in master.Should anyone else bump into this problem, you'll probably notice that nothing can delete the files (rm, ls, find etc all hang) after much fiddling, the solution is to use ls -U1, which will print the file list unsorted, and then pipe the output to rm, like so. ls -U1 /var/lib/php5 | xargs rm -f Gonna take me 5 days of rm to clean up all the session files, fun fun :D Besides this bug (which may just be some issue on my end) it seems great, I love how I can have pool redundancy in case one goes down. Title: Re: Flexible mining proxy Post by: kjj on June 13, 2011, 05:07:42 AM If you aren't using /var/lib/php5 for your session files, do "php -i | grep session.save_path" on the command line, or make a file with:
Code: <?php In your document tree, and load it in your browser, then look for session.save_path. An alternative removal method is: Code: find /tmp/ -name sess_\* -exec rm {} \;Title: Re: Flexible mining proxy Post by: cdhowie on June 13, 2011, 05:09:00 AM very strange.. it works fine if there is only one pool, but when I add multiples, it simply doesn't work. I have also noticed that my stale/rejected shares went up extremely high... In deepbit, for the last month I have been under 1% stales. Using the proxy, I'm at almost 20% rejected shares :( That's weird. :( The code path isn't any different if you have one or more pools configured...I was really hoping that this would work If any of you are still having stale share issues, would you mind taking a capture of the network traffic? On the proxy host, issue this as root: Code: tcpdump -w proxy.pcap -s 65535 'port 80 or portrange 8332-8337' Then email me the proxy.pcap file after an hour or two. Note that this file will contain all of the traffic on these ports and this means that I will have access to your proxy worker passwords as well as your pool worker passwords. You may certainly encrypt the file to my GPG key if you are worried about the file being intercepted. (Specifically, note that if you talk to a bitcoind from the web server, this will expose your RPC password! So be especially careful in this case. I'm not going to misuse this information, but... well, if I were you, I wouldn't trust me blindly either. ;)) If possible, capture traffic from the workers to the pools directly, without the proxy, and email me that as a separate file. Comparing the two files will help me determine what's going on. Even better, if you're using a pool supporting long-polling, have one miner going through the proxy and at the same time another one not using the proxy, and mail me both files. Seeing how the long-polling requests return in both cases (they should be identical...) will help me diagnose this. Thanks all for the feedback, donations, and support! It's heartening to know that the software is fulfilling its purpose and that people find it helpful. :) Title: Re: Flexible mining proxy Post by: tito13kfm on June 13, 2011, 05:24:49 AM Rebooted after crash and now Worker Status table on main dashboard is completely blank. Recent work submissions and recent failed submissions still show up just fine in the top 2 tables, but the bottom table is blank.
Any ideas? Edit: I was also having connection issues. I exported my workers and pools settings. Imported the original schema back, and imported those saved settings. All is well now. Title: Re: Flexible mining proxy Post by: cdhowie on June 13, 2011, 02:08:53 PM Rebooted after crash and now Worker Status table on main dashboard is completely blank. Recent work submissions and recent failed submissions still show up just fine in the top 2 tables, but the bottom table is blank. Sounds like the MySQL database got corrupted. Which is weird, since table logs are supposed to keep that from happening. :/Any ideas? Edit: I was also having connection issues. I exported my workers and pools settings. Imported the original schema back, and imported those saved settings. All is well now. Title: Re: Flexible mining proxy Post by: teknohog on June 13, 2011, 09:54:12 PM Has anyone got this working with Lighttpd (under Gentoo)? I am trying to rule out wider system-level errors before digging deeper into this code. Basically, my clients (Phoenix) are connecting to the proxy, but they are not getting any work. My PHP setup should be OK as I use it for other web content, and the admin interface seems to work fine. I have set this up in a separate port so there are no subdirectory issues for clients.
Title: Re: Flexible mining proxy Post by: Freakin on June 13, 2011, 09:59:27 PM Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or '}' in /home/REDACTED/public_html/admin/index.php on line 28
Hmm, not sure why i'm getting this. This is on the admin page. It complains about line 125 on the root Title: Re: Flexible mining proxy Post by: cdhowie on June 14, 2011, 02:05:30 PM Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or '}' in /home/REDACTED/public_html/admin/index.php on line 28 Which version of PHP? It's possible that your version does not support function access modifiers.Hmm, not sure why i'm getting this. This is on the admin page. It complains about line 125 on the root Title: Re: Flexible mining proxy Post by: Freakin on June 14, 2011, 04:23:10 PM Parse error: syntax error, unexpected T_STRING, expecting T_OLD_FUNCTION or T_FUNCTION or T_VAR or '}' in /home/REDACTED/public_html/admin/index.php on line 28 Which version of PHP? It's possible that your version does not support function access modifiers.Hmm, not sure why i'm getting this. This is on the admin page. It complains about line 125 on the root 4.4.8 5.x required? Title: Re: Flexible mining proxy Post by: exahash on June 14, 2011, 05:20:41 PM Do you want patches? I'm testing putting 50 miners through this proxy and was running into stability issues. I have mysql on a separate machine and am used to that being a bottleneck, so I looked into making it use persistent connections. I am not familiar with the PDO extension, but found the instantiator in common.inc.php on line 31; I added array(PDO::ATTR_PERSISTENT => true) to the new call and this took care of my problems.
The whole line now looks like this: return new PDO($BTC_PROXY['db_connection_string'], $BTC_PROXY['db_user'], $BTC_PROXY['db_password'], array(PDO::ATTR_PERSISTENT => true) ); I now have 32 connections from the proxy web server staying open (which is what I want). Hopefully this helps someone. Title: Re: Flexible mining proxy Post by: btcLeger on June 14, 2011, 08:23:40 PM 4.4.8 5.3 at least.5.x required? Title: Re: Flexible mining proxy Post by: Freakin on June 14, 2011, 11:17:57 PM Cool got it working on a Centos 5.6 VM w/ 5.3.6
had a bit of difficulty getting PDO installed but it went ok. looking forward to getting this working! Title: Re: Flexible mining proxy Post by: Freakin on June 14, 2011, 11:45:26 PM There should be a way to copy worker/pool configs
so instead of setting up 10x pools on 10x workers you can set it up once then copy it to the rest, then go in and change the logins as necessary. Or a way to use a variable in the config so you build X workers with the suffix X changed on each one Title: Re: Flexible mining proxy Post by: krom on June 14, 2011, 11:46:42 PM All working "quite" well with 1 miner and 1 pool.
Some things i've noticed:
I'll dig into the problem further, but have no time to do so yet. Besides that: Good work and keep it up! When i'll have more than some fractions of BC, i'll donate to you. :) Update: Long poll seem to be okay. Problem 3 was solved by adding 'http://' to the pool's url. Won't work without. Title: Re: Flexible mining proxy Post by: Freakin on June 15, 2011, 03:25:38 AM Really wish there was a way to monitor anything on solo mining besides getwork w/ the proxy
Title: Re: Flexible mining proxy Post by: kripz on June 17, 2011, 01:45:14 PM Testing out now, looks fairly solid.
What sort of values can i set for 'average_interval'? and what effects will it have? Also, please add rejected per hour and efficiency. Edit: I get alot of miner idle using this... Title: Re: Flexible mining proxy Post by: Freakin on June 17, 2011, 04:11:46 PM I had problems with this once i connected 4 machines to it (6 total miners).
Tried it soloing and on bitparking and while I slept and found that I wasn't getting what I was supposed to. turned off both namecoin options and was hitting slush and deepbit and getting about 2/3 of the shares I should have connections to the proxy started timing out, so restarting apache fixed it, but i didn't trust it enough to keep going This was on a VM w/ 1GB of ram What kind of hardware are you guys running on? Title: Re: Flexible mining proxy Post by: pwnyboy on June 17, 2011, 06:34:21 PM Really wish there was a way to monitor anything on solo mining besides getwork w/ the proxy There is indeed, it's called pushpool. But it's a real PITA to build. Once you have it going, it'll log shares to a text file, which you can use to dig out the info you want/need. You can also make it log shares to the database backend (SQLite, MySQL etc). Title: Re: Flexible mining proxy Post by: Freakin on June 17, 2011, 08:05:35 PM Really wish there was a way to monitor anything on solo mining besides getwork w/ the proxy There is indeed, it's called pushpool. But it's a real PITA to build. Once you have it going, it'll log shares to a text file, which you can use to dig out the info you want/need. You can also make it log shares to the database backend (SQLite, MySQL etc). yeah I think I'm going to set up a POC pool on a VM for personal use and play around. After 3 days soloing namecoin with 2GH and not finding a single block I don't plan on doing that again anyways... Wouldn't be so bitter if I hadn't found 23 blocks in the week before (half at ~3k difficulty) for BitParking Title: Re: Flexible mining proxy Post by: carlo on June 17, 2011, 10:00:50 PM Im testing your proxy now since a week. they main problem seems to be the apache.
i figured out that it does reject lesser shares if i run this settings for the apache: KeepAlive=On KeepAliveTimeout=60 MaxKeepAliveRequests=250 MaxClients=100 I run 8 GPU miners. But its still not the same reject rate if i connect directly to a pool. Im happy if somebody has other ideas, cause the fail-over is so nice in days of ddos. Problems during the last 2 days: Code: ERROR: Cannot connect to Bitcoin: Bitcoin returned error message: Work not found in proxy database. cheers Title: Re: Flexible mining proxy Post by: kripz on June 18, 2011, 02:10:38 AM Is it safe to drop the submitted work data? I want to reset my stats so i can use it for benchmarking.
Also what is average_interval? EDIT: Once a pool goes down, does it auto switch as a fail over and then switch back once it's back up? EDIT: How do you test if the pool is down? Mine seems to be constantly cycling through the pools even though they seem to be ok. Title: Re: Flexible mining proxy Post by: krom on June 19, 2011, 12:29:35 PM EDIT: Once a pool goes down, does it auto switch as a fail over and then switch back once it's back up? If a pool does not respond within a given time, which is about 3 seconds, the proxy will continue with the next pool in the priority list.EDIT: How do you test if the pool is down? Mine seems to be constantly cycling through the pools even though they seem to be ok. Title: Re: Flexible mining proxy Post by: kripz on June 19, 2011, 12:31:02 PM So when does it switch back to the highest priority?
Title: Re: Flexible mining proxy Post by: krom on June 19, 2011, 12:53:25 PM Every time the client requests new work from the proxy, so every 40-50 seconds, this will happen:
The proxy asks the first pool in his priority list for new work. If that pool does not respond within X seconds, the proxy goes on and asks the next pool in his priority list. This will repeat until a pool does respond and delivers new work which the proxy can relay to the client, or the end of the prioriry list is reached. Title: Re: Flexible mining proxy Post by: teknohog on June 21, 2011, 03:26:13 PM Has anyone got this working with Lighttpd (under Gentoo)? I am trying to rule out wider system-level errors before digging deeper into this code. Basically, my clients (Phoenix) are connecting to the proxy, but they are not getting any work. I got exactly the same error with Apache, and I even did some debugging via tcpdump. It seemed like the proxy was not even asking for any work from the pools. The solution turned out simple, I had to set allow_url_fopen = On in php.ini. So, lighttpd (which uses PHP via fastcgi) seems to work. As I just got it working, I have no long-term data which might be important for long polling, but so far things are OK :) Title: Re: Flexible mining proxy Post by: lachesis on June 22, 2011, 03:03:05 AM I'm getting the response
Code: {"error":"No enabled pools responded to the work request.","result":null,"id":1}How can I troubleshoot this? Title: Re: Flexible mining proxy Post by: lebuen on June 22, 2011, 01:25:00 PM Unfortunately, I get extreme stale rates (>10%) when using the proxy on my local mining-rig. But I'm behind a firewall - could it be that the Mining Proxy needs to be listening on port 80 from the "outside" to be able to receive LP-queries?
Title: Re: Flexible mining proxy Post by: kripz on June 22, 2011, 02:25:26 PM Randomly get this using DiabloMiner. Hashkill will just crash. Phoenix seems to be 100% stable, never crashes but i dont doubt it runs into the same problem behind the scenes.
Code: [23/06/11 12:20:47 AM] DEBUG: Attempt 26 found on Juniper (#1) Title: Re: Flexible mining proxy Post by: cdhowie on June 22, 2011, 10:17:37 PM Do you want patches? I'm testing putting 50 miners through this proxy and was running into stability issues. I have mysql on a separate machine and am used to that being a bottleneck, so I looked into making it use persistent connections. I am not familiar with the PDO extension, but found the instantiator in common.inc.php on line 31; I added array(PDO::ATTR_PERSISTENT => true) to the new call and this took care of my problems. Sounds like a good idea. I'll test this on my setup and commit if there are no issues.I now have 32 connections from the proxy web server staying open (which is what I want). There should be a way to copy worker/pool configs I might add one or more of these ideas in the future. Right now I'm trying to keep the number of features low and work on the ones that are having trouble.so instead of setting up 10x pools on 10x workers you can set it up once then copy it to the rest, then go in and change the logins as necessary. Or a way to use a variable in the config so you build X workers with the suffix X changed on each one Problem 3 was solved by adding 'http://' to the pool's url. Won't work without. Yup. :) I'll add something like this to the readme, and maybe add some validation too so that it will bail unless it sees a scheme in the URL.Really wish there was a way to monitor anything on solo mining besides getwork w/ the proxy I've no clue what you're asking for here.What sort of values can i set for 'average_interval'? and what effects will it have? When reporting on the number of shares submitted and the miner speed, this is the window of time it will look back to gather data. If you set it longer, the query will take longer and will generally even out more; if you set it shorter, the query will run faster but will result in wild fluctuations of the worker speed (for example) as the worker has lucky and unlucky periods.It is purely used for reports; it will have no effect on mining. Also, please add rejected per hour and efficiency. Another user has a nearly-ready patch that adds a lot of this information.connections to the proxy started timing out, so restarting apache fixed it, but i didn't trust it enough to keep going I'm running with 768MB, most of which is in use by a Minecraft server...This was on a VM w/ 1GB of ram What kind of hardware are you guys running on? Try increasing the number of worker children in your Apache configuration. With more miners, you need more workers to keep up with the demand. If Apache hits its limit then it will simply stop responding to requests. I'm getting the response If you can, run a sniffer on the web server and see if it even tries to connect. If it does, see if you can diagnose the problem from the content of the HTTP conversation.Code: {"error":"No enabled pools responded to the work request.","result":null,"id":1}How can I troubleshoot this? If it does not try to connect:
Unfortunately, I get extreme stale rates (>10%) when using the proxy on my local mining-rig. But I'm behind a firewall - could it be that the Mining Proxy needs to be listening on port 80 from the "outside" to be able to receive LP-queries? No, the proxy needs no ports open except for the miners themselves. It may be worth running a packet sniffer and seeing if the LP requests are actually making it through. If you are not running on Apache, some config tweaks may be necessary to get LP to work at all.Title: Re: Flexible mining proxy Post by: lachesis on June 22, 2011, 11:22:14 PM I'm getting the response If you can, run a sniffer on the web server and see if it even tries to connect. If it does, see if you can diagnose the problem from the content of the HTTP conversation.Code: {"error":"No enabled pools responded to the work request.","result":null,"id":1}How can I troubleshoot this? If it does not try to connect:
I wonder if there is an issue here: this server has two IPs on the same subnet. Is there a way to prioritize your proxy to choose one of them exclusively? I'm afraid it's expecting an answer on eth0 but receiving one on eth1 and not properly handling it. Title: Re: Flexible mining proxy Post by: cdhowie on June 23, 2011, 01:40:13 PM My client tried to connect twice. The proxy tried to connect a number of times and received several responses, but forwarded none on to my miner. If you can save a pcap file and email it to me, maybe I could have a look and see what's going on.I wonder if there is an issue here: this server has two IPs on the same subnet. Is there a way to prioritize your proxy to choose one of them exclusively? I'm afraid it's expecting an answer on eth0 but receiving one on eth1 and not properly handling it. The kernel should take care of routing stuff correctly. If traffic is coming back on the wrong interface, that would be a problem with an upstream router and not any particular configuration on your box. If you can load other websites fine then the network config shouldn't be interfering.Title: Re: Flexible mining proxy Post by: carlo on June 23, 2011, 01:51:22 PM Hello,
the Dashboard is getting slower and slower with the whole data. Does sombody have a quick and dirty soulution to erase the "submitted_work" and the "work_data" every 24h ? I thougt about dropping every 24h the data from hour 0 -> 23 ... so i keep some stats (like hashrate of the last hour). If i make my cronjob working ... i let you know ... if somebody does allready have one please let me know :) cheers Title: Re: Flexible mining proxy Post by: wyze on June 23, 2011, 02:48:44 PM Hello, the Dashboard is getting slower and slower with the whole data. Does sombody have a quick and dirty soulution to erase the "submitted_work" and the "work_data" every 24h ? I thougt about dropping every 24h the data from hour 0 -> 23 ... so i keep some stats (like hashrate of the last hour). If i make my cronjob working ... i let you know ... if somebody does allready have one please let me know :) cheers How big is your database? Mine is at 19MB and my dashboard still loads in under a second. I have about 4-5 days worth of data in there. This is the same speed I get when accessing the dashboard from work as well, while it is hosted on a little web server in my basement. Title: Re: Flexible mining proxy Post by: cdhowie on June 23, 2011, 03:29:16 PM Also, please add rejected per hour and efficiency. Another user has a nearly-ready patch that adds a lot of this information.Title: Re: Flexible mining proxy Post by: cdhowie on June 23, 2011, 03:33:50 PM the Dashboard is getting slower and slower with the whole data. There are some indexes missing from the schema. I need to add those and provide a migration script for older installs, but this will take a bit of work and I haven't finished it yet. Once this is done, running the upgrade script will create the indexes and the dashboard will load much faster.Does sombody have a quick and dirty soulution to erase the "submitted_work" and the "work_data" every 24h ? See the "Database maintenance" section of the readme (https://github.com/cdhowie/Bitcoin-mining-proxy/blob/master/README.markdown).I thougt about dropping every 24h the data from hour 0 -> 23 ... so i keep some stats (like hashrate of the last hour). Title: Re: Flexible mining proxy Post by: Salain on June 23, 2011, 04:34:39 PM Also, please add rejected per hour and efficiency. Another user has a nearly-ready patch that adds a lot of this information.Based on context, I think he's referring to a % good/stale shares stat. So (#good shares) / (#good + #stale shares). Any better efficiency metric would require you to load the current difficulty... Title: Re: Flexible mining proxy Post by: PulsedMedia on June 23, 2011, 06:34:46 PM Can this include some actual proxy kind of features, ie. caching?
So that this could be used behind a flaky internet connection to keep miners 100% at work, if the flakyness is in the seconds range? Or does this try to connect the pool directly on the same instance as the miner connects to etc.? What i'm wondering, would this enable me to run miners behind a 3G connection ;) I'll try this out soon, and i will then optimize the mysql db for you after it starts to slow off (one my of specialties is optimization, esp. mysql), to increase the performance. Not sure how well you've done performance wise, but we can make it work even with Tb sized datasets if you want to. Contact me via PM or freenode #PulsedMedia if you want to chat about it. I have a mining hosting idea (in the mining hw section) for which using this as intermediary could be PERFECT. Just need more stats etc. and eventually tagging, grouping nodes/gpus etc. "advanced workflow features", and RESTful API (or more like my own relaxed version using of which is way easier than full REST spec). Sorry for the basic questions, just yet is not time for me to research into this, i got other dev. tasks i need to finish first before i get to play around with this :) Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 01:07:36 AM Installed and seems to work. Tho weighting seems to be quite a bit off (haven't looked at that portion of code yet).
Quick glance at DB shows no reason why it should be slow, if queries match. So those who are having speed issues probably have bad MySQL config just. MySQL scales really efficiently and i don't see immediate reasons why this would be slow, but have to wait for data set to increase. If anyone got say 8G+ dataset they don't mind sharing, i would be willing to look into it. Or any size with which someone is having some serious perf issues. Code should be PLENTY more commented btw ;) Title: Re: Flexible mining proxy Post by: kjj on June 24, 2011, 01:10:22 AM Installed and seems to work. Tho weighting seems to be quite a bit off (haven't looked at that portion of code yet). Quick glance at DB shows no reason why it should be slow, if queries match. So those who are having speed issues probably have bad MySQL config just. MySQL scales really efficiently and i don't see immediate reasons why this would be slow, but have to wait for data set to increase. If anyone got say 8G+ dataset they don't mind sharing, i would be willing to look into it. Or any size with which someone is having some serious perf issues. Code should be PLENTY more commented btw ;) Check out the join that creates the status display. Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 01:21:03 AM Installed and seems to work. Tho weighting seems to be quite a bit off (haven't looked at that portion of code yet). Quick glance at DB shows no reason why it should be slow, if queries match. So those who are having speed issues probably have bad MySQL config just. MySQL scales really efficiently and i don't see immediate reasons why this would be slow, but have to wait for data set to increase. If anyone got say 8G+ dataset they don't mind sharing, i would be willing to look into it. Or any size with which someone is having some serious perf issues. Code should be PLENTY more commented btw ;) Check out the join that creates the status display. LOL! Yeah that would cause some serious issues (first query in admin/index.php) 3rd query is a monstrosity. Well there is the problem, using dynamic (on the fly created) tables etc. These queries is almost like SELECT *, wonder if they ever hit any indexes ... In any case need bigger data set before i can optimize them properly. But on first query FROM ( XXXX XXXX ) sw should be change to just choose the damn table, moving the LIMIT 10 at the whole query. Inner joins -> just choose from multiple tables, use same p.id = sw.pool_id So something like SELECT w.name AS worker, p.name AS pool, p.pool_id AS poolId, p.id AS poolId, sw.worker_id AS workerId [AND SO ON FOR ALL FIELDS REQUIRED] FROM submitted_work sw, pool p, woker WHERE pool.id = sw.pool_id AND worker.id=sw.workerid ORDER BY LIMIT INNER JOINS are like SELECT *, if i recall right :P Bottomline is that carefully crafted queries can do a full text match scored on a 100G dataset with multiple text matches, joining multiple tables (not using JOIN clause tho, but still called joining), on a Quad Core Xeon with 16G ram (pre-i7 xeon) in well under 100ms. (Target was 15searches/sec, achieved peak was above that, and bottleneck was actually the PHP syntax parsing for real world scenario which transformed our simplified custom query language into a mysql query, for easier use for the end users) Title: Re: Flexible mining proxy Post by: kjj on June 24, 2011, 01:47:12 AM Meh. Took me like 5 minutes to modify the work_data table, add new history tables, write a cron job to rotate the records out, and post my scripts (http://forum.bitcoin.org/index.php?topic=5506.msg164222#msg164222). Bonus: I didn't even have to think of any clever SQL tricks.
Title: Re: Flexible mining proxy Post by: nick5429 on June 24, 2011, 03:15:04 AM Just as another datapoint: loading Multipool into the proxy doesn't work for me, either. My miner sits there with 0Mhash/sec. Directly mining at Multipool directly works fine for me.
Also, I get weird authentication issues with hashkill. About half the time it will fail the initial auth, the other half it authenticates, does a little bit of work, and then claims it had authentication issues: Code: [hashkill] Version 0.2.5 Phoenix miner works fine for me with the same proxy settings. Title: Re: Flexible mining proxy Post by: nick5429 on June 24, 2011, 03:35:12 AM In case anyone is interested, I was able to get the mining proxy working on my Dreamhost shared server with a bit of tweaking. This should be equally applicable to other hosts that may impose similar restrictions.
Since Dreamhost forces PHP-CGI (http://blog.dreamhosters.com/kbase/index.cgi?area=2933) (rather than mod_php) on its shared hosting users, the .htaccess tricks don't work and the PHP_AUTH_USER / PHP_AUTH_PW variables in the scripts are empty. First, follow all the basic setup instructions in the standard Proxy guide. Then, you need to make sure that magic_quotes_gpc and allow_url_fopen are set to the defaults that we need, otherwise I don't think there's anything you can do. To do this, create a file called phpinfo.php in your htdocs with the following code: Code: <?php Search for magic_quotes_gpc (it needs to be off) and allow_url_fopen (it needs to be on). If these settings don't match, getting the proxy working is beyond the scope of this tweak. If those settings are okay, then we can start with the tweaking. The way the script is written won't allow you to authenticate, but everything else works fine. To fix this... Replace the contents of your .htaccess file with the following: Code: Options -Indexes Then, edit your common.inc.php file to include this code inside the "do_admin_auth()" function: Code: if (preg_match('/Basic\s+(.*)$/i', $_SERVER['HTTP_AUTHORIZATION'], $matches))This function will now look something like: Code: function do_admin_auth() {EDIT: You also need to include this code snippet near the top of index.php Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 06:53:57 AM Meh. Took me like 5 minutes to modify the work_data table, add new history tables, write a cron job to rotate the records out, and post my scripts (http://forum.bitcoin.org/index.php?topic=5506.msg164222#msg164222). Bonus: I didn't even have to think of any clever SQL tricks. I looked at the linked post... There is a bunch of bad things i could now say about that cron job ... Slow, prone to fault gum h4x to fix performance issues? People! This is how we create bloatware! Fix the problem, not the symptoms! Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 08:07:43 AM wow, there is many kinds of bad things going on code QA wise.
This code is what i call confused newbie abstraction. Using functions to output HTML in a view (already outputting HTML)? Yup! Using abstration function to output a simple form image button? Yup! Doing insane joined and dynamic tables query for a few fields of simple data? Yup! Anyways, 2 first queries optimized, one less table to lookup, indices being actually hit. Far from being completely optimized (still a filesort happening!), but should proof to be an order of magnitude faster, DESPITE hitting more rows. I have NO way to test however, nor profile correctly due to lack of dataset size, so measurements would be less than error of margin. Replace admin/index.php 2 first queries with those found at: http://pastebin.com/hwncLV1w NOTE: I have not done proper testing, results seem to be correct tho :) EDIT: Looking into 3rd query now, it's worse than expected. Will rework complete query and accompanying view portion. It actually checks all rows multiple times, assuming MySQL realizes how to optimize. It makes 5 dynamic (on-the-fly created) tables, hits *ALL* rows on submitted work, creates temp tables (lack of suitable index), 3 times filesort, 13 tables total, and if i interpret correctly total rows to go through is in the range of hundreds of thousands rows or more with my 2526 shares submitted test data! :O Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 10:38:55 AM well, i fooled around, according to profiling my changes are beneficial to the 3rd query but due to the really tiny sample data set i got the actual measurements mean nothing, the spent time might just be all in parsing the query and optimization engine, not the actual query.
So take even the earlier ones with a grain of salt: I've got no clue of the impact as i can't actually measure the difference, even tho i'm using for testing a ancient Athlon XP 1900+ ... As soon as i got enough data i will see again how my changed queries affect the performance. Title: Re: Flexible mining proxy Post by: kripz on June 24, 2011, 11:21:39 AM I will test, got a fix for the 3rd?
Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 11:28:56 AM I will test, got a fix for the 3rd? Yes, but approach was wrong so it might actually perform worse. Try also how these affect: http://pastebin.com/kcWN9gPH How many rows in your submitted_work and work_data tables? Title: Re: Flexible mining proxy Post by: kripz on June 24, 2011, 11:38:52 AM 30k and 50k
Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 11:41:09 AM 30k and 50k That might give some hint of the effect, but really we start seeing difference at around 10x that size ... Title: Re: Flexible mining proxy Post by: wyze on June 24, 2011, 12:09:33 PM Anyways, 2 first queries optimized, one less table to lookup, indices being actually hit. Far from being completely optimized (still a filesort happening!), but should proof to be an order of magnitude faster, DESPITE hitting more rows. I have NO way to test however, nor profile correctly due to lack of dataset size, so measurements would be less than error of margin. Replace admin/index.php 2 first queries with those found at: http://pastebin.com/hwncLV1w Looking at the image below, I fail to see how your 'optimized queries' are better. Maybe I just don't have a full grasp of the EXPLAIN command from phpMyAdmin, but it looks to me as your query scans all rows in the index, which comes out to 2436 with my current data set. That is way more than what the query was scanning before. Please correct me if I am wrong in reading the output. I only looked at the first query and stopped to get clarification. https://i.imgur.com/G4bJM.png Title: Re: Flexible mining proxy Post by: PulsedMedia on June 24, 2011, 12:13:50 PM those are not my queries! :O
Here they are: $viewdata['recent-submissions'] = db_query($pdo, ' SELECT w.name AS worker, p.name AS pool, sw.result AS result, sw.time AS time FROM submitted_work sw, pool p, worker w WHERE p.id=sw.pool_id AND w.id = sw.worker_id ORDER BY sw.time DESC LIMIT 10 '); $viewdata['recent-failed-submissions'] = db_query($pdo, ' SELECT w.name AS worker, p.name AS pool, sw.result AS result, sw.time AS time FROM submitted_work sw, pool p, worker w WHERE sw.result=0 AND p.id = sw.pool_id AND w.id = sw.worker_id ORDER BY sw.time DESC LIMIT 10 '); FYI, i have not actually even checked what they do exactly, just wrote simplified queries. EDIT: I just checked, and interestingly doesn't hit index, which is really wierd. Oh well, i check why at better time. Title: Re: Flexible mining proxy Post by: wyze on June 24, 2011, 12:36:05 PM those are not my queries! :O Didn't explain the image, lol. The top results were from your optimized version of the first query and the bottom result was from the query as it is now. It may be displayed differently by phpMyAdmin, but it still runs the same. Like I said, I didn't go past comparing the first query until I got some clarification on if I was reading the results of EXPLAIN correctly. Title: Re: Flexible mining proxy Post by: kripz on June 24, 2011, 12:39:52 PM Why is my proxy all of a sudden not submiting shares?
If i point my miner to the pool directly all is fine. EDIT: while im here, can somebody change the last submitted/request to say "X days/seconds/hours ago" Title: Re: Flexible mining proxy Post by: cdhowie on June 24, 2011, 03:59:51 PM Can this include some actual proxy kind of features, ie. caching? A non-PHP getwork backend is planned to resolve this and other LP issues. PHP is not well-suited to this kind of task.So that this could be used behind a flaky internet connection to keep miners 100% at work, if the flakyness is in the seconds range? Check out the join that creates the status display. LOL! Yeah that would cause some serious issues (first query in admin/index.php) 3rd query is a monstrosity. Well there is the problem, using dynamic (on the fly created) tables etc. These queries is almost like SELECT *, wonder if they ever hit any indexes ... Well there is the problem, using dynamic (on the fly created) tables etc. This in particular made me lol. Subqueries can be an effective optimization technique if you know how to use them correctly, and any DBA knows that. In the "last 10 submissions" cases, MySQL creates a plan that executes the LIMIT after the JOIN, which results in a full table scan of work_data/submitted_work. Querying those tables in a subquery with LIMIT forces it to execute the limit first, which results in a very fast join. This was a pain point until I refactored the query to use a subquery to derive the data tables. Please know WTF you are talking about and use EXPLAIN, kthx.EDIT: I just checked, and interestingly doesn't hit index, which is really wierd. Oh well, i check why at better time. Exactly. MySQL doesn't use the indexes in this case because it has decided to apply the LIMIT after the joins. So it does a table scan. And you don't need indexes to do a table scan, now do you? Essentially, MySQL's query analyzer sucks, and the subquery is the workaround.So let's do some investigation: Code: mysql> SELECT COUNT(*) FROM work_data; After executing the dashboard status query: Code: 3 rows in set (0.11 sec) EXPLAIN on the dashboard status query: Code: +----+-------------+----------------+--------+------------------------------------------------+-------------------------+---------+---------------------------------+------+----------------------------------------------+ Sorry, the query is fine. A bit big, but it's attempting to reduce quite a bit of data down to three rows. So meh. You do better and I'll take a patch. Title: Re: Flexible mining proxy Post by: cdhowie on June 24, 2011, 04:59:49 PM Note that a few indexes that are required to prevent table scans are not present in the schema; these were added later and I don't have a database migration script just yet, so it's expected that these queries will run a bit slow unless you've manually created the needed indexes. I just took a few minutes to knuckle-down and get this done (https://github.com/cdhowie/Bitcoin-mining-proxy/commit/022d547f4d9691b7935fe15d71b85ecd48b56adb). I recommend that all users get the latest from master and apply the DB migration script. Dashboard performance should significantly improve (to sub-second response times).Title: Re: Flexible mining proxy Post by: cdhowie on June 24, 2011, 05:08:54 PM Installed and seems to work. Tho weighting seems to be quite a bit off (haven't looked at that portion of code yet). There is no such thing as "weighting." Did you read the readme? Particularly the section on how priority works?Code should be PLENTY more commented btw ;) Probably, particularly index.php. The rest is fairly well-separated and should be readable as-is. My philosophy on comments is that if you have to comment a lot, your code doesn't explain itself well enough and should be refactored.Title: Re: Flexible mining proxy Post by: nick5429 on June 24, 2011, 07:46:54 PM Has anyone successfully used Multipool as a target pool through the Flexible Mining Proxy? cdhowie, have you tried it?
I've seen at least 2-3 reports in addition to mine claiming it doesn't work, and nobody saying it works for them. No signup or setup is needed -- just connect to http://multipool.hpc.tw:8337 with a Bitcoin address as your username, and anything for your password. I'd be interested to hear if multipool is broken for everyone.... Title: Re: Flexible mining proxy Post by: wyze on June 24, 2011, 09:07:31 PM Has anyone successfully used Multipool as a target pool through the Flexible Mining Proxy? cdhowie, have you tried it? I've seen at least 2-3 reports in addition to mine claiming it doesn't work, and nobody saying it works for them. No signup or setup is needed -- just connect to http://multipool.hpc.tw:8337 with a Bitcoin address as your username, and anything for your password. I'd be interested to hear if multipool is broken for everyone.... I will take a look at this for you over the weekend. Title: Re: Flexible mining proxy Post by: cdhowie on June 24, 2011, 09:10:00 PM Has anyone successfully used Multipool as a target pool through the Flexible Mining Proxy? cdhowie, have you tried it? I've seen at least 2-3 reports in addition to mine claiming it doesn't work, and nobody saying it works for them. No signup or setup is needed -- just connect to http://multipool.hpc.tw:8337 with a Bitcoin address as your username, and anything for your password. I'd be interested to hear if multipool is broken for everyone.... I will take a look at this for you over the weekend. Yes, if nobody can connect to multipool then it's likely some communication issue caused by the proxy code. If wyze figures it out then he'll probably fix it (he has a fork over on Github you could use until I merge his fix) and if not I'll have a look sometime this weekend too. Title: Re: Flexible mining proxy Post by: nick5429 on June 25, 2011, 04:53:24 AM Re: the multipool issue, I added some debugging dumps to the place_json_call function.
Slush's response to a getwork request looks like: Code: {"id": "1", "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "data": "00000001575fd9ef5901da864fff435588650d21f8619fa33cb3510f000006d2000000005fc23ab758404b1f6067f111978fcbcc1d4a227a377315ec590f71ed04bc0d8a4e0567941a0c2a1200000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "bd4be1f1643712bd150248e7e2d9ace616611d3b9e8ea76b6b76a0180f6b00ce", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000"}, "error": null}Whereas Multipool's looks like: Code: {"error":null,"id":"json","result":{"target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","midstate":"6dfada9f763c6ae458d123a4a9e71a56bf5fc65946d7b40c8b679e865d7ebad6","data":"00000001575fd9ef5901da864fff435588650d21f8619fa33cb3510f000006d200000000a006b13a7db011c6779926e01ec4f67bc3246bc44419c1b4d204c0650be396a64e0567021a0c2a1200000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000","hash1":"00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000"}}I haven't investigated enough to determine if this is the issue, but any chance it's due to Multipool's "id" field being non-numeric? edit: Doubt that's it; slush's pool sometimes returns id="json" as well. Either way, it appears (to my eyes, which aren't familiar with the pool mining protocol) that Multipool is returning valid data, but it isn't making its way into the work_data table. There are zero entries with the pool_id that corresponds to Multipool. edit2: the JSON response when Multipool is enabled is: {"error":"No enabled pools responded to the work request.","result":null,"id":1} Hope this data helped... Title: Re: Flexible mining proxy Post by: wyze on June 25, 2011, 06:25:10 AM It looks like when we do a getwork request to MultiPool, they are also returning extra headers after the JSON string. I am not sure if they are checking user agent or what on their side. I have a work around so that the data makes it into the work_data table. Now, it still looks like it has a problem submitting the data. I will look into this later in the day (Saturday), after I get some sleep. I'm thinking it is going to be along the same lines, where as, after we submit the work, MultiPool returns JSON + headers.
EDIT: I have a workable solution. I was able to successfully submit work with the proxy to MultiPool. I will discuss the fix further with cdhowie in the morning and see how he wants to go about implementing it. Title: Re: Flexible mining proxy Post by: cdhowie on June 25, 2011, 08:20:54 PM .
Title: Re: Flexible mining proxy Post by: kripz on June 26, 2011, 04:56:38 AM Well i think i found the bug cdhowie, i sent you a packet dump yesterday.
http://forum.bitcoin.org/index.php?topic=1721.msg285078#msg285078 lol pool went down at time of update, what a coincedence... Cant be me.. 90% shares arent being submitted? Bug? Miner 1 mhash 177.7/178.9 | a/r/hwe: 1/0/0 | ghash: 110.2 | fps: 30.0 Miner 2 mhash 819.5/826.6 | a/r/hwe: 0/3/0 | ghash: 114.0 113.3 112.3 | fps: 30.2 Quote [24/06/11 8:49:58 PM] DEBUG: Attempt 77 found on Cypress (#3) [24/06/11 8:50:02 PM] DEBUG: Attempt 78 found on Cypress (#2) [24/06/11 8:50:07 PM] DEBUG: Attempt 79 found on Cypress (#2) [24/06/11 8:50:12 PM] DEBUG: Attempt 80 found on Cypress (#3) [24/06/11 8:50:13 PM] DEBUG: Attempt 81 found on Cypress (#2) [24/06/11 8:50:14 PM] DEBUG: Attempt 82 found on Cypress (#3) [24/06/11 8:50:15 PM] DEBUG: Attempt 83 found on Cypress (#2) [24/06/11 8:50:18 PM] DEBUG: Attempt 84 found on Cypress (#3) [24/06/11 8:50:21 PM] DEBUG: Attempt 85 found on Cypress (#1) [24/06/11 8:50:23 PM] DEBUG: Attempt 86 found on Cypress (#2) [24/06/11 8:50:33 PM] DEBUG: Attempt 87 found on Cypress (#2) both updated to latest git Just updated windows machine Quote [24/06/11 8:56:17 PM] DEBUG: Attempt 3 found on Cayman (#2) [24/06/11 8:56:18 PM] DEBUG: Attempt 4 found on Cayman (#2) [24/06/11 8:56:20 PM] DEBUG: Attempt 5 found on Cayman (#2) [24/06/11 8:56:26 PM] DEBUG: Forcing getwork update due to nonce saturation [24/06/11 8:56:31 PM] DEBUG: Forcing getwork update due to nonce saturation [24/06/11 8:56:32 PM] DEBUG: Attempt 6 found on Cayman (#2) [24/06/11 8:56:32 PM] DEBUG: Attempt 7 found on Cayman (#2) [24/06/11 8:56:34 PM] DEBUG: Attempt 8 found on Cayman (#2) [24/06/11 8:56:38 PM] DEBUG: Attempt 9 found on Cayman (#2) mhash 364.5/362.8 | a/r/hwe: 0/1/0 | ghash: 30.1 | fps: 30.4 Nothing is being submitted? EDIT: now how do i go back to the old version? Nope, it's Diablominer and/or flexible proxy (though i never touched the proxy). It will find a few results, submit one or two and say accepted. After that "Attempt found" but never submit it? Phoenix works 100% rock solid The proxy probably does not correctly support things DiabloMiner does, such as time incrementing and returning multiple nonces for the same getwork over short periods. It looks like the sendwork thread is being choked by the proxy. So, clearly, its a proxy bug. Edit: IIRC i get the same behaviour with hashkill. Title: Re: Flexible mining proxy Post by: kripz on June 26, 2011, 08:34:45 AM Since the recent dashboard changes, the recent rejected is not working properly, seems to be missing alot of records.
This will display "5 seconds/minutes/days/weeks/years ago" instead of a full Date Time string. Unfortunately the database doesnt store milliseconds. http://img860.imageshack.us/img860/7343/unled1yf.jpg Code: (common.inc.php) function human_time($difference) Code: (config.inc.php) # Custom php date format or "human" for "x days/minutes/seconds ago" Title: Re: Flexible mining proxy Post by: wyze on June 26, 2011, 03:00:47 PM Since the recent dashboard changes, the recent rejected is not working properly, seems to be missing alot of records. No code was changed that would affect this. I simply removed the Result column as it was not really needed since they were all rejected anyways. :) feature request:
These look pretty good. Please open an issue here (https://github.com/cdhowie/Bitcoin-mining-proxy/issues) with some more detail and we can get that labeled properly for you. for the stale rate, won't it make more sense if it is daily or even life time instead of default interval 1h? but for submitted share alert, i would probably like 5min. to check whether one card is down. for hashing speed, 15min will be fine. TLDR: I recommend separate interval settings for submitted share alert, hashing speed and stale rate. You can always open an issue here (https://github.com/cdhowie/Bitcoin-mining-proxy/issues) for the recommended changes. :) Title: Re: Flexible mining proxy Post by: nick5429 on June 26, 2011, 09:55:22 PM . Just wanted to point out that this was an empty reply :) Title: Re: Flexible mining proxy Post by: cdhowie on June 27, 2011, 12:38:02 AM . Just wanted to point out that this was an empty reply :) Title: Re: Flexible mining proxy Post by: dishwara on June 27, 2011, 07:05:33 AM The option to delete post was their before. Now it was removed by admins/mods. QUOTE EDIT DELETE Now only QUOTE EDIT Title: Re: Flexible mining proxy Post by: PulsedMedia on June 27, 2011, 09:43:08 AM Installed and seems to work. Tho weighting seems to be quite a bit off (haven't looked at that portion of code yet). There is no such thing as "weighting." Did you read the readme? Particularly the section on how priority works?Code should be PLENTY more commented btw ;) Probably, particularly index.php. The rest is fairly well-separated and should be readable as-is. My philosophy on comments is that if you have to comment a lot, your code doesn't explain itself well enough and should be refactored.IMHO, code should be commented nevertheless, saves time reading. Ie. doc blocks to explain what for a method is for. Later on this saves time when you can just generate code documentation, and can refer to. Larger the project is, the more important this is. Tho this might be a bit small project for such level of documentation, nevertheless a short comment here and there is a good idea. If you check some major project commentation rules or philosophies, like Linux Kernel, they suggest that one comment per 10 lines is a good level, level which you should target. As for the queries, if you can avoid on the fly created tables, that's better. Indeed, i did not even check how my queries did, as i was expecting them to work as expected. I spent sometime working on the 3rd one, and i achieved in under an hour to make the work amount less, less subqueries and on the fly created tables. No temporarys, no filesorts etc. but on the testing machine i'm using (Athlon XP 1900+ so REALLY slow), i couldn't verify results as times went into margin of error easily, and might just be parsing the query & deciding indices etc. takes that damn long on that slow CPU, with my tiny dataset at the time. It was faster without mHash rate, slower with mHash rate added (comparing oranges to oranges, so i compared to yours without mhash rate, and with it). Now i have larger dataset so when i got time i'll check it out again. Tho, i might begin from scratch as approach is wrong. Now i got 50k rows in the table, but that's still too small, i want to get to atleast 150k+ to be able to properly go through it, despite handling on this slow machine. Why are you using echo_html function in the view templates? Have you looked into smarty btw? It's very easy to implement, and makes views damn much simpler to view (just view as html and most highlighters are perfect for it). And those saying it forces bad code behavior: It's all about how you use it, it doesn't force anything, it simply compiles the views you ask for, and actually provides an EXCELLENT "isolation layer" for different layers of code, and eases debugging quite a bit saving tons of time. Overhead is extremely minimal too, and the first time you want to even think about it is when you are hitting high requests per sec. Also, i noticed some core functionality issues with this: phoenix started crashing now and then, about once a day per instance. Using BTCGuild, Bitcoin.cz and Deepbit, got 2 workers put for it, another works with 2 discreet GPUs, another with just 1. Sometimes it will just not pass new work, and sometimes it pauses work for a while. Is it only me? I'm using LinuxCoin v0.2a and it's accompanying phoenix. EDIT: Still getting quite fast load on dashboard with this dataset. Infact most of the time in the ms range. At slowest nearly a second. What is the point where people are starting to experience serious performance degradation, all the time? Title: Re: Flexible mining proxy Post by: cdhowie on June 27, 2011, 12:44:08 PM I'll reply to the rest of the post shortly, but wanted to answer this question before I leave for work:
EDIT: Still getting quite fast load on dashboard with this dataset. Infact most of the time in the ms range. At slowest nearly a second. What is the point where people are starting to experience serious performance degradation, all the time? If you've fetched the latest code and applied the database migration script, then you shouldn't be seeing any degradation anymore. If you didn't, then you would certainly see horrible dashboard performance after a while, as there were some missing indexes that would cause table scans against submitted_work during the dashboard query. (As submitted_work tends to grow slower than work_data -- but proportionally to it -- you'll need a fast miner, or a bunch of slow ones, to start seeing the performance issues quicker.)Title: Re: Flexible mining proxy Post by: sodgi7 on June 27, 2011, 01:37:36 PM I have phoenix crashing issues too but I'm not really certain if it is this mining proxy causing it. Problem went away when I switched over to another miner(rpcminer). There are some known problems with phoenix. Then again without the proxy phoenix doesn't crash as often so I dunno.
I guess the truth is somewhere in the middle but since using other miners with this proxy works I don't see this as a problem. Title: Re: Flexible mining proxy Post by: PulsedMedia on June 27, 2011, 02:11:33 PM I'll reply to the rest of the post shortly, but wanted to answer this question before I leave for work: EDIT: Still getting quite fast load on dashboard with this dataset. Infact most of the time in the ms range. At slowest nearly a second. What is the point where people are starting to experience serious performance degradation, all the time? If you've fetched the latest code and applied the database migration script, then you shouldn't be seeing any degradation anymore. If you didn't, then you would certainly see horrible dashboard performance after a while, as there were some missing indexes that would cause table scans against submitted_work during the dashboard query. (As submitted_work tends to grow slower than work_data -- but proportionally to it -- you'll need a fast miner, or a bunch of slow ones, to start seeing the performance issues quicker.)Nope, not using the latest, but atleast the 2 first queries are mine, and my added indexes are in place. Haven't yet checked what the diffs are. I'm running about 1.1Ghash against this, and i noticed severe drop in earnings too, due to the more frequent phoenix crashes. Will need to put aside 2 identical rigs to properly test is there difference. My rejected rate is also "through the roof", at about 7% via this proxy! But again, need the testing setup of 2 identicals to verify. Need to go buy more 6950s or 6770s to setup 2 identical rigs ;) Title: Re: Flexible mining proxy Post by: sodgi7 on June 27, 2011, 02:50:30 PM Just remembered another thing. Even when I'm not using Phoenix miner I seem to get quite high rejected shares rate when checking from the mining proxy database. When I check mining pool dashboard though it is all fine. So I guess this proxy currently marks certain bad connection issues or something similar also as rejected shares while they are really not?
But yeah PulsedMedia.. I wouldn't be surprised if your issues are partly caused by phoenix. I'm going to testrun poclbm in Windows on my machines today.. this far it seems very stable with this proxy as rpcminer clients. Title: Re: Flexible mining proxy Post by: twmz on June 28, 2011, 04:45:36 PM Hey, guys. I am not sure if bitcoin mining proxy has support yet for X-Roll-NTime and for clients that increment the time header (DiabloMiner), but I added it to my ASP.NET implementation of the proxy last night and wanted to share what I had to do to make it work so that you guys to add it to the original PHP implementation if you like.
If you are not aware, the X-Roll-NTime header is a head that some pools may return to indicate to clients that they are allowed to just increment the time header when they have exhausted the nonce space in the getwork they have. This allows poclbm, for example, to do only 1 getwork request per minute instead of 1 every 5-10 seconds or so on fast GPUs. This is not only better for the pool because it has to respond to less getwork requests, but it appears to also dramatically reduce the occurrence of "the miner is idle" because the miner can always keep mining by just incrementing the time header while waiting for a new getwork. Not all pools include this header, so the first change I made was to look for it when proxying getwork (both an actual getwork and submitted share) and when proxying a long poll. I then made sure to pass the header through to the actual mining client when it was present. The second, related situation is that some mining clients (DiabloMiner) will now choose to increment the time header even when this header is not present in circumstances where the miner would otherwise be idle. They are doing this not as a mechanism to reduce the need for frequent getworks, but instead as a means of continuing looking for hashes even in the presence of network problems that are interfering with getwork requests (either erroring them out or just slowing them down). In both cases (X-Roll-NTime and DiabloMiner's new algorithm), what will happen is that a submitted share will come in with data that is not exactly the same as the data we returned in the getwork, even after only looking at the first 152 characters of the hex data string. This means that we won't find their submitted data in the work_data table and won't be able to determine which pool to submit to. I fixed this by changing what data is stored in the work_data table and stripping out the integer (8 hex characters) that represents the timestamp being changed. I don't claim to understand what all of the 19 integers (152 characters of hex) of the data are that we are saving, but I determinined experimentally that it was the second to last integer (8 hex characters) that was changing when the miners were incrementing the time header. So, my code looks like this for pruning both the data returned from getwork request to a pool and for pruning the data submitted by the client during a submit. Note, this is C#, but you can probably tell what the equvilent PHP code would look like: Code: string fullDataString = (string)parameters[0]; With this change (and with the change to correctly proxy the X-Roll-NTime header), my proxy now successfully proxies submits where the time header has been incremented. I hope this is useful to you. Update: See this thread for context on the DiabloMiner change: http://forum.bitcoin.org/index.php?topic=1721.msg290512#msg290512 Title: Re: Flexible mining proxy Post by: cdhowie on June 28, 2011, 07:44:17 PM Hey, guys. I am not sure if bitcoin mining proxy has support yet for X-Roll-NTime and for clients that increment the time header (DiabloMiner), but I added it to my ASP.NET implementation of the proxy last night and wanted to share what I had to do to make it work so that you guys to add it to the original PHP implementation if you like. I'll add support for this at some point. Right now I'm trying to get the new C# getwork backend finished up.Note that as long as X-Roll-NTime is sent as an HTTP header, the proxy should still work with DiabloMiner; since the proxy will not forward this HTTP header on to the miner, it should think that the pool doesn't support it. If DiabloMiner is assuming that the pool supports it (I can't find a reference to X-Roll-NTime in the DiabloMiner sources) then, well, that's a DiabloMiner bug. Title: Re: Flexible mining proxy Post by: twmz on June 28, 2011, 07:46:24 PM Hey, guys. I am not sure if bitcoin mining proxy has support yet for X-Roll-NTime and for clients that increment the time header (DiabloMiner), but I added it to my ASP.NET implementation of the proxy last night and wanted to share what I had to do to make it work so that you guys to add it to the original PHP implementation if you like. I'll add support for this at some point. Right now I'm trying to get the new C# getwork backend finished up.Note that as long as X-Roll-NTime is sent as an HTTP header, the proxy should still work with DiabloMiner; since the proxy will not forward this HTTP header on to the miner, it should think that the pool doesn't support it. DiabloMiner does it's time increment thing with or without the X-Roll-NTime header in any circumstance where it would otherwise have to be idle (getworks not returning fast enough, getworks erroring out, etc). So DiabloMiner is going to submit data that won't be found with or without that header at least some of the time. Title: Re: Flexible mining proxy Post by: cdhowie on June 28, 2011, 08:04:29 PM DiabloMiner does it's time increment thing with or without the X-Roll-NTime header in any circumstance where it would otherwise have to be idle (getworks not returning fast enough, getworks erroring out, etc). So DiabloMiner is going to submit data that won't be found with or without that header at least some of the time. I can't even find a reference to X-Roll-NTime in the DiabloMiner sources. As long as it only does this when otherwise idle, it's as though the pool didn't support the feature anyway. So the effect will be the same as if DiabloMiner didn't do this at all, since all of those shares will be rejected. Therefore this is effectively a feature request against the proxy and not a bug.Title: Re: Flexible mining proxy Post by: twmz on June 28, 2011, 08:10:23 PM DiabloMiner does it's time increment thing with or without the X-Roll-NTime header in any circumstance where it would otherwise have to be idle (getworks not returning fast enough, getworks erroring out, etc). So DiabloMiner is going to submit data that won't be found with or without that header at least some of the time. I can't even find a reference to X-Roll-NTime in the DiabloMiner sources. As long as it only does this when otherwise idle, it's as though the pool didn't support the feature anyway. So the effect will be the same as if DiabloMiner didn't do this at all, since all of those shares will be rejected. Therefore this is effectively a feature request against the proxy and not a bug.As far as I know, poclbm is the only miner that support X-Roll-NTime. I think the reason people complain about Diablo is that it makes the Diablo output look wrong (it shows massive rejected shares). That said, you are right that it should only happen in cases when it wouldn't have been productive anyway, so no harm done. Title: Re: Flexible mining proxy Post by: kripz on June 29, 2011, 03:50:56 AM diablo doesnt work at all, after a while it the proxy stops accepting shares. Rejected doesnt even show up.
Title: Re: Flexible mining proxy Post by: kripz on June 29, 2011, 01:24:08 PM What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account?
Title: Re: Flexible mining proxy Post by: teknohog on June 29, 2011, 02:59:13 PM What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account? Just try it. It's worked for me well enough. Title: Re: Flexible mining proxy Post by: Naven on June 29, 2011, 04:47:57 PM @cdhowie, this app is amazing, but need some work of sql optimization and hash-rate calculator
Title: Re: Flexible mining proxy Post by: cdhowie on June 29, 2011, 05:17:50 PM What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account? Then the proxy will use the same credentials for all three workers. If pointing all three miners at the same pool using the same account works fine, then pointing them at the same pool using the same account via the proxy will also work.@cdhowie, this app is amazing, but need some work of sql optimization and hash-rate calculator How so? The hash-rate calculation queries perform very quickly on my install, and EXPLAIN shows that the indexes are used appropriately. Which commit are you running from?Title: Re: Flexible mining proxy Post by: cdhowie on June 29, 2011, 05:19:41 PM diablo doesnt work at all, after a while it the proxy stops accepting shares. Rejected doesnt even show up. The proxy's lack of support for X-Roll-NTime may have been to blame for this. Pull down the latest code; I just added X-Roll-NTime support about 45 minutes ago.Title: Re: Flexible mining proxy Post by: Grinder on June 29, 2011, 10:28:11 PM EDIT: I have a workable solution. I was able to successfully submit work with the proxy to MultiPool. Did you get anywhere with this?I will discuss the fix further with cdhowie in the morning and see how he wants to go about implementing it. Title: Re: Flexible mining proxy Post by: kripz on June 30, 2011, 01:25:15 AM What happens if i have a single account on a pool and i create 3 miners on the proxy and point them to that one pool using the same account? Then the proxy will use the same credentials for all three workers. If pointing all three miners at the same pool using the same account works fine, then pointing them at the same pool using the same account via the proxy will also work.So no problems with getworks and the like? One card solves something and the others some how knows to getwork again? I'll try the ntime fix later when i get home. Title: Re: Flexible mining proxy Post by: cdhowie on June 30, 2011, 03:50:24 AM try to update to latest version now You could have just done what it suggested: TRUNCATE TABLE work_data;Code: Current database version is 2. Attempting migration to version 3. EDIT: I fixed it by deleting the whole database and import a fresh database and load backup to my worker/worker_pool/pool EDIT: I have a workable solution. I was able to successfully submit work with the proxy to MultiPool. Did you get anywhere with this?I will discuss the fix further with cdhowie in the morning and see how he wants to go about implementing it. So no problems with getworks and the like? One card solves something and the others some how knows to getwork again? I'm not sure what you mean. The mining software is responsible for determining how to allocate work to the various devices it supports. My proxy doesn't know or care how many GPUs you have.Title: Re: Flexible mining proxy Post by: kripz on June 30, 2011, 04:13:33 AM So if i have 2 or more cards, an instance of poclbm or phoenix (which requires one instance per device) hasher running for each card, each pointed to the one proxy account which points to the one miner account on a pool, it doesn't cause issues?
What about multiple accounts for the proxy (so a proxy account for each device) which all point to the one miner account on a pool? I was thinking the proxy may hand out the same work to multiple cards and if one of the cards solve it, then the others wont know about it and keep hasing. Title: Re: Flexible mining proxy Post by: Grinder on June 30, 2011, 07:26:53 AM Multipool is currently down, so I haven't been able to review the problem and proposed fix. He's put the code on Github, so someone else set up a server here: http://multiclone.us.to:18080/ (mining port 18337).It has the same problem. http://forum.bitcoin.org/index.php?topic=17970.msg302834#msg302834 Title: Re: Flexible mining proxy Post by: eusor on June 30, 2011, 12:38:13 PM I also just added the new s3.mining.eligius.st pool. For some weird reason, I'm getting a lot of invalid shares :
Submitted duplicate shares 17 4 Submitted stale shares 35 0 Which I don't get if I access directly. Anyone else noticed this behavior? The only thing I can see is that worker use the same login on the pool. Title: Re: Flexible mining proxy Post by: cdhowie on June 30, 2011, 01:45:31 PM I also just added the new s3.mining.eligius.st pool. For some weird reason, I'm getting a lot of invalid shares : If you ignore the duplicate/stale stats and focus only on the accepted shares, are the numbers very different over time from what you see mining without the proxy? I am wondering if the proxy code that retries share submissions on errors is successfully submitting it the first time, but thinks that there was some connection issue and tries again, thereby giving you a duplicate or stale share but also correctly giving you the one accepted share.Submitted duplicate shares 17 4 Submitted stale shares 35 0 Which I don't get if I access directly. Anyone else noticed this behavior? The only thing I can see is that worker use the same login on the pool. Title: Re: Flexible mining proxy Post by: SteveFL on June 30, 2011, 02:25:03 PM Been lurking in this thread for awhile now. Most of the issues I've encountered have already been reported by other users and patched, but I have two things I can't figure out:
1) I have a very disparate number of getworks to shares in the pool summary (251 to 26). 2) I noticed on Code: 30/06/2011 09:58:43, Setting pool test @ proxypool:8332 I don't know if these are all related or its just a poclbm issue. I was thinking a timeout in apache/php could be the cause but I don't know the best way to diagnose. I had noticed for the last two weeks I had an abnormally high number of apache2 processes running (150+) for about 20 active miners. I'd eventually have to restart after apache stopped responding. This seems to have cleared with the changes committed in the last 3 days. Again, also not sure if related. Excellent project by the way. I just wish I hadn't been tainted by coding microsoft.net all these years so I could contribute some code. Title: Re: Flexible mining proxy Post by: cdhowie on June 30, 2011, 03:13:30 PM Been lurking in this thread for awhile now. Most of the issues I've encountered have already been reported by other users and patched, but I have two things I can't figure out: The getwork code performs failover after 3 seconds if a pool doesn't respond. This means that a little lag in, say, the DNS lookup can trigger a failover. This will not be a problem with the C# backend since it will keep a queue of a few pieces of work per pool, populated on a background thread. Getworks from a worker will pull off of the queue and trigger the background thread to refill it.1) I have a very disparate number of getworks to shares in the pool summary (251 to 26). 2) I noticed on That's odd. poclbm is all I use and I haven't noticed too many problems with it. I do note that it tends to abort a getwork request after about 5 seconds and try again immediately. This means that poclbm and the PHP failover code are often at odds, since by the time that the proxy fails over to the second pool, poclbm has aborted the request and is trying again.I had noticed for the last two weeks I had an abnormally high number of apache2 processes running (150+) for about 20 active miners. I'd eventually have to restart after apache stopped responding. This seems to have cleared with the changes committed in the last 3 days. Again, also not sure if related. Hmm, I'm not sure what I could have changed that would fix this. The frequency of requests might be causing Apache to try to keep more HTTP worker processes running to serve them. And if your miners recognize the X-Roll-NTime header (which I just added), that means they will usually only do a handful of getworks per round, and so the number of requests to Apache will drop dramatically.Excellent project by the way. I just wish I hadn't been tainted by coding microsoft.net all these years so I could contribute some code. I don't think you'd be prevented from contributing to the PHP side of the project. :) And since I'm not building a compatible framework (like Mono is), I don't think I really need to exclude MS.NET developers from the C# side of the project either.Title: Re: Flexible mining proxy Post by: cdhowie on July 01, 2011, 08:20:36 PM The C# getwork/LP backend is now feature-complete and is being tested. If there are no major issues, expect a release of it before Tuesday.
Title: Re: Flexible mining proxy Post by: kripz on July 02, 2011, 02:46:15 AM Please consider porting to Java, now i'll have to install mono?
Everybody will probably have Java already. Title: Re: Flexible mining proxy Post by: cdhowie on July 03, 2011, 03:59:11 AM Please consider porting to Java, now i'll have to install mono? I thought about writing it in Java, until I realized that means I would have to write it in Java. Then I took some Pepto for the resulting nausea.Everybody will probably have Java already. [citation needed]If on a LAMP server, Java probably won't be installed by default. If on a desktop, Java still probably won't be installed by default (unless it's GNU classpath) and in some cases Mono will be installed too, for Banshee/Tomboy/F-Spot. Honestly, I'd rather focus my energy on adding support for PostgreSQL, which I should have done in the first place... Title: Re: Flexible mining proxy Post by: btc_man on July 03, 2011, 04:34:54 AM So when you say "At this point you should be able to point your workers at the proxy and they will start working." in the installation instructions/
What address should I be using? Title: Re: Flexible mining proxy Post by: kripz on July 03, 2011, 06:26:59 AM Your server IP or hostname.
Title: Re: Flexible mining proxy Post by: btc_man on July 03, 2011, 03:54:30 PM this is what is showing up in the apache logs
Code: 192.168.2.2 - default [03/Jul/2011:10:03:50 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4" And this is the command that i am using to launch the miners Code: python phoenix.py -u http://192.168.2.7:8337 -k poclbm DEVICE=2 BFI_INT WORKSIZE=128 AGGRESSION=8 FASTLOOP VECTORS Can you maybe try to tell me what the problem is or what other info i need to figure the problem out? Title: Re: Flexible mining proxy Post by: cdhowie on July 03, 2011, 05:56:29 PM this is what is showing up in the apache logs Response 401 is "unauthorized." Your command does not appear to include any authentication information. You need to add the username and password of one of the proxy's worker accounts.Code: 192.168.2.2 - default [03/Jul/2011:10:03:50 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4" And this is the command that i am using to launch the miners Code: python phoenix.py -u http://192.168.2.7:8337 -k poclbm DEVICE=2 BFI_INT WORKSIZE=128 AGGRESSION=8 FASTLOOP VECTORS Can you maybe try to tell me what the problem is or what other info i need to figure the problem out? Title: Re: Flexible mining proxy Post by: error on July 04, 2011, 05:44:21 AM Thanks for the proxy. Seems to work pretty well so far. I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST: Code: --- a/htdocs/common.inc.php 2011-05-29 22:20:55.000000000 -0400 Hey, you still haven't included this fix, as far as I can tell. I just pulled master and it isn't there. Title: Re: Flexible mining proxy Post by: cdhowie on July 04, 2011, 03:15:38 PM Thanks for the proxy. Seems to work pretty well so far. I did have to make one change to fix a problem with the proxy out of the box. Whenever I submitted a form, I'd be redirected to some other web site (localhost) which has no server running. It appears you're reading SERVER_NAME, which may or may not be correct. It's more reliable to read HTTP_HOST: Code: --- a/htdocs/common.inc.php 2011-05-29 22:20:55.000000000 -0400 Hey, you still haven't included this fix, as far as I can tell. I just pulled master and it isn't there. Patch applied and pushed, thanks! Title: Re: Flexible mining proxy Post by: btc_man on July 04, 2011, 05:14:47 PM this is what is showing up in the apache logs Response 401 is "unauthorized." Your command does not appear to include any authentication information. You need to add the username and password of one of the proxy's worker accounts.Code: 192.168.2.2 - default [03/Jul/2011:10:03:50 -0400] "POST / HTTP/1.1" 401 364 "-" "phoenix/1.4" And this is the command that i am using to launch the miners Code: python phoenix.py -u http://192.168.2.7:8337 -k poclbm DEVICE=2 BFI_INT WORKSIZE=128 AGGRESSION=8 FASTLOOP VECTORS Can you maybe try to tell me what the problem is or what other info i need to figure the problem out? how would I add the username and password in the string, could you give an example. I plan to donate when i get this running. Please and Thank You! Title: Re: Flexible mining proxy Post by: error on July 04, 2011, 06:35:05 PM how would I add the username and password in the string, could you give an example. I plan to donate when i get this running. Please and Thank You! The username and password go in the URL, just as it says in the main Phoenix miner thread (which is where you really should have gone first). Example: http://username:password@host:port Title: Re: Flexible mining proxy Post by: Mr2001 on July 05, 2011, 12:13:02 AM I've patched this proxy to perform automatic pool hopping, directing requests to the most profitable pools, and have been using it successfully for several days on 3 proportional pools (very hoppable), 2 score-based (slightly hoppable), as well as PPS and solo mining as fallbacks. I don't have exact statistics on the earnings yet, but I believe it's over 20% more than I was making before.
I've only shared my patch with one friend, but I'm considering releasing it to the public for a bounty. Would anyone be interested? Title: Re: Flexible mining proxy Post by: MiningBuddy on July 05, 2011, 01:03:16 AM I've patched this proxy to perform automatic pool hopping, directing requests to the most profitable pools, and have been using it successfully for several days on 3 proportional pools (very hoppable), 2 score-based (slightly hoppable), as well as PPS and solo mining as fallbacks. I don't have exact statistics on the earnings yet, but I believe it's over 20% more than I was making before. No, there are better free & opensource versions already available, thanks.I've only shared my patch with one friend, but I'm considering releasing it to the public for a bounty. Would anyone be interested? Title: Re: Flexible mining proxy Post by: Mr2001 on July 05, 2011, 01:16:06 AM No, there are better free & opensource versions already available, thanks. Cool, I'd love to see what they do differently. Any besides Multipool?Title: Re: Flexible mining proxy Post by: Disposition on July 05, 2011, 06:53:33 PM I been doing some research into the Long Polling issue, wouldn't be possible to just let the miner connect to the LP address directly instead of going through the proxy?
Title: Re: Flexible mining proxy Post by: cdhowie on July 05, 2011, 07:16:25 PM I been doing some research into the Long Polling issue, wouldn't be possible to just let the miner connect to the LP address directly instead of going through the proxy? No. The LP request will return work, and if the work doesn't pass through the proxy then it can't be recorded in the database. And if it can't be recorded in the database, then work submissions against that work cannot be routed to their source pool, and the work the miner has done will have been for nothing.Further, the LP specification implies that the X-Long-Polling header must return a host-relative URI. (Though some pools do not follow this.) Title: Re: Flexible mining proxy Post by: Disposition on July 05, 2011, 07:18:42 PM No. The LP request will return work, and if the work doesn't pass through the proxy then it can't be recorded in the database. And if it can't be recorded in the database, then work submissions against that work cannot be routed to their source pool, and the work the miner has done will have been for nothing. I see. Further, the LP specification implies that the X-Long-Polling header must return a host-relative URI. (Though some pools do not follow this.) I thought that was changed, most miners supports full url PL. Title: Re: Flexible mining proxy Post by: kripz on July 06, 2011, 09:59:00 AM How goes the C# backend?
Title: Re: Flexible mining proxy Post by: cdhowie on July 06, 2011, 01:50:03 PM How goes the C# backend? I've run into a few bugs and have been fixing them and doing a lot of testing. It's getting there quickly.Title: Re: Flexible mining proxy Post by: callmeivan on July 07, 2011, 08:34:02 PM I used xampp 1.7.4 to set it up but after a couple of minutes (after adding some workers) i get an error.
Until then everything works, does somebody know how to fix it ? Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[08004] [1040] Too many connections' in /opt/lampp/htdocs/common.inc.php:33 Stack trace: #0 /opt/lampp/htdocs/common.inc.php(33): PDO->__construct('mysql:host=loca...', 'user', 'password') #1 /opt/lampp/htdocs/admin/pool.php(31): db_connect() #2 [internal function]: AdminPoolController->indexGetView() #3 /opt/lampp/htdocs/mvc.inc.php(142): ReflectionMethod->invoke(Object(AdminPoolController)) #4 /opt/lampp/htdocs/mvc.inc.php(178): Controller->execute() #5 /opt/lampp/htdocs/admin/pool.php(140): MvcEngine::run(Object(AdminPoolController)) #6 {main} thrown in /opt/lampp/htdocs/common.inc.php on line 33 Title: Re: Flexible mining proxy Post by: nux on July 08, 2011, 04:13:02 PM I'm using the latest version of the proxy software, and the latest poclbm. Since updating to the more verbose poclbm (week or two ago) I see this error all the time:
proxy.hostname.net:80 Problems communicating with bitcoin RPC 0 2 Then it recovers from 0mhash/s and looks like this proxy.hostname.net:80 93870 khash/sunicating with bitcoin RPC 0 2 until it gets back up to the normal speed (which takes just a second or so). I see this happen every few seconds give or take. Has anyone else seen issues like this? Since the last week or so I've had to stop using this awesome software as that error slows down my hashing. Thanks Edit: I'm running Apache on Debian. Server is tuned to handle a lot of load, and is nowhere near capacity. I'm not finding any issues in the error_log files and not sure how I can debug further at this point. Title: Re: Flexible mining proxy Post by: geek-trader on July 10, 2011, 02:23:30 AM Multipool is currently down, so I haven't been able to review the problem and proposed fix. He's put the code on Github, so someone else set up a server here: http://multiclone.us.to:18080/ (mining port 18337).It has the same problem. http://forum.bitcoin.org/index.php?topic=17970.msg302834#msg302834 Do these comments mean this project doesn't work with Multiclone.us.to ? Because otherwise, this looks awesome. Title: Re: Flexible mining proxy Post by: cdhowie on July 10, 2011, 09:03:49 AM Do these comments mean this project doesn't work with Multiclone.us.to ? Because otherwise, this looks awesome. Looking at the getwork response from Multiclone, I see no reason why it should not work with my proxy.Title: Re: Flexible mining proxy Post by: nux on July 11, 2011, 04:36:01 PM Anyone else having this issue? I tried a fresh install of the proxy and still having the problem.
This only started after upgrading to the latest poclbm. Here's my output: proxy.hostname.net:80 11/07/2011 11:31:01, Setting pool g11 @ proxy.hostname.net:80 proxy.hostname.net:80 11/07/2011 11:31:02, Using new LP URL /index.php/1/aHR0cDovL3VzZWFzdC5idGNndWlsZC5jb206ODMzMi9MUA%3D%3D proxy.hostname.net:80 11/07/2011 11:31:02, LP connected to proxy.hostname.net:80 proxy.hostname.net:80 229016 khash/snicating with bitcoin RPC 0 2 proxy.hostname.net:80 229331 khash/snicating with bitcoin RPC 0 2 proxy.hostname.net:80 11/07/2011 11:32:15, 02463d19, accepted 0 2 proxy.hostname.net:80 Problems communicating with bitcoin RPC 0 2 proxy.hostname.net:80 11/07/2011 11:32:34, c30efce9, accepted 0 2 proxy.hostname.net:80 228730 khash/snicating with bitcoin RPC 0 2 proxy.hostname.net:80 11/07/2011 11:33:56, 1888d04e, accepted 0 2 proxy.hostname.net:80 11/07/2011 11:34:42, 92512293, accepted 0 2 proxy.hostname.net:80 215716 khash/snicating with bitcoin RPC 0 2 I'm using the latest version of the proxy software, and the latest poclbm. Since updating to the more verbose poclbm (week or two ago) I see this error all the time: proxy.hostname.net:80 Problems communicating with bitcoin RPC 0 2 Then it recovers from 0mhash/s and looks like this proxy.hostname.net:80 93870 khash/sunicating with bitcoin RPC 0 2 until it gets back up to the normal speed (which takes just a second or so). I see this happen every few seconds give or take. Has anyone else seen issues like this? Since the last week or so I've had to stop using this awesome software as that error slows down my hashing. Thanks Edit: I'm running Apache on Debian. Server is tuned to handle a lot of load, and is nowhere near capacity. I'm not finding any issues in the error_log files and not sure how I can debug further at this point. Title: Re: Flexible mining proxy Post by: nux on July 11, 2011, 04:56:32 PM After watching tshark, poclbm.py output, and tailing the access log for the proxy, I have determined that a low KeepAliveTimeout setting in the Apache config was the culprit.
The KeepAliveTimeout on my server was 3 seconds, and the ask rate was 5. This caused the miner to try to use a KeepAlive session, only to find it timed out. Then it had to establish a new connection. This was giving the connection issues I was seeing. Essentially what this means is the KeepAliveTimeout setting has to be higher than the ask rate. I would suggest adding a few seconds on top to make sure there's a little wiggle room. Title: Re: Flexible mining proxy Post by: cdhowie on July 11, 2011, 06:52:38 PM After watching tshark, poclbm.py output, and tailing the access log for the proxy, I have determined that a low KeepAliveTimeout setting in the Apache config was the culprit. Thanks for the info! I'll add this to the readme, and perhaps to the .htaccess file.The KeepAliveTimeout on my server was 3 seconds, and the ask rate was 5. This caused the miner to try to use a KeepAlive session, only to find it timed out. Then it had to establish a new connection. This was giving the connection issues I was seeing. Essentially what this means is the KeepAliveTimeout setting has to be higher than the ask rate. I would suggest adding a few seconds on top to make sure there's a little wiggle room. Title: Re: Flexible mining proxy Post by: nux on July 11, 2011, 07:36:49 PM One feature I thought would be handy is something like a worker group.
Essentially a way to mass manage a handful of workers. For example, create a pool, and update 9 workers at once to all use it with the same login, password, priority etc. I currently just use SQL to do it manually. Add a worker_id/pool_id combo manually, and then an UPDATE .. WHERE pool_id = to add a pool to every one of those workers. Title: Re: Flexible mining proxy Post by: NickW on July 12, 2011, 01:02:52 AM I'm running this fine (so far) with nginx + php-fpm as the webserver.
One thing to note to get LP working is to set fastcgi_read_timeout in the nginx virtual host to something high (such as 3600) along with max_execution_time in the php.ini file. You also need some fairly specific settings to get the php parameters such as path_info passed along correctly. This is what I have. Code: location ~ ^(.+\.php)(.*)$ {Title: Re: Flexible mining proxy Post by: kripz on July 12, 2011, 12:22:41 PM Everytime a block is solved i get this after long poll event:
Quote [2011-07-12 22:22:02] JSON-RPC call failed: { "code": 0, "message": "No enabled pools responded to the work request." } Title: Re: Flexible mining proxy Post by: shotgun on July 22, 2011, 06:55:37 PM I've been having some table locking issues with the curent database since the schema creates all of the tables as MyISAM. As a MySQL DBA per profession, I would recommend that anyone running more than a handful of miners to switch to InnoDB. You can run the following commands on the mysql command line to convert your tables.
Code: alter table pool engine=innodb; The schema file can be changed to create tables as InnoDB from the start by changing each line that says "ENGINE=MYISAM" to "ENGINE=INNODB". Title: Re: Flexible mining proxy Post by: shotgun on July 23, 2011, 07:15:58 PM I've created a health monitoring script and a new database table for the Proxy so that you can see your worker "clock speed, mem speed, fan speed, card temperature" from the Proxy Dashboard page. I have the schema change (just an added table) and the health reporting script all finished. Just need to modify the dashboard to support a couple of new columns for that data.
I'll post the changes required when I'm done - hopefully people enjoy the improvements. cdhowie - if you like my work with this can you include it in the github repo or should I branch the project? I really love the work you did with the proxy server, it's made my life a lot easier. :) Title: Re: Flexible mining proxy Post by: shotgun on July 23, 2011, 07:17:22 PM I'm running this fine (so far) with nginx + php-fpm as the webserver. One thing to note to get LP working is to set fastcgi_read_timeout in the nginx virtual host to something high (such as 3600) along with max_execution_time in the php.ini file. You also need some fairly specific settings to get the php parameters such as path_info passed along correctly. This is what I have. Code: location ~ ^(.+\.php)(.*)$ {Can you tell me why this stuff is required and why I would want to implement it over Apache (what I'm currently using to run the proxy app). Title: Re: Flexible mining proxy Post by: NickW on July 23, 2011, 11:10:22 PM I'm running this fine (so far) with nginx + php-fpm as the webserver. One thing to note to get LP working is to set fastcgi_read_timeout in the nginx virtual host to something high (such as 3600) along with max_execution_time in the php.ini file. You also need some fairly specific settings to get the php parameters such as path_info passed along correctly. This is what I have. Code: location ~ ^(.+\.php)(.*)$ {Can you tell me why this stuff is required and why I would want to implement it over Apache (what I'm currently using to run the proxy app). It's just the way Nginx works. It doesn't handle PHP directly like how Apache does, PHP is in an external process (fastcgi) instead and you have to tell Nginx how to pass the relevant parameters and forward the request to this fastcgi backend. It's just an alternative web server and some people like myself prefer it over Apache. Primarily it is a high performance lightweight static file server and can easily handle 30000+ requests per second on static files, and 10000+ concurrent downloads, while using very little RAM (tens of megabytes) and works extremely well in these high load environments. It is not a fully featured beast like Apache, which is why you have to forward requests for things such as PHP to another backend process. Why implement it over Apache? If it's just you using it then no reason at all. However I have set it up on one of my web servers which runs Nginx anyway. Also, it could be advantageous to run it if you have many (hundreds or even thousands) of miners connected due to the memory usage compared to Apache. Check out some benchmarks for it :). Title: Re: Flexible mining proxy Post by: shotgun on July 24, 2011, 04:12:19 AM I've created a health monitoring script and a new database table for the Proxy so that you can see your worker "clock speed, mem speed, fan speed, card temperature" from the Proxy Dashboard page. I have the schema change (just an added table) and the health reporting script all finished. Just need to modify the dashboard to support a couple of new columns for that data. I'll post the changes required when I'm done - hopefully people enjoy the improvements. cdhowie - if you like my work with this can you include it in the github repo or should I branch the project? I really love the work you did with the proxy server, it's made my life a lot easier. :) So I finished the health monitor script and integrated the status display with the main admin dashboard. Here's a screenshot showing the added columns: "Temp, Fan Speed, Clock(core) Speed, Mem Speed". Anyone interested in this being added to the github code? I have two tables on the database "worker_health_current" and "worker_health_archive" - the current table tracks the most recent entry for each GPU, and the archive table tracks all entries so that (when I code this later) I can display graphs for trending of GPU temperature over time. Let me know what you all think. https://i.imgur.com/RKQGc.png Title: Re: Flexible mining proxy Post by: kripz on July 25, 2011, 09:49:54 AM Looks good, please fork and consider making it optional (via the conf file).
Is this being updated still? Title: Re: Flexible mining proxy Post by: exahash on July 25, 2011, 04:11:44 PM So I finished the health monitor script and integrated the status display with the main admin dashboard. Here's a screenshot showing the added columns: "Temp, Fan Speed, Clock(core) Speed, Mem Speed". Anyone interested in this being added to the github code? I have two tables on the database "worker_health_current" and "worker_health_archive" - the current table tracks the most recent entry for each GPU, and the archive table tracks all entries so that (when I code this later) I can display graphs for trending of GPU temperature over time. Let me know what you all think. @Shotgun - I am VERY interested! Please post a patch or a fork and let me know where to find it. Title: Re: Flexible mining proxy Post by: shotgun on July 25, 2011, 05:56:52 PM So I finished the health monitor script and integrated the status display with the main admin dashboard. Here's a screenshot showing the added columns: "Temp, Fan Speed, Clock(core) Speed, Mem Speed". Anyone interested in this being added to the github code? I have two tables on the database "worker_health_current" and "worker_health_archive" - the current table tracks the most recent entry for each GPU, and the archive table tracks all entries so that (when I code this later) I can display graphs for trending of GPU temperature over time. Let me know what you all think. @Shotgun - I am VERY interested! Please post a patch or a fork and let me know where to find it. Forked the project and added my files. The bin/worker_health.sh file has the instructions and pertinent info. https://github.com/btc-shotgun/Bitcoin-mining-proxy If you like it, feel free to send me some coins ;) 1BUV1p5Yr3xEtSGbixLSospmK6B8NCdqiW Title: Re: Flexible mining proxy Post by: shotgun on July 26, 2011, 03:54:39 AM I added support for graphing temperature. There's a link for each worker on the "temp" column which will show you the last 24 hours of temperatures for each card. You can get the changes via a git clone or update. Later on I'll add the ability to specify a date range for graphing as well as support for graphing the other health stats.
Here's an example. It uses DyGraphs. https://i.imgur.com/DsrPR.png Title: Re: Flexible mining proxy Post by: dlasher on July 26, 2011, 08:48:29 PM I've created a health monitoring script and a new database table for the Proxy so that you can see your worker "clock speed, mem speed, fan speed, card temperature" from the Proxy Dashboard page. I have the schema change (just an added table) and the health reporting script all finished. Just need to modify the dashboard to support a couple of new columns for that data. I love what you're doing... that's awesome.. for my 2cents, I'd like to see the temps/load/etc separated from the "worker" name, since multiGPU boxes will generally run (1) worker per box... more than that gets unmanagable quickly. Either that or you poll all the GPU's/CPU/etc in the box, and log the average.. Title: Re: Flexible mining proxy Post by: dlasher on July 26, 2011, 08:49:23 PM I added support for graphing temperature. There's a link for each worker on the "temp" column which will show you the last 24 hours of temperatures for each card.... How tough would it be to add graphs for both "worker" and "aggregate" mining data, like Mhash, shares, rejects, etc? Title: Re: Flexible mining proxy Post by: shotgun on July 26, 2011, 10:57:40 PM I added support for graphing temperature. There's a link for each worker on the "temp" column which will show you the last 24 hours of temperatures for each card.... How tough would it be to add graphs for both "worker" and "aggregate" mining data, like Mhash, shares, rejects, etc? I'll have to look at the schema tables that store that data. Once I get a query to support gathering the data over time then pushing it to the graphing system is trivial. I'll update my findings soon. Title: Re: Flexible mining proxy Post by: shotgun on July 26, 2011, 11:25:45 PM I love what you're doing... that's awesome.. for my 2cents, I'd like to see the temps/load/etc separated from the "worker" name, since multiGPU boxes will generally run (1) worker per box... more than that gets unmanagable quickly. Either that or you poll all the GPU's/CPU/etc in the box, and log the average.. Glad you enjoy it :) In regard to separating the workers.. well, that's not really possible with the current schema. Also I'm not sure it makes sense to store multiple GPU device temperature stats for one worker. It would require a 1:N schema which would require a completely different approach for the monitoring script, schema, and reporting aspects as well as graphing. You can get around this by making dummy workers in the proxy and having the health monitor script upload it's data for those dummy workers. So your single-worker-per-box can connect to the proxy as usual, but you'd have like "box-name-GPU0", "box-name-GPU1" etc and look at the graphs that way. But I can't store multiple GPU temperatures for a single worker. As to administrative hassle of multiple workers per box, my boxes are all running 3-4 cards each with one instance of phoenix per GPU. Having an aggregate worker for all devices on a box makes it very difficult to see how an individual GPU is performing, and Phoenix can't use multiple devices per process instance. The only miner I know that will use multiple devices is Diablo, which runs about 5-10% slower on my GPUs than Phoenix. Managing multiple devices per box is actually very simple, I have a script per instance that gets started via the LXDE autostart script: Code: > cat /etc/xdg/lxsession/LXDE/autostart Each of the /home/user/p<GPU_DEVICE_NUMBER>.sh script contains the following code to start up fans, core/mem speed, and connect to flexible proxy. If I reboot the box the miners get started automatically without any input from me. If my UPS dies the BIOS on each server is set to restore power to "LAST STATE", so they will turn on as soon as there is power available, then they boot LinuxCoin into init-level-5 and run the autostart script which in turn runs the miner via this code: Code: #!/bin/sh Title: Re: Flexible mining proxy Post by: shotgun on July 30, 2011, 08:03:25 PM I added correlation graphs so you can see Mhash values alongside your GPU temp in the same graph. Requires the Phoenix logfile modification to grab the mhash value, as the schema in Flexible doesn't provide an easy method to get periodic mhash values that correspond to the temp values. Maybe I'll figure that out later... but in the meantime if you use Phoenix the logfile mod is super simple. https://forum.bitcoin.org/index.php?topic=27761.0
https://i.imgur.com/w1CkQ.png Title: Re: Flexible mining proxy Post by: Brian DeLoach on August 13, 2011, 11:48:02 PM Code: [Sat Aug 13 18:32:06 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting How do I increase this setting? When this error is thrown, apache screeches to a halt and is unusable until restarted. Title: Re: Flexible mining proxy Post by: shotgun on August 14, 2011, 01:42:32 AM Code: [Sat Aug 13 18:32:06 2011] [error] server reached MaxClients setting, consider raising the MaxClients setting How do I increase this setting? When this error is thrown, apache screeches to a halt and is unusable until restarted. Google is your friend. Open the httpd.conf file and change the max clients value to something higher. Title: Re: Flexible mining proxy Post by: Brian DeLoach on August 14, 2011, 02:00:17 AM the httpd.conf in /etc/apache2 is blank. Is there another file by that name somewhere else?
edit: config file is apache2.conf for Debian. Google is pretty useful. Title: Re: Flexible mining proxy Post by: Brian DeLoach on August 14, 2011, 02:25:02 AM Of course, fix a problem and another shows up
Code: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [1040] Too many connections' in Title: Re: Flexible mining proxy Post by: shotgun on August 14, 2011, 06:01:00 PM Of course, fix a problem and another shows up Code: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [1040] Too many connections' in Open the /etc/mysql/my.cnf (debian) or /etc/my.cnf (rhel) and set max_connections=250 or to something higher than what you have currently. I'm not going to get into the per-thread buffer memory usage stuff when you increase max_connections... so either read up on it or just be aware that each connection allocates ram so if you are low on memory you might want to consider that impact. The default mysql max_connections is 100, fyi. I am not sure why flexible proxy uses connections so much - maybe it's not making persistent or pooled connections. Title: Re: Flexible mining proxy Post by: virtualcodewarrior on August 16, 2011, 12:39:59 AM I have the same problems as alfalfa. Every so often apache runs out of clients and has to be restarted. Increasing the clients doesn't help because then PHP throws the exception. I run 4 separate miners through the proxy and use the latest phoenix release. I mine mostly on Eligius.
It looks like new connections keep being opened without closing the old ones (or maybe it closes the old ones slower then it opens the new ones) Sometimes it recovers on its own. Title: Re: Flexible mining proxy Post by: shotgun on August 16, 2011, 03:08:34 AM I have the same problems as alfalfa. Every so often apache runs out of clients and has to be restarted. Increasing the clients doesn't help because then PHP throws the exception. I run 4 separate miners through the proxy and use the latest phoenix release. I mine mostly on Eligius. It looks like new connections keep being opened without closing the old ones (or maybe it closes the old ones slower then it opens the new ones) Sometimes it recovers on its own. This is likely due to a bug in the application where it's not closing connections properly. Simple solution: create a /etc/crontab entry that restarts apache and/or mysql every N hours. Code: 01 * * * * root /etc/init.d/httpd restart > /dev/null Title: Re: Flexible mining proxy Post by: bitfoo on August 24, 2011, 01:22:06 AM After watching tshark, poclbm.py output, and tailing the access log for the proxy, I have determined that a low KeepAliveTimeout setting in the Apache config was the culprit. Thanks for the info! I'll add this to the readme, and perhaps to the .htaccess file.The KeepAliveTimeout on my server was 3 seconds, and the ask rate was 5. This caused the miner to try to use a KeepAlive session, only to find it timed out. Then it had to establish a new connection. This was giving the connection issues I was seeing. Essentially what this means is the KeepAliveTimeout setting has to be higher than the ask rate. I would suggest adding a few seconds on top to make sure there's a little wiggle room. In general, is it preferable to configure Apache with KeepAlive enabled or disabled for this proxy? Title: Re: Flexible mining proxy Post by: shotgun on August 24, 2011, 04:13:10 AM After watching tshark, poclbm.py output, and tailing the access log for the proxy, I have determined that a low KeepAliveTimeout setting in the Apache config was the culprit. Thanks for the info! I'll add this to the readme, and perhaps to the .htaccess file.The KeepAliveTimeout on my server was 3 seconds, and the ask rate was 5. This caused the miner to try to use a KeepAlive session, only to find it timed out. Then it had to establish a new connection. This was giving the connection issues I was seeing. Essentially what this means is the KeepAliveTimeout setting has to be higher than the ask rate. I would suggest adding a few seconds on top to make sure there's a little wiggle room. In general, is it preferable to configure Apache with KeepAlive enabled or disabled for this proxy? enabled afaik, ymmv Title: Re: Flexible mining proxy Post by: Intention on September 01, 2011, 09:42:42 PM So I got this up and running on just a temporary vps before I commit to running this on a local machine since I'm very bad at these things and things usually end up getting reinstalled.
I have the web portion up and running with the dashboard etc it is currently installed in the root of the machine's apache directory. I have pointed my miner to the ip of my vps however I keep getting the same loop of errors. 2011-09-02 05:36:39: Listener for "test": myip:8332 02/09/2011 05:36:39, Problems communicating with bitcoin RPC 1 2 2011-09-02 05:36:44: Listener for "test": myip:8332 02/09/2011 05:36:44, Problems communicating with bitcoin RPC 2 2 2011-09-02 05:36:49: Listener for "test": myip:8332 02/09/2011 05:36:49, Problems communicating with bitcoin RPC 3 2 2011-09-02 05:36:49: Listener for "test": myip:8332 02/09/2011 05:36:49, No more backup pools left. Using primary and starting over. I have made sure magic_quotes_gpc is off by manually entering the option and I've also changed the keepalive time out to 10 seconds however I am not sure what else could be causing this. Title: Re: Flexible mining proxy Post by: skEwb on September 02, 2011, 07:50:22 PM Ran it on my own private VPS for a while and had no issues with it. It was working as it should. Thanks for the proxy.
Title: Re: Flexible mining proxy Post by: kripz on September 03, 2011, 05:03:23 AM This is long dead, though still working. I've read reports on this proxy loosing up to 10% of shares due to the code and/or apache not keeping up with the number of requests.
Please check your miner/proxy/pool stats and compare. Title: Re: Flexible mining proxy Post by: coblee on September 08, 2011, 07:45:03 AM This is long dead, though still working. I've read reports on this proxy loosing up to 10% of shares due to the code and/or apache not keeping up with the number of requests. Please check your miner/proxy/pool stats and compare. I've seen it slow down due to db getting too large. So I have to clean them out at least once a week doing this: DELETE FROM `work_data` WHERE DATEDIFF(UTC_TIMESTAMP(), `time_requested`) > 0; DELETE FROM `submitted_work` WHERE DATEDIFF(UTC_TIMESTAMP(), `time`) > 0; Title: Re: Flexible mining proxy Post by: Cdecker on September 08, 2011, 09:16:19 AM The proxy is quite good AFAIK, I have a consistent rejection rate of ~1%, but that's to be expected.
Title: Re: Flexible mining proxy Post by: Naven on November 05, 2011, 08:00:05 PM @cdhowie, are u planing any updates?
Title: Re: Flexible mining proxy Post by: btc_artist on November 30, 2011, 04:45:34 PM Watching
Title: Re: Flexible mining proxy Post by: siggy on February 04, 2012, 12:00:23 AM apologies for resurecting an old thread..
is this still being updated / mantained? Is there a better mining proxy out there? I've now got 6 rigs I'm managing, and soon to be expanding further... having one central location to view stats & manage pools would be a godsend... thanks, Sigg Title: Re: Flexible mining proxy Post by: fizzisist on February 04, 2012, 09:21:10 AM apologies for resurecting an old thread.. is this still being updated / mantained? Is there a better mining proxy out there? I've now got 6 rigs I'm managing, and soon to be expanding further... having one central location to view stats & manage pools would be a godsend... thanks, Sigg I use this proxy and it works well enough for me. It's a shame that it's not still actively developed, though. I took a look at the various forks on Github and it looks like virtualcodewarrior's (https://github.com/virtualcodewarrior/Bitcoin-mining-proxy) is the most recently updated, but the last commit was Aug 21, 2011... Still, it might be worth a look (I haven't looked at it yet). I second the question, is there a better proxy out there? Should we start a bounty or something? Are there any major problems with the proxy that need fixing? Title: Re: Flexible mining proxy Post by: ummas on April 05, 2012, 08:20:16 AM I`m on the topic now.
I`ll try to run that, and if it will work for me, i`ll start thinking for better solution. Is there any other as fizzisist said ? Title: Re: Flexible mining proxy Post by: Red Emerald on April 30, 2012, 05:14:38 AM apologies for resurecting an old thread.. is this still being updated / mantained? Is there a better mining proxy out there? I've now got 6 rigs I'm managing, and soon to be expanding further... having one central location to view stats & manage pools would be a godsend... thanks, Sigg I use this proxy and it works well enough for me. It's a shame that it's not still actively developed, though. I took a look at the various forks on Github and it looks like virtualcodewarrior's (https://github.com/virtualcodewarrior/Bitcoin-mining-proxy) is the most recently updated, but the last commit was Aug 21, 2011... Still, it might be worth a look (I haven't looked at it yet). I second the question, is there a better proxy out there? Should we start a bounty or something? Are there any major problems with the proxy that need fixing? Title: Re: Flexible mining proxy Post by: Lomax42 on June 14, 2012, 06:13:01 AM Does anyone can tell what's the problem?? Why do so many Getworks * and so little Shares *
Pool Latest work requested Getworks * Shares * Rejected * Deepbit By 123 at 2012-06-14 06:10:56 UTC 7939 1936 1.86% Use cgminer-2.3.4-win32 & 15 comp Title: Re: Flexible mining proxy Post by: MasterX on November 15, 2012, 09:09:57 AM Thanks for the release! I'll dedicate some time to pulling and installing it tomorrow, and pass that money along once it's up. After all, a promise is a promise :) Thanks again. Cool, thanks! Hope the software works for you. If you run into any trouble, let me know. I intend to support the software I publish. (That goes for everyone.) Maybe i'm missing something, but after i installed the database and changed mysql parameters (login&passwd); When typing the url in my browser http://127.0.0.1 i get the apache login window, askin for credentials, but whatever i try, it keeps coming back and i'm unable to login. I tried changing the credentials in the config file, but even with default 'admin' + 'changeme' it does not work. Is there anything else i should check or do before i'm able to login/use this software? Any help would be great! PS: i'm testing this on the most recent stable release of Xampp. Title: Re: Flexible mining proxy Post by: Baazee on May 03, 2013, 05:41:20 PM I use this software on my server.
I am actually very happy with it, but have the problem that more and more pools switch to stratum. Is there an update? |
