I've pushed a series of changes that should clear up several issues. Long-polling should be much more stable now; I've been running my miner all night and have seen plenty of new block notifications in the miner log, and a significant reduction in rejected shares. To address Deepbit specifically, it has started returning an absolute URL in the X-Long-Polling header, and the proxy did not support this. Now it does. Note that Deepbit is currently infinitely redirecting long-polling requests for some reason; I am going to contact Tycho about this and see what's up. In the meantime, long-polling against Deepbit will not work -- but it will appear to! The proxy will now sleep for 30 minutes after a failed long-polling request and then pretend that it was a getwork request. This is because some miners will give up on long-polling altogether if any long-polling request ever fails. By pretending that the request succeeded, the miner will continue to send long-polling requests, so when long-polling starts working again (or the proxy fails-over to a different pool) the miner will begin benefiting from long-polling again without needing to be restarted. I believe that some of these problems were the cause of Phoenix hammering Apache with long-polling requests. Those of you who were having trouble with Phoenix, consider giving the proxy another go with the latest code. Please thoroughly stress-test the code in the repository and let me know what other issues you find.  UPDATE: Tycho fixed long-polling. The proxy should now work correctly with Deepbit. |
|
|
|
Just pushed some updates. - Applied patch from cdecker implementing hash speed estimation for workers (issue 5).
- Implemented timezone conversion (issue 7).
These changes require modifications to your config.inc.php. If you pull without updating your config.inc.php, the software will break.To address some posts: Actually this just didn't work very well for me  Set it up with BTC Guild as primary and Slush as secondary. BTC Guild is having intermittent connection issues, so I wanted a backup. However, overnight all of my GUIMiner clients got stuck with a "connection problem". My one phoenix client (which is also on same machine as my proxy) continued working fine however. Some miners will bail on a connection if it doesn't return work within 3-5 seconds. This is not always enough time for the proxy to detect a fail condition, and so the miner gives up before the proxy has a chance to try another pool. Actually this did not run as well as expected. I've got a very high rate of rejected (about 10%) and after running for two hours phoenix stalled again:
I have started seeing similar problems on my install. This is being actively researched. I did see an occasional LP message, but not frequently enough as compared to blocks being found in the network. I also regularly got errors about "NoneType" not being indexable.
I have been seeing the same with poclbm lately. I'm not sure what changed to make this happen, but I suspect it is related to the stale shares problem. i believe ufa cpu-miner is not supported at the moment? all my tests ran in timeouts I haven't used that particular miner and so I'm not sure if the proxy supports it. If the miner isn't doing anything too wacky it should work... If I get some free time I might investigate this. long polling does not seem to work with phoenix through the proxy but works fine when connected directly.
There are some issues with long polling that are not easily addressed by the current model. Long polling should in theory work just fine if your preferred pool doesn't go down, but during a failover scenario it can get a little bit wonky. Some miners behave differently with respect to how they handle changing long-polling URLs, and so this is something that will take a lot of engineering to fix. In the meantime I may supply a config option to disable long polling entirely; broken long polling is worse than none at all. phoenix miner *hammers* the web server with requests to index.php. Something like 10-20 requests per second as seen in the apache access.log. When connected directly and observed with tcpdump it behaves much more normally (1 request every 5 seconds or so).
Sounds like a bug in Phoenix. Can you take some sniffs of this behavior with tcpdump -w bmp.pcap port 80 and email me the pcap file(s)? This will help me figure out what's going on. (Please compress them, and you can optionally encrypt them to my GPG key. This should not expose your mining account passwords, only your local proxy worker and possibly admin passwords.) Btw if I have access to computers at school and they are behind a firewall and so cannot mine, can I use this software to get around that?
As long as the computers can make outbound connections on whatever port you are running your web server on, yes. Transparent (or non-transparent) HTTP proxies should be able to process these requests too, though they might strip long-polling info. |
|
|
|
I assume users are likely to get other peoples dirty coins. It's still useful, but it isn't as good as clean coins. It would be cool if the payments came from a mining pool or something like that. It would add the problem of what to do with the dirty coins you get in though.
The easiest thing to do would be to accept incoming coins into wallet A. You would send out payments from wallet B (a different bitcoind). If you don't have enough coins in wallet B to do the next payment, send the entire balance of wallet A to a new address in wallet B. This will lump together all of the dirty coins. Then when you send the coins back out, they will essentially be split off of one big transaction. In other words, this effectively mixes all the coins together -- you can tell all of the inputs to the big transaction, and you can tell all of the outputs, but you can't tell which inputs mapped to which outputs. If all of the coins coming in were being traced by law enforcement, the outgoing coins wouldn't be useful to them in a legal sense, because while they would have a strong suspicion that the receiving party is tied up in something illegal, they don't know what anymore. And that's not enough to prosecute successfully. |
|
|
|
The switch should occur either upon work request or work submission.
Switches will never occur on work submission. Submissions always route back to the pool that the work came from. |
|
|
|
That's not true. The client must consider alternate chains as both at least potentially valid, and therefore both copies of that one transaction must be potentially valid as well. Until the network has voted which chain to build on, either one could be "the one".
Your algorithm breaks down because you are requiring a one-to-one link with transactions and blocks, but there is a one-to-many relationship (for a short time at least) between transactions and blocks.
No, that's wrong. The client cannot maintain multiple "block chain candidates," and there is no "voting procedure." The client will always consider one chain to be the main chain, and that is the longest chain, or in the case of two equally-long chains, the one it was aware of first. There is a one-to-one link with transactions and blocks that are in the main chain (the main chain from the perspective of the client, of course). |
|
|
|
Wow ... umm ... I downloaded the MP3 in the OP to listen to some Bitcoin stuff. Then some poor woman called in to talk about the Osama news, and the hosts just ... ripped her apart. They were rude and outright vicious. They barely listened to her, cut her off all the time, and even started yelling. It got so bad that at the end they started coming up with hypothetical situations where they were going to kill her or steal all her money!?!  They were obviously in jest, of course, but that seems like an inappropriate thing to say to someone who calls into your show. Is this just Fox News with an opposite political compass? No, that would be Ed Schultz. |
|
|
|
Ahhh I see the problem. From your proxy to deepbit:
GET /listenChannel HTTP/1.0
Authorization: Basic youcanthavemypassword;]
Host: deepbit.net:8332
I watched it send this to deepbit once and it worked, but the next time it didn't. The request was the same, but it appears to be failing because you're sending the request as HTTP 1.0. Based on the reply from the pool, you can see that the pool wants to serve HTTP/1.1. So my guess is we're hitting nginx, which is setup as a loadbalancer, and one of the backend machines is configured to work with HTTP 1.0 and the other is not. You can also see from my previous post that the client normally asks in HTTP/1.1 (though in the long polling specification at deepbit's website, they don't mention that it needs HTTP/1.1). If that all looks sensible, can you commit a fix to make the long polling requests in HTTP/1.1?
Thanks!
Hmm... I'm not going to discount this explanation, but I don't see any evidence that sending an HTTP 1.0 request is causing problems. My workers are issuing long-polling requests to deepbit through the proxy and they appear to be working just fine. |
|
|
|
|
I've just done a push. I recommend that everyone update from master.
This update fixes some PHP warnings that would get triggered on a failover, effectively breaking failover entirely on configurations where PHP warnings are emitted. The pool timeout has also been lowered from 5 seconds to 2 since the default configuration of some miners is to request work every 5 seconds. These miners will break if the proxy takes too long to respond, terminating the active getwork request and issuing another. (This timeout will be made configurable in a later push so that you can customize it to your configuration.)
An error_reporting() call has also been added to the config.inc.php.sample file. It's strongly recommended that you copy this into your config.inc.php to mitigate any possible future issues caused by PHP warnings getting mixed into the JSON response for workers.
|
|
|
|
Maybe you want to set the error_reporting level somewhere centrally to avoid this problem happening to others and also to avoid having to explain all the notices.
Hmm, I thought I did. Maybe I just did that in my own php.ini or something. Anyway, thanks for the comment. I'll add this in a future push. |
|
|
|
Great, it works for me too, just had to change the URL in my scripts. I think using such an API to run cronjobs against is more useful than having it crammed into the web interface.
Yup, that's why it's there. Not all of the pages have proper JSON support implemented, simply because for most of them it wouldn't be terribly useful. But at some point, the proxy should allow full manipulation of the database objects via JSON, allowing you to code an alternative interface (like a desktop app) against the JSON API. That probably wouldn't be a good thing to do, but it would at least be possible. I guess I can drop the stats branch then, since it already works out of the box from your dashboard. On the worker-hashing branch I made the average interval configurable because it strongly depends on the hashing speed (high hashing speeds can get a more accurate reading with shorter intervals, while slow miners will have the average jump wildly around for small values). I think it is ready to be pulled. Once again sorry for all the noise I created in here ^^
Alright, I'll have a look at it at some point later. Thanks for the patch! |
|
|
|
Being a stats junky I'm working on a simple page that dumps the data in the dashboard to JSON so that I can then write a munin plugin to monitor the status of my miners, I guess we could use that one with a cronjob to kick off custom scripts. It would make it much easier and more flexible since the web server user is usually quite constrained in what he can do ^^
There is support in the MVC framework for JSON output. The dashboard view page already implements IJsonView so try visiting /admin/?format=json. This works for me. BTW: htdocs/index.php@71 is throwing a lot of errors since in getwork requests the lpurl parameter is not set, and since we already check 2 lines below and then set $lpurl again it is redundant and should be removed ^^
You're right, I'm not sure why that's there. That being said, PHP is a whiny little bitch and it likes to complain about things that aren't problems.  I haven't looked at it in-depth yet. Let me know when you are done with it and have tested it and I'll review the pull request. |
|
|
|
Correct. One could specify a shell script in the database, and perhaps two other user-defined arguments which would be called like:
`$script $minerip $timedelta $userdefinedarg1 $userdefinedarg2`
And would hopefully cause the script to send me an email letting me know that miner 1.2.3.4 has been down for 300 seconds.
I'm not sure I could include the IP since that's not logged anywhere. But I could include the worker's database ID as well as the worker name. Or the script could be a field in the database for workers, i.e. if the miner is a Windows box you might want a simple email, but if it's Linux, you might want to execute a different script that runs 'expect' and attempts to restart the miner by logging in via ssh and giving it a kick. For that matter, the other two could be database arguments, but if it's too difficult to be practical from a programmatical standpoint, no worries.
To keep database schema changes to a minimum, I'm not sure if I would want to do this. But if I pass the worker's ID to the script, you could have your script look at the ID and figure out what to do. Useful indeed, if only I could make it function as flawlessly in my environment as it does in yours. So as I mentioned, Phoenix 1.2 prints periodic "disconnected" messages. Phoenix 1.45 seems to sporadically connect to the long-polling URL. Phoenix 1.2 will connect, but it disconnects after a period of time. I think I see why. I hit my local mining proxy and fetched the long polling URL. The first time it worked, and actually sent me info on a new block. But the next time I tried it, I encountered a problem that almost surely explains what I'm seeing in Phoenix:
...
Based on the 3 or 4 times I've done this, it seems to happen when there is in fact a new block. This should be easily duplicable for you. If not, I can PM you the info on my proxy and let you see for yourself. Hope that helps.
Hmm. This capture is helpful, but without seeing the long-polling request and response that the proxy itself sends out, I can't really say why it's doing this. It is possible that deepbit itself is returning an error from the long-polling request and I'm just forwarding it on to the worker. |
|
|
|
Just wanted to update this to let you know that Phoenix 1.2 (version I've run for the last couple weeks) seemed to exhibit the problem, while version 1.45 has not yet exhibited the problem during several hours of testing. I'm going to let this thing run for several more hours, then switch over a few more miners to the proxy and see how it goes.
Cool, keep me posted on that. 1) Can you add in code (maybe configured by config.inc.php) to adjust the timestamp according to a user-defined timezone? I've become accustomed to looking at UTC and knowing the offset, but still would be a nice thing to have.
Sure! I planned to do this at one point but forgot about it later. Thanks for reminding me. 2) The purpose of my push for your software, and what makes it valuable to me, is being able to know if a miner stops pulling and submitting shares (in addition to all of the wonderful features you've coded in so far). Is this a feature you plan to add in the future, i.e. with simply listing all of the miners in red/green status like deepbit does, and having the option to act on it somehow, like with a shell script?
Yeah, this is probably something that could be done. The red display is pretty easy to do; the shell script will require DB schema changes and a bit more work, but will still be possible. I assume this would be used to, for example, email you if one of the miners stops requesting work? I could define three new configuration options: one for the amount of time a miner must have not requested work for it to display differently, one for the amount of time a miner must have not requested work for the script to be run, and one for the script to run. Thanks again for all of your hard work!
No problem! I'm glad you find the software useful. |
|
|
|
Ok further to my issue, I captured one of those sessions per below:
If I'm reading this right, the miner is actually closing the long-polling connection of its own accord, mid-stream. What miner are you using? Perhaps I can reproduce the issue locally. |
|
|
|
It's possible that you left around the same time that we starting logging each ban independently based on reason and your ban was mis-categorized. I didn't say you were wrong, I said our logs showed otherwise, stating, in hand, that upon inspecting your account history and our logs, it showed that you were banned for low efficiency. A lot of those logs were recategorized by hand when we first implemented the system. Fair enough. The logs are simply wrong. However, let me return to the point I was trying to make (and that you seem to only be helping me make): The only people who seem to dislike, distrust, or belittle our pool are those who haven't tried it themselves.
This is a demonstrably false wide-sweeping statement. I am one counter-example; there are likely others. It's best to avoid making categorical statements of the form "All S are P" unless you can prove it. Upset because we banned you for being inefficient?
It's a rule. You broke it.
A rush to judgement, using data generated from code that you have since admitted you knew to be buggy. Now, to return to your latest reply: It says nothing about the rest of the pool functionality or code, nothing about my ability to be logically competent, and doesn't say that I'm lying either.
You're right there. But this combined with the SQL injection vulnerability that was already exploited only furthers my case for avoiding your pool. Where there's smoke... However, if you choose to read that far into it, I can into you saying that by choosing to leave a pool that has no fees, consistently improves and implements new features, and grows on a daily basis in favor of any of the other options that were available a month ago, you're displaying that you personally are not capable of doing basic math in regard to net gains, as is the case with the better portion of the bitcoin community, as they left one pool that charges excess fees for another that charges even more.
The pool I'm in takes a flat fee of 0.02592 BTC from the entire pool per 30 days, which is less than 0.01 BTC when you consider that it is divided among all workers. Again, you rush to judgement without getting your facts straight. I don't think I can make my point any better than you already have. |
|
|
|
In your Install file you say:
2. Create a MySQL user for the proxy, giving it the privileges SELECT, INSERT, UPDATE, DELETE, and LOCK TABLES on the proxy's database only. (Optional, but recommended.)
Is it the same user than:
3. Set the admin user and password. This will be used when logging in to the web management console.
No. The first is a user in the MySQL database; all of the workers as well as the admin account will cause the proxy to authenticate to the database server with these credentials. The admin user is a separate set of credentials that you use to log in to the proxy management console. You can pick whatever you want for the admin credentials, but the MySQL credentials must match a MySQL user. and an other question is, when i connect on the "/" of my proxy, it ask me a login and password.
Is it the same than in the admin section ?
No. When you use the "/admin" URI, it will ask for admin credentials. When you use the "/" URI it will ask for worker credentials. |
|
|
|
How to set up a "no admin ask password" to test it
I'm not sure what you mean. Can you elaborate? |
|
|
|
Works like a charm, and it fixed my 100% problem with Diablo too ^^
Neat, I'll merge the fix into master. |
|
|
|
I'm not really familiar with the framework you're using, so I started by implementing the hash speed for the workers. As for the routing of the post request, I guess you'd be faster than me :-)
It's a custom lightweight MVC framework that doesn't do terribly fancy routing, but still allows me to separate my code better. I've pushed a fix to the diablo-lp-fix branch. Can you try that and see if it fixes the issue for you? Just issued a pull request for the hashing speed and shares in the last hour on the dashboard for each worker. Hope you like it ^^
Thanks! I'll have a look later today. |
|
|
|
I should have seen it immediately: Diablo uses POST request while well behaving clients (phoenix, poclbm, ...) use GET requests. Should be a simple matter of routing POST to the same handler Ah, good catch. I should have followed the "be lenient in what you accept" rule when coding... In the meantime I'm working on a hashrate estimate for the workers, which I'd find quite usefull. Do you accept patches/pull requests?
Absolutely, as long as you release your patches under the AGPLv3. Have you patched it to accept POST for long-polling requests, or should I fix that? |
|
|
|
|
Show Posts
