[EXPERIMENTAL] poclbm-autohop: Yet Another Automatic Pool Hopper

[Aexoden]

Can you explain what the different fields mean on the last line of the display?
Am I only getting an effective rate of 294 MHash/sec?

Code:

                                     
bitcoins.lc [426.863 MH/s (~294 MH/s Eff:0.10)] [Rej: 0/10 (0.00%)] [GW: 104 (Eff:0.10)]            

You've only submitted 10 shares, so the averages haven't really had time to stabilize. Here's a brief explanation of the various fields. Some are in stock poclbm, and some are from luke-jr's version.

The second section is your estimated hash rate based on accepted shares in some recent time period. Over time, it should be roughly equal to the first field, unless you're getting a lot of rejected shares. Eff stands for efficiency and refers to your recent ratio of shares per getworks. On some pools this will be low, and on others it may be 5 or greater.

The next section tells you the number of rejected and total shares you've had. (In this case, you've submitted 10 shares, all accepted.)

The final section tells you the number of getworks the client has issued and the overall efficiency. Better efficiency is primarily better for the pool, rather than for the miner, but it doesn't hurt, either.

[MrWizard]

Finally got it running on Windows 7.  I like it.  However, my hash rate is not as high as poclbm.exe Windows native executable.  My hash rate dropped to 305 from 316 MH/s.  So probably will not use your auto-hopper unless you can get the hash rate a little higher.

[iopq]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

[hawks5999]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

"it does reward every share equally" != "because every share has the same chance of being inside and outside the window"

That's like saying every lottery ticket is a winner because every lottery ticket has a chance to win

or

Every candidate is hired because every candidate has a chance to be hired

or

Everyone is a millionaire because everyone has a chance to be a millionaire

[iopq]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

"it does reward every share equally" != "because every share has the same chance of being inside and outside the window"

That's like saying every lottery ticket is a winner because every lottery ticket has a chance to win

or

Every candidate is hired because every candidate has a chance to be hired

or

Everyone is a millionaire because everyone has a chance to be a millionaire

you're just arguing semantics

[Aexoden]

I have modified the block probability API to at least partially score the blocks to determine the probability that each pool produced it, rather than simply using each pool's estimated hash rate. However, both factors are considered.

This should increase overall efficiency by some unknown degree, but since I have no easy way to measure such a thing (other than creating new accounts on all the pools and mining using only this miner), we may never know for sure. The most likely result of this change is that BTC Guild and bitcoins.lc should see increased time as the pool with highest utility.

Since this change is on my backend, no upgrade is necessary. Clients will automatically pick up the new data.

[MrWizard]

I have modified the block probability API to at least partially score the blocks to determine the probability that each pool produced it, rather than simply using each pool's estimated hash rate. However, both factors are considered.

This should increase overall efficiency by some unknown degree, but since I have no easy way to measure such a thing (other than creating new accounts on all the pools and mining using only this miner), we may never know for sure. The most likely result of this change is that BTC Guild and bitcoins.lc should see increased time as the pool with highest utility.

Since this change is on my backend, no upgrade is necessary. Clients will automatically pick up the new data.

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

[Aexoden]

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

I have seen that, and I have no idea why it happens. poclbm is such a mess in general, that it's hard to do any real sort of debugging. When it's happened to me, I've noticed that I can make it go away by running a CPU-bound process in the background (like a CPU miner). I also haven't the foggiest idea why that would help, but without knowing the reason for the rejected shares, it's hard to debug.

I utilize poclbm's built-in multi-server support to handle the multiple servers, and I don't think it handles switching very well, sometimes leading to additional rejects. However, fixing poclbm itself isn't very fun for me.

Also, I just noticed your message about hash rates. I haven't personally noticed too much of a drop, but my Windows miner has never done very well at staying at peak, even using stock poclbm. Linux seems fine with both, in any case.

[techwtf]

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

Maybe the pool disabled N time rolling. and I don't really know why "nonces_left += 0xFFFFFFFFFFFF" is required, so I removed these 2 parts of code. one getwork, one response expected. that's the normal behaviour in phoenix miner.

Add: I still using a heavily modded poclbm/2011.b7. No plan to catch up to the new version ...

[Artefact2]

Hello. I see you are using pident for choosing where to hop. I feel obliged to remind you that accuracy is fairly low ( http://pident.artefact2.com/accuracy ).

But hey, you said you tweaked it so it gathers more data… If you managed to improve the accuracy, could you care to share the source so that I can merge it? I know the WTFPL doesn't require you to do that, but it would be nice. If your public domain miner relies on a closed-source web API, then it's of no use

[Aexoden]

Hello. I see you are using pident for choosing where to hop. I feel obliged to remind you that accuracy is fairly low ( http://pident.artefact2.com/accuracy ).

But hey, you said you tweaked it so it gathers more data… If you managed to improve the accuracy, could you care to share the source so that I can merge it? I know the WTFPL doesn't require you to do that, but it would be nice. If your public domain miner relies on a closed-source web API, then it's of no use

The only reason it isn't released anywhere is because it's not up to my own standards for sharing (it has a bit of site-specific code in it, and isn't very well organized, because I changed how I implemented it halfway through). If I have a chance, I'll try to refactor it a bit and fork the pident repository on github. However, even if I do, it probably wouldn't be too mergeable. I tried to work entirely on top of pident, rather than modifying it, for two reasons:

• First, this made it easier to merge any changes you made.
• Second, I gave up trying to figure out exactly where in the code the scoring was happened. I think I figured it out since then, but as you'll see below, I'm not sure the work I've done is directly applicable to improving the scores themselves.

Regardless, I can at the very least attempt to describe my current thought process and algorithm.

The first instinct is to just take whichever pool has the highest score and assume they found the block. As you've said, however, the accuracy of that method isn't incredibly high. For the purposes of pool hopping, we don't want to guess which pool found the block. Rather, we want to take all the available information, and assign probabilities to each pool. Using these probabilities, we can calculate the utility of mining at each pool.

At first, I just used each pool's percentage of the overall network, which is accurate, but I figured it should be possible to do better with additional information, namely pident's scoring. The trick is figuring out how to massage the scores pident produces into probabilities. To that end, I sampled an equal number of blocks from each pool's set, producing a block universe where each pool is equally likely to find a block. pident's scoring works based on identifying unique characteristics of each pool's blocks. It (to my knowledge) has no idea that some pools are simply more likely to find blocks, so modeling on this sampled block universe makes sense to me.

Treating the scores, then, as relative confidences that each pool found the given block, we rescale them so they sum to 1, so they at least have a chance of representing probabilities. Once I did this, and plotted the likelihood that each range of scores meant a pool did find the block, there was a definite pattern. (In the particular data set I looked at, a rescaled score of 0.06-0.07 implied a ~13.5% chance that the pool found that block.) By the time we reach scores of 0.1, the probability is almost 100%. Ultimately, I modeled the data using the atan function (which matches the curve). (Does this make theoretical sense? Who knows, but as long as it works practically, I'll go with it.) The idea here is to scale the scores so that a score of 0.5 implies a 50% probability. I eyeballed the function, and it's not very forum friendly, but it matches the data fairly closely.

The end result of all this massaging of the original scores produced by pident is a rough probability that the block belongs to each pool, but only in the magical "every pool is equally likely" universe. To get back to the real world, we multiply each score by the pool's percentage of the overall hash rate. The last step is rescale these final values so they again sum to 1. That's the final probability I use in calculating the data in the API.

The next step is to test these numbers and make sure they at least come close to matching the ultimate results. I'll probably wait a few days so I have some out-of-sample data before I embark on that quest, however.

Other next steps:

• Getting the code postable. I have no desire to keep it that close to my vest.
• Deciding on the best way to implement the hopping strategy implied by the API. poclbm is kind of a pain. I contemplated modifying cgminer, but that's quite a chore. If recent improvements to bitHopper have made it more usable (e.g. unbroken LP support, and not spamming pools with high-frequency getwork requests), it might be best to implement it there. I'd have to compare its average reject percentage with poclbm.

[Artefact2]

Okay, thanks for the explanation! I'm very interested in any pool using pident, and if you need help to hack it (like, where/how are the scores calculated ), ask me on IRC or by email

[bitcoindaddy]

I noticed PolMine, MainframeMC and bitcoinpool are in your data-feed, but not in the pools.py code.  Are those not ready for prime-time or were they omitted for a reason? I've had good luck with PolMine.

Another one to consider adding is pool.bloodys.com, though they are fairly small. Is there a lower limit on the GH/s that you consider reasonable?

[Aexoden]

I noticed PolMine, MainframeMC and bitcoinpool are in your data-feed, but not in the pools.py code.  Are those not ready for prime-time or were they omitted for a reason? I've had good luck with PolMine.

Another one to consider adding is pool.bloodys.com, though they are fairly small. Is there a lower limit on the GH/s that you consider reasonable?

The main thing deciding which pools I support is the pool support in pident. I could add them myself, but it seems like there are new ones popping up every day, so keeping track and writing the parsers might be more than I want to take on. Practically, I probably wouldn't bother adding a pool that's average block solve time is longer than a week or so.

Before adding any pool to the actual miner, I have to review it and determine (as best I can) the reward system used and its expected utility. (For instance, I don't currently support Slush in the miner because I haven't yet figured out a way to calculate its utility. I haven't tried very hard yet, either, but it may not even be worth the effort.) So, reviewing the ones you mentioned:

• PolMine: I can't actually in a quick search verify their reward system. If I can figure that out, I'm not opposed to adding it, but I probably wouldn't sign up there myself.
• MainframeMC: Claims to use a cheat-proof scoring system. I haven't investigated further. At best, it'd be a good backup pool. Not opposed to adding it, if I can confirm the reward system.
• bitcoinpool: Last time I checked, they use a hopping deterrent (halving the value of shares if you mine less than 50% of the round). This should still be hoppable (but less so), but I'd have to come up with a good utility function. Since rounds shorter than an hour aren't subject to this, it's not quite as cut and dry as halving the utility of a normal prop pool (but that could be a good starting point). By the way, the hop point in the "halve the rewards" scenario is ~10%.
• bloodys: Not currently in pident, I don't think.

Also, my long-term rejected shares seem to be somewhere in the neighborhood of 4-5%. This is definitely more than mining a single pool with poclbm, but presumably the hopping bonus makes up for it. It's still worth working on, however. For me, most of them seem to happen right after switching pools, so something's probably going wrong there.

I'm also checking my efficiency on BTC Guild to see if the new hopping method is giving positive results, but it'll be several days before there's enough data to even consider. Successfully defeating the one hour delay will be necessary, otherwise more pool operators may consider it as a solution, rather than changing to a non-broken reward system.

[Aexoden]

The code used to generate the API is now available on github: https://github.com/aexoden/pident

The code is located entirely in the new printpooldata file, which prints out the JSON object. I haven't included my custom update script, at this time, however. (It's badly organized and has some site-specific code in it.) The update script is executed by cron once a minute, and approximately does the following things:

• Executes updateblocks
• If a new block was detected, queues an updatepools to run in two minutes (who knows if this delay is long enough/too long--I'm just trying to minimize unnecessary network access).
• If there were new blocks, or if updatepools was executed, a new copy of the pool data is generated and copied to my server, which then serves it up.

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

By the way, I'm under no illusion that printpooldata couldn't use some optimization. However, doing so isn't really worth the effort for me. It might seem kind of haphazard because of changing my approach a couple of times in the middle, and not completely rewriting to match.

[Artefact2]

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

Hmm, it doesn't appear to be the case in your repo. Also, unless you already use them, declareCache() and declareCacheExpiration() will do the job in 2 lines of code (Etags: and Expires: fields for client-side cache and server-side caching of the generated page). In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

[hawks5999]

In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

Now you are throwing down some massive geek braggin'

"Oh yeah, well my cache is so awesome, apache tell me to keep the change. BOOM!"

[Aexoden]

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

Hmm, it doesn't appear to be the case in your repo. Also, unless you already use them, declareCache() and declareCacheExpiration() will do the job in 2 lines of code (Etags: and Expires: fields for client-side cache and server-side caching of the generated page). In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

Pident itself was far too I/O heavy to run on my VPS, at least during the import. Maybe I just had too little RAM or needed to optimize postgres. Either way, for the moment I generate the data locally and upload it to the server where it's served by a short custom script to handle the headers and gzip.

Meanwhile, I am working on reducing rejects, but it'll be a few hours before I can see if my changes have improved things.

[Aexoden]

I've pushed a commit that may help to reduce rejected shares. Before this patch, my rejected shares ranged from 3-6%, depending on the frequency of hopping. My limited test suggests this has been drastically reduced, but the old bitcoins.lc URL wasn't working all day, so the test may be flawed. I'll post more results in about 12-18 hours.

[Aexoden]

I've added support for MainframeMC to the miner. Note that since it carries a 1.5% fee, it won't be used unless you have no other fair pools configured.

I've also added the ability to set a donation percentage for each pool in pools.conf. This will allow the miner to more accurately calculate the pool's utility. The field is optional, so your old pools.conf will continue to work with the new miner. If you are, for instance, donating 1.0% to a pool, you should add 1.0 to the end of the line.

Finally, the reject reduction work seems to be working well. Long-term reject percentage on my miners seems to be somewhere between 0.5% and 1.5%.

There are still several pools that are not supported:

• Bitcoinpool: Uses a modified proportional system for which I need to derive an accurate utility function. Given the Draconian TOS they have, I'm unlikely to bother.
• BTCMine: Uses the exponential decay scoring system with an unknown magic number. It may be possible to estimate the utility function, but I haven't looked into it.
• BTCMP: May add at some point.
• DeepBit: I will not add this as long as it makes up more than 33% of the network.
• PolMine: As long as I can confirm and keep updated on its reward system, I can add it.
• Slush: Like BTCMine, I'd have to evaluate the scoring system.

If a pool is not listed, it's not currently supported by pident. (Bitclockers is not supported by pident because it obfuscates its blocks, so no one knows exactly which blocks belong to it.) You can presumably talk to Artefact2 about getting pools added to pident.