[EXPERIMENTAL] poclbm-autohop: Yet Another Automatic Pool Hopper

[Aexoden]

The past few weeks, I have been slowly working on modifying poclbm to be a full-fledged automatic pool hopper. I think it's finally to the point where it might be usable enough to distribute, but I'm still choosing to label it as EXPERIMENTAL.

Motivation

Many mining pools are still using the inherently unfair proportional payout system. My primary intent is to encourage pool operators to migrate their pools to fair systems. It seems as if the only way this will happen is if miners themselves see that the proportional system is unfair. (Unfortunately, the fair systems often seem unfair at first glance, making explanations more difficult.)

Methodology

Originally, poclbm-autohop operated similarly to bitHopper, retrieving share counts from the various pools, and calculating a utility for each one. Recently, however, instead of replacing their broken payout systems with provably fair systems, pools have begun to implement various obfuscation measures, such as delayed statistics. It is apparent that another approach is needed.

Using a local copy of pident to identify the pools that have claimed each block, I calculate the probability that each pool's last block occurred at a given time index. This probability is currently based on each pool's relative hash rate, but work is progressing on improving these probabilities using a heuristic scoring system. (Limited tests show that it may ultimately be possible to predict the finder of a block with at least 30-40% accuracy. Given that most pools make up less than 10% of the network, this is a marked improvement.) I upload these probabilities as well as estimated hash rates for each pool and it available via a JSON API.

(As a quick example, if there's a 10% chance that a share will be worth 10x, and a 90% chance it will be worth 1x, its expected value is 1.9x.)

poclbm-autohop itself then downloads these estimated hash rates and probabilities, and uses them to calculate the utility of each pool. This method will never be as accurate as using the share counts directly, but since many pools are obfuscating or delaying this information, this new approach may be the best we can do.

Caveats

• The estimated hash rates are currently slightly inaccurate for some pools. This will improve within the first week of August, as my local copy of pident collects more data.
• The current code estimates hoppers provide a 30GH/s boost above the average. This is probably a lowball figure, so the utility of pools may be overestimated.
• The probabilities in the API are currently based only on each pool's share of the network.

In addition, while the logical next step for proportional pools would be to replace their broken payout systems, I fully expect pool operators will instead move to no longer report ownership of blocks. Personally, I have to wonder about any pool that would resort to this level of obfuscation rather than simply change their payout model. (Bitclockers already seems to obfuscate their blocks in this way.)

Bugs

• I don't know if it's something my code did or just a problem with poclbm itself, but occasionally it might go into massive reject sprees. This only happens with some pools, so who knows.
• Sometimes things just stop working properly, causing the miner to miss long polls or such, and I'm probably too lazy to dig into poclbm to fix all its problems. I recommend restarting the program periodically (once every day or two, perhaps).

There are probably others that I'm not thinking of, but it should largely work, in any case.

Usage

The code is available via GitHub:

https://github.com/aexoden/poclbm

It should run in any environment that poclbm itself will run, but it does have two additional dependencies: scipy and httplib2. Both should be readily available for both Linux and Windows.

For detailed information, you should definitely consult the README.mkd file. Basically, however, it accepts the same parameters as standard poclbm, except it no longer takes a list of servers as a parameter. Instead, you need to create a pools.conf file, which contains a line for each pool. Each line should say the name of the pool, your worker username, and your worker password, separated by whitespace (I recommend tabs). You may list the pools in your order of preference. If two pools have the same utility, the first pool listed will be preferred. Only pools listed in your pools.conf file will be used.

For the list of supported pools, please consult the README.mkd file.

Conclusion
Comments, suggestions, and questions are welcomed. I can only hope this software will be obsolete very shortly.

[1713449452]

1713449452

[1713449452]

1713449452

[1713449452]

1713449452

[Sukrim]

Have you tried to connect to all pools in the list, timing LongPollings and (as an easy start) assuming the pool from which the first LP comes has found the block? This could also be factored in additionally with your scoring approach and should probably give some better results.

[Aexoden]

Have you tried to connect to all pools in the list, timing LongPollings and (as an easy start) assuming the pool from which the first LP comes has found the block? This could also be factored in additionally with your scoring approach and should probably give some better results.

I meant to write something to test that strategy, but I haven't gotten around to it yet. Though, the data from https://forum.bitcoin.org/index.php?topic=32664.0 suggests it would at best be another piece of the puzzle, rather than a complete solution. One of the problems is that you're still trusting pools to not obfuscate things. (At least one pool was talking about delaying long polls to known hoppers--I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.) Then again, we're still trusting pools to report which blocks they've solved accurately.

[hawks5999]

The hopping monster is getting to a Gojira size. Nice.

[m3ta]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

[iopq]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

[1bitc0inplz]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin' 

Though, I agree, Prop is broken. Miners need a fair reward system (*cough* ESMPPS *cough*), and fighting pool hoppers by hiding stats, faking stats, or anything other than just moving away from Prop is not solving the problem.

[hawks5999]

Agree with bitp op, PPLNS is probably the least fair even including Prop. With prop every share is worth 'something'. With PPLNS a lot of work can end up completely wasted and worth 0.

Bitp is a great backup pool by the way.

[bitcoindaddy]

I'm getting the following errors. I tried a full pools.conf at first, then reduced it down to a single mtred xxxx xxxx line and it still doesn't work for me.

./poclbm.py -d 3  -v -f 1 -w 128
Traceback (most recent call last):
  File "./poclbm.py", line 68, in <module>
    miner = BitcoinMiner(devices[options.device], options, VERSION, HttpTransport.HttpTransport)
  File "/home/user/Downloads/poclbm-aexoden/BitcoinMiner.py", line 37, in __init__
    self.transport = transport(self)
  File "/home/user/Downloads/poclbm-aexoden/HttpTransport.py", line 21, in __init__
    super(HttpTransport, self).__init__(miner)
  File "/home/user/Downloads/poclbm-aexoden/Transport.py", line 37, in __init__
    say_line('Switching to {} with utility {:.3f}'.format(self.best_pools[0].name, self.best_pools[0].utility), show_server=False)
ValueError: zero length field name in format



sudo apt-get install python-scipy python-httplib2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python-httplib2 is already the newest version.
python-scipy is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.

[Aexoden]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin' 

Sure, any given share may not be paid, but the expected value for every share is (for the moment) 50 / difficulty. (In the case of mineco.in, it's actually a little greater, since they return transaction fees.) Remember, if (for example), N is set to difficulty / 2 (as at mineco.in), shares will be paid out with the following probabilities:

• No payout: ~60.65%
• 1 payout: ~30.33%
• 2 payouts: ~7.58%
• 3 payouts: ~1.26%
• 4 or more payouts: ~0.18%

The weighted average here is 0.4999 payouts per share. (If I had bothered calculating X number of payouts up to infinity, the sum would be 0.50.) Since each share will be paid out double the expected 50 / difficulty reward (because N is half the difficulty), the expected value per share is 1.0. Some work may seem wasted, but over time, you'll get exactly what you expect to get.

Ultimately, PPLNS definitely has higher variance than *SMPPS, but it's not statistically any more unfair. With a large pool like Deepbit, Slush, or BTC Guild, the variance would be quite low, even for intermittent miners. I also don't think it's vulnerable to the whole "unlucky streak killing the pool" problem. In any case, I currently treat PPLNS, Geometric, and SMPPS pools as "fair" in the hopper. Of course, if any SMPPS pool finds itself deep in a hole, I may have to adjust the utility of that pool.

[Aexoden]

I'm getting the following errors. I tried a full pools.conf at first, then reduced it down to a single mtred xxxx xxxx line and it still doesn't work for me.

./poclbm.py -d 3  -v -f 1 -w 128
Traceback (most recent call last):
  File "./poclbm.py", line 68, in <module>
    miner = BitcoinMiner(devices[options.device], options, VERSION, HttpTransport.HttpTransport)
  File "/home/user/Downloads/poclbm-aexoden/BitcoinMiner.py", line 37, in __init__
    self.transport = transport(self)
  File "/home/user/Downloads/poclbm-aexoden/HttpTransport.py", line 21, in __init__
    super(HttpTransport, self).__init__(miner)
  File "/home/user/Downloads/poclbm-aexoden/Transport.py", line 37, in __init__
    say_line('Switching to {} with utility {:.3f}'.format(self.best_pools[0].name, self.best_pools[0].utility), show_server=False)
ValueError: zero length field name in format

Before I ask anything else, what version of Python are you using?

[bitcoindaddy]

2.6.6-2ubuntu2

[Aexoden]

2.6.6-2ubuntu2

I've pushed a fix that may solve your problem (due to using Python 2.6). I use Python3 for my personal projects almost exclusively, and only use python 2.7.x when I'm forced to use Python2. If the fix doesn't work, let me know, and I'll try to get a 2.6.x testing environment.

[c00w]

Um if by geometric you mean slush's algorithm it is hoppable.

[Aexoden]

Um if by geometric you mean slush's algorithm it is hoppable.

The Geometric method refers to Meni's scoring system.

[Sukrim]

*SMPPS is broken/high risk of not returning 100% of expected value per share as soon as the pool hits negative, which should happen ~50% of the time.

PPLNS can choose arbitrarily large Ns (the highest useful N is 5 000 000 000, awarding each share with 1 Satoshi when blocks are found) if people really feel left out or are not able to get that they have a chance to get paid multiple times, equalizing the chance of not getting paid at all.
All that is needed is a proper interface, that shows percentages instead of "expected payout", as PPLNS looks forward to define rewards, not so much backwards as prop.


That said, as soon as pool operators start delaying LongPolls they really start to hurt their own profit + their own miners. Just think of the fact, that every millisecond in delay lets the miners compute 100 000 hashes per 100 MH/s in vain and will create more stale shares.
Also, if you just have to submit a valid solution every 10 minutes or so to count as an "active miner", one might be able to just pull a getwork from there every 10 minutes, solve + submit it, to not get throttled. The shorter this window, the more miners the operators hit + screw over, the longer, the easier it gets to fake a CPU miner.

[bitcoindaddy]

I've pushed a fix that may solve your problem (due to using Python 2.6). I use Python3 for my personal projects almost exclusively, and only use python 2.7.x when I'm forced to use Python2. If the fix doesn't work, let me know, and I'll try to get a 2.6.x testing environment.

Yes, it's working now, thanks

[Clipse]

This is getting interesting, may the most efficient hopper win

[hawks5999]

So when does hopping get its own section under mining?

[bitcoindaddy]

Can you explain what the different fields mean on the last line of the display?
Am I only getting an effective rate of 294 MHash/sec?

Code:

./poclbm.py -d 3  -v -f 1 -w 128
31/07/2011 18:05:09, Switching to bitcoins.lc with utility 1.057                                   
31/07/2011 18:05:09, Current utilities:                                                             
31/07/2011 18:05:09,   bitcoins.lc    : 1.057                                                       
31/07/2011 18:05:09,   arsbitcoin     : 1.000                                                       
31/07/2011 18:05:09,   bitpit         : 1.000                                                       
31/07/2011 18:05:09,   eligius        : 1.000                                                       
31/07/2011 18:05:09,   triplemining   : 0.721                                                       
31/07/2011 18:05:09,   mtred          : 0.705                                                       
31/07/2011 18:05:09,   btcguild       : 0.630                                                       
31/07/2011 18:05:09,   ozco.in        : 0.390                                                       
31/07/2011 18:05:09,   nofeemining    : 0.251                                                       
31/07/2011 18:05:09,   rfcpool        : 0.248                                                       
31/07/2011 18:05:09, Setting server (xxxxx @ bitcoins.lc)                                         
bitcoins.lc 31/07/2011 18:05:11, LP connected to bitcoins.lc                                       
bitcoins.lc [426.863 MH/s (~294 MH/s Eff:0.10)] [Rej: 0/10 (0.00%)] [GW: 104 (Eff:0.10)]           

[Aexoden]

Can you explain what the different fields mean on the last line of the display?
Am I only getting an effective rate of 294 MHash/sec?

Code:

                                   
bitcoins.lc [426.863 MH/s (~294 MH/s Eff:0.10)] [Rej: 0/10 (0.00%)] [GW: 104 (Eff:0.10)]           

You've only submitted 10 shares, so the averages haven't really had time to stabilize. Here's a brief explanation of the various fields. Some are in stock poclbm, and some are from luke-jr's version.

The second section is your estimated hash rate based on accepted shares in some recent time period. Over time, it should be roughly equal to the first field, unless you're getting a lot of rejected shares. Eff stands for efficiency and refers to your recent ratio of shares per getworks. On some pools this will be low, and on others it may be 5 or greater.

The next section tells you the number of rejected and total shares you've had. (In this case, you've submitted 10 shares, all accepted.)

The final section tells you the number of getworks the client has issued and the overall efficiency. Better efficiency is primarily better for the pool, rather than for the miner, but it doesn't hurt, either.

[MrWizard]

Finally got it running on Windows 7.  I like it.  However, my hash rate is not as high as poclbm.exe Windows native executable.  My hash rate dropped to 305 from 316 MH/s.  So probably will not use your auto-hopper unless you can get the hash rate a little higher.

[iopq]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

[hawks5999]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

"it does reward every share equally" != "because every share has the same chance of being inside and outside the window"

That's like saying every lottery ticket is a winner because every lottery ticket has a chance to win

or

Every candidate is hired because every candidate has a chance to be hired

or

Everyone is a millionaire because everyone has a chance to be a millionaire

[iopq]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin'  

it does reward every share equally because every share has the same chance of being inside and outside the window

most *PPS variants are vulnerable to BLOCK OF DEATH problem where a block or several long blocks could delay or otherwise affect the payouts for new members which discourages new people from joining the pool since the pool has to pay the old members back for their work despite being in the hole

PPLNS doesn't suffer from this problem, because when a block is long, the older members just don't get paid - they didn't get lucky
this means when a new member joins he has the same chance as anyone else to get paid and get paid promptly/in full

"it does reward every share equally" != "because every share has the same chance of being inside and outside the window"

That's like saying every lottery ticket is a winner because every lottery ticket has a chance to win

or

Every candidate is hired because every candidate has a chance to be hired

or

Everyone is a millionaire because everyone has a chance to be a millionaire

you're just arguing semantics

[Aexoden]

I have modified the block probability API to at least partially score the blocks to determine the probability that each pool produced it, rather than simply using each pool's estimated hash rate. However, both factors are considered.

This should increase overall efficiency by some unknown degree, but since I have no easy way to measure such a thing (other than creating new accounts on all the pools and mining using only this miner), we may never know for sure. The most likely result of this change is that BTC Guild and bitcoins.lc should see increased time as the pool with highest utility.

Since this change is on my backend, no upgrade is necessary. Clients will automatically pick up the new data.

[MrWizard]

I have modified the block probability API to at least partially score the blocks to determine the probability that each pool produced it, rather than simply using each pool's estimated hash rate. However, both factors are considered.

This should increase overall efficiency by some unknown degree, but since I have no easy way to measure such a thing (other than creating new accounts on all the pools and mining using only this miner), we may never know for sure. The most likely result of this change is that BTC Guild and bitcoins.lc should see increased time as the pool with highest utility.

Since this change is on my backend, no upgrade is necessary. Clients will automatically pick up the new data.

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

[Aexoden]

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

I have seen that, and I have no idea why it happens. poclbm is such a mess in general, that it's hard to do any real sort of debugging. When it's happened to me, I've noticed that I can make it go away by running a CPU-bound process in the background (like a CPU miner). I also haven't the foggiest idea why that would help, but without knowing the reason for the rejected shares, it's hard to debug.

I utilize poclbm's built-in multi-server support to handle the multiple servers, and I don't think it handles switching very well, sometimes leading to additional rejects. However, fixing poclbm itself isn't very fun for me.

Also, I just noticed your message about hash rates. I haven't personally noticed too much of a drop, but my Windows miner has never done very well at staying at peak, even using stock poclbm. Linux seems fine with both, in any case.

[techwtf]

Thanks for the update.  How are your stale shares?  In the Windows version that I built the program usually gets into a mode where it continuously produces stale shares after running for a few minutes.

Maybe the pool disabled N time rolling. and I don't really know why "nonces_left += 0xFFFFFFFFFFFF" is required, so I removed these 2 parts of code. one getwork, one response expected. that's the normal behaviour in phoenix miner.

Add: I still using a heavily modded poclbm/2011.b7. No plan to catch up to the new version ...

[Artefact2]

Hello. I see you are using pident for choosing where to hop. I feel obliged to remind you that accuracy is fairly low ( http://pident.artefact2.com/accuracy ).

But hey, you said you tweaked it so it gathers more data… If you managed to improve the accuracy, could you care to share the source so that I can merge it? I know the WTFPL doesn't require you to do that, but it would be nice. If your public domain miner relies on a closed-source web API, then it's of no use

[Aexoden]

Hello. I see you are using pident for choosing where to hop. I feel obliged to remind you that accuracy is fairly low ( http://pident.artefact2.com/accuracy ).

But hey, you said you tweaked it so it gathers more data… If you managed to improve the accuracy, could you care to share the source so that I can merge it? I know the WTFPL doesn't require you to do that, but it would be nice. If your public domain miner relies on a closed-source web API, then it's of no use

The only reason it isn't released anywhere is because it's not up to my own standards for sharing (it has a bit of site-specific code in it, and isn't very well organized, because I changed how I implemented it halfway through). If I have a chance, I'll try to refactor it a bit and fork the pident repository on github. However, even if I do, it probably wouldn't be too mergeable. I tried to work entirely on top of pident, rather than modifying it, for two reasons:

• First, this made it easier to merge any changes you made.
• Second, I gave up trying to figure out exactly where in the code the scoring was happened. I think I figured it out since then, but as you'll see below, I'm not sure the work I've done is directly applicable to improving the scores themselves.

Regardless, I can at the very least attempt to describe my current thought process and algorithm.

The first instinct is to just take whichever pool has the highest score and assume they found the block. As you've said, however, the accuracy of that method isn't incredibly high. For the purposes of pool hopping, we don't want to guess which pool found the block. Rather, we want to take all the available information, and assign probabilities to each pool. Using these probabilities, we can calculate the utility of mining at each pool.

At first, I just used each pool's percentage of the overall network, which is accurate, but I figured it should be possible to do better with additional information, namely pident's scoring. The trick is figuring out how to massage the scores pident produces into probabilities. To that end, I sampled an equal number of blocks from each pool's set, producing a block universe where each pool is equally likely to find a block. pident's scoring works based on identifying unique characteristics of each pool's blocks. It (to my knowledge) has no idea that some pools are simply more likely to find blocks, so modeling on this sampled block universe makes sense to me.

Treating the scores, then, as relative confidences that each pool found the given block, we rescale them so they sum to 1, so they at least have a chance of representing probabilities. Once I did this, and plotted the likelihood that each range of scores meant a pool did find the block, there was a definite pattern. (In the particular data set I looked at, a rescaled score of 0.06-0.07 implied a ~13.5% chance that the pool found that block.) By the time we reach scores of 0.1, the probability is almost 100%. Ultimately, I modeled the data using the atan function (which matches the curve). (Does this make theoretical sense? Who knows, but as long as it works practically, I'll go with it.) The idea here is to scale the scores so that a score of 0.5 implies a 50% probability. I eyeballed the function, and it's not very forum friendly, but it matches the data fairly closely.

The end result of all this massaging of the original scores produced by pident is a rough probability that the block belongs to each pool, but only in the magical "every pool is equally likely" universe. To get back to the real world, we multiply each score by the pool's percentage of the overall hash rate. The last step is rescale these final values so they again sum to 1. That's the final probability I use in calculating the data in the API.

The next step is to test these numbers and make sure they at least come close to matching the ultimate results. I'll probably wait a few days so I have some out-of-sample data before I embark on that quest, however.

Other next steps:

• Getting the code postable. I have no desire to keep it that close to my vest.
• Deciding on the best way to implement the hopping strategy implied by the API. poclbm is kind of a pain. I contemplated modifying cgminer, but that's quite a chore. If recent improvements to bitHopper have made it more usable (e.g. unbroken LP support, and not spamming pools with high-frequency getwork requests), it might be best to implement it there. I'd have to compare its average reject percentage with poclbm.

[Artefact2]

Okay, thanks for the explanation! I'm very interested in any pool using pident, and if you need help to hack it (like, where/how are the scores calculated ), ask me on IRC or by email

[bitcoindaddy]

I noticed PolMine, MainframeMC and bitcoinpool are in your data-feed, but not in the pools.py code.  Are those not ready for prime-time or were they omitted for a reason? I've had good luck with PolMine.

Another one to consider adding is pool.bloodys.com, though they are fairly small. Is there a lower limit on the GH/s that you consider reasonable?

[Aexoden]

I noticed PolMine, MainframeMC and bitcoinpool are in your data-feed, but not in the pools.py code.  Are those not ready for prime-time or were they omitted for a reason? I've had good luck with PolMine.

Another one to consider adding is pool.bloodys.com, though they are fairly small. Is there a lower limit on the GH/s that you consider reasonable?

The main thing deciding which pools I support is the pool support in pident. I could add them myself, but it seems like there are new ones popping up every day, so keeping track and writing the parsers might be more than I want to take on. Practically, I probably wouldn't bother adding a pool that's average block solve time is longer than a week or so.

Before adding any pool to the actual miner, I have to review it and determine (as best I can) the reward system used and its expected utility. (For instance, I don't currently support Slush in the miner because I haven't yet figured out a way to calculate its utility. I haven't tried very hard yet, either, but it may not even be worth the effort.) So, reviewing the ones you mentioned:

• PolMine: I can't actually in a quick search verify their reward system. If I can figure that out, I'm not opposed to adding it, but I probably wouldn't sign up there myself.
• MainframeMC: Claims to use a cheat-proof scoring system. I haven't investigated further. At best, it'd be a good backup pool. Not opposed to adding it, if I can confirm the reward system.
• bitcoinpool: Last time I checked, they use a hopping deterrent (halving the value of shares if you mine less than 50% of the round). This should still be hoppable (but less so), but I'd have to come up with a good utility function. Since rounds shorter than an hour aren't subject to this, it's not quite as cut and dry as halving the utility of a normal prop pool (but that could be a good starting point). By the way, the hop point in the "halve the rewards" scenario is ~10%.
• bloodys: Not currently in pident, I don't think.

Also, my long-term rejected shares seem to be somewhere in the neighborhood of 4-5%. This is definitely more than mining a single pool with poclbm, but presumably the hopping bonus makes up for it. It's still worth working on, however. For me, most of them seem to happen right after switching pools, so something's probably going wrong there.

I'm also checking my efficiency on BTC Guild to see if the new hopping method is giving positive results, but it'll be several days before there's enough data to even consider. Successfully defeating the one hour delay will be necessary, otherwise more pool operators may consider it as a solution, rather than changing to a non-broken reward system.

[Aexoden]

The code used to generate the API is now available on github: https://github.com/aexoden/pident

The code is located entirely in the new printpooldata file, which prints out the JSON object. I haven't included my custom update script, at this time, however. (It's badly organized and has some site-specific code in it.) The update script is executed by cron once a minute, and approximately does the following things:

• Executes updateblocks
• If a new block was detected, queues an updatepools to run in two minutes (who knows if this delay is long enough/too long--I'm just trying to minimize unnecessary network access).
• If there were new blocks, or if updatepools was executed, a new copy of the pool data is generated and copied to my server, which then serves it up.

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

By the way, I'm under no illusion that printpooldata couldn't use some optimization. However, doing so isn't really worth the effort for me. It might seem kind of haphazard because of changing my approach a couple of times in the middle, and not completely rewriting to match.

[Artefact2]

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

Hmm, it doesn't appear to be the case in your repo. Also, unless you already use them, declareCache() and declareCacheExpiration() will do the job in 2 lines of code (Etags: and Expires: fields for client-side cache and server-side caching of the generated page). In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

[hawks5999]

In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

Now you are throwing down some massive geek braggin'

"Oh yeah, well my cache is so awesome, apache tell me to keep the change. BOOM!"

[Aexoden]

The API on my server honors the If-Modified-Since HTTP header, and will send a 304 unless the data has been updated. It will also gzip compress the data if the user agent supports it.

Hmm, it doesn't appear to be the case in your repo. Also, unless you already use them, declareCache() and declareCacheExpiration() will do the job in 2 lines of code (Etags: and Expires: fields for client-side cache and server-side caching of the generated page). In fact, my cache is so awesome it's probably the piece of code in pident I'm the most proud of.

Pident itself was far too I/O heavy to run on my VPS, at least during the import. Maybe I just had too little RAM or needed to optimize postgres. Either way, for the moment I generate the data locally and upload it to the server where it's served by a short custom script to handle the headers and gzip.

Meanwhile, I am working on reducing rejects, but it'll be a few hours before I can see if my changes have improved things.

[Aexoden]

I've pushed a commit that may help to reduce rejected shares. Before this patch, my rejected shares ranged from 3-6%, depending on the frequency of hopping. My limited test suggests this has been drastically reduced, but the old bitcoins.lc URL wasn't working all day, so the test may be flawed. I'll post more results in about 12-18 hours.

[Aexoden]

I've added support for MainframeMC to the miner. Note that since it carries a 1.5% fee, it won't be used unless you have no other fair pools configured.

I've also added the ability to set a donation percentage for each pool in pools.conf. This will allow the miner to more accurately calculate the pool's utility. The field is optional, so your old pools.conf will continue to work with the new miner. If you are, for instance, donating 1.0% to a pool, you should add 1.0 to the end of the line.

Finally, the reject reduction work seems to be working well. Long-term reject percentage on my miners seems to be somewhere between 0.5% and 1.5%.

There are still several pools that are not supported:

• Bitcoinpool: Uses a modified proportional system for which I need to derive an accurate utility function. Given the Draconian TOS they have, I'm unlikely to bother.
• BTCMine: Uses the exponential decay scoring system with an unknown magic number. It may be possible to estimate the utility function, but I haven't looked into it.
• BTCMP: May add at some point.
• DeepBit: I will not add this as long as it makes up more than 33% of the network.
• PolMine: As long as I can confirm and keep updated on its reward system, I can add it.
• Slush: Like BTCMine, I'd have to evaluate the scoring system.

If a pool is not listed, it's not currently supported by pident. (Bitclockers is not supported by pident because it obfuscates its blocks, so no one knows exactly which blocks belong to it.) You can presumably talk to Artefact2 about getting pools added to pident.

[hawks5999]

Have you collaborated at all with c00w, bloodred, ryouiki or other hopping developers? Seems like some of them have different takes on various pools and are able to support some that you aren't. I also think their projects would benefit from your work with pident. Eventually I'd like to see all your projects roll together into one super hopping client.

[Aexoden]

Have you collaborated at all with c00w, bloodred, ryouiki or other hopping developers? Seems like some of them have different takes on various pools and are able to support some that you aren't. I also think their projects would benefit from your work with pident. Eventually I'd like to see all your projects roll together into one super hopping client.

My primary goal was to distrust the pools as much as possible. It'd be easy to hop a bunch of other pools if I were to just use their APIs. (In fact, older versions of poclbm-autohop did just that. Only pool I lost in the transition was bitclockers, though.) Of course, I'm trusting pools to report their blocks accurately. They might delay those stats, but I think most pools are unlikely to hide them completely. However, they might take a bitclockers type approach, where they make it almost impossible to associate the blocks with real blocks (by not providing the block number or hash).

In other news, I'm ready to report the first results of efficiency hopping BTC Guild. At first glance, it won't look good. My efficiency has been a paltry 81%. However, it's not as bad as it sounds. BTC Guild is stuck in a period of bad luck. A full-time BTC Guild miner's efficiency over the same period is only 72%. (Both of these numbers are relative to what the miner would have earned at a 0% PPS pool.) In the end, it looks like we should still be able to effectively hop pools that are delaying their statistics, if at a reduced efficiency.

[Aexoden]

Recent updates:

• Support for Bitminter was added.
• Since RFCPool has closed, support has been removed.

Otherwise, not much is happening. BTC Guild results are still significantly better than for a full-time miner.

[roos]

I get the following when trying to start poclbm for the first time after adding the following pools to pools.conf

bitpit
bitcoins.lc
bitminter
btcguild
mtred
ozco.in


Traceback (most recent call last):
  File "./poclbm.py", line 67, in <module>
    miner = BitcoinMiner(devices[options.device], options, VERSION, HttpTransport.HttpTransport)
  File "/home/roos/src/aexoden-poclbm-2889219/BitcoinMiner.py", line 33, in __init__
    self.transport = transport(self)
  File "/home/roos/src/aexoden-poclbm-2889219/HttpTransport.py", line 21, in __init__
    super(HttpTransport, self).__init__(miner)
  File "/home/roos/src/aexoden-poclbm-2889219/Transport.py", line 32, in __init__
    self.best_pools = self.pools.get_best_pools()
  File "/home/roos/src/aexoden-poclbm-2889219/pools.py", line 97, in get_best_pools
    return sorted(self.pools.values(), key=lambda pool: (pool.utility, pool.priority), reverse=True)
  File "/home/roos/src/aexoden-poclbm-2889219/pools.py", line 97, in <lambda>
    return sorted(self.pools.values(), key=lambda pool: (pool.utility, pool.priority), reverse=True)
  File "/home/roos/src/aexoden-poclbm-2889219/pools.py", line 125, in utility
    self.update_data()
  File "/home/roos/src/aexoden-poclbm-2889219/pools.py", line 116, in update_data
    self.rate = get_pool_rate(self.pident_name)
  File "/home/roos/src/aexoden-poclbm-2889219/pools.py", line 31, in get_pool_rate
    return _pool_data[0]['rates'][pool] if pool in _pool_data[0]['rates'] else 0.0
KeyError: 'rates'

[Aexoden]

I get the following when trying to start poclbm for the first time after adding the following pools to pools.conf

bitpit
bitcoins.lc
bitminter
btcguild
mtred
ozco.in

My copy of pident is currently having problems with its scoring system, causing it to not properly update the API. I'm attempting to fix it at the moment. It should work again in a few minutes, though.

EDIT: API is running again. Still not sure what caused pident to start having lots of division by zero errors. Since its base scoring code is opaque to me at the moment, I'm not in much of a position to debug it effectively. However, I have avoided the error as a stopgap measure.

[Aexoden]

As it turns out, the donation percentage code never actually correctly read the donation percentage from the configuration file. This has been fixed.

In other news, there will be approximately a one week period in late September where the API will not be updated. As a result, anyone using this miner will most likely only mine at fair pools for that week, rather than hop.

The ultimate goal, however, should be to somehow improve the whole process so that it doesn't rely on a third-party API. However, the reliance on pident makes this quite difficult. I have little time for bitcoin at the moment, so I'm probably not going to come up with anything myself anytime soon.

[roos]

This doesnt look right, btcguild had a new block 20mins ago and several others have had blocks too.


30/08/2011 21:31:18,   bitpit         : 1.000                                   
30/08/2011 21:31:18,   eligius        : 1.000                                   
30/08/2011 21:31:18,   mtred          : 0.310                                   
30/08/2011 21:31:18,   ozco.in        : 0.298                                   
30/08/2011 21:31:18,   bitcoins.lc    : 0.211                                   
30/08/2011 21:31:18,   triplemining   : 0.191                                   
30/08/2011 21:31:18,   bitminter      : 0.190                                   
30/08/2011 21:31:18,   btcguild       : 0.079                                   
30/08/2011 21:31:18,   nofeemining    : 0.054   

[Aexoden]

This doesnt look right, btcguild had a new block 20mins ago and several others have had blocks too.


30/08/2011 21:31:18,   bitpit         : 1.000                                   
30/08/2011 21:31:18,   eligius        : 1.000                                   
30/08/2011 21:31:18,   mtred          : 0.310                                   
30/08/2011 21:31:18,   ozco.in        : 0.298                                   
30/08/2011 21:31:18,   bitcoins.lc    : 0.211                                   
30/08/2011 21:31:18,   triplemining   : 0.191                                   
30/08/2011 21:31:18,   bitminter      : 0.190                                   
30/08/2011 21:31:18,   btcguild       : 0.079                                   
30/08/2011 21:31:18,   nofeemining    : 0.054   

Looks like the update process locked itself up ~18 hours ago. It should be catching up now, unless there's a bigger problem, which I'll find out shortly, I assume. Consider this a preview of how things will look when updates are suspended for ~8 days in late September.

[organofcorti]

New "how to hop" blog post:

How to hop part2: More on score

This week we discover Slush-type score systems fatal flaw, an update on the Slush hop point (previous estimate nearly 50% out), and what 'c' really means, and how to turn that old newspaper into something beautiful!

[Aexoden]

New "how to hop" blog post:

How to hop part2: More on score

This week we discover Slush-type score systems fatal flaw, an update on the Slush hop point (previous estimate nearly 50% out), and what 'c' really means, and how to turn that old newspaper into something beautiful!

Though, in order to add slush to this program, at least, I'd have to figure out how to calculate the utility of a given share. It should be possible once I've done the relevant analysis, but I don't exactly have the motivation to do such a thing at the moment.

As a second point, I'm worried that the list of pools I'm operating with is languishing a bit. It seems NoFeeMining changed to RSMPPS at some point, so it's been changed to a fair pool. Have there been any other proportional pools started? I hope we're past that point, given how easy it is to hop them. I'm unsure about adding any additional fair pools, as I already support several. In any case, they'd have to be added to pident first.

[organofcorti]

Though, in order to add slush to this program, at least, I'd have to figure out how to calculate the utility of a given share. It should be possible once I've done the relevant analysis, but I don't exactly have the motivation to do such a thing at the moment.

How do you define utility? If it is the expected value of a share if a given number of shares in a round have already occurred, I can send you a score table, or you can generate one yourself. Otherwise, let me know if I can help.

[Aexoden]

Though, in order to add slush to this program, at least, I'd have to figure out how to calculate the utility of a given share. It should be possible once I've done the relevant analysis, but I don't exactly have the motivation to do such a thing at the moment.

How do you define utility? If it is the expected value of a share if a given number of shares in a round have already occurred, I can send you a score table, or you can generate one yourself. Otherwise, let me know if I can help.

Based on available data, usually things like the current number of shares (but in slush's case, also the time since the last block and the hash rate, I believe), the expected value of a share, relative to PPS. I prefer a mathematical function to a table. For instance, the relevant function for proportional pools is equal to 1.0 at ~0.4348.

[organofcorti]

No problem. I'll run eureqa on Monday and make you a utility function for slush that will be accurate to at least 0.5*diff. That ok? It won't be derived from the calculus but it will work.

[Aexoden]

No problem. I'll run eureqa on Monday and make you a utility function for slush that will be accurate to at least 0.5*diff. That ok? It won't be derived from the calculus but it will work.

As long as it's relatively accurate up until the crossover point (when it drops below 1.0) it should be fine for my purposes.

[Vladimir]

Motivation

Many mining pools are still using the inherently unfair proportional payout system. My primary intent is to encourage pool operators to migrate their pools to fair systems. It seems as if the only way this will happen is if miners themselves see that the proportional system is unfair. (Unfortunately, the fair systems often seem unfair at first glance, making explanations more difficult.)

Good idea. The more the merrier. In all seriousness.

[MrWizard]

PPS/variation and prop is the most fair reward system and pplns is the most dangerous reward system and you miners should run away from pplns, maybe just maybe you need a slap to learn a lesson

+1

[cuqa]

Everyone should read this

After much consideration and review of how our anti-pool hopping system is looked upon by members of the Bitcoin community, we have chosen to change the way in which it is working.

In the Past:
If a user participated in less than 50% of the round, their shares would be reduced by 10%, unless a donation was set, in which case, no penalty was applied.

This allowed for our users with legitimate disconnects, or reasons for being unable to mine during long rounds to avoid being hit with a penalty.

Some people seem to belive that this is a negative aspect of the anti-hopping system due to the easily cheatable nature of the system and the fact that pool hoppers are still given a large incentive to exploit our pool.

From Now on:
If a user participates in less than 50% of the round, their shares will be reduced by 50%, regardless of donation. 50% of the penalty fee will be directed toward the donations account and will be applied to server costs and future monthly contests. The other 50% of the penalty will be removed from the total shares for the round, which will in-hand cause the value of all remaining shares in the round to increase.

Example -

100,000 shares are in the round.

User A has 5000 shares and is NOT a hopper. Their estimated earnings before penalties are applied is 2.5 BTC.

User B has 5000 shares and IS A hopper. Their estimated earnings before penalties are applied is 2.5 BTC.

Once penalties are applied, User B's shares are reduced to 2500 where 1250 has been credited to donations and 1250 has been removed from the round. After the penalty is applied, User B's unconfirmed earnings are 1.265 BTC and User A's unconfirmed earnings are 2.5316 BTC.

At this time, any donation currently set on your account will remain in place. Donations will be processed prior to any penalties. As a reminder, donations are entirely voluntary and are not required to use this pool.

We will use this new method of calculating penalties for a one week trial period, at which time we will decide whether or not it is effectively deterring pool hopping and effectively rewarding loyal members of the pool.

Regards,

BitcoinPool Staff

[iopq]

pplns is the most dangerous reward system and you miners should run away from pplns

why is that? because pool owners can hide solved blocks?

[Aexoden]

PPS/variation and prop is the most fair reward system and pplns is the most dangerous reward system and you miners should run away from pplns, maybe just maybe you need a slap to learn a lesson

I can't think of any reason PPLNS would be more dangerous than any other system. Assuming you can trust the operator, it is both hopping-proof and much more resilient to long periods of bad luck than any of the PPS variants.

If you can't trust the pool operator, then all systems have potential for abuse.

Everyone should read this

I had avoided adding bitcoinpool completely because of its Draconian measures designed to fight pool hoppers (rather than changing reward systems). Of course, as hopping becomes more prevalent, you can expect more pools to go down this road. Hopefully, they'll have to become so awful that users won't want anything to do with them, but I don't have high hopes in that regard. Users seem willing to tolerate a lot of abuse.

[iopq]

users still use slush, which is hoppable, api-friendly, and still doesn't have LP afaik
worse, it has a 2% commission and a high-variance payment method

[organofcorti]

...and I'll have that slush hop function for you ASAP, Aexoden, I just have to deal with this bcp brushfire first.

[organofcorti]

New 'how to hop' blog post - How to hop 3: the 50-50 tax

This week I take a break from exponentially scored pools to look at how to hop this new type of scored pool. Then I  show you the best way to determine when to hop from one type of pool to another (and it's not always the pool with the least shares). Read it and reap!

Aexoden, I've also included the first round of utility functions. Hop point functions to follow

[Aexoden]

The API is currently not being updated due to a problem with my bitcoind. I will attempt to fix it as soon as I can, but it may be a few hours. Until then, miners will mine at fair pools most likely.

UPDATE: API was fixed a few hours later. Of course, remember once again that it will not update for approximately a week in late September.

[organofcorti]

New How to hop out now!

'How to hop' - now with even more chart porn.

[roos]

Looks like it stopped updating again...

[Aexoden]

Looks like it stopped updating again...

Looks like the update process froze again. Sometimes, the pident script that checks which pools have solved which blocks never finishes, and since I've implemented some locking, so two updates never run simultaneously, it gets jammed up. I should probably set up a watchdog at some point, or at least figure out why updatepools doesn't ever quit sometimes.

Anyway, the situation is fixed, and updates should be coming in the next few minutes as pident catches up with the block chain.

However, please note, once again, that there will be no updates for about 8-9 days starting in about 28 hours. I advise users to investigate other options like bithopper or cherrypicking for that duration. I thought about putting together some kind of limited update script, but collecting all the data from the pools might be more than I can churn out in the next 24 hours (especially since my time is limited).

At some point, I'm going to have to find a way to either modify pident or write a replacement that isn't so hard on disk I/O so I can put it on a VPS. Relying on my local machine is too hit or miss.

[Aexoden]

The API will cease updating sometime in the next hour or two. If you wish to continue hopping effectively, you should probably convert to an alternate solution such as bithopper or cherrypicking at this time. Otherwise, your miner will likely continue to mine at a fair pool such as mineco.in, eligius, or arsbitcoin (if you have one configured). This isn't a bad thing, but you won't be maximizing your reward.

I expect the API to begin updating again on either the 29th or 30th, or sometime before that if I manage to come up with an alternate solution (but I wouldn't count on it).

[DrHaribo]

BitMinter switched to a variation of PPLNS today, so if you were pool hopping us you may want to remove us from your setup.

We'll still be paying 5% extra for over 40 more blocks, so we can be a good default, though.

[Aexoden]

BitMinter switched to a variation of PPLNS today, so if you were pool hopping us you may want to remove us from your setup.

We'll still be paying 5% extra for over 40 more blocks, so we can be a good default, though.

I think BitMinter was hopped, so I'll make the change whenever I get a chance. Thanks for letting me know, because it can be hard to keep track of all the pools myself.

Congratulations on switching to a fair reward system. I'll make sure to leave bitminter in as an option for people to use as their fair backup.

[Aexoden]

The API is currently in the process of being updated. The scripts are currently catching up with the last week's worth of blocks, but it should be caught up shortly.

[organofcorti]

How to hop 5 out now! Ever wonder why pool hopping works? Wonder no longer. No math required.

[organofcorti]

I'm now posting update announcements about new 'how to hop' posts at here. Make a comment to subscribe!

Also, there's a new 'How to hop', and the last one has been rewritten with errors removed.

Enjoy!

[Aexoden]

At this point, I am neither using poclbm-autohop or poclbm at all, so this program is likely to slowly enter a state of bit rot. I will continue to update the API as long as it is feasible to do so, but since pident is seemingly no longer maintained, that may not be as long as we think.

I still find the current methodology interesting, and if proportional pools continue to propagate, I would like to reimagine the implementation. Most likely, I would write a custom (and open source) program to generate the API automatically without reliance on external software like pident, and design it to be non-resource intensive, to allow it to be deployed by multiple people if necessary. The biggest problem is determining what kind of resources are needed to store block information, and whether or not to implement some kind of scoring algorithm. (It's worth investigating, in any case.)

The second component would be a new miner. At this point, it's likely I would attempt to modify cgminer to access the API and use its data. I did a cursory look into what would be needed a couple of months ago, but I suspect cgminer has changed slightly since then. In any case, it's dubious that I'll have the time to get any of this done anytime soon.

[organofcorti]

PPS/variation and prop is the most fair reward system and pplns is the most dangerous reward system and you miners should run away from pplns, maybe just maybe you need a slap to learn a lesson

I can't think of any reason PPLNS would be more dangerous than any other system. Assuming you can trust the operator, it is both hopping-proof and much more resilient to long periods of bad luck than any of the PPS variants.

If you can't trust the pool operator, then all systems have potential for abuse.

Everyone should read this

I had avoided adding bitcoinpool completely because of its Draconian measures designed to fight pool hoppers (rather than changing reward systems). Of course, as hopping becomes more prevalent, you can expect more pools to go down this road. Hopefully, they'll have to become so awful that users won't want anything to do with them, but I don't have high hopes in that regard. Users seem willing to tolerate a lot of abuse.

I have a hop function for bcpool's 50-50 tax that you can use if that helps, Aexoden.