[EXPERIMENTAL] poclbm-autohop: Yet Another Automatic Pool Hopper

[Aexoden]

The past few weeks, I have been slowly working on modifying poclbm to be a full-fledged automatic pool hopper. I think it's finally to the point where it might be usable enough to distribute, but I'm still choosing to label it as EXPERIMENTAL.

Motivation

Many mining pools are still using the inherently unfair proportional payout system. My primary intent is to encourage pool operators to migrate their pools to fair systems. It seems as if the only way this will happen is if miners themselves see that the proportional system is unfair. (Unfortunately, the fair systems often seem unfair at first glance, making explanations more difficult.)

Methodology

Originally, poclbm-autohop operated similarly to bitHopper, retrieving share counts from the various pools, and calculating a utility for each one. Recently, however, instead of replacing their broken payout systems with provably fair systems, pools have begun to implement various obfuscation measures, such as delayed statistics. It is apparent that another approach is needed.

Using a local copy of pident to identify the pools that have claimed each block, I calculate the probability that each pool's last block occurred at a given time index. This probability is currently based on each pool's relative hash rate, but work is progressing on improving these probabilities using a heuristic scoring system. (Limited tests show that it may ultimately be possible to predict the finder of a block with at least 30-40% accuracy. Given that most pools make up less than 10% of the network, this is a marked improvement.) I upload these probabilities as well as estimated hash rates for each pool and it available via a JSON API.

(As a quick example, if there's a 10% chance that a share will be worth 10x, and a 90% chance it will be worth 1x, its expected value is 1.9x.)

poclbm-autohop itself then downloads these estimated hash rates and probabilities, and uses them to calculate the utility of each pool. This method will never be as accurate as using the share counts directly, but since many pools are obfuscating or delaying this information, this new approach may be the best we can do.

Caveats

• The estimated hash rates are currently slightly inaccurate for some pools. This will improve within the first week of August, as my local copy of pident collects more data.
• The current code estimates hoppers provide a 30GH/s boost above the average. This is probably a lowball figure, so the utility of pools may be overestimated.
• The probabilities in the API are currently based only on each pool's share of the network.

In addition, while the logical next step for proportional pools would be to replace their broken payout systems, I fully expect pool operators will instead move to no longer report ownership of blocks. Personally, I have to wonder about any pool that would resort to this level of obfuscation rather than simply change their payout model. (Bitclockers already seems to obfuscate their blocks in this way.)

Bugs

• I don't know if it's something my code did or just a problem with poclbm itself, but occasionally it might go into massive reject sprees. This only happens with some pools, so who knows.
• Sometimes things just stop working properly, causing the miner to miss long polls or such, and I'm probably too lazy to dig into poclbm to fix all its problems. I recommend restarting the program periodically (once every day or two, perhaps).

There are probably others that I'm not thinking of, but it should largely work, in any case.

Usage

The code is available via GitHub:

https://github.com/aexoden/poclbm

It should run in any environment that poclbm itself will run, but it does have two additional dependencies: scipy and httplib2. Both should be readily available for both Linux and Windows.

For detailed information, you should definitely consult the README.mkd file. Basically, however, it accepts the same parameters as standard poclbm, except it no longer takes a list of servers as a parameter. Instead, you need to create a pools.conf file, which contains a line for each pool. Each line should say the name of the pool, your worker username, and your worker password, separated by whitespace (I recommend tabs). You may list the pools in your order of preference. If two pools have the same utility, the first pool listed will be preferred. Only pools listed in your pools.conf file will be used.

For the list of supported pools, please consult the README.mkd file.

Conclusion
Comments, suggestions, and questions are welcomed. I can only hope this software will be obsolete very shortly.

[Sukrim]

Have you tried to connect to all pools in the list, timing LongPollings and (as an easy start) assuming the pool from which the first LP comes has found the block? This could also be factored in additionally with your scoring approach and should probably give some better results.

[Aexoden]

Have you tried to connect to all pools in the list, timing LongPollings and (as an easy start) assuming the pool from which the first LP comes has found the block? This could also be factored in additionally with your scoring approach and should probably give some better results.

I meant to write something to test that strategy, but I haven't gotten around to it yet. Though, the data from https://forum.bitcoin.org/index.php?topic=32664.0 suggests it would at best be another piece of the puzzle, rather than a complete solution. One of the problems is that you're still trusting pools to not obfuscate things. (At least one pool was talking about delaying long polls to known hoppers--I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.) Then again, we're still trusting pools to report which blocks they've solved accurately.

[hawks5999]

The hopping monster is getting to a Gojira size. Nice.

[m3ta]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

[iopq]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

[1bitc0inplz]

I'm amazed at the effort being put into fighting pool hopping with methods other than changing reward systems.

+1....

+ONE MIRRION
PPLNS is 0 variance for the pool operator and rewards every share equally for the miners

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin' 

Though, I agree, Prop is broken. Miners need a fair reward system (*cough* ESMPPS *cough*), and fighting pool hoppers by hiding stats, faking stats, or anything other than just moving away from Prop is not solving the problem.

[hawks5999]

Agree with bitp op, PPLNS is probably the least fair even including Prop. With prop every share is worth 'something'. With PPLNS a lot of work can end up completely wasted and worth 0.

Bitp is a great backup pool by the way.

[bitcoindaddy]

I'm getting the following errors. I tried a full pools.conf at first, then reduced it down to a single mtred xxxx xxxx line and it still doesn't work for me.

./poclbm.py -d 3  -v -f 1 -w 128
Traceback (most recent call last):
  File "./poclbm.py", line 68, in <module>
    miner = BitcoinMiner(devices[options.device], options, VERSION, HttpTransport.HttpTransport)
  File "/home/user/Downloads/poclbm-aexoden/BitcoinMiner.py", line 37, in __init__
    self.transport = transport(self)
  File "/home/user/Downloads/poclbm-aexoden/HttpTransport.py", line 21, in __init__
    super(HttpTransport, self).__init__(miner)
  File "/home/user/Downloads/poclbm-aexoden/Transport.py", line 37, in __init__
    say_line('Switching to {} with utility {:.3f}'.format(self.best_pools[0].name, self.best_pools[0].utility), show_server=False)
ValueError: zero length field name in format



sudo apt-get install python-scipy python-httplib2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python-httplib2 is already the newest version.
python-scipy is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.

[Aexoden]

Not to be picky, but PPLNS does *not* reward *every* share equally. Shares that fall out of the PPLNS window are worth 0. Just sayin' 

Sure, any given share may not be paid, but the expected value for every share is (for the moment) 50 / difficulty. (In the case of mineco.in, it's actually a little greater, since they return transaction fees.) Remember, if (for example), N is set to difficulty / 2 (as at mineco.in), shares will be paid out with the following probabilities:

• No payout: ~60.65%
• 1 payout: ~30.33%
• 2 payouts: ~7.58%
• 3 payouts: ~1.26%
• 4 or more payouts: ~0.18%

The weighted average here is 0.4999 payouts per share. (If I had bothered calculating X number of payouts up to infinity, the sum would be 0.50.) Since each share will be paid out double the expected 50 / difficulty reward (because N is half the difficulty), the expected value per share is 1.0. Some work may seem wasted, but over time, you'll get exactly what you expect to get.

Ultimately, PPLNS definitely has higher variance than *SMPPS, but it's not statistically any more unfair. With a large pool like Deepbit, Slush, or BTC Guild, the variance would be quite low, even for intermittent miners. I also don't think it's vulnerable to the whole "unlucky streak killing the pool" problem. In any case, I currently treat PPLNS, Geometric, and SMPPS pools as "fair" in the hopper. Of course, if any SMPPS pool finds itself deep in a hole, I may have to adjust the utility of that pool.

[Aexoden]

I'm getting the following errors. I tried a full pools.conf at first, then reduced it down to a single mtred xxxx xxxx line and it still doesn't work for me.

./poclbm.py -d 3  -v -f 1 -w 128
Traceback (most recent call last):
  File "./poclbm.py", line 68, in <module>
    miner = BitcoinMiner(devices[options.device], options, VERSION, HttpTransport.HttpTransport)
  File "/home/user/Downloads/poclbm-aexoden/BitcoinMiner.py", line 37, in __init__
    self.transport = transport(self)
  File "/home/user/Downloads/poclbm-aexoden/HttpTransport.py", line 21, in __init__
    super(HttpTransport, self).__init__(miner)
  File "/home/user/Downloads/poclbm-aexoden/Transport.py", line 37, in __init__
    say_line('Switching to {} with utility {:.3f}'.format(self.best_pools[0].name, self.best_pools[0].utility), show_server=False)
ValueError: zero length field name in format

Before I ask anything else, what version of Python are you using?

[bitcoindaddy]

2.6.6-2ubuntu2

[Aexoden]

2.6.6-2ubuntu2

I've pushed a fix that may solve your problem (due to using Python 2.6). I use Python3 for my personal projects almost exclusively, and only use python 2.7.x when I'm forced to use Python2. If the fix doesn't work, let me know, and I'll try to get a 2.6.x testing environment.

[c00w]

Um if by geometric you mean slush's algorithm it is hoppable.

[Aexoden]

Um if by geometric you mean slush's algorithm it is hoppable.

The Geometric method refers to Meni's scoring system.

[Sukrim]

*SMPPS is broken/high risk of not returning 100% of expected value per share as soon as the pool hits negative, which should happen ~50% of the time.

PPLNS can choose arbitrarily large Ns (the highest useful N is 5 000 000 000, awarding each share with 1 Satoshi when blocks are found) if people really feel left out or are not able to get that they have a chance to get paid multiple times, equalizing the chance of not getting paid at all.
All that is needed is a proper interface, that shows percentages instead of "expected payout", as PPLNS looks forward to define rewards, not so much backwards as prop.


That said, as soon as pool operators start delaying LongPolls they really start to hurt their own profit + their own miners. Just think of the fact, that every millisecond in delay lets the miners compute 100 000 hashes per 100 MH/s in vain and will create more stale shares.
Also, if you just have to submit a valid solution every 10 minutes or so to count as an "active miner", one might be able to just pull a getwork from there every 10 minutes, solve + submit it, to not get throttled. The shorter this window, the more miners the operators hit + screw over, the longer, the easier it gets to fake a CPU miner.

[bitcoindaddy]

I've pushed a fix that may solve your problem (due to using Python 2.6). I use Python3 for my personal projects almost exclusively, and only use python 2.7.x when I'm forced to use Python2. If the fix doesn't work, let me know, and I'll try to get a 2.6.x testing environment.

Yes, it's working now, thanks

[Clipse]

This is getting interesting, may the most efficient hopper win

[hawks5999]

So when does hopping get its own section under mining?

[bitcoindaddy]

Can you explain what the different fields mean on the last line of the display?
Am I only getting an effective rate of 294 MHash/sec?

Code:

./poclbm.py -d 3  -v -f 1 -w 128
31/07/2011 18:05:09, Switching to bitcoins.lc with utility 1.057                                    
31/07/2011 18:05:09, Current utilities:                                                             
31/07/2011 18:05:09,   bitcoins.lc    : 1.057                                                       
31/07/2011 18:05:09,   arsbitcoin     : 1.000                                                       
31/07/2011 18:05:09,   bitpit         : 1.000                                                       
31/07/2011 18:05:09,   eligius        : 1.000                                                       
31/07/2011 18:05:09,   triplemining   : 0.721                                                       
31/07/2011 18:05:09,   mtred          : 0.705                                                       
31/07/2011 18:05:09,   btcguild       : 0.630                                                       
31/07/2011 18:05:09,   ozco.in        : 0.390                                                       
31/07/2011 18:05:09,   nofeemining    : 0.251                                                       
31/07/2011 18:05:09,   rfcpool        : 0.248                                                       
31/07/2011 18:05:09, Setting server (xxxxx @ bitcoins.lc)                                          
bitcoins.lc 31/07/2011 18:05:11, LP connected to bitcoins.lc                                        
bitcoins.lc [426.863 MH/s (~294 MH/s Eff:0.10)] [Rej: 0/10 (0.00%)] [GW: 104 (Eff:0.10)]