bitHopper: Python Pool Hopper Proxy

[c00w]

It currently has to have the payout entered manually. with a bitcoind interface it could be automatic but I haven't written it yet.

[burp]

Seems bitHopper loses more and more flexibility. With this config file even more is gone, that is the possibility to write more complicated functionality to get the share counts, which I relied on for some pools I added myself. I will stick to older versions.

[Clipse]

Wouldnt it be possible to allow the config for "less knowledable" users and leaving the pure python code for those who want to tinker more in depth?

[burp]

My changes:

I don't think regex api_index is necessary if you write proper regular expressions to match, as in my example.
I also added api_method cmd which runs a command in the shell where you can use arbitrary pipe, curl,grep,sed,awk magic.

api_method:cmd
api_cmd:curl -s -k https://pool.bitp.it/leaderboard | … more shell magic that returns a number

api_method:cmd
api_cmd:echo 1234

Code:

diff -rN -u old-bitHopper_darcs/pool.cfg.default new-bitHopper_darcs/pool.cfg.default
--- old-bitHopper_darcs/pool.cfg.default    2011-07-23 13:34:45.675261168 +0200
+++ new-bitHopper_darcs/pool.cfg.default    2011-07-23 13:34:45.679261217 +0200
@@ -51,7 +51,7 @@
 mine_address: eu1.triplemining.com:8344
 api_address:https://www.triplemining.com/stats
 api_method:re
-api_key:-->\d+</td>
+api_key:<td>.*?(\d+)</td>
 api_index:3,-5
 role:mine
 #CHANGE THIS
diff -rN -u old-bitHopper_darcs/pool.py new-bitHopper_darcs/pool.py
--- old-bitHopper_darcs/pool.py 2011-07-23 13:34:45.675261168 +0200
+++ new-bitHopper_darcs/pool.py 2011-07-23 13:34:45.679261217 +0200
@@ -107,15 +107,15 @@

         elif server['api_method'] == 're':
             output = re.search(server['api_key'],response)
-            s,e = server['api_index'].split(',')
-            output = output.group(0)
-            s = int(s)
-            if e == '0' or e =='':
-                output = output[s:]
-            else:
-                output = output[s:int(e)]
-            round_shares = int(output)
+            round_shares = int(output.group(1))
             self.UpdateShares(args,round_shares)
+
+        elif server['api_method'] == 'cmd':
+            import subprocess
+            round_shares = int(subprocess.Popen(server['api_cmd'], shell=True,
+                stdout=subprocess.PIPE).communicate()[0])
+            self.UpdateShares(args,round_shares)
+
         else:
             self.bitHopper.log_msg('Unrecognized api method: ' + str(server))

[flower1024]

very nice idea!
what about just placing a command line in pools.conf which gets executed every time round-shares are updated?

that way we could completly decouple bithopper from pools - and even change them without an restart.

shell command could return sth like "poolhashrate,roundshares,false|true" (false|true: is the api lagging)
which could easily be parsed by python

[organofcorti]

Anyone else have triplemining broken in the new bitHopper? Getting 'error in api'.

Also I love polmine, great service and good efficiency so far. But the new config broke the regex that ryouiki did. Anyone fix it? The old version (with what I assume is a commented out practice go) is:

Code:

            	'polmine':{'shares':default_shares, 'name':'polmine',
		    'mine_address':'polmine.pl:8347', 'user':polmine_user, 'pass':polmine_pass,
		    'lag':False, 'LP':None,'api_address':'http://polmine.pl/?action=statistics',
		    'role':'mine' },

    def polmine_sharesResponse(self, response):
        #info = re.search(r"stkich:</th><td> [0-9]*</td>", response).group(0)[4:-5]
        round_shares = int(re.search(r"stkich:</th><td>[ 0-9]*</td>", response).group(0)[16:-5].replace(' ',''))
        self.UpdateShares('polmine',round_shares)

worked fine, had some error at the start of bitHopper which I paid no attention to but worked.

So can anyone fix Triplemining and Polmine for me?

Cheers!

[frisco]

They changed the format again, new code for triple:

Code:

    def triple_sharesResponse(self, response):
        output = re.search(r"-->([0-9\s]+)</", response)
        if output != None:
            match = output.group(1)
            match = match.replace(' ','')
            round_shares = int(match)
            self.UpdateShares('triple',round_shares)

I don't think this can be done with the new config, so I am still at yesterday's version.

[Sukrim]

Other scoring methods
Many scoring systems are not statistically analyzed and hence still offer opportunity for hopping. For example, Pay-Per-Last-N-Shares is still quite hoppable (with typical N), as follows. Let's renormalize N by dividing by the difficulty, to give us the normalized last N shares, M. Using the same analysis as above, the speedup can be computed as shown below in Figure 3. PPLNS strongly deviates from proportional only during rounds which pay (per share) below the statistical average. Because the hopper "suffers" from PPLNS only during sub-par rounds, the absolute difference between proportional and PPLNS is small. One can calculated the expectation values of PPLNS shares (as a function of x and M), with the following formula (Wolfram link): exp(x) ( Gamma(0,x) - Gamma(0,M+x)) which is valid for x < M. A maximum speedup (at current difficulty) is still 13x with M=0.5. A quick'n'dirty way to implement PPNLS pools in hopping software is to penalize the PPNLS pool by multiplying round_shares by .43/.21. Slush's pool is also hoppable under the same mechanism, although the curves depend on hashing rate as well. Almost any proportional-based pool will be hoppable at x<.10


Figure 3. Comparison of expectation value from share submission from proportional and PPNLS (M=0.5, see text) payout schemes. Notice the intersection with y=1, where share expectation value is equal to solo mining. These intersections are respectively equal to ~0.21 and ~0.428 for PPNLS and proportional.

Conclusion
I hope this clears up some confusion about expectation values of shares and pool hopping. If anything is unclear feel free to reply or send me a PM.

Although this should REALLY go in a seperate topic... What's your understanding of PPLNS?
I got it like this, that you _always_ consider the last N shares and don't reset at block boundaries. Yes, this means you can get paid twice or more often for a single share but also that you can be not paid at all for some others. I suspect you do in your model reset your shares when a block was found. Is this true?

[paraipan]

nice work streblo, just managed to put my concepts straight thanks. I added you on my future donations list if you don't mind

@c00w kudos to you to man, I was waiting for this to happen coz my python skills really suck and many times I was just breaking the script after editing, it's a more portable now... thanks really , btw I already have you on donation list

[organofcorti]

I don't see how his model could include anything from previous rounds - I think it resets at boundaries. That just means the model gives a low bound for speedup/efficiency.

I'd really like some details (that i probably won't follow!) though. Or at least more details about

PPLNS strongly deviates from proportional only during rounds which pay (per share) below the statistical average. Because the hopper "suffers" from PPLNS only during sub-par rounds, the absolute difference between proportional and PPLNS is small.

because it's tantalising me. I'd like to know why the gamma functions allow us to model the PPLNS behaviour.

[bb]

With the newest version I am getting this error:

Code:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 1371, in dataReceived
    finishCallback(data[contentLength:])
  File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 1585, in _finishRequestBody
    self.allContentReceived()
  File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 1641, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.7/dist-packages/twisted/web/http.py", line 807, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 125, in process
    self.render(resrc)
  File "/usr/lib/python2.7/dist-packages/twisted/web/server.py", line 132, in render
    body = resrc.render(self)
  File "/usr/lib/python2.7/dist-packages/twisted/web/resource.py", line 210, in render
    return m(request)
  File "/opt/bitHopper-4/bitHopper.py", line 370, in render_POST
    return bitHopper_Post(request)
  File "/opt/bitHopper-4/bitHopper.py", line 227, in bitHopper_Post
    bithopper_global.log_msg('RPC request [' + rep + "] submitted to " + str(pool_server['name']))
exceptions.TypeError: 'NoneType' object is not subscriptable

When this happened, current in line 212 in bitHopper.py was 'mtred', however, pool_server in line 213 turned out to be None.

The mtred entry in the config file looks ok.

[streblo]

Very nice work, Streblo!

Thanks, this pool math is really interesting to me because the results are sometimes bizzare!

edit: Streblo!
I love the explanation. And I'm adding ppln.

Sounds good. Remember, the quick'n'dirty method with the factor of ~2 for shares adjustment only works with M=0.5. Though, I (or anybody with the mathematica link) can easily calculate numbers for arbitrary M.

Amazing streblo. Really great stuff.
The thing I'm still unclear on though (and maybe I'm just dense) is:
Based on your analysis, given the situation of hopping two pools having different speeds, should I stay with the faster pool even though it is further from share 1 and then hop to slower later. Or do I hop to pool 2 immediately after it solves a block. It seems to me the downside is getting stuck in a slow pool and missing the spread of shares to other available pools ( even backup)

Assuming you switch pools with no increase stales and you have accurate round_shares information, ALWAYS send shares to the pool with least number of shares (unless, of course, all pools are above x=.428, in which case you should solo mine [read: use a "backup" pool]).

nice work streblo, just managed to put my concepts straight thanks. I added you on my future donations list if you don't mind

Thanks man!

Although this should REALLY go in a seperate topic... What's your understanding of PPLNS?
I got it like this, that you _always_ consider the last N shares and don't reset at block boundaries. Yes, this means you can get paid twice or more often for a single share but also that you can be not paid at all for some others. I suspect you do in your model reset your shares when a block was found. Is this true?

Hmm, should I start another topic? I started a similar thread in Mining->Pools, but that is more for a new payout system which I call normalized Proportional Payout (nPP), which is almost identical to proportional, except it cannot be exploited by hoppers.

I don't see how his model could include anything from previous rounds - I think it resets at boundaries. That just means the model gives a low bound for speedup/efficiency.

I'd really like some details (that i probably won't follow!) though. Or at least more details about

PPLNS strongly deviates from proportional only during rounds which pay (per share) below the statistical average. Because the hopper "suffers" from PPLNS only during sub-par rounds, the absolute difference between proportional and PPLNS is small.

because it's tantalising me. I'd like to know why the gamma functions allow us to model the PPLNS behaviour.

You are correct, this analysis resets at the pools found block boundaries. I could run Monte Carlo simulations of the PPLNS system, because handling multiple payouts per share (technically, upto N times!) would be quite difficult, I suspect.
Yes, this analysis does include reseting shares after the pool finds a block (not when the system finds a block). Whoops. I don't included: multiple blocks found per share or the decreased payout from including the last N shares (including previous round(s)). These effects work in opposite directions, although I don't know which one is stronger.

PP expectation value was calculated by taking the payout (1/(n+x)) multiplied by the distribution of number of shares (x) required to find a block, and integrating over all possibilities.
n shares are already submitted
the payout after x more shares are submitted (1/(x+n))
probability of a pool solving a block after x shares = exp(-x)
PP = Integrate exp(-x)/(x+n) from x=0 to Infinity = exp(x) E1(x) = exp(x) Gamma(0,y) (this is the incomplete gamma function)

PPLNS (when incorrectly resetting shares after every block)
n shares are already submitted
the payout after x more shares are submitted: 1/(x+n) (x < M)
PPLNS = Integrate exp(-x)/(x+n) from x=0 to M = exp(x) ( Gamma(0,x) - Gamma(0,M+x)) (x < M)


Cheers

1FQoC3zsos22QxZC35fMfP8JteMK9nEQPU

[paraipan]

I'm trying to add polmine.pl to "pool.cfg" in last version (0dd8833) without success, I think the problem comes from the spaces in page source, just guessing

[polmine]
name: polmine.pl
mine_address: polmine.pl:8347
api_address:https://polmine.pl/?action=statistics
api_method:re
api_key:stkich:</th><td>\d+</td>
api_index:16,-5
role:mine
#CHANGE THIS
#https://polmine.pl/?action=signup
user: x
pass: x

any help would be appreciated



edit: minor changes

[enmaku]

Getting regex failures on triple with Ryouiki's latest. Did they change their stats page or something?

[Aexoden]

The expected value of a share in a (properly implemented) PPLNS pool is exactly 1.0 (relative to solo or PPS).

Once a share is submitted, the fundamental question of its value is: how many blocks will be found before N more shares have been submitted. This can be modeled as a Poisson process with the expected number of blocks equal to N / difficulty. (If N is the difficulty, you'd expect to find one block. If N is twice the difficulty, you'd expect to find two blocks, and so on.) If (for example), N is one-half the difficulty, you'd expect a share to be paid approximately as follows:

• No payout: ~60.65%
• 1 payout: ~30.33%
• 2 payouts: ~7.58%
• 3 payouts: ~1.26%
• 4 or more payouts: ~0.18%

Regardless of the value of N or the difficulty, the expected value will sum out to 1, however.

To make it relevant for this thread, PPLNS pools should be used only as backups, and definitely not as hopping targets.

[frisco]

enmaku read my last post

[enmaku]

enmaku read my last post

Yeah, I see that now. I already fixed it myself, but in a different way - I added one more replace() to pools.py line 241:

Code:

parsed = response.replace('\n', '').replace('\r','').replace('\t','').replace(' ','')

Took care of the extra spaces they added in the middle of the number as well as the space between the </td> and <td>

Glancing over yours I can kind of see how it's working, regex isn't my strong suit so if I can fix it ANYWHERE but the regex I'll gladly do so 

[c00w]

1)Pool.cfg is less flexible?
Well you can still add methods if you need some custom python code.
see json_ec. So no. Its actually more flexible.

2)bb nothing is happening?
You need to mv pools.cfg.default to pools.cfg
I should probably have a error output telling people that.

3)Cmd api_method?
Awesome. Can I get a copyright release burp?

4)Different triple mining code?
I'm going to test that but that works? Really? What I have now for re works also.

[EskimoBob]

Can you add http://mining.bitcoin.cz/ to the list?

[streblo]

The expected value of a share in a (properly implemented) PPLNS pool is exactly 1.0 (relative to solo or PPS).

Once a share is submitted, the fundamental question of its value is: how many blocks will be found before N more shares have been submitted. This can be modeled as a Poisson process with the expected number of blocks equal to N / difficulty. (If N is the difficulty, you'd expect to find one block. If N is twice the difficulty, you'd expect to find two blocks, and so on.) If (for example), N is one-half the difficulty, you'd expect a share to be paid approximately as follows:

• No payout: ~60.65%
• 1 payout: ~30.33%
• 2 payouts: ~7.58%
• 3 payouts: ~1.26%
• 4 or more payouts: ~0.18%

Regardless of the value of N or the difficulty, the expected value will sum out to 1, however.

To make it relevant for this thread, PPLNS pools should be used only as backups, and definitely not as hopping targets.

You're completely right, PPLNS isn't hoppable. (Although PPLNS is subject to a share-withholding attack)