bitHopper: Python Pool Hopper Proxy

[enmaku]

As far as fail over goes, try using the new poclbm. It haa built in failover to whatever your back up pool its at whatever -f you have for your main pool. Even if poclbm isn't as fast for you as whatever you're using, it might be worthwhile until bitHopper is working properly for you.

Don't really feel like changing all of my miners and re-building all my scripts around a new executable. Might be minor for those of you who have two or three nodes but I'm playing with 14  
Edit: For the record, since I'm pretty sure I haven't stated before, I'm running puddinpop's opencl miner under winxp

[GoMaD]

nice update thx.

But wish i had more options to pass like: --port --hideRPCrequests --showServerShares..

And it would be great for statistic if you could count each valid share for each server and save it in a stats.blub file or something like that. 

From time to time i get:

Code:

2011-07-13 09:20:14+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:20:14+0200 [-] User timeout caused connection failure.
2011-07-13 09:20:18+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:20:18+0200 [-] User timeout caused connection failure.
...
2011-07-13 09:23:07+0200 [-] User timeout caused connection failure.
2011-07-13 09:23:15+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:23:15+0200 [-] User timeout caused connection failure.

In that time my workers feel bored and didn't work.

[nob]

nice update thx.

But wish i had more options to pass like: --port --hideRPCrequests --showServerShares..

And it would be great for statistic if you could count each valid share for each server and save it in a stats.blub file or something like that.  

From time to time i get:

Code:

2011-07-13 09:20:14+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:20:14+0200 [-] User timeout caused connection failure.
2011-07-13 09:20:18+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:20:18+0200 [-] User timeout caused connection failure.
...
2011-07-13 09:23:07+0200 [-] User timeout caused connection failure.
2011-07-13 09:23:15+0200 [-] Caught, jsonrpc_call insides
2011-07-13 09:23:15+0200 [-] User timeout caused connection failure.

In that time my workers feel bored and didn't work.

Same here.
I'm on mtred now, and everythings fine.
But on bitclockers before, there where a lot of these errors. And about 20% stales.
Maybe some kind of DDOS-protection from Bitclockers?

phoenix wrote this in the console:

Code:

2011-07-13 09:22:38: Traceback (most recent call last):
2011-07-13 09:22:38: File "phoenix.py", line 125, in <module>
2011-07-13 09:22:38: File "twisted\internet\base.pyo", line 1166, in run
2011-07-13 09:22:38: File "twisted\internet\base.pyo", line 1175, in mainLoop
2011-07-13 09:22:38:  --- <exception caught here> ---
2011-07-13 09:22:38: File "twisted\internet\base.pyo", line 779, in runUntilCurrent
2011-07-13 09:22:38: File "minerutil\RPCProtocol.pyo", line 130, in callback
2011-07-13 09:22:38: File "minerutil\RPCProtocol.pyo", line 352, in handleWork
2011-07-13 09:22:38: exceptions.TypeError: string indices must be integers

DiabloMiner was idle after the rp_call connection lost erros:

Code:

java.lang.NullPointerException

Only m0mchilds poclbm seems to run smooth.

//EDIT:
Oops you uploaded a new Version, i'll test it


//EDIT2:
Thanks, works great now.

[paraipan]

mt.red stats failed me couple a minutes ago

Code:

Time: UTC+2

[12:09:06] mining.mainframe.nl :3622349
[12:09:06] Server change to eligius, telling client with LP
[12:09:06] mineco :66088
[12:09:06] bitcoin.lc :480294
[12:09:06] Server change to bclc, telling client with LP
[12:09:06] mtred :754610
[12:09:06] pool.bitp.nl :1569718
[12:09:06] btcguild :57456
[12:09:06] Server change to btcg, telling client with LP
[12:09:06] bitclockers :611500
[12:09:07] eclipsemc :2549766
[12:09:16] RPC request [] submitted to BTC Guild
[12:09:17] LP Call us.btcguild.com:8332/LP
[12:09:37] RPC request [] submitted to BTC Guild
[12:09:57] RPC request [] submitted to BTC Guild
[12:09:59] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000ad6cbce0f8afebe71d19f71464eb622a01d17c41cd014c16256f5b84bd869a4c4e1d6e8b1a0abbcf6b779c16000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:10:08] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000ad6cbce0f8afebe71d19f71464eb622a01d17c41cd014c16256f5b84bd869a4c4e1d6e8b1a0abbcf3dedf17f000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:10:18] RPC request [] submitted to BTC Guild
[12:10:38] RPC request [] submitted to BTC Guild
[12:10:59] RPC request [] submitted to BTC Guild
[12:11:02] mining.mainframe.nl :3622490
[12:11:03] mineco :75772
[12:11:03] bitcoin.lc :567786
[12:11:03] bitclockers :616410
[12:11:03] mtred :0
[12:11:03] pool.bitp.nl :1571346
[12:11:03] btcguild :116551
[12:11:04] eclipsemc :2551778
[12:11:10] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000e68dc07f3973e7d42b69c79942a63eb7de1b5e4577b53412795421af314056204e1d6ec91a0abbcff6130688000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:11:14] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000e68dc07f3973e7d42b69c79942a63eb7de1b5e4577b53412795421af314056204e1d6ec91a0abbcf9e724bbb000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:11:19] RPC request [] submitted to BTC Guild
[12:11:40] RPC request [] submitted to BTC Guild
[12:12:00] RPC request [] submitted to BTC Guild
[12:12:15] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c900000150000000003ff9095133cdd7ce5a5db8c9836cd9cc1dd2e9abde9a4a1c32e66c9890a3aab24e1d6f061a0abbcf8251a8b4000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:12:21] RPC request [] submitted to BTC Guild
[12:12:33] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000a166b70401e554527ff71b723a75f693e3a8798c6a38f140968f5478b4b4159c4e1d6f1b1a0abbcf36bbc88e000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:12:37] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000a166b70401e554527ff71b723a75f693e3a8798c6a38f140968f5478b4b4159c4e1d6f1b1a0abbcf8faca9c0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:12:39] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000a166b70401e554527ff71b723a75f693e3a8798c6a38f140968f5478b4b4159c4e1d6f1b1a0abbcfc0a47fe2000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to BTC Guild
[12:12:41] RPC request [] submitted to BTC Guild
[12:13:00] mining.mainframe.nl :3622490
[12:13:00] mineco :75772
[12:13:00] bitcoin.lc :567786
[12:13:00] bitclockers :621155
[12:13:00] mtred :0
[12:13:00] pool.bitp.nl :1572960
[12:13:00] btcguild :176268
[12:13:00] Server change to mtred, telling client with LP
[12:13:01] eclipsemc :2553709
[12:13:02] RPC request [] submitted to mtred
[12:13:02] LP Call mtred.com:8337/LP
[12:13:17] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c900000150000000008e9838f1e4fd79ceecf50420e84a588ecb9213a20131f2820eb1afc84ee9be7c4e1d6f441a0abbcf0d19aeb8000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:13:22] RPC request [] submitted to mtred
[12:13:43] RPC request [] submitted to mtred
[12:13:49] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000a64dfa76d8141ce09c2fc668f42a267db3f249c8a46a21fa81d2c8f26250c70a4e1d6f6d1a0abbcf52c9eb4b000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:13:52] RPC request [u'000000014610341ceb92ab33cbe0e555229c8e5999d99021bbc1f0c90000015000000000a64dfa76d8141ce09c2fc668f42a267db3f249c8a46a21fa81d2c8f26250c70a4e1d6f6d1a0abbcf3a83066a000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:14:03] RPC request [] submitted to mtred
[12:14:17] LP triggered from server mtred
[12:14:17] LP triggering clients manually
[12:14:17] mining.mainframe.nl :3622587
[12:14:18] mineco :75772
[12:14:18] bitcoin.lc :567786
[12:14:18] mtred :0
[12:14:18] pool.bitp.nl :1573711
[12:14:18] bitclockers :623600
[12:14:18] btcguild :235357
[12:14:19] LP triggered from server mtred
[12:14:19] LP triggering clients manually
[12:14:19] mining.mainframe.nl :3622587
[12:14:19] mineco :75772
[12:14:20] mtred :0
[12:14:20] pool.bitp.nl :1573711
[12:14:20] bitclockers :623600
[12:14:20] btcguild :235357
[12:14:22] bitcoin.lc :567786
[12:14:24] RPC request [] submitted to mtred
[12:14:24] LP Call mtred.com:8337/LP
[12:14:24] RPC request [] submitted to mtred
[12:14:25] RPC request [] submitted to mtred
[12:14:25] eclipsemc :2555007
[12:14:25] RPC request [] submitted to mtred
[12:14:25] RPC request [] submitted to mtred
[12:14:26] RPC request [] submitted to mtred
[12:14:28] eclipsemc :2555044
[12:14:45] RPC request [] submitted to mtred
[12:14:50] RPC request [u'0000000117fd24eaa050282730a84371a5eb57bdaeb390cfe5a0b11000000a340000000014572add3e2f6d6397f9de08cbd0f3e36738d1551c7a47f73181288f9ce459594e1d6ffa1a0abbcfa0041633000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:14:57] mining.mainframe.nl :3622587
[12:14:57] mineco :75772
[12:14:57] bitcoin.lc :567786
[12:14:57] mtred :0
[12:14:57] pool.bitp.nl :1574462
[12:14:57] bitclockers :625442
[12:14:58] btcguild :235357
[12:14:58] eclipsemc :2555547
[12:15:06] RPC request [] submitted to mtred
[12:15:27] RPC request [] submitted to mtred
[12:15:29] RPC request [u'0000000117fd24eaa050282730a84371a5eb57bdaeb390cfe5a0b11000000a34000000008d36a7ea7eab4dd2d6f93e7c3044846ec2d95def6ae21d2c11f07bdc79343c8f4e1d6ffb1a0abbcf3e79a311000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:15:39] RPC request [u'0000000117fd24eaa050282730a84371a5eb57bdaeb390cfe5a0b11000000a34000000008d36a7ea7eab4dd2d6f93e7c3044846ec2d95def6ae21d2c11f07bdc79343c8f4e1d6ffb1a0abbcf693abb94000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred
[12:15:44] RPC request [u'0000000117fd24eaa050282730a84371a5eb57bdaeb390cfe5a0b11000000a34000000008d36a7ea7eab4dd2d6f93e7c3044846ec2d95def6ae21d2c11f07bdc79343c8f4e1d6ffb1a0abbcf66ea9ece000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'] submitted to mtred

restarted and everything goes fine

[paraipan]

just downloaded the last version and have a small request: please do not hide the pool share count from time to time, please. It helps allot if everything keeps transparent and for debugging purposes too. Thanks

[anty]

Just a little feedback:
The latest version seems pretty stable. It didn't crash for me in the last 4 hours.

[nob]

Yes, the new Version runs great for about 6 Hours and 9 Workers connected.

No Errors, fast pool switching and nearly 0 stales.

[Sukrim]

I started writing a google spreadsheet to have a few stats form the pools. The biggest problem currently is that the pools don't show which shares were mined with which difficulty --> you usually get a "total_shares" or something like that, most of the time also a total payout or similar but you can't calculate efficiency over several rounds (as you don't know how many shares were mined with which difficulty).
For this you would need a "canonical" metapool, that does not really give out work or accept it but just takes an arbitrary number of shares for each round and outputs the expected 0% PPS payout after the block was confirmed (to make sure also transaction fees are accounted for - important in the long run). This data should then be queriable aggregated via APIs like in most other pools. From this you could then determine if your pool cheats you or not (ideally for non-hoppers it should show about the same payouts on the metapool and the real pool, if it's a 0% fee pool).

Also some pools don't have a (useful) API at all, you'd need to login and scrape their website...

All in all, I fear that only client side stats from bithopper will be useful (but might not reflect the reality 100%, as in the end only what reaches your wallet counts!). The ones from pools might be interesting for basic statistics, but you can't even see how lucky/unlucky you were each round (something that I'm interested in). They could be used to verify the local stats though and/or to correct them.


Edit:
It seems to me that bitHopper hops to early! According to https://forum.bitcoin.org/index.php?topic=3165.0 line 169 should read min_shares = difficulty*.435, not min_shares = difficulty*.40! We still loose some "hot" share this way in the worst case.

[zybron]

Hey, c00w, what's the chance we could have the code a bit refactored so each pool has an include or something similar? Something along the lines that the user/pass, server definition, shares definition, etc. are in a single location? The reason I ask is that it would then be very easy to 'plug in' another pool option just by adding a new .py file for that pool.

I think that would make it pretty easy for some of us that just play around with python to add a custom pool into the rotation.

[organofcorti]

Edit:
It seems to me that bitHopper hops to early! According to https://forum.bitcoin.org/index.php?topic=3165.0 line 169 should read min_shares = difficulty*.435, not min_shares = difficulty*.40! We still loose some "hot" share this way in the worst case.

the min_shares = difficulty*.435 comes from a special case of hopping from a 'contributed' model to a 'connected' model. It was a proof-of-concept, and I don't know how applicable the exact figure is to hopping multiple proportional pools. I was trying to figure it out and then thought it might just be easier to run a simulation and vary the percentages to see what the best outcomes were.

Unfortunately I can't find a good poisson distribution random number generator for non-integers (ie fractions of difficulty). Any ideas?

[roos]

[17:05:01] Error in pool api for eclipsemc
[17:05:01] [Failure instance: Traceback: <type 'exceptions.ValueError'>: No JSON object could be decoded
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1076:gotResult
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1063:_inlineCallbacks
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks
pool.py:309:selectsharesResponse
pool.py:250:eclipsemc_sharesResponse
/usr/lib/python2.7/json/__init__.py:326:loads
/usr/lib/python2.7/json/decoder.py:360:decode
/usr/lib/python2.7/json/decoder.py:378:raw_decode
]

[organofcorti]

All in all, I fear that only client side stats from bithopper will be useful (but might not reflect the reality 100%, as in the end only what reaches your wallet counts!). The ones from pools might be interesting for basic statistics, but you can't even see how lucky/unlucky you were each round (something that I'm interested in). They could be used to verify the local stats though and/or to correct them.

A whole heap of Multipool is devoted to this. I couldn't do it, but Nick from Multiclone seems like a nice guy, and he's on the case and might be able to give a hand adapting it by explaining the bits to do with html scraping.

[koopa]

[17:05:01] Error in pool api for eclipsemc
[17:05:01] [Failure instance: Traceback: <type 'exceptions.ValueError'>: No JSON object could be decoded
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1076:gotResult
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1063:_inlineCallbacks
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks
pool.py:309:selectsharesResponse
pool.py:250:eclipsemc_sharesResponse
/usr/lib/python2.7/json/__init__.py:326:loads
/usr/lib/python2.7/json/decoder.py:360:decode
/usr/lib/python2.7/json/decoder.py:378:raw_decode
]

I think this is caused by the fact that eclipse mining website https://eclipsemc.com/ is not currently working 100%, so no API data is currently available from them. Bithopper sees this and is reporting it as an API error; expected behavior I would say.

[roos]

ah, ofcourse.

[neptop]

Instead of or in addition to --disbale what about just evaluating the existence of username and/or password. If it doesn't exist ignore it. 

[Sukrim]

I think that would make it pretty easy for some of us that just play around with python to add a custom pool into the rotation.

It's the same difficulty as now, only that you have to write the function(s) in pool.py, not in a seperate file.

There's not much to do anyways: A sharesResponse function, a entry in the dictionary and the user/pass info.

I am writing one for triplemining right now, the only problem is that I need to load their stats-page and manually scrape it (with beautiful soup it would be easier but that adds another library - I'll try to do it with some regex kung-fu) instead of having it in json. That can all go in the sharesResponse though, so no big deal.

[zybron]

I actually have a sharesResponse definition for triplemining. I used regex and I'm sure it would break if the layout changed at all, but it's functional. I'll post the code when I get home.

The main point for refactoring the code is that changes in the main pool.py routines wouldn't require merging those changes with my own changes. Maybe I'll see if I can do some of that myself and do a proper fork.

[Sukrim]

I guess I am faster with my code!

in pools.py:

Code:

@@ -9,6 +9,7 @@ import sys
 import exceptions
 import optparse
 import time
+import re
 
 from twisted.web import server, resource
 from client import Agent
@@ -68,7 +69,11 @@ servers = {
         'bitp':{'shares': default_shares, 'name': 'bitp.it',
            'mine_address': 'pool.bitp.it:8334', 'user': bitp_user,
            'pass': bitp_pass, 'lag': False, 'LP': None,
-            'api_address':'https://pool.bitp.it/api/pool'}
+            'api_address':'https://pool.bitp.it/api/pool'},
+        'triplemining':{'shares': default_shares, 'name': 'triplemining.com',
+           'mine_address': 'eu.triplemining.com:8344', 'user': triplemining_user,
+           'pass': triplemining_pass, 'lag': False, 'LP': None,
+            'api_address':'https://www.triplemining.com/stats'}
         }
 
 current_server = 'btcg'
@@ -289,6 +294,14 @@ def bitclockers_sharesResponse(response):
     servers['bitclockers']['shares'] = round_shares
     log_dbg( 'bitclockers :' + str(round_shares))
 
+def triplemining_sharesResponse(response):
+    global servers
+    statpage = response
+    shares = re.search(r"<td>[0-9]*</td>", statpage).group(0)[4:-5]
+    round_shares = int(shares)
+    servers['triplemining']['shares'] = round_shares
+    log_dbg( 'triplemining :' + str(round_shares))
+
 def errsharesResponse(error, args):
     log_msg('Error in pool api for ' + str(args))
     log_msg(str(error))
@@ -305,7 +318,8 @@ def selectsharesResponse(response, args):
         'btcg':btcguild_sharesResponse,
         'eclipsemc':eclipsemc_sharesResponse,
         'miningmainframe':mmf_sharesResponse,
-        'bitp':bitp_sharesResponse}
+        'bitp':bitp_sharesResponse,
+        'triplemining':triplemining_sharesResponse}
     func_map[args](response)
     server_update()

and you need a triplemining_user and a triplemining_pass of course in the passwords file.

Th regex currently just takes the first number in a <td> tag which luckily is the current shares number. A bit more safe would be to check for the "MINING" line and then take the next number or so... however: It works for the time being!

[c00w]

1) Arsbitcoin as the second backup?
Sweet. I'll add it.

EDIT: The website is broken and I can't login. So I haven't added it. Will try later on a windows machine.

2) Refactor code?
Yeah. I want to do it. I was going to move everything thats not explicitly pool related out of pool.py and into bithopper.py and lp.py

3) Pool share count?
I can reenable it.

4) dumping the stats to a file?
Yeah I was going to try and make an sqlite database and then well LP errors and crazy cycling.

EDIT: I added it as --statsdump

5) Triplemining?
Well it seems like a little bit of a ponzi scheme. I have no issues hopping them however.

[zybron]

I guess I am faster with my code!

Yep, but good to see we used the same methods.