bitHopper: Python Pool Hopper Proxy

[TeaRex]

It might be interesting to reduce this to just the website scraping part of it. Then run in combination with Chris Howie's AMP based Proxy and just let it change the priorities in the MySQL database instead of doing its own proxying, simply reordering them in the order of number of shares (just set priority in the DB to negative number of shares or sth like that). That should relieve you of re-inventing complicated (and proven to work) proxying code.

I *might* make an attempt at that later today or tomorrow. Note I said *might*. No promises whatsoever.

As for license I strongly suggest the AGPL (GNU Affero General Public License). It has an additional rule that if you host it on a server that others can access, you also have to give them any modifications to the source that you made, even if they themselves don't run it. Makes lots of sense for server apps (or potential server apps, as this thing is).

[c00w]

Hi,

I wrote this program so that each client could run their own pool hopper proxy. If you want a hopping pool multipool and multiclone already fill that gap.

However I am going to implement basic statistics and it should be pretty simple to add in the ability to record the user name and password people logged in with and store that with their shares.

So this should work pretty well as a pool. You are going to need to write your own custom payment functions however.

The license will be changed to something more pool friendly when this is in a decent state.

-c00w

[enmaku]

Does it only bind to localhost or is there anything stopping me from running this on one system and pointing all of my rigs at it?

(read: I don't want to deal with the dependencies eight times)

[organofcorti]

I'm running this on one machine, 3 clients pointed at it. Seems to run well, only sometimes it will go for the one from the list of pools with the second lowest shares, or it wont change over. Eg last night it chose btcguild over eclipse (btc guild on over 300000, eclipse at 20000), but after switching off and restarting it changed to eclipse. This morning it stayed with eclipse at 280000 when it should have moved to mineco.in at 240000.

Otherwise stable enough for last 12 hours - no crashes.

[c00w]

Hi,

1) Does it only bind to localhost?
It uses the default twisted http server and should bind to all interfaces. So you can run it on one computer and have multiple workers, on other computers, pointed to it. This assumes your firewall doesn't get in the way.

2) Why doesn't it always hop to the lowest pool.
I currently have it set to only change pools if the shares of the current pool is over 40% of difficulty. However it would make sense to switch anytime a new block comes out. I'll add that once I get server side LP working. Its a pretty simple switch if I have server side LP.

The reason it only switches pools at 40% is I didn't want two pools with close amounts but desynced json api update rates to cause the proxy to hop back and forth between them wasting time and shares.

-c00w

[organofcorti]

2) Why doesn't it always hop to the lowest pool.
I currently have it set to only change pools if the shares of the current pool is over 40% of difficulty. However it would make sense to switch anytime a new block comes out. I'll add that once I get server side LP working. Its a pretty simple switch if I have server side LP.

The reason it only switches pools at 40% is I didn't want two pools with close amounts but desynced json api update rates to cause the proxy to hop back and forth between them wasting time and shares.

Ah well, expected behaviour then! You could make it so that it only switches when the pool with lesser shares has say less than 50% of the pool you're on to avoid the jumping back and forth.

The earlier you come in on a round, the greater the value of a share on a prop pool. So you should really be hopping to whichever pool has started on a new block.

I currently have bit hopper running with 6 pools and it never has to go to eligius, so I'd say bithopper's working well so far. No set up issues like I had (still having) with multipool.

[c00w]

Ah well, expected behaviour then! You could make it so that it only switches when the pool with lesser shares has say less than 50% of the pool you're on to avoid the jumping back and forth.

The earlier you come in on a round, the greater the value of a share on a prop pool. So you should really be hopping to whichever pool has started on a new block.

Well I almost have Long polling support enabled on the server side so it'll switch everytime a block is found.

Your idea of switching if its under 50% of the current pool sounds like a smart idea as well. I'll add that in.

Thanks
-C00w

[organofcorti]

Well I almost have Long polling support enabled on the server side so it'll switch everytime a block is found.

looking forward to LP support, but how will you know where to switch from LP? Won't it just tell you that a block was found somewhere in the network? I suppose you could use that info to determine when bithopper checks to see which pool has the least shares to see if one of them was the one that found the block?  Or maybe I'm just misunderstanding LP.

Again, thanks for such a simple and easy to use/modify proxy.

[c00w]

1) How/why does LP switching help?
Well LP means a server found a block. And it may be one of ours. So we run a server switch to the server with the lowest share-count because it may be a new round and either way all the current get-works are invalid so the overhead is minor.

[burp]

Does the current version work?

[2011-07-09 11:00:46] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:46] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:47] LONGPOLL detected new block, flushing work queue
[2011-07-09 11:00:48] LONGPOLL detected new block, flushing work queue

It doesn't seem to choose a pool.

[ahitman]

This thing is great! I would like to be able to see better what its doing, is there any way to hide the Starting factory and Stopping factory messages?

[c00w]

1) Stopping and starting messsages are annoying. Help?
Currently I started using the twisted logging system because it made finding bugs easier with the timestamps. The twisted logging system has no simple way that I can find to stop those messages from appearing. It hasn't gotten a lot of love.  I'm planning on adding an actual debug option and using print otherwise for purely informational messages.

If you want to write me a subclass of httpclient with those messages turned off though I'll happily accept it.

2) Server side Long Polling is working?
It does now. I had an issue with longpoll handing because I inverted a logic statement. Its pushed to the repo in the correct form and running on my servers.

3)What happened to Burps machine?
From the log it looks like the server started cycling endlessly because there was no internet connection? I'm guessing my changes making the server swap if another server has less than 25% of its shares is at fault since if there are multiple 0 share connections it would try each, lag them out, issue a LP to the client because it swapped server, try the next and rinse and repeat through the whole list. Which certainly looks like what happened. I'll redo that code to instead first check that the share > 10% of difficulty.

[burp]

Doesn't make a difference, here is the complete logfile: http://pastebin.com/1iY1pS3M

PS: It's no internet connectivity problem

[organofcorti]

Just using new bithopper - how can you tell which pool you're connected to?

[c00w]

1) How can you tell what pool you are connected to?
It should say rpc request [] from blah.blah.
That blah.blah is the server it is pulling from.

2) Burf:
Its going to sound horrible but can you run git pull again? I just rewrote the client LP system to have a lot less bugs. It appears that miners implement LP in all sorts of ways because its actually not clearly specified anywhere.

Also what miner are you using? That might have a disturbingly significant effect. The logs look like nothing is wrong.

[organofcorti]

1) How can you tell what pool you are connected to?
It should say rpc request [] from blah.blah.
That blah.blah is the server it is pulling from.

Sorry, I can only see an RPC request to mtred which hasn't yet recorded any shares from me. SHould I just dl and try the lastest?

Code:

011-07-09 10:47:37+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:36 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] LP triggered <twisted.web._newclient.Response object at 0x1ae6490>
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:38+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mineco.in/stats.json>
2011-07-09 10:47:38+0000 [HTTPChannel,1,127.0.0.1] RPC request [] From mtred
2011-07-09 10:47:38+0000 [HTTPChannel,6,127.0.0.1] RPC request [] From mtred
2011-07-09 10:47:38+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:38+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP Call mtred.com:8337/LP
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:38 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP Call mtred.com:8337/LP
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] 127.0.0.1 - - [09/Jul/2011:10:47:38 +0000] "POST / HTTP/1.1" 200 608 "-" "poclbm/2011.beta4"
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] mineco :1415190
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://mineco.in/stats.json>
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] bitclockers :1087070
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP triggered <twisted.web._newclient.Response object at 0x1c2a810>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mineco.in/stats.json>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] LP triggered <twisted.web._newclient.Response object at 0x196af90>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:39+0000 [HTTP11ClientProtocol,client] Starting factory <HTTPClientFactory: https://mineco.in/stats.json>
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:39+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] mtred :0
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://mtred.com/api/user/key/d91c52cfe1609f161f28a1268a2915b8>
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] bitclockers :1087134
2011-07-09 10:47:40+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitclockers :1087143
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://bitclockers.com/api>
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] bitcoin.lc :1485982
2011-07-09 10:47:41+0000 [HTTPPageGetter,client] Stopping factory <HTTPClientFactory: https://www.bitcoins.lc/stats.json>
2011-07-09 10:47:41+0000 [HTTPChannel,1,127.0.0.1] RPC request [u'00000001cce6f8a2cf82f32896d9cd75aaa1e86d626c29045094b6bd0000097e00000000ab84c24b1c41ebde8fef201eb19ba0b460acc681849e6d21d21e2a47892e05eb4e1831c81a0abbcfb83a58d6000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000'] From mtred

[burp]

2) Burf:
Its going to sound horrible but can you run git pull again? I just rewrote the client LP system to have a lot less bugs. It appears that miners implement LP in all sorts of ways because its actually not clearly specified anywhere.

Also what miner are you using? That might have a disturbingly significant effect. The logs look like nothing is wrong.

Nope, no effect. I'm running cgminer (http://forum.bitcoin.org/index.php?topic=21275.0).

[c00w]

Hmmm. I see the problem and It has something to do with cgminer managing to call bitHopper quickly enough to make it make two LP connections one of which gets returned. I'll figure out why my code to stop this doesn't work tommorow. For now use poclbm or phoenix.

[burp]

Are the pool stats updated after each new share now, or am I misinterpreting the log spam? At least when I run the client without long-polling, or disable the long-polling header at bitHopper.

[OCedHrt]

[7/9/11 6:11:07 AM] Added Barts (#1) (14 CU, local work size of 128)
Waiting...Exception in thread "DiabloMiner LongPollAsync" java.lang.NullPointerE
Waiting...
        at com.diablominer.DiabloMiner.DiabloMiner.forceUpdate(DiabloMiner.java:
Waiting...
        at com.diablominer.DiabloMiner.DiabloMiner$LongPollAsync.run(DiabloMiner
Waiting...)
Waiting... java.lang.Thread.run(Unknown Source)

I get this exception on the DiabloMiner side when connecting to BitHopper. But it still seems to run after that.

Also, it seems btcguild doesn't have a get shares function? The user api json is broken when there are no workers, but luckily the site-wide api is functioning. And, server_update() should not be called in each get response actually, and should be done at the end of update_servers because each server_update does a comparison against the rest of the server and calls select_best_server() - logging is also much quieter this way

Code:

def btcguild_sharesResponse(response):
    global servers
    info = json.loads(response)
    round_shares = int(info['round_shares'])
    servers['btcg']['shares'] = round_shares
    log.msg( 'btcguild :' + str(round_shares))

def btcguild_getshares():
    d = getPage('https://www.btcguild.com/pool_stats.php')
    d.addCallback(btcguild_sharesResponse)
    d.addErrback(log.err)

def update_servers():
    global servers
    bitclockers_getshares()
    mineco_getshares()
    btcguild_getshares()

    server_update()

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return