bitHopper: Python Pool Hopper Proxy

[ahitman]

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833  its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools?  Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

Well I almost have Long polling support enabled on the server side so it'll switch everytime a block is found.

looking forward to LP support, but how will you know where to switch from LP? Won't it just tell you that a block was found somewhere in the network? I suppose you could use that info to determine when bithopper checks to see which pool has the least shares to see if one of them was the one that found the block?  Or maybe I'm just misunderstanding LP.

Again, thanks for such a simple and easy to use/modify proxy.

I think you understand correctly, and this is a good idea in theory, but if you are using the api of the pool to figure out which pool it is and it takes a while for some pools to update the stats, then your probably best off staying in the current pool until bithopper picks up a low block count from the winning pool through its regular checks, and switches anyways.

How often does bithopper check each pools apis?

[ahitman]

Here is some logic that popped in my head while looking through the code (I've never programmed in python but I have done some programming at school). Please let me know if this makes sense:

1. Check the api of each pool for share_count
2. If share_count[of the pool you are checking] < 1 then share_count[of the pool you are checking] = 10**10
3. Pick the pool with the lowest share count

Step 2 will catch any misbehaving api or a pool that is down and is returning a 0 for the share count.

Let me know if the code is doing this or something like this already, I have a hard time following some of the if statements, but you've inspired me to go and learn some python syntax!

[ahitman]

Sorry to take spam this thread with so many posts but I found this useful and maybe someone else will as well.

Just using new bithopper - how can you tell which pool you're connected to?

I added a line to my pool.py file that will display the chosen server name with **** around it so one can pick it out of the torrent of information flowwing through the log file. Please let me know if there is a better spot for this as Im not good at python (yet) so I just stuck it in to see what happens.

Code:

    for server in servers:
        info = servers[server]
        if info['shares']< min_shares and info['lag'] == False:
            log.msg( '***min_shares :' + str(min_shares))
            min_shares = servers[server]['shares']
            server_name = server

    if server_name == None:
        server_name = 'eligious'

    """This line added to show current server in a more visible way"""
    log.msg( '******New server **** :' + str(server_name))

    global new_server
    global lp_set

[OCedHrt]

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833  its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools?  Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

No I don't think so. As written it wouldn't do that. Rather it actually prevents a switch until the lowest share server has at least 50% of the current server, not the difficulty. The other problem is that it's doing an update_server every share submit, not just on each LP.

[ahitman]

And, what's the reasoning for waiting until the server with the least shares to have more than half of the current server before switching?

Code:

        if min_shares < servers[current_server]['shares']:
            if servers[current_server]['shares'] - min_shares < .50 * servers[current_server]['shares']:
                select_best_server()
                return

Check out http://forum.bitcoin.org/index.php?topic=26866.msg342833#msg342833  its to avoid hopping back and fourth between two pools.
How much overhead is there when hopping between pools?  Would it be limited to how many work units a miner had queued up? Or is there some other losses Im missing?

No I don't think so. As written it wouldn't do that. Rather it actually prevents a switch until the lowest share server has at least 50% of the current server, not the difficulty. The other problem is that it's doing an update_server every share submit, not just on each LP.

I think your right, should it be

Code:

if servers[current_server]['shares'] - min_shares > .50 * servers[current_server]['shares']:

?

I think I would prefer my miners to switch as soon as there is anything lower, to maximize effectiveness. Unless the switching lowers your hasrate too much, but I dont see how it would.

[OCedHrt]

I think your right, should it be

Code:

if servers[current_server]['shares'] - min_shares > .50 * servers[current_server]['shares']:

?

I think I would prefer my miners to switch as soon as there is anything lower, to maximize effectiveness. Unless the switching lowers your hasrate too much, but I dont see how it would.

Actually, I changed it to:

Code:

            if servers[current_server]['shares'] / 2 > min_shares:

Which really achieves the same thing as using >. But this means it only switches if a pool has less than half the shares of the current pool. Though I think to prevent pool hopping backing back and forth you only need to require a 1-5% difference in shares.

The bigger issue I'm seeing is that it's doing an API stats pull every share accepted and that's going to raise some eyes from the pool operator side. I'm not sure if server side LP is even working.

[c00w]

1) Server side LP is broken?
Yeah. It is. I'm disabling it until I can figure out why twisted is ending the calls early.

2) Screwy logic with regards to switching?
Yeah. I missed that. I'm switching it over.

3) Better location of pool update function?
Well thats the wrong location because it gets called before we get new information. Where it is now if the logic wasn't broken it would work a lot better.

4) How often does bitHopper check pool API's?
EDIT: Changed to every 117 seconds.

5) BTCguild is broken?
I was waiting for their api to come up so that I could add the update function.

[burp]

Here is how to make twisted less noisy:

http://stackoverflow.com/questions/5078980/twisted-starting-stopping-factory-protocol-less-noisy-log-messages

EDIT: I just noticed you fixed a couple of things, nice

[c00w]

Here is some logic that popped in my head while looking through the code (I've never programmed in python but I have done some programming at school). Please let me know if this makes sense:

1. Check the api of each pool for share_count
2. If share_count[of the pool you are checking] < 1 then share_count[of the pool you are checking] = 10**10
3. Pick the pool with the lowest share count

Step 2 will catch any misbehaving api or a pool that is down and is returning a 0 for the share count.

Let me know if the code is doing this or something like this already, I have a hard time following some of the if statements, but you've inspired me to go and learn some python syntax!

I implemented something similar to this. Now if the API call fails it changes that server's difficulty to something along the line of 10^10. Your code is a good idea but wouldn't work because when the calls fail the server returns nothing and the error functions are called. Not a json api repsonse which can be parsed into a share count of 0.

[c00w]

Here is how to make twisted less noisy:

http://stackoverflow.com/questions/5078980/twisted-starting-stopping-factory-protocol-less-noisy-log-messages

EDIT: I just noticed you fixed a couple of things, nice

Well the issue is that the noisy factories are embedded in getPage. I could rewrite it but its a lot simpler to not bother.

I'm going to add a option parser so I can have a flag to turn off client side LP for cgminer and diablominer users as well as a debug flag for me which turns on the twisted output.

[burp]

Here is how to make twisted less noisy:

http://stackoverflow.com/questions/5078980/twisted-starting-stopping-factory-protocol-less-noisy-log-messages

EDIT: I just noticed you fixed a couple of things, nice

Well the issue is that the noisy factories are embedded in getPage. I could rewrite it but its a lot simpler to not bother.

I'm going to add a option parser so I can have a flag to turn off client side LP for cgminer and diablominer users as well as a debug flag for me which turns on the twisted output.

You can just turn off LP support with --no-longpoll for cgminer, so it's no problem now. (A:4339  R:77, and accepted/rejected ratio is still good) But fixing the issue doesn't harm either

[c00w]

The issue is that when I use a twisted deffered mechanism to handle LP requests, which is the only way I can see to do it, twisted sends back a packet with no http data to say," We're alive but we are not done processing no http response"

All the python based miners are cool with this.

Diablominer takes this packet and tries to operate on it and breaks.

Cgminer accepts this packet, goes LP WAS FIRED and grabs a new block via getwork. Which isn't how LP is supposed to work either. They should try and read a getwork from the response like Diablominer and crash or they should check for a null packet and do something like oh not cycle endlessly through getworks.

[burp]

Btw, this is missing to make btcguild actually work:

Code:

 def get_difficulty():
@@ -171,6 +171,7 @@ def btcguild_sharesResponse(response):
     round_shares = int(info['round_shares'])
     servers['btcg']['shares'] = round_shares
     log.msg( 'btcguild :' + str(round_shares))
+    server_update()

 def bclc_sharesResponse(response):
     global servers
@@ -213,7 +214,7 @@ def errsharesResponse(error, args):

 def btcg_getshares():
     d = getPage('https://www.btcguild.com/pool_stats.php')
-    d.addCallback(bclc_sharesResponse)
+    d.addCallback(btcguild_sharesResponse)
     d.addErrback(errsharesResponse, ('btcg'))
     d.addErrback(log.err)

@@ -247,6 +248,7 @@ def update_servers():
     mtred_getshares()
     bitclockers_getshares()
     mineco_getshares()
+    btcg_getshares()

[c00w]

Fixed.

[OCedHrt]

The issue is that when I use a twisted deffered mechanism to handle LP requests, which is the only way I can see to do it, twisted sends back a packet with no http data to say," We're alive but we are not done processing no http response"

All the python based miners are cool with this.

Diablominer takes this packet and tries to operate on it and breaks.

Cgminer accepts this packet, goes LP WAS FIRED and grabs a new block via getwork. Which isn't how LP is supposed to work either. They should try and read a getwork from the response like Diablominer and crash or they should check for a null packet and do something like oh not cycle endlessly through getworks.

I actually encountered a new problem with DiabloMiner where it loops RPC Request in BitHopper continuously. This only happens IF DiabloMiner doesn't error out as described earlier.

[c00w]

I actually encountered a new problem with DiabloMiner where it loops RPC Request in BitHopper continuously. This only happens IF DiabloMiner doesn't error out as described earlier.

Hmm. I ran diablo miner and didn't get the LP problem. I did see an abnormally high number of RPC requests. I am seeing an issue where my code is submitting 2-4 times as many RPC request as it should. I'm debugging.

EDIT: Turns out my logging function double printed certain things. Fixed now. It looks like Diablominer just pulls a lot of work. I'll keep playing with it and see if I can find the LP issue though.

EDIT AGAIN: Actually I remember causing that error in one of my commits. I fixed it once I tested it on my server and it didn't work roughly 2 minutes later. Git Pull and see if it works?

[OCedHrt]

I actually encountered a new problem with DiabloMiner where it loops RPC Request in BitHopper continuously. This only happens IF DiabloMiner doesn't error out as described earlier.

Hmm. I ran diablo miner and didn't get the LP problem. I did see an abnormally high number of RPC requests. I am seeing an issue where my code is submitting 2-4 times as many RPC request as it should. I'm debugging.

EDIT: Turns out my logging function double printed certain things. Fixed now. It looks like Diablominer just pulls a lot of work. I'll keep playing with it and see if I can find the LP issue though.

If Diablo Miner errors on start, it's ok:

Code:

[7/9/11 4:29:32 PM] Started
[7/9/11 4:29:32 PM] Connecting to: http://localhost:8337/
[7/9/11 4:29:32 PM] Using AMD Accelerated Parallel Processing OpenCL 1.1 AMD-APP
-SDK-v2.5 (684.211)
[7/9/11 4:29:34 PM] BFI_INT patching enabled, disabling hardware checking
[7/9/11 4:29:34 PM] Added Barts (#1) (14 CU, local work size of 256)
Waiting...Exception in thread "DiabloMiner LongPollAsync" java.lang.NullPointerE
Waiting...
        at com.diablominer.DiabloMiner.DiabloMiner.forceUpdate(DiabloMiner.java:
504)
        at com.diablominer.DiabloMiner.DiabloMiner$LongPollAsync.run(DiabloMiner
.java:1363)
        at java.lang.Thread.run(Unknown Source)
[7/9/11 4:29:37 PM] Accepted block 1 found on Barts (#1)

If it doesn't error, then we have problem. Log @ paste bin below. There may be some logging differences since I've made some small changes, but it still has same problem with the pool.py I downloaded.

Code:

[7/9/11 4:29:47 PM] Started
[7/9/11 4:29:47 PM] Connecting to: http://localhost:8337/
[7/9/11 4:29:47 PM] Using AMD Accelerated Parallel Processing OpenCL 1.1 AMD-APP
-SDK-v2.5 (684.211)
[7/9/11 4:29:49 PM] BFI_INT patching enabled, disabling hardware checking
[7/9/11 4:29:49 PM] Added Barts (#1) (14 CU, local work size of 256)
[7/9/11 4:30:35 PM] Accepted block 1 found on Barts (#1)

http://pastebin.com/e4hUXD6R

[c00w]

Its an LP issue. Diablominer doesn't actually read from the LP channel. Once it gets a packet it triggers and asks for more work from the normal channel. There is a way to rewrite the clientside LP and I'll add it to the list but for now just run with pool.py --noLP.

[OCedHrt]

Its an LP issue. Diablominer doesn't actually read from the LP channel. Once it gets a packet it triggers and asks for more work from the normal channel. There is a way to rewrite the clientside LP and I'll add it to the list but for now just run with pool.py --noLP.

That's fine. I'm actually using poclbm with phatk because DiabloMiner is returning high rejects (but maybe this is also due to LP).

Code:

mhash: 279.6/280.2 | accept: 32 | reject: 5 | hw error: 0

I get 1-2% reject on poclbm.

[toasty]

Code:

def selectsharesResponse(response, args):
    log_msg('Calling sharesResponse for '+ args)
    func_map= {'bitclockers':bitclockers_sharesResponse,
        'mineco':mineco_sharesResponse,
        'mtred':mtred_sharesResponse,
        'bclc':bclc_sharesResponse,
        'bitclockers':bitclockers_sharesResponse,
        'btcg':btcguild_sharesResponse}

Bitclockers is defined twice here. Not hurting anything, but might cause a bug down the road if only one of them gets changed.