[Claimed - 6 BTC bounty] New RPCProtocol for Phoenix

[jedi95]

For the past week I have been working on a new RPC implementation for Phoenix since the current one has several issues I have not been able to fix. It doesn't help matters that the original RPC code author, CFSworks, has been MIA for awhile now.

So I decided to rewrite the RPC code, without using the Twisted.web library. I think most of the issues with the current RPC protocol are due to the hacked-in support for keep-alive.

Anyway, what I have right now is basically an adaption of poclbm's RPC code to Phoenix's interface. It makes use of the Twisted library, but only for handling threads/deferreds.

The 6 BTC bounty will go to whoever can write code that meets the requirements below. You can use what I have or write it from scratch, so long as it meets the requirements. The current code successfully connects and gets work, but it doesn't correctly send shares back. Obviously the code must be distributed under the X11 license like the rest of Phoenix. The bounty is payable immediately to the address of your choice once I can confirm that the code meets the requirements.

Requirements:
1. Calls to requestWork() and sendResult() must not block while sending/receiving requests from the RPC server. (these don't need to be thread-safe, they are always called from the main thread)
2. For sent work, a boolean must be returned to the caller of sendResult() indicating accepted/rejected.
3. Sending and receiving work should use separate connections. (but only 1 each, to keep pool load down)
4. Sending and receiving work must not block each other.

Existing code:
http://xp-dev.com/svn/phoenix-miner/branches/NewRPC/RPCProtocol.py

Networking code isn't a strong point of mine, and I just don't have the time to do this myself.

[fpgaminer]

So I decided to rewrite the RPC code, without using the Twisted.web library. I think most of the issues with the current RPC protocol are due to the hacked-in support for keep-alive.

Keep Alive is only used for the Long Polling connections, as far as I know. So, do you suspect that whatever hack Phoenix uses to enable Keep Alive is also interfering with getwork requests and submissions? Or is Long Polling the only issue?

I'm just curious and trying to flesh out more information in this thread.

[jedi95]

So I decided to rewrite the RPC code, without using the Twisted.web library. I think most of the issues with the current RPC protocol are due to the hacked-in support for keep-alive.

Keep Alive is only used for the Long Polling connections, as far as I know. So, do you suspect that whatever hack Phoenix uses to enable Keep Alive is also interfering with getwork requests and submissions? Or is Long Polling the only issue?

I'm just curious and trying to flesh out more information in this thread.

The keep-alive connections are required for stable operation on some pools due to anti-DDoS protection. Without keep-alive the miner makes a new connection for each getwork() as opposed to re-using the existing one. A good example is Slush's pool, which many users were having problems with prior to adding keep-alive. The current RPCProtocol implementation uses the Twisted.web library, which does not support keep-alive. CFSworks "hacked-in" keep-alive support for Twisted.web so that we could continue to use it. However, I suspect that the idle issues people have been having with Phoenix are caused by a bug in this modification. All my miners run through MMP and I have yet to see a single idle problem, which eliminates the rest of the components as the source of the problem.

Basically, what I want is a new RPCProtocol that doesn't use Twisted.web.

[fpgaminer]

Okay so, given your code, you do this for sending work:

Code:

d = threads.deferToThread(self.getwork, self.sendConnection, data)

Which means that it actually calls getwork. I guess you didn't implement a submitwork function and just used that as a placeholder?

EDIT: I take that back; getwork should be compatible with doing submitwork requests, because of the data parameter.


I don't know Phoenix's code well enough to immediately work on it myself. Seems overly complicated, compared to just having two threads (mining thread and network thread) and two Queues (workQueue and resultsQueue). I actually moved away from Phoenix because of the idle issues, after I hacked phatk into poclbm. poclbm's code is quite simple and indeed uses the two threads with Queues architecture.

[fpgaminer]

Okay I've fixed your code so that it submits shares correctly, tested against eligius. Here are the changes:

Line 82 and 83 changed to:

Code:

        self.postdata = {"method": 'getwork', 'id': 1}
        self.headers = {"User-Agent": USER_AGENT, "Authorization": 'Basic ' + b64encode('%s:%s' % (url.username, url.password)), "Content-Type": 'application/json'}

Changed the id to 1. I doubt it makes a difference, but Phoenix uses 1 normally.
Added a Content-Type header, which is necessary for submitting work, but not getting work ... not sure why. That's just how the pool operates I guess.

Line 290:

Code:

        d = self.sendWork(result.encode('hex'))

You already wrap the data in an array, so that wasn't necessary, and we need to hex the whole result, not just the first byte.

And that's it! I have no tested if this solves the idle problem. Someone else who experiences the idle problem will probably have to test it out for an extended period of time to see if your code fixes it.

[jedi95]

Okay so, given your code, you do this for sending work:

Code:

d = threads.deferToThread(self.getwork, self.sendConnection, data)

Which means that it actually calls getwork. I guess you didn't implement a submitwork function and just used that as a placeholder?

EDIT: I take that back; getwork should be compatible with doing submitwork requests, because of the data parameter.


I don't know Phoenix's code well enough to immediately work on it myself. Seems overly complicated, compared to just having two threads (mining thread and network thread) and two Queues (workQueue and resultsQueue). I actually moved away from Phoenix because of the idle issues, after I hacked phatk into poclbm. poclbm's code is quite simple and indeed uses the two threads with Queues architecture.

I don't like how poclbm works, since it sends and recieves work in the same thread. This means that if sending results stalls for even a few seconds, the miner is unable to get more work. This is made worse by the fact that poclbm lacks a queue, so it doesn't have a buffer of work for when this occurs.

This is one of the main reasons I decided to make another miner.

[jgarzik]

Oh good grief, let's not reinvent the wheel again.

Just use the efficient binary protocol already supported by pushpoold.

[fpgaminer]

I don't like how poclbm works, since it sends and recieves work in the same thread. This means that if sending results stalls for even a few seconds, the miner is unable to get more work. This is made worse by the fact that poclbm lacks a queue, so it doesn't have a buffer of work for when this occurs.

Good points. I'll have to add those features to my modified poclbm client 

Did the changes I posted work for you, by the way?

[jedi95]

Okay I've fixed your code so that it submits shares correctly, tested against eligius. Here are the changes:

Line 82 and 83 changed to:

Code:

        self.postdata = {"method": 'getwork', 'id': 1}
        self.headers = {"User-Agent": USER_AGENT, "Authorization": 'Basic ' + b64encode('%s:%s' % (url.username, url.password)), "Content-Type": 'application/json'}

Changed the id to 1. I doubt it makes a difference, but Phoenix uses 1 normally.
Added a Content-Type header, which is necessary for submitting work, but not getting work ... not sure why. That's just how the pool operates I guess.

Line 290:

Code:

        d = self.sendWork(result.encode('hex'))

You already wrap the data in an array, so that wasn't necessary, and we need to hex the whole result, not just the first byte.

And that's it! I have no tested if this solves the idle problem. Someone else who experiences the idle problem will probably have to test it out for an extended period of time to see if your code fixes it.

Nice work! With these changes I was able to finish a complete RPCProtocol including long polling. Let me know what address you want payment sent to.

[fpgaminer]

Nice work! With these changes I was able to finish a complete RPCProtocol including long polling. Let me know what address you want payment sent to.

Yay! I'm glad I was able to help, and thank you in advance for the bounty 
1FEmGDo7Csi1S3w3pYuWwoV885UkiWSwx5