DiabloMiner GPU Miner

[DiabloD3]

@DiabloD3: I'm currently writing my own pool software and i was wondering if DiabloMiner conforms with the longpoll protocol description at https://deepbit.net/longpolling.php ? because currently my code works perfectly with Phoenix but Diablominer screws up:

Code:

[09-07-11 04:08:06] Started
[09-07-11 04:08:06] Connecting to: http://localhost:8000/
[09-07-11 04:08:06] Using AMD Accelerated Parallel Processing OpenCL 1.1 AMD-APP
-SDK-v2.4 (650.9)
[09-07-11 04:08:08] Added ATI RV770 (#2) (10 CU, local work size of 256)
[09-07-11 04:08:15] DEBUG: Enabling long poll support
[09-07-11 04:08:16] DEBUG: Long poll returned
[09-07-11 04:08:17] DEBUG: Long poll returned
[09-07-11 04:08:23] DEBUG: Long poll returned
[09-07-11 04:08:24] DEBUG: Long poll returned
[09-07-11 04:08:25] DEBUG: Long poll returned
[09-07-11 04:08:26] DEBUG: Long poll returned
[09-07-11 04:08:27] DEBUG: Long poll returned
[09-07-11 04:08:28] DEBUG: Long poll returned
[09-07-11 04:08:29] DEBUG: Long poll returned
[09-07-11 04:08:30] DEBUG: Long poll returned
[09-07-11 04:08:30] DEBUG: Long poll returned
[09-07-11 04:08:32] DEBUG: Long poll returned
[09-07-11 04:08:33] DEBUG: Long poll returned
mhash: 51,1/47,0 | a/r/hwe: 0/0/0 | ghash: 1,2 | fps: 29,8

I guess you're using the protocol differently because i never actually recieves any GET request from diablominer...
Could you prehaps give me the specifications you go by?

I follow the URL RFC and the LP specification. What URL are you giving it in the LP header?

Code:

response.AppendHeader("X-Long-Polling","");

AKA same URL but i guess it probably would be a good idea to add /LP just to avoid confusing miners But none the less i'm never getting a GET request which i technically should :p

Okay so i changed my code around a bit to use the incomming RawUrl as the determining factor for wether or not it's the LP request wich works fine for both phoenix and diablominer BUT diablominer starts spamming getwork requests after ~57 seconds @ 60-70 mhash/s
Any idea what that could be?

"" makes no sense, because then it would use the existing URL with no modifications. How would the pool server be able to tell the difference? My miner immediately starts a new LP attempt once the existing one completes. So, yes, it starts spamming because LP is not supposed to return immediately.

"foo" adds /foo to the end of the existing URL's path. "/foo" replaces the entire path, and then it also accepts a fully qualified URL. Other miners seem to only accept "foo" in violation of any sane interpretation of the URL RFC.

[Zagitta]

-snip old stuff-

"" makes no sense, because then it would use the existing URL with no modifications. How would the pool server be able to tell the difference? My miner immediately starts a new LP attempt once the existing one completes. So, yes, it starts spamming because LP is not supposed to return immediately.

"foo" adds /foo to the end of the existing URL's path. "/foo" replaces the entire path, and then it also accepts a fully qualified URL. Other miners seem to only accept "foo" in violation of any sane interpretation of the URL RFC.

https://deepbit.net/longpolling.php[/url]]Long polling protocol description

1) Miner initiates connection to the mining pool just as usual, requests getwork and starts working on it.

2) If mining pool does supports Long Polling, it should include a special header:
X-Long-Polling: /long-polling-url
where /long-polling-url is a path for long polling connection. It can be a full URL with separate port too.

3) Miner starts a request to long polling URL with GET method and same basic authorization as on main connection.
This request is not answered by server until new block is found by bitcoin network. The answer is the same as getwork on the main connection. Upon receiving this answer, miner should drop current calculation in progress, discard it's result, start working on received data and make a new request to a long polling URL.

4) If all the nonce space is exhausted during calculation or 60 seconds passed since receiving the data, the miner should request new one by means of main connection. 60 seconds limit is set to allow adding new transactions into the block.

So yes "" does make sense because the pool can just check wether it's a GET or POST request...
Anyway that's not important because i changed my code...

Also we're talking past eachother here i think... Because it doesn't matter wether i add /foo, foo or http://localhost:8000/foo to the header, it still connects just fine...

the Longpolling issue is already fixed!

The spamming im talking about now is this:

Code:

[18:28:24] {"method":"getwork","params":[],"id":1}
[18:28:34] {"method":"getwork","params":[],"id":1}
[18:28:35] {"method":"getwork","params":[],"id":1}
Longpoll connection added
[18:28:44] {"method":"getwork","params":["0000000150038abc2f93cb863de66465583803
652bc9b21f3914467a0000043a00000000293f71de5bfea504c4dd9bedd2f29788c74d310d6a00ba
823ae665894adaeb7b4e1881b71a0abbcf4655401100000080000000000000000000000000000000
0000000000000000000000000000000000000000000000000080020000"],"id":1}
Solved work submitted
[18:29:34] {"method":"getwork","params":[],"id":1}
[18:29:35] {"method":"getwork","params":[],"id":1}
[18:29:35] {"method":"getwork","params":[],"id":1}
[18:29:35] {"method":"getwork","params":[],"id":1}
[18:29:36] {"method":"getwork","params":[],"id":1}
[18:29:36] {"method":"getwork","params":[],"id":1}
[18:29:37] {"method":"getwork","params":[],"id":1}
[18:29:37] {"method":"getwork","params":[],"id":1}
[18:29:37] {"method":"getwork","params":[],"id":1}
[18:29:38] {"method":"getwork","params":[],"id":1}
[18:29:38] {"method":"getwork","params":[],"id":1}
[18:29:38] {"method":"getwork","params":[],"id":1}
[18:29:39] {"method":"getwork","params":[],"id":1}
[18:29:39] {"method":"getwork","params":[],"id":1}
[18:29:39] {"method":"getwork","params":[],"id":1}
[18:29:40] {"method":"getwork","params":[],"id":1}

diablominer output says nothing about the issue... And phoenix doesn't do the same, nor does poclbm so i has to assume you're doing something special?

[DiabloD3]

3) Miner starts a request to long polling URL with GET

So yes "" does make sense because the pool can just check wether it's a GET or POST request...

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

If you want LP to work, it must be a unique URL.

Try running DM with -dd, you'll see a matching flood of LP returned messages because your server is not holding the LP request open correctly.

[DiabloD3]

Update: Further fix against broken pools

[wasabi]

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

Can you clarify this a bit? My understanding is that a LP URL does not have a body, and thus using GET is just fine.

[DiabloD3]

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

Can you clarify this a bit? My understanding is that a LP URL does not have a body, and thus using GET is just fine.

Nope, LP expects a standard JSON RPC request as far as I can tell.

[wasabi]

From deepbit: Miner starts a request to long polling URL with GET method and same basic authorization as on main connection.

It does not specify whether there is a body. I should note, that I do not use a body, and it works fine, with every pool I've seen. If it does expect a body, it is a contradictory spec, as you said.

[Zagitta]

3) Miner starts a request to long polling URL with GET

So yes "" does make sense because the pool can just check wether it's a GET or POST request...

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

If you want LP to work, it must be a unique URL.

Try running DM with -dd, you'll see a matching flood of LP returned messages because your server is not holding the LP request open correctly.

Could you at least try reading my whole post? This has nothing todo with longpoll anymore because i implemented it differently...

proff:

[kaosbit]

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

You are correct regarding the HTTP spec, of course. The LP specification is rather ambiguous at this point. However, a GET would actually make sense when the reply is a JSON-RPC notification (id: null). Then the procedure would be like this:

• The client does a simple GET on the LP URL, indicating its interest in "interruption" by notification
• Once the network finds a new block, the server replies with the fresh hash, as if the client just issued a getwork request
• The client aborts/discards the obsolete calculation and starts on the received fresh one - no need for a getwork on the main connection!
• The client re-reisters for "interruption" by starting again with the first GET step.

Of course this is just my interpretation of the LP spec. But it seems that at least poclbm agrees with me, from a quick glance at the sources.

Anyway, thanks for this nice piece of software, keep up the good work!

[Zagitta]

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

You are correct regarding the HTTP spec, of course. The LP specification is rather ambiguous at this point. However, a GET would actually make sense when the reply is a JSON-RPC notification (id: null). Then the procedure would be like this:

• The client does a simple GET on the LP URL, indicating its interest in "interruption" by notification
• Once the network finds a new block, the server replies with the fresh hash, as if the client just issued a getwork request
• The client aborts/discards the obsolete calculation and starts on the received fresh one - no need for a getwork on the main connection!
• The client re-reisters for "interruption" by starting again with the first GET step.

Of course this is just my interpretation of the LP spec. But it seems that at least poclbm agrees with me, from a quick glance at the sources.

Anyway, thanks for this nice piece of software, keep up the good work!

This is indeed the same interpretation i had of it, which Phoenix also seems to follow... The GET request is indeed without a body so one might even be bold enough say to DiabloD3 that phoenix implements it correctly

[kaosbit]

Something is extremely wrong with OSX. Its saying its executing kernels, but then not, and then doesn't emit any errors.

DM will wildly spin like that when kernels take zero time to execute. Problem is, the implementation MUST emit errors.

I so very much hate OSX.

Well, yeah, OSX is special. I am experiencing the same problem with the master of DiabloMiner. Strangely enough, a previous version (the one packaged as a Mac App) works nicely! Playing around a bit, I found the problem seems to have occurred between these git commits:

commit 354b53286af2e982e5c8aa030784a785ebc5032b
Date:   Sat Jun 25 02:17:14 2011 -0400

    Switched to an unholy union of the current kernel and phatk. Dear Lord,
    forgive me for I have sinned.

commit 2d0ed8523c17d011ae5e755936baab01becab171
Date:   Fri Jun 24 01:22:45 2011 -0400

    Move execution threads to 2 down from 3 since getwork can no longer
    block execution threads

I branched from the "pre-phatk" checkout and patched all later non-kernel changes into it. Works great!

So the new kernel must at least be part of the issue. Figures, poclbm with phatk actually freezes my WindowServer, grrr.

But you're right, OSX really should do a better job at error reporting. With the way it behaves now, I have no idea how to fix this.

Oh well, praise git. I'll stick with my branch for now.

[DiabloD3]

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

You are correct regarding the HTTP spec, of course. The LP specification is rather ambiguous at this point. However, a GET would actually make sense when the reply is a JSON-RPC notification (id: null). Then the procedure would be like this:

• The client does a simple GET on the LP URL, indicating its interest in "interruption" by notification
• Once the network finds a new block, the server replies with the fresh hash, as if the client just issued a getwork request
• The client aborts/discards the obsolete calculation and starts on the received fresh one - no need for a getwork on the main connection!
• The client re-reisters for "interruption" by starting again with the first GET step.

Of course this is just my interpretation of the LP spec. But it seems that at least poclbm agrees with me, from a quick glance at the sources.

Anyway, thanks for this nice piece of software, keep up the good work!

The LP spec doesn't say "the LP request is not a JSON RPC request" explicitly. If it did, then, yes, I could get away with doing GET. Receiving a JSON RPC response is questionable at best in the case that this is true (although normal JSON over HTTP obviously is GETtable).

I don't particularly like the idea of mixing JSON RPC and non JSON RPC in the same protocol. If the LP spec says that you don't need to send a JSON RPC request, then the LP spec is wrong.

[DiabloD3]

3) Miner starts a request to long polling URL with GET

So yes "" does make sense because the pool can just check wether it's a GET or POST request...

ANY client that EVER uses GET for JSON RPC is wrong. Read the HTTP spec, GETs should never have message bodies. DiabloMiner follows the HTTP specification and uses POST for all requests.

If you want LP to work, it must be a unique URL.

Try running DM with -dd, you'll see a matching flood of LP returned messages because your server is not holding the LP request open correctly.

Could you at least try reading my whole post? This has nothing todo with longpoll anymore because i implemented it differently...

proff:

Print the responses to those requests including the HTTP status codes.

[DiabloD3]

Update: Multiple pool support, use commas

[kaosbit]

The LP spec doesn't say "the LP request is not a JSON RPC request" explicitly. If it did, then, yes, I could get away with doing GET. Receiving a JSON RPC response is questionable at best in the case that this is true (although normal JSON over HTTP obviously is GETtable).

I don't particularly like the idea of mixing JSON RPC and non JSON RPC in the same protocol. If the LP spec says that you don't need to send a JSON RPC request, then the LP spec is wrong.

Agreed, this would be questionable at best. The JSON-RPC spec clearly states "A response may only be sent in reply to a request." Just to see what happens, I modified the miner to actually use GET without a request body on the LP connection. The response from pushpool is still a JSON-RPC response, just as if it had received the request:

Code:

GET /LP HTTP/1.1
Authorization: Basic a2FvX2dwdTpmUjMzbDAwN3I=
Accept-Encoding: gzip,deflate
Content-Type: application/json
Cache-Control: no-cache
User-Agent: Java/1.6.0_26
Host: pit.x8s.de:8337
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive

HTTP/1.1 200 ok
Content-Type: application/json
X-Roll-NTime: Y
Date: Mon, 11 Jul 2011 11:54:01 GMT
Content-Length: 591

{"id":1,"error":null,"result":{"midstate":"d3eed22cc5909d76ac118412da850d8dfabc49687f1674fbcc73cecc45166cbd","target":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000","data":"00000001faca27b55e2524d24df3ab7ef9392b5c85a11bb006de813b000001d00000000035940bd70e45c1e87c69af951db0705d893945dda73c0202ad691730070e15044e1ae4571a0abbcf00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000","hash1":"00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000"}}

Bottom line: It seems that pools really don't care one way or another. So let's stick with proper JSON-RPC.

[ancow]

Update: Multiple pool support, use commas

You initialise the networkStates array with a length of 0 if the url command line option is not used, resulting in an ArrayIndexOutOfBoundsException on line 427:

Code:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
        at com.diablominer.DiabloMiner.DiabloMiner.execute(DiabloMiner.java:427)
        at com.diablominer.DiabloMiner.DiabloMiner.main(DiabloMiner.java:133)

Initialising it to 1 with the command line arguments:

Code:

-dd -o "swepool.net" -r 8337 -p "mypwd" -u "myuser" -w 128

I get the following exception:

Code:

Exception in thread "DiabloMiner GetWorkAsync for swepool.net" java.lang.NullPointerException
        at java.io.StringReader.<init>(StringReader.java:33)
        at org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:427)
        at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1285)
        at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1007)
        at com.diablominer.DiabloMiner.DiabloMiner$NetworkState.doJSONRPC(DiabloMiner.java:688)
        at com.diablominer.DiabloMiner.DiabloMiner$NetworkState$GetWorkAsync.run(DiabloMiner.java:815)
        at java.lang.Thread.run(Thread.java:680)

A little more testing without the url option might be in order...

[DiabloD3]

Update: Multiple pool support, use commas

You initialise the networkStates array with a length of 0 if the url command line option is not used, resulting in an ArrayIndexOutOfBoundsException on line 427:

Code:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 0
        at com.diablominer.DiabloMiner.DiabloMiner.execute(DiabloMiner.java:427)
        at com.diablominer.DiabloMiner.DiabloMiner.main(DiabloMiner.java:133)

Initialising it to 1 with the command line arguments:

Code:

-dd -o "swepool.net" -r 8337 -p "mypwd" -u "myuser" -w 128

I get the following exception:

Code:

Exception in thread "DiabloMiner GetWorkAsync for swepool.net" java.lang.NullPointerException
        at java.io.StringReader.<init>(StringReader.java:33)
        at org.codehaus.jackson.JsonFactory.createJsonParser(JsonFactory.java:427)
        at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1285)
        at org.codehaus.jackson.map.ObjectMapper.readTree(ObjectMapper.java:1007)
        at com.diablominer.DiabloMiner.DiabloMiner$NetworkState.doJSONRPC(DiabloMiner.java:688)
        at com.diablominer.DiabloMiner.DiabloMiner$NetworkState$GetWorkAsync.run(DiabloMiner.java:815)
        at java.lang.Thread.run(Thread.java:680)

A little more testing without the url option might be in order...

... shit. I know exactly what I did too.

Note to self: Don't code for a few hours, then release it.

Update: Fixed the obvious bug.

[DiabloD3]

Update: Fix up the X-Switch-To code

[wasabi]

The LP spec doesn't say "the LP request is not a JSON RPC request" explicitly. If it did, then, yes, I could get away with doing GET. Receiving a JSON RPC response is questionable at best in the case that this is true (although normal JSON over HTTP obviously is GETtable).

I don't particularly like the idea of mixing JSON RPC and non JSON RPC in the same protocol. If the LP spec says that you don't need to send a JSON RPC request, then the LP spec is wrong.

I agree. It's not much of a spec. It's more like, 5 half-assed written paragraphs. But that's fine. We do with it what we can.

Perhaps the problem is you shouldn't consider LP as JSON-RPC. Maybe it's not. After all, if it was, it would not specifically mention GET. It's just a hack. I don't see any other way to interpret it. It's a GET request whose response is a JSON-RPC method response packet, completely outside of the normal JSON-RPC flow.

It should also be noted that no matter what you POST to it, it works. You can post completely arbitrary data, and it will still respond with a JSON-RPC response. Or you can post nothing. It is clearly not JSON-RPC.

[kaosbit]

You can post completely arbitrary data, and it will still respond with a JSON-RPC response. Or you can post nothing. It is clearly not JSON-RPC.

Probably a case of "be strict in what you send, but generous in what you receive". And good practice too for some piece of pool software, considering there are so many ways to interpret the LP spec.