[ANN] Stratum mining protocol - ASIC ready

[DrHaribo]

I believe I saw somewhere a way for a server to ask a client to identify itself (software name and version). Is this something the current crop of clients support?

[1713960417]

1713960417

[1713960417]

1713960417

[1713960417]

1713960417

[1713960417]

1713960417

[-ck]

I believe I saw somewhere a way for a server to ask a client to identify itself (software name and version). Is this something the current crop of clients support?

client.get_version

Supported by cgminer.

[fireduck]

I have made a java stratum pool server implementation.  It seems to be working pretty well.  Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right?  This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex.  If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

[kano]

...
Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.
...

As far as I know, none of the pools support this yet

cgminer currently assumes that a disconnect means throw away the work, since all pools do that.

Be the first! Make the others do it too!

[eleuthria]

I have made a java stratum pool server implementation.  It seems to be working pretty well.  Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right?  This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex.  If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

For #2)  Yes, work with Stratum is defined per connection, so is difficulty.  If people pool multiple workers over one connection, they have to work at whatever the highest difficulty worker is set to.

[fireduck]

I have made a java stratum pool server implementation.  It seems to be working pretty well.  Once I pull my passwords and settings out of the source into a config file I'll put it up on github.

Questions:

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

2) I should be able to support a bunch of authenticated workers on a single TCP connection, right?  This means they are all going to work with the same mining.notify messages and with the same extranonce1 values?

If I do this, it makes me dedup code a little more complex.  If I don't do it, it probably disables some interesting proxy use cases.

I could also see an issue with select-able difficulty pools (like mine) where a user creates a stratum connection with a bunch of workers at different difficulty levels and then when they find a share they submit it as whichever worker has the highest difficulty that it matches, thus greatly increasing their pay at my expense.

I've put my code on github in case anyone is interested:
https://github.com/fireduck64/SockThing

It seems to work but is certainly not done.  I need to add metrics, probably some job management and clean up the multi-connection same user story a bit.

[eleuthria]

I've put my code on github in case anyone is interested:
https://github.com/fireduck64/SockThing

It seems to work but is certainly not done.  I need to add metrics, probably some job management and clean up the multi-connection same user story a bit.

Ahh, the Java, it burns!  Glad to see another fresh implementation of Stratum.  I'll try to take a deeper look at it tomorrow, but so far I must say it's one of the cleaner java-based pool servers I've seen in quite some time.

[slush]

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

[kano]

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

Except ... for anyone with an unreliable internet connection (happens with wireless and 3g and such) they lose more shares the more often they lose the connection.
Of course if you have a dynamic IP address, you also lose shares during that change, however often that happens - but obviously that's not often.
... pity it's a new problem added by Stratum (that didn't already exist)

Yes in general it is a small number of shares, but it also means that if the pool itself does a restart, everyone loses the work they are working on and can never submit it.
Yes a proper implementation would have to handle a pool restart also - but that is the pools problem not the protocol's problem.

However, the problem is simply that it is hard for the pools to implement, so the pools have shied away from doing it or are unable to do it due to their own design.

[slush]

properly implemented server and client and standard internet connection won't drop the connection for days.

Except ... for anyone with an unreliable internet connection (happens with wireless and 3g and such) they lose more shares the more often they lose the connection.

Important part highlighted.

Of course if you have a dynamic IP address, you also lose shares during that change

Why? Providers are dropping TCP connection because of moving the customer to new IP address?

... pity it's a new problem added by Stratum (that didn't already exist)

A lot of other and more important issues existed before, which are solved by Stratum.

Yes a proper implementation would have to handle a pool restart also - but that is the pools problem not the protocol's problem.

Ok, adding session identifier to the response of mining.subscribe() is trivial, this identifier can be used by mining.resume() on the beginning of another TCP connection. But you're literally opening Pandora box:

a) Backends must have some backbone channel for retrieving session data of given session identifier
or
b) Backends must use exactly the same template, so session identifier can be the same as extranonce1, but backends must have backbone for validating duplicated submissions.

Both solutions have drastical impact to scalability and both solutions are extremely hard to implement. It also open another doors for attacks (DoS, share double spends). That additional infrastructure is definitely not worth of <0.001% lost shares of users with non-standard mining setup.

I agree with you kano, that it's not perfect. But adding another magnitude of complexity won't help much.

[Luke-Jr]

I'd suggest just adding an optional field to the end of mining.submit for extranonce1 that the miner can use for old share submissions on reconnect.
Let the pool decide whether to honour them or not, and for how long.

[slush]

I'd suggest just adding an optional field to the end of mining.submit for extranonce1 that the miner can use for old share submissions on reconnect.
Let the pool decide whether to honour them or not, and for how long.

But pool cannot validate the share unless the backend has the same block template (with the same job ID). Yes, we can start adding "optional fields", but that's not the direction of development which I want to go.

If something, then go with mining.resume() and let the pool decide if he'll accept it or not. However I don't think anybody sane will want to implement full mechanism of session resuming.

[-ck]

If something, then go with mining.resume() and let the pool decide if he'll accept it or not. However I don't think anybody sane will want to implement full mechanism of session resuming.

You have certainly expressed your disinterest in implementing it, but that is not the case for all other pool ops. If just one pool op decides to implement this, I will support it in cgminer, and as per always, once one pool has set the standard, it would be crazy for others to not follow. So who's first?

[slush]

If just one pool op decides to implement this, I will support it in cgminer.

No problem. I just hope that the first daredevil will implement it properly, so response of mining.subscribe will have one extra argument 'session-id':

{"id": 1, "result": [[], "08000002", 4, "some-session-id"], "error": null}\n

If this parameter is set and non-null, the miner will store sesssion id internally and then, after the crash of the connection, miner will call mining.resume('session-id') instead of mining.subscribe(...). Pool will respond with true/false. If true, miner can continue in the previous session (so re-upload pending shares, ...), if false, miner must call mining.subscribe and throw away pending shares.

Correct?

[fireduck]

1) Should a work unit (from a mining.notify message) be usable by a client if the TCP connection breaks and they reconnect?  It makes sense to me that it should, but I am having trouble because the mining.subscribe seems to occur before the mining.authorize so I need to issue (or reissue) the same extranonce1 before I know which user it is.

Please propose some protocol extension before you implement anything in this area. I was thinking about some session identifiers and resuming the connection, but I didn't find any usable (simple and scalable) solution yet. From my perspective this is actually much smaller problem that it seems to be; properly implemented server and client and standard internet connection won't drop the connection for days. And one second of lost work per day is definitely NOT worth all the effort.

So, I've just made SockThing use the same extranonce1 for all connections.  I am also adding an extra random number to the coinbase and generate a new coinbase for each job.

Now reconnects should work fine, except clients have no way of knowing that they will.

I'm thinking maybe adding a field to mining.subscribe that indicates this feature exists.

However, I don't imagine other pool operators are inclined to build a new coinbase for each user so I don't imagine my solution will be for everyone.

[fireduck]

If just one pool op decides to implement this, I will support it in cgminer.

No problem. I just hope that the first daredevil will implement it properly, so response of mining.subscribe will have one extra argument 'session-id':

{"id": 1, "result": [[], "08000002", 4, "some-session-id"], "error": null}\n

If this parameter is set and non-null, the miner will store sesssion id internally and then, after the crash of the connection, miner will call mining.resume('session-id') instead of mining.subscribe(...). Pool will respond with true/false. If true, miner can continue in the previous session (so re-upload pending shares, ...), if false, miner must call mining.subscribe and throw away pending shares.

Correct?

That makes sense to me.  I know the ship has long since sailed, but I kinda with these weren't json arrays but instead where key value pairs so that it would be a little more self describing and easier to add various extensions.  Maybe any extensions should be done as pairs rather than tacking values on to keep things clear?

[slush]

So, I've just made SockThing use the same extranonce1 for all connections.  I am also adding an extra random number to the coinbase and generate a new coinbase for each job.

Why is this better than generating custom extranonce1? Extranonce1 has been designed exactly for this. If you can achieve the same block template on all backends, then you can make session_id==extranonce1 and my proposal will work for you.

[slush]

That makes sense to me.  I know the ship has long since sailed, but I kinda with these weren't json arrays but instead where key value pairs so that it would be a little more self describing and easier to add various extensions.  Maybe any extensions should be done as pairs rather than tacking values on to keep things clear?

Then next logical step is to get rid of json entirely and propose protobuf messages. I picked json for good reasons and I still think it was smart choice, but for future development protobuf would be better. If anyone thinks about protocol change, then please take protobuf in account.

[fireduck]

So, I've just made SockThing use the same extranonce1 for all connections.  I am also adding an extra random number to the coinbase and generate a new coinbase for each job.

Why is this better than generating custom extranonce1? Extranonce1 has been designed exactly for this. If you can achieve the same block template on all backends, then you can make session_id==extranonce1 and my proposal will work for you.

Yeah, your proposal is a clean way of doing it.  Mine was just a hack to work within the existing protocol.

[-ck]

If just one pool op decides to implement this, I will support it in cgminer.

No problem. I just hope that the first daredevil will implement it properly, so response of mining.subscribe will have one extra argument 'session-id':

{"id": 1, "result": [[], "08000002", 4, "some-session-id"], "error": null}\n

If this parameter is set and non-null, the miner will store sesssion id internally and then, after the crash of the connection, miner will call mining.resume('session-id') instead of mining.subscribe(...). Pool will respond with true/false. If true, miner can continue in the previous session (so re-upload pending shares, ...), if false, miner must call mining.subscribe and throw away pending shares.

Correct?

Nice.

If mining.resume is successful, does this mean the miner does not need to do mining.authorize afterwards?