[ANN] Stratum mining protocol - ASIC ready

[-ck]

There is no guarantee of serialising of data sent/received so the mining software has to handle it asynchronously and should be able to cope with a few queued messages at once. Most pools currently implementing stratum do give difficulty changes back to back with new work information. The only vardiff pool which currently seems to have its variable difficulty fluctuating wildly is EMC. The others seem to be solid after a very short period of adaptation. I did ask Inaba about hysteresis in his calculations for var diff and he insisted it was there, but the amount of diff changes I see on EMC have been enough for me to not mine stratum there. As far as I can see, this problem occurs on EMC with GBT as well.

In cgminer's case, if the difficulty has risen, it still submits the shares of lower difficulty if it was already work in progress. This ends up creating a little burst of rejects when diff rises. If the difficulty has dropped since the work was done, cgminer retargets to the new difficulty. Regardless, the separation of work and difficulty means there are still issues, but I have tried my best to alleviate the loss to miners in cgminer.

[Luke-Jr]

I have read the discussion about separating target/difficulty and work data. I have to say I think this was a mistake as it opens possibilities for abuse. I think something about these issues should go in the docs. Otherwise someone is going to create a naive implementation with some nasty issues.

Issue #1: As Eleuthria mentioned miners can hold back proofs of work. Keep results that are above the target, and submit them later if the difficulty decreases. Keep results that are far below the target. Submit them later if difficulty increases, getting better pay for them. By varying your hashrate between two Stratum pools you can make the two pools raise and lower your difficulty frequently. This enables a kind of Stratum pool hopping.

You can prevent this on the server side by always following the change of difficulty with sending out new work and flushing old work. Effectively negating the separation of diff/target and work data.

Issue #2: As Kano mentioned an evil pool op can steal from you. When the miner sends in a hash that is below target but not too far below it, simply reply with "1. I'm raising the difficulty. 2. your share is rejected due to the new difficulty". This will happen naturally once in a while as there is a race condition inherent in the protocol. But as a pool op you can make it happen on purpose to steal from miners. Then reduce difficulty again, and repeat. Raising difficulty once in a while will also make the miner throw away work it was just about to send in. Lowering the difficulty again could make up for it if the miner has kept old low difficulty shares. But oops, the server is flushing old work to prevent issue #1, so they are no longer valid.

With getwork and getblocktemplate a reject due to the hash being above target means there is a bug in your miner. With Stratum it means "it could be nothing - but maybe the pool op is stealing from you."

While issue #1 is fixable, issue #2 can't be fixed 100% without changing the protocol. The best you can do is to always follow up every difficulty change with quick new work data that flush the old. This should reduce the frequency of the problem. Then just hope the miner won't tell the user too often that you are stealing.

Is it ok to pipeline commands to the client? That is, send the difficulty change and new work (2 RPC calls) in quick succession without waiting for a reply to the first.

That should at least reduce the amount of visible rejects that cause miners to think you are stealing. However it won't fix it 100% and having to flush old work to change the difficulty will also waste some hashpower.

These sound like excellent points to bring up when slush publishes the first draft of the stratum BIP.
I've been making a list of possible improvements in stratum myself.

[Inaba]

There is no guarantee of serialising of data sent/received so the mining software has to handle it asynchronously and should be able to cope with a few queued messages at once. Most pools currently implementing stratum do give difficulty changes back to back with new work information. The only vardiff pool which currently seems to have its variable difficulty fluctuating wildly is EMC. The others seem to be solid after a very short period of adaptation. I did ask Inaba about hysteresis in his calculations for var diff and he insisted it was there, but the amount of diff changes I see on EMC have been enough for me to not mine stratum there. As far as I can see, this problem occurs on EMC with GBT as well.

In cgminer's case, if the difficulty has risen, it still submits the shares of lower difficulty if it was already work in progress. This ends up creating a little burst of rejects when diff rises. If the difficulty has dropped since the work was done, cgminer retargets to the new difficulty. Regardless, the separation of work and difficulty means there are still issues, but I have tried my best to alleviate the loss to miners in cgminer.

I'm not sure why the raising/lowering of the difficulty is a problem, can you elaborate?   With GBT, the difficulty is tied to the work sent out, so regardless of how often the difficulty changes, as long as you submit the work back and it's under the difficulty it was sent out with, it's accepted.  This is also the way Stratum should handle it, but apparently doesn't, which is a deficiency.

[-ck]

I'm in agreement difficulty not being tied with work on stratum is a problem. However the harm in it is greatly alleviated on other pools which quickly converge to one difficulty target. I think I confused EMC's behaviour with stratum with its GBT behaviour, so ignore that claim about it on GBT, sorry.

[slush]

I have read the discussion about separating target/difficulty and work data. I have to say I think this was a mistake as it opens possibilities for abuse. I think something about these issues should go in the docs. Otherwise someone is going to create a naive implementation with some nasty issues.

Well, after discussion with Kano I slightly changed my mind. As I described few posts ago, with slightly modified meaning of "set difficulty" message, everything should be fixed.

Instead of meaning "stop sending lower difficulty shares than X immediately", set_difficulty should mean "all next jobs are using difficulty X". This still keeps the concept of separated difficulty message from the job itself, it doesn't require sending both messages together, fixes the problem of roundtrips (sending some bad-diff shares by the client because of latency) etc. It also gives the possibility to choose if pool wants to change difficulty in aggressive or in lazy way; by just sending "set difficulty" message without new job with clean_jobs=True, clients will slowly adopt higher difficulty as miners will start using new jobs, but pool can enforce higher difficulty with immediate effect by sending clean_jobs=True, as it is now.

This change in protocol is also backward compatible, because pools using current meaning of set_difficulty don't need to change anything, they just *may* stop sending clean_jobs job, if they want. Because of this, it is win-win solution for me. I described this already, but the idea didn't get any significant attention yet.

Is it ok to pipeline commands to the client? That is, send the difficulty change and new work (2 RPC calls) in quick succession without waiting for a reply to the first.

Yes, commands (notifications) can be pipelined. set_difficulty() and notify() don't expect any response from the client. Just sending one packet with both commands serialized is possible.

[DrHaribo]

Instead of meaning "stop sending lower difficulty shares than X immediately", set_difficulty should mean "all next jobs are using difficulty X".

Nice way to fix it without breaking current implementations. Thanks

[slush]

However to make this change happen, we'll need modified miners. Poclbm already does it in this way, so wink wink, ckolivas ;-).

[-ck]

However to make this change happen, we'll need modified miners. Poclbm already does it in this way, so wink wink, ckolivas ;-).

Who said cgminer doesn't do this already?

[slush]

So cgminer does apply the difficulty only to the future jobs?

[-ck]

So cgminer does apply the difficulty only to the future jobs?

To get the best of both worlds, if difficulty rises, cgminer submits all jobs from the previous stratum base at the previous lower difficulty. If difficulty drops, cgminer submits *all* work at the new lower difficulty. That way the miner can't lose any shares that the pool might accept, at the risk of one or 2 extra rejects.

[kano]

...
These sound like excellent points to bring up when slush publishes the first draft of the stratum BIP.
I've been making a list of possible improvements in stratum myself.

Try fixing GBT first ... you've got a list of MANDATORY changes there already that you're ignoring ...

Stay away from Stratum - you're the last person anyone would want near the better pool protocol.
You'll just fuck it up if anyone lets you change it
(like that crap Stratum pull you tried to fuck cgminer over with)

... at least the problems with Stratum are WAY less severe than GBT.

[slush]

So cgminer does apply the difficulty only to the future jobs?

To get the best of both worlds, if difficulty rises, cgminer submits all jobs from the previous stratum base at the previous lower difficulty. If difficulty drops, cgminer submits *all* work at the new lower difficulty. That way the miner can't lose any shares that the pool might accept, at the risk of one or 2 extra rejects.

Sounds good. So we can safely change the specs and not break anything.

[slush]

Keep calm, guys.

[Zwenny]

Hi,

i have a problem to start the mining_proxy. Everytime when i run the program i get:

Code:

bash$ mining_proxy.py    
2012-11-19 19:19:15,867 INFO proxy jobs.<module> # C extension for midstate not available. Using default implementation instead.
2012-11-19 19:19:15,869 INFO proxy mining_proxy.main # Stratum proxy version: 1.1.2
2012-11-19 19:19:15,869 INFO proxy mining_proxy.main # Trying to connect to Stratum pool at api-stratum.bitcoin.cz:3333
2012-11-19 19:19:15,973 INFO stats stats.print_stats # 1 peers connected, state changed 1 times
2012-11-19 19:19:15,973 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at api-stratum.bitcoin.cz:3333
2012-11-19 19:19:15,973 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs
Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 368, in callback
    self._startRunCallbacks(result)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1101, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/dist-packages/stratum_mining_proxy-1.1.2-py2.7.egg/EGG-INFO/scripts/mining_proxy.py", line 166, in main
    interface=args.getwork_host)
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py", line 476, in listenTCP
    p.startListening()
  File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 969, in startListening
    raise CannotListenError, (self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on 0.0.0.0:8332: [Errno 98] Address already in use.
2012-11-19 19:19:16,034 INFO proxy client_service.handle_event # New job 1584a for prevhash d2e47965, clean_jobs=True
2012-11-19 19:19:31,438 INFO proxy client_service.handle_event # New job 1584b for prevhash d2e47965, clean_jobs=False

[slush]

Hi,

i have a problem to start the mining_proxy. Everytime when i run the program i get:

Code:

Address already in use.

You have another application running on port 8332 (bitcoind?). Please specify alternate port by using "-gp <port>" parameter.

[Zwenny]

Thanks, now it works:-)

[antirack]

Is there any way to enable logging in the proxy? I've tried the following without success:

Code:

$ /home/dev/stratum-mining-proxy/mining_proxy.py -rt -o pool -p port 2>&1 |tee /home/dev/logs/stratum.log

$ /home/dev/stratum-mining-proxy/mining_proxy.py -rt -o pool -p port >> /home/dev/logs/stratum.log

Creates a file with 0 bytes but that's about it. The latter one would still output to the screen, while I don't think it is supposed to.

A Python "problem"?

[slush]

Is there any way to enable logging in the proxy? I've tried the following without success:

Does it print something to the screen, if you remove >> or tee? I don't see how this can be a python problem.

[antirack]

Both are still writing to the screen.

the proxy is up and running for a few weeks if not longer, but I have some random connectivity problems in my miners logs so I wanted to see if I can find anything on the proxy's end during the time it happens.

I have a network monitoring program running for a few days now, with no problems recorded at all. So I am ruling out network problems in my LAN for the moment.


The errors range from 'read timeout' to 'connection refused' (the miner talking to the proxy). And it happens a couple of times a day.

Having the proxy log would definitely be helpful.

[slush]

Hm, proxy is writing to stdout and errout, no idea why redirection does not work for you...