-ck
Legendary
Offline
Activity: 4214
Merit: 1644
Ruu \o/
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 18, 2012, 11:52:58 PM |
|
There is no guarantee of serialising of data sent/received so the mining software has to handle it asynchronously and should be able to cope with a few queued messages at once. Most pools currently implementing stratum do give difficulty changes back to back with new work information. The only vardiff pool which currently seems to have its variable difficulty fluctuating wildly is EMC. The others seem to be solid after a very short period of adaptation. I did ask Inaba about hysteresis in his calculations for var diff and he insisted it was there, but the amount of diff changes I see on EMC have been enough for me to not mine stratum there. As far as I can see, this problem occurs on EMC with GBT as well.
In cgminer's case, if the difficulty has risen, it still submits the shares of lower difficulty if it was already work in progress. This ends up creating a little burst of rejects when diff rises. If the difficulty has dropped since the work was done, cgminer retargets to the new difficulty. Regardless, the separation of work and difficulty means there are still issues, but I have tried my best to alleviate the loss to miners in cgminer.
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
Luke-Jr
Legendary
Offline
Activity: 2576
Merit: 1186
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 18, 2012, 11:54:33 PM |
|
I have read the discussion about separating target/difficulty and work data. I have to say I think this was a mistake as it opens possibilities for abuse. I think something about these issues should go in the docs. Otherwise someone is going to create a naive implementation with some nasty issues.
Issue #1: As Eleuthria mentioned miners can hold back proofs of work. Keep results that are above the target, and submit them later if the difficulty decreases. Keep results that are far below the target. Submit them later if difficulty increases, getting better pay for them. By varying your hashrate between two Stratum pools you can make the two pools raise and lower your difficulty frequently. This enables a kind of Stratum pool hopping.
You can prevent this on the server side by always following the change of difficulty with sending out new work and flushing old work. Effectively negating the separation of diff/target and work data.
Issue #2: As Kano mentioned an evil pool op can steal from you. When the miner sends in a hash that is below target but not too far below it, simply reply with "1. I'm raising the difficulty. 2. your share is rejected due to the new difficulty". This will happen naturally once in a while as there is a race condition inherent in the protocol. But as a pool op you can make it happen on purpose to steal from miners. Then reduce difficulty again, and repeat. Raising difficulty once in a while will also make the miner throw away work it was just about to send in. Lowering the difficulty again could make up for it if the miner has kept old low difficulty shares. But oops, the server is flushing old work to prevent issue #1, so they are no longer valid.
With getwork and getblocktemplate a reject due to the hash being above target means there is a bug in your miner. With Stratum it means "it could be nothing - but maybe the pool op is stealing from you."
While issue #1 is fixable, issue #2 can't be fixed 100% without changing the protocol. The best you can do is to always follow up every difficulty change with quick new work data that flush the old. This should reduce the frequency of the problem. Then just hope the miner won't tell the user too often that you are stealing.
Is it ok to pipeline commands to the client? That is, send the difficulty change and new work (2 RPC calls) in quick succession without waiting for a reply to the first.
That should at least reduce the amount of visible rejects that cause miners to think you are stealing. However it won't fix it 100% and having to flush old work to change the difficulty will also waste some hashpower. These sound like excellent points to bring up when slush publishes the first draft of the stratum BIP. I've been making a list of possible improvements in stratum myself.
|
|
|
|
Inaba
Legendary
Offline
Activity: 1260
Merit: 1000
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 03:57:45 AM |
|
There is no guarantee of serialising of data sent/received so the mining software has to handle it asynchronously and should be able to cope with a few queued messages at once. Most pools currently implementing stratum do give difficulty changes back to back with new work information. The only vardiff pool which currently seems to have its variable difficulty fluctuating wildly is EMC. The others seem to be solid after a very short period of adaptation. I did ask Inaba about hysteresis in his calculations for var diff and he insisted it was there, but the amount of diff changes I see on EMC have been enough for me to not mine stratum there. As far as I can see, this problem occurs on EMC with GBT as well.
In cgminer's case, if the difficulty has risen, it still submits the shares of lower difficulty if it was already work in progress. This ends up creating a little burst of rejects when diff rises. If the difficulty has dropped since the work was done, cgminer retargets to the new difficulty. Regardless, the separation of work and difficulty means there are still issues, but I have tried my best to alleviate the loss to miners in cgminer.
I'm not sure why the raising/lowering of the difficulty is a problem, can you elaborate? With GBT, the difficulty is tied to the work sent out, so regardless of how often the difficulty changes, as long as you submit the work back and it's under the difficulty it was sent out with, it's accepted. This is also the way Stratum should handle it, but apparently doesn't, which is a deficiency.
|
If you're searching these lines for a point, you've probably missed it. There was never anything there in the first place.
|
|
|
-ck
Legendary
Offline
Activity: 4214
Merit: 1644
Ruu \o/
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 04:10:09 AM |
|
I'm in agreement difficulty not being tied with work on stratum is a problem. However the harm in it is greatly alleviated on other pools which quickly converge to one difficulty target. I think I confused EMC's behaviour with stratum with its GBT behaviour, so ignore that claim about it on GBT, sorry.
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 06:51:16 AM |
|
I have read the discussion about separating target/difficulty and work data. I have to say I think this was a mistake as it opens possibilities for abuse. I think something about these issues should go in the docs. Otherwise someone is going to create a naive implementation with some nasty issues.
Well, after discussion with Kano I slightly changed my mind. As I described few posts ago, with slightly modified meaning of "set difficulty" message, everything should be fixed. Instead of meaning "stop sending lower difficulty shares than X immediately", set_difficulty should mean "all next jobs are using difficulty X". This still keeps the concept of separated difficulty message from the job itself, it doesn't require sending both messages together, fixes the problem of roundtrips (sending some bad-diff shares by the client because of latency) etc. It also gives the possibility to choose if pool wants to change difficulty in aggressive or in lazy way; by just sending "set difficulty" message without new job with clean_jobs=True, clients will slowly adopt higher difficulty as miners will start using new jobs, but pool can enforce higher difficulty with immediate effect by sending clean_jobs=True, as it is now. This change in protocol is also backward compatible, because pools using current meaning of set_difficulty don't need to change anything, they just *may* stop sending clean_jobs job, if they want. Because of this, it is win-win solution for me. I described this already, but the idea didn't get any significant attention yet. Is it ok to pipeline commands to the client? That is, send the difficulty change and new work (2 RPC calls) in quick succession without waiting for a reply to the first.
Yes, commands (notifications) can be pipelined. set_difficulty() and notify() don't expect any response from the client. Just sending one packet with both commands serialized is possible.
|
|
|
|
DrHaribo
Legendary
Offline
Activity: 2730
Merit: 1034
Needs more jiggawatts
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 07:00:19 AM |
|
Instead of meaning "stop sending lower difficulty shares than X immediately", set_difficulty should mean "all next jobs are using difficulty X".
Nice way to fix it without breaking current implementations. Thanks ![Smiley](https://bitcointalk.org/Smileys/default/smiley.gif)
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 07:22:02 AM |
|
However to make this change happen, we'll need modified miners. Poclbm already does it in this way, so wink wink, ckolivas ;-).
|
|
|
|
-ck
Legendary
Offline
Activity: 4214
Merit: 1644
Ruu \o/
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 07:54:32 AM |
|
However to make this change happen, we'll need modified miners. Poclbm already does it in this way, so wink wink, ckolivas ;-).
Who said cgminer doesn't do this already?
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 08:39:49 AM |
|
So cgminer does apply the difficulty only to the future jobs?
|
|
|
|
-ck
Legendary
Offline
Activity: 4214
Merit: 1644
Ruu \o/
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 09:27:44 AM |
|
So cgminer does apply the difficulty only to the future jobs?
To get the best of both worlds, if difficulty rises, cgminer submits all jobs from the previous stratum base at the previous lower difficulty. If difficulty drops, cgminer submits *all* work at the new lower difficulty. That way the miner can't lose any shares that the pool might accept, at the risk of one or 2 extra rejects.
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
kano
Legendary
Offline
Activity: 4592
Merit: 1851
Linux since 1997 RedHat 4
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 09:49:15 AM |
|
... These sound like excellent points to bring up when slush publishes the first draft of the stratum BIP. I've been making a list of possible improvements in stratum myself.
Try fixing GBT first ... you've got a list of MANDATORY changes there already that you're ignoring ... Stay away from Stratum - you're the last person anyone would want near the better pool protocol. You'll just fuck it up if anyone lets you change it (like that crap Stratum pull you tried to fuck cgminer over with) ... at least the problems with Stratum are WAY less severe than GBT.
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 10:17:22 AM |
|
So cgminer does apply the difficulty only to the future jobs?
To get the best of both worlds, if difficulty rises, cgminer submits all jobs from the previous stratum base at the previous lower difficulty. If difficulty drops, cgminer submits *all* work at the new lower difficulty. That way the miner can't lose any shares that the pool might accept, at the risk of one or 2 extra rejects. Sounds good. So we can safely change the specs and not break anything.
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 10:18:21 AM |
|
Keep calm, guys.
|
|
|
|
Zwenny
Newbie
Offline
Activity: 52
Merit: 0
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 06:22:19 PM |
|
Hi, i have a problem to start the mining_proxy. Everytime when i run the program i get: bash$ mining_proxy.py 2012-11-19 19:19:15,867 INFO proxy jobs.<module> # C extension for midstate not available. Using default implementation instead. 2012-11-19 19:19:15,869 INFO proxy mining_proxy.main # Stratum proxy version: 1.1.2 2012-11-19 19:19:15,869 INFO proxy mining_proxy.main # Trying to connect to Stratum pool at api-stratum.bitcoin.cz:3333 2012-11-19 19:19:15,973 INFO stats stats.print_stats # 1 peers connected, state changed 1 times 2012-11-19 19:19:15,973 INFO proxy mining_proxy.on_connect # Connected to Stratum pool at api-stratum.bitcoin.cz:3333 2012-11-19 19:19:15,973 INFO proxy mining_proxy.on_connect # Subscribing for mining jobs Unhandled error in Deferred: Unhandled Error Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 368, in callback self._startRunCallbacks(result) File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 464, in _startRunCallbacks self._runCallbacks() File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 551, in _runCallbacks current.result = callback(current.result, *args, **kw) File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1101, in gotResult _inlineCallbacks(r, g, deferred) --- <exception caught here> --- File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks result = g.send(result) File "/usr/local/lib/python2.7/dist-packages/stratum_mining_proxy-1.1.2-py2.7.egg/EGG-INFO/scripts/mining_proxy.py", line 166, in main interface=args.getwork_host) File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/posixbase.py", line 476, in listenTCP p.startListening() File "/usr/local/lib/python2.7/dist-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/tcp.py", line 969, in startListening raise CannotListenError, (self.interface, self.port, le) twisted.internet.error.CannotListenError: Couldn't listen on 0.0.0.0:8332: [Errno 98] Address already in use. 2012-11-19 19:19:16,034 INFO proxy client_service.handle_event # New job 1584a for prevhash d2e47965, clean_jobs=True 2012-11-19 19:19:31,438 INFO proxy client_service.handle_event # New job 1584b for prevhash d2e47965, clean_jobs=False
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 19, 2012, 06:29:34 PM |
|
Hi,
i have a problem to start the mining_proxy. Everytime when i run the program i get:
You have another application running on port 8332 (bitcoind?). Please specify alternate port by using "-gp <port>" parameter.
|
|
|
|
Zwenny
Newbie
Offline
Activity: 52
Merit: 0
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 20, 2012, 01:43:46 PM |
|
Thanks, now it works:-)
|
|
|
|
antirack
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 21, 2012, 09:53:41 AM |
|
Is there any way to enable logging in the proxy? I've tried the following without success: $ /home/dev/stratum-mining-proxy/mining_proxy.py -rt -o pool -p port 2>&1 |tee /home/dev/logs/stratum.log
$ /home/dev/stratum-mining-proxy/mining_proxy.py -rt -o pool -p port >> /home/dev/logs/stratum.log
Creates a file with 0 bytes but that's about it. The latter one would still output to the screen, while I don't think it is supposed to. A Python "problem"?
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 21, 2012, 09:55:27 AM |
|
Is there any way to enable logging in the proxy? I've tried the following without success:
Does it print something to the screen, if you remove >> or tee? I don't see how this can be a python problem.
|
|
|
|
antirack
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 21, 2012, 03:01:56 PM |
|
Both are still writing to the screen.
the proxy is up and running for a few weeks if not longer, but I have some random connectivity problems in my miners logs so I wanted to see if I can find anything on the proxy's end during the time it happens.
I have a network monitoring program running for a few days now, with no problems recorded at all. So I am ruling out network problems in my LAN for the moment.
The errors range from 'read timeout' to 'connection refused' (the miner talking to the proxy). And it happens a couple of times a day.
Having the proxy log would definitely be helpful.
|
|
|
|
slush (OP)
Legendary
Offline
Activity: 1386
Merit: 1097
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
November 21, 2012, 03:05:15 PM |
|
Hm, proxy is writing to stdout and errout, no idea why redirection does not work for you...
|
|
|
|
|