Ok, fair enough. Maybe Con can chime in then and tell me what the difference is between EMC and Slush's pool as viewed by CGMiner? All this is ultimately due to CGMiner apparently having trouble with EMC because the difficulty changes too rapidly - but if that's the case, wouldn't the same problem be evident on Slush's pool? If not, why not? I'll be glad to fix it if there's a problem somewhere.
Actually, I wasn't comparing you to slush's pool because ironically... slush's pool doesn't do vardiff yet. ozcoin and btcguild are the other pools I can compare to. Make what you want of that fact.
So the issue really comes down to the round trip of network time between pool sending out diff and miner receiving it, and the miner sending out a share. Most of us have round trips of ~500ms across real internets, though some of you are seeing ping times of <50ms but this does not translate to a processing, submission and return time of 50ms. Luckily c is fast so cgminer intrinsically doesn't add *that* much processing time, but it does add time.
There are 4 distinctly different scenarios with changing diff separate from work.
A. Diff rises before cgminer submits lower diff share and cgminer is aware of it
B. Diff falls before cgminer submits now adequate diff share and cgminer is aware of it
C. Diff rises in the round-trip window period as cgminer has already submitted a share below difficulty and cgminer is not aware of the new diff
D. Diff falls in the round-trip window period as cgminer has discarded a share that would now be adequate as cgminer is not aware of the new diff
cgminer can only do something about scenario A and B. C and D are entirely out of its control. The way I decided to manage these situations is based on maximising the possible paid share submission.
Scenario A:
cgminer can decide to retarget for the higher difficulty and not submit the share. It currently does NOT do this, instead keeping the difficulty target at the original lower value, in case pools have a grace period at the lower difficulty. It risks rejects but no shares are lost in this way. This is what causes more rejects on EMC when the difficulty is retargeted often. However no paid work is lost here.
Scenario B:
cgminer can decide to retarget for the lower difficulty and submit the share. cgminer does this since clearly this share is now work it will get paid for. No paid work is lost here, and no rejects are induced.
Scenario C:
cgminer can't do anything about it, but this registers as a reject. Any increase of diff by a pool can elicit this, and the more often the difficulty is raised, the more rejects will be seen. Once again, no paid work is lost here.
Scenario D:
cgminer can't do anything about it, and this work is lost. Any decrease of diff by a pool can elicit this, and the more often the difficulty is lowered, the more shares are lost. No rejects are induced.
So there is a round-trip window period for scenario D where shares are potentially lost. If you have a 500ms round trip window, then you lose .5 seconds worth of possible lower difficulty shares with each drop. The more decreases, the more lost.
The only changes I can make to cgminer's default behaviour to help here is scenario A, and retarget to the higher difficulty. Note that the emphasis is always on submitting shares that are possibly paid for over minimising rejects. B is fine. Nothing can be done about the rejects in C, but no paid shares are lost. D is the real issue that nothing can be done about, and unless difficulty is tied to the work, it ends up being dependent on how often the pool ends up changing difficulty that determines how much work is lost, and secondarily the visible rejects.
I agree that retargeting with vardiff is important or floods of low diff shares will be a real issue. However retargeting for small changes in difficulty based on share submission, which is subject to large amounts of variance, will see lots of changes of difficulty and the inherent rise in rejects and lost shares.
EDIT: Note that my scenario A management is in line with slush's change to the protocol stipulating changes to diff apply to next work sent by server, not existing work.
