The fact that it isn't part of the work definition in your protocol is what creates the issues.
I can agree that it creates the issue in your implementation. It don't create any issue in other miners or in my proxy, which has support of the same protocol.
It's a separate, global per worker, independent piece of information according to your protocol.
Yes, it is independent information from job definition. As I said previously, there's no need for bundling it with job definition, it can be freely used independently.
Basically you are defining work that you will reject - and that you must reject, since the work returned cannot prove the difficulty that it was processed at - work difficulty is not encoded anywhere in the hash either (you left it out of the hash to gain performance ...)
Basically - you're right. In some edge cases, like connecting 2THash miner with difficulty 1 to the pool, it may happen that miner will waste first few seconds of the work. However I'm thinking about "mining.propose_difficulty" command, where miner will be able to propose some minimum difficulty in connection instead of "default" 1, so with proper implementation no waste will happen.
but has connectivity problems, and during that time they were sent a difficulty increase, they will lose work that was valid at the lower difficulty, bit not at the new difficulty. Late submission of work is not handled by the protocol in this case.
This is TCP, not HTTP. You cannot "lose" part of the data transmitted over the socket. You'll receive retransmissions of TCP packets (so you'll receive difficulty adjustment) or the socket will be dropped and the "late submission" won't be accepted in any way. So it's no argument here.
A difficulty change does indeed mean throwing away work that was valid prior to receiving the difficulty change ...
Every sane pool implementation will send standard retargeting command together with new job (and clean_jobs=True). So miner have to drop previous jobs anyway, so no work lost here.
since the work is missing the difficulty information at both the pool and the miner.
We're repeating here. You don't need to know target difficulty when you're creating work in miner, right?
The time from starting work, to it being confirmed, by the pool is quite long ... it includes the network delay from the miner to the pool and back ... which when hashing at 54GH/s using an ASIC device, is certainly the slowest part of the whole process, not the mining.
Not I probably don't understand your point (language barrier). How is network latency during share submission related to difficulty change?
This also means that even during normal connectivity, work will often be in transit already when a difficulty change is received
Most common case - yes. But not necessary, there're use cases when sending it separately has some advantage, as I described above.
You must already keep information valid per worker: the difficulty ... as well as a bunch more: like who they are and where they are, that must be looked at in order to send the work out.
You simply add the work difficulty to the information sent - rather than send it separately.
Your code MUST already process through levels to get from the job definition to sending it to a worker.
Why MUST? You didn't tell me the real reason why it MUST be known on the beginning of the job. Except that threading issue, which is implementation specific and can be solved easily.
Why Per worker? Difficulty is related to the connection, not per worker. Btw bundling difficulty to job definition don't change it either.
As far as I can say, me and m0mchil implemented the same thing already (me in proxy, m0mchil in poclbm) and we don't have such issues at all. All this sounds to me that your complains are related to the miner implementation.
... and suggesting that a software 'change' is a reason to not implement something is really bad form
From the architecture view, keeping it separately is much cleaner. Maybe some things lead to ugly hacks in some implementation, but optimizing *protocol* for some implementation *is* bad example of doing software architecture.
Adding a small amount of information per worker is a negligible hit on the pool software since the pool must already have that information per worker and it is simply added to the message, not a regeneration of the message.
Again, I don't understand your point here. Of course that pool knows difficulty per connection. But what is this related to your issue in retargeting? I was saying about fact that creating of *jobs* is pool-wide, which is different story.
I was looking for reasons and stating why I wanted them - I had heard none reasonable so far at that point
I think I'm giving you a lot of reasonable points. I'm just aware that you are too much focused to implementation in cgminer.
No it's not trickier in cgminer.
It's a performance hit due to making something global for all worker's work, yet the value can change at any time, it's not an attribute of the work according to the pool, yet in reality it indeed is.
How the hell can be lookup for single value (read only!) became a "performance bottleneck"?
Basic database 101 - 3rd normal form - 2 phase commit - and all that jazz
Don't teach me over database design, please ;-).
I'm all fine with this. Just make additional filtering of difficulty when you're sending shares to the network and I'll be happy :-).
It's simply a case of any miner that isn't brain dead and does use threading properly (like any good software has used for a VERY long time
) to deal with work properly, has a locking issue dealing with the fact that with testing the validity of a share, the test definition can unknowingly change before the test starts (the pool sends the difficulty change) and the change can be known during the test, but before, the test completes (arrival of the difficulty change) and thus the result is no longer true (which will also not be rare when a difficulty change arrives)
It forces a global thread lock on access to the work difficulty information - since it is global information - you can't put it in the work details since the pool doesn't do that either.
Do you know term "over-optimization"? It seems to be this case. You can safely forget about the case of race condition when miner will receive new difficulty *exactly* at the same time when some thread is checking share validity. Nothing serious will happen if you send one, two or ten of low-diff shares in this case.
It forces a global thread lock on access to the work difficulty information
This catched my eyes. You don't have write-only locks in C/C++? :-P
It's discussing why the protocol should or shouldn't include the difficulty as part of the work information.
I hope I explained this clearly already. Job definition and difficulty are logically two separate things (although maybe it don't look like in your implementation). Short summary: Job may change without difficulty change, difficulty may change without job change. And job is the same for all miners on the poo, but difficulty is defined per-connection. Is this enough for not bundling them together?
However, I will also add, that this part of the protocol definition seems to be directly aimed at helping the pool (but in reality very little performance gain)
Yeah, this is valid point! Of course it is helping pool, I'm not hiding it! But I'm strongly against saying "this feature is helping pool" and "this another feature is helping miners". Both pool and miners have the same goal - have the highest real hashrate, lowest resource consumption and highest block reward. Period. There's no real reason to fight miners against pool ops. When the protocol gives tools to pool ops to drive resource consumption in some nice way, then miners will benefit from this also - by faster replies of server, lower stale rate and potentially by lower fees, because pool don't need to handle ugly peeks in performance like it is nowadays.
at the expense of the miner losing shares unnecessarily sometimes.
This may happen only on some pool implementations and only in some edge cases. Losing jobs is definitely not "by design" and expected.
