Miner-defined difficulty for stratum protocol

[zephen]

I've been lurking for awhile, and this is my first post, which apparently has to be in a newbie section.

I like the stratum protocol, and I like anonymous pools like eligius.

But with a fast miner, it would be nice not to have to do a start-up dance on the share difficulty, and it would be really nice to be able to dynamically raise or lower difficulty based on adding/removing hardware, without the pool's involvement.

If the miner sets the difficulty higher than the pool would otherwise require, the miner might be artificially increasing his own variance, but that is not a problem of the pool.  If the miner sets the difficulty lower than the pool would otherwise require, then obviously that's a problem for the pool, and the pool can (as it does right now) give the miner a minimum difficulty and ignore shares with a difficulty that is too low.

I know that with some pools that require registration, such as slush's pool, the miner can request a difficulty using out-of-band information, but AFAIK there is no in-band way for the miner to set a minimum difficulty or to explain that the miner's conditions have changed.

There are two parts to this proposal.  The first part requires a mechanism, either upon subscribe or upon some other request from the client, for the pool to give information to the client that the client is allowed to set a minimum difficulty.  I am agnostic about how this is accomplished, but it would be nice if the pool could also describe a target submission rate during the process.

The second part is that, once the client has established that the pool will let it define the difficulty, the client needs to start sending shares of the client's defined difficulty.  Again, I am agnostic about how this should proceed, but the client needs to be able to prove to the pool that it is, in fact, targeting a higher difficulty -- otherwise, the client could cheat.

So, for the second part, one way that this could work is that, when the miner is defining its own difficulty, it adds one or two extra bytes at the start of extranonce2.  The simple existence of these extra two bytes shows that the miner is defining his own difficulty.  A possible encoding for the difficulty would be byte1 * (2 ** byte2), which would let the miner fine-tune his difficulty quite nicely.  He could even change it for every submission.

This part of the proposal makes it impossible for the miner to cheat.  If there are two extra bytes in extranonce2, and the hash difficulty is greater than the user-defined hash difficulty, then the user did, in fact, generate that hash when he was looking for hashes of that difficulty or greater.

And if the miner's submit rate is below the rate that the pool is targeting, then the share should be accepted.

[moszis]

I'm guessing the people that run the pool know more about difficulty requirement for the coin you are mining than a miner.   But I could be wrong..

[zephen]

> I'm guessing the people that run the pool know more about difficulty requirement for the coin you are mining than a miner.

The difficulty for a block is known, and changes approximately every two weeks.

There are very few pool-specific things that the pool knows that miners doesn't know, such as the pool's current bandwith and CPU usage (both in an absolute sense, and as a percentage of the maximum current capability of the pool), and the number of current miners and their current combined hashrate.

Based on these factors, the pool might want to constrain the amount of communication with each miner.  (It also might not want to communicate at all with some miners, but that is a difficult decision to make that will soon be widely known in most cases.)

There are reasons why a pool might want to increase the amount of communication with a miner, as well.  I think they would boil down to wanting to eliminate stale shares, and to insure that the miner is actually attempting useful work.  In most cases, communication every 30 seconds to every couple of minutes should suffice for both these reasons.

So if a miner unilaterally decides he's mining at a hashrate that should give him a reportable hash once every 30 seconds or so, it's hard to imagine why the pool would have a problem with that.  But, if the pool _did_ have a problem with that, there is already a mechanism to let the pool increase the difficulty (reducing the communication rate) for the miner.

[moszis]

Are you saying that the only purpose of Target Diff/Vardiff setup for stratum is to identify how "often" miners respond to server?

I'm not sure if this is what you are looking for, but there is actually a way to allow for user defined difficulty in stratum settings.

[zephen]

> Are you saying that the only purpose of Target Diff/Vardiff setup for stratum is to identify how "often" miners respond to server?

What other reason would there be?  Every returned hash consumes bandwidth and CPU, on both ends.  If it didn't, there would be no reason to bump the difficulty.

> I'm not sure if this is what you are looking for, but there is actually a way to allow for user defined difficulty in stratum settings.

IIUC, the protocol allows the pool to define a minimum difficulty to the miner, and there is an out-of-band indication at some pools that the user can give to request a different minimum difficulty, but it's not actually part of the stratum protocol itself.  I am proposing that the protocol itself incorporate a way for a miner to always mine shares of a higher difficulty, and prove that he is doing so.

[moszis]

Thank you

What about the first part of my question.. just curious.

Lets say you are mining a coin with difficulty of 2(or even less).  The miner is chugging along at difficulty of 1000.  Would it not reduce the pools chances of finding blocks?    Also, wouldn't the number of rejected blocks increase drastically?

[zephen]

> Lets say you are mining a coin with difficulty of 2(or even less).

1) You don't mine "coins".  You mine "blocks".
2) The difficulty of finding a block is set by the network, based on total hash rate, and is currently around 1.5e9 (1.5 US billion).
3) The entire purpose of a mining pool is to reduce the variability to miners by rewarding them more consistently for finding less difficult blocks.  A block that meets the 1.5e9 difficulty also happens to meet a smaller difficulty, such as 2, so if everybody is mining for blocks of difficulty 2, eventually someone will find one that meets the higher difficulty (probably).

> The miner is chugging along at difficulty of 1000.  Would it not reduce the pools chances of finding blocks?

No, it reduces the number of blocks reported by the miner to the pool.  But since the miner is at 1000 and the pool is at 2, the pool will reward the miner 500x a normal difficulty 2 block for every difficulty 1000 block it produces.

> Also, wouldn't the number of rejected blocks increase drastically?

Not at all, as long as the miner keeps current on transactions and timestamps.

[moszis]

Thank you, that's very informative.

Much appreciated.

[moszis]

Actually, one more clarification if you dont mind..

> No, it reduces the number of blocks reported by the miner to the pool.  But since the miner is at 1000 and the pool is at 2, the pool will reward the miner 500x a normal difficulty 2 block for every difficulty 1000 block it produces.
> Not at all, as long as the miner keeps current on transactions and timestamps.

I understand what you are saying there.. but it seems to me that if the coin has a network difficulty of finding a block at 1-2 and you are processing 500-1000 blocks at a time, by the time you send your work back to the server, the block may have already been found half way through your work and new one was released half way through your work.   Which means that you were hashing 250-500 extra blocks for nothing?  It may be just my lack of understanding how this works... 

I guess what i'm not clear on is the frequency of valid blocks generation by the network.  Are they 'generated' after previous valid block was found (x seconds after) or they are generated on set schedule every x seconds, regardless if blocks are found or not?

Thank you again.

[zephen]

> I understand what you are saying there.. but it seems to me that if the coin has a network difficulty of finding a block at 1-2 and you are processing 500-1000 blocks at a time, by the time you send your work back to the server, the block may have already been found half way through your work and new one was released half way through your work.

It doesn't work like that.  Each hash a miner tries either meets some difficulty or it doesn't.  Each has has the same probability as any other hash of meeting that difficulty, so the average block rate the miner will expect to produce will be based on his own hash rate and the difficulty target he is trying to hit.  But there is no such thing as being 50% of the way to producing a block that meets any particular difficulty.  It's just like buying a million or a billion or a trillion lottery tickets a second.  Each one stands alone.

> Which means that you were hashing 250-500 extra blocks for nothing?  It may be just my lack of understanding how this works...

There will always be some stale shares based on the miner finding hashes after the network has accepted a new block.  But if the pool and miner are operating properly, there should never be stale shares based on not keeping up with the clock and the transactions, because the pool should accept oldish work for a bit of time after sending out new transactions and/or time.

But the difficulty the miner is targeting should not affect what percentage of his shares are stale over the long-term.

> I guess what i'm not clear on is the frequency of valid blocks generation by the network.  Are they 'generated' after previous valid block was found (x seconds after) or they are generated on set schedule every x seconds, regardless if blocks are found or not?

A valid block for the block chain is found whenever it is found.  The definition of a valid share is somewhat up to the pool, but should in general be any share generated by a miner that (a) would have been a valid block if it met the network difficulty at the time it was reported to the pool; and (b) did in fact meet the difficulty that the pool required of that miner at that time.

Again, the pools _already_ can set per-miner difficulties, and they do this to cope with bandwidth problems, but there is a spin-up time when the pool might be changing the difficulty for a miner, and might reject shares because they weren't difficult enough, even though the miner had been previously told they _were_ difficult enough.  This proposal lets the miner who knows he has a high hash rate start off at a higher difficulty.

[eleuthria]

EDITED because I wrote most of this forgetting the "main" focus of the question was related to Anonymous pools.


For Anonymous Pools:
A lot of anonymous pools in the past made it possible to request a difficulty with your worker authentication, either through the password field or by appending something to the end of the username.  You can also ask them if they support suggest_difficulty, though I'm not sure of how you actually use that in any mining software.



For Registration Pools:
Most pools already let you set difficulty through their interface, setting a floor to how low your difficulty can go.  At least when implemented like BTC Guild, changing the setting in the Web UI instantly applies your setting to any active connections to stratum servers that you already have with that worker, on top of future connections.



Obviously Variable Difficulty is always going to override your minimum settings if it isn't high enough.

[zephen]

> What is it you're trying to do that isn't already possible through pools that allow you to set your difficulty through Web UI?

Not every pool supports a web UI; if it were part of the protocol then it wouldn't matter.

> At least when implemented like BTC Guild, changing the setting in the Web UI instantly applies your setting to any active connections to stratum servers that you already have with that worker, on top of future connections.

That's nice to know.  Does BTC guild treat this as a minimum difficulty, and never set the difficulty below it?

>  And don't say you want to force it to stay at a certain level, variable difficulty is there for a reason and it would still take over if you're trying to get a difficulty from the pool that is too low for your hash rate.

Obviously.  In fact, quite the opposite -- it would be nice for the miner to be able to specify a hash rate he does not want to mine below (so the miner's final hash rate would be the maximum of his suggestion and the pool's requirement).

In addition to the nicety of the miner being able to set a lower bound on his difficulty, there is also the issue of when the pool wants to dynamically change the difficulty.

Whenever the pool wants to increase the difficulty, you occasionally have the small issue of stale shares that were mined before the miner received the updated difficulty message.  With the proposed protocol enhancement, the pool could accept these older shares, secure in the knowledge that the miner wasn't scamming the pool by holding them back until it received a higher difficulty -- because the difficulty targeted by the miner is actually declared in the new portion of the extranonce.  So with this format enhancement, there is no reason for the pool to ever reject a share with the with the old difficulty after changing difficulties (unless, of course, the miner doesn't adjust its rate within a few seconds or a minute, or whatever time the pool deems reasonable).

Likewise if the pool, for some reason, wanted to decrease the difficulty.  With the current stratum protocol, the pool will accept a share that was mined at the higher previous difficulty, but will probably only credit it at the new lower difficulty, thus potentially cheating the miner (by the difference in the difficulties).  If the difficulty that the miner was targeting when mining is embedded in the share in a way that cannot be forged, then there is no reason for the pool not to credit the share at the rate it was mined at.

EDITED to respond to your edits:


> Obviously Variable Difficulty is always going to override your minimum settings if it isn't high enough.

Absolutely.  And here, as I am trying to explain, the enhancement still helps, because if a pool supports it and a miner is using it, then the pool will never cheat a miner out of any work that he did during the window between the time the pool server decides on a new difficulty and the time the miner has had a chance to propagate that new difficulty to all its mining hardware.

[eleuthria]

> At least when implemented like BTC Guild, changing the setting in the Web UI instantly applies your setting to any active connections to stratum servers that you already have with that worker, on top of future connections.

That's nice to know.  Does BTC guild treat this as a minimum difficulty, and never set the difficulty below it?

Yes.  If you authorize a worker on your connection with a difficulty set higher than what it was already running at, it will immediately increase your difficulty to the new minimum (since stratum difficulty is per-connection).  It will never drop below that setting for the remainder of the session even if vardiff would normally decrease it.