Double geometric method: Hopping-proof, low-variance reward system

[Meni Rosenfeld]

I'm a bit confused about the term "fixed fee" vs average total fee" mentioned in the OP and what they mean. Is the "average total fee" the average fee the pool operator receives over the long term? Or is 'f' the fee the pool operator expects to receive over the long term?

I ask because I notice, for example, a pool that states it has a 1% DGM fee but parameters "f=-0.25, c=0.2, o=0.8". This makes the "average total fee" 0 which makes me think that that isn't the fee the pool is expecting to get on average.

What the pool operator receives over the long term is the average total fee, f+c-fc.

A pool with f=-0.25, c=0.2 indeed has 0% fee. You might be talking about OzCoin; the parameters displayed might be a relic from the time it was 0-fee, I don't know what parameters they actually use now. It's possible they added an additional 1% fee on top of the method formula; that defeats the purpose of having a parameter f for that.

[1714689331]

1714689331

[1714689331]

1714689331

[1714689331]

1714689331

[doublec]

A pool with f=-0.25, c=0.2 indeed has 0% fee. You might be talking about OzCoin; the parameters displayed might be a relic from the time it was 0-fee, I don't know what parameters they actually use now. It's possible they added an additional 1% fee on top of the method formula; that defeats the purpose of having a parameter f for that.

Yes it was OzCoin, and I think Coinotron used the same parameters when it was running DGM. Thanks for the information.

[doublec]

4. If the share found happens to be a valid block, then after doing #3, also do the following for each participant: Give him a payout of (exp(lS-ls)*(r-1)*(1-f))/p. Set lS = lS + log(o).

Is the 'p' used here '1.0/D' or is it 'd/D' (where d is user's difficulty of submitted share). If it's 'd' how is it affected if a user has their difficulty changed part way through a block? Is it the 'd' for the most recent share submission?

[Meni Rosenfeld]

4. If the share found happens to be a valid block, then after doing #3, also do the following for each participant: Give him a payout of (exp(lS-ls)*(r-1)*(1-f))/p. Set lS = lS + log(o).

Is the 'p' used here '1.0/D' or is it 'd/D' (where d is user's difficulty of submitted share). If it's 'd' how is it affected if a user has their difficulty changed part way through a block? Is it the 'd' for the most recent share submission?

It's d/D, and yes, d is for the share that solved the block.

[doublec]

It's d/D, and yes, d is for the share that solved the block.

Thanks, I assume the 'r' used in that calculation should be calculated using a 'p' adjusted for the difficulty of the share that solved the block too?

[Meni Rosenfeld]

It's d/D, and yes, d is for the share that solved the block.

Thanks, I assume the 'r' used in that calculation should be calculated using a 'p' adjusted for the difficulty of the share that solved the block too?

True.

[roy7]

Or, if the total is higher than the block reward (only possible if f<0), the operator pays the difference out of his own funds.

In any case where f>=0, the operator will not be paying out of his own funds, is that accurate? I think I was misunderstanding "operator variance" as meaning the operator may be paying money out of his own funds like PPS in long rounds. Which I'm not willing to do. However, if it is simply variance in how much the operator earns without ever going negative, that's a totally different situation. I wouldn't mind my earnings (as an operator) going all over the place as long as the pool cannot ever go bankrupt in terms of payouts to miners like PPS.

[Meni Rosenfeld]

Or, if the total is higher than the block reward (only possible if f<0), the operator pays the difference out of his own funds.

In any case where f>=0, the operator will not be paying out of his own funds, is that accurate? I think I was misunderstanding "operator variance" as meaning the operator may be paying money out of his own funds like PPS in long rounds. Which I'm not willing to do. However, if it is simply variance in how much the operator earns without ever going negative, that's a totally different situation. I wouldn't mind my earnings (as an operator) going all over the place as long as the pool cannot ever go bankrupt in terms of payouts to miners like PPS.

Correct. But it wasn't a misunderstanding on your part either - if f<0 the operator can indeed pay out of his own pocket.

You can choose f=0 to avoid having to pay out of pocket, but it limits your options for reducing variance and maturity for miners. You could use something like f=0, c=0.01, o=0.99, it turns out similar to PPLNS. in core parameters.

[roy7]

In other news, I've thought of a new framework which includes as special cases double geometric, 0-1 and linear PPLNS, and their extension to operator variance. Some parts of this I've known for months, others for weeks and a key part I've just thought about. I think in particular the extension of linear PPLNS will give the ultimate variance tradeoff (though it's not pure score-based). I'm so excited... And yet I really need to work on Bitcoil now. So much to do, so little time...

Did you ever write this up someplace?

[roy7]

Correct. But it wasn't a misunderstanding on your part either - if f<0 the operator can indeed pay out of his own pocket.

You can choose f=0 to avoid having to pay out of pocket, but it limits your options for reducing variance and maturity for miners. You could use something like f=0, c=0.01, o=0.99, it turns out similar to PPLNS. in core parameters.

So instead of a PPLNS pool with a fixed 3% (f=.03) fee, one could run DGM with f=0, c=.03, o=.97 to have a similar pool with average fee to operator of 3%, but less variance for the miners because the operator is taking on that variance.

I didn't see it explicit in the OP, but does o need to equal 1-c?

I skimmed back over the thread, I thought I saw this discussed before but missed it just now if so, but if you are adjusting the parameters per-miner are there any specific constraints that must be observed? I'm specifically thinking of pools where the fee is based on the difficulty you are mining at. So as the user selectable difficulty rises, the pool's fee for that miner goes down. I wasn't sure if multiple miners all with f=0 but a variable c (and o=1-c?) would pose a problem.

[Meni Rosenfeld]

In other news, I've thought of a new framework which includes as special cases double geometric, 0-1 and linear PPLNS, and their extension to operator variance. Some parts of this I've known for months, others for weeks and a key part I've just thought about. I think in particular the extension of linear PPLNS will give the ultimate variance tradeoff (though it's not pure score-based). I'm so excited... And yet I really need to work on Bitcoil now. So much to do, so little time...

Did you ever write this up someplace?

Yeah, it's discussed in AoBPMRS.

Correct. But it wasn't a misunderstanding on your part either - if f<0 the operator can indeed pay out of his own pocket.

You can choose f=0 to avoid having to pay out of pocket, but it limits your options for reducing variance and maturity for miners. You could use something like f=0, c=0.01, o=0.99, it turns out similar to PPLNS. in core parameters.

So instead of a PPLNS pool with a fixed 3% (f=.03) fee, one could run DGM with f=0, c=.03, o=.97 to have a similar pool with average fee to operator of 3%, but less variance for the miners because the operator is taking on that variance.

Yes.

I didn't see it explicit in the OP, but does o need to equal 1-c?

It doesn't need to be but that generally gives a reasonable tradeoff between variance and maturity time. X in PPLNS is in some ways analogous to approximately c/(1-o).

I skimmed back over the thread, I thought I saw this discussed before but missed it just now if so, but if you are adjusting the parameters per-miner are there any specific constraints that must be observed? I'm specifically thinking of pools where the fee is based on the difficulty you are mining at. So as the user selectable difficulty rises, the pool's fee for that miner goes down. I wasn't sure if multiple miners all with f=0 but a variable c (and o=1-c?) would pose a problem.

It wasn't designed to work with variable c; I think we could make it happen if we change the way some things are parameterized, but it's easier to vary f (make it 0 for the high-difficulty-share miners, and slightly higher for the low-difficulty shares).

[roy7]

4. If the share found happens to be a valid block, then after doing #3, also do the following for each participant: Letting S be his score, give him a payout of (S(r-1)(1-f))/(ps). Set S=S*o. The remaining reward is given to the operator.

Two implementation questions, if you have ever thought about this.

For miners that have a very small payout due, it's best to not pay them at this time and rather let their payout grow higher first. Otherwise they collect too much "dust" and when they try to go spend those coins, the transfer fees eat it all up (or cost more than the inputs, making the dust useless). Is handling this as easy as skipping their payment and leaving their score alone?

I realize even with f=0 future blocks may have more rewards to pay than the block contains. The other alternative is I could "pay" it to a separate dust database (keep the coins for a pool "dust reserve") then once the dust has gotten big enough do a payout from pool funds. Then the DGM scoring system/database isn't effected and knows nothing about it.

If you are paying miners in the coinbase of the block found, you can't wait for confirmation of the block to adjust their scores (or you might pay them multiple times if you find multiple blocks in a row). In a *PPS system you mark the found block as a "payout" for the relevant shares, and if that block later becomes an orphan you mark those share back to unpaid. Piece of cake, conceptually. In DGM, is the way to handle this to go through everyone paid in that block and set S=S/o? The payout in effect never happened, and the only change in database before was S=S*o. Seems simple, but wanted to know if you agreed.

Thanks!

[Meni Rosenfeld]

For miners that have a very small payout due, it's best to not pay them at this time and rather let their payout grow higher first. Otherwise they collect too much "dust" and when they try to go spend those coins, the transfer fees eat it all up (or cost more than the inputs, making the dust useless). Is handling this as easy as skipping their payment and leaving their score alone?

Leaving the score alone won't work.

You could add to the worker's score in a way that compensates for the lost payment, but that is more complicated. It is simpler to keep a record for each miner of the total deferred payment, and when it is high enough make the physical payment via a normal transaction. Deferred payments will be kept in a designated address in your control.

If you are paying miners in the coinbase of the block found, you can't wait for confirmation of the block to adjust their scores (or you might pay them multiple times if you find multiple blocks in a row). In a *PPS system you mark the found block as a "payout" for the relevant shares, and if that block later becomes an orphan you mark those share back to unpaid. Piece of cake, conceptually. In DGM, is the way to handle this to go through everyone paid in that block and set S=S/o? The payout in effect never happened, and the only change in database before was S=S*o. Seems simple, but wanted to know if you agreed.

It's even simpler than that - you don't need to set S=S/o. The score will remain in its reduced state and only the actual payment will be cancelled, and if you pay in the coinbase this happens naturally when the block is orphaned.

I can see why it is intuitive to treat orphaned block as if they never happened - thus cancelling S=S*o - but the method invariant is based on the assumption that a share has a probability of d/D to be a block. If orphans exist this is no longer the case, shares actually have a lower chance of becoming a valid block. If orphans are ignored completely, the operator will pay out more than he should (the block rewards he actually receives are lower than the method assumes). I've done the math and if you simply cancel the payment specified for the orphan blocks, everything is just right. This assumes, however, that every block, once received by the pool, has the same chance to become an orphan.

A tricky part is how to combine this with the dust payments mentioned earlier. You can no longer rely on the network to cancel the orphan payments - you'll need to actually remember for each worker the reward per specific block, and if a block becomes an orphan remove that record yourself.

[roy7]

It's even simpler than that - you don't need to set S=S/o. The score will remain in its reduced state and only the actual payment will be cancelled, and if you pay in the coinbase this happens naturally when the block is orphaned.

What I'm struggling with getting my mind around is that work done before the orphan are at S*o and work done after the orphan is at S. When the orphan goes away and the payment is cancelled, aren't shares after the orphan worth more than the shares before it improperly? (If the orphan never happened and scores were never updated, then all work since last block (before or after the orphan that never happened) are all at S.)

I'm hoping you are indeed correct, since being able to ignore the score changes from the orphan event would be so easy. And you are normally correct.

A tricky part is how to combine this with the dust payments mentioned earlier. You can no longer rely on the network to cancel the orphan payments - you'll need to actually remember for each worker the reward per specific block, and if a block becomes an orphan remove that record yourself.

Dust is an issue for everyone eventually I think (if I'm understanding correctly) since there's no 0-1 cutoff. We are going down by S=S*o every payout the scores for a miner who stops mining will approach zero but never actually get there, they will just have ever decreasing scores. (Eventually so low it's just dust, forever?)

[roy7]

I've done the math and if you simply cancel the payment specified for the orphan blocks, everything is just right. This assumes, however, that every block, once received by the pool, has the same chance to become an orphan.

I think I'll just take your word for it and run with it. As they said in Star Trek II, "Yours is the superior intellect."

[roy7]

a. Periodic rescaling: The only thing that matters is the values of the scores relative to the variable s. Thus, if the values grow to large, all that is needed is to rescale them. This means dividing the scores of all participants by s, and then setting s=1. This should be done once in a while, the exact times do not matter (it can even be done for every share).

I was going to use logs so I didn't need to rescale, but it occurred to me that if I rescale every share I don't need to store s. Since s=s*r and s is always 1 going into the share, s is now always r. Thus, the payout becomes (S(r-1)(1-f))/(pr).

I guess the question is, do I really want to divide every score by r every share (updating hundreds of rows if there are hundreds of workers) vs storing s and using logs. It'd seem logs would be less impact to the database.

Edit: Final question for now. Is there a formula to calculate the total expected future payouts for a miner if they stop mining? That is, given the current state of things like s, r, p, B, and their score, can you approximate "if you stop mining right now the sum of all of your future payments is expected to be X". Basically, the value of their capacitor discharging from the current state down to zero. I think miners might find that useful (in addition to expected payout on next block from S/s), so they have a sort of idea of the long-term trailing reward that offsets the slow startup as the capacitor charged.

Thanks.

[Meni Rosenfeld]

It's even simpler than that - you don't need to set S=S/o. The score will remain in its reduced state and only the actual payment will be cancelled, and if you pay in the coinbase this happens naturally when the block is orphaned.

What I'm struggling with getting my mind around is that work done before the orphan are at S*o and work done after the orphan is at S. When the orphan goes away and the payment is cancelled, aren't shares after the orphan worth more than the shares before it improperly? (If the orphan never happened and scores were never updated, then all work since last block (before or after the orphan that never happened) are all at S.)

I'm hoping you are indeed correct, since being able to ignore the score changes from the orphan event would be so easy. And you are normally correct.

Shares submitted after the orphan will be worth more than those submitted before, and this is proper.

The orphan did happen and this is information that needs to be taken into account. The orphan blocks are not in addition to the valid blocks - they are instead, some of the blocks which should have been valid end up orphans instead. The correct comparison is not orphan vs. no block, but rather orphan vs. valid block.

When a block is found, all past shares have a chance to get a payday if it ends up valid. The cost of this chance at payment is the score reduction - even if, contingently, it ends up an orphan. Later shares never had a chance to profit from the block, and thus don't get their scores reduced for it.

This is similar to the situation with shares in general in a pool - when you submit a share, you are rewarded even if the particular share was of no use, because in finding a share you had a chance to benefit the pool - and you get an expected reward equal to your expected contribution.

Dust is an issue for everyone eventually I think (if I'm understanding correctly) since there's no 0-1 cutoff. We are going down by S=S*o every payout the scores for a miner who stops mining will approach zero but never actually get there, they will just have ever decreasing scores. (Eventually so low it's just dust, forever?)

Right. In practice you could trim scores which are below a very low threshold.

a. Periodic rescaling: The only thing that matters is the values of the scores relative to the variable s. Thus, if the values grow to large, all that is needed is to rescale them. This means dividing the scores of all participants by s, and then setting s=1. This should be done once in a while, the exact times do not matter (it can even be done for every share).

I was going to use logs so I didn't need to rescale, but it occurred to me that if I rescale every share I don't need to store s. Since s=s*r and s is always 1 going into the share, s is now always r. Thus, the payout becomes (S(r-1)(1-f))/(pr).

I guess the question is, do I really want to divide every score by r every share (updating hundreds of rows if there are hundreds of workers) vs storing s and using logs. It'd seem logs would be less impact to the database.

It seems you are suggesting that rescaling on every share is easier than rescaling once in a while, I don't see why.
I do recommend logs, they're less intuitive but more robust and probably less resource intensive.

Edit: Final question for now. Is there a formula to calculate the total expected future payouts for a miner if they stop mining? That is, given the current state of things like s, r, p, B, and their score, can you approximate "if you stop mining right now the sum of all of your future payments is expected to be X". Basically, the value of their capacitor discharging from the current state down to zero. I think miners might find that useful (in addition to expected payout on next block from S/s), so they have a sort of idea of the long-term trailing reward that offsets the slow startup as the capacitor charged.

S/s is not the expected payout from the next block. S/s (or more accurately (1-f)*(1-c)*S/s, accounting for fees) is what you asked about just now - the expected total eventual payout for already submitted shares.

The expected payout for the next block is S/s * (1 - c/(1-o+co)) * (1-f).

[roy7]

Right. In practice you could trim scores which are below a very low threshold.

I think maybe I was paranoid about changing values in the database somehow making it unstable. Given your answer below, I'm thinking if a miner hasn't mined for a bit of time (to not trim brand new low hash rate miners) then do a single manual payout if S/s gets too low, and delete them from database completely.

S/s is not the expected payout from the next block. S/s (or more accurately (1-f)*(1-c)*S/s, accounting for fees) is what you asked about just now - the expected total eventual payout for already submitted shares.

The expected payout for the next block is S/s * (1 - c/(1-o+co)) * (1-f).

*bonks self on head*. Thanks.

[roy7]

TLDR: If you jump down to Edit #2, I figured it out.

The expected payout for the next block is S/s * (1 - c/(1-o+co)) * (1-f).

This is giving me values about half of the actual amount paid in the blocks. To use some real numbers on one specific block, here is my amount paid out formula. I am re-scaling $s to 1 every share. So it is $s=$s*$r in the figures below.

Code:

Actual payout: $score * ($r - 1.0) * (1 - $f) / ($p * $s)
(12.932020560171 * (1.0000315757467 - 1.0) * (1 - 0) / (3.2552316167526E-5 * 1.0000315757467)) = 12.543663867791

If I also calculate the estimated payout for the next block at the same time, I get (leaving out /s since I re-scale to 1 constantly.)

Code:

Estimated payout: $score * (1.0 - $c/(1.0-$o+$c*$o)) * (1.0-$f) 
12.932020560171 * (1.0 - 0.03/(1.0-0.97+0.03*0.97)) * (1.0-0) = 6.36754311846

For S=12.932020560171, r=1.0000315757467, c=.03, f=0.0, o=.97, p=3.2552316167526E-5, s=1.0000315757467. The estimated equation reduces to 12.932020560171 * 0.4923857868 which is why it's about half. (With re-scaling every share, the payouts are extremely similar to the actual score value itself.)

I think I have the math coded properly. So either I misunderstand what "expected payout for next block" means or I have a blatant bug in how I'm coding the formula. Assuming formula is correct.

(To try and get a handle on how DGM operates, I'm using my share database to recreate "what would have happened" with DGM from start to finish. Since I'm only doing private testing, my test data is limited with about a week of mining and 3 blocks found, on an alt coin. Maybe the sample size is too small. 99K shares recorded with average block difficulty 36K, the block difficulty changes every block.)

Edit: Just occurred to me. Is the expected payout for next block given above if the miner stopped mining? So on average, D shares from now the payout to the miner would of course be much lower than the numbers I'm seeing (since the miner in my data sample kept mining away).

Edit #2: Yes that's it. I should have worked on it more before posting. Sorry for all the noise. It's spot on.

I injected an extra 41000 shares of work on a 41000 difficulty block into my database, and then injected a new "found share".

Before these changes, up to the final point in time for a worker (not the one listed above):
"Expected remaining total payout 6.1845766103505, next block pays 3.0451976203249"

After my artificial data injection,

Block Found: (score 3.1393789900256) pays out: 3.0451255771104

Nailed it within .00008. And I bet that's because there actually were a few thousand shares in the database after the last real block I left alone (so it was a slightly long round I created). So if someone stops mining, I can tell them their expected remaining total payout and their expected payout in the next block too. Awesome.

[Meni Rosenfeld]

Edit: Just occurred to me. Is the expected payout for next block given above if the miner stopped mining? So on average, D shares from now the payout to the miner would of course be much lower than the numbers I'm seeing (since the miner in my data sample kept mining away).

Yes, I should have clarified that, sorry.

If you want to assume he keeps mining, there are two factors to consider - his current score and his future proportion of the pool's hashrate. Taking them both into account requires some calculations. It's easier to just show what his reward will be if a block is found right now.