Bitcoin Forum

This is a new reward system for mining pools, a hybrid between PPLNS (https://bitcointalk.org/index.php?topic=39832) and the geometric method (https://bitcointalk.org/index.php?topic=4787.0) which combines advantages of both. It starts with PPLNS-like low share-based variance without operator risk, and then allows the operator to absorb some variance to decrease pool-based variance (which the geometric method can do, but not PPLNS). It is of course hopping-proof - the expected payout per share is always the same no matter when it was submitted. The variance and the maturity time (the time it takes to receive the reward) are independent of the pool's history and almost completely independent of future difficulty changes.

The method is purely score-based, which means that all the information required to calculate payouts can be encoded with a single score value per participant. There is no fundamental need to keep a history of shares. However, because the scores grow exponentially, it is advised to use a logarithmic scale to store their values and do the calculations.

We will denote by B the block reward (assumed constant) and p = 1/Difficulty. In addition there are 3 parameters which can be adjusted to balance average fee, operator variance, share- and pool-based participant variance, and maturity time:
f - Fixed fee.
c - Average variable fee. The average total fee will be (c+f-cf)B per block. Increasing c reduces participants' variance but increases operator's variance.
o - Cross-round leakage. Increasing o reduces participants' share-based variance but increases maturity time. When o=0 this becomes the geometric method. When o->1 this becomes a variant of PPLNS, with exponential decay instead of 0-1 cutoff (note that "exponential" does not mean "rapid", the decay can be chosen to be slow). For o=1, c must be 0 and r (defined below) can be chosen freely instead of being given by a formula.

The method is as follows:
1. When the pool first starts running, initialize s=1. Initialize the scores of all workers to 0.
2. Set r = 1 + p(1-c)(1-o)/c. If at any point the difficulty changes, p and then r should be recalculated.
3. When a share is found, increase by p*s*B the score of the participant who found it. Set s=s*r.
4. If the share found happens to be a valid block, then after doing #3, also do the following for each participant: Letting S be his score, give him a payout of (S(r-1)(1-f))/(ps). Set S=S*o. The remaining reward is given to the operator. Or, if the total is higher than the block reward (only possible if f<0), the operator pays the difference out of his own funds.

The inutition is this: Instead of keeping the score unchanged when a block is found (as in PPLNS) or setting all scores to 0 and effectively transferring them to the operator (as in the geometric method), a part of the score is transferred to the operator. When rounds are long, participants get to keep most of their score between rounds and this is similar to PPLNS. However, if several blocks are found in rapid succession, the operator will collect a large portion of the score and thus be the primary beneficiary of the good fortune. The fees collected this way allow letting f be negative, sweetening the rewards of long rounds. Overall, this decreases the dependence of participants' rewards on the pool's luck, thus reducing the variance caused by it.

The variance of the payout for a single submitted share is

(1-c)^4(1-o)(1-p)p^2(1-f)^2B^2
------------------------------
   (2-c+co)c+(1-c)^2(1-o)p

Notably, the factor of (1-o) allows this to be much smaller than the variance of the geometric method, if o is chosen close to 1. This value is of relevance, though, only to small miners, and the potential advantage of this system is for large miners (which suffer from pool-based variance but not so much from share-based variance). It would be of interest to evaluate the total variance for a participant constituting a given portion of the pool, and the variance of the operator. So far I was unable to derive symbolic expressions for these, but they can be evaluated in simulation. For c = 0.5, o = 1-c = 0.5, f = (-c)/(1-c) = -1 I got that a miner constituting the entirety of the pool has about 30% of solo variance (instead of 100% as in PPLNS), and the operator's variance is about 25% of PPS variance.

The geometric method was so called because shares decay in a geometric sequence, and its analysis crucially depends on summation of geometric series and the fact that round length follows the geometric distribution. Because in this new method shares decay geometrically along two directions, both for every share found and for every block found, I call it the double geometric method.

The variables used in the calculations grow rapidly, so if the method is implemented naively, they will overflow the data types used. Thus the implementation should use one of the following:

a. Periodic rescaling: The only thing that matters is the values of the scores relative to the variable s. Thus, if the values grow to large, all that is needed is to rescale them. This means dividing the scores of all participants by s, and then setting s=1. This should be done once in a while, the exact times do not matter (it can even be done for every share).

b. Logarithmic scale: Instead of maintaining the variables themselves, their logarithms are stored and updated. The following is the method expressed in logarithmic scale, denoting by log the natural logarithm function and by exp the natural exponentiation function:
1. When the pool first starts running, initialize ls=0. For every worker define a variable lS and initialize it to negative infinity (or a negative number of very large magnitude, say -1000000), and do this also for every worker which later joins.
2. Set r = 1 + p(1-c)(1-o)/c, lr = log(r). If at any point the difficulty changes, p, r and lr should be recalculated.
3. When a share is found, let lS = ls + log(exp(lS-ls) + pB) for the participant who found it. Set ls = ls + lr.
4. If the share found happens to be a valid block, then after doing #3, also do the following for each participant: Give him a payout of (exp(lS-ls)*(r-1)*(1-f))/p. Set lS = lS + log(o).

For display purposes, the quantity that should be shown as the score of a worker is S/s (or exp(lS-ls) in logarithmic scale). This represents the expected payout the worker should receive in addition to any confirmed rewards (before fees). To display the expected payout after deducting fees, use (1-f)*(1-c)*S/s, or (1-f)*(1-c)*exp(lS-ls).

How about just break the block down into 10 minute intervals and thus divide the block total (e.g. 50) by the number of intervals and that fraction divided up over the shares during that interval.
(or even use the network block times themselves)

Anyone want to do the math? :)

Does it solve the supposed 43.5% issue mathematically?

I don't think so. It is an interesting proposition though and it made me think some.

The issue is still that it's always better to mine at a pool with fewer intervals, all things being equal, because your shares will still be worth more than they are at a pool with more intervals, but a lower hash rate (so you would get a higher per interval payout).

Not to mention that a 10 minute interval at a place like deepbit is basically an entire round. While at a small pool 10 minute intervals are very few shares.

don't break rounds, merge 3 rounds and pay 150btc

Then it's more profitable to mine at the beginning of an interval.
Actually it will create interesting dynamics depending on the number of hoppers. But it's still more profitable to mine early in the round.

It's more profitable to mine in the beginning of the 3-round batch, or after 1 or 2 blocks were found if there are relatively few shares.


Anyway, this is off-topic, hopping-proof methods have been known for months, no need to reinvent the square wheel. This post is about a breakthrough in low-variance reward systems.

How does this deal with variable or unknown block rewards? Already now block rewards are rarely 50 BTC and the endgame scenario is anyways random block rewards (which should just be profitable).

Some comparisons between Geometric, PPLNS and PPS:

Share decay (If I stop mining, how will my expected balance decay):
Geometric: Exponential
PPLNS: Linear
PPS: None

Maturity time (How long after I submit a share will I be able to claim the full reward for this share):
Geometric: after block was found + calculations for payout have finished (+120 confirmations eventually)
PPLNS: N more shares have been submitted after my share (+eventually some confirmations)
PPS: Instant (or until a block has been confirmed, though a PPS pool should already start with a buffer!)

Payout estimates:
Geometric: Only estimates, will only be correct if the next submitted share solves the block - otherwise decay
PPLNS: Can only give "baseline" + probabilities for additional earnings (first baseline = 0 BTC/share, after a block was found [blockreward/N] BTC/share etc.) - gets more exact over time the more shares are being submitted after my share
PPS: exact pre-known price per share

Pool can go in minus if there are some long rounds:
Geometric: As far as I understood it - yes
PPLNS: Never
PPS: Biggest threat to true PPS actually

Your system wants to be somewhere in between Geometric and PPLNS.
How is maturity time affected? How exact and how "stable" are payout estimates displayed on a pool website (if I stop mining, how fast do shares decay)? How far can the pool in theory go into minus to "absorb some variance"?

Oh, and it might be great to have a google spreadsheet with example calculations + chart porn! ;)

It doesn't. For now I recommend (for all reward systems) that the operator keeps transaction fees, and take this into account in his business plan when he decides what pool fees to take. I'll investigate this more thoroughly as it becomes a bigger problem.

EDIT: As discussed below, this turns out not to be a serious problem. Though variability in block rewards does increase operator risk.

You should look not at the time until all the reward is obtained, but rather at the average time it is received. So if your reward is 2 BTC, and you receive 1 BTC a million shares from now and 1 BTC two million shares from now, I count it as a maturity time of 1.5M shares. If you consider time to be fully repaid important for reasons of making a clean break from the pool, I have a different suggestion to address this which I will post about later.

When o=1, you can choose r to have whatever tradeoff you want between maturity time and variance. It turns out you get the same tradeoff as with 0-1 cutoff. A good choice is r=1+p, resulting in decay by a factor of e every D (difficulty) shares, with maturity time of 1 (expressed in multiples of D) and variance of (pB)^2/2. This is the same as with PPLNS when N = 2D. Then you can decrease o (and change r accordingly), keeping maturity time constant while decreasing variance of participants (at the cost of increased operator variance).

The decay rate, and the accuracy of the estimates (which is basically the inverse variance), are tunable, and as discussed above, start as good as PPLNS and can then be improved.

The operator can only lose if f is negative, and in this case, can lose up to (-f)B per block found. Compare with PPS, which is equivalent to f -> -Inf and the operator's loss per round is unbounded.

I'll think about it.

In order simulate efficiency and the variability of efficiency for this scoring method, what f c and o values would you suggest I use?

Edit: For use in the simulator I define 'efficiency' to be (reward for a hopping miner)/(reward for a fee-free pps miner).

Use f = -1, c = 0.5, o = 0.5. With these parameters this method is the most different from other methods, so if there are any problems they should show up there. Also, with this the average fee is 0, so efficiency will be exactly 1.

Remember that to simulate/implement this, you need to use either logarithmic scale, or periodic rescaling (once in a while (which could be every share if you're only keeping track of a few workers), divide all scores by s and set s to 1).

Thanks Meni, and thanks for the reminder about log scale.

As far as efficiency goes, it's the variation in efficiency I want to simulate, which I think would really be the stand out reason to use this method over other hopper proof methods.

By this, do you mean simply the variance of the reward? In this case you need parameters that are realistic with respect to the risk operators are willing to take, but for an operator who is already seriously considering offering PPS, I think these parameters are entirely viable (perhaps with a slightly higher f if he wants to gain on average).

It wont really be the statistical parameter 'variance' I'm recording in simulation, just the variation in the reward for this system compared to baseline pps, run for a number of rounds equivalent to days or weeks. But yes, it would record something similar to variance, so I think the parameters you suggest will be useful. The plots generated would look similar to the graph below.
https://i.imgur.com/rfqUH.png

On second thought, it is trivial to deal with variable block rewards - at least, in a framework such as this which allows operator variance, and as far as hopping-proofness and expectation is concerned. Analysis of things like variance is harder though. I've changed the wording to be more friendly to incorporating this - basically the score given to a share should be based on the block reward at the time of submitting it.


In other news, I've thought of a new framework which includes as special cases double geometric, 0-1 and linear PPLNS, and their extension to operator variance. Some parts of this I've known for months, others for weeks and a key part I've just thought about. I think in particular the extension of linear PPLNS will give the ultimate variance tradeoff (though it's not pure score-based). I'm so excited... And yet I really need to work on Bitcoil now. So much to do, so little time...

Is your geo method and this double geo method essentially PPLNS but N is the number of shares in the last x hours?

Can you please graph the decay rate?

Absolutely not.

Time never plays a factor in a hopping-proof method.

To clarify the picture, all these methods fall into a spectrum with two axes:

1. Decay function: 0-1 (step function), exponential, linear etc.
2. Block finding effect: No effect, resetting the round completely, or something in between.

PPLNS is 0-1 with no effect for block finding. Geom is exponential with complete round reset. Double geom is exponential with a partial effect of block finding, the magnitude of which is controlled with a parameter.

It's exponential decay, you can see a graph of how that looks like here (http://www.wolframalpha.com/input/?i=e^%28-x%29+x+from+0+to+4). The exact decay rate depends on the parameters. With double geom it's more complicated because there is also decay when blocks are found, but that's also exponential in nature.

This is a very interesting method. But there are two aspects I am unsure about:

1. The reward is assumed constant. After some time fewer and fewer coins will be minted and transaction fees will be a larger part of the income. Eventually the only income is transaction fees. At some point variable income will be a must.

Perhaps you can just take the payout from the formula today, divide it by 50 and multiple with the actual income.

2. Proofs of work must be handled sequentially. This could impede performance. It could also make it difficult to have multiple backend servers spread around the world for redundancy.

Most of the analysis assumes constant block reward, but the method doesn't require it. When you calculate the score for a share, you use the block reward in this share (viewed as a potential block). This guarantees fair expected return, but adds some variance to the operator.

It needs to be the potential income when the share was submitted, not the received income when a block is found.

I don't know much about the technicalities of this, but from an algorithmic standpoint, having shares go slightly out of sync only causes a negligible error in the payouts.

This was my conclusion as well when coding the backends for EMC.  However, there are ways to account for that by storing shares and calculating them according to time submitted.  Unless your hashrate is in the multiple TH/s range, accommodating multiple back ends is not too difficult.

I'am looking for a way to determine the estimated block reward B when a share (which is not solving the block) was submitted. Is that even possible?

A possible solution i have in mind: we increase the user score for every submitted share on block x as soon as block x was solved and we know exactly how much B was.

Is that's the way it should be done?

That's exactly the thing I said shouldn't be done. The correct way is much simpler, it's to use the block reward of the share itself. That is, the share is a hash of a block header you hand out for the getwork, which has a merkle root of a transaction list with a given total transaction fee, which together with generation is the total block reward. That's the value of B which should be used for this share.

Ozcoin Pooled Mining now offers DGM  ;D

Thanks Meni  :D

Cool. :) Good luck.

So ... how does this fare with someone who only mines 18 hours a day?

On average you get exactly your fair reward, with a relatively low variance.

"Fair reward" is a meaningless statement as you well know.

But these 'geometric' methods are affected by start/stop times to deal with hoppers.
Basically the whole reason they exist.

So it has NO negative effect at all on someone who stops and starts mining?
Or are you just using that vague "Fair" word to mean your definition of "Fair" is that yes you will get less BTC because that's fair?

Could you be a little clearer? It's easier to show you where you're wrong if I can follow what you mean.

To start with, what does
mean?

By "fair reward" I mean pB per share, where p = 1/difficulty at the time the share was submitted and B is the block reward at the time the share was submitted. This is what you'd get with PPS (disregarding fees).

If you use a mining reward calculator to find what you'll get per day for mining with your hashrate, then for mining 18 hours per day you'll get on average exactly 3/4 of that amount - the reward is proportional to the work you put in. If you mine for a minute a day you'll get on average 1/1440 (1440 minutes in a day) of what you'd get for mining the whole day with the same hashrate, though your variance will be high (though not nearly as high as with high-variance methods like Geometric).

This indicates a misunderstanding of how these methods work. DGM doesn't care at all when you started or stopped mining. It doesn't try to detect hoppers and punish them. It rewards each share separately, in a way that is independent of the past and thus does not offer any incentive to hop.

Correct.

I think you'll find that my interpretation of "fair" as described above is reasonable.

Some of the confusion I think I'm seeing with DGM is the lag time between cause and effect.  Most people seem to have figured it out, but occasionally there will be some new people or what have you that don't understand that what you do right now won't necessarily affect your payout for another 4 - 7 blocks.

For example, on EMC, if you stop mining right now, your Proportional Differential will be higher than if you had kept mining for the current block.  However, 4 - 7 blocks down the road, your prop differential is going to start taking a bit of a dive until it evens out the increase you saw on the block you stopped mining on, then it will start picking back up to neutral territory.

It makes for some confusing cause and effect if you aren't prepared for it or understand what's going on. 

I like to think of it as a capacitor. If you have a light bulb and capacitor in parallel connected to a DC source, the bulb won't be lit right away. As the capacitor charges more current is routed through the bulb and it becomes brighter, until it reaches its maximum brightness as if there was no capacitor. If the power source is disconnected, the bulb gradually fades as the capacitor discharges its stored energy through the bulb. No energy is lost.

Of course there are many other real-life examples of this dynamic. Electric radiative space heaters come to mind - you turn them on and only after a while they become hot and heat the room. The dynamic is the same also quantitatively - in all 3 cases it can be modeled with (dx / dt) = a - bx, where a becomes 0 when the source is discontinued, and where x is itself the derivative of some quantity of interest.

This comment confused me until I realized you meant that you stop mining and resume sometime later.

Yes, a capacitor is a good analogy, though I think the heater is a better layman description for people to get the concept.  I think I'll use that going forward. 

Sorry about being confusing, I actually thought to myself I should clarify that but then promptly forgot to do so before posting. But yes, I did mean to imply that you resumed mining at some point.

Sorry - they were also meant to be "?" like the 2 lines after them :)

Don't forget to poll every DGM pool to find which is currently paying the most per share and hop to that site till the value drops on the next block they find...

Edit: If it wasn't obvious, this was me explaining how DGM is hoppable.

Yes, it was obvious that you do not understand how DGM works. DGM is hopping-proof, the expected payout per share is constant.

I don't see how you explain DGM is hoppable - can you explain exactly which/what pools list an amount they are paying per share?  There's exactly two DGM pools that I know of and neither of them pay a set amount per share.

Presumably he means checking the parameters of each pool and its current state and calculate the reward from this... Which leads us back to the the expected reward being completely independent of the current state, since it depends only on the future and not the past.

Hmm will be interesting to see my total payout after I stop mining Ozcoin some time in the future (also after I get all the after amounts)
Coz I joined them a few days ago when their luck was showing high, but I guess that doesn't mean I get more ...

http://ozco.in/content/share-payout-comparison-bitcoin

It seems to me if a pool has a long enough run of luck, you join in and benefit while the value slowly works its way back down as the luck turns.

By the way, I'm not saying this disadvantages the long term miners in any way...

Yep Hopping doesn't have to mean your gain is someone else's loss.
Maybe many think those two ideas are directly connected - but obviously in this case - they aren't.
What I guess DGM does is allow you to capitalise on the fact that when a pool is lucky, that luck extends beyond the time when it occurs
Thus you can get part of that luck after the fact.

In the capacitor analogy, every miner has a capacitor which is charged as he submits shares, and its charge level affects how much he gets per future block found. The pool at large does not have a capacitor which charges as blocks are found and affects the reward for future shares submitted. The pool's history has no effect whatsoever on the rewards for future shares.

The pool's luck in the past has no effect on your rewards. The pool's luck only affects you if your capacitor is charged at the time.

No. If you read the method's description you'll see that finding a block has two effects:
1. Giving out rewards to miners who currently have a score
2. Decrease the score of miners who currently have a score
It has no effect whatsoever on future miners (or on the rewards of current miners for shares they will submit in the future - only those that they submitted in the past).

Yes it does. With a given total hashrate, a pool's average reward is constant. If one miner gets on average more than his fair share, it means another gets less. In a hopping-proof method like DGM everyone gets their fair share on average.

Absolutely not. The luck extends backward in time, not forward. You can't get part of the luck after the fact.

To be completely honest, I'm growing weary of having to repeat the same thing over and over again.

Thanks very much for taking the time to explain. I misunderstood completely where the capacitor was. Sent you a little something for your troubles and work on developing this pay scheme.

Thanks, much appreciated!

How is that true? When you hop, the bitcoin pie does not end up getting bigger. But your piece does grow (hence why you hop), meaning that at least one person's piece has to shrink.

Pool hopping at a proportional pool does mean that other lose. There's only 50btc per round, and if you are getting more than expected of that 50btc, then someone else is getting less.

The first two charts here (https://bitcointalk.org/index.php?topic=47411.msg614637#msg614637) should give you an indication of just how much fulltime miners lose at a proportional pool if there are pool hoppers.

If by 'pool hopper' you mean someone without a share payment maximisation strategy -  who bounces from pool to pool or round to round without regard to how many shares have been submitted for the round or what kind of scoring system is being used - then I think that 'intermittent miners' is a better name since 'pool hopper' has come to mean someone who has a maximisation strategy based on using pools with broken payout systems.

DGM is working very well for us, miners are happy and well paid, pool has grown considerably.

Thanks again Meni

Graeme

You're welcome, I'm glad you're enjoying it.

OK here's some obviously flawed logic of my own but I'm curious where the flaws actually are:

In a 0% fee standard proportional pool, each person is paid the same amount per share when the block is found - I'll call it just "Prop"

In a 0% fee PPS pool each person is paid the same amount per share (during a 2 week difficulty) no matter how many blocks are found - I'll call it just "PPS"

In a 0% fee DGM pool each person is paid, well, you work it out :) - I'll call it just "DGM" - but it is apparently the same as PPS in the long run

If I were to mine Prop for 129 minutes and then mine PPS for 171 minutes with X GH/s where Prop has an average 300 minutes per block for 2 weeks, then I would receive an amount that is higher than mining PPS constantly (or Prop constantly)
Of course it wouldn't be 171 minutes PPS every block, but it would average out to that.
This is called hopping (I've got that right?)

i.e. although I would get paid the same per share that I mined as anyone else who mined on Prop, that is considered "unfair" because although it is quite logical that each share should be worth the same since you spent the same amount of effort on it as everyone else with the same hardware setup, I still get more than expected for my X MH/s due to the mathematics involved.
"fair" seems to be that if you leave early your shares are worth less per share than someone who stays, since this stops you from getting more than your expected payout for your mining X GH/s

So yeah the first part seems odd that you should get paid less per share than everyone else - to make it "fair" so that you don't actually get more than the expected amount for your X GH/s

Logic fails, maths wins.

Now, no matter how you word it, when someone is mining PPS they get the same amount.
When someone is mining Prop they get different amounts that adds up to more than the PPS rate ... otherwise there would be no point hopping.

If on the other hand, they did the 129 minutes on DGM and 171 minutes on PPS they would still get the same amount from PPS.
However, DGM resolving: "hopping getting extra", means they MUST get less than mining Prop thus they must get less per share than mining Prop, thus they must get less per share than everyone else mining DGM.

How does DGM ensure people who hop will get less per share than the people who don't hop?
Why doesn't that happen to anyone who doesn't mine 24/7 but say instead mines 2/3 of each day? Or does it and that process of mining only 18 hours a day means they would get more on Prop than they should, if not for DGM?

Yeah that's actually some questions also ... if anyone feels like explaining that so that others who use DGM and didn't do university level statistics can understand, that would be good :)
(and also point out the flaws in what I said)

I'll let Meni do the real 'splainin but DGM pays fairly based on contribution, not like PPS really.  It's more like a 100% fair prop with no hoppers.  DGM pays more on short rounds and less on long rounds (except on EMC, where I have a bit of a kicker for longer rounds that comes out of my own pocket, so the pool pays more than 50 BTC on long rounds, slightly less than 50 BTC on short rounds.  I think after a final tally, I've paid a few BTC out of my pocket recently due to the longer rounds) when compared to PPS (on a per share basis).

Hoppers don't get paid less than non hoppers, they just don't get paid more... bit of a subtle difference, but it's really the essence of a fair pool reward system.  DGM does not "punish" hoppers, which seems to be a common misconception.  It just pays hoppers for the work they put in compared to everyone else.  

Hmm unless I'm wrong in what I wrote ... to repeat: hoppers do get paid less per share than non-hoppers.
The problem is that if they get paid the same per share (when nothing intervenes with their hopping) then they get more than expected.

That's where the maths wins logic loses bit applies.

Well, I suppose that depends on how you define it.  Later shares are worth more than earlier shares (in a PPS sense), so in that sense, then yes they get paid less, but then their contribution to the block is also less.

@kano: Your error is that you do not properly consider the effect of round length on how much shares are worth.

In a proportional pool, shares in a short round are worth more than in a long round. If you look at the intra-round level then yes, hoppers get per share the same as anyone else in the round. But the trick is that hoppers get more of their shares in short rounds than normal miners, thus profit more per share.

So to be fair in a proportional system you need not only for all shares in a round to be rewarded equally, you also need each share to have the same chance to go into a short round as other shares. And this simply doesn't hold true for hoppers, so it's unfair.

To give an analogy: At a soup kitchen people are randomly assigned to a line where potatoes are handed out or a line where meat (which is superior) is handed out. Suppose someone sneaks into the meat line although he was assigned to the potato line. He gets the same food as everyone else in the line - but it's unfair because he cheated his way into the better line. (Though this example will only make sense to people who understand that true randomness is fair).

In DGM the concept of rounds isn't rigidly defined. If a hopper tries to hop a DGM pool as if it was proportional then yes, in every "round" he'd get per share on average less than normal miners. But this is only because the artificial division to rounds is flawed. What matters is the reward per share on the global level. If you must, you can say that hoppers get more of their shares in the shorter, more lucrative rounds, but in those rounds get less per share than other shares in the round. The two effects cancel each other to result in average payout per share equal to (1-f)pB, no matter the mining pattern.


And I disagree vehemently with your sentiment that "maths wins logic loses". If a mathematical result seems illogical the art did not fail you, it is you who failed the art.

I'm not trying to give you a new definition of the word fair - I'll use your one for this:

And you didn't actually point out what was wrong with what I said, you instead gave a different reason why you call anti-hopping fair.

I guess that means what I said is actually correct?

OK I'll put it this way:

If I use the same times I gave and mine 129 minutes Prop and then 171 minutes PPS on a Prop that takes 300 minutes to get a block
(on average for all those 3 numbers) and lets say I get 10 shares a minute and did this for 100 blocks.
(all easy numbers)
Then on average I will mine 12,900,000 Prop shares and 17,100,000 PPS shares.

For block 1 mined on Prop each of my shares will be paid the same as everyone else's shares.
For block 2 mined on Prop each of my shares will be paid the same as everyone else's shares (but this number may be different to block 1)
all the way up to block 100.

For block 1 mined on PPS each of my shares will be paid the same as everyone else's shares on PPS.
For block 2 mined on PPS each of my shares will be paid the same as everyone else's shares on PPS.
all the way up to block 100.

Logically that makes sense (and that's the reason Prop pools existed in the first place)

Mathematically it is unfair since over those 100 blocks I will get more BTC than if I only mined Prop or PPS.
The extra BTC is of course gained from Prop

Maths wins logic fails.

On average, I will get the same amount of BTC from the PPS pool if I mine Prop as the hop pool or I mine DGM instead.

If I mine DGM instead I will get less BTC since I will lose my unfair hop advantage.

The ONLY way that is possible is if my average amount paid per non-PPS share drops.

If it drops, then I am being paid less per non-PPS share than the people who are mining 24/7

Again that last statement logically seems unfair, but it isn't. Again Logic fails and maths wins.

Edit: corrected some mixed up wording at the end

"Seems" is the key word, to a flawed logic it might appear unfair, correct logic will show it's fair.

I already answered your questions. You basically repeated what you said - I'll spare us both repeating what I said.

But you might want to read up on Simpson's paradox (http://en.wikipedia.org/wiki/Simpson%27s_paradox).

Kano, have a read of  this (http://hoppersden.info/entries/20-How-to-hop-5-Back-to-basics),  this (http://hoppersden.info/entries/25-How-to-hop-6-Basic-probability-for-bitcoin-miners), and then this (http://hoppersden.info/entries/24-How-to-hop-7-Expected-share-values)  and I'll be happy to answer any questions you have on the 'How to hop' thread.

This looks cool.

Yes I do realise that this is your definition of fair, and it is actually what you are referring to.
I also realise that it is logically correct when you consider the maths ...

But it is the point that most people won't see that without the maths and why I say Logic loses Maths wins.

Though I guess that's also follows why you say:

I'd also call using the word "Fair" as failing the art since that word is purely subjective.

Again, the reason why Prop pools exist in the first place is missing that which you seem to suggest is Logically obvious ...

Your "unfair" mining is in different words:
Hoppers hedging their bets by aiming for higher value shares.

They are being "unfair" coz they don't hang around for the bad, they just aim for the good.

Please note that next time you buy any stock-exchange shares at a lower price and sell them at a higher price ... that you are effectively doing the same thing ... being "unfair" :)
(and make sure you don't ever buy BTC, and later sell it at a price higher than you bought it ...)

The only non-subjective problem with hoppers (and the reason why people have no reason to complain about anti-hopping techniques) is that if everyone who used the Prop pool hopped (and of course you cannot guarantee to aim for good without doing it from the start), the pool would die on the the first block that reached 43% of the difficulty in shares and that would be the end of it and none would get paid anything.

You could almost apply the same stock exchange analogy here also, you must trade shares and lose money in the companies you are involved in during the good and bad times ... wait? what? That's not called "fair" :D
Damn unfair stock exchanges ...

At the moment on P2Pool people might be gaining better than expected by using cgminer to reduce their reject rate ...
Should people be using this "unfair" word there also - and should the pool be penalising people using cgminer's to make it "fair"?

I agree that dwelling too much on fair/unfair is a red herring. But I'd say people have enough of their perceived meaning of "fair" in common that it does carry some information content, and useful in the conversation.

Fairness is relative to the intended purpose of the mechanism in question and what people can reasonably expect from it. Stock exchanges are known to be a mechanism for people who have beliefs about future price changes of stocks to profit from these beliefs (or lose if the beliefs are wrong). There is thus no unfairness in doing just that.

But pooled mining is a mechanism to allow miners to join forces to reduce individual variance. Miners expect their average reward to be equal to solo mining (minus fees). Given that, a strategy that increases your own average reward at the expense of others is unfair.

This argument doesn't hold any water and you're not the first to suggest it.
1. Using cgminer increases your expected payment per share you find, but it does not reduce other miners' expected payment per share they find. It also causes no ill-effects if done by everyone.
2. A pool is a give-and-take relationship. You give shares which could end up a valid block and thus benefit the pool. In return you get payment for these shares. When hopping you contribute the same to the pool but receive more in return (at the expense of others). Using cgminer or any other similar technique increases the total quantity of shares you submit and benefit the pool, but you get the same payment for each.
3. Mining exists to secure the Bitcoin network. Using cgminer increases the total amount of valid proof-of-work you submit to the network, and warrants a corresponding increase in your reward. Hopping does not increase your contribution to network security.

If 2 people are hashing at the same rate, one with 9% rejected (as per forrestv says is expected) and the other with cgminer on 0% then of course one has an advantage.
One will have, on average, a higher ratio of shares per block than the other.
Thus they will get paid more per block than the other due to having more shares accepted.
Are you suggesting this extra payment appears out of thin air?

Where does this extra come from? The others of course who will get a reduced payment because of having mined fewer shares than the other.
That's all pretty simple maths there ...
(P2Pool is PPLNS)

"It also causes no ill-effects if done by everyone." Yep - but that's got nothing to do with the discussion ... since people accept that 9% is OK ... or are you suggesting that forrestv has caused the problem by saying everyone should expect 9%?

Also: "but you get the same payment for each" - well you've already argued that that statement is not relevant on it's own - so I'm not sure why you are using it yourself ... :D

3) Yep but cgminer increases it at the disadvantage of others also.
You can't get more per block out of thin air - sorry :P

The extra comes from being able to find more valid blocks for the pool. Let's say each share has 1/100 chance of being a block. Miners A and B have the same hashrate and find 1 share per second. B uses cgminer, A doesn't.

Miner A: Each share has 91% to be unrejected, and hence 0.91% chance to be a valid block which increases the pool's total rewards. Miner A's payment for this share is on average 0.91% of block reward.
Miner B: Each share has 100% to be unrejected, and hence 1% chance to be a valid block which increases the pool's total rewards. Miner B's payment for this share is on average 1% of block reward.

It's just like if a miner adds or removes mining hardware. He changes his contribution to the pool, and accordingly his reward.

All this assumes there isn't something very strange with the whole p2pool/cgminer issue I'm missing.

You can get more blocks if you avoid wasting your hashes on something that will be invalid.

Unlike this quote from me a few months ago, when you physically add mining capability (or lower your rejects - they are the exact same conceptually), the bitcoin hashing pie really does grow. People still get what they deserve based on what proportion of the network they are. When you hop, you grow your reward per share submitted, which MUST mean other people's rewards per share shrink, right?

My head hurts...  :D

Yep I got that one wrong big time :)
Oh well.

I guess it's just forrestv being evil and telling people with bad reject ratios, that can be fixed, it's OK :)

The gain is out of thin air :D
(Well it sort of is coz it was just wasted hashes before)

So I was thinking about DGM the other day coz I've swapped back to Ozcoin for a while ...

and I was thinking about how it compares to a modified version of PPS
... or more specifically what I guess is SMPPS though I'm not sure but what I mean is:
a PPS that pays (50/difficulty)*0.985 per share i.e. with a 1.5% pay back to the payout fund (to cover orphans)
and the pool effectively has outstanding debt that is paid later when the fund is less than the amount due to be paid at any payout time

Anyway is there a reason to use DGM instead of an appropriate modified version of PPS?
Since I'm thinking that the point of DGM is to match some modified PPS scheme, why not just use the much simpler scheme it is trying to match and have everyone be able to calculate their expected payments?

This is indeed very similar to SMPPS. SMPPS is broken as I've already written about at length. The short story is that when the debt grows large (which will happen with 100% probability), miners will have greatly delayed payments and thus move on to a different pool, causing the collapse of this one.

Even under the unrealistic assumption that miners will not leave, they will just suffer unbounded maturity time. The purpose of a reward method is to balance variance, fees and maturity time, not to try eliminating any two for an extreme increase in the third.

Other than orphans (as I said with the 98.5% with 1.5% back into the fund that pays) what else will make the 'SMPPS' pool's expected balance average less than zero (plus some initial balance to help startup) over time?

again I'm only referring to SMPPS for a base of discussion but using the description I mentioned above - in case there is any difference

Not expected balance average. The balance it will reach eventually.

When a miner decides if to mine in a pool, his maturity time is affected by the balance now, not by the expected balance in the future. It takes just one point in time when the balance is low enough to cause a collapse.

If the fee exactly matches the losses (so the operator doesn't get anything on average), this is Brownian motion and it is guaranteed to eventually be low enough, no matter how low is "low enough".

If there is an additional fee which stays within the pool, the negative balance likely reached is inversely proportional to the extra. It will still cause large maturity time. The accumulated funds that will help the balance remain positive could have instead been handed out to miners.

Also, this method is hoppable. Mining when the balance is high results in reduced maturity time, which means continuous miners will suffer from more than their fair share of increased maturity time.

You didn't specify what your method is. All *SMPPS match your description, the difference between them is how to distribute new funds.

Hey Meni!

Coinotron's BTC pool rewards are now calculated using DGM :)

Great, enjoy! (And watch out for the high risk with your chosen parameters.)

But that collapse is the same with DGM also.
Simple example, last week or the week before DeepBit has 26 blocks with an average across those 26 of 200% shares.
DGM would also have the same problem of the balance getting low enough to cause a collapse.

DGM doesn't have a balance. It is hopping-proof. Bad luck will cause reduced payouts for past shares but will not affect future shares.

If the parameters are too aggressive the operator has a risk of losing his reserves, but this risk can be kept arbitrarily low with a choice of parameters.

Also, assuming the operator keeps a cashout reserve (which he should), operator bankruptcy causes no direct losses to miners.

OK I don't get that one.

Since the amount it pays is not exactly the block amount each time a block is found, it must have a balance.

When it pays above the block amount (bad luck) that amount is obviously not covered by the block so must come from a 'balance' stored somewhere.
Then when the pool gets a short block and pays less than the block amount, that is effectively increasing the balance.

I'm simply looking at the payouts on Ozcoin - so unless they are doing it completely wrong, there must be a balance by the fact that the block amount doesn't match the payout amount.

It's the operator's reserve. In bad luck the operator pays from his reserve, in good luck he builds up his reserve.

You could call this "balance" if you insist but the difference from SMPPS is that the current reserve has no effect on the payment for future shares. That is, the reserve is not a miner-facing property and having a low reserve does not mean miners are better off elsewhere (especially since even in the event of bankruptcy, an honest operator will offer PPS cashout before discontinuing the pool).

Heh - that sounds like just giving it a different name for the sake of making it sound better :)

If the pool doesn't have the BTC to make DGM payments during bad luck, it's not gonna appear from no where - the payments will be late :P

As for bankruptcy payout - the main problem I'd see there from DGM vs PPS (or some of the different PPS) is that no one can probably work out the owed amount on DGM since there is a future component to that, that depends on the future also.
On PPS it's pretty simple.

Maybe. It's an important distinction and using a different name could prevent confusion.

The payments are not delayed on any case. Either they are paid on time or the pool declares bankruptcy and shuts down. In practice, the operator may be willing to liquidate personal assets and add it to the pool's reserves to keep it running. With parameters like in EMC the risk is minimal. Ozcoin is more aggressive, cointron even more so.

You can calculate the expected future payment for a given score. (Actually, you can parametrize it so that the expected payout is the score). The operator should pay that out so that miners won't suffer (I'm assuming a deterministic payout is considered better than a variable payout of the same expectation). The total score of all miners is bounded, and the operator should have this as an additional reserve - he should declare bankruptcy before he becomes unable to offer a cashout.

Seem to be more miners asking questions about DGM.
Thought I would bump this a bit so its easier for them to find  ;)

If I'm shopping for a pool to join, how do I use the information on a pools page to calculate my expected payout?
If a pool is hiding information that would make comparison possible, then why should I join their pool when they are not being transparent enough for me to be comfortable?
I've been parametrying alot and I just scratch my head. :D

You shouldn't join a pool which hides information. Fortunately, the only pools doing that are the ones you shouldn't be joining anyway.

Your expected payout is the same in every hopping-proof pool with 0 fee (assuming the same number of invalids), and it is equal to p*B per share.

At a guess (and seeing a number of conversations about it) he's probably concerned about variance vs maturity time. It would be nice if pools posted these values based on their chosen parameters. It would be a point of differentiation for them, anyway.

I thought the other pools were too.
We have since we moved to DGM shown
Double Geometric Variables
f=-0.25, c=0.2, o=0.8
in our forum thread 1st post and on Ozcoin's homepage
are there others we should be showing?

I see, I interpreted "expected payout" literally. I use the term "performance metrics" for the entire range of payout details.

One problem is that I don't know of a simple calculation that will give the variance and maturity time as a function of the parameters. Also, there is more than one kind of variance. If there is demand for this I'll try to work on some approximations or charts or something. For now there's the variance for a single share in the OP, it's relevant for very small or intermittent miners, and as you can see it's a mouthful.

I do think that's a great start, but it occurred to me that miners just want to know how the variable choice will affect them. Many miners will confuse expectation with variance and maturity time. Having the variables listed is good, no doubt, but for a naive miner it will be difficult to understand the relevance.


Thinking about it some more, even listing the single share variance wouldn't be a very good indicator for a general miner. Charted data from a simulation might be a better bet, but I'm not sure how you'd select a set of exemplars that would cover all (or most) situations of interest to a miner. For example, would you need to have separate large and small miner simulation results? Plus, each pool would have to run their own simulation to illustrate what effects their variable choices have on the performance metrics of the examplars selected.

It should be possible to create a dataset by running for various parameters, and then interpolating for all other values. Then each pool can show values or graphs for their chosen parameters, and allow the user to set his hashrate with a slider to his how specific results. I have reason to believe that the variance is affine in the miner's hashrate (at least approximately), so that simplifies things. If the interpolation can be done easily enough, maybe the pool can also give sliders for seeing the results for various DGM parameters.

By the way, in AoBPMRS I mentioned that it's not strictly necessary that all miners in a pool will have the same parameters. So it could be possible for miners to choose their own parameters based on their desired performance metrics.

Just thought I'd mention that you seem to be discussing exactly what I was referring to here ...

That's not really what I meant. I don't want to see how parameters affect the 'owed amount' , but rather the variance in payout and  maturity time. If the score is calculated to represent the 'owed amount', then a graphical illustration of the effects of changing parameters wouldn't be necessary. Unless I'm not following you?

The important parameters being f, c, o, and miner hashrate as a proportion of pool hashrate? Let me know if I missed anything.

I didn't see that mentioned explicitly - is it implied?

is this assuming a "stable " hashrate through the round
with proxy miners, gpumax miners and some leftover hoppers using us as backup etc coming and going our hashrate can vary significantly within 1 round
eg
https://ozcoin.net/content/hashrate

I think you need to assume a miner's hashrate is a constant proportion of the pool's hashrate, excepting when variance in share submission is a variable of interest.

so miner average hashrate and pool average hashrate over the round
to be able to  make that assumption - cheers :)

Expected payout is one thing, variance is another, maturity time is another. Expected payout is the most important and easiest to calculate, and it's safe to refer to it as the amount owed.

A measure of the miner's intermittency would also be good to have.
 
It's fairly explicit in section 7.4, "Heterogeneous pools".

Variability in pool hashrate is in a sense equivalent to variability in miner hashrate, which I think should be included. One issue is that the specific form of variability will be different, but perhaps it is possible to encode it approximately with a single value.

Lots of miners moving around atm, questions on DGM being asked
time for a bump to make this thread easier to find :D

What changes should be made to the DGM calculation in the presence of miners that are mining at different difficulty levels? For example, a pool that dynamically adjusts difficulty based on the users hashrate, or one that allows the user to select their own target value?

It's more or less the same, but instead of p=1/D you should use throughout p=d/D, where d is the difficulty of the current share submitted.

Pardon my ignorance to this issue but I was looking over your initial post and I was trying to understand why with a variable share difficulty that p would turn into d/D? Wouldn't it be 1/d? I guess I don't understand where D comes into play with variable share difficulty.

p is the probability that a share will be a block. If the global difficulty is D and the share difficulty is the standard 1 then p=1/D. If the share difficulty is d then p=d/D.

Ah, so it should have always been d/D? Where d was always 1 until now?

Exactly.

Sweet explanation... thanks Meni.

(some code snippet...)

Did I handle the var diff right? in linear model. p is always 1/D.

A small but important correction: The last line should be
(Extra diff in the denominator.)

In addition:

1. The function "estimate" seems to return the reward that should be given to the user for a block just founds, but the name seems to suggest something else.

2. Make sure the r used in the reward calculation is based on the current p.

I'm a bit confused about the term "fixed fee" vs average total fee" mentioned in the OP and what they mean. Is the "average total fee" the average fee the pool operator receives over the long term? Or is 'f' the fee the pool operator expects to receive over the long term?

I ask because I notice, for example, a pool that states it has a 1% DGM fee but parameters "f=-0.25, c=0.2, o=0.8". This makes the "average total fee" 0 which makes me think that that isn't the fee the pool is expecting to get on average.

What the pool operator receives over the long term is the average total fee, f+c-fc.

A pool with f=-0.25, c=0.2 indeed has 0% fee. You might be talking about OzCoin; the parameters displayed might be a relic from the time it was 0-fee, I don't know what parameters they actually use now. It's possible they added an additional 1% fee on top of the method formula; that defeats the purpose of having a parameter f for that.

Yes it was OzCoin, and I think Coinotron used the same parameters when it was running DGM. Thanks for the information.

Is the 'p' used here '1.0/D' or is it 'd/D' (where d is user's difficulty of submitted share). If it's 'd' how is it affected if a user has their difficulty changed part way through a block? Is it the 'd' for the most recent share submission?

It's d/D, and yes, d is for the share that solved the block.

Thanks, I assume the 'r' used in that calculation should be calculated using a 'p' adjusted for the difficulty of the share that solved the block too?

True.

In any case where f>=0, the operator will not be paying out of his own funds, is that accurate? I think I was misunderstanding "operator variance" as meaning the operator may be paying money out of his own funds like PPS in long rounds. Which I'm not willing to do. However, if it is simply variance in how much the operator earns without ever going negative, that's a totally different situation. :) I wouldn't mind my earnings (as an operator) going all over the place as long as the pool cannot ever go bankrupt in terms of payouts to miners like PPS.

Correct. But it wasn't a misunderstanding on your part either - if f<0 the operator can indeed pay out of his own pocket.

You can choose f=0 to avoid having to pay out of pocket, but it limits your options for reducing variance and maturity for miners. You could use something like f=0, c=0.01, o=0.99, it turns out similar to PPLNS. in core parameters.

Did you ever write this up someplace? :)

So instead of a PPLNS pool with a fixed 3% (f=.03) fee, one could run DGM with f=0, c=.03, o=.97 to have a similar pool with average fee to operator of 3%, but less variance for the miners because the operator is taking on that variance.

I didn't see it explicit in the OP, but does o need to equal 1-c?

I skimmed back over the thread, I thought I saw this discussed before but missed it just now if so, but if you are adjusting the parameters per-miner are there any specific constraints that must be observed? I'm specifically thinking of pools where the fee is based on the difficulty you are mining at. So as the user selectable difficulty rises, the pool's fee for that miner goes down. I wasn't sure if multiple miners all with f=0 but a variable c (and o=1-c?) would pose a problem.

Yeah, it's discussed in AoBPMRS.

Yes.

It doesn't need to be but that generally gives a reasonable tradeoff between variance and maturity time. X in PPLNS is in some ways analogous to approximately c/(1-o).

It wasn't designed to work with variable c; I think we could make it happen if we change the way some things are parameterized, but it's easier to vary f (make it 0 for the high-difficulty-share miners, and slightly higher for the low-difficulty shares).

Two implementation questions, if you have ever thought about this.

For miners that have a very small payout due, it's best to not pay them at this time and rather let their payout grow higher first. Otherwise they collect too much "dust" and when they try to go spend those coins, the transfer fees eat it all up (or cost more than the inputs, making the dust useless). Is handling this as easy as skipping their payment and leaving their score alone?

I realize even with f=0 future blocks may have more rewards to pay than the block contains. The other alternative is I could "pay" it to a separate dust database (keep the coins for a pool "dust reserve") then once the dust has gotten big enough do a payout from pool funds. Then the DGM scoring system/database isn't effected and knows nothing about it.

If you are paying miners in the coinbase of the block found, you can't wait for confirmation of the block to adjust their scores (or you might pay them multiple times if you find multiple blocks in a row). In a *PPS system you mark the found block as a "payout" for the relevant shares, and if that block later becomes an orphan you mark those share back to unpaid. Piece of cake, conceptually. In DGM, is the way to handle this to go through everyone paid in that block and set S=S/o? The payout in effect never happened, and the only change in database before was S=S*o. Seems simple, but wanted to know if you agreed.

Thanks!

Leaving the score alone won't work.

You could add to the worker's score in a way that compensates for the lost payment, but that is more complicated. It is simpler to keep a record for each miner of the total deferred payment, and when it is high enough make the physical payment via a normal transaction. Deferred payments will be kept in a designated address in your control.

It's even simpler than that - you don't need to set S=S/o. The score will remain in its reduced state and only the actual payment will be cancelled, and if you pay in the coinbase this happens naturally when the block is orphaned.

I can see why it is intuitive to treat orphaned block as if they never happened - thus cancelling S=S*o - but the method invariant is based on the assumption that a share has a probability of d/D to be a block. If orphans exist this is no longer the case, shares actually have a lower chance of becoming a valid block. If orphans are ignored completely, the operator will pay out more than he should (the block rewards he actually receives are lower than the method assumes). I've done the math and if you simply cancel the payment specified for the orphan blocks, everything is just right. This assumes, however, that every block, once received by the pool, has the same chance to become an orphan.

A tricky part is how to combine this with the dust payments mentioned earlier. You can no longer rely on the network to cancel the orphan payments - you'll need to actually remember for each worker the reward per specific block, and if a block becomes an orphan remove that record yourself.

What I'm struggling with getting my mind around is that work done before the orphan are at S*o and work done after the orphan is at S. When the orphan goes away and the payment is cancelled, aren't shares after the orphan worth more than the shares before it improperly? (If the orphan never happened and scores were never updated, then all work since last block (before or after the orphan that never happened) are all at S.)

I'm hoping you are indeed correct, since being able to ignore the score changes from the orphan event would be so easy. And you are normally correct. ;)


Dust is an issue for everyone eventually I think (if I'm understanding correctly) since there's no 0-1 cutoff. We are going down by S=S*o every payout the scores for a miner who stops mining will approach zero but never actually get there, they will just have ever decreasing scores. (Eventually so low it's just dust, forever?)

I think I'll just take your word for it and run with it. :) As they said in Star Trek II, "Yours is the superior intellect."

I was going to use logs so I didn't need to rescale, but it occurred to me that if I rescale every share I don't need to store s. Since s=s*r and s is always 1 going into the share, s is now always r. Thus, the payout becomes (S(r-1)(1-f))/(pr).

I guess the question is, do I really want to divide every score by r every share (updating hundreds of rows if there are hundreds of workers) vs storing s and using logs. It'd seem logs would be less impact to the database.

Edit: Final question for now. Is there a formula to calculate the total expected future payouts for a miner if they stop mining? That is, given the current state of things like s, r, p, B, and their score, can you approximate "if you stop mining right now the sum of all of your future payments is expected to be X". Basically, the value of their capacitor discharging from the current state down to zero. I think miners might find that useful (in addition to expected payout on next block from S/s), so they have a sort of idea of the long-term trailing reward that offsets the slow startup as the capacitor charged.

Thanks.

Shares submitted after the orphan will be worth more than those submitted before, and this is proper.

The orphan did happen and this is information that needs to be taken into account. The orphan blocks are not in addition to the valid blocks - they are instead, some of the blocks which should have been valid end up orphans instead. The correct comparison is not orphan vs. no block, but rather orphan vs. valid block.

When a block is found, all past shares have a chance to get a payday if it ends up valid. The cost of this chance at payment is the score reduction - even if, contingently, it ends up an orphan. Later shares never had a chance to profit from the block, and thus don't get their scores reduced for it.

This is similar to the situation with shares in general in a pool - when you submit a share, you are rewarded even if the particular share was of no use, because in finding a share you had a chance to benefit the pool - and you get an expected reward equal to your expected contribution.

Right. In practice you could trim scores which are below a very low threshold.

It seems you are suggesting that rescaling on every share is easier than rescaling once in a while, I don't see why.
I do recommend logs, they're less intuitive but more robust and probably less resource intensive.

S/s is not the expected payout from the next block. S/s (or more accurately (1-f)*(1-c)*S/s, accounting for fees) is what you asked about just now - the expected total eventual payout for already submitted shares.

The expected payout for the next block is S/s * (1 - c/(1-o+co)) * (1-f).

I think maybe I was paranoid about changing values in the database somehow making it unstable. Given your answer below, I'm thinking if a miner hasn't mined for a bit of time (to not trim brand new low hash rate miners) then do a single manual payout if S/s gets too low, and delete them from database completely.


*bonks self on head*. Thanks.

TLDR: If you jump down to Edit #2, I figured it out.


This is giving me values about half of the actual amount paid in the blocks. To use some real numbers on one specific block, here is my amount paid out formula. I am re-scaling $s to 1 every share. So it is $s=$s*$r in the figures below.


If I also calculate the estimated payout for the next block at the same time, I get (leaving out /s since I re-scale to 1 constantly.)


For S=12.932020560171, r=1.0000315757467, c=.03, f=0.0, o=.97, p=3.2552316167526E-5, s=1.0000315757467. The estimated equation reduces to 12.932020560171 * 0.4923857868 which is why it's about half. (With re-scaling every share, the payouts are extremely similar to the actual score value itself.)

I think I have the math coded properly. So either I misunderstand what "expected payout for next block" means or I have a blatant bug in how I'm coding the formula. Assuming formula is correct.

(To try and get a handle on how DGM operates, I'm using my share database to recreate "what would have happened" with DGM from start to finish. Since I'm only doing private testing, my test data is limited with about a week of mining and 3 blocks found, on an alt coin. Maybe the sample size is too small. 99K shares recorded with average block difficulty 36K, the block difficulty changes every block.)

Edit: Just occurred to me. Is the expected payout for next block given above if the miner stopped mining? So on average, D shares from now the payout to the miner would of course be much lower than the numbers I'm seeing (since the miner in my data sample kept mining away).

Edit #2: Yes that's it. I should have worked on it more before posting. Sorry for all the noise. It's spot on.

I injected an extra 41000 shares of work on a 41000 difficulty block into my database, and then injected a new "found share".

Before these changes, up to the final point in time for a worker (not the one listed above):
"Expected remaining total payout 6.1845766103505, next block pays 3.0451976203249"

After my artificial data injection,

Block Found: (score 3.1393789900256) pays out: 3.0451255771104

Nailed it within .00008. And I bet that's because there actually were a few thousand shares in the database after the last real block I left alone (so it was a slightly long round I created). So if someone stops mining, I can tell them their expected remaining total payout and their expected payout in the next block too. Awesome.

Yes, I should have clarified that, sorry.

If you want to assume he keeps mining, there are two factors to consider - his current score and his future proportion of the pool's hashrate. Taking them both into account requires some calculations. It's easier to just show what his reward will be if a block is found right now.

Yup. I just wanted to have an accurate way to show a miner their future remaining balance to offset the capacitor charging up originally. I have to imagine there's some learning curve for most miners who have never used a pool with DGM before.

I've just added DGM to my pool, and yes there's a learning curve. Since introducing it the pool hasn't solved a block with less than 1x difficulty shares for the last 6 or so blocks. The new DGM users are struggling with the concept of lower earnings why they'd be earning more on PPS. It's probably easier with a pool that starts with DGM vs switching.

For readability the first formula matches up to lS = log( exp(lS) + exp(ls) * pB ) but I assume you are doing it your way to have one less exp() call per calculation and/or to avoid huge/tiny numbers? My math skills are failing me to be sure both expressions are identical.

Edit: I understand setting lS = -Inf for new miners if you are setting them up in your score table in advance. But if you aren't going to add them until they request their first block to work on, it is equivalent to change ls + log(exp(lS-ls) + pB) to ls + log(pB), right? Since exp(lS-ls) will be 0 for an unknown miner, with lS=-Inf.

It's to make it numerically robust and avoid overflow. Both lS and ls eventually get so large that exp(lS) and exp(ls) cause overflow. But their difference lS-ls always manageable.

That's why we're using logarithmic scale in the first place, if you could handle exp(lS) directly you could simply work directly with S as in the original formulation.

Right.

Yeah, I just wanted to be sure I'm handling the switch to using logs correctly. I'm trying to be very careful as I code it, that there's no mistakes. :) Any sort of error could be too subtle for me to spot in production just by looking at score numbers alone. I implemented it the way you wrote it, I was just trying to make sure I understood the math and didn't just plug things in blindly.

Thanks.

Has anyone implemented DGM in php or a similar language?  If so I would love to take a look.

Hey Meni, with f=0.0 and c=.03 I'm assuming in long rounds the pool makes little/nothing and in short rounds it makes more than 3% to offset that. However, it seems so far (now that the first few blocks are out of the way and the miners have plenty of score) that in a really long round I'm making a hair over 0% (which is ok) but in a really short round I made a hair under 3%. The short round making 3% of the pool is what had me confused. I was figuring in short rounds (CDF was 2.2%) the pool would make well over the value of c.

Any thoughts? It's possible I have something wrong, although my java and php implementations resulted in the same values and it seems to be functioning correctly for actual miner payments.

There's cross-round leakage, so the pool's proceeds depends not only on the length of the last round but the previous rounds as well. If the short round you looked at followed a succession of long rounds it's normal to have low revenue.

Ah ok. I'll wait to worry until a few dozen more rounds have happened to give me more data to look at. ;)

I've hit so many short blocks in a row the fee is up to about 10%. I can't see miners appreciating that, even if in the long run it will all even out. (My fee on a recent long block was 0%.) So I might retool a bit to move from c=3% to f=3%.

If I have c=0 though, what should I use for r since I can't divide by zero?

This sentence doesn't make sense. You're reducing the miners' variance and taking it on yourself. In PPS with similar luck your fee would have been 300% and miners would still appreciate giving them variance-free payouts.

If c=0 then o must be 1, and then you can choose anything for r. r = 1+p is a good choice, more generally you can use r = 1 + a p, where larger a gives more variance and less maturity time.

Oh I understand, and it's why I choose DGM. I'm just not sure the average miner will understand. I still see people on other pool's threads complain about DGM now and then for totally incorrect reasons (penalizes hoppers, penalizes people who don't mine 24 hours a day, etc etc).

I'm sticking with it after all. For those who understand DGM and why I'm doing it this way, they'll appreciate it. And those that don't will just mine elsewhere. :)

Ive been sticking with bitparking thru their problems.

Right now my mining has been pretty stable yet the 'reward' seems to have dropped thru the floor.  Is this normal for this kind of reward system or do they have a flaw or what?  Did the difficulty just jump an order of magnitude or what?

Aaron

If you mean the payout per round, take into account that Bitparking's hash rate has almost doubled since yesterday.

Indeed. If the pool's hashrate doubles, you have twice as many rounds with half the reward each.

But if that's not the issue, I'd need more details to know what is.

Pool luck has been pretty reasonable lately. See the blockstats here (http://mmpool.bitparking.com/blockstats). When exactly did your rewards reduce? And are you comparing to PPS, other pools, or historical DGM payout on bitparking?

Did I understand correctly: we need to recalculate the global variable "s" after each submitted share?

Suppose there is a large pool which handles a few TH/s (a hundreds shares per second). What if operator will recalculate 's' only on each 100th share? How it will affect miners?

And what if pool supports adaptive share difficulty (x2, x4..)? Can it just count each x16 diff share as 16 shares x1 diff in row and push it into DGM algorithm? Adaptive diff increases miner's variance, but is it the only problem?

The calculations are lightning fast, updating s hundreds of times per second shouldn't be a problem. Well, doing it locally in your front end system and then storing it/updating it every so often. I wouldn't want to do hundreds of sql updates per second just to update the value of s. Update it on every share and store it every 100 (or some number) of shares in your database. If you have 100 shares/sec and store it every 500-1000 shares, then if the front end crashes/etc you only lose 5-10 seconds of data which may not be all that bad. Or do it every 100 shares. 1 sql query per second is no issue.


https://bitcointalk.org/index.php?topic=39497.msg1928641#msg1928641

p = d/D where d is the difficulty of the work sent to the miner, and D is the difficulty of that block being worked on. So it'd just be 1/D or 2/D or 4/D etc. You don't need to update it 4 times in a row with 1/D, just use 4/D.

One error I had early on was to use the difficulty of the block being worked on now, not the difficulty of the block when the work was sent to the miner. So keep an eye on that. (On bitcoin it'd only matter every difficulty change, but for my TRC pool the difficulty changes every block.)

Thanks, but you answered a different question =) I agree that 100-1000 multiplications per sec is not a problem. But I'd like to know if recalculating 's' on every i'th step is a reasonable optimization for large pools?

Then that I'll have to leave to Meni but surely it'd break the math. To what extent, I can't say.

Not really.

I'm still not sure what is the motivation, "recalculating" s is more or less one multiplication, it's completely trivial. Storing it in the db might be more expensive, but aren't you storing each share in the db anyway?

Shift-PPLNS and shift-DGM are more friendly for scalability.