"How to hop" has moved

[Meni Rosenfeld]

I wouldn't be surprised if it's a real result caused by self-hashrate-hopping. As you recall, in temporal pools it is an advantage to mine when the hashrate is higher. When you disconnect from the pool its hashrate is reduced, so conversely, you mine when the hashrate is higher.

I haven't quantified this effect, and AFAIK it could just as easily have the opposite effect by expediting decay of share submitted before disconnecting. I'll try to look into it.

Ok, I've got some results, though I haven't spent enough time on this to be sure they're correct. Doing this for slush's method is complicated, so I've done this for PPLNM (the temporal version of PPLNS). Wlog assume that the window is 1 time unit. We have a miner with hashrate H2 mining intermittently for a pool which, besides him, has a constant hashrate H1. Once in a while he starts mining for some time and then disconnects. We will denote H = H2/H1. His efficiency in the first time unit he joins is
 (H (2 + 3 H) - 2 (1 + H) Log[1 + H])/(2 H^2)
And his efficiency in the last time unit before he disconnects is
 1/2 + 1/H + Log[1/(1 + H)]/H^2
So his average efficiency in these two time units is
 (Log[1/(1 + H)] + (1 + H) (2 H - Log[1 + H]))/(2 H^2)
This is always less than 1 so he earns less than normal. It achieves a minimum of 94.8064% at H=3.47669. As H->Infinity the efficiency approaches 1. For H~0 the efficiency is (1-H/12). This is only for the 2 time units spent in the beginning and end of each connected session; the rest of the time the efficiency is 1.

slush's method is a little different but I'm no longer confident it could create a positive effect.

I'd like to remind everyone that this happens only in temporal pools. Hopping-proof methods like unit-PPLNS and DGM are immune to the effect. slush's method is temporal which is one of its two problems, but since it's so big you likely won't see the effect - if you're mining at 1GH/s you'll only have 0.008% (1 part in 12000) reduction in the worst case (with the reasonable assumption that the numbers for slush and PPLNM are similar).

Edit: Fixed a critical error.

[organofcorti]

Thank you kindly for that, Meni - very quick. I hope that the analysis makes it into AoBpmrs, although I can't see it becoming a problem unless someone develops that exahash farm   Or hopping tiny PPLNS-H pool comes into vogue. So there's one answer for you, paraipan!

If my following excuses about the errors I had in my last simulator attempt are not interesting for you, please go to the Results so far: paragraph.

So I had some errors in my last attempt at a simulation (which showed that 0.7% increase in efficiency for non-intermittent fulltime miners). For the intermittent miner, that increase was more like 1%. I think I've fixed these particular errors:

1. I was trying to make the simulator as efficient as possible and I was using some old techniques to do this. One of them was to assume all miners submitted shares evenly each time period (say every second). I changed this to a different and much less efficient method using Poisson random variables to determine when and how many of each share would be received by the pool and the error disappeared. I really don't know why.

2. Because of the original naive method of simulating Slush's pool I used a simple method of creating 'random' periods of downtime which happened to slightly favour the latter parts of a round. This made the miner a hopper, and made the increase in efficiency about 1%, which is clearly wrong. In my new method I treat the rounds as a very long string of share submissions and overlay random periods of downtime for the fulltime miner under investigation.

Variables: For the non-intermittent fulltime miner I've used 1Ghps hash generation rate with a 1000Ghps pool, making a hashrate ratio of 0.001. For the intermittent miner downtime is 0.05% with a string of as many downtimes as there are rounds. The downtimes also appear at times based on a uniform random variable because I simply don't have enough information to decide on a different probability distribution. The downtimes are exactly D*0.05 long, giving an efective hashrate of 0.95Ghps for the intermittent miner when averaged over many rounds - but this can vary significantly by round, especially on shorter rounds - no downtime on some short rounds and completely preventing submission during other short rounds. Longer rounds are affected by this variability less.

Results so far: I'm finding that for an intermittent miner the Slush scored payouts are an average of about 99.9% of the proportional pool payout, although I have a thousand or so rounds to run in total with only 400 completed so that figure may change. It is significantly worse than Meni's estimate for an intermittent miner in a PPLNS-H pool. I don't know if this reflects irl results, but it isn't a large enough difference to make a significant difference to the charts.

Possible modifications: I still have a few tricks up my sleeve if necessary - I'd like to model the downtime onsets as a Poisson random variable with an estimated arrival time of once per round (I could be quite wrong with this oversimplification, but ignoring external factors I guess that the probability of downtime per unit time is constant - very wrong if external factors like brownouts due to excessive power usage at certain times of day are to blame for the downtimes). Unfortunately generating and managing long strings of numbers slows things down significantly so I wont be doing this unless I have to. I could also model the length of downtimes as a uniform random variable which creates downtimes of 0 - 10%, but again I don't know whether this models the irl downtimes.

I'll post the charts when I have the thousand rounds worth of data completed and the charts prettified.

Requests Anyone want to see something specific from the data? Make your requests in a form I can generate charts from, eg 'Comparison of payout for Slush intermittent miner compared to prop pool intermittent miner as a function of round length' or 'contour plot of intermittent payout/proportional payout as a function of change in miner hashrate  and round length'. etc. If like the latter example your request requires new data, be patient - this is a slow beast.

Thanks again Meni for your rapid analysis of the intermittent miner on a PPLNS-H pool - interesting as always, plus having someone else's (trusted) results makes working out if and where there might be errors in a simulation much easier.

Edit: I just realised that the variance over as few as a thousand rounds would probably mask the tiny reductions in payout as described by Meni.

[Meni Rosenfeld]

Thanks again Meni for your rapid analysis of the intermittent miner on a PPLNS-H pool - interesting as always, plus having someone else's (trusted) results makes working out if and where there might be errors in a simulation much easier.

Well, seeing as my results before the edit were completely nonsensical, maybe they shouldn't be trusted too much . In my defense I did disclaim I hadn't taken the time to verify this properly.

Edit: I just realised that the variance over as few as a thousand rounds would probably mask the tiny reductions in payout as described by Meni.

Yeah, I doubt you'll be able to simulate properly such small differences. Also maybe there's a difference between slush and PPLNM, and maybe there are additional discrepancies in the scenarios we've looked at.

[paraipan]

thank you both for working on this, @meni hope you appreciate the small donation a sent you for your time, in Europe that buys you a latte . I hope in the future you will try figuring out more things related to score type of payouts under real world production environments.

@organof, until now this seems to be an interesting project to graph and thanks again for starting the "ball rolling".
We only had theoretical figures on every payout system, so this small experiment should make us an idea on the subject and free us from the burden of testing it in a real mining rig for the time being.
I know that miners have to take into account 3 main parameters when choosing their mining profile: self mining efficiency (under miner control), Internet and power connectivity (not always under control), pool variance (pool speed and popularity are two factors that go hand in hand to affect it). Those would be the main variables that depend on lots of sub-parameters on their own. I supposed that any change in one of the three affects payouts some way and preliminary results you and meni posted seem to go that direction too.
Hope you show us some nice comparing graphs soon and help us get more educated in the process  

[Meni Rosenfeld]

thank you both for working on this, @meni hope you appreciate the small donation a sent you for your time, in Europe that buys you a latte .

Thanks, much appreciated!

I hope in the future you will try figuring out more things related to score type of payouts under real world production environments.

My curse is that I know what's important and what's not. What we've done here is a nice exercise but it's not important, because A) the effect is tiny and B) This only happens in temporal methods like slush's pool. I don't like it when slush's pool is subject to false criticism but I never recommended using his method, it has several problems and it's known that this is one of them. This simply cannot happen in hopping-proof methods where the average reward really is equal to the total worth of work submitted regardless of pattern. (Though I'm not talking about things like stale shares, invalid blocks, pool downtime etc. which have nothing to do with the reward system.) The geometric method is what slush's method would have looked like if it was done properly.

[organofcorti]

So, here are the first few results. Look first, description after.












Before I start, the mean payout for the intermittent  Slush miner was 0.0074% more than for the intermittent prop pool miner, and the mean payout for the non-intermittent Slush miner was actually 0.01164% less than for the non-intermittent prop miner. I think we can ignore these values given the variance involved.

The first two charts show the amount earned by an intermittent miner at a proportional pool (green) and the exact same share submissions at a Slush scored pool. The first chart shows results for one hundred rounds, the second chart are results for twelve hundred rounds. You can see clearly that the results in both cases are centred near the expected 0.0475 btc. However the scored payout shows a lot more variance.

The third chart shows the proportional payout minus the Slush payout for non intermittent miners. It has a fair amount of variance but doesn't seem skewed to either above or below expected, meaning that median and mean are similar, and we don't have a large amount of small increases one way and a few very large ones the other way to 'balance' it.

The fourth chart shows the proportional payout minus the Slush payout for non intermittent miners. There is skew in the data, with lots of small negative differences (where the Slush scored pool pays better than proportional) and a few very large positive differences (where the slush scored pool pays significantly worse). I think the density plot below that illustrates this point well.

Statistics:

Non-intermittent, Prop payout per round minus Slush payout per round
median: -0.0000470
mean:    0.0000058
sd:         0.0036246

Intermittent, Prop payout per round minus Slush payout per round
median: -0.0006029
mean:    -0.0000035
sd:         0.0061706

Summary:

In the long run and with the parameters previously mentioned, being an intermittent miner on a Slush scored pool is not significantly different from being an intermittent miner on a proportional pool. In the short term, a comparison between the non-intermittent and intermittent groups shows significantly increased variance for the prop minus Slush intermittent group as compared to the non-intermittent group, with some slight skew - which may not be significant given the variance in the data.

Edit: Fixed an error in mean intermittent prop minus Slush payout. Value should be -0.0000035 not 0.0000035.

[paraipan]

awesome work organof, just managed to pull a few great conclusions based on your data, hope others will too. I'm pretty sure i will be making better mining decisions from now on.
Sent you a tip, not much but expect more in the future because i will be closely watching your work.

[organofcorti]

awesome work organof, just managed to pull a few great conclusions based on your data, hope others will too. I'm pretty sure i will be making better mining decisions from now on.
Sent you a tip, not much but expect more in the future because i will be closely watching your work.

Well, thanks for that, paraipan! Your donation is much appreciated.

What I got from the simulation was (if you're a fulltime 1Ghps miner with 5% downtime) not to allow a few worse payouts than expected change your mind about mining at Slush's pool since this is offset by more regular and less noticeable better than proportional payments. The hoppable score system is something that should make you change your mind though.

Since you were kind enough to donate I rewrote the the simulation to allow for a changing hashrate and changing downtimes. I'll post a contour plot and a 3D perspective graph of the results when it's done. It will take a few days, I expect.

While I'm working on this simulator, any other requests? Keep in mind this simulator only includes variables which are important for the payout system. It doesn't include frequency of getwork requests, mining software variables, etc - just frequency of share submission, frequency of downtime, and the payout.

[deepceleron]

PPLNS systems, as long as they cross rounds (which all do, there is no concept of pool rounds when distributing payments) are all fair payout methods that do not positively or negatively affect the part-time miner. Since the work you submit now determines your percentage of reward on future blocks until the work expires, you don't know when blocks will be found, and past block finding doesn't affect future payments, all contributed shares have equal expected value.

I think it may be possible to game 'decaying' value pools or get a different average reward than you deserve when part-time mining, which is non-ideal. Lets say you have three different accounts on one pool, and switch between accounts every hour. It would seem that under an exponentially decaying reward or other custom poorly-informed formula (don't know if any pools do this) your expected reward may be different than constantly mining one account. Just thinking that 'pulsing' your hashrate (intermittent mining), may get you more or less expected reward than constant mining, depending on the decay rate, when we want every submitted share to have the same expected return.

Upon reflection, it seems like the second paragraph's concern is also unfounded. As long as each share's value is independently evaluated, it shouldn't make a difference. If a pool has some other policy, like "if you leave for three hours your reward is cut in half", this would "connect" the value of shares you submit now to future shares you submit.

[Meni Rosenfeld]

It would seem that under an exponentially decaying reward or other custom poorly-informed formula

PPLNS isn't the only hopping-proof method. As you can see in section "general unit-based framework" of AoBpmrs, a hopping-proof method can either cross rounds, include a variable fee, or both, and have either exponential, step, or any other decay. Exponential decay is far from poorly-informed - it is the only decay that allows encoding the score with a single value.

The problems with slush's method aren't that it's exponential - it's that it's temporal, and that it has neither round crossing nor variable fee.

[teukon]

Upon reflection, it seems like the second paragraph's concern is also unfounded. As long as each share's value is independently evaluated, it shouldn't make a difference. If a pool has some other policy, like "if you leave for three hours your reward is cut in half", this would "connect" the value of shares you submit now to future shares you submit.

Yes.  The goal of many of the reward systems is to ensure that each submitted share has the same expected value.  PPLNS does this (unless carelessly implemented), PPS does this (this too can be implemented in a way that is technically hoppable thanks to the transaction fee reward), and Geom does this.  As far as I understand it, DGM simply interpolates between PPLNS and Geom is a spectrum of different non-hoppable reward systems ranging from PPLNS to Geom and, consequently, takes an additional parameter.  If a reward system has this property and pays all rewards out in a timely manner then it simply cannot be hopped (for the time issue, compare DGM to SMPPS).

The interesting aspect is that, for a fixed difficulty and hashrate, the number of blocks generated in a certain period is very well modelled by the Poisson distribution (equivalently, the time until the next block is found is very well modelled by the exponential distribution).  Consequently there is a natural "decay", not built into Bitcoin by design, but simply a mathematical phenomenon.  If you just design a reward system with shares decaying in value at some random rate you prescribe then most likely the pool will be hoppable.  The decay must exactly match that required by the mathematics.  This is why the proportional reward system is hoppable; the decay is not the same as that required by the mathematics.  Nonsense

[Meni Rosenfeld]

As far as I understand it, DGM simply interpolates between PPLNS and Geom

That's not what it does. It's on a spectrum that has exponential-PPLNS on one end and Geom on the other, but it doesn't just pay out a convex combination of the respective rewards.

If you just design a reward system with shares decaying in value at some random rate you prescribe then most likely the pool will be hoppable.

That's not really it. You can use any decay function (and linear decay gives the best variance/maturity tradeoff). But you must maintain a steady state with some combination of round crossing and correctly tuned variable fee. Suppose you take the geometric method, in which the operator's score is specified as 1/(r-1). If you choose to make the score any less, you incentivize mining in young rounds; if you make it any higher, you incentivize mining in old rounds; only with this exact score the profitability is always the same.

This is why the proportional reward system is hoppable; the decay is not the same as that required by the mathematics.

There's nothing wrong with the decay in proportional. It's equivalent to exponential decay with r=1. If you do this with infinite score fee and correspondingly infinite negative fixed fee, you get PPS which is hopping-proof. The problem is that people try to use it with 0 score fee.

[teukon]

As far as I understand it, DGM simply interpolates between PPLNS and Geom

That's not what it does. It's on a spectrum that has exponential-PPLNS on one end and Geom on the other, but it doesn't just pay out a convex combination of the respective rewards.

If you just design a reward system with shares decaying in value at some random rate you prescribe then most likely the pool will be hoppable.

That's not really it. You can use any decay function (and linear decay gives the best variance/maturity tradeoff). But you must maintain a steady state with some combination of round crossing and correctly tuned variable fee. Suppose you take the geometric method, in which the operator's score is specified as 1/(r-1). If you choose to make the score any less, you incentivize mining in young rounds; if you make it any higher, you incentivize mining in old rounds; only with this exact score the profitability is always the same.

This is why the proportional reward system is hoppable; the decay is not the same as that required by the mathematics.

There's nothing wrong with the decay in proportional. It's equivalent to exponential decay with r=1. If you do this with infinite score fee and correspondingly infinite negative fixed fee, you get PPS which is hopping-proof. The problem is that people try to use it with 0 score fee.

My apologies, I had to dash while I was composing this post and didn't have time to read it.  By "interpolate" I didn't mean to suggest any particular mathematical way of calculating payouts but rather just meant to suggest that by varying one of the parameters you would obtain a continuum of reward systems from PPLNS to Geom.  I admit this was a poor choice of word but couldn't think of anything better at the time.  Spectrum is a good word for this.

As for the "decay" part I was being particularly dense when I wrote this and I'm sorry for any confusion caused (I'll edit my earlier post if I can).  I somehow managed to get the idea in my head that it would be possible to have a non-hoppable reward system where for each block the full reward is paid out according to the shares submitted in that round.  I was talking about the "decay" in value such a share submitted to such a pool must experience before the end of the round (and thought it would exist and be some kind of exponential decay).  Of course, the only reward system that satisfies these two properties is PPLNS with N=1.  I was flatly wrong, "That's not really it" was just kindness .

After a quick read of your initial post about the Geometric method I see what you are saying about Prop and PPS.

May I ask what you do for a living Meni?

[Meni Rosenfeld]

Of course, the only reward system that satisfies these two properties is PPLNS with N=1.

Yeah, that's what I call "the hopping immunity theorem". I'll add a proof in AoBpmrs as soon as I determine the most general conditions under which I am able to prove it.

May I ask what you do for a living Meni?

A bit off-topic isn't it?

I used to be an algorithm researcher in a small internet startup. I recently moved to a 20%-time position there so I can focus on Bitcoil.


This, by the way, is my 1000th post.

[teukon]

Of course, the only reward system that satisfies these two properties is PPLNS with N=1.

Yeah, that's what I call "the hopping immunity theorem". I'll add a proof in AoBpmrs as soon as I determine the most general conditions under which I am able to prove it.

Good.  The simple form of the theorem that is suggested by the two conditions I gave is easy to prove but it would be nice to say something more along the lines of "If X is a hop-proof pool then there must exist a function f and ...".  This shouldn't be tough (kind of fun really) but as I said a while back I've not done any of the maths for pooled mining and instead am enjoying just going with rough intuition (which frequently leads to my making false assertions but that's fine).  Good luck with this, not that you'll need it.

May I ask what you do for a living Meni?

A bit off-topic isn't it?

I used to be an algorithm researcher in a small internet startup. I recently moved to a 20%-time position there so I can focus on Bitcoil.

If you've been following my posts you'll know I'm a fan of "off-topic".

This, by the way, is my 1000th post.

Congrats!  It's a shame you don't get higher status than "Hero Member".

[organofcorti]

Want to try your hand at simulating pooled bitcoin mining? How to hop 9: Simulating miner earnings

I explain how to generate random variables and provide a step by step example of what to do with them. Feel free to ask any questions you might have.

[Meni Rosenfeld]

round(ln(U)/(ln(1-1/D)))

Round length follows a geometric distribution starting at 1. So to generate a sample accurately, you should use 1+round(ln(U)/(ln(1-1/D))) (assuming "round" rounds down). For an approximation that works well for large D you can simply use round(-D*ln(U)).

[organofcorti]

Thanks Meni - I'd forgotten to include the '1+' so you just saved blog readers who might want to simulate very large numbers of rounds from NAN-ville 

Also, the simplified version was great. I know I've seen that approximation somewhere, but how do you derive the

Code:

D approximates -1/ln((D-1)/D) approximates 1/ln((D+1)/D) 

I should probably know this!

[organofcorti]

I just realised that hardly any of the blog readers so far have accessed the extra charts for the post or the scripts I linked to, so I encourage everyone who might want to try their hand at simulating pooled mining earning to check them out:

'How to hop 9' album on imgur.com
Poisson random number generator
Mining simulator pseudo-code
Mining simulator

Also, if you want to generate geomterically distributed random numbers you approximate it (and compound the inaccuracies, but oh well) back-asswards using a Poisson distributed RNG:

Code:

 Pseudo-code:

set blockFound<-k<-0
while blockFound<1
set blockFound <- generate(Poisson random, mean = D)
set k <- k+1

return k

This is inaccurate (since the number generated ending the round could be > 1, and any inaccuracies in the Poisson RNG will be compounded), very inefficient and I guess not very interesting. But it is how I generated geometrically distributed random numbers for my first simulator and before I knew better! I'd better thank Meni for that, too.

[Meni Rosenfeld]

Also, the simplified version was great. I know I've seen that approximation somewhere, but how do you derive the

Code:

D approximates -1/ln((D-1)/D) approximates 1/ln((D+1)/D) 

I should probably know this!

From the Taylor series of ln(1+x) You know that for x~0 you have ln(1+x)=x+O(x^2). So for large D you have 1/D ~ 0 and so ln(1-1/D) ~ -1/D and ln(U)/ln(1-1/D) ~ ln(U)/(-1/D) = -D*ln(U).

If you don't know the Taylor series of ln you can simply use the fact that the derivative of ln at 1 is 1. Or you could use exp(x) ~ 1+x and take logs of both sides.