Does pool hopping really work?

[anon4758]

@Meni Rosenfeld
I mostly lurk. So

http://img25.mediafire.com/8abd9f4a6e982c940048f1937b8b857c9ca7fc090c5373d2c3afa2c565aaab9c6g.jpg

is an accurate description.

[Sukrim]

I just wonder how you realized your pool hopping method (manual might be interesting for a short time, but is not really feasible).

Also how to decide in reality to which pool to jump - imagine mining still in the 43% margin of pool 1 but then pool 2 finds a share. Should you then jump to pool 2 or mine until 43% on pool 1 and risk not having a pool to immediately jump to?

So in the end: Is it smarter to hop off as early as possible or as late as possible? Could/should it be weighted somewhere in between?

I bet you get also tons of PMs now!

[Meni Rosenfeld]

I just wonder how you realized your pool hopping method (manual might be interesting for a short time, but is not really feasible).

Also how to decide in reality to which pool to jump - imagine mining still in the 43% margin of pool 1 but then pool 2 finds a share. Should you then jump to pool 2 or mine until 43% on pool 1 and risk not having a pool to immediately jump to?

So in the end: Is it smarter to hop off as early as possible or as late as possible? Could/should it be weighted somewhere in between?

I bet you get also tons of PMs now!

In the simplest model, you always want to mine for the pool with the least shares in the current round.
So if your current pool has 30% and some other pool has 0% or 10%, you hop.
The next step is to take into account the time wasted reconnecting to a new pool. If the difference is very small it is generally better to stay.
It will probably be beneficial to have several miners pointed at different pools with similar age.
Thus goes the theory, anon will have to discuss how he implemented this.

[anon4758]

That's right, you jump to the shares-based pool with the presently lowest round_time/mean_round_time fraction, or score-based pool if none are below 43% fraction. The switching takes only a second and is not a problem. I use a script to query the pools periodically/whenever bitcoind reports a new block and redirect the miners as necessary. This also simultaneously solves the unplanned pool downtime issue. Interestingly, bitcoind always reports new blocks at the same time as pools pushing new longpoll work, or no more than 10 seconds apart. Since the daemon and the miners use entirely different connections, that means transactions/block solutions are being propagated across the entire bitcoin network within 10 seconds or less on average.

[grue]

That's right, you jump to the shares-based pool with the presently lowest round_time/mean_round_time fraction, or score-based pool if none are below 43% fraction. The switching takes only a second and is not a problem. I use a script to query the pools periodically/whenever bitcoind reports a new block and redirect the miners as necessary. This also simultaneously solves the unplanned pool downtime issue. Interestingly, bitcoind always reports new blocks at the same time as pools pushing new longpoll work, or no more than 10 seconds apart. Since the daemon and the miners use entirely different connections, that means transactions/block solutions are being propagated across the entire bitcoin network within 10 seconds or less on average.

isn't it better to stay with a pool until 43%, rather than switching to the lowest % pool? the graph in Raulo's paper shows that leaving before 43% carries a steep penalty.

[Sukrim]

isn't it better to stay with a pool until 43%, rather than switching to the lowest % pool? the graph in Raulo's paper shows that leaving before 43% carries a steep penalty.

Actually you should stay at least ~20% and max. ~60% of the expected round time in a proportional pool. Hopping off below ~20% is not a very smart decision, above 60 or 70% it gets more and more useless. 43% still is the optimum, but if you jump off at 40 it might be a better thing to do than to risk mining until 70%.

In practice you would probably have to rank pools according to their hash rate and if a pool finds a block, evaluate if it makes more sense to stay in the current pool or to hop on to the next candidate.

[Meni Rosenfeld]

That's right, you jump to the shares-based pool with the presently lowest round_time/mean_round_time fraction, or score-based pool if none are below 43% fraction. The switching takes only a second and is not a problem. I use a script to query the pools periodically/whenever bitcoind reports a new block and redirect the miners as necessary. This also simultaneously solves the unplanned pool downtime issue. Interestingly, bitcoind always reports new blocks at the same time as pools pushing new longpoll work, or no more than 10 seconds apart. Since the daemon and the miners use entirely different connections, that means transactions/block solutions are being propagated across the entire bitcoin network within 10 seconds or less on average.

isn't it better to stay with a pool until 43%, rather than switching to the lowest % pool? the graph in Raulo's paper shows that leaving before 43% carries a steep penalty.

Raulo discusses the case of one proportional pool, when the choice is to switch from it to solo/score-based.

You all need to remember there isn't a magical tooth fairy that comes and gives you monies when you do the act of switching pools. It's all a matter of finding the expected payout right now for every pool and going with the one for which it is highest. For proportional pools the payout is determined strictly as a decreasing function of the current round age. For >43.5% it becomes less than solo/score.

[anon4758]

isn't it better to stay with a pool until 43%, rather than switching to the lowest % pool? the graph in Raulo's paper shows that leaving before 43% carries a steep penalty.

Actually you should stay at least ~20% and max. ~60% of the expected round time in a proportional pool.

Not really - the pool that has just found a block is always the best, though it might be better to switch back to a slower pool at some later time. There is no penalty. The utility of each submitted share is independent of any other share (unless you are a dominant contributor to the pool).

For difficulty d and pool hashrate R, the probability that the round will end in exactly time t is:
$ p(t) = \frac{R}{2^{32}d} e^{\frac{-tR}{2^{32}d}} $
If we express time in units of mean lifetime $ Tm = \frac{2^{32}d}{R} = 1 $, this simplifies to:
$ p(t) = e^{-t} $
The expected reward for a single hash submitted T time into a round is the integral over all possible total round times of this probability divided by the total number of pool hashes in the round:
$ E(T) = \int_T^\infty \frac{e^{-(t-T)}}{Rt} dt = \frac{e^T}{R} \int_T^\infty \frac{e^{-t}}{t} dt = -\frac{e^T}{R} Ei(-T) $
Your solo expected reward is always 1/(d*2^32), so your pool/solo reward ratio, or efficiency, is (using mean lifetime units, where d*2^32/R = 1):
$ r(T) = -e^T Ei(-T) $
http://www.wolframalpha.com/input/?i=+-e^T+Ei%28-T%29+%3D+1
The efficiency drops below 100% at exactly 0.434818... mean lifetimes into a round. Efficiency is pretty high early on though. When mining solo, you earn $ \int_0^{0.4348} e^{-T} dT = 35.26\% $ of all your rewards during the 0 to 43.58% mean lifetime round interval, but in a pool you earn $ \int_0^{0.4348} -Ei(-T) dT = 63.41\% $ of your total rewards in that interval - a 79.83% bonus. Across all time, your average reward is highest if you hop at 43% point:
$ \int_0^{0.4348} -Ei(-T) dT + \int_{0.4348}^\infty e^{-T} dT = 1.2814 $ - a 28% bonus. Multi-pool hopping works even better, since you always pick the pool with the highest current expected reward, which is the pool with the smallest round time/mean round time ratio. As I said, I only rotate between four pools, and not always is there a pool with round time below 43%, so I average about 70% extra efficiency over entire time, but the bonus can be pushed above 100% using many more pools.

It is somewhat of a philosophical question of whether 1% here and there really makes a difference, and whether you should actually care whether it is 43% or 43.5% or whatever, but this is what the math says. Again, since hopping only takes a second, you take no time penalty from doing so anytime you wish.

For Slush: the score-based method, as implemented, is better, but not perfect. If a share at time t into a round gets a score $ s = u + e^{t/v} $ where v, the decay constant, is 300 seconds and u is 1 (it doesn't make much difference actually), the total score of a round of length t is:
$ s(t) = \int_0^t R (u+e^{t/v}) dt = R (ut + v(e^{t/v} - 1)) $
And the expected reward ratio for a share submitted T time into a round is (using the reduction d*2^32/R =1 ):
$ r(T) = \int_T^\infty \frac {e^{-(t-T)}(u + e^{T/v})} {ut + v(e^{t/v}-1)} dt $
I don't have a plot of this curve unfortunately, only some numerical approximations. For 542GHash/s, Tm is 3446 seconds, so decay constant v is 0.0871Tm. and assuming u=1:
r(0.01)=261.40%
r(0.10)=109.87%
r(0.14)=99.25%
r(0.43)=91.15%
r(1.00)=91.98%
r(10.00)=91.99%
The important point to see here is that the expected reward ratio is not 100%, as claimed, but rather drops below 100% at 14% round mean lifetime, and hovers around 92% from there on. Averaged over all time, I can make 10%-20% extra by only mining in slush's pool in the first 14% of a round.

Again, I recommend pool operators to familiarize themselves with the math that makes pool-hopping efficient, and implement an algorithm, like Meni Rosenfeld's geometric score, that is provably neutral. I also request people to review my constant-time proposal, since I believe it would be easier to implement/understand than exponential scores (linked lists instead of logarithm math).

[Raulo]

The important point to see here is that the expected reward ratio is not 100%, as claimed, but rather drops below 100% at 14% round mean lifetime, and hovers around 92% from there on. Averaged over all time, I can make 10%-20% extra by only mining in slush's pool in the first 14% of a round.

I haven't checked all your math but although slush's method is hopping prone and it can be proved, by no means you can get 10-20% more out of it. You must have forgotten how rare are the short rounds and for long rounds, you'll get virtually zero by staying during 14% of the round. We have done simulations with Slush and it resulted in about 2-3% gain at that time.

I also request people to review my constant-time proposal, since I believe it would be easier to implement/understand than exponential scores (linked lists instead of logarithm math).

It may be easier to understand but was much easier to implement by slush because he does not have to keep all the shares in memory, only the score. Moreover,
your proposal is nothing more than an elaborate pay par share method which may be good for pool users but is very risky for the pool operators.

[Sukrim]

isn't it better to stay with a pool until 43%, rather than switching to the lowest % pool? the graph in Raulo's paper shows that leaving before 43% carries a steep penalty.

Actually you should stay at least ~20% and max. ~60% of the expected round time in a proportional pool.

Not really - the pool that has just found a block is always the best, though it might be better to switch back to a slower pool at some later time. There is no penalty. The utility of each submitted share is independent of any other share (unless you are a dominant contributor to the pool).

Ah, thanks for the insights! I was already wondering why early hopping sould be bad at all, as early shares should be the most valuable anyways (as for example in a case where a block gets solved after just 10 shares by 10000 miners, the 10 lucky share submitters would get FAR more than expected).

[Raulo]

The important point to see here is that the expected reward ratio is not 100%, as claimed, but rather drops below 100% at 14% round mean lifetime, and hovers around 92% from there on. Averaged over all time, I can make 10%-20% extra by only mining in slush's pool in the first 14% of a round.

OK. Now I get it what you mean by 10-20% more. This is 10-20% more out of slush's pool. If you mined solo for the rest of your hashrate, it would be 6% of your total hashrate. When slush implemented the pool, there were no other pools and the hopping reward was somewhat smaller. With multiple pools, it might be worth the fuss. 

I've just did a Monte-Carlo simulation for decay constant 0.0871 and hopping cut-off 0.14 and you get
reward equal to 19.4% of your total hashrate out of the pool, while one should have gotten only 13.2%. So it is 6% of your hashrate extra (assuming you mined solo or pay par share pool the rest of the time) but more than 40% more out of the shares submitted to the pool. Pool owner can reduce it by decreasing decay constant while increasing variance but it's hard to reduce it completely without increasing the variance excessively.

Meni Rosenfeld's geometric score, that is provably neutral.

This geometric score requires either huge fees or introduce a payoff variance for the pool operators which may be unacceptable for them.

[Raulo]

Hopping away from a single pool (at 43% of mean round time) earns 30% more. Hopping between a couple pools (I have 4 currently automated) earns me 70% more. If you'd automate a dozen pools, you could earn more than 100% extra.

In the theoretical limit of infinitely many proportional pools and perfect hopping, I think you get log(difficulty) times normal, which currently is X13, or 1200% extra.

It can't be right. First of all, hopping reward is completely independent of difficulty.  1GH/s hopper @ difficulty 100000 gets the same reward as 2GH/s hopper @ difficulty 200000. Secondly, even with infinite number of pools, there is a finite number of blocks. With 6 blocks/hours you can only make 6 hops an hour and some of the hops are impossible to do (you cannot hop to solo miners). My guess based on the intuition I gained working on this problem is that in the limit of infinite number of proportional pools, all global hashrate contributed by the pools you can hop into, and optimal hopping, you can probably get about 100-150% more. I'm yet to derive it or program a simulator. This number is reduced if some hashrate is from solo mining and if the pool operators introduce countermeasures. I doubt that anon4758 can get further beyond his 70% by more than a few percent because he probably hops among almost all the hashrate available for hopping.

[Meni Rosenfeld]

Hopping away from a single pool (at 43% of mean round time) earns 30% more. Hopping between a couple pools (I have 4 currently automated) earns me 70% more. If you'd automate a dozen pools, you could earn more than 100% extra.

In the theoretical limit of infinitely many proportional pools and perfect hopping, I think you get log(difficulty) times normal, which currently is X13, or 1200% extra.

It can't be right. First of all, hopping reward is completely independent of difficulty.  1GH/s hopper @ difficulty 100000 gets the same reward as 2GH/s hopper @ difficulty 200000. Secondly, even with infinite number of pools, there is a finite number of blocks. With 6 blocks/hours you can only make 6 hops an hour and some of the hops are impossible to do (you cannot hop to solo miners). My guess based on the intuition I gained working on this problem is that in the limit of infinite number of proportional pools, all global hashrate contributed by the pools you can hop into, and optimal hopping, you can probably get about 100-150% more. I'm yet to derive it or program a simulator. This number is reduced if some hashrate is from solo mining and if the pool operators introduce countermeasures. I doubt that anon4758 can get further beyond his 70% by more than a few percent because he probably hops among almost all the hashrate available for hopping.

The model you've analyzed in the paper is continuous and does not take into account share granularity. If p=1/difficulty and you submit to a proportional pool the very first share in a round, you'll note that your expected payout is $\sum_{i=1}^{\infty}p(1-p)^{i-1}\frac1i \approx -\log p$.

However, you are correct that I have not taken into account that you will also need infinitely many blocks for this. So what we can say is: With perfect hopping, you can get X13, but not continuously - rather, you run your miner only when a proportional pool has a new round, which will give you the measly volume of 1 share per block found by a proportional pool. This result is of course of no practical significance, but generally analyzing intermittent mining may be important if electricity is a large portion of the cost.

This geometric score requires either huge fees or introduce a payoff variance for the pool operators which may be unacceptable for them.

With a correct choice of parameters, the variance can be acceptable for both the operator and the participants.
At current difficulty, with c = 0.00015 and f = 0.02-c, you get 2% average fee and a variance reduction of X133 for both operator and participants, compared to having the same payout mining solo. This isn't bad.
If you want to give up a little hopping-resistance to gain a little variance-reduction, you can decrease the operator's score for a given r. slush basically reduces it to 0 and thus has only a little resistance, but you can find a better balance.
But the decay based on shares rather than time is a pure advantage which should be adopted even if score fee is eschewed.
I don't know where you got the part about huge fees.

[anon4758]

I've just did a Monte-Carlo simulation for decay constant 0.0871 and hopping cut-off 0.14 and you get
reward equal to 19.4% of your total hashrate out of the pool, while one should have gotten only 13.2%. So it is 6% of your hashrate extra (assuming you mined solo or pay par share pool the rest of the time) but more than 40% more out of the shares submitted to the pool. Pool owner can reduce it by decreasing decay constant while increasing variance but it's hard to reduce it completely without increasing the variance excessively.

You are right. I finally found an integrator that works, and between 0% and 14% round time, I get 19.80% of my total reward in slush's pool, compared to 13.06% of total reward while mining solo - a 52% premium, but only 6.7% over all time. Of course, if the network hashing power were divided equally among 6 score-based pools, then instead of only 14% of the time, I could mine at a premium 84% of the time (or however much it adds up to), for a 40% (or however much) total premium. However, since the majority of the network hashing power is currently in share-based pools, that is not even necessary.

In the theoretical limit of infinitely many proportional pools and perfect hopping, I think you get log(difficulty) times normal, which currently is X13, or 1200% extra.

It can't be right. First of all, hopping reward is completely independent of difficulty.  1GH/s hopper @ difficulty 100000 gets the same reward as 2GH/s hopper @ difficulty 200000. Secondly, even with infinite number of pools, there is a finite number of blocks. With 6 blocks/hours you can only make 6 hops an hour and some of the hops are impossible to do (you cannot hop to solo miners). My guess based on the intuition I gained working on this problem is that in the limit of infinite number of proportional pools, all global hashrate contributed by the pools you can hop into, and optimal hopping, you can probably get about 100-150% more. I'm yet to derive it or program a simulator. This number is reduced if some hashrate is from solo mining and if the pool operators introduce countermeasures. I doubt that anon4758 can get further beyond his 70% by more than a few percent because he probably hops among almost all the hashrate available for hopping.

The model you've analyzed in the paper is continuous and does not take into account share granularity. If p=1/difficulty and you submit to a proportional pool the very first share in a round, you'll note that your expected payout is $\sum_{i=1}^{\infty}p(1-p)^{i-1}\frac1p \approx -\log p$.

That is all correct. Sometimes I forget that the real world is discrete. The utility of the very first share in a round is not infinite but very much finite (x13 seems reasonable), since shares are not infinitely divisible and integrable. Also I agree that the maximum theoretical efficiency of multi-pool hopping is limited by the fact that on average no more than 6 blocks can be found per hour.

I also request people to review my constant-time proposal, since I believe it would be easier to implement/understand than exponential scores (linked lists instead of logarithm math).

It may be easier to understand but was much easier to implement by slush because he does not have to keep all the shares in memory, only the score. Moreover,
your proposal is nothing more than an elaborate pay par share method which may be good for pool users but is very risky for the pool operators.

Keeping a stack of several million of the latest shares is not particularly memory-intensive. Even less-so if the oldest shares are offloaded to disk, before sometime later all record of them is finally deleted (very old shares are exponentially-unlikely to be needed for payout). Also, it is not pay-per-share and carries no risk for the operator, since all payouts come from each solved block and not out-of-pocket.

[Meni Rosenfeld]

I also request people to review my constant-time proposal, since I believe it would be easier to implement/understand than exponential scores (linked lists instead of logarithm math).

I think this indeed works. Your payout for a submitted share only depends on the future and not the past, so you can't hop. And it has a significant variance advantage over the geometric method. However:
1. You need to base the window on number of shares, not time. Time-based methods are a mess and open to hashrate fluctuation attacks.
2. You'll need to decide what to do with a short round when the pool starts, and there are no previous rounds to draw on. You could let the operator take the leftover, or distribute it between the shares (thus creating an expectation bonus early in the life of the pool).
3. You'll need to figure out how to make the payout invariant to changes in difficulty (should be doable).
4. You'll need to handle the case of changing the window length and fees. With other methods, each round is self-contained so you can change parameters between rounds. But here, a contingent future change may affect the current expectation.

[dayfall]

Yes, pool hopping works.  And it is lots of fun if done to pools where the owners don't believe it works.  I still see that people are successfully hopping on that certain unnamed pool.

I was going to write a simulation of the score based system, but I see that you guys have that well under control.  It simply amazes me that people are so clueless when it comes to (what seems to me) simple math.

[anon4758]

All right, taking Meni Rosenfeld's considerations into account, how is this for a formal proposal for a constant-share pool:

1. When the round ends, the pool will use the block reward (minus fee) to pay equally for all of the latest "difficulty/2" shares
The mean number of shares in a round is always equal to the difficulty, regardless of pool size, since each share is a solution of difficulty 1, and each share has 1/d probability of solving the block. In comparison to share-based pools, on average, you will be paid for exactly half of the shares you submit to the pool (twice as much, of course), and the reward will be constant if your hashing power as a fraction of the pool's total hashing power is constant.

2. If there are less than d/2 shares in the round, the pool with use the rollover reward to pay for the latest unpaid shares in the previous round(s)
The rollover eliminates the risk of pool hopping, since there is never an advantage or disadvantage to mine very early or very late in a round.

3. The pool will cache the last 10d shares, in case there is a very lucky streak of very short rounds
There needs to be a cache of sufficient size for the rollover to be effective. With a cache of size 10d and payout slices of size d/2, there is a (very roughly) 1/2^20 (one in a million) chance that the cache will soon be exhausted.

4. If there are lucky blocks at the very start of the pool's lifetime, the pool will hold on to the rollover to distribute to the miners in the unlikely/inevitable event that the pool is shut down permanently
This is a mostly trivial point and you have to trust the pool operator to carry through with this promise, but this terminal fund simultaneously resolves two problems:
a) The end-of-life problem: if a pool is shut down, the round never ends and the miners who contributed to it are left holding an empty bag. But with the terminal fund, the miners can continue to mine with security, even if there is uncertainty of whether the pool is about to close.
b) The initial pool problem - what to do with the rollover at the beginning of a pool's lifetime. The fund is a clever way to short circuit the end of the pool's lifetime to its beginning, tying up all loose ends. The other two alternatives are to distribute the bonus to the miners - the early adopter advantage - or for the operator to keep it.

5. When difficulty changes, the reward interval follows; the rollovers will occur as normal, except that older shares have a higher weight equal to "new difficulty/old difficulty"
The reward interval is fixed to difficulty/2 to maintain constant variance, regardless of pool or network size. This necessitates using correct transitions during difficulty changes. I believe the above is the correct way to transition. Shares after difficulty change are worth proportionally less than shares before change, because they are instantaneously less likely to solve the block. People complain why PPS dropped so much so quickly - it all has to be tied directly into the difficulty. This method also accurately handles difficulty decreases.

Finally, here is my reference implementation of the constant-share pool, written in perl. The first part is the actual algorithm demonstrating the details of the rollover and the difficulty changes. The second part is a simulation of a typical pool that uses constant-share payments. Run the program to see the pool operate. Notice the memory usage: the stack takes up about ~100 bytes per share, or 400MB for a full cache at current 400k difficulty - manageable even in the implementation's current state. Of course, perl is memory-inefficient, and in a production implementation, the stack should only take up 4 bytes per share (a single integer user id), be stored in a database, and be periodically saved to disk. As difficulty increases, the stack would grow, but as pools decide to increase work difficulties higher than 1 to relieve pool load, fewer shares will need to be cached for payout calculations.

[HanSolo]

Hi @anon4758!

I independently came up with something similar in an Eligius discussion, called it Pay-Per-Last-N-Shares (PPLNS), and documented it on the Eligius wiki..

http://eligius.st/wiki/index.php/Pay-Per-Last-N-Shares

Notably I don't think it changes the math if..

1. the 'lookback' is any consistent multiple of the difficulty

2. shares get paid multiple times in the event of quick hits, rather than searching back for the last still-unpaid shares

In both cases, the expected return for the marginal next-contributed share remains the same. Having a larger lookback can lessen some miners' unease with the idea recent shares get paid nothing. Paying shares within the lookback multiply puts a hard cap on the amount of sharedata to cache to calculate payouts.

I hadn't considered difficulty transitions but your adjustment factor may be the right thing.

The startup issue seems so small I would simply suggest that until N shares are contributed, the pool can be proportional.. an extra bonus for founding miners if a quick hit is made.

Similarly the shutdown issue.. a surprise shutdown doesn't benefit anyone disproportionately.. all participants pro-rata experience loss of expected return from shares never collected.