Mine in multiple pools to reduce variance

[Meni Rosenfeld]

tl; dr: If you distribute your hashrate among multiple pools in proportion to their respective total hashrate, your variance will be as if you mine in one pool with the total hashrate of all pools.

This is a point I was trying to make in AoBPMRS (section "Proxy mining") but I get the feeling people miss that so I import it as a forum post.

Mining pools are used primarily to reduce variance, and the larger the pool, the more effective it is for this purpose. There is a simple way to decrease the variance further: Mine in multiple pools. For example, if EMC is 450 GH/s and Ozcoin is 700 GH/s you should use 40% of your compute power to mine for EMC and 60% to mine for Ozcoin, and then your variance will be as if you mined for a single 1150 GH/s pool. It doesn't need to be exact, of course; if in this case you do a 50%/50% split, it will be like a 1100 GH/s pool.

This not only helps variance for individual miners, but is healthier for the network. In the current standard usage, there is a "the rich get richer, the poor get poorer" tendency where larger pools are more attractive and thus grow even larger, and all else being equal, the equilibrium is a single huge pool (thankfully, all else is not equal). If miners adopt the proposed strategy, the tendency will be to maintain the status quo distribution, so pools can rise and fall based on their merits. Miners will enjoy the low variance of a single huge pool, without the centralization of power problem.

I'm not an expert on how to technically implement this but it shouldn't be much of a problem. If you have multiple GPUs you can simply point each to a different pool. I think there is local proxy software that allows you to split your hashrate between multiple pools, and if not one can be made.

Miners will probably not want to bother with creating accounts in many pools so will only use 2-3 pools. Ideally, there will be proxy pools that optimally distribute hashrate among as many hopping-proof pools as possible, allowing miners to enjoy low variance with a single pool account.

Appendix: More generally, if you put a portion of p1 of your hashrate at a pool with total hashrate H1, and so on, your variance will be like in a pool with hashrate of 1 / (p1^2/H1 + ... + pn^2/Hn). That the optimal value of H1+...+Hn is obtained when pi = Hi / (H1+...+Hn) is left as an exercise to the reader.

Pools which have other advantages (for example, take less fees or absorb variance with their reward method) should be given a higher weight than their hashrate indicates.

[1713978122]

1713978122

[1713978122]

1713978122

[Mousepotato]

In for later...

[Red Emerald]

Interesting.  Now we just need a miner that makes this easy to do.

[enmaku]

Seems to me that something like bitHopper could be tweaked to do this with the timesliced scheduler pretty easily. It's intended for pool hopping but if you disable all of the hoppable pools and just timeslice between the "fair" pools it should have the effect you're after.

[Sukrim]

Well, shouldn't be too hard to modify a poolhopping client like Bithopper to hop based on valid shares submitted instead of round time...

In a world where every block is solved by a pool, you would get a payout on every block by mining at every pool in every round. In reality you can at least squash variance low enough to make it seem you're mining PPS every day or so I guess (= payouts per 24h are close to the theoretical value).

@enmaku:
Seems we both have the same idea...

[DeathAndTaxes]

One caveat is cgminer seems to have inefficiencies when using in round robin mode.  Maybe that will be improved in future versions but a proxy server should work fine.

[check_status]

If you have 2 rigs, 1 hashing 1500 MH/s and the other hashing at 2400 MH/s, you could point the 1 1500 rig at p2pool and the 2nd 2400 rig at EMC. Each rig has it's own instance of a miner, so just point each at a different pool. This is what I get out of the OP.

Is the proxy method being suggested for single rig miners?
What would be the minimum hashing ability that would benefit from splitting between 2 pools? Like those with one 6850/5830/5850?

[Meni Rosenfeld]

If you have 2 rigs, 1 hashing 1500 MH/s and the other hashing at 2400 MH/s, you could point the 1 1500 rig at p2pool and the 2nd 2400 rig at EMC. Each rig has it's own instance of a miner, so just point each at a different pool. This is what I get out of the OP.

Is the proxy method being suggested for single rig miners?

These are two means to the same end. The proxy method is more general, more accurate and potentially simpler. Manually pointing each rig to a different pool is more of a workaround if that's unavailable.

What would be the minimum hashing ability that would benefit from splitting between 2 pools? Like those with one 6850/5830/5850?

Assuming the pools use a reward method with moderate variance such as PPLNS with X=1, the miner needs to have a hashrate of at least on the order of H/D (where H is the pool hashrate and D is the difficulty) to see significant gains. So for example if the pool is 1 TH/s and D=1.5M then the miner needs to be more than about 1 MH/s, which is pretty much everyone.

[Red Emerald]

So will any changes need to be made to bithopper's code? Or should the user.cfg be enough?

I like this idea, not only for reducing variance, but also for having a bunch of backup pools (and of course p2pool )

It also looks like someone has done some math for us already.  https://bitcointalk.org/index.php?topic=77000.0

[grue]

brilliant!

[Red Emerald]

So bithopper needs some tweaks to work well with p2pool.  I was getting a lot of rejects.  Anyone have any experience with this?

[jkminkov]

edit user.cfg with an entry line for each pool

percent:XX

where XX = 100 / # of pools enabled

for p2pool you should use p2pmining as it is the only node offering difficulty 1 shares, if you don't use diff 1 shares BH will end in a mess as it hops between pools counting shares(diff 1 shares)

[Red Emerald]

edit user.cfg with an entry line for each pool

percent:XX

where XX = 100 / # of pools enabled

for p2pool you should use p2pmining as it is the only node offering difficulty 1 shares, if you don't use diff 1 shares BH will end in a mess as it hops between pools counting shares(diff 1 shares)

p2pmining.com would work, but it isn't needed.  You can just add "/1" after your worker's username and that does the same with the current version of p2pool.

However, I did set difficulty 1 shares and I still got a horrible reject rate.  I think its because of the massive amounts of longpolls.

[JayCoin]

edit user.cfg with an entry line for each pool

percent:XX

where XX = 100 / # of pools enabled

for p2pool you should use p2pmining as it is the only node offering difficulty 1 shares, if you don't use diff 1 shares BH will end in a mess as it hops between pools counting shares(diff 1 shares)

p2pmining.com would work, but it isn't needed.  You can just add "/1" after your worker's username and that does the same with the current version of p2pool.

However, I did set difficulty 1 shares and I still got a horrible reject rate.  I think its because of the massive amounts of longpolls.

Use +1 not /1 to the end of your username.

[Red Emerald]

edit user.cfg with an entry line for each pool

percent:XX

where XX = 100 / # of pools enabled

for p2pool you should use p2pmining as it is the only node offering difficulty 1 shares, if you don't use diff 1 shares BH will end in a mess as it hops between pools counting shares(diff 1 shares)

p2pmining.com would work, but it isn't needed.  You can just add "/1" after your worker's username and that does the same with the current version of p2pool.

However, I did set difficulty 1 shares and I still got a horrible reject rate.  I think its because of the massive amounts of longpolls.

Use +1 not /1 to the end of your username.

Oh nice. Thanks.

Edit: Nope Still > 30% rejects.

[DeathAndTaxes]

Edit: Nope Still > 30% rejects.

To clarify you are getting low reject rate using p2pool directly but higher reject rate when using the proxy?

If so the proxy may be adding too much latency.  Is the reject rate higher on other pools when also using p2pool?  If so the proxy may be getting "confused" by p2pool LP (~98% of p2pool LP involve no block change).

[Red Emerald]

Edit: Nope Still > 30% rejects.

To clarify you are getting low reject rate using p2pool directly but higher reject rate when using the proxy?

Correct

If so the proxy may be adding too much latency.  Is the reject rate higher on other pools when also using p2pool?  If so the proxy may be getting "confused" by p2pool LP (~98% of p2pool LP involve no block change).

I'm pretty sure it has to do with LP since the reject rate is horrible for all the pools when p2pool is enabled. I don't know what to do to BitHopper to make it not confused. I was going to try a different mining proxy, but none of them seemed actively maintained.

[enmaku]

Aside from the p2pool issue, does anyone have results to report?

[organofcorti]

This is a really important post right now, so I'm bumping this.

GHash.IO's recent hashrate increases mean most small pools are taking a big variance hit. As miners miners decide they can't take the variance on the smaller pools, they will move to the larger pools increasing the larger pools proportion of the network even further.

So now is a good time to remind miners that "load balancing" - mining on multiple pools, even small ones - will significantly reduce your variance.

Edit: Maybe this should also be stickied?

[Meni Rosenfeld]

As miners miners decide they can't take the variance the variance on the smaller pools

Duplicate duplicate?

So now is a good time to remind miners that "load balancing" - mining on multiple pools, even small ones - will significantly reduce your variance.

Are there software tools to facilitate this? To use the smaller pools effectively they need a smaller proportion of the miner's hashrate.

Anyway eventually a switch to Multi-PPS will be needed.