Miner Variance

[optimator]

How do you calculate the possible hashing variance of a miner?

Say to have an Avalon that's rated at 66 GH/s. You are on a pool giving you diff 64 shares.

What would be the minimum and maximum hash rates you'd expect to see?

[DrG]

Has rates won't vary on your end (other than +/- 2-3 MH/s)

Your submitted shares per minute will vary.  In CGMiner that's the "U" that's in the top right.  About 1 share per minute for every 71 MH/s, so a 600MH/s miner would average about 8.5 shares/minute.

That's the expected average.  Sometimes you'll submit 2 shares per minute, sometimes you'll submit 15.  That's why what the pool shows your hashrate to be will vary - it has to guess based on your submitted share count.  This won't show much variance after about an hour.

Does that answer your question?

[optimator]

Has rates won't vary on your end (other than +/- 2-3 MH/s)

Your submitted shares per minute will vary.  In CGMiner that's the "U" that's in the top right.  About 1 share per minute for every 71 MH/s, so a 600MH/s miner would average about 8.5 shares/minute.

That's the expected average.  Sometimes you'll submit 2 shares per minute, sometimes you'll submit 15.  That's why what the pool shows your hashrate to be will vary - it has to guess based on your submitted share count.  This won't show much variance after about an hour.

Does that answer your question?

Let me try my question again  

If you are solo mining the probability of finding a block is constant, but there is variance for when you will find the block. In a pool with Diff 1 shares the variance is almost eliminated.

My thinking is that with Diff64 shares there is a noticeable difference in variance (that is, when the diff64 share is reported to the pool). That variance will be reflected in the pool's reported hash rate (since pools calculate the hash rate based on submitted work).

For example, the Avalon is rated at 66,000 MH/s, but the pool reports between 63,000 MH/s and 71,000 MH/s.

My question is how can I calculated the anticipated pool reported hash rate?

There was a post, that I now can't find, that describe the effect variance had on pool dollar reserve requirements for PPS. That post had the math I think I need to calculate the anticipated pool hash rate for a give hash rate and share difficulty.

EDIT: here's the link to the paper I mentioned - http://arxiv.org/pdf/1112.4980.pdf

[organofcorti]

I'll have a bit more time to answer questions later on, but until then have  a read of this:

http://organofcorti.blogspot.com/2012/10/71-variable-pool-difficulty.html

It also explains why you see miner variance, and links to a look up table confidence intervals for hashrates.

[....]
EDIT: here's the link to the paper I mentioned - http://arxiv.org/pdf/1112.4980.pdf

There's a newer version of Meni's paper at https://bitcoil.co.il/pool_analysis.pdf

[optimator]

I'll have a bit more time to answer questions later on, but until then have  a read of this:

http://organofcorti.blogspot.com/2012/10/71-variable-pool-difficulty.html

It also explains why you see miner variance, and links to a look up table confidence intervals for hashrates.

[....]
EDIT: here's the link to the paper I mentioned - http://arxiv.org/pdf/1112.4980.pdf

There's a newer version of Meni's paper at https://bitcoil.co.il/pool_analysis.pdf

Wow! So, a 66Gh/s miner at a 64 difficulty would submit (66*10^9*60)/(2^32*64) = 14 Shares / Minute

Using your first table, submitting 14 shares / minute, over an hour would result in -6.7% to +6.8% variance. Or an observable hash rate of 61 Gh/s to 70 Gh/s.

I suppose the key would be to know what timeframe a pool is using for reporting the miner hash rate.

Thanks!

[optimator]

In CGMiner that's the "U" that's in the top right. 

So now, using the "U" in cgminer, defined as Shares per Minute, and using the 95% confidence threshold for variance, I could expect about a 50% swing either way.

[bcpokey]

On a practical level, reported hash rate is inconsequential. Every pool system relies on shares submitted to calculate rewards. Hashrate is simply a human vanity.

[optimator]

On a practical level, reported hash rate is inconsequential. Every pool system relies on shares submitted to calculate rewards. Hashrate is simply a human vanity.

Except pool reported hash rate is determined by submitted shares. The idea is to try to proactively determine a failing miner before it goes to zero.

If you are able to determine the expected variance of a miner then anything outside that variance is cause for concern.

[bcpokey]

On a practical level, reported hash rate is inconsequential. Every pool system relies on shares submitted to calculate rewards. Hashrate is simply a human vanity.

Except pool reported hash rate is determined by submitted shares. The idea is to try to proactively determine a failing miner before it goes to zero.

If you are able to determine the expected variance of a miner then anything outside that variance is cause for concern.

Again, on a practical level, if you are simply worried about a failing miner, then you could simply look at time since last share submitted. Estimated hashrate being more or less worthless. Your actual hashrate is zero when a miner fails, regarldess of what is being reported. Unless you mean something strange by "failing miner", as in it is stuttering or something, I don't know.

[optimator]

Again, on a practical level, if you are simply worried about a failing miner, then you could simply look at time since last share submitted. Estimated hashrate being more or less worthless. Your actual hashrate is zero when a miner fails, regarldess of what is being reported. Unless you mean something strange by "failing miner", as in it is stuttering or something, I don't know.

Yes, close to stuttering. Imagine an Avalon unit with 3 modules. How do you know that all three units are hashing? What if a chip on one module fails?

Again, the idea is to detect the failing unit before it's hash rate hits zero.

[jaywaka2713]

Again, on a practical level, if you are simply worried about a failing miner, then you could simply look at time since last share submitted. Estimated hashrate being more or less worthless. Your actual hashrate is zero when a miner fails, regarldess of what is being reported. Unless you mean something strange by "failing miner", as in it is stuttering or something, I don't know.

Yes, close to stuttering. Imagine an Avalon unit with 3 modules. How do you know that all three units are hashing? What if a chip on one module fails?

Again, the idea is to detect the failing unit before it's hash rate hits zero.

What do you plan on doing if one unit happened to actually fail?

[optimator]

What do you plan on doing if one unit happened to actually fail?

Good question! It depends on what fails. PDU, Power Supplies, and the TP-Link can be replaced. If the control unit fails, I think boards could be moved to other units - I'm not sure but I *think* the Avalon will run with 4 modules. So some shuffling could be done. Worst case would be to try and have someone repair the board.

[organofcorti]

I'll have a bit more time to answer questions later on, but until then have  a read of this:

http://organofcorti.blogspot.com/2012/10/71-variable-pool-difficulty.html

It also explains why you see miner variance, and links to a look up table confidence intervals for hashrates.

[....]
EDIT: here's the link to the paper I mentioned - http://arxiv.org/pdf/1112.4980.pdf

There's a newer version of Meni's paper at https://bitcoil.co.il/pool_analysis.pdf

Wow! So, a 66Gh/s miner at a 64 difficulty would submit (66*10^9*60)/(2^32*64) = 14 Shares / Minute

Using your first table, submitting 14 shares / minute, over an hour would result in -6.7% to +6.8% variance. Or an observable hash rate of 61 Gh/s to 70 Gh/s.

I suppose the key would be to know what timeframe a pool is using for reporting the miner hash rate.

Thanks!

Most pools have that information somewhere, but it varies by the pool.

If you can use the Gnu Scientific Library, you can calculate the confidence intervals for yourself using gsl_cdf_poisson_Q (unsigned int k, double mu).

[optimator]

If you can use the Gnu Scientific Library, you can calculate the confidence intervals for yourself using gsl_cdf_poisson_Q (unsigned int k, double mu).

And this is where I wish I would have taken more than the basic statistics course 

Thank's for your input, it was incredibly helpful! I'll read up a bit more (enough to become dangerous) on Poisson Distribution and play around with the variables.

Cheers!

[organofcorti]

If you can use the Gnu Scientific Library, you can calculate the confidence intervals for yourself using gsl_cdf_poisson_Q (unsigned int k, double mu).

And this is where I wish I would have taken more than the basic statistics course 

Thank's for your input, it was incredibly helpful! I'll read up a bit more (enough to become dangerous) on Poisson Distribution and play around with the variables.

Cheers!

PM me if you need help.

[eleuthria]

Most pools (from my observations, note this may be very outdated now that pools have vardiff) display miner hash rates as either a 15 minute or 1 hour average.  Some use slightly different windows.  BTC Guild for example uses a variable window based on available data, starting with just 2 minutes worth of shares, and extending up to a full hour [and then the graphs page can give you 4-hour average hash rates].

However, hourly variance *shouldn't* mean much to a miner.  Most pools have vardiff configured in a way the leads to a ~1% +/- variance on the miner's share submissions over the course of a day assuming you're mining at a point where vardiff is needed.  In comparison, most GPU miners are seeing only ~6 shares per minute from a 420 MH/s GPU, which is a ~2% +/- miner variance in 24 hours.

Obviously you're still at the mercy of pool variance for non-PPS mining as well.

[organofcorti]

Most pools (from my observations, note this may be very outdated now that pools have vardiff) display miner hash rates as either a 15 minute or 1 hour average.  Some use slightly different windows.  BTC Guild for example uses a variable window based on available data, starting with just 2 minutes worth of shares, and extending up to a full hour [and then the graphs page can give you 4-hour average hash rates].

However, hourly variance *shouldn't* mean much to a miner.  Most pools have vardiff configured in a way the leads to a ~1% +/- variance on the miner's share submissions over the course of a day assuming you're mining at a point where vardiff is needed.  In comparison, most GPU miners are seeing only ~6 shares per minute from a 420 MH/s GPU, which is a ~2% +/- miner variance in 24 hours.

Obviously you're still at the mercy of pool variance for non-PPS mining as well.

The tricky thing this guy wants to do though is figure out if his miner is starting to fail. In this case the daily average is too late - he needs something that will let him know, real time, whether or not his submission rate is falling outside a particular confidence interval. So he needs to find out how many shares he's submitting (on average) in the time during which the hashrate is being averaged. As soon as the average hashrate drops lower than the calculated confidence interval, he'd get an alert.