Earnings lower than estimate?

[mgio]

I've only been mining for about a month and a half now (45 days). I have an Avalon, two BFL Jalapenos and a BFL FPGA which total about 100 GHash/sec if everything is working correctly.

I've been mining mostly on Ozcoin and I keep very detailed statistics including the daily hashrate for my miners and the payout for every block found for the pool.

Ozcoin charges a 1% fee and uses DGM payout method (there is also POT but I always use DGM). DGM is similar to PPLNS. Well kind of.

I also keep track of the current difficulty and what my expected earnings for each day should be (minus the 1% fees).

I use that to calculate a "luck percentage" where 100% is when I hit my expected earnings target.

What I've noticed is that of the past 45 days only 9 of the days did I ever reach 100% or above. The other 36 days I was below 100%. There are several days where I was 50% or less and only about 2 or 3 days where I was significantly above 100%. Overall, my payout is just 85.59% of what was expected.

Now, I realize there is a huge amount of variance because this pool pays based on blocks found and every pool has unlucky days when there are fewer blocks found than expected.

What I'm trying to figure out is if it is just the case that Ozcoin has been very unlucky the past 6 weeks or if I am calculating something incorrectly. FYI, I am using the hashrate reported to me by Ozcoin to calculate what my expected earnings should be. 85.59% seems particularly bad. I don't see why I would stick with Ozcoin (as much as I do love it) and not switch to a pool like 50BTC where I would be guarenteed to make 97% of my expected earnings. It seems that extra 2% isn't really worth it when the variance can really hurt you.

As far as I can tell, the reasons for the much lower than expected earnings are one of the following:

1) Variance in the pool. Basically, just bad luck. I would have expected this to have averaged out over 45 days but it seems to me to consistently be underperforming. Maybe I haven't waiting long enough. I dunno.

2) The hashrate Ozcoin is reporting for my miners isn't the exact hashrate that I am getting when it comes to the number of valid shares I'm submitting. I'm not sure why this would be the case. My "efficiency" is always about 99.5% across all my miners when it comes to valid shares. Maybe everything is actually correct but I'm not actually doing the 100 GH/s worth of work Ozcoin is telling me.

3) DGM is underpaying because that of the nature of the algorithm (which is more complicated than most). It seems to underpay even when the pools finds the expected number of blocks each day. I'm not sure why.

4) People are cheating the pool. I always wondered how a pool prevented miners from lying and submitting invalid shares. Obviously the pool can't check every single one, right? They would require to have a hashrate as high as all the miners participating in order to do that, correct? The pools seems to have many more days where it doesn't find the expected number of blocks given the pools reported hashrate. What if people are submitting fake shares making the pool like it has more hashrate that it really does and taking a bigger share of the Ozcoin pie.

5) There are other hidden fees I don't know about. Or perhaps I am calculating my expecting earnings incorrectly. My calculations do seem to match up with what ozcoins own utility bots (in its IRC channel) seem to tell me.

6) Ozcoin itself has some problems. The pool has gone down on occasion and has had problems where the statistics were unavailable for a time (though the owners claimed it was still processing shares correctly). There could be some bugs in its software.

Which of these seem most likely? I'm probably going to give another pool a shot for a while and see if it is any better. I do really like Ozcoin though and I want to continue mining there.

What is the experience that other people have when it comes to expected earnings vs actual earnings?

[eleuthria]

1) Variance in the pool. Basically, just bad luck. I would have expected this to have averaged out over 45 days but it seems to me to consistently be underperforming. Maybe I haven't waiting long enough. I dunno.

45 days isn't that long when you're mining on a pool that is a significantly small share of the overall network like Ozcoin.  organofcorti would be needed to give any hard numbers, but being ~15% under in a 45 day period doesn't sound that unusual when dealing with a pool that has only been 2-5% of network hash rate during that time frame.

2) The hashrate Ozcoin is reporting for my miners isn't the exact hashrate that I am getting when it comes to the number of valid shares I'm submitting. I'm not sure why this would be the case. My "efficiency" is always about 99.5% across all my miners when it comes to valid shares. Maybe everything is actually correct but I'm not actually doing the 100 GH/s worth of work Ozcoin is telling me.

Every pool has its own slightly different method of estimating miner hash rates.  In the end they can only guess based on submitted shares.  Beyond that, each pool does their own slight tweaks in terms of how long they examine share submissions (longer time frame = larger sample = more accurate), whether or not they include rejected shares, etc.  If you are *always* showing up lower than your actual hash rate, and it's any kind of significant percentage, then it may be a concern.

3) DGM is underpaying because that of the nature of the algorithm (which is more complicated than most). It seems to underpay even when the pools finds the expected number of blocks each day. I'm not sure why.

DGM *should* pay out at expectation given proper implementation and neutral luck.

4) People are cheating the pool. I always wondered how a pool prevented miners from lying and submitting invalid shares. Obviously the pool can't check every single one, right? They would require to have a hashrate as high as all the miners participating in order to do that, correct? The pools seems to have many more days where it doesn't find the expected number of blocks given the pools reported hashrate. What if people are submitting fake shares making the pool like it has more hashrate that it really does and taking a bigger share of the Ozcoin pie.

The pool can check every single share, and should be.  When you submit a share, that is a single hash you found that is appropriate for difficulty.  The pool only has to do one double-sha256 hash to determine if the share was valid or not, whereas the miner (on average) does ~4.2 billion hashes in order to find that share.

5) There are other hidden fees I don't know about. Or perhaps I am calculating my expecting earnings incorrectly. My calculations do seem to match up with what ozcoins own utility bots (in its IRC channel) seem to tell me.

No hidden fees on Ozcoin.

6) Ozcoin itself has some problems. The pool has gone down on occasion and has had problems where the statistics were unavailable for a time (though the owners claimed it was still processing shares correctly). There could be some bugs in its software.

While pool downtime will hurt your earnings if you don't have a backup, pool downtime should not actually impact overall pool luck.  Mining has no "memory", so if the pool goes down, there was no "progress" towards the next block that was lost.  Having a proper backup pool is always recommended to avoid downtime affecting you, no matter which pool you use.  For the second part, it is always possible that software bugs are causing the downtime, so there's always a chance that can lead to negative trending luck.

Which of these seem most likely? I'm probably going to give another pool a shot for a while and see if it is any better. I do really like Ozcoin though and I want to continue mining there.

What is the experience that other people have when it comes to expected earnings vs actual earnings?

Mining in the pool you like the most (interface, features, etc.) is always preferred for someone who isn't in it as a primary source of income.  The more of a business mining is to you, the more other factors come into play (variance, reliability, notification features, security, etc.).

[dbitcoin]

I've only been mining for about a month and a half now (45 days). I have an Avalon, two BFL Jalapenos and a BFL FPGA which total about 100 GHash/sec if everything is working correctly.

......

What is the experience that other people have when it comes to expected earnings vs actual earnings?

1) Yes, variance is a always a problem, but 45 days not enough.
2) 10% it's normal deviation for short time, and 5% for several hours average value, If pool software not broken and your hashrate it's stable.
3) You just do not understand how DGM work. short round always underpaid, because these money used for long round reward compensation. 
4) You do not able submit fake shares (if pool software not broken), and yes every share is verified.
    You may do not submit winning shares (withholding attack), but you only lose money in this case, if pool use DGM payment method.
5) Nope, if pool op not cheating you
6) Yes, this is a problem and your expected payout depend on that, but if only web fronted is down, this do not affect your reward.

p.s. you should give up, and understand what your expected reward has nothing to do with real long term mining... 

[mgio]

I've only been mining for about a month and a half now (45 days). I have an Avalon, two BFL Jalapenos and a BFL FPGA which total about 100 GHash/sec if everything is working correctly.

......

What is the experience that other people have when it comes to expected earnings vs actual earnings?

1) Yes, variance is a always a problem, but 45 days not enough.

ok, I'll let it go for longer and see what happens. Unfortunately, rising difficulty means it is very unlikely that I will be able to overcome the variance I've seen already in the past.

2) 10% it's normal deviation for short time, and 5% for several hours average value, If pool software not broken and your hashrate it's stable.
3) You just do not understand how DGM work. short round always underpaid, because these money used for long round reward compensation. 

I understand this. But when the rounds are found at their expected rate, shouldn't the payouts be what is expected.

In other words, if the pools hashrate means that you would expect to find a block every 6 hours, and you do find 4 blocks in a day roughly every 6 hours, wouldn't you expect the payouts to match? In other words, wouldn't you expect that each round would payout 1/4 of your expected rate. The amount always seems to be less than that.

4) You do not able submit fake shares (if pool software not broken), and yes every share is verified.
    You may do not submit winning shares (withholding attack), but you only lose money in this case, if pool use DGM payment method.
5) Nope, if pool op not cheating you
6) Yes, this is a problem and your expected payout depend on that, but if only web fronted is down, this do not affect your reward.

p.s. you should give up, and understand what your expected reward has nothing to do with real long term mining... 

I'm not sure what you mean there at the end.

[mgio]

1) Variance in the pool. Basically, just bad luck. I would have expected this to have averaged out over 45 days but it seems to me to consistently be underperforming. Maybe I haven't waiting long enough. I dunno.

45 days isn't that long when you're mining on a pool that is a significantly small share of the overall network like Ozcoin.  organofcorti would be needed to give any hard numbers, but being ~15% under in a 45 day period doesn't sound that unusual when dealing with a pool that has only been 2-5% of network hash rate during that time frame.

I'm going to give it another few months, I suppose. Rising difficulty means that I will never overcome past bad luck though, and I expect I will always be below "average" from now on.

2) The hashrate Ozcoin is reporting for my miners isn't the exact hashrate that I am getting when it comes to the number of valid shares I'm submitting. I'm not sure why this would be the case. My "efficiency" is always about 99.5% across all my miners when it comes to valid shares. Maybe everything is actually correct but I'm not actually doing the 100 GH/s worth of work Ozcoin is telling me.

Every pool has its own slightly different method of estimating miner hash rates.  In the end they can only guess based on submitted shares.  Beyond that, each pool does their own slight tweaks in terms of how long they examine share submissions (longer time frame = larger sample = more accurate), whether or not they include rejected shares, etc.  If you are *always* showing up lower than your actual hash rate, and it's any kind of significant percentage, then it may be a concern.

The hashrate I expect and the hashrate ozcoin reports to me does indeed match up, which is good.

3) DGM is underpaying because that of the nature of the algorithm (which is more complicated than most). It seems to underpay even when the pools finds the expected number of blocks each day. I'm not sure why.

DGM *should* pay out at expectation given proper implementation and neutral luck.

I think DGM is probably fine. I'm guessing it seems to underpay mostly because I suppose Ozcoin has been unlucky the past couple of months and hasn't found the expected number of blocks.
And when there is an "average" day and it finds the number of blocks expected, payouts are lower to make up for large payouts from previous long rounds. I suppose that is what I am seeing.

4) People are cheating the pool. I always wondered how a pool prevented miners from lying and submitting invalid shares. Obviously the pool can't check every single one, right? They would require to have a hashrate as high as all the miners participating in order to do that, correct? The pools seems to have many more days where it doesn't find the expected number of blocks given the pools reported hashrate. What if people are submitting fake shares making the pool like it has more hashrate that it really does and taking a bigger share of the Ozcoin pie.

The pool can check every single share, and should be.  When you submit a share, that is a single hash you found that is appropriate for difficulty.  The pool only has to do one double-sha256 hash to determine if the share was valid or not, whereas the miner (on average) does ~4.2 billion hashes in order to find that share.

The problem here is I don't really understand how pools work. So if someone could explain it to me or point me to some web page that explains it, that would be great. I understand that a miner has a block and choses nonces until they double sha256 hash is under a given a threshold. That much for solo mining makes sense for me. But I guess I really don't know what a "share" is. Is it a set or range of nonces the miner hashes? And how can the pool operator verify a share faster than a miner can computer them? If someone would explain the math behind it to me that would be great. I naively assumed that pools would assign miners ranges of nonces to try and miners would return the hashed "solutions" and the pool would just accept their word for it. I'm guessing the reality is more complicated than that. 

5) There are other hidden fees I don't know about. Or perhaps I am calculating my expecting earnings incorrectly. My calculations do seem to match up with what ozcoins own utility bots (in its IRC channel) seem to tell me.

No hidden fees on Ozcoin.

6) Ozcoin itself has some problems. The pool has gone down on occasion and has had problems where the statistics were unavailable for a time (though the owners claimed it was still processing shares correctly). There could be some bugs in its software.

While pool downtime will hurt your earnings if you don't have a backup, pool downtime should not actually impact overall pool luck.  Mining has no "memory", so if the pool goes down, there was no "progress" towards the next block that was lost.  Having a proper backup pool is always recommended to avoid downtime affecting you, no matter which pool you use.  For the second part, it is always possible that software bugs are causing the downtime, so there's always a chance that can lead to negative trending luck.

The thing is when ozcoin has gone down, it still has accepted shares which has meant my miners have never failed over. So even though the pool looks like it is done (from looking at the webstats), maybe it actually wasn't. But it is unclear whether it was working properly at that time. For example, there were times when the webstats would show everyones hash rate at half of what was expected. Was this due to a network problem that caused everyone to only be able to submit half the number of shares and thus the pool was running at half its expected rate. Or was everything working correctly and just the hashrate being calculated incorrectly. It's hard, if not impossible to tell.

Which of these seem most likely? I'm probably going to give another pool a shot for a while and see if it is any better. I do really like Ozcoin though and I want to continue mining there.

What is the experience that other people have when it comes to expected earnings vs actual earnings?

Mining in the pool you like the most (interface, features, etc.) is always preferred for someone who isn't in it as a primary source of income.  The more of a business mining is to you, the more other factors come into play (variance, reliability, notification features, security, etc.).

I agree. As long as I can get withing a few percent of what I expect in the long run, I'm going to chose the pool that I like the best over one the one that might pay slightly more. But 15% is too much to ignore if it continues. It's meant 9 btc over the past 45 days which is not insignificant.

[dbitcoin]

In other words, if the pools hashrate means that you would expect to find a block every 6 hours, and you do find 4 blocks in a day roughly every 6 hours, wouldn't you expect the payouts to match? In other words, wouldn't you expect that each round would payout 1/4 of your expected rate. The amount always seems to be less than that.

....

I'm not sure what you mean there at the end.

You do not tell a numbers (2, 5, 10, 50% ?). Pool hashrate it's not stable value, so your reward always fluctuate.
I mean - there is too much factors affecting your real reward.

[Graet]

The thing is when ozcoin has gone down, it still has accepted shares which has meant my miners have never failed over. So even though the pool looks like it is done (from looking at the webstats), maybe it actually wasn't. But it is unclear whether it was working properly at that time. For example, there were times when the webstats would show everyones hash rate at half of what was expected. Was this due to a network problem that caused everyone to only be able to submit half the number of shares and thus the pool was running at half its expected rate. Or was everything working correctly and just the hashrate being calculated incorrectly. It's hard, if not impossible to tell.

Hi, maybe if I explain our setup a little it will help your understanding of what happens.
We have 5 mining nodes located in different places worldwide, one database/website server in Australia and another sever with a bitcoind we use for relaying blocks and providing statistics.
If a mining node fails you will fail-over to another node or pool if your miner is correctly setup.
In case of interruption to the internet or issues with db server mining nodes cache shares which are later forwarded and counted.

At times the internet just lags  http://www.internetweathermap.com/map is one site I look at often, when there is lots of orange or red displayed often statistics lag (eg: the day everyone showed 1/2 hashrate there were 4 orange lines connecting Australia to the rest of the world)
the other thing that happens is as shares counting "catches up" hashrates displayed are unusually high.
Earlier this week the whole dc that hosts the separate bitcoind was taken offline when street works cut not 1 but 2 cables to the DC, this broke displayed stats for an extended time but shares were still counted and paid as usual (this did not just affect Ozcoin but all of that datacentre's customers) - this was an unusual situation but we are looking at setting up a 2nd stats bitcoind for situations like these.

Thanks to others helping answer the questions

As an aside the pool wallet doesn't enjoy extended bad luck either. As can be seen here https://ozcoin.net/content/overview-bitcoin we are paying well over 25Btc/round a lot lately, hopefully the swing to good luck is very soon

[mgio]

The thing is when ozcoin has gone down, it still has accepted shares which has meant my miners have never failed over. So even though the pool looks like it is done (from looking at the webstats), maybe it actually wasn't. But it is unclear whether it was working properly at that time. For example, there were times when the webstats would show everyones hash rate at half of what was expected. Was this due to a network problem that caused everyone to only be able to submit half the number of shares and thus the pool was running at half its expected rate. Or was everything working correctly and just the hashrate being calculated incorrectly. It's hard, if not impossible to tell.

Hi, maybe if I explain our setup a little it will help your understanding of what happens.
We have 5 mining nodes located in different places worldwide, one database/website server in Australia and another sever with a bitcoind we use for relaying blocks and providing statistics.
If a mining node fails you will fail-over to another node or pool if your miner is correctly setup.
In case of interruption to the internet or issues with db server mining nodes cache shares which are later forwarded and counted.

At times the internet just lags  http://www.internetweathermap.com/map is one site I look at often, when there is lots of orange or red displayed often statistics lag (eg: the day everyone showed 1/2 hashrate there were 4 orange lines connecting Australia to the rest of the world)
the other thing that happens is as shares counting "catches up" hashrates displayed are unusually high.
Earlier this week the whole dc that hosts the separate bitcoind was taken offline when street works cut not 1 but 2 cables to the DC, this broke displayed stats for an extended time but shares were still counted and paid as usual (this did not just affect Ozcoin but all of that datacentre's customers) - this was an unusual situation but we are looking at setting up a 2nd stats bitcoind for situations like these.

Thanks to others helping answer the questions

As an aside the pool wallet doesn't enjoy extended bad luck either. As can be seen here https://ozcoin.net/content/overview-bitcoin we are paying well over 25Btc/round a lot lately, hopefully the swing to good luck is very soon

Thanks for responding Graet. In your opinion, does it seem like the pool has been having a lot of bad luck the past few weeks or so? It would be nice if there was a stat like Eligius's "luck" that told you how the pool was doing that day or round relative to the estimated rate it would be expected to find blocks. That way I could tell whether it was just bad luck or if my miners were underperforming.

[juhakall]

It would be nice if there was a stat like Eligius's "luck" that told you how the pool was doing that day or round relative to the estimated rate it would be expected to find blocks. That way I could tell whether it was just bad luck or if my miners were underperforming.

http://ozco.in/content/luck-bitcoin
http://ozco.in/content/share-payout-comparison-bitcoin

Looks like the latter needs some work, though. Recent difficulty periods have such small bars because of the huge difficulty spike, so only the tooltips tell anything useful now.