BTC Guild has 4% efficiency/stales/fee problem?

[cosurgi]

Hi,

I did some analysis of pool efficiency for: Tycho, BTCGuild, slush and luke-jr, and I have found a problem with BTCGuild. I think I need your help to verify this, to be sure that I make no mistake here.

Here is what I did:
download all rounds history for each pool:

• Tycho: https://deepbit.net/stats , in crontab:

Code:

5       0            *     *     *    /root/bin/getTycho 2>&1 > /dev/null

, where getTycho script is

Code:

wget -O /backup/btc/deepbit-`date +"%Y-%m-%d"`.log 'http://deepbit.net/stats.php'

• BTCGuild: https://www.btcguild.com/all_blocks.php
• slush: same thing with cron, but on http://mining.bitcoin.cz/stats/
• luke-jr: http://eligius.st/~artefact2/blocks/

I did setup cron about half a year ago, to be able to verify pool stats some day. And I didn't setup it for BTCGuild because they provide all_blocks.php webpage with full history. Now it's only them that appear to be problematic in those calculations. I suppose that other pool owners could also provide their history to verify my crontab downloading.

Next thing, the calculations for each pool are following:
i - round number,
j - difficulty index (all rounds with the same difficulty have the same j) ):

1. calculate average pool's luck as sum_by_i(round_i_shares)/sum_by_i(round_i_difficulty), expected number is 1.0
2. calculate standard deviation as sqrt( sum_by_i(difficulty_i*difficulty_i) ) / sum_by_i(difficulty_i)=sqrt(sum_by_j(number_of_rounds_with_difficulty_j * difficulty_j*difficulty_j))/sum_by_j(number_of_rounds_with_difficulty_j * difficulty_j), value depends on number of rounds

You can verify all my calculations by downloading this openoffice spreadsheet: https://sites.google.com/site/cosurgi/check-pools.ods

To summarize here are all results:

• Tycho: 0.9949 +/- 0.011, which gives 0.51% pool luck with 1.1% error
• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error
• slush: 0.9999 +/- 0.016, which gives 0.01% pool luck with 1.6% error
• luke-jr: 0.9602 +/- 0.040 which gives 4% pool luck with 4% error

luke has biggest error uncertainty because he has only 770 rounds. But his luck still fits in standard deviation.

[1714475790]

1714475790

[1714475790]

1714475790

[1714475790]

1714475790

[1714475790]

1714475790

[slush]

(As I'm one of operators, I won't comment results, however I'm subscribing this thread)

[cosurgi]

To get difficulty of blocks, when pool didn't provide block number but hash I was using http://blockexplorer.com/ like that:

http://blockexplorer.com/block/00000000000008bf00635be7fba910747e69b383db828ddcdd7ec2b41975764d

There you can see "Difficulty". I downloaded all blocks from blockexplorer for each pool, with a short script, then filtered them on "Difficulty" with grep, sorted it by hash and pasted into spreadsheet. Then I sorted pool stats by hash. And compared if those hashes are equal. You can exactly see this in tycho's case, Because I wrote extra column for hash comparison, which writes "OK" when they match. For others I didn't need to do this, because there were fewer blocks. In tycho there were 12000

[kano]

Just a minor point, when difficulty changes, it means the pool rounds may have 2 difficulties, not just one.
The longer the pool rounds, the more likely.
You'd have to break it down at the per block level (easiest) or break those rounds that cross the difficulty change or calculate a difficulty for those rounds (worst idea of the three )

[cosurgi]

Just a minor point, when difficulty changes, it means the pool rounds may have 2 difficulties, not just one.
The longer the pool rounds, the more likely.
You'd have to break it down at the per block level (easiest) or break those rounds that cross the difficulty change or calculate a difficulty for those rounds (worst idea of the three )

currently I was using the difficulty of generated block. And this difficulty is in denominator so it makes the resulting fraction `shares/diff` to be smaller when difficulty goes up (which was the case most of the time). So this approach makes the pools a little more lucky than in reality. But total effect on pool luck is very small, like 0.001 or something like that. It's easy to check - get that openoffice spreadsheet, change difficulties of those blocks and see what is the result.

Maybe somebody can calculate for arsbitcoin too? I have them archived also, but didn't have time yet to calculate.

[DeathAndTaxes]

Interesting.  At first glance having bad luck 3x standard deviation looks pretty bad.  I will reserve final judgement until I look at the workflow but statistically peaking there is only a  1/2000 chance of a pool being that unlucky.   

OP you got a donation address?
I hope this is something you intended to do periodically.

[shads]

-ve luck could be caused by a pool accepting shares that shouldn't have been.  e.g. duplicates or if there is a delay between new block on the network and the pool detecting it and marking all new shares issued before that time as stale.

conversely +ve luck could be caused by pool rejecting shares that are valid (but still potentially submitting to bitcoin network).  e.g. pools share cache has expired the work so the share is counted as unknown.   Or blkmond detecting a new block before bitcoind is ready to serve work from the new block so longpolls go out with 'new' work from previous block but when they come back as a share it's rejected as stale due to prevblockhash not matching.

[Vladimir]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

[Nachtwind]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

GO back to the hole you crawled out of, please.. this has been dealt with and you were proven time and again to have simply miscalculated your stuff.

[Vladimir]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

GO back to the hole you crawled out of, please.. this has been dealt with and you were proven time and again to have simply miscalculated your stuff.

yep, and still it somehow matches the above closely enough and no credible information about this "miscalculation" have ever been posted or brought to my attention.

[kano]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

GO back to the hole you crawled out of, please.. this has been dealt with and you were proven time and again to have simply miscalculated your stuff.

yep, and still it somehow matches the above closely enough and no credible information about this "miscalculation" have ever been posted or brought to my attention.

So would you wonder why he is moving to only PPS soon when, according to you, he is scamming people in the pool not using PPS?

Try facts rather than complaining about random numbers being random

P.S. the pool in your sig has shut down.

[Vladimir]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

GO back to the hole you crawled out of, please.. this has been dealt with and you were proven time and again to have simply miscalculated your stuff.

yep, and still it somehow matches the above closely enough and no credible information about this "miscalculation" have ever been posted or brought to my attention.

So would you wonder why he is moving to only PPS soon when, according to you, he is scamming people in the pool not using PPS?

Re scamming, your words not mine. I never said that. Switch to PPS proves nothing. Go get some logic classes.

Too bad MMC pool is down. Nobody is perfect it seems.

[kano]

• BTCGuild: 1.0442 +/- 0.018, which gives 4.4% pool problems with 1.8% error

Surprise, surprise! see my sig.

GO back to the hole you crawled out of, please.. this has been dealt with and you were proven time and again to have simply miscalculated your stuff.

yep, and still it somehow matches the above closely enough and no credible information about this "miscalculation" have ever been posted or brought to my attention.

So would you wonder why he is moving to only PPS soon when, according to you, he is scamming people in the pool not using PPS?

Re scamming, your words not mine. I never said that. Switch to PPS proves nothing. Go get some logic classes.

Too bad MMC pool is down. Nobody is perfect it seems.

Well your sig says: 108-is-your-pool-cheating-you-essential-self-defence-course-for-bitcoin-miners-part-1
And you post about this regularly and BTC Guild is your target of the 4% accusation ...

So ... unless English isn't well known by you and thus we can excuse the lack of understanding ...
that means the same thing as scamming to anyone but an idiot.

[Vladimir]

Dear Kano,

It is impossible to have a coherent discussion with someone who is unable to build up a logical argument. There is no logical link leading from whatever I ever said using  my imperfect English to your claim that I allegedly said "BTC Guild is cheating someone". BTC Guild cheating someone may or may not be the case, I have not seen any 100% proof of either.

After you take some logic classes go read up on Stockholm Syndrome. Meanwhile, welcome to my ignore list.