Could BTCGuild be cheating its miners?

[Mad7Scientist]

Here is my advice:
Don't mine on a proportional pool.

On a *PPS pool if the owner steals blocks he will have to cover the losses himself which makes stealing useless.

Although analysis of the luck on BTCGuild (and possibly other pools which could be working with BTCGuild) suggests that there is a very high probability that the day to day luck has been manipulated so it jumps to unnaturally high and low values, it is not possible to be 100% sure of this, just as we can't know that the Dilbert random number generator is truly random. [edit]But we may be able to be 99.8% sure.

update:
You can see the historical luck of several pools here http://l0ss.net/ However, it appears that the site isn't working anymore. It used to show a graph where BTCGuild was quite a bit lower in payout for quite a long time (like a few weeks) and then it went up to normal. I checked back recently and for some reason much of the historical data had been removed so I could only see back for a much shorter time. The bad luck streak of BTCGuild.com was cut off. Today I went back and now the site seems to be gone completely. I wish I would have saved a screenshot.

I told Eleuthria, the owner of BTCGuild, on IRC that if he converted his pool(s) to PPS that it would stop all further suspected cheating or manipulation, but he didn't want to and I got kicked from the channel. He didn't like it because he would have to be responsible for the variation in luck instead of the users of the pool.

I have been mining on bitp.it and it has been working out well for me. [edit 10/2012: bitp.it is now gone but abcpool.co has been working great for a long time] It also uses a custom software that is different from pushpool which is used on other mining pools. Because it uses different software I think it has a lower chance of being a part of some possible mining pool conspiracy. [update:] bitp.it got hit by a massive DoS attack and most of the users left. I wonder why they would get attacked?

The original post starts here:
I'm concerned that BTCGuild.com may be cheating its miners.

When I talked about this on IRC most everyone just made fun of me and used ad-hominem attacks and not a single person even attempted to do any calculations themselves.

When it comes to bad luck in the mining pool, sadly many people have the attitude that is shown in the Dilbert comic below. Well I don't think like that! If i went to the #btcguild IRC channel and said that I thought the random number generator that keeps saying nine over and over wasn't really random they would say something like "it's luck! You can't calculate it's probability!". Then if I said "Yes you can calculate it's probability" they would then go on and on about how I am stupid because I said "it's" instead of "its".


Anyway, from June 22 to June 27 BTCGuild had a 5 day streak of very bad luck. That's when I began to get concerned. Since then they have had another streak of bad luck.

The 5 day streak of bad luck:
Block statistics: https://www.btcguild.com/all_blocks.php
My calculation started: Block 137428   2011-07-22 02:49:15
My calculation ended:  Block 138235   2011-07-27 02:25:29

With the constant difficulty during that time and the 2250 GH figure I used for hashing power I figured they should have gotten 133 solved blocks, and solved a block every 53m47s. They only got 105 blocks. That's 28 missing blocks, or $19,000. About 807 blocks went by in that time. That number is important because you need to know the number of trials. Probability (p) of finding a block is 0.165 (that's 133/807).

Edit: I believe I made a mistake. I should have been using Poisson instead of Binomial Distribution. So my 0.0036 (277:1) figure was incorrect it should have been 0.00695 (144:1) as Vladimir said.

The probability of that happening is 1:277. That means that there is a 1 in 277 chance of that happening on any given 5 day period. So it should happen every 4 years. Then you have to balance that with the probability of the owner of the pool being honest. I know that a lot of people will not hesitate to steal if there are no legal ramifications for doing so, such as in a situation like this. Let's say there is a 1:10 chance that the owner of the pool is dishonest, or that the pool has been hacked. Now if we were to look at just a single 5 day segment we would compare (multiply) the 277:1 odds in favor of manipulation to the 1:10 chance of dishonesty and we would get 27:1 odds that the pool is stealing from us. Now really it isn't 27:1 because BTCGuild has been around for a lot longer than 5 days (it started 2011-5-9). But it is highly unlikely that somebody would start stealing from the pool when it was band new, and I haven't calculated anything before this. If we look at a 30 day window we could say the odds of having this 5 day streak of bad luck is 1:46, so 46:1 * 1:10 = 4.6:1 odds that there is stealing going on.

Even 1 in 2 odds that there is stealing going on is more than enough for me to move to a different pool.

You can put those numbers in the Binomial Calculator here to do it yourself. http://stattrek.com/tables/binomial.aspx . When you have a set of trials and successes with a known probability of having a success on each trial, the binomial calculator can tell you the odds of a certain amount of deviation occurring. Insurance companies would use this to analyze data from factory worker accidents. If there is an increase in factory worker accidents they would want to know what the probability of that happening by chance is verses the probability of it being that the working conditions in the factory have become more dangerous.

I think when I calculated a single day I found that the chance of having -40% luck on a single given day was around 2%. Somebody else can calculate again it's getting late.

Now shortly after that 5 day streak of very bad luck, the luck suddenly shot up to 70% for a day! I found that kind of odd. It could have just happened by chance, or it could be manipulation. Diverting some mining power away from the pool in to your own private pool is a piece of cake, but adding blocks to the list of blocks that the pool solved is not! So it could have just happened that way by chance. Or BTCGuild could be claiming blocks that were solved by solo miners or another pool (if you see another pool claiming to have solved the same block then we can know for sure!). Or BTCGuild could be colluding with another pool and that pool would have had bad luck during the time that BTCGuild's luck went positive because blocks from that pool would be given away to BTCGuild. In any case I really don't know.

Now lately, the luck on BTCGuild.com has gone way down again over a several day period. So now this is looking serious.

Now when I calculated those 5 days I just put everything together and calculated over 5 days. If I had calculated the odds for each day independently and then multiplied them together (using some special statistical multiplication that I don't know how to do) the probability would have been even worse since the probability increases exponentially as the number goes away from the expected deviation. It's like Root Mean Square verses normal mean average. For instance, if there was a single day with -100% luck the odds of that happening would be astronomical but if I just calculated out over 5 days with 0%, 0%, 0% -100%, 0% it would just look like -20% every day and it would look bad but not really bad. Better yet would be to calculate each block and then put all those numbers together somehow. Of course you would need to know the difficulty and hash rate for each block if you were doing it over a long period of time or if the hash rate wasn't constant enough. But I am not skilled enough to do that so I was hoping someone else would.

I know that each time the luck deviates, either positive or negative, from the expected positive or negative deviation it exponentially increases the probability that there is manipulation going on. Maybe somebody will know how to do these kinds of calculations, especially knowing how to take multiple independent calculations of probability and be able to combine them.

BTCGuild.com pool chart: https://www.btcguild.com/pool_charts.php

As you can see from the Dilbert cartoon, with randomness we can never be sure. But I think the metaphorical random number generator on BTCGuild.com has said "NINE" 5 times in a row now.

Update:
I posted this later on but I'll edit it in again here because I'd like to have the important things all on one page. This is important for explaining how positive luck would be added to the pool. Somebody else mentioned that on the days that BTCGuild was low on luck some of the other pools were having good days.

Here is what I think may be happening after thinking about it for a little while. BTCGuild and some other pools, such as Deepbit, or Slush or Ars Bitcoin are actually run by the same people.

The operator of BTCGuild is intentionally manipulating the luck in the pool making it drop very low and then jump up to unexpected highs. On low luck days, they would be doing this by having the pool generate some shares which are valid for a different pool and then capturing the winning shares and sending them over to the other pool where that share would be sent out to one of the miners to win. On high luck days, the reverse would happen.

During this time, some of the winning shares may be sent off to a private pool where some of the loot is kept, or are sent off to a PPS pool like Ars Bitcoin and help build up that big 800 BTC buffer that they have. Then somebody gets to keep that buffer.

But the amount of actual stealing that takes place depends on weather or not people notice. By manipulating the pool and adding positive luck, it really a great way to make most people not notice that something is wrong. So right now there is probably very little stealing actually going on but there is still a high degree of manipulation!

As far as I know, this share trading scheme is possible. Nobody has yet told me why it couldn't work. I think it would also be possible to simply reroute hashing power between pools.

[Maged]

OR someone is trying to kill the pool by withholding shares that meet the network difficulty.

[JoelKatz]

By my math, such an attacker would have to have at least 1/7th of the pool's hashing power.

[silverchair]

Could be.

[brocktice]

I left btcguild recently because of this. Could be a technical problem too.

[noob_jul11]

I left BTCGUILD.com for the same reason. I noticed the beginning string of "bad lucks" back when they implemented the 1 hour delay announcement to avoid pool hoppers. If they can delay announcement, can they "forgot" to announce?

Edit: And now, high stale shares - https://bitcointalk.org/index.php?topic=35507.0

[brocktice]

I left BTCGUILD.com for the same reason. I noticed the beginning string of "bad lucks" back when they implemented the 1 hour delay announcement to avoid pool hoppers. If they can delay announcement, can they "forgot" to announce?

Edit: And now, high stale shares - https://bitcointalk.org/index.php?topic=35507.0

Could be botnet miners, they cause high load and generally screw with things.

[Vladimir]

With the constant difficulty during that time and the 2250 GH figure I used for hashing power I figured they should have gotten 133 solved blocks, and solved a block every 53m47s. They only got 105 blocks. That's 28 missing blocks, or $19,000. About 807 blocks went by in that time. That number is important because you need to know the number of trials. Probability (p) of finding a block is 0.165 (that's 133/807).

If we assume that you are correct on expectation of 133 while only 105 blocks found (I have not checked it) than probability of finding at most 105 blocks out of expected 133 is 0.6952%. This is rather high certainty i.e. more than 99.3% that something is badly wrong there. I'd suggest anyone to jump the ship ASAP (I would).

Check out http://mining.mainframe.nl my math shows that no cheating going on there.

On longer time period it is very easy to spot cheating. Take as long period as you have data for. Calculate expected number of blocks solved, calculate actual number of blocks solved. It should converge. The law of large numbers is working. A few such bad streaks as described above and certainty that a pool is cheating quickly goes over 99.99%.

[noob_jul11]

I had posted July 30th https://bitcointalk.org/index.php?topic=32933.0 where they had an 11 million shares round of no block found.

[Vladimir]

I had posted July 30th https://bitcointalk.org/index.php?topic=32933.0 where they had an 11 million shares round of no block found.

There is not enough data to make conclusions. For 11 million shares (5.7 blocks expected, at most  1 found) probability is about 2% i.e. this is expected to happen every 50*11 = 550 million shares or so.

It is when we are talking about hundreds of blocks the probabilities could get much more damning quickly due to law of large numbers.

[brocktice]

I have a lot of respect for El and the other guys at BTCguild, and they claim they're running standard pushpool and there's nothing they can do about the bad luck.

I believe them.

Assuming that is true, what else could be causing such a run of bad luck, both in terms of technical glitches and malicious activity?

EDIT:, I'm an idiot, and I've been looking at the wrong chart and not paying attention to differences in scale bars. Ignore me.

[eleuthria]

My silence on this matter has been the result of having 0 control over it.  I made my posts last time these threads started [which went away immediately when a few days later our luck surged the opposite direction for ~34 hours].

BTC Guild uses the following:

  Pushpool 0.5.1 [only modifications are adding in a long-poll disable bit to specific workers, and DB Schema changes].  6 smaller pool servers each running against their own bitcoind/wallet.dat file, so there is never "duplicated" work across the servers.

  Bitcoin 0.3.24 with JoelKatz's patches [upgraded to his 0.99 patch on Saturday]


So unless somebody finds a flaw in one of those two (pushpool or JoelKatz's patch), I can only shrug it off and wait for better days.  To those leaving over it:  I'll see you again after our next positive streak, just like what happened during the +54% day.

Regarding the 11m share round, within 24 hours of that round completing DeepBit reported a 16m round, just for reference regarding extremely long rounds.

[zerokwel]

Luck is luck.. And el is a very transparent person so not for one second do I believe he would cheat anyone. and yes there has been days where the luck has hit 50%+ and well no one complained then.


All I am gonna say is prove that he's cheating..


I call bullshit..

[Vladimir]

Assuming that is true, what else could be causing such a run of bad luck, both in terms of technical glitches and malicious activity?

Just to clarify. Basically, my math (rounded to integer) shows that when we expect 133 blocks and 105 get solved it means that:

99% chance that here is either malicious activity or technical glitch or both (I have no idea which one it is)
1% chance that it is simply bad luck.

1% chance shall not be discounted since it is something that happens on about every 100th try.

However, no matter whether it is malicious or technical glitch these are equally good reasons to jump the ship unless one wants to play with odds stacked against him.

If someone would like to make his own DD, simply break available data over difficulty periods. You would need number of shares submitted for each difficulty period. Than divide this number of shares by difficulty, it will be your expected number of solved blocks for the period. Sum all the expected blocks for all the periods. Now you have both expected number of blocks and actual one. The longer time period you use the better.

Than read up http://en.wikipedia.org/wiki/Poisson_distribution or if you really lazy (like me) use one of online calculators like http://www.sbrforum.com/betting-tools/poisson-calculator/

Simply enter expected and actual blocks there and see 'at most' number. This will be a chance of pool finding at most so many actual blocks for given number of expected ones.

All you need to know for this calculation is number of shares and difficulty and number of actual blocks found.

[Mad7Scientist]

Luck is luck..

How many times would the random number generator have to say "NINE" before you believe something is wrong?

there has been days where the luck has hit 50%+ and well no one complained then.

I did. Deviation in either the  positive or negative direction is a sign of manipulation.

All I am gonna say is prove that he's cheating..

"That's the problem with randomness, you can never be sure."

NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE NINE

99% chance that here is either malicious activity or technical glitch or both (I have no idea which one it is)
1% chance that it is simply bad luck.

How does Poisson work for this? During that time that we only got 105 blocks 807 blocks went by. With the binomial distribution calculator you put that 807 number in as the number of trials and that is a major factor affecting the probability. In the binomial calculator try dividing p by 10 and multiplying 807 by 10 and seeing how much it affects the outcome.

Here is the binomial calculator: http://stattrek.com/tables/binomial.aspx. Try putting in 0.1648, 105 and 807 in there from top to bottom. If you put in 133 instead of 105 that would be for a 0% luck 5 day period.

[nosro]

Now shortly after that 5 day streak of very bad luck, the luck suddenly shot up to 70% for a day! I found that kind of odd.

I think questions should always be asked, so I don't begrudge you for putting forth the notion that BTC might be cheating.

However, people are notoriously poor judges of randomness. The "hot hand" in basketball is a classic example that statisticians use of sports fans attributing skill to what can be explained by a good run of randomness.

As you said, the luck suddenly shot up 70% for a day. Rather than focusing on the five days, it might make more sense to take an entire month, which includes the one good day and all the bad days. Put another way, I'm sure you know that flipping a coin and coming up with five heads proves nothing, even though the probability is 3%, which in the parlance of this post implies a 97% chance of cheating.

As a final note, your calculated probability of 0.6952% (which I have not verified) is well within a 4-sigma event (0.0165%). Four standard deviations is hardly considered extreme.

[JoelKatz]

I think the best analysis would be to start by taking the longest period possible. If the odds of performance that low by luck alone over that longest possible interval is not less than 5%, I'd say you can pretty much ignore the possibility of cheating on the part of either miners or the pool. If the pool is stealing from miners, it's not like sometimes they're going to cheat and sometimes they're going to sneak in bonuses.

[codyrbrown]

Just wanted to say I for one don't have much complaints with btcguild *I think ad-funded sites are tacky/1990's though*.  At the very least my "24 Hour Rewards" at btcguild is 99% of the time above the "Reward estimation PPS: X.XX BTC/24h" at deepbit.

I realize the PPS has much higher fee's to negate the risk, but, it makes me happy.

[lemonginger]

online poker is rigged!

[nosro]

If the pool is stealing from miners, it's not like sometimes they're going to cheat and sometimes they're going to sneak in bonuses.

I think this is an excellent point. Pretend you are a pool operator out to cheat. Having 5 visibly bad days seems like a bad way to steal. It would be both easier and less visible to skim off a small amount on a constant basis rather than inserting code that robs a large amount on a set schedule.