Bitcoin Forum

.

According to that shouldn't people be pool hopping?

Well written...but one forum is enough for me

Self defence against hopping? WTF?

Firstly: I don't hop, I will never hop (coz it doesn't change anything), small non-PPS pools should like hoppers (especially if the hoppers would not be present otherwise)

In PPS if the share value is fixed, then hopping makes no difference at all - just find the pool with the highest PPS value :P

CAN ANYONE give me a mathematical proof that hopping is any of the following:

1) Bad for a pool that pays out blocks based on share%
2) Advantageous for a hopper in these pools

I can think of no statistical proof at all.

The only mathematical side effect that I can think of is the effect it has on the standard deviation of the pools block finding rate.

But this is obvious:
When the Gh/s rate of a pool is low (e.g. under 100Gh/s) the standard deviation is higher ... and it is not linear.
As the Gh/s drops the probability of failing to find a block in a given time increases more than linearly i.e. when the hoppers leave.
As the Gh/s increases the probability of failing to find a block in a given time decreases more than linearly i.e. when the hoppers arive.
(e.g. check my calculator on that subject in my sig: http://tradebtc.net/bitprob.php )

Everyone who I have asked says "oh it is" - but cannot prove it (coz it is false)

The first step to understanding why people say it is, but it isn't, is to understand this wikipedia page:
http://en.wikipedia.org/wiki/Gambler%27s_fallacy

Now unless someone can apply the Monty Hall Problem to hopping, I cannot even see it as anything but a fallacy.

I would place it under the heading of an attempt at taking control of something for which there is actually no control - and that makes people feel better thinking they have control over something rather than being at the whim of statistics :)

http://bitcointalk.org/index.php?topic=3165.0

Again:

The paper at that link seems to be all about PPS.
My questions specifically said "based on share%"

I read the first page and found it matched exactly my two php web pages (yes I was surprised to see similar wording I use for my page "Pool Failure Calculator" page http://tradebtc.net/bitprob.php - he obviously wrote the paper long before I wrote that 2nd PHP page, but I had never seen this paper before today)

In a share% payment, your shares % slowly drops after you quit while mining continues until a block is found ...

My issue is that I see hoppers doing the 43.5% on a share% pool and wonder WTF they are doing?
They clearly do not understand mathematics.

To prove the point I will state the simplest example:
If you mine with a share% of 10% at a site for 50% of the expected time to find a block, then, your shares will be worth on average 5% instead of 10% (since your % will slowly drop, after you leave, to 5% until the block is found)
During this time you can go to another pool with the same hash rate and do the same thing ... and thus get a total of 10% ... which is what you would have got to start with :P

Adjusting the numbers actually gives the same results - I just used simple numbers to calculate so that anyone should be able to understand it.

Edit: fixed a wording there that wasn't right (i.e. not what I was thinking :)

I'm guessing you didn't actually read through the paper (I had only skimmed before, but went back and read it).  It clearly talks about proportional payout pools, shares are mentioned a lot (I'm guessing that's why you're thinking it's talking about PPS), but it's definitely dealing with prop payout.  The shares are mentioned as being used to pay prop. for shares submitted/total block shares.

How about "don't let one miner take over 80% of the hashpower of one pool" vlad?

Forget that one?  Ooops :)

No I said it based on the mathematics.
Yes I've already written two PHP pages that use the same mathematics at that first page coz it is very simple maths.

I'd not bother with a PHP page like the 2nd page coz the maths is just about PPS or it's wrong.

Again read my simple example.
In the paper he says that any % will work, so I've given a 50% example that DOESN'T work.
So clearly either it is only about PPS or it's wrong.
(I can see how it would work with PPS)

Don't I remember you being (personally) pro hopper at some point? I can't find the post so I could be wrong.


Mind posting your proof? With chart porn if you please. I'd like to see what you have that proves hopping isn't an issue. Might be able to use it in some arguements with prop pool ops.

I'm especially interested in you showing me how it's possible to hop PPS, and why prop hopping won't work.

Apologies then, Vladimir.

Good call. Full time miners have to remember that hoppers do not cost the pool any money (unless the hoppers come down on an unprepared  pool like a very small ddos attack). In fact pool ops can benefit from the increased hashrate at the start of a round by using that hashrate for advertising.

Full time miners do however make proportionally less than hoppers and with hash rate decreasing once hoppers leave, they get increased variance too.

Some pools are considering a hoppers fee and redistributing the whole fee to their full time miners. With full disclosure, and extra coins every round, full time miners at these pools might start wondering where the money is coming from.

Another point is that pools often stay prop because their members demand it. Maybe your thread might reverse the trend.

This is untrue. This is like those disseminating anti-global warming or anti-evolution information, there is little doubt in the scientific community, but it serves to create a perception that there is even another side besides the one based on fact-finding.

I will post pretty pictures for those who need it. These were done by forum user (and poclbm-autohop creator) Aexoden (https://bitcointalk.org/index.php?topic=28652.msg360992#msg360992) (not me, conspiracy theorists), who has run simulations here (http://www.calindora.com/tmp/bitcoin/index.php) of different payout systems for full-time miners, on a 200ghash pool vs. a 200ghash pool + 100ghash of hoppers. Blue line is what you make.

http://i236.photobucket.com/albums/ff99/qubez1/prop200-nohop.png

http://i236.photobucket.com/albums/ff99/qubez1/prop200-withhop.png

(edit: I do see something I don't like in these graphs, rounds longer than 1/2 difficulty should be shorter by a .14 difficulty factor, since we assume the first dif*.43 of a round had 50% more hashrate. If the X coordinate is time (not shares), that should result in a subtle skew to the left. Perhaps I will do my own Mathematica simulation, since I know how to generate an accurate "hashes before blockfind" distribution, but then again I don't care that much..)


(edit again: I guess the expected skew is there, in the "with pool hoppers" example, the pool has solved two more blocks by the end.)

Why, I even asked Google  :P:
http://i236.photobucket.com/albums/ff99/qubez1/google-poolhop.png

Yes, they do cost non-intelligent miners money - but not the pool itself! The pool actually gains, if it swings from occassionally 260GH/s to 60 GH/s sustained vs. only 60 GH/s all the time.

As pool operators charge fees based on income percentages (instead of just fixed fees, like 5 BTC or 50 USD in BTC or whatever per week) they do earn more, the more hash rate they have. Pool hoppers provide hash rate, so they increase the income of operators.


@Vlad:
Doing statistical analysis is also only useful to a certain degree, since a pool operator can do that as well and just configure his share stealing to be within error rates. Also anything displayed on a website can be faked, and if a pool launders it's newly found blocks before paying them out to it's miners, there's nearly no way for users to find out which pool has found which block for sure.

Bottom line is that you have to trust your pool operator. A lot. There are a lot more ways to cheat your users as pool operator and if you do it carefully, it's also maybe detectable, but not provable (if you stay within certain limits). It is for example possible to just mark shares as "stale" at will for the operator and there's no way to dispute this for miners.

Watch your operator(s) closely, pay them reasonable fees (ideally fixed per week/month, NOT percentages of found blocks, as it might make them greedy/desperate) and try to invent/improve new things, like P2Ppool did.

Read it again carefully... i think he really does mean to say they dont cost the POOL any money.  They do however cost the miners IN the pool money.

Please read the part i bolded. Actually, perhaps you should just read things properly the first time. You do have a tendency tend to fly off the handle a bit without reading things properly, deepceleron.

Even if it doesn't cost the pool directly it will cost indirectly as their 24/7 miners wise up and leave for a hop-proof pool.

If this happens, the pool can still switch and maybe win these miners back.

What I meant by fixed amounts was that currently mining is on operator side in any case a game of luck, there's no "PPS" for pools themselves. If this variance is too high for pool operators to bear, this might lead to problems. If you however design a pool that for example only pays out (automatically) once a week, you can then deduct fixed fees transparently from everything mined during that week and distribute the rest. Depending on how hight your expenses + pool luck are, this might shift the effective percentage taken a bit, but allows for more sustainable planning and expansion and might also lead to less accusations of greed/cheating (there is for example NO way to tell for sure if [Tycho] is manipulating stats/mining every 1000th block for himself/doing other evil things as this would all just go down in noise - again, I'm NOT saying he is doing any of these things, just using him as an example). With fixed fees you could be sure that the pool operator always gets what he needs - not more, not less. With just a fixed percentage it's a bit tricky: Is 1% enough? Would 0.95% also be enough? What about 0.94321%? What about the time the Block reward halves? Double the percentage?

you've kind of just pointed one of the more insidious potential effect of hopping.  For many small pools switching to PPS is simply not an option due the much higher variance risk they have to bear.  Only last week we saw a small pool shutdown for just that reason.  The endgame is that small pools will gradually lose 24/7 miners to pool large enough to handle PPS.  Leaving them with only hoppers... When they have no 24/7 miners left to leech off the hoppers won't have any advantage and will move onto the next pool.

It's currently very hard for small pools to break in a get a reasonable market share but continue to above scenario to it's logical conclusion and you won't have any more small pools.  Competition and innovation will be seriously damaged.

You say "full-time miners have to remember" - that shows we are discussing the interests of full-time miners, not the pool operator, so "pool" defines "miners pooling their resources". This thread's topic also shows we are discussing benefits to bitcoin miners and their concerns; this is not a guide for pool ops to maximize their profits by getting more block solves and fees by green-lighting hopping. The reply sounds revisionist.

Marking valid shares as stale or bad accounting, or not reporting a block find to pool members, are not the the likeliest attack vectors of an unscrupulous pool op. The least transparent (although it will still show over time as lowered luck) is for the pool op to add nonexistant mining shares into the pool rounds, getting paid for mining work not done.

You don't have to generate all getworks yourself, so you could just proxy some shares from bigger pools in the beginning to not run out of funds. Also Bitcoins are quite cheap atm. - getting 100 of them to pay for nearly 4 million shares is not that expensive... shutting down because of running out of funds leaves me asking why someone starts a pure PPS pool in the first place, where they know they might have to have an initial bank roll to cover everything until the first block is found etc.

I think that many PPS pools will come up as "meta" pools (maybe even doing pool hopping in the background) and charging for example a 2% fee (VERY low for pure PPS) but mining at 0% PPLNS/SMPPS/... pools to build up funds for a real PPS endeavour in the future. For this you just need a handful of coins to battle the pooled pools variance (which is usually not as huge as block finind variance on your own) and nearly no infrastructure. On the other hand as soon as one of the other pools scams the metapool owner, it's likely that he'll either have to shut down or otherwise go out of business, so it's important that he'll get his funds quickly and as often as possible from the other pools.

Im curious how that would work? Did you mean proxy some shares TO bigger pools?   Wouldnt you still need to charge a premium on top of whatever you are making from the bigger pool per share?   Is this ethical?

OK then, how about reading the paragraph in full:



So:
1. I mention full time miners
2. Hoppers do not cost pool
3. pool operators can benefit
4. Full time miners make less than hoppers

Since I mention that full time miners lose out just after the part where I said the pool can benefit I don't think there's any confusion there.

Just admit you didn't read the post and we can kiss and make up.

Since he is not claiming that #4 is false like other hoppers, I will just have to concede that this learned poster is too cunning to be understood.

I have moved all but one out of my miners out known hopped pools.  The last will move out around the weekend. 

what always gets me is when hoppers call 24/7 single pool miners "lazy"...  A "hard-working" thief is still a thief.

Interesting thread.  Past performance doesn't indicate future "luck" and that is easily shown by the average number of shares taken (on average) to find the next block for a single user or pool.

Possibly the argument in favour of pool hoppers is that when they jump to small pools, their impact is proportionally higher.  Adding 1 Ghash to a 5000 Ghash pool has less impact than the other way around.  The flip side is that if 5000 Ghash hopped into a small 1 Ghash pool, then it would change the average finding rate (Poisson lambda) for that pool.  You also need to consider how long a hopper might stick with a "hopped" pool and that could be done on the basis of pool size over time.

Fundamentally, over time, any one returned share has the same probability of "finding" a block as any other.  Across all pools and all users, all the coins are distributed.  The ability to consistently get a higher payout is probably beyond the calculation ability of many, and if pool hopping makes them think it helps, then maybe they are not valuing their own time and effort properly.

Just for the record, I decided to join a larger pool and proportional payout rather than pay per share or a small pool/solo mining.  I know too much statistics to rely on luck  :)

???

If you know statistics well, what is your reason behind not using PPS? It has the lowest possible variance, is not dependent on pool size and can be easily calculated + verified.

Yes, with proportional payouts a bigger pool gives lower variance due to more "events" happening during the same time frame, but if you want to reduce variance, you would use pools with completely different payout methods anyways.

Also there were simulations done with distributing hash rate between several pools while hopping based on various criteria - bottom line is that it still is the best possible approach to just always jump to the pool with the lest shares, no matter how small (assuming all pool operators play/pay fair).

http://www.youtube.com/watch?v=4KoKWf6pLs8

that guide is a shameless promotion of your own pool, claiming that *MPPS pools are crap and should be avoided, how come ? ALL shares are paid * 0.00002768 BTC, for ALL miners!

http://abcpool.com/

that is a good one Vladi!  just take a swim instead of fiddeling with the rig.  ;D


besides the link: good thing to point all that out loud and clear

haha!  I clicked the link and was like "Hey! what happened to abc pool?" and then i was all like "oh...doh!"

A) Rational people act rationally - hahahahaha. That's a good one.
B) I'm likely the source of the "lazy" tag for miners who just sit on one pool and don't try to optimize their mining revenue. I'll add that if these miners are anti-hopper then they are more than likely intellectually dishonest as well as lazy. I only have to reference the polmine miners who are fleeing the pool now that they are stuck on a 9 million block. There are numerous examples of miners "hopping" away from pools when they hit long blocks. Those are just slow hoppers. The deepbit 24/7 miners really are lazy and are detrimental to bitcoin. We hoppers provide a valuable service to bitcoin itself, which is infinitely more important to me than some whiny miners.

I can't say it better than Vladimir himself:  "Guess just like predators in nature you serve your purpose, eliminating stupid miners and advancing evolution and making bitcoin stronger. Whoever mines constantly in one of the pools successfully hopped by you surely deserves what he gets."

Well, actually I guess I can say it better and did privately to Vladimir 2 hours before he started this thread:
"Taking your point to its logical conclusion though you have to acknowledge that it is we hoppers who are the ethical ones. We are working literally around the clock (check the irc channel and git commits) to improve our methods and that in turn improves bitcoin. The miners who just buy a GPU, point it at a pool and walk away are the true leeches on the system, the unethical ones. They are the ones looking for money for nothing. We hoppers are at least putting in the time to justify our small percentage of extra earnings. We improve and strengthen bitcoin - they add nothing. If we can drive them out, fine. Hell, you do your best to convince every other miner in the world to quit. We are simply aiding in your effort, we are an extension of your effort.

I understand you have to take a public stance against us. That's alright. We both know that you benefit from us. We can continue to play the game. We aren't enemies, we are both on the side of making bitcoin dominant. Everybody plays a part. You keep playing yours and we'll keep playing ours."

Vladimir launching a PR campaign to try to paint himself as the defender of miners is a little on the nose. He's the most predatory of miners and for that I applaud him. At least he gets it and is self-serving even in his "self-defense" diatribe. You bleeding hearts with your "concern" for those poor miners hurt by hopping are dangerous to bitcoin. The sooner you get out of the evolutionary path and the bitcoin gene pool, the better. You want to compete against world currencies and yet you weep for the weak and ignorant? You will be devoured by the global financial market and spit out before you can even get your boots on.

because no matter what, hoppers gonna hop.  ;D

Difference in fee structure made the expected payoff for proportional higher than pay per share, plus I can withstand the increased variance over time.

Finally a rational post without crazy claims and ridiculous accusations.  There is some logic and truth in PIECES of what you say.  If you speak this way im much more inclined to at least see your side of this issue.   I dont have to agree but if you take the time to explain yourself i can at least understand where you are coming from.  

A few key points;
1) working hard is not the same thing as being ethical.
2) if everyone hopped and there were no loyal miners no blocks would be solved.  In time, you all would eventually hop yourselves and bitcoin to death if there were no 24/7 miners to save you.

I still believe that in the long term you are hurting and not helping only to maximize your own profits.  As an individual i can understand your motivation for doing this and can appreciate the effort it takes.  But i still stand by my opinion that you are exploiting a system for personal gain with no regard to the long term consequences.  

As a pool operator, on the other hand,  we have a fiduciary duty to protect our miners (non-hoppers) from the consequences of pool-hoppers.  The loyal miners are the ones that will solve that elusive block when all the hoppers are long gone because the round lasted longer than they were willing to stick around for.   They should be rewarded for that.  A share has no value.  Its an arbitrary proof of work and nothing more.  Its the blocks that matter and if you wont stick around until one is found why should you share in the bounty?
Whether you agree with that or not - surely you at least see the honesty in it.

As for the bleeding heart garbage... you can throw that out the window... i protect my miners so they can maximize their income so that in turn i can maximize pool revenue and charity donations to bitcoin developers.  Simple, really.

You and I are on flip sides of the same bitcoin.

When the easy prey dies off then we just take our strength, knowledge and numbers and begin hunting elephants. It's not even fun until that point, anyway :)

Nice tusks, btw.

:D :D

Interesting, the system throws random answers (returned shares) at a problem and people attribute long solves to "luck" rather than just the normal probability distribution of solving a block.

If I jumped to a new pool that found the block as soon as I joined, I might get 0 shares out of 100, because the pool was "lucky" to find it right then, but my 100000 shares in the previous pool will contribute to the next block found there anyway.

Willing to be proved wrong, someone might want to supply a link that shows, over a period of a month or longer, for 1000Mhash mining (ie, non trivial) the average return from sitting versus hopping?

Well written and gave me a chucle.  But I do not think the anaolgy is correct.

You started with the elephants and got em all, did the cows and pigs.  Now you are on the sheep and running low.  Hoping does make money now, but like mining will become less and less.

Yeah, on deepbit...

There are pure PPS pools with 0% fee out there too, just sayin'... since PPS is NOT dependent on pool size, you have only little to loose.

There's a bunch of chartporn on the bitHopper thread, or you can try this simulator.
https://github.com/organofcorti/byteHopper-s (https://github.com/organofcorti/byteHopper-s)


The actual Mhps is not important, just the efficiency.

I suggested a reasonable hash rate because operating at 1 Mhash there are appreciable probabilities of not completing shares or making a reasonable contribution.  At least when calculating at a reasonable speed and reasonable time span, so of those noise effects drop away.

I there were no pools (everyone was their own pool- solo), or one pool, I have a chance of finding the block for every share I crunch.  If I am in a pool where people come in and out, or I jump around, my chance of finding a block does not change, nor does it change for anyone else.  A bit like playing roulette (which I don't do because of the stacked odds) - one spin or table is much like another.  If hopping worked so well, people would be doing this in casinos.

Great thing random chance.  Full points to the poster mentioning "gambler's fallacy".

OK, try this link then:

https://bitcointalk.org/index.php?topic=32814.msg478871#msg478871

Oh, don't worry. We ethical miners have seen the huge red flag and are hopping on Mt Red. That huge swing in hashing power has led to 6 blocks in the last 24 hours. Again, compare to the anti-hopping polmine who hasn't solved a block now in 5 days (corresponding with their full implementation of anti-hopping measures). It's all coincidence, of course.

But if hashing power = work done then the Samuel Goldwyn quote seems apropos:

"The harder I work, the luckier I get"

I looked at https://bitcointalk.org/index.php?topic=32814.msg478871#msg478871 and thinking it over while walking to work gave me some entertainment.

It becomes farcical at the point it talks about "Take the extreme example of a pool hitting a 10x difficulty block"  LOL  between difficulty resets, they all have the same difficulty.  Such screwed logic is not backed by the input mechanics or the observed results.

If you want to pick on my strategy, I mine at deepbit because it was easy to set up and the fee was reasonable.  Last 24 hours average time per block is 25 minutes (yesterday 21 minutes).  Roughly 5Ghash or 1/3rd so should be getting a couple of blocks per hour.  It tells me the variance in shares submitted per block is +9.6% at the moment and I get similar payments for a 2 minute or 100 minute block, but on average 20-ish minutes I get a proportional share of what I've contributed.  If I skipped out on a block after an hour and mined somewhere else, my shares still contribute to the next block found (whenever that is, and proportionally), and same with a pool I might jump to.  My averages don't change (assuming fair pools) because my work and the distribution lambda have not changed.

btw, if the pools are set up correctly, pool hopping should make little actual difference over the long term - the notion of ethics is odd unless somehow brand loyalty counts for something - do we get bitcoin-miles for staying with one pool, I haven't looked.

Back to the point of this thread - Vladimir posted up something to help tell if your pool is biased, and added in the variable covering hopping because some people think it makes a difference.  Maybe they're the same ones who like losing money in ponzi's or gambling.

Yeah it's sad that a p2p, decentralized currency is at risk from an attack against one organization (deepbit). If hopping deepbit can help drive away some of the hashrate currently there, it's just that much more rewarding.

That refers to the length of a pool round by comparing the number of round shares to the difficulty. This is a common way of discussing the length of rounds, because when calculating a pool's average round length, it may be a different amount of time depending on pool hashrate, or a different number of shares depending on difficulty, however a round always has a 50% chance of being solved in the same number of user-submitted shares (which are difficulty 1) as the current Bitcoin difficulty. Thus, a 2x difficulty round would be twice as many shares as was expected to find a block solve.

Your strategy is to use the pool with the highest fees (https://en.bitcoin.it/wiki/Comparison_of_mining_pools)? Kudos! That's worthy of being picked on!

 A 10 x difficulty block is one that requires 10 times the share contributions of a normal difficulty block in order to solve it. ATM that means a block of 18 057 000 shares. Can you explain to me where the screwed logic is?

You're suggesting that between difficulty resets every pool solves a block at difficulty shares every time? Surely you've been involved with bitcoin mining to believe that! A quick look at btc-poolwatch.com gives bitclockers total shares at 78% of difficulty, bitpit total shares at 19% difficulty, mineco.in total shares at 51% and triplemining total shares 460% of difficulty. All different totals. If you meant they solve blocks at the same number of shares - difficulty*1 - then that is also just plain wrong.

But I'm glad I was able to give you some enjoyment at work.


BTW - ever get around to running a simulator so you could work out your answers for yourself? Instead of just shooting everyone else's down, I mean.

I'm not here to shoot down others without having some basis for my arguments - there are more than enough people who do that in the BTC forums.  As for simulation, I can look at actual results to see outcomes.  I could set something up quickly that would simply demonstrate that across all users and all pools, with a randomly sampled number of shares to solve blocks, that the average number of shares will equal the current difficulty.  If I participate proportionally, I will get a proportional payout.  I think the underlying point that hasn't been made well is that some pools might pay unfairly.  That harks back to the OP, which makes the point if your pool or strategy is consistently delivering poor results, then look for something better.  As I said, I'm churning shares through deepbit and it tells me at the moment it is about 3% longer per block than expected, which is nice because earlier it was at +10% (and average block time fallen from 25 minutes to 23 minutes).

The point I was making was that even for a block that has been running 18 million shares, there is a probability for the next one share to solve the block.  Blocks don't require a set number of shares to be returned before "solving", and it could continue to run for another 18 million shares to.  The distribution isn't perfectly Poisson, but close enough.

Every looked at the distribution of solve times in the block-chain, a nice sequence of values there.  Some long, some short, but as happily reported and adjusted occasionally, resets aiming to preserve 10 blocks per hour.

I wasn't claiming you could know whether or not the round would be 18000000 shares - just that if you had happened to move to another pool at 1800000 and it had had 9 rounds, then you would have made more btc.

As for the actual reason poolhopping works without knowing the future, well if you know enough about statistic and understand how the number of shares submitted before a pool finds a block follows a *geometric* distribution, then you should be able to figure out the expected value of any given share for yourself. I couldn't, and had to run simulations to check.

Why would I care if Vlad supplies say, 100ghash to a pool I mine in?

If it's not PPS based, then all it means is every member gets paid faster and more frequently.
It also lowers the huge variance that occurs at 1.8 million difficulty.

Really.. This, and pool hopping etc.. Are not advanced math. You can figure these out with common sense.

Some people here seem to oppose everything, for the sake of being against something.

Well, he could run a withholding attack (in any pool/payout scheme), for whatever reason - so it's not too desirable to have HUGE miners in your pool as you have to trust them too.

I'll thank you to keep your size-ist remarks to yourself, Sukrim. Vlad looks quite a healthy weight in his photo.

Spending some of my weekend looking at this and went across to https://en.bitcoin.it/wiki/Comparison_of_mining_pools.  The range of different payment systems introduces the advantages and that was probably the argument that should have been put forward earlier.

If the payment methods were logical and consistent, then the hop vs non-hop largely disappears.  However, having systems where, for example,  "Each submitted share is worth more in the function of time t since start of current round." provides the opportunities to maximise returns.  That's more an issue with pool rules and the motivations of operators to attract people and earn fees.

Over time, miners would probably drift to an equilibrium state where consistent payout pools and hop-friendly pools separate, and then converge as the hopped pools then confer reduced advantage because they rely on non-hoppers to extract their advantage.  Simple - if you want to give away work, belong to a pool with exploits.  Again, that was the point Vladimir was trying to make.

Vladimir: After looking at your spreadsheet, I found a flaw in your analysis. You need to calculate the Poisson for all blocks found by all pools (and expected), not just the ones found by btcguild. You then need to determine what is the likelyhood that out of all the difficulty periods in all the pools, one will exhibit a run of "bad luck" such as the ones you found in BTCguild. You are failing to account for the selection bias effect.

For instance:

I separate people into many groups and have them all flip coins. I then tally each groups expected number of heads and tails. I see that one group has more heads than tails, and I calculate the Poisson distribution of that group as if those were the only coin flips thrown. Of course, this will show that groups efforts as unlikely to have occurred because I have masked out the efforts of the other groups.

Edit:

Also, you are comparing a real world empirical measurement to a perfect world theoretical average. Since there is no magic fairy dust that would result in a computer solving blocks at an average rate consistently faster than this theoretical average, we can ignore positive biases. If there was an operational friction that was not accounted for, it would exert a negative bias to all results. The way to account is to examine the poisson of all blocks found vs expected, btcguild blocks found vs expected, and all blocks without btcguild vs expected. That may produce a baseline would could illuminate any unseen frictions. It might be interesting to do that exercise for each pool, if deepbit had less friction its size may mask higher rates of friction in other pools.

What if they flipped a coin made by a "friend" who claims that he made the coin right. It is the first time he has ever made a coin, but he did his best to make sure it was flat and looked good. We should first test the performance of this coin against the universe of coins made by friends, not against the theoretical 50/50. If we need to choose a coin to flip, we should compare then to each other and then choose one.

Also, the point is not that all pools are having bad luck. The point is that the universe of pools was examined and a single unlucky one was chosen. A better analog than coins might be dice. Imagine throwing six dice. One die comes up with 1 and we calculate the odds of that die landing on 1 correctly as 1 in 6. But that is not what we saw, we saw 6 dice land and the odds of any one of those dice showing a 1 is 1 in 1. Now let us suppose that these dice were not bought from the store, but were each made by a different individual. Now we are no longer sure what the correct distribution of die faces is for any of the dice.

Finally, the pools use different infrastructure. There is work to be done by the server when a share is submitted. What if pool A waits until all that work is complete before acknowledging to the client that it has received a submitted share. If clients cannot begin work on a second share until the first is acknowledged, this serializes the share submission with server share collation at the cost of latency. If pool B allows clients to submit their share and then immediately gives them a new one to work on before the collation work is done on the server, the share submission and processing are overlapped. Pool A's method has the effect of a natural governor on the speed that clients can use the server. Pool B's method minimizes latency but relies on the server to never get bogged down. Under maximum sustained load, these two systems will measure the same. But under failure conditions, they will measure differently.


BTCguild did experience a DoS situation in June, I do not have the exact dates. But the scuttlebutt was that it was from a botnet, that it occurred over the course of a couple of days and peace finally "broke out". I am not sure just what the effect this might have on the "efficiency" of a pool, but I suppose it is possible it could skew the data.


Actually, they do. http://www.l0ss.net/index30.php
As you can see, there is very little if any "good" luck to be found in any of these 6 pools over the last 30 days.
Note btccoins.lc, btcmine, and mtred are all well below the expected luck line. What if there is friction that derives from a common component in the make up of these 3 pools? The luck for these 3 pools is demonstrably worse than btcguild, perhaps you should calculate what the "odds" of their bad luck occurring as well. Then we can calculate the conditional probabilities of those 3 pools + btcguild all having this sort of bad luck all at the same time.  :o

Also, one side question. How did you account for the invalid blocks?

"quote" we saw 6 dice land and the odds of any one of those dice showing a 1 is 1 in 1. "quote"


WHAT?  After the event, looking at the dice, that's allowable, but rolling six dice and getting at least one "1" is [1 - (1 - 1/6)^6] ~= 0.66

Yep, you are right. one roll of 6 dice is will show a 1 66.51% of the time. /sadface
My point was the probabilities were different. I assure you, my point was not to belabor the fact that I am a moron.  ;D

lol - it gave me a laugh earlier today.  I don't think you are a moron, but the expression just jumped off the page at me.

Simulations of pool block finding for different lengths of time gives a good idea of how 'lucky' pools might be in the real world. In the histograms below, the 'luck' index = log((number of shares to find x blocks by pool)/(x * difficulty)).

http://imgur.com/MwGr2.png  https://i.imgur.com/iTmyV.png

Slower pools will have much larger variability than faster ones, but even after 1000 blocks found some pools will look 'unluckier' than others.

The odds of finding outcome in a dataset is different than that outcome occurring in a single monte carlo run. You need to calculate what the odds were of finding a pool with luck that bad, not what the odds were that a specific pool would experience that luck in the future. One is more likely than the other, and misrepresenting the data in this fashion doesn't do anyone any favors.


If one were to go back to June 1st and then decide a pool to mine in, you would be absolutely correct. Going forward, what sort of prediction would you make about the performance of BTCguild? I went and asked the pool operator about events that had an outsized effect on performance that were unlikely to recur. There were 3 and they were very visible in the data set. (DDos July 4th-7th, DDoS Aug 12th, a bad pushpoold patch from June 29th - July 2nd). Also, you didn't treat the invalid blocks correctly in your data set, but that is another conversation.


The historical performance is definitely subpar. However, after dropping those 3 periods the picture changes quite a bit. I do mine with them currently, although I didn't point any significant hashing power at them until August, after the dust had settled from the DDoS and patch issue.


You are going to throw out 20 days worth of data that demonstrates a clear trend in 4 pools (including the one your are discussing now)? When I come back in 10 days with 30 days worth of data, will you dismiss that as well? You were not even curious as to why 4 pools were clearly below trend and the other 2 were flat?


First point:: I believe you calculated the odds incorrectly regarding finding a set of bad luck in the search space of pools

Second point: I believe you treated the invalid blocks found incorrectly in your data set

Third point: I believe you represented past data that contained extreme and discontinuous events as historical norms. If you going looking for a problem in the past and show how the performance deviated from expected as a result of those problems, that is fine. You shouldn't represent that search as the result of a true Monte Carlo run to be expected in future performance.

Fourth point: I believe you are dismissing data that disagrees with you because it is not "as significant". Your response should have been: "Oh, that is interesting, I wonder why that is." I wonder why those pools are under-performing and I believe that someone who claims to be providing a defense course for pool-shopping should wonder as well.

You are very right about one thing, with hindsight we should have absolutely avoided BTCGuild on 7 out of the last 60 days. I am trying to keep this discussion collegial and courteous, I am not trying to pick gnat scat out of pepper to just to troll you  ;)

No. But I don't think I can ever convince you of the difference between finding history vs the odds of it repeating itself.


If every block was being marked as invalid, would that be a sign of "bad luck" or of a software problem. If, after the pool operator applied a patch and blocks were now being found successfully, would you expect the future runs to revert to every block as invalid? Or would you treat that as a discontinuous function instead of a continuous one?


They don't change the yield, but understanding the events that were causal is much more logical than just assuming you understand the distribution perfectly. Congratulations, you found a pool that were you to possess a time machine you would be able to warn people from the past not to use it.


Well, I suppose I should just let you cherry pick your results and pass them off as statistics.


You don't understand how a DDoS from a botnet could affect performance? Or a patch that prevented submitted shares from being aggregated correctly might skew results? I think I see now why your view on this subject is so narrow.

I think what he meant is:

You do calculate the probability of finding a valid hash but comparing it with finding a valid block (which is lower by design because of the probability of chainsplits/orphans being > 0).

So, if I DDoS the smaller pools that can't afford to buy as much bandwidth in perpetuity as I can afford to rent from botnets for a week and they experience degraded performance as a result. This is either bad luck or the pool operators fault according to you. I don't agree with that assessment. Does it degrade performance and harm miners? Absolutely. Is it predictable according to expected performance by a bitcoin pool? Not in my opinion.


You don't want me to answer the first question because it would be in the form of your quotes:
As an example, of a simple check any miner could to do in order to be confident in his pool
I do not see anything wrong with my suggestion to avoid pools which had extremely bad yield in the past.
And of course, this all started with your essay on "Is your pool cheating you."

You don't want me to answer the second question because it would also be in the form of your quotes:
I have specifically chosen the most "unlucky" pool to illustrate my point.
I have calculated odds for 3 month performance of some pools on that chart..

"Chosen the most unlucky pool" from what dataset? The dataset (http://www.l0ss.net/index30.php) I posted shows a strong negative bias over the last 20 days against the theoretical 0 luck line. Does the set of data that you examined (and then presented us only the small cherry flavored segment) correspond exactly to the theoretical 0 luck?

And what do you have against BTCguild anyway? You seem inordinately hostile and intransigent on this subject, declaring repeatedly that "excuses" do not change the yield". Well, nobody here is offering excuses. I am offering explanations. Bitcoin probability does not account for faulty software patch probabilities. You presenting the aftereffects of one as "some mysterious force" or "excuses" or "incompetence" makes it seem like you have a vendetta against this pool you have chosen.

You can't even calculate the odds of finding an "unlucky pool" when you go looking for one. Why should any of us believe you can find one that is cheating you?

No pool can eliminate invalid blocks. Invalids are expected and are a direct consequence of the 10-minute target and network propogation effects because solved blocks do not propogate instantly to every node on the network. Another blockchain with a lower target speed, say 3-minutes, would expect to have more invalid blocks.

Your spreadsheet is slightly inaccurate because you are including invalids in the 'actual' column, but not in the 'expected' column. Until you can compensate correctly for the 'expected' number of invalids, it's best to leave them out of the 'actual' column calculations.

Is there any non-falsifiable proof that a given block *was* actually invalid?

Not entirely accurate.  WIthout going into details,  there are things which can be done to minimize and practically eliminate a block your pool has found from being orphaned (strictly speaking about BTC here and not other shitcoin chains).   In very simple terms its just making sure that you arrange very good connectivity between your pool and as many other nodes on the net as possible (especially the other largest nodes and especially geographically well spaced out).  

To see some effects of doing this, have a look at this post:
https://bitcoin.org.uk/forums/topic/117-mmc-one-of-the-fastest-bitcoin-mining-pool-in-the-world-with-a-proof/

You can see that in many cases Mainframe is the first to announce a new block to the network even when we didnt even find it.   Of course this can depend on the location from which these measurements are taken but i think it illustrates my point that measures can be taken and gives evidence that those measures can be effective.

Yes.  Although it could take some research, i should think that it should be able to be verified by collating data from other pools and blockexplorer if a pool has falsely marked a block invalid/orphaned. 

Yes, what I said was entirely accurate, and I'll re-word it in case I wasn't clear enough. You cannot eliminate invalids. You can, at best, reduce them.

fair enough. 

Thanks for the article. I enjoyed it!

God, Vladimir. You throw libel accusations around far too often. Please go learn a new legal term. You have completely worn out this one through cavalier misuse.

It is difficult to get a man to understand something, when his salary depends upon his not understanding it!
-Upton Sinclair

This quote can be used over and over in the BitCoin community, it seems.

Don't worry Vladimir. Some of the silent guys around here already read your article, found it interesting and well-thought, and have followed your advices. I've quitted BTCGuild, and I found a new pool I'm really satisfied with.

It's normal to have criticism when you do something interesting (like your articles), it's part of the game. Just don't blow a fuse over them ;)

Let's see:

DDoS:
Might make accepting shares difficult/impossible, maybe even sending out valid blocks to the network --> more invalids, but surely not more shares/block

Bugs:
They could lead to accepting + paying solutions twice, acepting + paying invalid solutions etc --> Might increase the shares per block

Fees:
Are an arbitrary concept, do not influence the shares per block, only the payment per share

Latency:
Maybe from miner <--> pool? Increases the stale share rate and might even cause some valid solutions not being pushed to the net (if pushpoold is not "brave" enough to dare to cause a chain split) --> might lead to more invalid blocks or even more shares/block

The real question is: How high is this persistent negative bias (that HAS to exist, with solo mining on a very well connected node as a baseline) throughout all pools and which pool is best there? Also: Why do some pools have a bigger negative bias consistently?

Oh and by the way:
Long Polls are also far too often VERY ineffective or too late, if you look at statistics by pool hoppers! These are the main cause for stale shares and can lead to significantly lower payouts, if you have a pool that performs porly there.

Hahahahahaha!  :D

Never thought about that. Sir, you have my utmost respect!  ;D

So the Jr. Member of the forums, k9quaint, is supposed to be viewed as such an expert on the matters of the article and comments of the Hero Member, yourself, that we the unwashed masses would mistake his statement that you are a fraud as a statement of fact instead of a mere opinion?

Further we are to suppose that k9quaint's opinion is in fact an intentional false communication, and that it harms your reputation(assumes you have a good reputation, no?)? decreases the respect, regard, or confidence in which you are held(assumes you are held in any regard, respect or confidence, no?)? or induces disparaging, hostile, or disagreeable opinions or feelings against you(more than you induce yourself?)?

This is now twice that I've observed you reference libel as a defense in a debate. Here is the other: https://bitcointalk.org/index.php?topic=38668.msg474351#msg474351. Calls of libel are the last defense (or perhaps in your case, the first defense) of the coward. The only thing that appears to be factual in this whole exchange is the level of your pusillanimity and the weakness of your argument.

Oh and you are welcome for the bump.

Vlad the Petulant had cited overall return as the gold standard for pools. I was pointing out that overall returns are likely never to be perfectly efficient for pools given all the things that can go wrong and the lack of known incidents/methods/etc that increase positive luck. That was why I was interested in the distribution of all the other pools Vladdie searched through. The data at http://www.l0ss.net/index30.php shows this bias for the previous 20 days and as more data is gathered, it should give us an idea of the size of the bias.

As for Vladdums claims to not understand how system failures could lead to invalid blocks, Mainframe Mining Cooperative cited the following:
"...with some unfortunate side effects of reduced I/O to one of the storage array disk clusters the pool cluster is using which was causing some latency and load issues for the pool. At the very moment we found block 142,496 we were getting this array back up to speed and im convinced that everything running a bit slower is what caused us to lose our first race."
https://bitcoin.org.uk/forums/topic/125-the-story-behind-our-little-orphan-block-142496/

@Vlad, now that you cry libel defense you can either come at me with a lawyer by filing a john doe lawsuit and compel bitcointalk.org to reveal my IP address and then subpoena my ISP for my identity and then proceed with the lawsuit to recover damages. Or you can add your claims of a libel defense to the list of things you have posted in this thread that are not correct.  ;D

If Vlad ever bothers to post the data he supposedly gathered on the other pools we might get to the bottom of this. I doubt he will, since it will likely make his claims look baseless. Until that time, http://www.l0ss.net/index30.php speaks volumes with nothing to refute it.

Translation: Vlad will not post the data he claimed to have for reasons he does not want to get into.

I am waiting for the process server to notify me I am the defendant in a lawsuit.

I'd watch your going to end up paying his legal costs and probably a fine for a frivolous filing. You might want to look up the definition if libel first.

here's a good start
Judicial and statutory definitions of words and phrases Volume 5
http://books.google.com/books?id=cJENAAAAYAAJ&printsec=titlepage#v=onepage&q&f=false

It should be even more simple than that. Since you claimed to have already prepared the data, you can just upload it to the internet and post a link to it here. Or was that claim false?

Your claim:
"I actually have started from that chart, and than I have calculated odds for 3 month performance of some pools on that chart."
https://bitcointalk.org/index.php?topic=38785.60 (post #80)

I already told you how to go about finding me with the courts. Any competent lawyer should be able to accomplish it.

First the courts aren't going to use your definition. They will use the legal one.
Second , truth is not a defense.
Third the only definition that ultimately matters would be the one the jury gets.


Libel is about malicious intent not truth.

Let me raise the ante and post the statute that you appear to have violated by your actions in this thread:

Title 18 U.S.C. § 1343
http://www.law.cornell.edu/uscode/18/1343.html

Specifically, the part about transmitting false representations for the purpose of obtaining money.

You won't post the data on the other pools you claim to have compiled in order to determine that BTCguild is the "worst pool".
You now are backpedaling from your claim of libel, wisely as it turns out.
You should quit while you are behind.

You forgot the link to the "one more".

You said:
"I actually have started from that chart, and than I have calculated odds for 3 month performance of some pools on that chart. However, one pool have shown extremely high "bad luck" while others, even though, below average in last few weeks over longer period of time hit almost exactly 0 luck line."

All you have to do is post the 3 month datasets for the pools you compiled data for which supposedly contradicts the data from http://www.l0ss.net/index30.php (which I posted and you refute) and shows those pools having zero luck instead of a bias toward negative luck.


 

@Vladimir:
I see you have edited your post to include the data you already presented about BTCguild.
That is not the data from the other pools you compiled stats for. The data from which you supposedly selected BTCguild as the one pool having extraordinary bad luck. The data which you claimed contradicts http://www.l0ss.net/index30.php.

I have to assume you never actually compiled it. That would leave you making false statements about BTCguild, from which you are likely to benefit materially. Not good.

So you claim. You refuse to post the data you claim to have, that you claim is more significant, that you claim BTCguild was the worst out of, and that you claimed contradicted the data I have posted. You alleged you compiled 3 months worth of data on many other pools from which you selected  BTCguild. Where is this data and why do you refuse to upload it to the internet?

Why won't you release the rest of the data?

Perhaps you should contact Mad7Scientist who first brought this exact issue to light on these boards in this thread:
https://bitcointalk.org/index.php?topic=35505.0

His claims are nearly identical to yours, and you participated in that thread, but you did not credit him for bringing this issue to light. Curiouser and curiouser. Could it be that his claims did not receive a good reception either? Or did you want your representations of BTCguild to gain a wider circulation?

Another example of Vladimir plagiarizing Mad7Scientist and passing off that individuals efforts as his own.
Another example of Vladimir refusing to show work he claims to have done.
Another example of Vladimir dodging and hoping to smear BTCguild for his own gain.

I am happy to point out your bad behavior as long as you would like to continue to engage in it.

More dodging and backpedaling from Vladimir.

http://www.l0ss.net/index30.php clearly shows that BTCguild is middle of the road at worst for pool luck. This directly contradicts your claims to the contrary. You claimed you have data to back this up but you refuse to post it.

The data at http://www.l0ss.net/index30.php is compiled out of more than 3000 blocks found. We are to believe Vladimir that 2400 blocks is significant when it shows what he likes but 3000+ blocks is not significant when it shows something he doesn't like?

It must be so humiliating for you to be shown to have plagiarized your article from Mad7Scientist at https://bitcointalk.org/index.php?topic=35505.0 and caught misrepresenting the data you compiled in support of it.

I am glad to hear I finally ran you off the boards. Next time you come back with stuff of this nature, you may expect the same treatment.

Don't worry, I will keep this thread alive so people can see you as you really are.

P.S. I will also keep my eyes peeled for that process server you promised me with a libel lawsuit in tow.  ::)

you two should get a room !  :) :)

The first problem with your statistics is you are just using shares in your calculations. This is not how you would calculate the expected round length. When mining Bitcoins, round length percentiles (the chances a round will go as long as it did) are a binomial distribution based on the number of total independent hashes. What you are doing is equivalent to taking interest compounded monthly, using stats about the interest where it is compounded yearly, and trying to fit it to a curve that would be for interest compounded continuously. Actually, it looks like you are just adding up the total number of shares, instead of fitting them to a round quantile, which is even farther from what you should be doing.

Here is an example of how percentile calculations using just the number of shares and ignoring the underlying difficulty 1 binomial distribution of share finding will skew your results. This is using real binomial percentile calculation instead of whatever you are doing...
This is a table of all block finds for the last difficulty at BTCg.

#   Block   Date/Time   Duration   Shares   Total Hashes percentile   share percentile
2362   141124   08/16/11 12:11 AM   04:56:00   7609114   02.656%   01.479%
2363   141125   08/16/11 12:37 AM   00:26:00   0681963   72.239%   68.546%
2364   141130   08/16/11 01:29 AM   00:52:00   1350501   52.520%   47.335%
2365   141135   08/16/11 02:25 AM   00:56:25   1484443   49.271%   43.951%
2366   141136   08/16/11 02:31 AM   00:05:30   0143481   93.387%   92.361%
2367   141147   08/16/11 05:31 AM   03:00:05   4832611   09.982%   06.882%
2368   141153   08/16/11 06:09 AM   00:38:01   1054539   60.481%   55.766%
2369   141160   08/16/11 07:12 AM   01:03:00   1735924   43.703%   38.237%
2370   141162   08/16/11 07:36 AM   00:24:31   0698169   71.683%   67.933%
2371   141165   08/16/11 07:56 AM   00:19:28   0529012   77.705%   74.605%
2372   141168   08/16/11 08:44 AM   00:48:00   1358208   52.328%   47.134%
2373   141170   08/16/11 09:07 AM   00:23:14   0667927   72.724%   69.080%
2374   141173   08/16/11 10:16 AM   01:08:56   1926729   39.902%   34.403%
2375   141179   08/16/11 11:05 AM   00:49:23   1385716   51.646%   46.421%
2376   141182   08/16/11 11:41 AM   00:35:28   0972989   62.879%   58.342%
2377   141183   08/16/11 11:46 AM   00:04:59   0135719   93.733%   92.759%
2378   141188   08/16/11 12:21 PM   00:35:54   1012043   61.719%   57.094%
2379   141198   08/16/11 01:41 PM   01:19:07   2209830   34.864%   29.411%
2380   141203   08/16/11 02:05 PM   00:23:59   0667575   72.737%   69.094%
2381   141209   08/16/11 02:38 PM   00:33:11   0942983   63.785%   59.320%
2382   141211   08/16/11 02:45 PM   00:06:49   0183003   91.644%   90.362%
2383   141215   08/16/11 03:10 PM   00:25:02   0700124   71.616%   67.860%
2384   141218   08/16/11 03:32 PM   00:21:58   0607330   74.856%   71.438%
2385   141220   08/16/11 03:53 PM   00:21:39   0616836   74.518%   71.063%
2386   141223   08/16/11 04:19 PM   00:25:21   0696526   71.739%   67.995%
2387   141230   08/16/11 05:49 PM   01:30:01   2501908   30.331%   25.018%
2388   141242   08/16/11 08:00 PM   02:11:00   3533660   18.545%   14.129%
2389   141245   08/16/11 08:21 PM   00:21:25   0578235   75.902%   72.598%
2390   141247   08/16/11 08:48 PM   00:27:14   0677616   72.389%   68.711%
2391   141252   08/16/11 09:42 PM   00:53:20   1266941   54.655%   49.577%
2392   141256   08/16/11 10:06 PM   00:24:05   0575149   76.014%   72.723%
2393   141266   08/16/11 11:48 PM   01:42:11   2462278   30.910%   25.573%
2394   141274   08/17/11 12:38 AM   00:49:44   1166794   57.329%   52.405%
2395   141277   08/17/11 01:25 AM   00:47:00   0946058   63.692%   59.219%
2396   141278   08/17/11 01:32 AM   00:07:01   0105215   95.107%   94.340%
2397   141279   08/17/11 01:42 AM   00:10:00   0206608   90.618%   89.188%
2398   141283   08/17/11 02:30 AM   00:47:59   1016167   61.598%   56.964%
2399   141295   08/17/11 04:55 AM   02:25:01   3412584   19.647%   15.109%
2400   141306   08/17/11 07:21 AM   02:26:52   3623270   17.769%   13.445%
2401   141308   08/17/11 07:29 AM   00:07:08   0173062   92.079%   90.861%
2402   141310   08/17/11 07:39 AM   00:10:30   0258664   88.396%   86.654%
2403   141326   08/17/11 11:04 AM   03:24:44   5139667   08.623%   05.806%
2404   141331   08/17/11 12:00 PM   00:56:01   1413260   50.972%   45.719%
2405   141333   08/17/11 12:52 PM   00:52:33   1350274   52.526%   47.341%
2406   141341   08/17/11 02:27 PM   01:34:12   2381984   32.116%   26.736%
2407   141342   08/17/11 02:33 PM   00:06:00   0145661   93.290%   92.250%
2408   141358   08/17/11 06:12 PM   03:39:04   5684853   06.649%   04.293%
2409   141380   08/17/11 09:04 PM   02:52:42   4480611   11.806%   08.363%
2410   141382   08/17/11 09:11 PM   00:07:00   0170823   92.177%   90.973%
2411   141384   08/17/11 09:40 PM   00:29:01   0756007   69.733%   65.792%
2412   141389   08/17/11 10:11 PM   00:30:46   0794891   68.452%   64.390%
2413   141392   08/17/11 10:39 PM   00:28:07   0727708   70.681%   66.831%
2414   141401   08/17/11 11:41 PM   01:01:20   1562514   47.470%   42.092%
2415   141411   08/18/11 01:18 AM   01:37:00   2482370   30.615%   25.291%
2416   141419   08/18/11 02:21 AM   01:02:55   1661534   45.281%   39.846%
2417   141430   08/18/11 03:28 AM   01:07:04   1768458   43.030%   37.555%
2418   141431   08/18/11 03:31 AM   00:03:01   0074000   96.533%   95.985%
2419   141445   08/18/11 03:31 AM   00:00:02   0001780   99.915%   99.901%
2420   141478   08/18/11 05:53 AM   02:21:57   3883147   15.698%   11.643%
2421   141450   08/18/11 06:53 AM   01:00:01   1699965   44.459%   39.006%
2422   141432   08/18/11 08:13 AM   01:19:59   2297726   33.433%   28.013%
2423   141457   08/18/11 09:15 AM   01:02:00   1725020   43.931%   38.469%
2424   141461   08/18/11 10:54 AM   01:39:51   2664088   28.074%   22.869%
2425   141470   08/18/11 12:08 PM   01:13:10   2005153   38.438%   32.941%
2426   141487   08/18/11 01:34 PM   01:25:59   2386828   32.042%   26.665%
2427   141496   08/18/11 03:20 PM   01:46:43   2997992   23.942%   19.008%
2428   141501   08/18/11 04:00 PM   00:39:40   1124282   58.503%   53.653%
2429   141510   08/18/11 05:30 PM   01:29:38   2507835   30.245%   24.936%
2430   141512   08/18/11 05:38 PM   00:08:01   0226044   89.782%   88.233%
2431   141514   08/18/11 05:47 PM   00:08:58   0257777   88.434%   86.696%
2432   141526   08/18/11 07:39 PM   01:52:10   3112459   22.670%   17.841%
2433   141532   08/18/11 08:20 PM   00:40:50   1120369   58.612%   53.770%
2434   141545   08/18/11 10:46 PM   02:26:13   3924076   15.395%   11.382%
2435   141546   08/18/11 11:28 PM   00:42:22   1175575   57.089%   52.151%
2436   141552   08/19/11 01:02 AM   01:33:25   2600589   28.937%   23.688%
2437   141558   08/19/11 01:45 AM   00:43:00   1188897   56.728%   51.767%
2438   141566   08/19/11 03:03 AM   01:18:12   2131499   36.190%   30.715%
2439   141569   08/19/11 03:32 AM   00:28:48   0775002   69.105%   65.103%
2440   141573   08/19/11 04:10 AM   00:38:01   1043501   60.800%   56.108%
2441   141574   08/19/11 04:26 AM   00:16:34   0467272   80.026%   77.200%
2442   141575   08/19/11 04:42 AM   00:15:25   0415757   82.017%   79.434%
2443   141586   08/19/11 06:27 AM   01:45:01   2984100   24.101%   19.155%
2444   141600   08/19/11 09:12 AM   02:45:39   4779202   10.240%   07.088%
2445   141605   08/19/11 09:39 AM   00:26:20   0753706   69.810%   65.875%
2446   141615   08/19/11 11:44 AM   02:05:04   3631839   17.697%   13.381%
2447   141616   08/19/11 12:00 PM   00:15:53   0482024   79.466%   76.571%
2448   141618   08/19/11 12:25 PM   00:25:03   0733702   70.479%   66.609%
2449   141619   08/19/11 12:32 PM   00:07:01   0205693   90.657%   89.234%
2450   141622   08/19/11 01:18 PM   00:46:11   1371842   51.989%   46.779%
2451   141628   08/19/11 01:46 PM   00:27:48   0797782   68.358%   64.287%
2452   141631   08/19/11 02:47 PM   01:01:00   1799350   42.401%   36.918%
2453   141644   08/19/11 04:51 PM   02:04:24   3688955   17.221%   12.965%
2454   141671   08/19/11 10:38 PM   05:46:41   9937498   00.875%   00.407%
2455   141673   08/19/11 10:53 PM   00:14:52   0430853   81.428%   78.772%
2456   141674   08/19/11 11:03 PM   00:10:28   0299371   86.697%   84.722%
2457   141695   08/20/11 02:28 AM   03:25:21   5818787   06.237%   03.986%
2458   141701   08/20/11 03:17 AM   00:48:15   1391035   51.515%   46.285%
2459   141703   08/20/11 03:54 AM   00:36:59   1074718   59.902%   55.146%
2460   141709   08/20/11 04:29 AM   00:35:01   1003017   61.985%   57.380%
2461   141711   08/20/11 04:44 AM   00:15:10   0449461   80.709%   77.965%
2462   Invalid   08/20/11 06:38 AM   01:53:50   3305441   20.677%   16.032%
2463   141733   08/20/11 08:25 AM   01:47:02   3212302   21.616%   16.881%
2464   141734   08/20/11 08:31 AM   00:06:34   0189122   91.377%   90.056%
2465   141747   08/20/11 10:14 AM   01:42:29   3054329   23.307%   18.424%
2466   141756   08/20/11 11:44 AM   01:30:44   2728916   27.219%   22.063%
2467   141759   08/20/11 11:50 AM   00:05:29   0157588   92.761%   91.643%
2468   141760   08/20/11 12:18 PM   00:27:47   0835741   67.132%   62.950%
2469   141780   08/20/11 02:20 PM   02:02:25   3677391   17.316%   13.048%
2470   141782   08/20/11 03:47 PM   01:26:34   2558765   29.520%   24.243%
2471   141784   08/20/11 03:56 PM   00:09:00   0274134   87.747%   85.915%
2472   141767   08/20/11 04:08 PM   00:12:00   0360555   84.204%   81.900%
2473   141781   08/20/11 04:10 PM   00:02:01   0039201   98.148%   97.852%
2474   141802   08/20/11 07:51 PM   03:41:47   6420919   04.681%   02.856%
2475   141810   08/20/11 09:23 PM   01:31:16   2588165   29.109%   23.851%
2476   141815   08/20/11 09:37 PM   00:13:56   0380546   83.405%   80.998%
2477   141818   08/20/11 09:48 PM   00:11:01   0308384   86.325%   84.300%
2478   141821   08/20/11 10:35 PM   00:47:01   1333068   52.959%   47.795%
2479   141828   08/20/11 11:29 PM   00:53:58   1546878   47.826%   42.458%
2480   141834   08/21/11 12:26 AM   00:57:00   1621428   46.155%   40.740%
2481   141836   08/21/11 12:53 AM   00:27:07   0782568   68.856%   64.831%
2482   141854   08/21/11 04:32 AM   03:38:53   6243885   05.093%   03.150%
2483   141855   08/21/11 04:46 AM   00:14:44   0444035   80.918%   78.199%
2484   141864   08/21/11 06:13 AM   01:26:55   2516246   30.124%   24.820%
2485   141869   08/21/11 06:34 AM   00:20:26   0580633   75.816%   72.502%
2486   141870   08/21/11 06:49 AM   00:14:45   0451507   80.630%   77.877%
2487   141874   08/21/11 07:36 AM   00:47:36   1400203   51.290%   46.050%
2488   141877   08/21/11 07:57 AM   00:21:03   0616930   74.515%   71.059%
2489   141894   08/21/11 10:35 AM   02:37:31   4647383   10.904%   07.625%
2490   141896   08/21/11 10:38 AM   00:02:51   0088585   95.864%   95.213%
2491   141901   08/21/11 11:24 AM   00:46:10   1363241   52.202%   47.003%
2492   141903   08/21/11 12:24 PM   00:59:59   1674707   44.998%   39.556%
2493   141911   08/21/11 01:23 PM   00:59:00   1711755   44.210%   38.753%
2494   141915   08/21/11 01:39 PM   00:16:00   0474834   79.738%   76.877%
2495   141914   08/21/11 01:40 PM   00:00:50   0021798   98.966%   98.800%
2496   141916   08/21/11 01:48 PM   00:08:10   0232932   89.488%   87.898%
2497   141923   08/21/11 03:09 PM   01:21:01   2249399   34.212%   28.773%
2498   141924   08/21/11 03:10 PM   00:01:00   0031841   98.493%   98.252%
2499   141935   08/21/11 05:03 PM   01:53:06   3301770   20.713%   16.065%
2500   141951   08/21/11 07:14 PM   02:11:24   3699692   17.133%   12.888%
2501   141955   08/21/11 07:40 PM   00:25:29   0668345   72.710%   69.064%
2502   141968   08/21/11 09:38 PM   01:58:07   3137588   22.400%   17.594%
2503   141980   08/21/11 11:11 PM   01:32:54   2377221   32.189%   26.807%
2504   141990   08/21/11 11:57 PM   00:46:09   1176870   57.054%   52.113%
2505   141991   08/21/11 11:59 PM   00:01:50   0045022   97.876%   97.538%
2506   141993   08/22/11 12:07 AM   00:08:01   0205596   90.662%   89.238%
2507   142001   08/22/11 12:54 AM   00:47:44   1245173   55.226%   50.179%
2508   142003   08/22/11 01:02 AM   00:07:15   0174270   92.026%   90.800%
2509   142004   08/22/11 01:07 AM   00:05:00   0136568   93.695%   92.716%
2510   142005   08/22/11 01:21 AM   00:14:39   0388225   83.100%   80.654%
2511   142011   08/22/11 02:12 AM   00:50:22   1298670   53.835%   48.714%
2512   142015   08/22/11 02:42 AM   00:30:38   0813555   67.846%   63.728%
2513   142016   08/22/11 02:45 AM   00:02:21   0049561   97.664%   97.293%
2514   142025   08/22/11 04:23 AM   01:38:01   2551980   29.615%   24.334%
2515   142033   08/22/11 05:20 AM   00:57:38   1542625   47.923%   42.558%
2516   142055   08/22/11 07:48 AM   02:27:22   4054779   14.465%   10.587%
2517   142059   08/22/11 08:38 AM   00:50:20   1424492   50.700%   45.435%
2518   142061   08/22/11 09:42 AM   01:03:54   1845962   41.469%   35.977%
2519   142064   08/22/11 09:53 AM   00:10:45   0284980   87.294%   85.400%
2520   142065   08/22/11 10:00 AM   00:07:02   0195652   91.093%   89.731%
2521   142088   08/22/11 12:55 PM   02:54:58   4960234   09.393%   06.412%
2522   142099   08/22/11 03:04 PM   02:09:43   3686401   17.242%   12.983%
2523   142116   08/22/11 05:49 PM   02:44:55   4656867   10.855%   07.585%
2524   142118   08/22/11 06:09 PM   00:19:22   0543406   77.173%   74.012%
2525   142121   08/22/11 06:31 PM   00:22:36   0635583   73.855%   70.329%
2526   142129   08/22/11 08:03 PM   01:31:24   2545096   29.713%   24.427%
2527   142140   08/22/11 09:28 PM   01:25:00   2364648   32.383%   26.994%
2528   142141   08/22/11 09:36 PM   00:08:04   0233339   89.470%   87.878%
2529   142143   08/22/11 09:50 PM   00:14:00   0375147   83.620%   81.240%
2530   142146   08/22/11 10:00 PM   00:09:56   0271610   87.852%   86.035%
2531   142168   08/23/11 12:57 AM   02:57:12   4958392   09.401%   06.419%
2532   142173   08/23/11 01:58 AM   01:01:36   1762323   43.156%   37.682%
2533   142174   08/23/11 02:06 AM   00:07:03   0200533   90.881%   89.489%
2534   142184   08/23/11 02:48 AM   00:42:09   1185587   56.817%   51.862%
2535   142194   08/23/11 04:43 AM   01:54:50   3324331   20.491%   15.866%
2536   142195   08/23/11 04:50 AM   00:07:26   0216254   90.202%   88.713%
2537   142202   08/23/11 05:56 AM   01:05:37   1913737   40.150%   34.651%
2538   142211   08/23/11 07:16 AM   01:20:48   2284651   33.642%   28.217%
2539   142213   08/23/11 07:42 AM   00:25:52   0753785   69.807%   65.873%
2540   142221   08/23/11 08:30 AM   00:47:27   1348127   52.580%   47.398%
2541   142230   08/23/11 09:33 AM   01:03:01   1811406   42.158%   36.672%
2542   142231   08/23/11 09:39 AM   00:06:00   0170895   92.174%   90.970%
2543   142244   08/23/11 10:59 AM   01:20:00   2283738   33.656%   28.231%
2544   142251   08/23/11 12:14 PM   01:15:21   2200739   35.015%   29.559%
2545   142258   08/23/11 01:06 PM   00:51:53   1510217   48.669%   43.328%
2546   142278   08/23/11 03:21 PM   02:15:12   3917862   15.440%   11.421%
2547   142300   08/23/11 08:49 PM   05:28:01   6801826   03.903%   02.312%
2548   142301   08/23/11 08:56 PM   00:06:28   1028646   61.232%   56.571%
2549   142303   08/23/11 09:30 PM   00:34:01   2438248   31.266%   25.916%
2550   142322   08/24/11 12:33 AM   03:03:04   5186095   08.434%   05.658%
2551   142328   08/24/11 02:03 AM   01:29:59   2511933   30.186%   24.880%
2552   142333   08/24/11 02:45 AM   00:42:00   1156510   57.610%   52.704%
2553   142346   08/24/11 05:23 AM   02:38:00   4420084   12.152%   08.648%
2554   142349   08/24/11 05:36 AM   00:13:00   0358914   84.270%   81.974%
2555   142358   08/24/11 07:08 AM   01:32:15   2660622   28.120%   22.913%
2556   142363   08/24/11 07:34 AM   00:26:28   0756036   69.732%   65.791%
2557   142365   08/24/11 07:42 AM   00:08:01   0228131   89.693%   88.132%
2558   142368   08/24/11 07:58 AM   00:15:09   0435106   81.263%   78.587%
2559   142376   08/24/11 09:49 AM   01:50:58   3195791   21.787%   17.036%
2560   142391   08/24/11 11:35 AM   01:46:10   3025469   23.630%   18.721%
2561   142408   08/24/11 02:58 PM   03:22:58   5791868   06.318%   04.046%
2562   142421   08/24/11 06:26 PM   03:28:18   5851416   06.141%   03.914%
2563   142425   08/24/11 08:07 PM   01:40:54   2775808   26.617%   21.497%
2564   142427   08/24/11 08:23 PM   00:15:49   0418619   81.905%   79.308%
2565   142439   08/24/11 09:51 PM   01:28:24   2402909   31.797%   26.428%
2566   142445   08/24/11 11:27 PM   01:35:36   2540231   29.782%   24.493%
2567   142456   08/25/11 01:05 AM   01:38:00   2559087   29.515%   24.239%
2568   142466   08/25/11 02:26 AM   01:21:04   2116696   36.447%   30.968%
2569   142472   08/25/11 04:17 AM   01:50:57   2931173   24.717%   19.725%
2570   142478   08/25/11 04:43 AM   00:26:17   0715194   71.104%   67.296%
2571   142485   08/25/11 05:35 AM   00:51:42   1379627   51.796%   46.578%
2572   142489   08/25/11 06:23 AM   00:48:00   1294650   53.938%   48.822%
2573   142495   08/25/11 07:31 AM   01:08:00   1824487   41.896%   36.407%
2574   142501   08/25/11 08:36 AM   01:05:47   1772265   42.952%   37.475%
2575   142503   08/25/11 08:45 AM   00:08:13   0202233   90.807%   89.405%
2576   142504   08/25/11 08:51 AM   00:06:39   0188678   91.396%   90.078%
2577   142509   08/25/11 09:33 AM   00:41:31   1112043   58.845%   54.018%
2578   142520   08/25/11 11:11 AM   01:37:50   2622165   28.641%   23.406%
2579   142525   08/25/11 12:29 PM   01:18:22   2152234   35.834%   30.364%
2580   142536   08/25/11 03:07 PM   02:37:40   4291742   12.919%   09.285%
2581   142542   08/25/11 05:03 PM   01:56:19   3162430   22.136%   17.354%
2582   142549   08/25/11 06:35 PM   01:31:39   2456122   31.000%   25.661%
2583   142563   08/25/11 09:01 PM   02:26:00   3838146   16.039%   11.936%
2584   142564   08/25/11 09:03 PM   00:02:01   0054250   97.446%   97.040%
2585   142572   08/25/11 10:09 PM   01:05:59   1697227   44.517%   39.066%
2586   142580   08/25/11 11:05 PM   00:56:20   1440643   50.311%   45.030%
2587   142579   08/25/11 11:05 PM   00:00:17   0005253   99.750%   99.710%
2588   142585   08/26/11 12:03 AM   00:57:23   1457687   49.903%   44.607%
2589   142589   08/26/11 12:24 AM   00:21:14   0549533   76.948%   73.762%
2590   142591   08/26/11 12:50 AM   00:25:46   0657659   73.081%   69.474%
2591   142601   08/26/11 01:40 AM   00:50:00   1268784   54.607%   49.527%
2592   142614   08/26/11 03:08 AM   01:28:01   2284627   33.642%   28.217%
2593   142625   08/26/11 05:27 AM   02:18:59   3666314   17.408%   13.128%
2594   142677   08/26/11 05:27 AM   00:00:03   0000663   99.968%   99.963%
2595   142641   08/26/11 06:20 AM   00:52:58   1410142   51.048%   45.798%
2596   142626   08/26/11 07:08 AM   00:47:59   1265316   54.698%   49.622%
2597   142632   08/26/11 01:53 PM   06:45:37   10648774   00.623%   00.275%
2598   142685   08/26/11 04:05 PM   02:11:23   3311627   20.616%   15.978%
2599   142690   08/26/11 04:40 PM   00:35:00   0856172   66.481%   62.241%
2600   142699   08/26/11 06:03 PM   01:22:50   2045320   37.709%   32.216%
2601   142698   08/26/11 06:16 PM   00:13:09   0224481   89.849%   88.310%
2602   142700   08/26/11 06:35 PM   00:19:20   0559432   76.586%   73.358%
2603   142712   08/26/11 08:05 PM   01:30:30   2196947   35.078%   29.621%
2604   142715   08/26/11 08:43 PM   00:37:57   0925473   64.320%   59.898%
2605   142731   08/26/11 11:33 PM   02:49:44   4061634   14.417%   10.547%
2606   142737   08/27/11 12:46 AM   01:12:30   1704244   44.368%   38.914%
2607   142740   08/27/11 01:40 AM   00:54:01   1269611   54.586%   49.504%
2608   142747   08/27/11 04:00 AM   02:19:59   3332412   20.413%   15.795%
2609   142748   08/27/11 04:20 AM   00:19:57   0475795   79.702%   76.836%
2610   142759   08/27/11 05:08 AM   00:48:08   1141242   58.031%   53.152%
2611   142760   08/27/11 05:30 AM   00:22:14   0545689   77.089%   73.919%
2612   142767   08/27/11 06:31 AM   01:00:36   1461931   49.803%   44.503%
2613   142775   08/27/11 08:50 AM   02:19:05   3378500   19.969%   15.397%
2614   142793   08/27/11 11:52 AM   03:02:00   4441617   12.028%   08.545%
2615   142801   08/27/11 01:28 PM   01:36:01   2381076   32.130%   26.750%
2616   142811   08/27/11 03:21 PM   01:52:56   2817633   26.092%   21.005%
2617   142824   08/27/11 04:36 PM   01:15:04   1852772   41.335%   35.841%
2618   142825   08/27/11 04:59 PM   00:22:59   0586722   75.596%   72.258%
2619   142837   08/27/11 07:55 PM   02:56:00   4293643   12.907%   09.275%
2620   142840   08/27/11 08:07 PM   00:12:18   0293147   86.955%   85.015%
2621   142842   08/27/11 08:34 PM   00:26:50   0642278   73.619%   70.069%
2622   142847   08/27/11 10:18 PM   01:43:48   2473242   30.748%   25.419%
2623   142850   08/27/11 11:20 PM   01:02:07   1454249   49.985%   44.692%
2624   142852   08/27/11 11:30 PM   00:10:02   0241919   89.105%   87.461%
2625   142854   08/28/11 12:02 AM   00:32:22   0759580   69.615%   65.662%
2626   142862   08/28/11 02:38 AM   02:35:48   3650511   17.540%   13.244%
2627   142863   08/28/11 02:46 AM   00:07:56   0183507   91.622%   90.337%
2628   142865   08/28/11 03:01 AM   00:15:12   0356376   84.372%   82.089%
2629   142866   08/28/11 03:04 AM   00:02:51   0063392   97.022%   96.550%
2630   142897   08/28/11 08:44 AM   05:39:46   8016526   02.187%   01.180%
2631   142907   08/28/11 11:09 AM   02:24:50   3461476   19.194%   14.705%
2632   142913   08/28/11 12:59 PM   01:50:06   2670659   27.986%   22.786%
2633   142917   08/28/11 01:27 PM   00:27:57   0675141   72.475%   68.805%
2634   142921   08/28/11 02:49 PM   01:22:33   1998135   38.567%   33.069%
2635   142934   08/28/11 04:26 PM   01:36:34   2294743   33.480%   28.060%
2636   142954   08/28/11 08:24 PM   03:58:00   5595685   06.937%   04.510%
2637   142957   08/28/11 08:44 PM   00:20:12   0494324   79.001%   76.052%
2638   142964   08/28/11 10:56 PM   02:11:48   3201088   21.732%   16.986%
2639   142967   08/28/11 11:22 PM   00:26:10   0654882   73.178%   69.581%
2640   142972   08/29/11 12:11 AM   00:49:36   1232654   55.556%   50.528%
2641   142974   08/29/11 12:40 AM   00:28:14   0699448   71.640%   67.885%
2642   142984   08/29/11 01:56 AM   01:16:40   1928985   39.859%   34.360%
2643   142990   08/29/11 02:41 AM   00:44:20   1098627   59.223%   54.421%
2644   142993   08/29/11 03:06 AM   00:25:04   0648936   73.386%   69.811%
2645   142994   08/29/11 03:14 AM   00:08:00   0201389   90.844%   89.446%
2646   142996   08/29/11 03:44 AM   00:30:19   0768774   69.310%   65.328%
2647   143002   08/29/11 05:57 AM   02:12:50   3437147   19.418%   14.905%
2648   143016   08/29/11 09:12 AM   03:14:48   5176607   08.472%   05.688%
2649   143028   08/29/11 10:55 AM   01:43:31   2795485   26.369%   21.264%
2650   143043   08/29/11 12:02 PM   01:06:57   1796155   42.466%   36.983%
2651   143048   08/29/11 12:40 PM   00:37:39   1030350   61.182%   56.518%
2652   143052   08/29/11 01:36 PM   00:56:22   1550462   47.744%   42.373%
2653   143062   08/29/11 04:16 PM   02:39:51   4299717   12.870%   09.244%
2654   143074   08/29/11 06:44 PM   02:27:59   3974780   15.027%   11.067%
2655   143087   08/29/11 09:11 PM   02:27:11   3895349   15.607%   11.564%
2656   143091   08/29/11 10:14 PM   01:02:23   1644946   45.641%   40.213%
2657   143092   08/29/11 10:23 PM   00:09:27   0261903   88.260%   86.499%
2658   143093   08/29/11 10:30 PM   00:06:59   0184834   91.564%   90.270%
2659   143110   08/30/11 01:08 AM   02:38:13   4194390   13.533%   09.799%
2660   143111   08/30/11 01:17 AM   00:08:21   0214580   90.274%   88.795%
2661   143114   08/30/11 01:41 AM   00:24:31   0650079   73.346%   69.767%
2662   143115   08/30/11 01:50 AM   00:09:10   0242691   89.072%   87.424%
2663   143120   08/30/11 02:48 AM   00:57:39   1536122   48.072%   42.711%
2664   143127   08/30/11 03:58 AM   01:10:14   1872587   40.946%   35.450%
Mean percentile using estimated total hashes: 52.41%
Mean percentile using shares vs difficulty: 48.60%
Mean # of shares:1831414.1320132      

(We can see above that accusations of the pool being unlucky are already specious, as the pool found blocks with 101.4% of the expected shares, pretty darn close.)

As you can see, the last two percentile columns do not match. in the first, I have estimated the number of shares per difficulty 1 block solve to get the total hashes computed at that round's difficulty, and then determined the percentile for the full probability; in the second I have naively just used the number of shares. You can see that I get different results. This is the problem with just using somebody's CDF calculator without understanding the statistics behind it.

Secondly, you are making some kind of statement that BTC guild has been improbably unlucky. Could we can blame the luck of the miners? Maybe their submitted shares are unlucky? We must first analyze if we can expect the difficulty 1 block solves to correlate to hash rate.

For this I use the R statistics package. The probability of a single hash producing a block solve at difficulty 1 is 2^-32, and I am going to find the 95% confidence interval of probability for a single block find:

R version 2.13.1 (2011-07-08)
Copyright (C) 2011 The R Foundation for Statistical Computing

> dif1prob <- 2^-32
> dif1trials <- 2^32
> oneblock <- 1
> binom.test(x = oneblock, n = dif1trials, p = dif1prob, conf.level = 0.95)

        Exact binomial test

data:  oneblock and dif1trials 
number of successes = 1, number of trials = 4294967296, p-value = 1
alternative hypothesis: true probability of success is not equal to 2.328306e-10 
95 percent confidence interval:
 5.894762e-12 1.297249e-09 
sample estimates:
probability of success 
          2.328306e-10 

Here we can see that if we find a single block at difficulty 1 with the expected average number of hashes, we would have a very wide number of hashes possible for a block solve where we can still say with 95% confidence that the probability is correct. We also know intuitively that a lucky person could find a block in just a few hashes, which would lie outside the confidence interval, showing that even statistics can't absolutely doubt random events. How about if we increase this to 100,000 blocks at the same block solve rate?:

> binom.test(x = oneblock*100000, n = dif1trials*100000, p = dif1prob, conf.level = 0.95)

        Exact binomial test

data:  oneblock * 1e+05 and dif1trials * 1e+05 
number of successes = 1e+05, number of trials = 4.294967e+14, p-value = 1
alternative hypothesis: true probability of success is not equal to 2.328306e-10 
95 percent confidence interval:
 2.313898e-10 2.342786e-10 
sample estimates:
probability of success 
          2.328306e-10 

Here we can see that the confidence interval lies between 2.31 and 2.34 (x 10^-10), a much smaller range, so we can intuit that with over 100,000 shares submitted, with 95% confidence that the luck of share finding shouldn't deviate more than 1% (but in 1 of 20 cases we can predict it will).

So now we look at the blocks of BTC guild and see if we have enough statistics to make any conclusion. Let's pick a rather "unlucky" day, 8/24, where 17 blocks were found with 43715542 shares (an average of 2571502 shares per round, at difficulty 1805700.83). I'm going to compute with just shares first so I don't run out of memory...:

> x<-17
> N<-43715542
> prob<-(1/1805700.83)
> dbinom(x, N, prob)
[1] 0.02901398

Here we get the percentile of 2.9%; these odds can be expected averaging 1 in 34 days.
Can we say though that the game is "fixed", that the probability this day is not what is expected?

> binom.test(x, N, prob)

        Exact binomial test

data:  x and N 
number of successes = 17, number of trials = 43715542, p-value = 0.1547
alternative hypothesis: true probability of success is not equal to 5.538016e-07 
95 percent confidence interval:
 2.265356e-07 6.226308e-07 
sample estimates:
probability of success 
          3.888777e-07

The "alternative hypothesis" as represented above is what would be outside the confidence level, that the probability is not the Bitcoin probability, maybe luck being affected by the pool op. We can see that both the bitcoin probability and the probability estimate from the results are both within the confidence interval, meaning that these results do not deviate outside an expected block solve rate for the day.

These numbers are now large enough that the discrete binomial distribution has approached the continuous poisson model for the same data (shown below), and we can then do even bigger math:

> poisson.test(x, N, prob)

        Exact Poisson test

data:  x time base: N 
number of events = 17, time base = 43715542, p-value = 0.1547
alternative hypothesis: true event rate is not equal to 5.538016e-07 
95 percent confidence interval:
 2.265356e-07 6.226309e-07 
sample estimates:
  event rate 
3.888777e-07 

Now let's look at the data for the whole two-week difficulty period where 303 blocks were found with 554918482 shares:

> x2w <-303
> N2w <-554918482
> poisson.test(x2w, N2w, prob)

        Exact Poisson test

data:  x2w time base: N2w 
number of events = 303, time base = 554918482, p-value = 0.8417
alternative hypothesis: true event rate is not equal to 5.538016e-07 
95 percent confidence interval:
 4.862695e-07 6.110991e-07 
sample estimates:
  event rate 
5.460261e-07

Lets do the full expansion based on the total hashrate, and even go a step further and say that the shares being submitted this whole time were 0.5% luckier than expected, resulting in more hashrate trials per submitted share:

> poisson.test(x2w, ((N2w*2^32)*1.005), (prob*2^-32))

        Exact Poisson test

data:  x2w time base: ((N2w * 2^32) * 1.005) 
number of events = 303, time base = 2.395274e+18, p-value = 0.776
alternative hypothesis: true event rate is not equal to 1.28942e-16 
95 percent confidence interval:
 1.126552e-16 1.415747e-16 
sample estimates:
  event rate 
1.264991e-16

I think you get the idea, the estimated block win rate of BTC Guild is well within the expected confidence range, the Vladimir hypothesis is busted.

Edit: maybe clearer, probably not...

Well done.

Just as an aside, I believe the Poisson test is not appropriate since the shares can interact with each other in the form of orphaned blocks. The since the events can interact with each other they are not completely independent from one another. The Poisson distribution is one of events that are independent from one another.

Also, extra funny is Vladimir using the Poisson distribution, which is also known as Poisson's law of small numbers, and then demanding large numbers.  ;D

The Vladimir myth is thoroughly busted.

@Vladimir
Numerous people have, in multiple threads. You just fail to learn from your repeated schoolings.
More appropriate treatment of your dataset has shown repeatedly in multiple threads that BTCguild performance is well within the expected behavior.
Expanded datasets that include other pools have shown that you are completely off base. You deny datasets larger than your own as "insignificant" in attempts to warp the data to fit your hypothesis.

You keep pasting your tired old story, and it never stops being wrong. You keep making yourself look foolish. At this point, anyone who trusts you with their bitcoins is even more foolish.

Holy crap, don't quote my whole post again!

Invalid block solves or orphaned blocks, of which there was one in the period I showed above, is a completely different mechanism than the statistics of block finding. If your pool finds a block solution, but another pool finds another essentially simultaneously, the winner is the one who gets it published and distributed on the p2p network first and fastest, so that future block hashes are built on it. You don't get the 50BTC if your block is not included in the final blockchain. This generally means that the non-orphaned solution is the one found first, but it is also could be that the ultimate winner had higher connectivity and was able to get their solution to more p2p nodes faster. This is hard to statistically model, but we could at least say that if pool 1 had five orphaned block solves (which they can publish to prove that the invalid was actually found within seconds of the winner), and the block solve winner was always pool 2, we might suggest that pool 1 should optimize their connectedness or miner work pushes to get their block finds out sooner on the Bitcoin p2p network.

I could run a exact confidence level model on several months of block finding, but I'm not enough of a math genius that I can immediately see how to also model the changing probability (difficulty) for the N trials in three months in code. It would probably involve making a data frame of all the block solves and running the vectors of shares and round difficulties through the poisson test though. We can see from my second code snippet above, with 100,000 difficulty 1 block solves (many more than the 2275 rounds) at a probability 1/2^32 (much lower than 1/difficulty) we still have a +/- 1% confidence window. Besides, it is improbable, but not impossible, for a pool to never solve another block in your lifetime.

http://www.random.org/analysis/dilbert.jpg

It is possible to look at the histogram of number of shares in a round for a large number of rounds to get a visual idea of pool luck luck. Further, you can calculate expected round length by total shares in a round. From Meni Rosenfeld's excellent work in progress:

 (http://bitcoil.co.il/pool_analysis.pdf)
and the likelihood of a round being solved but a certain number of submitted shares is given by the CDF (using R)

for 100000 shares:
> pgeom(100000,1/1777774)
[1] 0.05469788

for 10000000 shares:
> pgeom(10000000,1/1777774)
[1] 0.9963935

So a round has a 5.5% likelihood of being solved by 100000 shares, and a 99.63% likelihood of being solved by 10000000 shares.

This is not a criticism of your long term approach, I haven't yet read through it all. But your opening paragraph does leave you open to misinterpretation.

The binomial cumulative distribution function differs depending on the number of trials. This is the source of my "interest rate" analogy.

Bitcoin probability of a single hash solving a block is 1/(Difficulty*2^32). Pool statistics obfuscate the actual number of hashes by being a 1 in difficulty problem wrapped around a 1 in 2^32 problem.

The function (1-(1/Dif))^(total shares) does not have the same quantiles as (1-(1/(Dif*2^32)))^(total hashes).

OK, got it. This ->


is a bit of a broad statement and confused me a bit.


Edit: This is totally off topic, and you all have my apologies, but why isn't


suitable for the same job? I thought that was what Vladimir was doing and I'm not sure I understand your reasoning behind not using it.

deepceleron already answered that. Including data from multiple difficulty changes complicates things.

I don't feel like busting out my stats, but just a visual examination shows difficulty 567269 to look really bad. I think an analysis of that period whould show a significant deviation from expected results. Something happened to BTCGuild between 6/6 and 6/15.

There are ephemeral effects that can take place which can degrade a pools ability to win this race. They can be both external and out of the operators control (DDoS or a macro-network failure) or internal and under control (such as a subsystem failure or degradation). Modelling the probability of these events is difficult since they are usually large and have few recurrences. It is also not clear what the impact for any particular event was on any particular pool. Also, a software patch applied to some pools but not others could degrade the ultimate process of obtaining the 50BTC without anyone quantifying the extent of the degradation or the period of occurrence at all.

My point was that since block finds can collide, we cannot assume the events are independent. Nor can we assume that the variance of that dependence can be known at a given point of time with any certainty. Rather than supposing we understand how the distribution of rewards for found blocks may vary from one that is appropriate for Poisson, we can use a different method and be certain.

However, it may not be necessary to model that. Instead we can look at how close each pool comes to the expected block solves for a perfect pool with zero hardware, software, and network failures. Then we can have a degree of confidence about expected performance in practice. This would be a very annoying data collection however, given that not all pools provide the data. If only we knew someone who had already compiled 3 months worth of data from the other pools and could post that data on the internet. Sadly, no such figure exists. The closest I have been able to find is at http://www.l0ss.net/index30.php. Of course, this analysis would only be useful as hindsight and have little in the way of predictive powers.

One thing we can be absolutely certain of: no matter how many people explain to Vlad how his conclusions are flawed, he won't change his story in the slightest. The narrative is likely to increase the size of his pocketbook and thus have "zero variance".  ;D

Wasnt that the whole point of this exercise anyway (and the original post for that matter?) 

And if not, why not? 

What do I,  as a miner,  care about your hardware, software, or network failures?  I want a return on my electricity costs + margin and the rest is your problem and why i pay you a fee/donation.  I dont care about normalized statistical calculations about the probability of x and y happening if the end result is the pool  solving less blocks than expected or hoppers stealing a share of my earnings.

Judging from the growth we have seen at Mainframe,  im guessing we arent the only ones with this viewpoint.

I think we need to remember that whether or not Vladimir's math and methods are 100% agreeable to everyone,  the basic point of what he was trying to say is valid and sound, good advice for miners.  Beware pools which appear to under earn and pools which allow hopping (unless you plan to hop yourself).   Not because they are evil or cheating or anything else (while its possible they could be) but because it eats into your earnings.  This is just common sense and good advice and all the math in the world cant replace it.

There is a lot of stuff i don't understand in this thread, in Vlad's defence i might say that he kinda makes it a little bit easier for me to digest and this thread made me stop trusting blindly in pool ops.

what do you guys have to say about rfcpool, it's not in the list. thanks

You could also run a quick simulation (see https://bitcointalk.org/index.php?topic=38785.msg486238#msg486238) and do a visual comparison on real data, which yes, you'd need to renormalise for each difficulty period. The advantage of using shares per round as a measure of luck is that renormalising the data is quite simple (list of difficulties and change dates here (http://blockexplorer.com/q/nethash))

Note that this wont prove anything, but if you can model your simulation data accurately enough (http://creativemachines.cornell.edu/eureqa) you'll be able to measure how far the real world data varies from it.

He only compared one pool to the perfect record. We only have his word that btcguild is significantly worse than the other pools in this comparison. He refuses to post the data. For all we know, btcguild could be middle of the road compared to other pools as is shown at http://www.l0ss.net/index30.php.


Software, hardware, and network failures do not follow the binomial distribution of solving bitcoin blocks. Lumping them in as if they were recurring phenomena subject to a known distribution is just pseudo-math. The point is that you don't care to understand why, and that is fine. Feel free to hash where ever you like, for whatever reasons you like.


Glad you guys managed to scare up some business.

Without having precise details and reports on failure dates, times, and durations for each pool its impossible to do anything other than lump them in.   As long as this is done for ALL pools, its fair, isnt it?  The point im trying to make is that removing REAL occurences that affect availability and earnings only because they cant be predicted or determined to be recurring isnt really the right way to determine where to mine (or not to mine) either.   Your method suggests to completely abandon meaningful data simply because it may be phenomena (at least if if understand you correctly).  It could also be incompetence, you know (which does tend to be recurring and probably even with a known distribution) :D :D

I think you should read my post again. I was showing deepceleron how it could be possible to look at variation in pool luck using shares per block.

I have never mentioned wanting to include 'irrelevant' variables.

I never mentioned that I was not happy with your efforts.

I also have never made any indication that number of blocks found in a time period is a good indicator of pool luck, when average shares per round seems a more easily calculated index. So, no, I won't do calculations based on blocks per unit time. I can calculate for you the number of difficulty 1 hashed expected to find a block, but I'm hoping I don't have to. 

Was this just to get your thread to the top of the forum? Shame on you!

I'll give your thread a bump, and summarize my points:

- Bitcoin block finding is random - pools may have runs of bad or good luck, but past results can't predict how a pool will perform in the future,

- To say a pool is cheating users based on luck is speculative, because even with many rounds of data, the statistics can't say that the probability is outside of expected variance. If I flip "tails" 10 times in a row in my coin-flip game, how do I then defend against allegations of impropriety just because it is unlikely? If my pool has 10 unlucky rounds in a row, should users be entitled to accuse me of stealing their earnings?

- Cheating by pool operators is hard to detect. Many mechanisms could be employed, such as only withholding very lucky block finds, diverting some user's shares to a second 'dark pool', adding non-existent shares to the pool to get paid for work not done, etc. Improving mining clients can help - it is possible for a miner to know that they submitted a winning share of higher difficulty than the bitcoin target, and the submission of a winning share can be locally logged; when the pool op knows that users know when a block was found, cheating is harder.

- When calculating "luck", even for curiosity's sake, one must convert the shares per round to a percentile or sigma of expected length, using good math aware of the true Bitcoin probability underlying pool shares, otherwise you will have skewed findings.

- Losses from pool hoppers is not hard to detect. Is the pool's hashrate graph jumping up at the start of every new round? In a proportional pool, your round earnings should be inversely proportional to round length. When you find that you are not making as much per submitted share as you should in short rounds, and are also able to submit fewer shares than you should have been able to in short rounds, it's time to jump ship.

Losing 50% of his BTC on short rounds at deepbit?  Sounds like someone doesn't know how to do math....  There would have to be a ridiculous number of hoppers for that to happen.  They may hurt some on shorter rounds, but at a giant pool like deepbit it should barely be felt.

That guy probably just had some bad luck with those short rounds where "he lost 50% from hoppers being in the pool".

The PPS fee on Deepbit is way too high, that would just be stupid. Hopping is done to maximize profits, and you don't do that by using a pool with a 10% fee. The pools are envious of Deepbit's market share and lots of people thinks it's too close to 50%, so they will say anything to get miners to move to other pools.

I don't know and I have no idea how to find out, so take the following figures as untested approximations.

If you look at other proportional pools, the increase from auto-hopping is in the order of 150 - 300 Ghps (correct me if I'm wrong). We could expect a similar amount from deepbit, making the hashrate increase from hopping only ~ 6%. There is no way this will reduce payoffs for fulltimers on prop by 50%.

If there are some miners who only hop from deepbit PPS to deepbit prop and back very few would be using an auto-hopper to do it (since so many more options are available).

Finally if you look at a graph of deepbit's hashrate over time any significant increase in hashrate that coincided with a found block and the start of a new round would stick out like a sore thumb. You don't see anything like that on deepbit hashrate graphs.

From what I've seen irl and in sim, the loss of income due to hoppers increases when the hashrate of the hoppers increases compared to the hashrate of the pool, and is only significant when the hashrate is a large fraction of the overall pool hashrate. I could be wrong though - if you think I am please explain why.

rfcpool is most excellent for a 24/7 miner imo :)

PPLNS FTW!