Bitcoin scraped pool luck data/graphics

[topheroly]

Hello all, I've decided to share something that I think would be interesting to this community.  Over this past weekend I wrote a script that scrapes the biggest pools for blocks (deepbit/btcguild/slush/mtred/btcmine for now) and uses that information to get a running estimate of the pool's luck.  I know that some pools delay their stats so this information comes from the reported blocks when they are gathered.  Another key thing to note, when the difficulty switches, the first tick of information from this switch will be off, bare that in mind.  Otherwise all other information should be accurate.

About the graphic:
The images are just plots of luck as time goes on.  The largeness of the points themselves indicate how long of a time length for that estimate.  The smallest points are 6 hr windows, the second being 12 hrs and the largest dots are 24 hr time window estimates of luck.  There is a horizontal dotted rule added as a reference line as well as a fit to the data (the solid black line that is very thin).  Around the fit, there is a 95% confidence region.

The second image is simply a histogram of the luck data collected in the past 5 days (the limit I'm holding this graphic plotting region).  From this you can see roughly how each pool is doing overall. You can find this image in l0ss.net/images under the histogram png.  I moved it because it didn't fit with the other ones.

Site is now hosted here, updated daily at 5pm central:
http://www.l0ss.net/

For the inclined:
 I wrote this completely in R http://www.r-project.org/ and rastered the graphics using the package ggplot2 http://had.co.nz/ggplot2/.  The formula used to calculate luck is

z <- (x/y)
if z < 1
    luck <- (1 - z)*(-1)
if z > 1
    luck <- (z - 1)

x == Actual Blocks found within Time Frame
y == Estimated Blocks found within Time Frame

To calculate estimated blocks, use the formula (difficulty * 2^32)/hashrate.  These numbers are also grabbed from each site when the script is run.


Future work to do on this script:
1 - get more data from pools that are sized around ~250 GHash/s or more currently.
2 - write in a difficulty indicator on the graphs
3 - have a separate long luck running graph for > 30 days

If you have questions about any of this please let me know and I will do my best to answer them.

[1715025895]

1715025895

[1715025895]

1715025895

[1715025895]

1715025895

[1715025895]

1715025895

[grod]

These figures make sense since BTCguild and slush are both part of a popular pool hopping script.

Hoppers join BTCguild for fresh blocks, stay for the "lucky" blocks, leave for the "unlucky" ones which means the short blocks get the most computing power and more "lucky" blocks get found than "unlucky" ones, though the "unlucky" ones take disproportionately longer to solve.

Slush's pool pays better for people joining late for "unlucky" blocks, so people hop out to fresh blocks in other pools and hop back in to slush otherwise, meaning fewer computing resources get used to find "lucky" blocks there.

[topheroly]

I forgot to mention that you can see a very cool effect of variance reduction among the pools.  If memory serves right, variance goes off as 1/sqrt(n) where n is the sample size, in our case number of miners.  Thus we see deepbit having the sharpest peak of the pools followed by btcguild and then finally slush.

[Clipse]

These figures make sense since BTCguild and slush are both part of a popular pool hopping script.

Hoppers join BTCguild for fresh blocks, stay for the "lucky" blocks, leave for the "unlucky" ones which means the short blocks get the most computing power and more "lucky" blocks get found than "unlucky" ones, though the "unlucky" ones take disproportionately longer to solve.

Slush's pool pays better for people joining late for "unlucky" blocks, so people hop out to fresh blocks in other pools and hop back in to slush otherwise, meaning fewer computing resources get used to find "lucky" blocks there.

WOW, wOOOw and WooooooW !!!

You have absolutely no understanding of how hopper works, let me try and educate you.

1.) BTCguild stats delayed by 1hr, effectively hoppers dont know when a fresh block starts.
2.) "lucky" blocks? Say what? Hoppers have no idea if the current running block will be long or short block, they cant magically know to stay because its going to magically be a lucky block.

You should really get a better understanding but with the logic you just posted I dont think any proper understanding will help you.

[Sukrim]

Exactly - these stats are nice but have 0 prediction value for the future!

Could you do a month, quarter, year and alltime stats overview too? A week is a bit short...

[Meatpile]

Those are very interesting graphs

Even more interesting is all the people who will make "intelligent claims" on why this graph of luck is shaped the way it is.

[topheroly]

If anything I wanted to show that luck ... is luck.  It's all basic probability, anytime makes some silly claim about biased pool luck, they need only look to this image now.  If there was something fishy going on, we would be able to see it.  I also have updated my to-do list, including adding a long-running graph and stats, etc.

[Sukrim]

If there was something fishy going on, we would be able to see it.

Would it really show up, if a pool operator would steal (with a random function) every 300th block for example?

[topheroly]

Sukrim, good point.  I guess if they took a small block duration out then no, It would be very hard to identify.  I guess what I was more eluding to were errors on the pool's fault, like a consistently running bad luck due to some software error in pushpool or what have you.  I guess we'll see how useful this information really is in the future.

[Meatpile]

I would assume this graphing would only show systemic problems, Like if someone figured out how to cheat the system as a whole.

At which point they wouldn't be a member of a pool i would assume... So besides aesthetically interesting I sort of question the usefulness of the graphs for identifying such a purpose.  (now if you could graph every single individual on the network wowee)

[grod]

These figures make sense since BTCguild and slush are both part of a popular pool hopping script.

Hoppers join BTCguild for fresh blocks, stay for the "lucky" blocks, leave for the "unlucky" ones which means the short blocks get the most computing power and more "lucky" blocks get found than "unlucky" ones, though the "unlucky" ones take disproportionately longer to solve.

Slush's pool pays better for people joining late for "unlucky" blocks, so people hop out to fresh blocks in other pools and hop back in to slush otherwise, meaning fewer computing resources get used to find "lucky" blocks there.

WOW, wOOOw and WooooooW !!!

You have absolutely no understanding of how hopper works, let me try and educate you.

1.) BTCguild stats delayed by 1hr, effectively hoppers dont know when a fresh block starts.
2.) "lucky" blocks? Say what? Hoppers have no idea if the current running block will be long or short block, they cant magically know to stay because its going to magically be a lucky block.

You should really get a better understanding but with the logic you just posted I dont think any proper understanding will help you.

Why does pool hopping work?  Because working on the freshest blocks in proportional pools pays more than working on stale blocks till the cows come home. 

Think about the effect of mining power being added only when the pool starts on a new block.  If that block is 'lucky' (read: finishes before 43.5% of difficulty shares) then the pool starts on a new block.  If the block is NOT 'lucky', the mining power leaves, and the 'unlucky' block gets mined for a longer period.   The pool has the most processing power ONLY for 'lucky' or short blocks, meaning the pool has higher odds of having MORE of them.  And fewer 'unlucky' blocks which take far longer to process. 

In other workds, the 'luck' did not change, but the speed at which 'lucky' and 'unlucky' blocks get solved did affecting the distribution over time.

Since it is pointless to be in a PPLNS pool early on, hoppers will be mining in proportional pools.  Once those are all stale they switch to a PPLNS pool.  Since nobody has announced a solved block by definition they're joining a stale or 'unlucky' block, where their payout is not affected as much by all the other work.  So the reverse happens - less mining power during 'lucky' blocks means they take longer, so they appear less frequently in the distribution.

As far as not knowing when a fresh block starts at BTC -- that is a difficult problem, you got me there.  It seems like there was no 1 hour delay in crediting my account for solved blocks when I was donating 2.5%, but that could have changed since I last mined there.

[Clipse]

These figures make sense since BTCguild and slush are both part of a popular pool hopping script.

Hoppers join BTCguild for fresh blocks, stay for the "lucky" blocks, leave for the "unlucky" ones which means the short blocks get the most computing power and more "lucky" blocks get found than "unlucky" ones, though the "unlucky" ones take disproportionately longer to solve.

Slush's pool pays better for people joining late for "unlucky" blocks, so people hop out to fresh blocks in other pools and hop back in to slush otherwise, meaning fewer computing resources get used to find "lucky" blocks there.

WOW, wOOOw and WooooooW !!!

You have absolutely no understanding of how hopper works, let me try and educate you.

1.) BTCguild stats delayed by 1hr, effectively hoppers dont know when a fresh block starts.
2.) "lucky" blocks? Say what? Hoppers have no idea if the current running block will be long or short block, they cant magically know to stay because its going to magically be a lucky block.

You should really get a better understanding but with the logic you just posted I dont think any proper understanding will help you.

Why does pool hopping work?  Because working on the freshest blocks in proportional pools pays more than working on stale blocks till the cows come home. 

Think about the effect of mining power being added only when the pool starts on a new block.  If that block is 'lucky' (read: finishes before 43.5% of difficulty shares) then the pool starts on a new block.  If the block is NOT 'lucky', the mining power leaves, and the 'unlucky' block gets mined for a longer period.   The pool has the most processing power ONLY for 'lucky' or short blocks, meaning the pool has higher odds of having MORE of them.  And fewer 'unlucky' blocks which take far longer to process. 

In other workds, the 'luck' did not change, but the speed at which 'lucky' and 'unlucky' blocks get solved did affecting the distribution over time.

Since it is pointless to be in a PPLNS pool early on, hoppers will be mining in proportional pools.  Once those are all stale they switch to a PPLNS pool.  Since nobody has announced a solved block by definition they're joining a stale or 'unlucky' block, where their payout is not affected as much by all the other work.  So the reverse happens - less mining power during 'lucky' blocks means they take longer, so they appear less frequently in the distribution.

As far as not knowing when a fresh block starts at BTC -- that is a difficult problem, you got me there.  It seems like there was no 1 hour delay in crediting my account for solved blocks when I was donating 2.5%, but that could have changed since I last mined there.

Again I will add, you sir have no clue.

[Sukrim]

Again I will add, you sir have no clue.

I second that. Read a bit more about statistics + true random events. What you say would only (maybe) be true in a pool that pays everyone for the last N minutes of mining work, not for the shares.

Oh and I throw in the 9 second block recently at slush for good measure.

[grod]

You are right, there is nothing to see here. 

Correlation does not imply causality, and just because btcguild and slush were part of bithopper's default config and deepbit wasn't, combined with rise in popularity of pool hopping mapping roughly to the 'luck' growth trend is just a series of coincidences.

However, if it is by some miracle that is not a series of happy coincidences but instead some interaction I'm not smart enough to figure out then it's just that much more reason to run a pool hopper.

[topheroly]

Made a site, hosted, and looks better.  Check it out.  Updated to-do list.

http://www.l0ss.net/

[topheroly]

Script now includes mtred and btcmine in addition to deepbit, btcguild and slush.

[TraderTimm]

Very nice, certainly useful when it comes to the solo-vs-pooled mining arguments. Good job!

[topheroly]

Script now generates some basic descriptive statistics on the sidebar.

[Clipse]

You are right, there is nothing to see here.  

Correlation does not imply causality, and just because btcguild and slush were part of bithopper's default config and deepbit wasn't, combined with rise in popularity of pool hopping mapping roughly to the 'luck' growth trend is just a series of coincidences.

However, if it is by some miracle that is not a series of happy coincidences but instead some interaction I'm not smart enough to figure out then it's just that much more reason to run a pool hopper.

Hey guess what, I and many others have other means of hopping even deepbit. So again your theory is loaded with cow maneure.

Of course there is no reason not to run a pool hopper, the question is will you drop your faith of being against it or join it making you a hypocrit like so many others? :p

[Clipse]

Script now generates some basic descriptive statistics on the sidebar.

Btw , love the new site layout with the graphs, good job.