Bitcoin Forum

Hello all, I've decided to share something that I think would be interesting to this community.  Over this past weekend I wrote a script that scrapes the biggest pools for blocks (deepbit/btcguild/slush/mtred/btcmine for now) and uses that information to get a running estimate of the pool's luck.  I know that some pools delay their stats so this information comes from the reported blocks when they are gathered.  Another key thing to note, when the difficulty switches, the first tick of information from this switch will be off, bare that in mind.  Otherwise all other information should be accurate.

About the graphic:
The images are just plots of luck as time goes on.  The largeness of the points themselves indicate how long of a time length for that estimate.  The smallest points are 6 hr windows, the second being 12 hrs and the largest dots are 24 hr time window estimates of luck.  There is a horizontal dotted rule added as a reference line as well as a fit to the data (the solid black line that is very thin).  Around the fit, there is a 95% confidence region.

The second image is simply a histogram of the luck data collected in the past 5 days (the limit I'm holding this graphic plotting region).  From this you can see roughly how each pool is doing overall. You can find this image in l0ss.net/images under the histogram png.  I moved it because it didn't fit with the other ones.

Site is now hosted here, updated daily at 5pm central:
http://www.l0ss.net/ (http://www.l0ss.net/)

For the inclined:
 I wrote this completely in R http://www.r-project.org/ (http://www.r-project.org/) and rastered the graphics using the package ggplot2 http://had.co.nz/ggplot2/ (http://had.co.nz/ggplot2/).  The formula used to calculate luck is

z <- (x/y)
if z < 1
    luck <- (1 - z)*(-1)
if z > 1
    luck <- (z - 1)

x == Actual Blocks found within Time Frame
y == Estimated Blocks found within Time Frame

To calculate estimated blocks, use the formula (difficulty * 2^32)/hashrate.  These numbers are also grabbed from each site when the script is run.


Future work to do on this script:
1 - get more data from pools that are sized around ~250 GHash/s or more currently.
2 - write in a difficulty indicator on the graphs
3 - have a separate long luck running graph for > 30 days

If you have questions about any of this please let me know and I will do my best to answer them.

These figures make sense since BTCguild and slush are both part of a popular pool hopping script.

Hoppers join BTCguild for fresh blocks, stay for the "lucky" blocks, leave for the "unlucky" ones which means the short blocks get the most computing power and more "lucky" blocks get found than "unlucky" ones, though the "unlucky" ones take disproportionately longer to solve.

Slush's pool pays better for people joining late for "unlucky" blocks, so people hop out to fresh blocks in other pools and hop back in to slush otherwise, meaning fewer computing resources get used to find "lucky" blocks there.

I forgot to mention that you can see a very cool effect of variance reduction among the pools.  If memory serves right, variance goes off as 1/sqrt(n) where n is the sample size, in our case number of miners.  Thus we see deepbit having the sharpest peak of the pools followed by btcguild and then finally slush.

WOW, wOOOw and WooooooW !!!

You have absolutely no understanding of how hopper works, let me try and educate you.

1.) BTCguild stats delayed by 1hr, effectively hoppers dont know when a fresh block starts.
2.) "lucky" blocks? Say what? Hoppers have no idea if the current running block will be long or short block, they cant magically know to stay because its going to magically be a lucky block.

You should really get a better understanding but with the logic you just posted I dont think any proper understanding will help you.

Exactly - these stats are nice but have 0 prediction value for the future!

Could you do a month, quarter, year and alltime stats overview too? A week is a bit short...

Those are very interesting graphs

Even more interesting is all the people who will make "intelligent claims" on why this graph of luck is shaped the way it is.

If anything I wanted to show that luck ... is luck.  It's all basic probability, anytime makes some silly claim about biased pool luck, they need only look to this image now.  If there was something fishy going on, we would be able to see it.  I also have updated my to-do list, including adding a long-running graph and stats, etc.

Would it really show up, if a pool operator would steal (with a random function) every 300th block for example?

Sukrim, good point.  I guess if they took a small block duration out then no, It would be very hard to identify.  I guess what I was more eluding to were errors on the pool's fault, like a consistently running bad luck due to some software error in pushpool or what have you.  I guess we'll see how useful this information really is in the future.

I would assume this graphing would only show systemic problems, Like if someone figured out how to cheat the system as a whole.

At which point they wouldn't be a member of a pool i would assume... So besides aesthetically interesting I sort of question the usefulness of the graphs for identifying such a purpose.  (now if you could graph every single individual on the network wowee)

Why does pool hopping work?  Because working on the freshest blocks in proportional pools pays more than working on stale blocks till the cows come home. 

Think about the effect of mining power being added only when the pool starts on a new block.  If that block is 'lucky' (read: finishes before 43.5% of difficulty shares) then the pool starts on a new block.  If the block is NOT 'lucky', the mining power leaves, and the 'unlucky' block gets mined for a longer period.   The pool has the most processing power ONLY for 'lucky' or short blocks, meaning the pool has higher odds of having MORE of them.  And fewer 'unlucky' blocks which take far longer to process. 

In other workds, the 'luck' did not change, but the speed at which 'lucky' and 'unlucky' blocks get solved did affecting the distribution over time.

Since it is pointless to be in a PPLNS pool early on, hoppers will be mining in proportional pools.  Once those are all stale they switch to a PPLNS pool.  Since nobody has announced a solved block by definition they're joining a stale or 'unlucky' block, where their payout is not affected as much by all the other work.  So the reverse happens - less mining power during 'lucky' blocks means they take longer, so they appear less frequently in the distribution.

As far as not knowing when a fresh block starts at BTC -- that is a difficult problem, you got me there.  It seems like there was no 1 hour delay in crediting my account for solved blocks when I was donating 2.5%, but that could have changed since I last mined there.

Again I will add, you sir have no clue.

I second that. Read a bit more about statistics + true random events. What you say would only (maybe) be true in a pool that pays everyone for the last N minutes of mining work, not for the shares.

Oh and I throw in the 9 second block recently at slush for good measure.

You are right, there is nothing to see here. 

Correlation does not imply causality, and just because btcguild and slush were part of bithopper's default config and deepbit wasn't, combined with rise in popularity of pool hopping mapping roughly to the 'luck' growth trend is just a series of coincidences.

However, if it is by some miracle that is not a series of happy coincidences but instead some interaction I'm not smart enough to figure out then it's just that much more reason to run a pool hopper.

Made a site, hosted, and looks better.  Check it out.  Updated to-do list.

http://www.l0ss.net/

Script now includes mtred and btcmine in addition to deepbit, btcguild and slush.

Very nice, certainly useful when it comes to the solo-vs-pooled mining arguments. Good job!

Script now generates some basic descriptive statistics on the sidebar.

Hey guess what, I and many others have other means of hopping even deepbit. So again your theory is loaded with cow maneure.

Of course there is no reason not to run a pool hopper, the question is will you drop your faith of being against it or join it making you a hypocrit like so many others? :p

Btw , love the new site layout with the graphs, good job.

Added bitcoins.lc and changed the layout so is now more visually appealing.

Added pool growth hash rate graphs to information provided.  Might include some sort of calculator next or trade data.

Added Trade data bubble plots/loess lines to graphs available.

This is very cool, thanks for putting the time into this.

It might provide circumstantial evidence in case of DDOS - i.e. which pool the attacker is using (since that one will likely not be affected).

Nice work. I salute you sir!

I note that slush's pool has been lucky lately, which I was just noticing yesterday in my own spreadsheets. It's nice to have that independently confirmed. I also note that slush's pool always has a nice bell-curve look to it. It's not clear to me why everyone else has their peak on the unlucky side with a bump way up on the lucky side. I wonder if it has to do with slush's algorithm to make pool hopping unprofitable on his server . . .

Anyone who finds these graphs useful should also check out:
http://bitcoin.sipa.be/
http://bitcoinx.com/charts/

Cool project.

Beyond pretty graphs, what have you learned? Are some pools luckier? Do some pools out perform the average?

Before everyone starts lecturing me about statistics and probabilities, all pools are not created equal. Bitcoins.lc runs very very low invalids. I've seen other polls run as high as 3%. They're running different backends. The code that manages the workers I'm sure varies and has an effect on how effective the pool is.

Maybe there's not enough data to draw any conclusions, blocks don't happen very often. But if anyone has any theories on this data do share. :P

My personal theory is that slush has the best pool for people who aren't pool-hopping. My guess is that the uneven luck distribution for other pools is a result of pool-hoppers making more on short blocks at the expense of non-pool hoppers.

slush has an algorithm that weights recent work higher than older work, which means that pool hopping doesn't get you anything. Other pools don't do this, to my knowledge.

I included histograms for people to look at to actually see what effect a larger pool has on the variance of luck.  As we can see, deebit has the sharpest peak around 0 implying they are running a very steady ship for payouts.  When looking at the smaller pools however, you can see that there is a MUCH larger variance for their ability to find a block.  Interesting to see anyway.

The stats page also includes the mean/sd/quantiles for those that want to look at the descriptive statistics.

Very nice graphs! And they clearly show that ALL pools on average are unlucky!
Either pool owners or block hiders screw the small miner.

It doesnt show that.

Good site, great graphs and useful information. Thanks!

Added an about page and a contact form.  Probably going to add something for people who are interested in the code as well.  I'm still debating that.

Since it is only updated daily at 5 pm now, what is the point of having 6 and 12 hour results on the charts?

It's to give an idea of the magnitude of the variance that pools have.

Added another geom_line to the plots, it helps demonstrate how much everything changes.  The "Actual Luck" line is in red and is drawn through the 24hr points.  I also changed the smoother formula so it only uses 12/24 scrapes because of how much the smaller pools vary.

Added a spiffy break-even calculator to find out what cost it starts becoming unprofitable.

That is good quality work sir.

What happened to the charts? Any plans on putting them back up? Are you still collecting the data the charts use?

You like my histograms of hashes and nonces posted on this BitcoinTalk thread (https://bitcointalk.org/index.php?topic=985846.msg10742132#msg10742132).