Neighbourhood Pool Watch

[organofcorti]

20% fee pool last 18 moths? WOW!

I thought this post would get a bit more traction to be honest. There was quite an uproar when I found that Bitclockers.com was either underpaying their miners by ~21% or incorrectly reporting their stats to avoid paying pool hoppers. BitcoinPool's data shows that miners there have definitely been paid  ~ 80% PPS over the last 18 months - of that there is no doubt. It's also extremely unlikely that it's just bad luck - the data isn't even distributed as it should be, aside from the higher mean shares per round /Difficulty.

Maybe it's just that in a low hashrate pool there aren't many miners, and maybe they don't read this board.

The how and why of the glaring anomaly I don't know, and I don't think the pool ops are ripping their miners off - the data doesn't seem to support that.  I've made the anomalies I found as clear as I could and I've provided the dataset so hopefully someone a bit more cunning than I can figure out what the problem is.

[organofcorti]

20% fee pool last 18 moths? WOW!

deepbit version 2.0?

Well, it's not really a pool fee - BitcoinPool only take a donation. But on average, miners there have been paid 20% less than PPS since July 2011.

[organofcorti]

New post:

Neighbourhood Pool Watch 12.1 The pre-ASIC network hashrate distribution.

5. Conclusions:

• Hashrates are distributes amongst contributors as a Pareto II random variable.
• Hashrate ownership percentages follow the Pareto Principle: ~ 20 percent of pool accounts contribute ~ 80% of the hashrate.
• Minimum hashrates apparently vary by groups - four pools have hashrate averages per contributor approaching 2Ghps, and some have averages less than a quarter of that.

6. Next:
The reason for creating a sample of fulltime miners also had another purpose - to allow an estimation of the post-ASIC network hashrate, when the next plateau in hashrate will occur. I will do just that and also  explain how you can estimate this for yourself in the next post 12.2 Predicting post-ASIC network hashrates.

[organofcorti]

New post:

Neighbourhood Pool Watch 12.2 Pool and network miner hashrate distributions.

5. Conclusions.

• Most miners still have sub 1Ghps mining tools.
  
• A visibly large number of user accounts are at ~65Ghps, likely as a result of using an Avalon ASIC mining tool.
  
  • Pre-ASIC, hashrates were continuously distributed from small to large, leading to a very smooth and rounded density plot. Post-ASIC, the distribution becomes quite lumpy - since Avalon ASICs were the only ones available up until this point, large amounts of hashrate exist in the "hips", at integer multiples of 65 Ghps or thereabouts.
    
  • The distribution of users accounts and user account hashrates is very much in flux, and given the combined distributions "lumpiness" it's unlikely that a Pareto II distribution will model the current user account hashrate distribution very well.
    

[organofcorti]

New post:

Neighbourhood Pool Watch 13.1: BitMinter and "luck".

9. Conclusions.

•   BitMinter's "luck" both over the history of the pool and over the last two months is not unusual. While this does not prove that the pool is running normally, there is no evidence that it is not running normally.
•   Shares per round / Difficulty are distributed as expected for pooled bitcoin mining.
•   The claim that miners at BitMinter have earned 10% less than expected over the past month are untrue.
•   When transaction fee earnings and merged mining are taken into account, earnings are greater than for a fee free PPS pool that doesn't merged mine or include transaction fees. Miners can be confident that even in when recent luck has been poor, they're still earning more than at a PPS pool that doesn't pay transaction fees and doesn't merged mine.

I think there are several reasons that concerns arose about BitMinter's luck.
1.   There was a large influx of new miners who have not yet learned all there is to know about pooled bitcoin mining, and many older miners who have had to answer these questions repeatedly have started to avoid answering the questions, since from previous experience these explanations become long and time consuming. Perhaps I'll write a post on "luck" and pooled bitcoin mining  - then new miners could simply be directed to it when the "luck" questions arise.
2.   Many miners seem unaware of the extra income from merged mining and transaction fees.
3.   BitMinter's charts are also part of the problem In order for miners to intuitively understand whether or not statistics are unusual, confidence intervals must be provided. Without that guide, miners have no way to judge how bad or good "luck" has been over a given period of time. To my knowledge there are no public pools that that provide this data, so BitMinter is not the only pool with this problem. Another smaller issue is that on charts that do not provide time axis dates, miners find it hard to judge over how long a period of time luck has been either good or bad.

[kano]

Meh - I always shudder when people use numbers other than shares/difficulty
Anything else is effectively either derived from that (usually with added inaccuracies) or estimated (and even more inaccurate).
I was wondering about that in the BitMinter thread for a while ... until you chimed in with the results

[organofcorti]

Meh - I always shudder when people use numbers other than shares/difficulty
Anything else is effectively either derived from that (usually with added inaccuracies) or estimated (and even more inaccurate).

Shares/difficulty is definitely my preferred metric, and the one with which I have the most experience. But there is a another metric not directly derived from shares/difficulty, and that is payments per share (for PPLNS pools). Payments per share is Poisson distributed with a mean of N, so it's just as amenable to analysis as shares/difficulty.

But I get what you mean - there are a bunch of ways people have tried to figure out luck, most of them assuming that the variables are normally distributed which they are emphatically not.

Figuring out probabilities is not hard once you know what you're doing. I'll write a post explaining how to do it using Wolfram Alpha if I get a chance - then no one has to listen to "Argh my pool paid out 10% less than normal over the last month!", since they can figure out the truth of the statement and if true how probable it is. Then it's just a matter of getting used to interpreting the results.

I was wondering about that in the BitMinter thread for a while ... until you chimed in with the results

.... and now it's gone all quiet over there

[organofcorti]

New post, and a new way to visualise changes in the user hashrate distribution. There's and excerpt below, but there's lots more at http://organofcorti.blogspot.com/2013/05/124-miner-hashrate-distributions-28th.html

0. Introduction
Although the violin plots used previously were quite good at showing trends, they weren't so good for showing clear changes to total pool hashrates and total numbers of miners, or how particular amounts of hashrate changed over time. As well as this, the violins were not a good method to present an open ended chronological series.

So instead I grouped user hashrates and plotted by time:

The number of user accounts in that group.
The total hashrate attributable to that group.
All the charts in this post represent this, using various colours to represent various groups of hashrate. Think of them as a layer cake, building upwards to a total number of miners or total hashrate and to the right over time.

In doing this I gained new insight into how users hashrates were changing, and also noticed a problem with Itzod's miner count - it seemed to top out at 500 miners. On investigation I realised this was a problem with the way I'd been scraping the data which is now fixed.



1. Change in miner count and grouped hashrates by pool.
Each coloured layer in the charts below represents either the number of miners (left plot) or the total hashrate contributed by that group (right plot) - ASICMiner solo will of course only ever have a miner count of 1.

Some trends are evident:
Nearly all miners have a hashrate less than 5 Ghps, but user accounts with more than 5Ghps contribute far more to the total hashrate of each pool.
The amount of very low hashrate miners at BitMinter has decreased and the amount of hashrate contributed by high hashrate miners has increased significantly, and the same is true for BTCGuild.
The number of low hashrate miners at Eligius has increased.
At Ozcoin and HHTT, miners with hashrates greater than 5 Ghps make up more of the user base than at other pools.
1 - 4.99 Ghps miners at P2Pool, Itzod and Polmine contribute more to the total hashrate than any other group.
I'll leave it to the reader to find the other trends in visible in the charts.

It should be noted that the data from the end of January to mid April is interpolated. I have no data point in that time frame. After that point data points are weekly (or every few days).

[organofcorti]

New post: Neighbourhood Pool Watch 16.1 The network: Orphaned blocks part 1

http://organofcorti.blogspot.com/2013/10/161-network-orphaned-blocks-part-1.html

It's just part 1, I hope to get the rest out soon.

0. Introduction
Most miners become fascinated with orphaned blocks at some point - What are they? This can be simply answered, and also answered simply. But then the questions escalate (or at least mine did): Why do orphaned blocks occur? Can they be minimised or predicted? Which pool has the fewest? What factors most influence the number of orphans produced by the network?  Miners then find very few answers and mostly forget about orphaned blocks (until they lose income to an orphan race), putting them in the same category as forces of nature.

[organofcorti]

New post: 16.2 A short post on mapping the historical pool hashrate

Except below:

0. Introduction

For a while I've been trying to find better ways to present pool hashrate distributions. Pie charts are OK - most people understand them (although as you should all be aware, pie charts have severe problems and can mislead you), but I wanted something that a) was not misleading and b) could show proportions of the network attributable to specific pools. I also wanted to show cumulative data, rather than the plots I usually use to show the historical hashrate distribution.

.....

[ok]

Very good quality posts on this thread, keep it up.

[eleuthria]

Sorry I can't give you the May 2011 through ~October 2011 speeds for BTC Guild.  All that information was wiped when the pool switched from free proportional to PPS .

One request/suggestion:  Perhaps a graph which only shows a smaller subset of pools?  It's virtually impossible to distinguish them when their colors are so similar.

[organofcorti]

Sorry I can't give you the May 2011 through ~October 2011 speeds for BTC Guild.  All that information was wiped when the pool switched from free proportional to PPS .

I thought this would be a problem, so the charts don't show hashrate - just blocks per week and then blocks per week as a percentage of total of network blocks per week. Do you have block history going back before that?


 chart doesn't show hashrate

One request/suggestion:  Perhaps a graph which only shows a smaller subset of pools?  It's virtually impossible to distinguish them when their colors are so similar.

I did try that, but then you lose a feel for how much of the network the pools represent, and how many smaller pools there are, when they started and when they died etc.

What I'd like to do is learn how to make it interactive. I'd like to be able to mouse over the chart, the whole band corresponding to the colour over which you've moused gets highlighted and the pool name appears. I think this could be done using JS (which I've never used) and some JS charting tools (that I've never used), so there's a big learning curve ahead of me.

Very good quality posts on this thread, keep it up.

Cheers!

[eleuthria]

Sorry I can't give you the May 2011 through ~October 2011 speeds for BTC Guild.  All that information was wiped when the pool switched from free proportional to PPS .

I thought this would be a problem, so the charts don't show hashrate - just blocks per week and then blocks per week as a percentage of total of network blocks per week. Do you have block history going back before that?

Unfortunately, BTC Guild history pre-October 2011 is pretty much lost .  It was only about 5 months old, and I was still learning as I went along back then.  I didn't quite realize importance of keeping all that just for historical accuracy back then, after only being involved in BTC for half a year, paired with the long sustained decline in value after the 2011 bubble popped.  We didn't start signing the coinbase until PoolServerJ was released.

[organofcorti]

Sorry I can't give you the May 2011 through ~October 2011 speeds for BTC Guild.  All that information was wiped when the pool switched from free proportional to PPS .

I thought this would be a problem, so the charts don't show hashrate - just blocks per week and then blocks per week as a percentage of total of network blocks per week. Do you have block history going back before that?

Unfortunately, BTC Guild history pre-October 2011 is pretty much lost .  It was only about 5 months old, and I was still learning as I went along back then.  I didn't quite realize importance of keeping all that just for historical accuracy back then, after only being involved in BTC for half a year, paired with the long sustained decline in value after the 2011 bubble popped.  We didn't start signing the coinbase until PoolServerJ was released.

When was PoolServerJ released? The early data I have is from coinbase signatures only - do you have much data before that? (I can't load up the dataset atm, and I can't tell from the chart - there might be a month I don't have).

[organofcorti]

New post:

17 Goodbye HHTT. The coinbase will never be as strange again.

Here's the whole thing in full (although with most of the links missing)

NOTE: There will be no Weekly Pool and Network Statistics this week, as I'm going to having a communications-free week. Instead, I've written a short post commemorating one of the strangest and most fun pools of the last couple of years.

0. Introduction

Recently, fireduck (the pool operator of the HHTT bitcoin pool), announced that he would be closing the pool on New Year's Day. This made me a little sad, since it was a great pool which had some innovative fee mechanisms, and was one of the first pools to take variable share submission difficulty seriously. The open source stratum server he developed has been used by at least one other pool (just grep "SOCK" in the coinbase and you'll see which one) and saw very little downtime.

From my (non-mining) point of view, however, the best thing about HHTT was the sporadic and weird coinbase messages. I had been planning a post about my favourite bitcoin blockchain coinbase messages, but as a remembrance of all things HHTT I decided just to focus on theirs.

The chart below shows HHTT's percentage of weekly network blocks solved, and if you click on the image you can read the coinbase messages that were added at that point in time. If you haven't got a magnifying glass handy (since Blogger seems to compress the chart somewhat) then HHTT's unicode coinbase messages, along with the block height of the block with which they were included, are here: http://bitbin.it/VOXWstm9. I will leave it up to the reader to translate the Japanese keyboard emoticons (such as ╯°□°）╯︵ ┻┠┠) . It should also be noted that block 229417 message appears to be not a penis, but an ASCII sword. Finally, "Finders keepers" had half a coin in it and was found not long after the message appeared. More like that, I say!

So long and thanks for all the weirdness, fireduck. I had lots of fun wondering what weird message you'd come up with, and was more than a bit disappointed that I hadn't seen any recently. There should be more creativity and fun in the blockchain!



organofcorti.blogspot.com is a reader supported blog:
12QxPHEuxDrs7mCyGSx1iVSozTwtquDB3r

[organofcorti]

....
The funny thing is, my hate is from being a pool hopper yet i supported you more than almost all!

Is this addressed to me, fireduck, or are you proposing your own coinbase message? I'll admit you have me stumped, since I thought we were chums.

[organofcorti]

....
The funny thing is, my hate is from being a pool hopper yet i supported you more than almost all!

Is this addressed to me, fireduck, or are you proposing your own coinbase message? I'll admit you have me stumped, since I thought we were chums.

We are chums. I respect you more than almost anyone in BTC land. I am under attack from gmaxwell right now so I am a bit unpolite. Sorry.

I figured I'd just misunderstood your meaning. I should have put a in there somewhere.

So, my two most favourite firebrands in a duel Or something more serious?

Can you post a link?

[organofcorti]

Here's a treat for all you tinfoil hat wearers worried about the "unknown" hashrate owners:

16.3 The unknown network hashrate.

0. Introduction
When miners start looking at the weekly pie charts, they often start to wonder "Who is this "unknown"? What is their nefarious purpose?". If they start to read the daily pie charts from blockchain.info and notice "unknown" at 10 to 20 %, they may even start to become concerned.

My recent work on providing unified block attribution history (last week's blocks here: http://bitbin.it/3iIbl6tA) has made it simple to find a bit more about the "unknown".

.....



.....

2. I hate not knowing! So here's some bounties.
I think most of these will end up being pools, a few early adopters of FPGAs and ASICs, and some of the recent ones will be businesses that don't sign the coinbase.


To get the ball rolling, I'll pay 0.005 btc for the first readers to identify any of the addresses in the table below, on the conditions that they can be identified conclusively (proof required).

.....

[organofcorti]

If you just bought a Gridseed and were wondering whether or not it might be more profitable to mine bitcoin, then this post is for you.

http://organofcorti.blogspot.com/2014/03/918-gridseed-should-you-mine-bitcoin-or.html

0. Introduction

It has been a while since I last mined, and when I heard that Gridseed ASICs could mine both bitcoin and litecoin, I decided to throw caution to the wind, ignore the fact that I may never get a return on my investment, and purchase a batch of ten through seeds over at bitcointalk.org (in case you're in the market for a Gridseed miner, I can recommend seeds: he delivered within ten days, and when I had problems setting the miners up he had an engineer help me out remotely)

These ASICs can mine either bitcoin or litecoin or both simultaneously, although the point of simultaneous ltc and btc mining eludes me - surely one would simply mine whichever coin was most profitable at the time?  When mining bitcoin (SHA256 mode) one device can produce 8 Ghps at 53W; mining litecoin (scrypt mode) one device can produce 300 Mhps at 7W.

So, which is more profitable to mine with a Gridseed miner - bitcoin or litecoin?