Neighbourhood Pool Watch

[organofcorti]

Thanks very much to the person that last donation was from!

[organofcorti]

Neighbourhood Pool Watch 5.1 p2Pool - bad luck or flawed? is posted.

There have been concerns by miners on the bitcointalk.org forum thread for p2Pool that there has been significantly poor luck at the pool for an extended period, and that the number of orphaned blocks created by the pool is higher than it should be. In this post we will investigate the published pool statistics generally, and then specifically with regard to these two concerns.

7. Conclusions

• Although p2Pool's round lengths appear normally distributed, they have a mean value of 1.104D, and are greater than the expected value of D with 95% confidence. The expected round lengths at p2Pool are about 10% more than expected. This is not "bad luck" but inherent in the pool.
• Orphan production seems to have been increasing,  although with a small sample size this conclusion may be erroneous.
• Orphan production rate does not appear to correlate with changes in P2pool hashrate.
• If the usual orphan rate at other pools is 1.5%, and the mean round length remains approximately 1.1D, then miners will earn 11% less here than their expected share values.
• If the orphan rate is normal, then miners are earning approximately 9% less than expected.
• Variance is significantly larger than for pooled mining using difficulty 1 shares, but much less than for solo mining.
• Donations to p2Pool over the past 5 months 131.13 btc, increasing miner earnings by an average of 0.52%
• Overall, miner share values have been either 8.5% or 10.5% less than expected.
• Given the decentralised nature of the pool, some proponents feel that this is an acceptable price.
• DDoS attacks can also lead to downtime and loss of income for miners. This will not occur at p2Pool.
• The concept of p2Pool is laudable, and I hope it continues to be developed and the round length problem solved.

[Meni Rosenfeld]

Neighbourhood Pool Watch 5.1 p2Pool - bad luck or flawed? is posted.

There have been concerns by miners on the bitcointalk.org forum thread for p2Pool that there has been significantly poor luck at the pool for an extended period, and that the number of orphaned blocks created by the pool is higher than it should be. In this post we will investigate the published pool statistics generally, and then specifically with regard to these two concerns.

7. Conclusions

• Although p2Pool's round lengths appear normally distributed, they have a mean value of 1.104D, and are greater than the expected value of D with 95% confidence. The expected round lengths at p2Pool are about 10% more than expected. This is not "bad luck" but inherent in the pool.
• Orphan production seems to have been increasing,  although with a small sample size this conclusion may be erroneous.
• Orphan production rate does not appear to correlate with changes in P2pool hashrate.
• If the usual orphan rate at other pools is 1.5%, and the mean round length remains approximately 1.1D, then miners will earn 11% less here than their expected share values.
• If the orphan rate is normal, then miners are earning approximately 9% less than expected.
• Variance is significantly larger than for pooled mining using difficulty 1 shares, but much less than for solo mining.
• Donations to p2Pool over the past 5 months 131.13 btc, increasing miner earnings by an average of 0.52%
• Overall, miner share values have been either 8.5% or 10.5% less than expected.
• Given the decentralised nature of the pool, some proponents feel that this is an acceptable price.
• DDoS attacks can also lead to downtime and loss of income for miners. This will not occur at p2Pool.
• The concept of p2Pool is laudable, and I hope it continues to be developed and the round length problem solved.

I didn't read the entire analysis but I disagree with your conclusion that round lengths are inherently longer, there is hardly enough evidence to rule out bad luck. Especially if there is no plausible explanation how can round length mean be higher.

[organofcorti]

I didn't read the entire analysis but I disagree with your conclusion that round lengths are inherently longer, there is hardly enough evidence to rule out bad luck. Especially if there is no plausible explanation how can round length mean be higher.

There's enough evidence to show that the round length mean is greater than D to a 95% confidence level. It would have to be quite bad luck.

But I agree there's no mechanism I know of that could plausibly cause such a result. But I'm not a p2Pool expert, and I'm hoping that such an expert will come on board and discuss it. The p2Pool share chain and dynamic difficulty work so differently to a normal pool that I really can't comment. I was thinking that some shares may be unaccounted for and not reported locally, or some such thing. Most likely improbable, and if someone can prove the impossibility then I will gladly edit the post.

[Inaba]

Did you factor in the high stale rate compared to a normal pool? 

[organofcorti]

Did you factor in the high stale rate compared to a normal pool? 

DOA and orphan shares don't make it into the sharechain and so can't increase the numbers of shares in a round.

One possibility is that the method used to estimate the equivalent difficulty 1 shares to solve a block using the pool hashrate is buggy. If for example the pool hashrate is actually 9% lower than reported, then the number of shares to solve a block would be D.

[Meni Rosenfeld]

I didn't read the entire analysis but I disagree with your conclusion that round lengths are inherently longer, there is hardly enough evidence to rule out bad luck. Especially if there is no plausible explanation how can round length mean be higher.

There's enough evidence to show that the round length mean is greater than D to a 95% confidence level. It would have to be quite bad luck.

But I agree there's no mechanism I know of that could plausibly cause such a result. But I'm not a p2Pool expert, and I'm hoping that such an expert will come on board and discuss it. The p2Pool share chain and dynamic difficulty work so differently to a normal pool that I really can't comment. I was thinking that some shares may be unaccounted for and not reported locally, or some such thing. Most likely improbable, and if someone can prove the impossibility then I will gladly edit the post.

That's why Bayesian statistics is superior to frequentist statistics. All the data gives us is about 10:1 likelihood ratio for 1.1D over 1D, which doesn't mean much when the prior probability of 1.1D is so low.

One possibility is that the method used to estimate the equivalent difficulty 1 shares to solve a block using the pool hashrate is buggy. If for example the pool hashrate is actually 9% lower than reported, then the number of shares to solve a block would be D.

That makes more sense. But even then, not much evidence.

[organofcorti]

That's why Bayesian statistics is superior to frequentist statistics. All the data gives us is about 10:1 likelihood ratio for 1.1D over 1D, which doesn't mean much when the prior probability of 1.1D is so low.

10:1 likelihood ratio? Isn't it 20:1? But I get your point. Perhaps I was a little too rash in discussing what it might mean in terms of possible future miner earnings. But it does mean that either:

1. p2Pool's had a very poor sort of luck.
2. p2Pool's round length mean is larger than it should be.
3. There's a problem with the reported pool statistics.

I think it's worth pointing out the possibility of either 2 or 3, even in absence of a mechanism known to me. Many p2Pool miners are concerned about the pool's luck, and I've read posts by a few that have left the pool for that reason. If the p2Pool devs can come up with an explanation then it might save the pool losing even more miners.

Edit: if there is a problem with the reported stats then p2Pool miners should be receiving on average the expected value for their hashes. Anyone from p2Pool how has been mining consistently care to share their earnings history and hashrate?

[Meni Rosenfeld]

10:1 likelihood ratio? Isn't it 20:1?

According to http://p2pool.info/luck/, 3851 PH were calculated but 3491 PH worth of blocks were found. Assuming difficulty = 1.6M this means 560.4 blocks expected and 508 blocks found. So the likelihood ratio between poisson distributions with mean 560.4 and 508 for 508 found is (508/560.4)^508*exp(560.4-508) ~ 12.5.

[Shadow383]

Where are you getting 5% probability from? I'm seeing more like 1% here.

Left p2pool a while ago, might go back to using it as a backup at least as/if it improves...

[Smoovious]

Um... perhaps something to consider...

When you were evaluating the other pools, one variable that didn't come into play, was the propagation of the new block among the miners.

Since with a centralized pool, long-polling delays aside, everyone is basically mining off of the same daemon.

With P2Pool, this is not the case. We're effectively all solo-miners, with pooled payouts. When one node finds a block, it takes time to propagate, so for a period of time, you would have part of the pool still working on one block, while a growing portion of the pool is working on the new block as the found block gets distributed.

As far as orphans go, a centralized pool is in itself, a single solo miner, albeit one with lots of connections, and bandwidth to distribute the new block efficiently. P2Pool, on the other hand, is a whole bunch of solo miners cooperating.

So, one thought that occurred to me, is when comparing stats, shouldn't solo mining be considered the normal, with the centralized pools, being above normal, due to the efficiency they have in their miners being notified that they should be working on the next block all at once? If they weren't pooled, all of those miners, would be notified at different rates, which may have an affect on orphan rates.

Perhaps, the orphans P2Pool have been experiencing were simply normal for solo mining...

(I'm no mathematician compared to your level, so, I may not be quite clear on what I'm getting at...)

-- Smoov

[Meni Rosenfeld]

@Smoovious: I think it's the other way around. A centralized pool should have more latency than solo because first the pool needs to learn about the block, then it has to long-poll you. When solo mining you just need to learn about the block.

[bitfoo]

@Smoovious: I think it's the other way around. A centralized pool should have more latency than solo because first the pool needs to learn about the block, then it has to long-poll you. When solo mining you just need to learn about the block.

If I understand this right, you're talking about stale shares, not orphans. If I find a block on p2pool, but my bitcoind is only connected to 8 peers, it might take a while for the block I found to propagate through the network. Meanwhile Deepbit finds a block and sends it to 1000 different nodes all at once, orphaning my block.

I don't know the workings of p2pool fully. Do they send newly found blocks to each other as well, to aid in propagating it?

[organofcorti]

Where are you getting 5% probability from? I'm seeing more like 1% here.

Left p2pool a while ago, might go back to using it as a backup at least as/if it improves...

The 95% CI excluded a mean of D, median of ~ 0.693D, and an sd of ~ D. I didn't check the 99% CI. It may also exclude the expected values.

[Meni Rosenfeld]

@Smoovious: I think it's the other way around. A centralized pool should have more latency than solo because first the pool needs to learn about the block, then it has to long-poll you. When solo mining you just need to learn about the block.

If I understand this right, you're talking about stale shares, not orphans. If I find a block on p2pool, but my bitcoind is only connected to 8 peers, it might take a while for the block I found to propagate through the network. Meanwhile Deepbit finds a block and sends it to 1000 different nodes all at once, orphaning my block.

Well, if a stale share happens to be a block, it is technically an orphan block, so this is still an increase in the orphan rate, depending on how you count it (maybe orphan blocks which are known to be orphaned when the pool receives them are not counted). What we care about here is the total rewards of the pool, not how they are distributed internally.

[bitfoo]

Well, if a stale share happens to be a block, it is technically an orphan block, so this is still an increase in the orphan rate, depending on how you count it (maybe orphan blocks which are known to be orphaned when the pool receives them are not counted).

Agreed. So both situations actually affect p2pool negatively. Assuming that the average p2pool node has lower bandwidth and lower connectivity than a big pool, it will generally receive blocks and send out new blocks slower, resulting in more orphans overall.

[Smoovious]

Well, if a stale share happens to be a block, it is technically an orphan block, so this is still an increase in the orphan rate, depending on how you count it (maybe orphan blocks which are known to be orphaned when the pool receives them are not counted).

Agreed. So both situations actually affect p2pool negatively. Assuming that the average p2pool node has lower bandwidth and lower connectivity than a big pool, it will generally receive blocks and send out new blocks slower, resulting in more orphans overall.

Yeah... I think... and, I wasn't around in the early days, not beginning to mine until the last quarter of last year, so, dunno about the early intentions of bitcoin...  but...

When it was designed, I think the intention was for individual peers mining individually. Not sure if pooled mining was the intent from the start. (I'm sure I'll be corrected on this if I'm mistaken )

So, a higher level of orphaned blocks, when scaled up to a lot of peers, may have been an expected metric, and considered the normal situation. Perhaps P2Pool's orphan rate, better reflects the way this would have been, without the centralization of mining?

What the centralized pools did, was cut out the latency from when found blocks, propagated to the other people mining. Since they would inform the mining peers more or less, simultaneously, it would have the effect of near-instant propagation among a large number of peers. Since none of them would have to sanity-check the block against their own blockchain, avoiding the latency that would come with it while distributing it, and not having to pass it along to the next set of peers afterwards, you have a large segment of miners abandoning the current block and starting the new one, much sooner than would have been expected, had everyone been solo-mining. This alone, would have eliminated a portion of what would have been a normal orphan rate.

All of those mining peers, on a centralized pool, would be considered collectively, a single bitcoin peer as far as the network is concerned. Take the top 75% of the hashpower on the bitcoin network, and you end up with, what... a few dozen bitcoin peers? Perhaps a little more? Certainly a tiny fraction of all of the individual bitcoin peers that are mining as a whole, by node count.

I don't know how this could be reflected in useful data, but perhaps this has an affect on assumptions of expected levels of good blocks and orphans within a given period of time?

I think what I'm getting down to, is... perhaps instead of looking to the average pool results to find normal, the pool results, should be considered above-normal instead of the baseline, which would be pure solo mining on a more distributed network?

-- Smoov

[Smoovious]

@Smoovious: I think it's the other way around. A centralized pool should have more latency than solo because first the pool needs to learn about the block, then it has to long-poll you. When solo mining you just need to learn about the block.

If I understand this right, you're talking about stale shares, not orphans. If I find a block on p2pool, but my bitcoind is only connected to 8 peers, it might take a while for the block I found to propagate through the network. Meanwhile Deepbit finds a block and sends it to 1000 different nodes all at once, orphaning my block.

I don't know the workings of p2pool fully. Do they send newly found blocks to each other as well, to aid in propagating it?

forrestv added some protocol changes to aid this very thing, but we're still dealing with a distributed network for the daemon's connectivity, and a separate distributed network for p2pool's connectivity. We basically have another level of notification, letting us know to work on a new block, but we still have the latency of a distributed network, getting that notification propagated. I think I am understanding it right tho, that p2pool's notification, will go much faster, as it deals with less data in the transmission, and less sanity-checking at each node before it gets passed on.

The main difference, the way I see it tho, amounts to the way a centralized vs decentralized system handles how the data gets around. How much it is handled between hand-offs, and how many hand-offs between the finder and the rest of the pool.

Take deepbit as an example... when one of their miners find a block, it is a very short path for the rest of the pool to be notified...

miner1!deepbit!{miner2|...|minerA}

With P2Pool and no centralized bitcoin, and with forrestv's protocol change from last week, we have 2 paths, but they can be pretty long.

miner1!p2pool1!bitcoind!bitcoind2!bitcoind3!bitcoind4!p2pool2!miner2
or
miner1!p2pool1!p2pool2!p2pool3!p2pool4!p2pool5!miner2

The paths the data needs to take, through either route, can be even longer. The route via bitcoind would be slower, and via p2pool would be faster, but both would be much slower than the deepbit path above.

If I remember right, we (p2pool users), have orphaned one of our own blocks before. Something that couldn't happen at the centralized pools.

-- Smoov

[kano]

Did you factor in the high stale rate compared to a normal pool?  

DOA and orphan shares don't make it into the sharechain and so can't increase the numbers of shares in a round.

One possibility is that the method used to estimate the equivalent difficulty 1 shares to solve a block using the pool hashrate is buggy. If for example the pool hashrate is actually 9% lower than reported, then the number of shares to solve a block would be D.

Although they don't affect the number of shares per block difficulty
(which is really the only valid measurement)
DOA and orphan shares reduce the 'calculated' hash rate of people using p2pool.

P2Pool itself does apparently send out DOA and orphan shares to the bitcoind to ensure that shares that are no good for the P2Pool share-chain but are actually a valid BTC-block will still go out.

My other suggestion (that I've said many times) of the cause of p2pool's "misfortune" is that a 'typical' p2pool is quite literally crap compared to a well tuned pool.
Thus the delay between the miner finding a share and the share making it 'out onto the net' would of course be worse on average with p2pool users vs a well tuned, well connected, high performance pool server.
This 2nd item would of course show up in terms of more orphans compared to other well configured pools.

Recent BTC problems may hide this

[organofcorti]

I included orphaned blocks in the average round length calculations, since it was the number of shares taken to create a valid block - so the number of orphaned blocks do not affect the round length / D average. They do still affect miner earnings though.

Rate of orphan production for the first hundred blocks solved was 0%, for the next hundred 1%, for the third hundred 3%, the fourth hundred 6%, and for the first 35 blocks of the sixth hundred blocks, 8.5%. Although there are only 17 orphaned blocks altogether, is does seem that orphan production has increased. This could just be variance though - I have no idea what the mean or variance of the orphaned block distribution should be, and not enough data to even guess.


Edit: and a quick thank you to the recent very generous donator, whoever you are. Much appreciated.
Edit2: actually thanks to the last few donators, especially whomever sent exp(1)/10 btc - I got a laugh out of that