A guide for mining efficiently on P2Pool, includes FUD repellent and FAQ

[maqifrnswa]

Solely looking at p2pool stats, not gut feeling.  You said a week was bad, on what basis do you make that statement?  

M

I said last week was bad based on the 7 day sliding average data presented on p2pool. I was comparing data from Sunday the 17th (the original date it was posted) GMT to data from the Thursday the 14th GMT. It's semantics and counterproductive to discuss what dates were chosen for the illustration.

What you're nitpicking really doesn't matter at all, this is not a discussion of whether or not we are lucky. I was just illustrating that people confuse "luck" with "it's broken." And I gave an example how over the course of one week we switched from "it's crap" to "it's the best pool in the history of the universe."

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining. If not enough people are mining on it, it will have huge variance. You were unlucky, but someone else who mined p2pool probably had inverse luck. For example, over the past 7 days (March 22), p2pool paid you 185% of what you would have gotten from PPS (including fees). Last week (March 15th), people mining on p2pool were only paid 85% compared to PPS (including fees). Variance (luck) is a problem for all pools that are underpowered and doesn't mean that it "pays out crap."

[gyverlb]

Scrypt+Stratum fix is in, git users should pull. Others should block stratum by using "--fix-protocol" in cgminer/bfgminer until a new official release comes out.

As always try to keep an eye on new miner and p2pool releases.

[mdude77]

You keep referring to something I didn't write. I hate that: you are really trying my patience.

Sorry, I'm not good with names.  No offense intended.  

What you're nitpicking really doesn't matter at all, this is not a discussion of whether or not we are lucky. I was just illustrating that people confuse "luck" with "it's broken." And I gave an example how over the course of one week we switched from "it's crap" to "it's the best pool in the history of the universe."

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining. If not enough people are mining on it, it will have huge variance. You were unlucky, but someone else who mined p2pool probably had inverse luck. For example, over the past 7 days (March 22), p2pool paid you 185% of what you would have gotten from PPS (including fees). Last week (March 15th), people mining on p2pool were only paid 85% compared to PPS (including fees). Variance (luck) is a problem for all pools that are underpowered and doesn't mean that it "pays out crap."

I wouldn't call it nitpicking.  The fact is under a week ago, 90+ day performance was < 90%.  That is awful in my book.  Is it really luck that suddenly changed this last week?  While I find that hard to believe (and I'm not convinced), I'll defer to the mathematical experts who claim it's all variance and shut up.

M

[organofcorti]

You keep referring to something I didn't write. I hate that: you are really trying my patience.

Sorry, I'm not good with names.  No offense intended.  

What you're nitpicking really doesn't matter at all, this is not a discussion of whether or not we are lucky. I was just illustrating that people confuse "luck" with "it's broken." And I gave an example how over the course of one week we switched from "it's crap" to "it's the best pool in the history of the universe."

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining. If not enough people are mining on it, it will have huge variance. You were unlucky, but someone else who mined p2pool probably had inverse luck. For example, over the past 7 days (March 22), p2pool paid you 185% of what you would have gotten from PPS (including fees). Last week (March 15th), people mining on p2pool were only paid 85% compared to PPS (including fees). Variance (luck) is a problem for all pools that are underpowered and doesn't mean that it "pays out crap."

I wouldn't call it nitpicking.  The fact is under a week ago, 90+ day performance was < 90%.  That is awful in my book.  Is it really luck that suddenly changed this last week?  While I find that hard to believe (and I'm not convinced), I'll defer to the mathematical experts who claim it's all variance and shut up.

M

Oh dear. I come to this thread to read about how to optimise p2Pool. I don't understand the optimisations very well - I just don't have sufficient geek, or a sufficiently stable / low latency network. But it is an interesting read none-the-less.

What aren't interesting reads are the continual claims that something is wrong with the pool, and causing it "bad luck". Here are a few things which anyone should consider when making a "luck" claim one way or the other:

1. Talking about average "luck" over a number of days leads to claims that bad luck periods go for a long time, and good luck for only short periods. This is because when fewer blocks are solved by the pool, the "bad luck" stretches in time. If you instead look at a graph of average luck per block, or better yet the boxplots I posted, you can see that the recent "bad luck" is a small deviation that has actually been shorter than some other deviations. I'll just bold that: Average luck per time period is misleading.

2. This is something Meni Rosenfeld brought up when I was first analysing p2Pool's luck - how could a period of "bad luck" be a problem with the pool? I don't see anyone proposing any mechanisms.

3. The published stats are going to be a little inaccurate. If anonymised earnings per hashrate for a large number of miners could be posted somewhere I think the "luck" discussion would end for all time.

[maqifrnswa]

I wouldn't call it nitpicking.  The fact is under a week ago, 90+ day performance was < 90%.  That is awful in my book.  Is it really luck that suddenly changed this last week?  While I find that hard to believe (and I'm not convinced), I'll defer to the mathematical experts who claim it's all variance and shut up.

M

I am a mathematical expert. If your hash rate solves 2 blocks every day over 90 days, there is a 9.4% chance that in any given 90 day period you'll have < 90% "luck"

If your hash rate solves 1 block every day, there is a 19% chance that you will have less than <90% luck in a 90 day window.

So when p2pool's hash rate was low the pool was more likely to have < 90% luck over 3 months than someone throwing a 6 on a 6 sided dice.

My point is that I'm kind of surprised people think something that has between a 1/10 and 1/5 chance of occurring is a sign it is broken.

[zvs]

i guess i'll say it again, since it's still on there

the statement:

The best results (but probably by a negligible amount) can be obtained when the bitcoind data directory is on tmpfs

is incorrect.  one can make a tmpfs drive that runs "low" on memory ("low" default being 50%?), in which case it'll start using swap space, slowing bitcoind down

with ramfs you have total control over how much RAM to use on the drive and that's all it'll ever use.  it's not too hard to tell how large you should make a ramfs drive.  at this moment, around 8.5gb

[gyverlb]

i guess i'll say it again, since it's still on there

the statement:

The best results (but probably by a negligible amount) can be obtained when the bitcoind data directory is on tmpfs

is incorrect.  one can make a tmpfs drive that runs "low" on memory ("low" default being 50%?), in which case it'll start using swap space, slowing bitcoind down

with ramfs you have total control over how much RAM to use on the drive and that's all it'll ever use.  it's not too hard to tell how large you should make a ramfs drive.  at this moment, around 8.5gb

There are 2 problems with your arguments.

• I use "can be obtained" because it involves more than simply using tmpfs. If you have enough RAM, you indeed can obtain the best results. It's left to the system administrator to find out how to tune and monitor a system using tmpfs.
• tmpfs can indeed use swap, but unlike you wrote last time I checked ramfs isn't bounded. So if your RAM isn't large enough to fit your tmpfs system at least you can set a boundary and avoid unpredictable behaviour. If you use ramfs and bitcoind's data directory grows to a point where it fills your whole RAM it will put everything else in swap and eventually trigger the kernel OOM killer. What will be killed then can only be guessed.

So ramfs isn't a clear-cut better solution here: it depends on how the system admin is configuring the whole setup. Anyway this is for advanced admins: people shouldn't follow a guide for this but fully understand the implications themselves. I wouldn't recommend it but I'll add ramfs as an option in the guide.

To check what I just wrote about ramfs:

Code:

mkdir /mnt/t
mount -t ramfs -o size=20m ramfs /mnt/t
dd if=/dev/zero of=/mnt/t/a bs=1M count=30

You can write 30M to a 20M ramfs (at least on a 3.7.9 kernel)...

Edit: man mount clearly shows that there's no options for the ramfs filesystem. So "-o size=20m" isn't even an initial value or a hint, it's completely ignored.

[brox]

To be fair, the only bad thing about p2pool is Python language. Which makes it completely useless on low-end PCs.
Hope some day p2pool will be integrated efficiently into bitcoind; reserved my donations for that...

[dooferorg]

Scrypt+Stratum fix is in, git users should pull. Others should block stratum by using "--fix-protocol" in cgminer/bfgminer until a new official release comes out.

As always try to keep an eye on new miner and p2pool releases.

Question.. in the OP it didn't mention about setting up a stratum proxy so that miners futher away from your p2pool instance can connect via stratum. How would I do that? I know how to set up a proxy for local getworks TO a stratum receiver, but it's the setting up of a stratum receive to p2pool I don't understand yet.

[gyverlb]

Scrypt+Stratum fix is in, git users should pull. Others should block stratum by using "--fix-protocol" in cgminer/bfgminer until a new official release comes out.

As always try to keep an eye on new miner and p2pool releases.

Question.. in the OP it didn't mention about setting up a stratum proxy so that miners futher away from your p2pool instance can connect via stratum. How would I do that? I know how to set up a proxy for local getworks TO a stratum receiver, but it's the setting up of a stratum receive to p2pool I don't understand yet.

I'm not sure I understand the question. Why do you need a proxy? Do you have a firewall blocking access to your P2Pool node? P2Pool supports both direct getwork and stratum connexions from miners so you shouldn't need a proxy at all.

[davidkassa]

This a great guide - thanks!

I have a couple of questions.

This thread, and others, have said:

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining.

I'm new to bitcoin (as my rep will tell you), but I understand the concept of luck and variance. How specifically does p2pool "out pay" others? I currently mine on a no-fee (keep trans fees) PPLNS pool. My analysis shows that the trans fees are ~1% so p2pool will out pay there. How else? Is there some sort of reduced latency bonus? Or just a "no-downtime" gain? I read somewhere that >100% efficiencies are "stealing" somehow gaining from others with low pool efficiency by getting their hash power but not always giving credit -- or something.

Also, this guide mentions using:

Code:

-Q 0 -g 1 --intensity d --gpu-dyninterval 50

Can you elaborate? the -g sets the miner to a single thread (opposed to 3 on my machine). Isn't that going to reduce my hashrate? the -Q 0 seems inefficient too but I assume that I'm missing something. The --gpu-dyninterval default is 7 but you recommend 50. I don't understand why. These setting seem really important so I really want to understand.

Finally, how is difficultly handled? Hosted pools can auto-adjust this "magically". How can I tell what I should set mine to (or leave it at) and how do I do it?

[organofcorti]

This a great guide - thanks!

I have a couple of questions.

This thread, and others, have said:

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining.

I'm new to bitcoin (as my rep will tell you), but I understand the concept of luck and variance. How specifically does p2pool "out pay" others? I currently mine on a no-fee (keep trans fees) PPS pool. My analysis shows that the trans fees are ~1% so p2pool will out pay there. How else? [....]

Which fee free PPS pool is that? Are there any left? Mining at a fee-free PPS pool might cost less now, but the risks of the pool failing are high so the cost may be more than you imagine.

[davidkassa]

Which fee free PPS pool is that? Are there any left? Mining at a fee-free PPS pool might cost less now, but the risks of the pool failing are high so the cost may be more than you imagine.

Eligius - It's actually CPPSRB - Capped Pay Per Share with Recent Backpay

I understand the value of p2pool but I don't think that a failing pool is all that costly. If you have a failover you shouldn't have any issues when the primary is down. Worst case is that you'd lose out on anything that wasn't distributed to you yet - hopefully < 0.2

[organofcorti]

Which fee free PPS pool is that? Are there any left? Mining at a fee-free PPS pool might cost less now, but the risks of the pool failing are high so the cost may be more than you imagine.

Eligius - It's actually CPPSRB - Capped Pay Per Share with Recent Backpay

I understand the value of p2pool but I don't think that a failing pool is all that costly. If you have a failover you shouldn't have any issues when the primary is down. Worst case is that you'd lose out on anything that wasn't distributed to you yet - hopefully < 0.2

Yes, Eligius is not actually a PPS pool and so my comments don't really apply to it. In general however, I was referring to the risks PPS pools take and the chances of them going into bankruptcy owing you coins. That is costly and has happened, and has more of a chance happening in a fee-free PPS pool, since a long negative buffer will probably wipe them out.

[gyverlb]

This a great guide - thanks!

I have a couple of questions.

This thread, and others, have said:

The mathematical fact still remains that a well tuned p2pool will out pay any other pool. In fact, it will approach solo mining.

I'm new to bitcoin (as my rep will tell you), but I understand the concept of luck and variance. How specifically does p2pool "out pay" others? I currently mine on a no-fee (keep trans fees) PPLNS pool. My analysis shows that the trans fees are ~1% so p2pool will out pay there. How else? Is there some sort of reduced latency bonus? Or just a "no-downtime" gain? I read somewhere that >100% efficiencies are "stealing" somehow gaining from others with low pool efficiency by getting their hash power but not always giving credit -- or something.

Among p2pool users there's indeed a competition for the best efficiency. It shouldn't be a problem though: people with really bad efficiency unable to make it better shouldn't mine on p2pool leaving only the most efficient ones (which will bring them closer to 100% efficiency). This is assuming their goal is maximum rewards, you can mine on p2pool for other reasons (you like the concept or find it a better solution for Bitcoin's overall health/robustness).

The advantage of p2pool vs classic pools is stated in the guide: it should have less orphans than classic pools because it's better connected to the Bitcoin network and thus should propagate blocks faster (this is a theoretical advantage, I'm not aware of anyone having measured it).

One thing I didn't write yet is the ability of merged mining. If you have moderately high hashpower (currently ~5GH/s) it should bring you additional coins. Namecoin has been between 5 and 13% as profitable as Bitcoin recently, being able to merge-mine it (Devcoin and Ixcoin are possible too) is an additional bonus. For example I set NMC merged-mining up early just to tinker with it (it wasn't really worth my time) and found a quite nice surprise when looking at my NMC wallet last month: 1000+ NMC where waiting there...

Also, this guide mentions using:

Code:

-Q 0 -g 1 --intensity d --gpu-dyninterval 50

Can you elaborate? the -g sets the miner to a single thread (opposed to 3 on my machine). Isn't that going to reduce my hashrate?

It depends on your setup, but I didn't measure any speedup with -g 2 on mine. The higher the number of threads and the higher your latency is, so if it doesn't bring any measurable benefit, you should keep it down.

I'm not sure about --gpu-dyninterval default value but the goal is to keep the GPU latency low without sacrificing too much hashrate. I've measured a ~0.3% lower hashrate when going from 250ms to 50ms but the amount of stales/doa was much lower too. Going lower was eating too much hashrate and I believe 50ms to be much lower than most people latencies with the P2Pool network.

These are good starting point, but if you want to grab a 1% worth of Bitcoin more you might want to study the impact of modifying them by comparing the <hashrate>*<efficiency> (which is what matters for your p2pool's income) value for long runs (at least 24h).

the -Q 0 seems inefficient too

This is what is advised by p2pool. It seemed inefficient to me too, but I couldn't measure any impact vs -Q 1.

Finally, how is difficultly handled? Hosted pools can auto-adjust this "magically". How can I tell what I should set mine to (or leave it at) and how do I do it?

Difficulty is auto-adjusted automatically so that your miners keep submitting ~1 share every second (to avoid too much traffic and load and still get a good estimate of your hashrate).

[davidkassa]

Great! I think I'm going to try p2pool this weekend and keep Eligius as my backup.

Interesting that you mentioned merged mining because I just asked this question on StackExchange today too:
http://bitcoin.stackexchange.com/questions/9999/merged-mining-disadvantages

[mdude77]

Great! I think I'm going to try p2pool this weekend and keep Eligius as my backup.

Interesting that you mentioned merged mining because I just asked this question on StackExchange today too:
http://bitcoin.stackexchange.com/questions/9999/merged-mining-disadvantages

Note: if you use cgminer, and probably bfgminer, you don't want to mix p2pool and "conventional" pools.  They don't mix well.

M

[davidkassa]

hmmm, can you elaborate? if my p2pool process falls over what will behave differently with my backup server? I know that I lowered my queue and thread count slightly, but inefficient hashing is better than downtime, right? TIA.

[gyverlb]

Great! I think I'm going to try p2pool this weekend and keep Eligius as my backup.

Interesting that you mentioned merged mining because I just asked this question on StackExchange today too:
http://bitcoin.stackexchange.com/questions/9999/merged-mining-disadvantages

Note: if you use cgminer, and probably bfgminer, you don't want to mix p2pool and "conventional" pools.  They don't mix well.

M

They do. I've seen my rigs fallback to backup pools without problems when I stopped my p2pool node on numerous occasions.

[mdude77]

Great! I think I'm going to try p2pool this weekend and keep Eligius as my backup.

Interesting that you mentioned merged mining because I just asked this question on StackExchange today too:
http://bitcoin.stackexchange.com/questions/9999/merged-mining-disadvantages

Note: if you use cgminer, and probably bfgminer, you don't want to mix p2pool and "conventional" pools.  They don't mix well.

M

They do. I've seen my rigs fallback to backup pools without problems when I stopped my p2pool node on numerous occasions.

I'm just going by what the cgminer authors have said.  cgminer goes into different logic for p2pool than it does for conventional pools.  I believe the words were "do so at your own risk".

M