[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[Richy_T]

Any idea why payouts drop off so precipitously when I have any issues with my setup. I mean, obviously the payout builds up slowly over time but surely they should drop of at a similar rate? I made some changes which apparently stopped p2pool from producing valid shares then shortly after that, my potential payout dropped from near .025 btc to around .002 in almost no time.

[Richy_T]

This is what I'm talking about.



If it's shares over the last 24 hours, surely the payout should drop slowly as old shares expire (which it seems to do on Fri 16th to Sat 17th)?

[idonothave]

question was: Have one p2pool process connecting to multiple bitcoind processes via RPC at the same time?
you have answered: No.
how should I undersand this: [BITCOIND_RPCUSERPASS [BITCOIND_RPCUSERPASS ...]]

Fallback option?

You might be able to connect to multiple bitcoinds with a load balancer, maybe.

you mean this one http://lnlb.sourceforge.net/ for example?

[Richy_T]

you mean this one http://lnlb.sourceforge.net/ for example?

Maybe. I don't know enough about it. I would imagine that p2pool talking to bitcoind is simple enough that simple load balancing would be enough but I don't know enough to say for sure.

[yslyung]

windpath, your site is partially working ? seems like it may not be updating.

yeah another block, more pls to cover the dry spell.

[windpath]

windpath, your site is partially working ? seems like it may not be updating.

yeah another block, more pls to cover the dry spell.

Thanks,

Not sure what went down, something crashed bitcoind, which is unusual on my server.

Restarted, will keep an eye on it today.

[jtoomim]

Not sure what went down, something crashed bitcoind, which is unusual on my server.

The congested mempool that we've had recently from the 14kB blocks is causing excessive memory consumption. From my investigation so far, it looks like there's a memory leak in getblocktemplate/CreateNewBlock. This makes bitcoind's memory consumption increase by as much as 1 GB per day. I've been restarting my bitcoind processes every 2 days or so.

[jtoomim]

This is what I'm talking about.



If it's shares over the last 24 hours, surely the payout should drop slowly as old shares expire (which it seems to do on Fri 16th to Sat 17th)?

That looks like the default payout address used by that node was changed. The graph shows the payout for the address that was being used at the time the data point was collected, not at the time the graph was generated.

[jtoomim]

yeah another block, more pls to cover the dry spell.

http://173.48.117.206:9332/static/graphs.html?Day

This one was found by a 5 TH/s miner.

[Richy_T]

This is what I'm talking about.



If it's shares over the last 24 hours, surely the payout should drop slowly as old shares expire (which it seems to do on Fri 16th to Sat 17th)?

That looks like the default payout address used by that node was changed. The graph shows the payout for the address that was being used at the time the data point was collected, not at the time the graph was generated.

It does look that way but is definitely not the case. p2pool has been running since Oct16 with the same address...

user 1677  1668 15 Oct16 pts/3    09:28:11 python ./run_p2pool.py -a 1address

What happened was that someone was saturating my upstream by with a bitcoind connection (presumably downloading the blockchain). I ran a tc.sh script which was apparently over-conservative with the bandwidth settings (both for bitcoind and p2pool). I can understand why that would stop me getting new shares but not that precipitous drop.

[windpath]

Not sure what went down, something crashed bitcoind, which is unusual on my server.

The congested mempool that we've had recently from the 14kB blocks is causing excessive memory consumption. From my investigation so far, it looks like there's a memory leak in getblocktemplate/CreateNewBlock. This makes bitcoind's memory consumption increase by as much as 1 GB per day. I've been restarting my bitcoind processes every 2 days or so.

We keep relatively strict limits for bitcoind's mempool on the node, granted below is only since this mornings restart, I'll keep an eye on it.

Code:

bitcoin-cli getmempoolinfo
{
    "size" : 1967,
    "bytes" : 1745856
}

[jonnybravo0311]

Not sure what went down, something crashed bitcoind, which is unusual on my server.

The congested mempool that we've had recently from the 14kB blocks is causing excessive memory consumption. From my investigation so far, it looks like there's a memory leak in getblocktemplate/CreateNewBlock. This makes bitcoind's memory consumption increase by as much as 1 GB per day. I've been restarting my bitcoind processes every 2 days or so.

We keep relatively strict limits for bitcoind's mempool on the node, granted below is only since this mornings restart, I'll keep an eye on it.

Code:

bitcoin-cli getmempoolinfo
{
    "size" : 1967,
    "bytes" : 1745856
}

I'm inclined to believe the memory leak... although I've only seen it on my Linux (Ubuntu 14.04) node.  I watched bitcoind spool up to consume 8GB of RAM before it was shut down.  On my Mac, using the same codebase, I regularly only see about 400MB of RAM being consumed by the bitcoind.  Both Linux and Mac are built from source.

Anyway, back on topic... I wonder why my reject rate is always so high mining on your node windpath.  My ping time to your server is about 17ms, yet I'm constantly seeing 10% or higher DOA.  When I was running my own VPS node (with about the same ping time), I would typically see about 3-4%.  Yes, running my own local node when I'm at home is considerably lower (usually about 1% or less), and that's to be expected since my miners and my node are all hardwired into the same internal gigabit network.

I'm wondering if it is caused by the traffic on the node, and p2pool's inherent single-threaded execution.  Anyone have thoughts on it?

[windpath]

I'm inclined to believe the memory leak... although I've only seen it on my Linux (Ubuntu 14.04) node.  I watched bitcoind spool up to consume 8GB of RAM before it was shut down.  On my Mac, using the same codebase, I regularly only see about 400MB of RAM being consumed by the bitcoind.  Both Linux and Mac are built from source.

Anyway, back on topic... I wonder why my reject rate is always so high mining on your node windpath.  My ping time to your server is about 17ms, yet I'm constantly seeing 10% or higher DOA.  When I was running my own VPS node (with about the same ping time), I would typically see about 3-4%.  Yes, running my own local node when I'm at home is considerably lower (usually about 1% or less), and that's to be expected since my miners and my node are all hardwired into the same internal gigabit network.

I'm wondering if it is caused by the traffic on the node, and p2pool's inherent single-threaded execution.  Anyone have thoughts on it?

Yea, I'll keep an eye on it, so far mempool seems to be behaving normally.

As to the high reject rate on my node I suspect you are correct.

In addition to the # of miners a lot of folks seem to park their browsers on the stats pages which hit both p2pool and bitcoind to pull some of the data for the front end.

There are some relatively easy fixes like setting the data refresh to stop in the browser after say 30 minutes with a "I'm still here" button to restart it....

I'll look into it.

[jonnybravo0311]

Yea, I'll keep an eye on it, so far mempool seems to be behaving normally.

As to the high reject rate on my node I suspect you are correct.

In addition to the # of miners a lot of folks seem to park their browsers on the stats pages which hit both p2pool and bitcoind to pull some of the data for the front end.

There are some relatively easy fixes like setting the data refresh to stop in the browser after say 30 minutes with a "I'm still here" button to restart it....

I'll look into it.

If I remember that UI correctly, there's a config file that defines the stats refresh rate, which defaults to like 10 seconds for the stats and 30 for the graphs (or something like that).  Of course, you've customized it so much that file has probably been rendered irrelevant at this point .

[jtoomim]

I'm inclined to believe the memory leak... although I've only seen it on my Linux (Ubuntu 14.04) node.  I watched bitcoind spool up to consume 8GB of RAM before it was shut down.  On my Mac, using the same codebase, I regularly only see about 400MB of RAM being consumed by the bitcoind.  Both Linux and Mac are built from source.

Are both bitcoind nodes being used for p2pool? My testing indicates that the use of getblocktemplate is necessary for the memory leaking to occur.

[Meuh6879]

[HS]
i have opened a thread for the memory (and CPU) problem.
https://github.com/bitcoinxt/bitcoinxt/issues/82

every day, i change the maxmempooltx after viewing the history of the CPU (less 100% freeze = increase the setting).

actually, it fluctuate from 800 to 1200.
[/HS]

This problem impact RPC server ... and so, the P2Pool process.

[jonnybravo0311]

I'm inclined to believe the memory leak... although I've only seen it on my Linux (Ubuntu 14.04) node.  I watched bitcoind spool up to consume 8GB of RAM before it was shut down.  On my Mac, using the same codebase, I regularly only see about 400MB of RAM being consumed by the bitcoind.  Both Linux and Mac are built from source.

Are both bitcoind nodes being used for p2pool? My testing indicates that the use of getblocktemplate is necessary for the memory leaking to occur.

Yes.  I was running a VPS (a DigitalOcean droplet) that I used as my backup p2pool node.  I had other miners besides my own on it pretty constantly.  It ran for quite a long time until a few weeks ago it started acting up.  The bitcoind process would die for no apparent reason.  Sometimes the p2pool process would just die off as well.  Finally spending time looking at the box, I watched as the bitcoind would slowly consume more and more memory, up until it took almost the entire 8G available, at which point it would invariably die off.

On my Mac, I also run a p2pool node that is local to my miners at my home.  That bitcoind, just like the one on my droplet, was compiled from source.  On my local node, I've never seen it utilize more than 1G of RAM.  More typically, it's around 400-500MB.  Granted, I have fewer miners on my local home node (2-3 at most with a total hash of about 5TH/s).  On my Linux node, I typically had 5-6 miners with a hash rate of about 15TH/s.

[Polyatomic]

...
 I watched as the bitcoind would slowly consume more and more memory, up until it took almost the entire 8G available, at which point it would invariably die off.

What was the magic you used to produce the output, (8G <-- bitcoind is using this much ram.)
Somethings not right there man.

Linux will try to use alot of the computers memory to keep the system fast. 8 gigs of ram
i'm thinking is below average for power linux users. Is adding more of it an option for you.

Maybe i'll bootstrap a build and run the latest bitcoind and the p2pool to see what the story is.

[p3yot33at3r]

I'm having no issues with my node, in fact, I'm really happy with how it & p2pool are performing in general - especially since I reverted back to the standard bitcoind settings - latency has reduced nicely.

@ windpath: Are the luck stats displayed on your web GUI correct?:

Seven Days    162.74%
Thirty Days    148.03%

If they are, I'm pretty sure there isn't another pool out there that can touch us atm 

[Richy_T]

This is what I'm talking about.



If it's shares over the last 24 hours, surely the payout should drop slowly as old shares expire (which it seems to do on Fri 16th to Sat 17th)?

Any other input on this? If shares are being dropped early, it seems like this is either a bug that is causing it or a bug that is being exploited by a malicious actor (?). Either way, not good. I'm willing to help with diagnosing this if anyone who knows what they're doing is interested. I suspect it's easy enough to duplicate.