[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[whalingoutbox]

Hello,

I'm still trying to understand how P2Pool works. How do I find a list of P2Pool servers? The idea is that I make a node and connect to a server, right? I can't quite visualize how all of this works, despite looking on Google for information visualizations. The image I saw on the Bitcoin wiki didn't make much sense, seemed too technical.

There are multiple servers, right?

Is there like one main server? Or do a bunch of nodes together make the server?

I read and read but it doesn't stick/make-sense. So, hopefully, I can talk to people about this.

[1713944432]

1713944432

[1713944432]

1713944432

[1713944432]

1713944432

[1713944432]

1713944432

[in2tactics]

Hello,

I'm still trying to understand how P2Pool works. How do I find a list of P2Pool servers? The idea is that I make a node and connect to a server, right? I can't quite visualize how all of this works, despite looking on Google for information visualizations. The image I saw on the Bitcoin wiki didn't make much sense, seemed too technical.

There are multiple servers, right?

Is there like one main server? Or do a bunch of nodes together make the server?

I read and read but it doesn't stick/make-sense. So, hopefully, I can talk to people about this.

Each node is its own pool. p2pool is a collection of pools. You can either make your own node or use another.

[frodocooper]

I'm still trying to understand how P2Pool works. How do I find a list of P2Pool servers? The idea is that I make a node and connect to a server, right? I can't quite visualize how all of this works, despite looking on Google for information visualizations. The image I saw on the Bitcoin wiki didn't make much sense, seemed too technical.

This P2Pool guide should be able to help. It's a lot more beginner-friendly than the Bitcoin wiki's article.
And here is an up-to-date list of public P2Pool nodes that you can connect your miners to, if you decide to not run your own P2Pool node.

There are multiple servers, right?

Yes, in a sense. There are multiple P2Pool nodes, and all of them make up the P2Pool network. It is very similar to how the Bitcoin network works, where multiple Bitcoin full nodes make up the Bitcoin network.

Is there like one main server? Or do a bunch of nodes together make the server?

P2Pool is a decentralized pool, similar to how Bitcoin is a decentralized network. There is therefore no main or central server, nor nodes that make up a main or central server. There is only the network of P2Pool nodes.

In other words, try to look at it according to what in2tactics said: each P2Pool node is its own pool. The P2Pool network connects these nodes together through the P2Pool sharechain, similar to how the Bitcoin network connects every Bitcoin full node together through the Bitcoin blockchain. P2Pool can therefore also be described as a collection of solo miners that pool block payouts and distribute them accordingly, since each P2Pool node is essentially doing its own thing. Contrast this to a traditional pool, where the central pool server dictates what its miners do.

[jtoomim]

Quick reminder: the jtoomimnet 1mb_hardforked branch of p2pool uses about 2x as much RAM, since the share chain contains around 2x as many transactions. If you are using pypy, this means that your memory consumption may get up to 6 GB. If you're using CPython, it's a little under 1 GB. Please make sure that you have enough RAM in your nodes. If you don't have enough RAM, you will get massive swapping, and you might notice that bitcoind is unable to keep up with blocks and may fall behind, causing p2pool to stop working.

Lowering RAM usage is my next goal for p2pool.

[tubexc]

Quick reminder: the jtoomimnet 1mb_hardforked branch of p2pool uses about 2x as much RAM, since the share chain contains around 2x as many transactions. If you are using pypy, this means that your memory consumption may get up to 6 GB. If you're using CPython, it's a little under 1 GB. Please make sure that you have enough RAM in your nodes. If you don't have enough RAM, you will get massive swapping, and you might notice that bitcoind is unable to keep up with blocks and may fall behind, causing p2pool to stop working.

Lowering RAM usage is my next goal for p2pool.

That's good.
And what about the p2pool ltc node testing and then the
merge into only one p2pool powerfull bitcoin sharechain?

[jtoomim]

And what about the p2pool ltc node testing

LTC now requires SegWit, which requires veqtrus's PR. It would take some more work to get LTC working with both my code and veqtrus's. It would need to be a different alt for testing.

At the moment, I'm more interested in the memory issues and the share size issues than in doing testing and merging. I'd also like to add some code that checks the system clock against NTP (when reachable) at startup in order to reduce the clock offset issues that people have been having, and I think that might be good to do before merging into p2pool master.

[Meuh6879]

I'd also like to add some code that checks the system clock against NTP (when reachable) at startup in order to reduce the clock offset issues

use a preformated NTP list : http://www.prunoqi.com/~probruno/WORD/Liste%20des%20serveurs%20de%20temps%20SNTP%20disponibles%20sur%20Internet.htm

[KorbinDallas]

Queue the block dance guy ......

[frodocooper]

Empty block by jtoomimnet.

[in2tactics]

Empty block by jtoomimnet.

Well, there was only 24 seconds between block 468172 and 468173.

[frodocooper]

Empty block by jtoomimnet.

Well, there was only 24 seconds between block 468172 and 468173.

With 110 MB of transactions waiting in the Bitcoin mempool during that time. jtoomimnet should've been able to fill its transaction cache back up to capacity with the share that found block 468173, but apparently did not.

[jtoomim]

It's more an issue of processing and propagation latency than mempool size. P2pool (along with most other pools) has code in it to generate empty blocks when the pool has heard of a new block header on the network, but has not yet gotten a block template from bitcoind. bitcoind can't generate a new block template until after it has fully downloaded and verified the previous block, since otherwise it might include transactions twice or include double-spends. Since downloading a block header is faster than downloading and verifying a full block, there can sometimes be a substantial amount of time in between where the best you can do is mine an empty block. The idea is that if you have a miner hashing on something, it's better to hash on an empty block at the right height than a full block at the wrong height (which would result in an orphan race). This condition should only last for a few seconds, though. The 23 actual seconds that elapsed seems excessive. It's possible that I need to do something to improve block propagation latency on that node. It's also possible that p2pool didn't poll getblocktemplate frequently enough or something.

If this happens when blocks are found in the first 30 15 seconds of mining, that would correspond to a 5% 2.5% reduction in average transactions included. That seems undesirable, but not excessive. The alternative -- mining on the wrong block height for up to 15 seconds -- sounds worse. I wish I had time to look into this more and improve performance, but at the moment I don't.

Edit: It appears that the timestamps on the blocks are wrong. This is pretty common, as generally timestamps are set when the stratum job is issued to the miner, not when the miner actually finds the block. Here's what shows up in my bitcoind logs:

Code:

2017-05-26 05:03:32 UpdateTip: new best=0000000000000000019c0b9124dd971ca31d59deac39f23cac484495895eb2af  height=468
172  log2_work=86.489404  tx=226059198  date=2017-05-26 05:02:56 progress=1.000000  cache=144.7MiB(44861tx)
2017-05-26 05:03:32 CreateNewBlock(): total size 999899 txs: 2197 fees: 296309213 sigops 7618
2017-05-26 05:03:35 Acceptable block: ver:20000000 time:1495775000 size: 1623 Tx:1 Sig:41
2017-05-26 05:03:35 UpdateTip: new best=00000000000000000117d37aba3cd8d9d1811f4f719834f5b8f71c78f8c504d7  height=468173  log2_work=86.489438  tx=226059199  date=2017-05-26 05:03:20 progress=1.000000  cache=144.7MiB(44877tx)

Then, in my p2pool logs:

Code:

2017-05-25 22:03:20.093931 Skipping from block 156cb130cb9ee78a01dd6d5ed4f144fb05c222f117829e7 to block 19c0b9124dd971ca31d59deac39f23cac484495895eb2af!
2017-05-25 22:03:20.102054 New work for 1PXxBrUbWUMZemAQknEqDTSHKaKmVmhJCK! Diff: 65681.48 Share diff: 9893593.66 Block value: 12.50 BTC (0 tx, 0 kB)
2017-05-25 22:03:34.275890 Generating a share with 998819 bytes (348901 new) and 2197 transactions (678 new)
2017-05-25 22:03:34.316686 New work for 1PXxBrUbWUMZemAQknEqDTSHKaKmVmhJCK! Diff: 69877.30 Share diff: 9893593.66 Block value: 15.46 BTC (2197 tx, 999 kB)
2017-05-25 22:03:34.384701 Generating a share with 998819 bytes (348901 new) and 2197 transactions (678 new)
2017-05-25 22:03:34.788256 
2017-05-25 22:03:34.788319 GOT BLOCK FROM MINER! Passing to bitcoind! https://blockchain.info/block/00000000000000000117d37aba3cd8d9d1811f4f719834f5b8f71c78f8c504d7
2017-05-25 22:03:34.788340 
2017-05-25 22:03:34.791621 GOT SHARE! 1EyWF5ZQ9BHxbLAKuFj2MfQT9daE1sVsTx f8c504d7 prev 91af17f0 age 14.68s DEAD ON ARRIVAL
2017-05-25 22:03:34.802307 

It's worth noting that this was a DOA share that happened to be a block. P2pool learned of the block at height 468172 at 3m20s, bitcoind learned of it 12 seconds later at 3m32s and had work for p2pool around that time, but p2pool didn't have it bundled into a stratum job until 3m34.7s. While p2pool was bundling the stratum job, a miner returned the block from the previous stratum job, and p2pool didn't process it until after it finished assigning the new work because p2pool is single-threaded. This means that the block was found at some point between 3m32s and 3m34.7s.

This node was running CPython, not pypy, because that server is running low on RAM. This may have contributed to the slow (2.7s) processing of new work, and increases the likelihood of a DOA share.

So in this case, we're talking about a 15 second time window, not 30 or 24. Of those 15 seconds, 12 seconds were due to bitcoind being a bit slow, and 3 seconds were due to p2pool being slow. Overall, I think finding an empty block in this scenario is a reasonable outcome, and do not consider this to be a priority for optimization right now.

[kano]

It's more an issue of processing and propagation latency than mempool size. P2pool (along with most other pools) has code in it to generate empty blocks when the pool has heard of a new block header on the network, but has not yet gotten a block template from bitcoind.
...

This is based on the fact that LukeJr is a crappy coder and came up with the idea that a block change with transactions is slow, so do not put transaction in the first work.

This is of course bullshit.

[jtoomim]

I'm having trouble following your assertion. Is 14.68 seconds not slow to you? Or is it bullshit that it's so slow? Or is it bullshit to mine transaction-less blocks during those 14.68 seconds? Or is it just bullshit because Luke-jr was involved?

[kano]

I'm having trouble following your assertion. Is 14.68 seconds not slow to you? Or is it bullshit that it's so slow? Or is it bullshit to mine transaction-less blocks during those 14.68 seconds? Or is it just bullshit because Luke-jr was involved?

Takes me less than 1 second - so I guess you have the same problem as LukeJr.
(less than 1 second includes: from block arrives, processed, work generated, new work sent out to miners)

[jtoomim]

Takes me less than 1 second - so I guess you have the same problem as LukeJr.
(less than 1 second includes: from block arrives, processed, work generated, new work sent out to miners)

We're not counting the same way. In my example above, it took 12 seconds (after p2pool received the header) for the block to arrive, less than one second for the block to be validated and a new block template to be pushed to p2pool, and 2.7 seconds (because CPython p2pool is slow) for p2pool to issue new work to the miners. You're not counting what took 12 seconds in the case above.

[kano]

Takes me less than 1 second - so I guess you have the same problem as LukeJr.
(less than 1 second includes: from block arrives, processed, work generated, new work sent out to miners)

We're not counting the same way. In my example above, it took 12 seconds (after p2pool received the header) for the block to arrive, less than one second for the block to be validated and a new block template to be pushed to p2pool, and 2.7 seconds (because CPython p2pool is slow) for p2pool to issue new work to the miners. You're not counting what took 12 seconds in the case above.

Thus you are saying there's some problem with p2pool receiving blocks.
That 12 seconds doesn't exist for me.

Edit: here's a link to make you worry more about p2pool ... https://poolbench.antminer.link/

[HeroC]

Takes me less than 1 second - so I guess you have the same problem as LukeJr.
(less than 1 second includes: from block arrives, processed, work generated, new work sent out to miners)

We're not counting the same way. In my example above, it took 12 seconds (after p2pool received the header) for the block to arrive, less than one second for the block to be validated and a new block template to be pushed to p2pool, and 2.7 seconds (because CPython p2pool is slow) for p2pool to issue new work to the miners. You're not counting what took 12 seconds in the case above.

Thus you are saying there's some problem with p2pool receiving blocks.
That 12 seconds doesn't exist for me.

Edit: here's a link to make you worry more about p2pool ... https://poolbench.antminer.link/

Maybe it's just my internet, but that link you sent never actually loaded for me. It's just been loading for like a minute.

EDIT: Another minute later it loaded, nevermind

[jtoomim]

Thus you are saying there's some problem with p2pool receiving blocks.
That 12 seconds doesn't exist for me.

No, the 12 seconds was due to my bitcoind process, not p2pool, as you would know if you had read all of my original post. I'm not running Falcon or FIBRE on my node, just theblumatt's old relay network, which is probably why it took 12 seconds. If you think it is bullshit that it took 12 seconds to download that block, then I'm inclined to agree with you. If you think that it's bullshit that p2pool choose to mine an empty block during those 12+2.7 seconds, then I'm inclined to disagree with you.

[kano]

Thus you are saying there's some problem with p2pool receiving blocks.
That 12 seconds doesn't exist for me.

No, the 12 seconds was due to my bitcoind process, not p2pool, as you would know if you had read all of my original post. I'm not running Falcon or FIBRE on my node, just theblumatt's old relay network, which is probably why it took 12 seconds. If you think it is bullshit that it took 12 seconds to download that block, then I'm inclined to agree with you. If you think that it's bullshit that p2pool choose to mine an empty block during those 12+2.7 seconds, then I'm inclined to disagree with you.

No, I'm saying there is no reason to mine empty blocks due to the first work generated on a block change.
If your stats say otherwise, then you need to fix your stats, not mine empty blocks to overcome a crappy setup or crappy code.