[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[frodocooper]

Lightning's Next Act: Decentralizing Bitcoin Mining?

https://www.coindesk.com/hub-and-spoke-could-bitcoins-lightning-network-decentralize-mining/

There is a dedicated thread for it here.

[1714112282]

1714112282

[1714112282]

1714112282

[1714112282]

1714112282

[1714112282]

1714112282

[jtoomim]

Oh, looks like jtoomimnet mined a block two days ago:

https://blockchain.info/block-index/1626545

1004.743 kB, 3999.157 kWU. Thanks to veqtrus for writing the code for p2pool to support Segwit.

15.1513745 BTC block reward.

[frodocooper]

Oh, looks like jtoomimnet mined a block two days ago:

https://blockchain.info/block-index/1626545

1004.743 kB, 3999.157 kWU. Thanks to veqtrus for writing the code for p2pool to support Segwit.

15.1513745 BTC block reward.

Very nice.

It didn't show up on my node's web front-end, though.

[jtoomim]

It didn't show up on my node's web front-end, though.

Yeah, it seems that might be a bug. I've noticed once before that we get paid without it showing on the front-end. Better that way than the other way around, I suppose.

[Comandante77]

Hi, everybody!

How can I configure p2pool with other coin? Just interesting fo CANN.

[sawa]

Hi, everybody!

How can I configure p2pool with other coin? Just interesting fo CANN.

https://github.com/ilsawa/p2pool-cann

[Crosshair]

I just finished a beta version of a merge of veqtrus's SegWit code with my 1mb_hardforked+lowmem code. I've also added a couple of things that should improve the likelihood of it working with altcoins. I'm testing with Bitcoin right now. I'll try to have the testing done and the Bitcoin code ready for others to use within a couple of days, and then I'll work on the altcoin stuff and test it with Litecoin.

[Comandante77]

https://github.com/ilsawa/p2pool-cann

Great! Thanks!

[giuliano_ojeda]

Hello Everybody, first of all I would like to thank the developers of p2pool for such a wonderful job. We are Raptor Pool and we just launched a p2pool node in Paraguay(South America) raptorpool.com or es.raptorpool.com (In Spanish). We invite everyone in south america to join us. One more thing, We would like to ask whoever is in charge of the node scanner dev if it's possible to add a flag (Paraguayan Flag) for 190.52.135.28 and the location pin in the map? that will help everyone that's nearby to find out about us.

Thanks again and Best regards, 

[Comandante77]

Hi, everybody!

How can I configure p2pool with other coin? Just interesting fo CANN.

https://github.com/ilsawa/p2pool-cann

 Hmm, trying to set pool for solo, but it still connect to other nodes.

 Changed in p2pool/networks.py

PERSIST = False
BOOTSTRAP_ADDRS = ''.split(' ')

  but it doesnt work.

[jtoomim]

I pushed some more code to https://github.com/jtoomim/p2pool/tree/1mb_segwit that adds more CPU and (maybe) RAM performance improvements for p2pool. I think it should improve CPU usage and latency by about 30% or more. These improvements should reduce DOA and orphan rates on the network a bit. It looks like running p2pool on CPython with medium-slow CPUs should now be viable without huge DOA/orphan costs, although I still recommend using pypy whenever possible. The new code makes fewer memory allocations when serializing objects for network transmission, which might reduce total memory consumption, or it might not. We'll see in a few days.

I also added a performance profiling/benchmark mode. If p2pool is too slow for you, I would find it helpful if you ran python run_p2pool.py --bench and then sent me a snippet of the output, especially if you can get the output near when a share is received over the network.

[Xantus]

I pushed some more code to https://github.com/jtoomim/p2pool/tree/1mb_segwit that adds more CPU and (maybe) RAM performance improvements for p2pool. I think it should improve CPU usage and latency by about 30% or more. These improvements should reduce DOA and orphan rates on the network a bit. It looks like running p2pool on CPython with medium-slow CPUs should now be viable without huge DOA/orphan costs, although I still recommend using pypy whenever possible. The new code makes fewer memory allocations when serializing objects for network transmission, which might reduce total memory consumption, or it might not. We'll see in a few days.

I also added a performance profiling/benchmark mode. If p2pool is too slow for you, I would find it helpful if you ran python run_p2pool.py --bench and then sent me a snippet of the output, especially if you can get the output near when a share is received over the network.

okey, thanks. i will take a look these days. would by nice to get it run on Win10 with Pyton 2.7 (pypy does not run here ...)

[Cryptonomist]

Hello,

I've got some questions related to P2Pool:

1) On the web interface of my node I find the following in relation to the shares: "Best share", "Verified heads", "Heads", "Verified tails", "Tails". I'm not sure I fully understand what they mean.
      *I suppose "best share" is the last share added to the sharechain (I'm I right about this???).
      *"Verified heads" contains several shares. I see that one of these shares is the one mentioned in "best share". So what are the other      shares mentioned? Other heads that are or will be orphaned or so?
      *What is the difference between "Verified heads" and "heads"?
      *What are the "Verified Tails" and "Tails"?

2) When I look to the "Best share" share, then I see stuff like "Time first seen", "Peer first received from", "Timestamp".
      *I suppose "Time first seen" gives the time that my node received a Inv (or the p2pool equivalent) from another node containing the hash of the share. Is this correct? Or is it the time when my node has completely downloaded the share?
      *"Peer first received from" gives then probably the node that has send my node the Inv containing the share.
      *"Timestamp" is the timestamp from the node that has found the share I guess. But this timestamp probably depends on the accuracy of that nodes clock, and can in theory deviate from reality?

3) Say for example that I want to know the time it takes my node to receive the latest share of the sharechain (so I want to know the time it takes my node to converge to the consensus sharechain). I probably can use the timestamp of the share gegenerated by the node that found the share, as it will be very close to the moment that it sends it to the rest of the network. So if I can also get the time that my node has downloaded the share, and restarts its mining work on top of the new share, I can in principal determine the time that it took the share to propagate from the node that has found the share to my node, and as such I can determine the "dwell time" of my node. Is this correct?

4) If my reasoning in point 3 is correct, I just need to find the time when my node is finished downloading the new share, and the share is added to the local sharechain. Is there a way to access the sharechain of my node directly through a log file or something? I know that p2pool creates logs in the directory /p2pool/data/bitcoin/, and occasionaly I find references to shares downloaded and stuff like that in the output. But is there another file for the sharechain, that contains time data of when a new share is downloaded and added to the share chain?

Thank you in advance

[jtoomim]

okey, thanks. i will take a look these days. would by nice to get it run on Win10 with Pyton 2.7 (pypy does not run here ...)

I was able to get pypy running on Windows 10 by using the Linux Subsystem for Windows:

https://bitcointalk.org/index.php?topic=18313.msg21025074#msg21025074

However, the reason I made these CPU improvement changes was in the hope that it would allow people to run CPython on medium-speed CPUs without crazy orphan rates. I'm very much interested in hearing whether it works.

[jtoomim]

1) On the web interface of my node I find the following in relation to the shares: "Best share", "Verified heads", "Heads", "Verified tails", "Tails". I'm not sure I fully understand what they mean.

P2pool has a share chain that is very much like the Bitcoin blockchain, with a few differences. The tip of a chain (a block/share with no children) is known as a head. The root of a chain is known as the genesis block in Bitcoin (the block with no parent), but in p2pool it's known as a tail. P2pool's tails *do* have parents, unlike bitcoin, and consequently have a height greater than 1; however, they're called tails in p2pool because their parents have been pruned off from the node's copy of the share chain, so as far as your node is concerned, they have no living parents.

The best share is always a head. It's usually the head with the most work behind it, but not always. The rules for determining the head are a little complicated, and are different in different versions of p2pool.

     *I suppose "best share" is the last share added to the sharechain (I'm I right about this???).

Not necessarily, but usually. Just because a share is recent doesn't mean that it is placed at the tip of the chain.

     *"Verified heads"  -- Other heads that are or will be orphaned or so?

Yes, precisely.

     *What is the difference between "Verified heads" and "heads"?
      *What are the "Verified Tails" and "Tails"?

I'm not sure. I think unverified heads or tails are shares that have not yet been connected to the share chain, such as for a share that is was just downloaded during initial sync but whose ancestors have not yet been downloaded. However, I haven't verified that hypothesis, as I have not yet needed to work on that part of the code.

     *I suppose "Time first seen" gives the time that my node received a Inv ... Or is it the time when my node has completely downloaded the share?

Completely downloaded and mostly deserialized.

     *"Peer first received from" gives then probably the node that has send my node the Inv containing the share.

Yes, but it's a sendShares or share_reply message.

     *"Timestamp" is the timestamp from the node that has found the share I guess. But this timestamp probably depends on the accuracy of that nodes clock, and can in theory deviate from reality?

Correct. In the current p2pool master, the timestamp is required to be between 1 second after the previous share's timestamp and 59 seconds after the previous share's timestamp. Since the timestamps are used for minimum share difficulty adjustments, this can result in a situation where the share difficulty is very slow to adjust downwards after a rapid decrease in the network hashrate. In the jtoomimnet fork, I got rid of this timestamp clipping, and replaced it with a rule that rejects shares that are timestamped to come from the future in order to prevent share difficulty manipulation.

Keep in mind that the timestamp is the time at which the mining job was sent to the hardware, not the time at which the hardware returned the nonce value.

3) Say for example that I want to know the time it takes my node to receive the latest share of the sharechain (so I want to know the time it takes my node to converge to the consensus sharechain). I probably can use the timestamp of the share gegenerated by the node that found the share, as it will be very close to the moment that it sends it to the rest of the network.

Nope, it's not close. It will usually be off by around 5-50 seconds. But what you can do is use the "Peer first received from" line to browse to the info page for that share on the peer's address (you may have to guess the port number), and repeat, until eventually you get to a peer that says "Peer first received from: Null" for that share, and then you can use the "Time first seen" shown by that ndoe (or just the node that you got the share from, if you're lazy).

When I've done this measurement in the past, it's usually around 1 second per hop for nodes running CPython with 100 ms network latency, and around half that for nodes running pypy.

4) If my reasoning in point 3 is correct, I just need to find the time when my node is finished downloading the new share, and the share is added to the local sharechain. Is there a way to access the sharechain of my node directly through a log file or something? I know that p2pool creates logs in the directory /p2pool/data/bitcoin/, and occasionaly I find references to shares downloaded and stuff like that in the output. But is there another file for the sharechain, that contains time data of when a new share is downloaded and added to the share chain?

If you run my code (e.g. 1mb_segwit), you can install the rfoo package for python, then run run_p2pool.py with the --rconsole command line option, and then run the rconsole command in a separate terminal. This will give you a python interpreter that effectively runs inside the p2pool process, and allows you access to all of the variables and functions of p2pool while it's running. The share objects can be found in the gnode.tracker.verified object.
1mb_segwit also now has a --bench command-line option, which I think you would find interesting.

You could also maybe parse the log in data/bitcoin/log. That's just the stdout output of p2pool for (usually) the last few days.

[Xantus]

However, the reason I made these CPU improvement changes was in the hope that it would allow people to run CPython on medium-speed CPUs without crazy orphan rates. I'm very much interested in hearing whether it works.

okey, first night run is done. Synchronising and getting down the share dificulty to test run with 4Th was take somethink about 4 hours. This morning i switched the
Miner Hash rate down to somethink about 1Th.
CPU last is now down to 10-15% avergade, put there are peaks to 30% :
http://www.c-xantus.de/Zwischenablage03.jpg

my Miner dont runs 24Hr a day but p2pool node is alwas on an reachabel at:
http://212.43.81.114:9332

config is Windows 10 64Bit
http://www.c-xantus.de/Zwischenablage02.jpg

here is the Log File (i dont Start the Debug mode ..):
www.c-xantus.de/log

i will take a look for a few more days. Efficenity is better now, but i could be better 

[Cryptonomist]

1) On the web interface of my node I find the following in relation to the shares: "Best share", "Verified heads", "Heads", "Verified tails", "Tails". I'm not sure I fully understand what they mean.

P2pool has a share chain that is very much like the Bitcoin blockchain, with a few differences. The tip of a chain (a block/share with no children) is known as a head. The root of a chain is known as the genesis block in Bitcoin (the block with no parent), but in p2pool it's known as a tail. P2pool's tails *do* have parents, unlike bitcoin, and consequently have a height greater than 1; however, they're called tails in p2pool because their parents have been pruned off from the node's copy of the share chain, so as far as your node is concerned, they have no living parents.

The best share is always a head. It's usually the head with the most work behind it, but not always. The rules for determining the head are a little complicated, and are different in different versions of p2pool.

     *I suppose "best share" is the last share added to the sharechain (I'm I right about this???).

Not necessarily, but usually. Just because a share is recent doesn't mean that it is placed at the tip of the chain.

     *"Verified heads"  -- Other heads that are or will be orphaned or so?

Yes, precisely.

     *What is the difference between "Verified heads" and "heads"?
      *What are the "Verified Tails" and "Tails"?

I'm not sure. I think unverified heads or tails are shares that have not yet been connected to the share chain, such as for a share that is was just downloaded during initial sync but whose ancestors have not yet been downloaded. However, I haven't verified that hypothesis, as I have not yet needed to work on that part of the code.

     *I suppose "Time first seen" gives the time that my node received a Inv ... Or is it the time when my node has completely downloaded the share?

Completely downloaded and mostly deserialized.

     *"Peer first received from" gives then probably the node that has send my node the Inv containing the share.

Yes, but it's a sendShares or share_reply message.

     *"Timestamp" is the timestamp from the node that has found the share I guess. But this timestamp probably depends on the accuracy of that nodes clock, and can in theory deviate from reality?

Correct. In the current p2pool master, the timestamp is required to be between 1 second after the previous share's timestamp and 59 seconds after the previous share's timestamp. Since the timestamps are used for minimum share difficulty adjustments, this can result in a situation where the share difficulty is very slow to adjust downwards after a rapid decrease in the network hashrate. In the jtoomimnet fork, I got rid of this timestamp clipping, and replaced it with a rule that rejects shares that are timestamped to come from the future in order to prevent share difficulty manipulation.

Keep in mind that the timestamp is the time at which the mining job was sent to the hardware, not the time at which the hardware returned the nonce value.

3) Say for example that I want to know the time it takes my node to receive the latest share of the sharechain (so I want to know the time it takes my node to converge to the consensus sharechain). I probably can use the timestamp of the share gegenerated by the node that found the share, as it will be very close to the moment that it sends it to the rest of the network.

Nope, it's not close. It will usually be off by around 5-50 seconds. But what you can do is use the "Peer first received from" line to browse to the info page for that share on the peer's address (you may have to guess the port number), and repeat, until eventually you get to a peer that says "Peer first received from: Null" for that share, and then you can use the "Time first seen" shown by that ndoe (or just the node that you got the share from, if you're lazy).

When I've done this measurement in the past, it's usually around 1 second per hop for nodes running CPython with 100 ms network latency, and around half that for nodes running pypy.

4) If my reasoning in point 3 is correct, I just need to find the time when my node is finished downloading the new share, and the share is added to the local sharechain. Is there a way to access the sharechain of my node directly through a log file or something? I know that p2pool creates logs in the directory /p2pool/data/bitcoin/, and occasionaly I find references to shares downloaded and stuff like that in the output. But is there another file for the sharechain, that contains time data of when a new share is downloaded and added to the share chain?

If you run my code (e.g. 1mb_segwit), you can install the rfoo package for python, then run run_p2pool.py with the --rconsole command line option, and then run the rconsole command in a separate terminal. This will give you a python interpreter that effectively runs inside the p2pool process, and allows you access to all of the variables and functions of p2pool while it's running. The share objects can be found in the gnode.tracker.verified object.
1mb_segwit also now has a --bench command-line option, which I think you would find interesting.

You could also maybe parse the log in data/bitcoin/log. That's just the stdout output of p2pool for (usually) the last few days.

Ok great. Thank you for the clarifications. I'll try the methodology you suggest and see if I can get the information I need.

[jtoomim]

However, the reason I made these CPU improvement changes was in the hope that it would allow people to run CPython on medium-speed CPUs without crazy orphan rates.

config is Windows 10 64Bit

A 2.0 GHz A10 is not what i would call a medium-speed CPU, unfortunately. Sure, it might have 4 cores, but Python is effectively single-threaded, so that doesn't help at all. Pre-Ryzen AMD chips have poor single-threaded performance, so a 2 GHz A10 is equivalent in performance to a roughly 1.3 GHz Core i3 (if such a thing existed). I wish I could get p2pool to run nicely on such a CPU, but I'm afraid that's out of reach at the moment.

You've got plenty of RAM, so I think you can get it to work decently if you go through the steps I described to get Pypy running on Windows.

Also, if you could run p2pool with the --bench option, that would make the log file much more useful to me.

[rav3n_pl]

okey, thanks. i will take a look these days. would by nice to get it run on Win10 with Pyton 2.7 (pypy does not run here ...)

I was able to get pypy running on Windows 10 by using the Linux Subsystem for Windows:

https://bitcointalk.org/index.php?topic=18313.msg21025074#msg21025074

However, the reason I made these CPU improvement changes was in the hope that it would allow people to run CPython on medium-speed CPUs without crazy orphan rates. I'm very much interested in hearing whether it works.

This "linux subsystem" is like 5-10x slower than you make a Vitrualbox VM and run Ubuntu in it.
Tested on cross-compiling coin daemons.
It would be way better choice to install Ubuntu desktop in VM, run it in seamless mode and use for whatever you want.

[jtoomim]

This "linux subsystem" is like 5-10x slower than you make a Vitrualbox VM and run Ubuntu in it.

That assertion is inconsistent with most of the benchmarks I have seen, in which WSL is often faster than native Linux and rarely more than 5% slower.
https://www.phoronix.com/scan.php?page=article&item=windows-10-lxcore&num=2

However, for some reason, code compilation appears to be the exception to that rule. For some strange reason, compiling code on WSL is extremely slow:
https://www.phoronix.com/scan.php?page=article&item=windows-10-lxcore&num=4

Since running p2pool is not code compilation, performance on WSL should be quite good. P2pool is mostly just a CPU-bound task with very little disk IO and a small amount of network IO, so the poor results from code compiling should not manifest here.