[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[jtoomim]

jtoomimnet found another block two days ago.

https://blockchain.info/block/00000000000000000096cef4bfd159307030923b0111acd39bfb6561b277a375

1014 kB, 3998 kWU, 13.858 BTC.

We also added a few hundred TH/s yesterday, and are now up to 2.2 PH/s.

[Xantus]

A 2.0 GHz A10 is not what i would call a medium-speed CPU, unfortunately. Sure, it might have 4 cores, but Python is effectively single-threaded, so that doesn't help at all. Pre-Ryzen AMD chips have poor single-threaded performance, so a 2 GHz A10 is equivalent in performance to a roughly 1.3 GHz Core i3 (if such a thing existed). I wish I could get p2pool to run nicely on such a CPU, but I'm afraid that's out of reach at the moment.

You've got plenty of RAM, so I think you can get it to work decently if you go through the steps I described to get Pypy running on Windows.

Also, if you could run p2pool with the --bench option, that would make the log file much more useful to me.

hey there,
here is the log file from p2pool on phyton2.7 on windows 10 with --bench option:
www.c-xantus.de/log2
by the way, there is a bug on the p2pool mining difficult. it dont will be set down if smaler miner are working, or it takes days !

i also tried to install these Linux box and followed the intall steps in the ubuntu bash box to test it.
installation runs fine without anny errors. i copied the bitcoin.conf file to /home/user/.bitcoin/bitcoin.conf
and was able to start P2Pool with --bench option on pypy. But seams that twisted dont work ? - i get no connetion to other nodes.
alwas show the twistet errors :

Code:

2017-09-20 22:26:57.168361 Peer 5.9.143.40:9335 says protocol version is 3301, client version 16.0-139-g49e1a94
2017-09-20 22:26:57.168361 Outgoing connection to peer 5.9.143.40:9335 established. p2pool version: 3301 '16.0-139-g49e1a94'
2017-09-20 22:27:18.544607 P2Pool: 0 shares in chain (0 verified/0 total) Peers: 1 (0 incoming)
2017-09-20 22:27:18.544607  Local: 0H/s in last 0.0 seconds Local dead on arrival: ??? Expected time to share: ???
2017-09-20 22:27:18.544607    0.000 ms for update_remote_view_of_my_known_txs
in handle_share_hashes:
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError: in GenericDeferrer

2017-09-20 22:27:18.560231 Lost peer 5.9.143.40:9335 - Connection was aborted locally, using.
2017-09-20 22:27:41.083009    0.000 ms for update_remote_view_of_my_known_txs
2017-09-20 22:28:02.193542 P2Pool: 0 shares in chain (0 verified/0 total) Peers: 0 (0 incoming)
2017-09-20 22:28:02.193542  Local: 0H/s in last 0.0 seconds Local dead on arrival: ??? Expected time to share: ???
2017-09-20 22:28:02.193542    0.000 ms for update_remote_view_of_my_known_txs
Error getting work from bitcoind:
Traceback (most recent call last):
Failure: twisted.internet.defer.TimeoutError: Getting http://127.0.0.1:8332/ took longer than 30 seconds.

[jtoomim]

Interesting, it seems that time.time() on Windows only has 15 or 16 ms resolution, so all of the short function calls are reported as taking either 0 ms, 15 ms, or 16 ms, even though they're actually probably taking about 4 ms each.

I also see a few lines like this one:

Code:

2017-09-20 07:00:17.936000  453.000 ms for 0 txs in p2p.py:handle_remember_tx (453.000 ms/tx)

453 ms for handle_remember_tx when no transactions are being added seems strange and significant. I'll have to reread the code and see if I can figure out what could be taking so long.

There's also this:

Code:

2017-09-20 07:00:37.877000  750.000 ms for work.py:get_work()
2017-09-20 07:00:37.877000 1734.000 ms for 1 shares in handle_shares (1734.000 ms/share)

That makes about 1.7 seconds of latency just in those two functions for your node switching work. (IIRC, handle_shares() calls get_work, so the 750 ms is already included in the 1734 ms.) There may be a few other functions that I didn't add the benchmarking code to that also play a role, but that latency alone should be responsible for about 5.8% of your DOA+orphan rates.

Code:

2017-09-20 07:02:24.241000 Decoding transactions took 31 ms, Unpacking 1188 ms, hashing 16 ms

The 1188 ms (unpacking) is something I think I know how to fix via a CPU/RAM tradeoff. That will probably be my next task. However, that will only reduce the CPU usage in processing new getblocktemplate responses, which I think is not in the critical code path for share propagation, so I don't expect it to help with DOA+orphan rates much. It's a pretty easy and obvious optimization, though. The node I do my testing on (4.4 GHz Core i7, pypy) only shows about 40 ms for unpacking instead of 1188 ms, so I wasn't sure it would be worthwhile. But if it's stalling your CPU for over a second, that sounds like it could be interfering with things.

Not sure what's the issue with pypy on your machine. Does your CPU show activity when you run it? How much?

[Xantus]

Not sure what's the issue with pypy on your machine. Does your CPU show activity when you run it? How much?

pypy cpu usage is lower than 5%, but it seams as anything is verry slow. it look like p2pool on pypy lags. and also get now connection.
but i did now "sudo apt-get update and "upgrade" and gave pypy another changse this day. loocks better for this moment.
But i have to go to work now and look this evening again does pypy work or not. - i will edit this post ....

thanks agein so far jtoomin

[jtoomim]

pypy cpu usage is lower than 5%, but it seams as anything is verry slow.

That sounds like you might be out of RAM and have swapping to disk. Can you check? Task manager -> Performance -> Memory. In Use, Available, Committed, Paged Pool.

[Xantus]

pypy cpu usage is lower than 5%, but it seams as anything is verry slow.

That sounds like you might be out of RAM and have swapping to disk. Can you check? Task manager -> Performance -> Memory. In Use, Available, Committed, Paged Pool.

no, system look fine, Memory is 50%free, CPU usage is now with pypy somethink around 15% (5%CPU for pypy avergage, and 20%peek)
but other strange thing happend, since i Wrote this morning, i esay let it run and go to work. now i am here again, take a look and p2pool is a bit online 
i cant connect the web interface from an other computer, and the miner cant connect to p2pool node. also i got no p2pool incomming connection. but it got 6 outgoing connection.
does this linux box have an own software firewall ? - i think that is an network problem ...


Log file from p2pool on pypy in an ubuntu bash box on windows 10 :
www.c-xantus.de/log3

System Preformance (using ubuntu bash and p2pool on pypy):
http://www.c-xantus.de/Zwischenablage0.jpg

[jtoomim]

Xantus, maybe what's happening is that there just aren't any exceptions in Windows Firewall for this. Native Windows apps usually trigger a request to open ports in the firewall when needed, but I maybe WSL pypy p2pool isn't doing that. Try disabling Windows Firewall for an hour to see if that helps. If it does, then you can manually add a firewall rule to allow incoming tcp connections on port 9332 and 9333 (or whatever you're using).

Edit: Pages like this suggest that you do indeed need to open up ports on the Windows Firewall if you want to allow WSL processes to serve as servers.

[Xantus]

well, that is strange,
i tested that. its true, i get more connection ...

but i also see ofen this :
"Warning: LOST CONTACT WITH BITCOIND for" is displayed ... perhaps these linux bash box dont runs cleanly.
it fells like it freeze every few minutes:

and after restart now in bench mode (i also put in new tcp rule for in- and outgoing tcp connection on port 9332 and 9333) :
without miner connected : www.c-xantus.de/log4
with miner connected : www.c-xantus.de/log5

by the way, P2Pool is using at linux bash with pypy 1,3Gb Memory an betwen 0 and 15% cpu

[jtoomim]

"Warning: LOST CONTACT WITH BITCOIND for" is displayed ... perhaps these linux bash box dont runs cleanly.

That's what happens when your CPU gets overloaded and stalled. Let's say you have enough incoming traffic in your TCP buffers to keep p2pool's CPU busy for 1.1 minutes. P2pool first sees that it has a queued need to get a new block template, so p2pool sends a getblocktemplate request to bitcoind. Bitcoind may respond immediately, but the response goes to the end of that 1.1 minute queue. P2pool then works through its queue of stuff, then gets to the bitcoind request, and process it, but notices that it took 1.1 minutes and so complains about it.

Same thing for web UI requests. When you point your browser to localhost:9332, that request goes into p2pool's queue of jobs to work on, and p2pool doesn't get around to it until e.g. 1.1 minutes has passed.

by the way, P2Pool is using at linux bash with pypy 1,3Gb Memory an betwen 0 and 15% cpu

The 15% CPU is the problem. 15% is above the 12.5% threshold for fully utilizing a single core on your machine, which means that the work in p2pool's to-do queue is snowballing.

The --bench output is somewhat helpful, but it seems that the bulk of the load is happening in a function that I didn't put benchmarking code into. In your log files, I frequently see line pairs like this one (note the timestamps):

Code:

2017-09-24 21:52:37.814726    0.000 ms for 7 txs in p2p.py:handle_remember_tx (0.000 ms/tx)
2017-09-24 21:52:58.864085    0.000 ms for 11 txs in handle_losing_tx (0.000 ms/tx)
...
2017-09-24 21:52:59.239189 > >>> Warning: LOST CONTACT WITH BITCOIND for 1.6 minutes! Check that it isn't frozen or dead!

The "Lost contact" message means that your CPU was stalled for 1.6 minutes. 1.6 minutes before the "Lost contact" message at 12:52:59 would be 12:51:20 or something like that. That means that your CPU was busy between 21:52:37 and 21:52:58. However, we don't know what was happening at that time, since the code that was running was something that I didn't add benchmarking code to.

Can you add this to your p2pool startup command line and send me the resulting profile1.log file? Try to run it for about an hour. If you run it for too much less than that, and the profile data will be dominated by the start-up share loading time.

Code:

pypy -m cProfile -o "profile1.log" run_p2pool.py [other options]

If anyone is curious, you can analyze those cProfile log files with a python script that looks like this:

Code:

import pstats, sys

if not len(sys.argv) > 1:
   print "Usage: cumtime.py input_file [-t] [lines]"
   sys.exit()

p = pstats.Stats(sys.argv[1])
if '-t' in sys.argv:
    p.sort_stats('tottime')
else:
    p.sort_stats('cumtime')

lines = 100
if len(sys.argv) > 2:
    try:
        lines = int(sys.argv[2])
    except:
        pass

p.print_stats(lines)

[davidadrake]

I haven't been on p2pool-bitcointalk for a while, but I remember reading through the posts every day, and jtoomin, I believe had the version of p2pool to run. On p2pool.org, it shows it has been 83 days since the last block, so are they not running jtoomin's fork?

Is there a list somewhere of pools that are running the fork? Sorry if this has been answered somewhere, but I looked through a bunch of pages without finding any info.

Thanks,

Dave

[Xantus]

That's what happens when your CPU gets overloaded and stalled. Let's say you have enough incoming traffic in your TCP buffers to keep p2pool's CPU busy for 1.1 minutes. P2pool first sees that it has a queued need to get a new block template, so p2pool sends a getblocktemplate request to bitcoind. Bitcoind may respond immediately, but the response goes to the end of that 1.1 minute queue. P2pool then works through its queue of stuff, then gets to the bitcoind request, and process it, but notices that it took 1.1 minutes and so complains about it.

Same thing for web UI requests. When you point your browser to localhost:9332, that request goes into p2pool's queue of jobs to work on, and p2pool doesn't get around to it until e.g. 1.1 minutes has passed.

by the way, P2Pool is using at linux bash with pypy 1,3Gb Memory an betwen 0 and 15% cpu

The 15% CPU is the problem. 15% is above the 12.5% threshold for fully utilizing a single core on your machine, which means that the work in p2pool's to-do queue is snowballing.

Hey big Programmer jtoomim,

i dont belive in these CPU overload thinks you talking about. first time you sayed quad core ist with 25% overloaded, now it is 12,5% an i use it with python2,7 an see load peaks of this singel task by 48% without any connection lost message. there is an other big problem a displayed but no one is intereted.

this linux think hangs and freeze for a few sec everey round about 30sec. and the tcp-connections in these linux box in an problem, perhaps because timeouts between linux box and system network interface.

i run now p2pool on phyton 2.7 an waste a day after start up until it reduce his mining difficult and hoping ther will be a good running windows version with Segwit in the future.

@davidadrake
i have no list, but for now these is a node : "212.43.81.114:9332"
p2pool.org i the "non segwit" version, jtoomim goes his own "segwit way" for now. - korregt me if i talking wrong things!

[Cryptonomist]

I haven't been on p2pool-bitcointalk for a while, but I remember reading through the posts every day, and jtoomin, I believe had the version of p2pool to run. On p2pool.org, it shows it has been 83 days since the last block, so are they not running jtoomin's fork?

Is there a list somewhere of pools that are running the fork? Sorry if this has been answered somewhere, but I looked through a bunch of pages without finding any info.

Thanks,

Dave

Hello,

There are a few websites that give the results from a node scanner. For example http://p2pool.jir.dk/bitcoin/ gives publicly reachable nodes in the P2pool main network and in the jtoomim network. Nodes with a version number 17 or 16 are nodes of the main network. Nodes with a version number of 15.0-33 are nodes from the jtoomim network.
Another useful site is https://p2pool.coinpool.pw/. This site also gives the publicly reachable nodes for both networks.
P2pool.org has also a node scanner, but this one only shows results for the p2pool main network. All the information shown on the p2pool.org website is currently that of the p2pool main network and not of the jtoomim network.
If you want to know the ip address of nodes that are not publicly reachable, the only way to do that is run your own node and look to which nodes it makes a connection.

I hope this answers your question.

[BitTeo]

pypy cpu usage is lower than 5%, but it seams as anything is verry slow.

That sounds like you might be out of RAM and have swapping to disk. Can you check? Task manager -> Performance -> Memory. In Use, Available, Committed, Paged Pool.

How many set virtual memory with 16gb of ram in windows 10 ? .

[jtoomim]

On p2pool.org, it shows it has been 83 days since the last block, so are they not running jtoomin's fork?

Correct. The most recent block found on jtoomimnet was found on September 18th. Jtoomimnet currently has around 2.6 PH/s and an expected time per block of around 22 days. Mainnet has around 0.7 PH/s and an expected time per block of around 90 days.

[belcher]

Has anybody thought about in detail how adding the uncles scheme to p2pool would work?

It was mentioned in the original bitcointalk thread about uncles in 2013 here and here. And also by jtoomim a few months ago here

But I haven't been able to find more details (or even mentions) than those. I've got a couple of ideas but it's good to read everything that's been written about it beforehand.


If uncles was applied to bitcoin then special care would have to be taken to make sure a transaction in an uncle doesn't conflict with a transaction in the main chain. This doesn't apply to p2pool's sharechain because the shares can't conflict with each other in the same way, each share just gets added to the total work that the hasher did. This is quite a cool property of applying uncles to p2pool.

If uncles are added then I think the share interval could be reduced even more to (say) 15 seconds. There'd be no more orphans at all. This would reduce the difficulty (by a constant factor but still) which would mean if p2pool gets really big then small hashers can still solve shares fairly often.

Another problem with the existing algorithm for choosing which share to base work off of is that all shares are treated the same, regardless of their difficulty, which makes low-diff shares have an advantage for selfish mining attacks. Large-diff shares contribute more work to p2pool, so they should have a greater chance of winning an orphan race, ceteris paribus. But you can't just pick the highest-diff share of all shares in an orphan race, since that would cause a race-to-the-top scenario where the optimal difficulty to use for your shares is the highest diff that a competitor uses plus one. That would also allow a selfish miner to reliably win orphan races against siblings: instead of working on a child share of the best known share, you could just orphan it by creating a sibling with +1 difficulty. So it needs to be probabilistic.

Another way of solving this apart from being probabilistic is to limit the difficulty that hashers can mine on. Only allow p2pool hashers to mine 1x difficulty, 10x difficulty, 100x difficulty and 1000x difficulty. Then it becomes impossible for a bad hasher to just mine difficulty + 1, they can only mine difficulty x 10.

Of course, with the scheme of adding uncles to p2pool there totally stop being orphans so these kind of difficulty games don't lead to extra profit.

[Polyatomic]

If uncles are added then I think the share interval could be reduced even more to (say) 15 seconds.

For what its worth, average time between blocks on the p2pool share chain was 10 seconds once upon a time.

[belcher]

If uncles are added then I think the share interval could be reduced even more to (say) 15 seconds.

For what its worth, average time between blocks on the p2pool share chain was 10 seconds once upon a time.

I suppose it was increased because of orphans and DOA shares. If so then using uncles should allow share intervals to go lower because there wouldn't be a race to publish shares anymore.

[Xantus]

How many set virtual memory with 16gb of ram in windows 10 ? .

3,6Gb memory free, 450Mb virtual Memory (see picture a few posts up)

[jtoomim]

If uncles was applied to bitcoin then special care would have to be taken to make sure a transaction in an uncle doesn't conflict with a transaction in the main chain.

Actually, you'd simply ignore all transactions in uncles except the coinbase transaction, which gets programmatically modified to reduce the amounts.

If uncles are added then I think the share interval could be reduced even more to (say) 15 seconds.

I'd really rather not, for CPU and RAM performance reasons in addition to orphan rate reasons. It takes a medium-slow CPU on Python2.7 up to 4 seconds to process a share and issue new work with 1 MB blocks. With 4 MB blocks, you would generally be unable to run p2pool on medium-slow CPUs with a 15 second share interval, as it would take 16 seconds to process each share. Share variance is currently insignificant compared to block variance on p2pool. Increasing the share interval to 60 seconds on average is more likely to be a good idea than decreasing it.

While there are a few major changes to p2pool's architecture that would drastically reduce p2pool's performance load, those changes are quite large and may never be done. It would certainly involve more code than just adding uncles.

There'd be no more orphans at all. This would reduce the difficulty (by a constant factor but still) which would mean if p2pool gets really big then small hashers can still solve shares fairly often.

There would be uncles instead. Uncles have a smaller reward for the person who mined them than normal shares, so there's still a potential fairness penalty to having high uncle rates. The game theory of uncles is complicated, and if you don't set the rules right, it can be in a miner's best interest to not mention other people's shares as uncles and orphan them instead. Giving the uncled shares a hit in revenue is necessary to making sure there's an incentive to include them.

Another way of solving this apart from being probabilistic...

A probably better solution is to just follow the share with the lowest hash, regardless of its difficulty. This inherently prefers high-diff shares, but is fair in that low-diff shares have a chance to win against a high-diff share if the low-diff share is particularly lucky in its hash. Basically, it is effectively equivalent to the algorithm I proposed in the post that you linked to, but where the source of randomness is the hashes themselves.

[davidadrake]

Are the blocks that are mined on jtoomin's fork empty, like they were on the main p2pool?

I haven't been on p2pool since around Christmas I think, but I remember them being mostly empty. I think when I was on, a few people were testing jtoomin's fork, but I thought that it was supposed to get merged with the main p2pool. Is that not the case anymore?

I found it hard to find, as there's no info about his fork anywhere. It's not really accessible, you have to know exactly what you're looking for - I had to ask on here. Maybe at least something on the first page, or are you trying to keep it small - I thought it was nice when the main p2pool was around 10PH.

It would be really cool if there was something like p2pool.org if it's not getting merged.

just some of my thoughts.