[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[p3yot33at3r]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

[1714018363]

1714018363

[1714018363]

1714018363

[Meuh6879]

  Yep ... it's good.

[jedimstr]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

More than likely a big player has us as a backup pool and they just rebooted/restarted their primary pool instance.  Definitely looks more like a failover spike to me.

[jtoomim]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

http://xtnodes.com/bip101_mining_pools.php

http://74.82.233.205:9334/static/graphs.html?Week

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

[p3yot33at3r]

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  

[jtoomim]

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  

Keep in mind that I have 350 TH/s on my node 24/7, and spikes up to 3.0 PH/s. We probably have about two hundred miners (i.e. computers, not owners) that use our node. I think that might have something to do with the CPU load.

Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz

I think the problem might be related to the use of the poll()-based reactor in twisted. The poll() function checks a bunch of file descriptors (e.g. network ports) to see if any have data available. When you have a lot of

http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.html

Here are the first few lines of my cProfile stats results for my non-XT node (slower CPU), sorted by the total time each function took (not including subroutines):

Code:

>>> p.print_stats()
Sun Sep 27 17:25:26 2015    cprofile-001.log

         1042548800 function calls (1022440006 primitive calls) in 9605.998 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1828987 7667.676    0.004 7667.676    0.004 {built-in method poll}
   133405  631.073    0.005  701.931    0.005 p2pool/p2p.py:189(update_remote_view_of_my_known_txs)
  2025600   55.807    0.000   91.726    0.000 p2pool/util/math.py:64(add_dicts)
6083060/521034   42.351    0.000  204.197    0.000 p2pool/util/pack.py:292(read)
8211943/2317050   41.329    0.000  187.231    0.000 p2pool/util/pack.py:298(write)
 50788863   38.770    0.000   45.809    0.000 p2pool/util/pack.py:13(read)
  1748230   33.862    0.000   35.874    0.000 p2pool/util/graph.py:14(_shift)
3454845/2954699   31.683    0.000  240.778    0.000 p2pool/util/pack.py:51(_pack)
 33374468   31.063    0.000   63.310    0.000 p2pool/util/pack.py:96(read)
 39492368   30.537    0.000   42.449    0.000 p2pool/util/pack.py:115(write)
    17576   29.261    0.002   36.366    0.002 p2pool/node.py:252(_)
2559639/2225541   27.652    0.000 1004.009    0.000 p2pool/util/variable.py:68(set)
    12883   25.897    0.002  298.297    0.023 p2pool/p2p.py:396(handle_remember_tx)
925154/859232   25.618    0.000  191.821    0.000 p2pool/util/pack.py:166(read)
  1748230   25.587    0.000  128.783    0.000 p2pool/util/graph.py:46(_add_datum)
  7155588   23.545    0.000   32.801    0.000 p2pool/util/pack.py:221(write)
1432678/1382638   22.485    0.000  153.893    0.000 p2pool/util/pack.py:174(write)
4503389/2916784   21.317    0.000   29.991    0.000 p2pool/util/pack.py:270(__eq__)
  6426883   21.155    0.000   33.183    0.000 p2pool/util/pack.py:215(read)
   515095   20.821    0.000   34.674    0.000 p2pool/util/math.py:186(natural_to_string)
 33493018   17.543    0.000   26.334    0.000 p2pool/util/pack.py:259(__getitem__)
101548128/101547801   17.280    0.000   18.491    0.000 {len}
2198269/2198243   16.876    0.000  101.744    0.000 {sorted}
 47813834   16.131    0.000   16.131    0.000 {_struct.pack}
      941   15.535    0.017   35.789    0.038 p2pool/data.py:119(generate_transaction)
  5306039   14.821    0.000   14.821    0.000 {_hashlib.openssl_sha256}
     5401   14.798    0.003   14.803    0.003 {method 'items' of 'dict' objects}
    44406   14.483    0.000   24.064    0.001 p2pool/bitcoin/sha256.py:17(process)
 49754459   14.364    0.000   15.180    0.000 {getattr}
 82295913   13.969    0.000   13.969    0.000 {method 'append' of 'list' objects}
  7485573   13.270    0.000   13.270    0.000 {binascii.a2b_hex}
 34547735   13.172    0.000   13.172    0.000 p2pool/util/math.py:69(<genexpr>)
     5850   11.564    0.002  407.450    0.070 p2pool/node.py:246(_)
  2712960   10.420    0.000   13.272    0.000 p2pool/data.py:1011(add_share)
  7433713   10.328    0.000   16.661    0.000 p2pool/util/forest.py:332(get_chain)
2534168/285416   10.066    0.000 1790.124    0.006 {method 'send' of 'generator' objects}
    16053    9.917    0.001    9.917    0.001 /usr/lib/python2.7/json/encoder.py:205(iterencode)
 43314052    9.827    0.000    9.827    0.000 {method 'get' of 'dict' objects}
    11226    9.816    0.001    9.816    0.001 /usr/lib/python2.7/json/decoder.py:371(raw_decode)
  6337858    9.336    0.000   17.570    0.000 p2pool/util/pack.py:188(read)
 35144317    9.291    0.000    9.291    0.000 p2pool/util/math.py:63(<lambda>)
  2712960    9.236    0.000   11.115    0.000 p2pool/data.py:1022(add_verified_hash)
     5103    8.790    0.002  175.572    0.034 p2pool/node.py:23(handle_shares)
 25577856    8.662    0.000    8.662    0.000 p2pool/bitcoin/sha256.py:18(rightrotate)
  5306038    8.635    0.000    8.635    0.000 {method 'digest' of '_hashlib.HASH' objects}
  1748230    8.348    0.000   27.323    0.000 p2pool/util/graph.py:23(_)
   185065    8.338    0.000 1097.603    0.006 p2pool/util/p2protocol.py:63(packetReceived)
  2236393    8.072    0.000   10.352    0.000 p2pool/util/datachunker.py:18(get)
    15805    7.840    0.000   23.109    0.001 p2pool/data.py:338(get_other_tx_hashes)
 16472083    7.647    0.000    7.647    0.000 {divmod}
 13992152    7.534    0.000   11.242    0.000 p2pool/util/pack.py:261(__setitem__)
  1748230    7.523    0.000   45.516    0.000 p2pool/util/graph.py:32(_shift_bins_so_t_is_not_past_end)
  2526120    7.308    0.000   12.750    0.000 p2pool/bitcoin/height_tracker.py:106(get_height_rel_highest)
  6389979    6.975    0.000    7.048    0.000 {method 'join' of 'str' objects}
   138138    6.974    0.000   75.463    0.001 p2pool/util/skiplist.py:12(__call__)
  1742141    6.708    0.000   37.944    0.000 p2pool/bitcoin/data.py:10(hash256)
  3859910    6.190    0.000   43.954    0.000 p2pool/util/pack.py:313(write)
  8056916    5.986    0.000    9.912    0.000 p2pool/util/pack.py:192(write)
      157    5.716    0.036   35.242    0.224 p2pool/main.py:199(save_shares)
6809454/4283334    5.614    0.000   29.820    0.000 {max}
  6911540    5.597    0.000    5.597    0.000 {binascii.b2a_hex}
      969    5.518    0.006   28.824    0.030 p2pool/bitcoin/data.py:180(calculate_merkle_link)
  3556045    5.510    0.000    6.936    0.000 p2pool/util/graph.py:24(<lambda>)
  1828987    5.438    0.000 9135.571    0.005 /usr/lib/python2.7/dist-packages/twisted/internet/pollreactor.py:147(doPoll)
3454845/2954699    5.414    0.000  245.595    0.000 p2pool/util/pack.py:73(pack)
  4553432    5.247    0.000   20.618    0.000 p2pool/util/pack.py:130(read)
 17540412    5.109    0.000    5.109    0.000 {method 'iteritems' of 'dict' objects}
       95    5.102    0.054    5.102    0.054 {posix.fsync}
2236477/2236471    5.097    0.000 1135.022    0.001 p2pool/util/p2protocol.py:29(dataReceiver)
  6729729    4.927    0.000   33.823    0.000 p2pool/util/pack.py:82(packed_size)
 33380877    4.815    0.000    4.815    0.000 {ord}
  1828987    4.779    0.000  459.169    0.000 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:762(runUntilCurrent)
 17987582    4.687    0.000    4.687    0.000 {setattr}
  2807093    4.630    0.000  223.412    0.000 p2pool/util/pack.py:41(_unpack)
438995/222789    4.491    0.000 1064.895    0.005 p2pool/util/variable.py:35(happened)
  3928877    4.374    0.000    9.756    0.000 p2pool/util/pack.py:134(write)
        1    4.369    4.369 9605.102 9605.102 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:1172(mainLoop)
   336129    4.218    0.000   37.090    0.000 {method 'update' of 'dict' objects}
 11691309    4.045    0.000    4.045    0.000 {isinstance}
  3042144    4.038    0.000    5.191    0.000 p2pool/bitcoin/data.py:217(target_to_average_attempts)
  5804151    3.976    0.000   23.636    0.000 p2pool/data.py:367(<genexpr>)

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I think I might try a

Code:

try:
    from twisted.internet import epollreactor
    epollreactor.install()
except:
    print "twisted.internet.epollreactor could not be loaded. "
    print "Falling back to pollreactor. Performance may suffer."

from twisted.internet import reactor

or something like that. It will probably take a little more than that, as I don't know if the APIs are compatible.

The biggest performance improvements are usually algorithmic, not language-based. You shouldn't change languages until you are pretty sure that you have the algorithm down. There are some big algorithmic improvements still to be had in p2pool, so I don't think the time is yet ripe for a rewrite in C.

(However, there are quite a few modules (mostly in the utils directory, like utils/math.py and utils/pack.py) that would make for straightforward C modules that could be called by python. But more on that later, maybe.)

edit: changed the code above. It seems to be working now. I don't know if the performance is improved yet. I'll check back in a few hours. This epollreactor thing will only run on Linux >= 2.6, by the way.

[geneda]

This error is normal when you have no data to display somewhere... it was told like 200 times already.

Wow, I can do without the snide remark.  If you you are upset, just don't post...geez.

Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Anyone else (besides rav3n) have any advice?

Thanks for the time.

[p3yot33at3r]

Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Thanks for the time.

What credentials are you using on your miner?

[p3yot33at3r]

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Have you tried contacting forrestv at all?

[geneda]

Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Thanks for the time.

What credentials are you using on your miner?

I wasn't sure what to put in there, but I have it running on my local machine with the credentials that are in my bitcoin.conf.
All I did was copy bitcoin-qt and the blockchain up to my internet server, start bitcoin and p2pool (changing the rcpip and rpcallowip to the server address) then I just changed the
URL in my miner.  But it won't connect.

thanks again for the time...

[btcscot]

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon

[p3yot33at3r]

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon

I can see it's working on the IP you posted, well done!

Glad you got it up & running - welcome to p2pool!  

[btcscot]

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon

I can see it's working on the IP you posted, well done!

Glad you got it up & running - welcome to p2pool!  

Yeah thanks now just need my miner to move onto it

[Meuh6879]

After the 5Ph/s peak ... we have waves ...   so, P2Pool is the best pool ?  

Sure and it can deserve more than that !

COME HERE !!!

[jtoomim]

I noticed on the p2pool scroll every time I hit the the stat page, I get the following errors:

...

2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "p2pool\web.pyc", line 338, in <lambda>
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 > exceptions.TypeError: %x format: a number is requir
ed, not NoneType

Here's line 338 of web.py:


    new_root.putChild('best_share_hash', WebInterface(lambda: '%064x' % node.best_share_var.value))

I think what it's saying is that you don't have any shares yet, so you don't have a best_share_var.

You should know that Antminer S5s have some significant problems with p2pool that causes (a) lost hashrate unless you replace the default cgminer binary, and (b) instability, which can result in (c) hardware damage due to the S5 shutting down the fan immediately when cgminer stops running but heat continuing to be generated by the ASICs for several minutes afterward.

[jtoomim]

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Have you tried contacting forrestv at all?

No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call.

On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading).

[idonothave]

thread should be renamed, I would recommend "p2pool: Decentralized, DoS-resistant, Hop-Proof pool"

[jonnybravo0311]

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Have you tried contacting forrestv at all?

No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call.

On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading).

You're spot on jtoomin.  The fact is that the code is single threaded, so even with a blazing fast CPU, you're going to see huge spikes in CPU load when you've got significant traffic (either because of transaction spamming of the network, or from a large number of connected miners).  While not an ideal solution, if you've got a large number of miners on your node, you might want to setup multiple nodes and spread the traffic load across them.  Not sure if that's an option for you, though... it's just a suggestion.

[Meuh6879]

[p3yot33at3r]

Is anyone here using the S7 on p2pool? If so, how are they performing? I'm interested to know if they are still using the crazy queue settings with their cgminer fork & if anyone has replaced it with the mainline cgminer yet. If bitmain lower their prices again soon, I might be tempted.....if they work OK with p2pool that is.