p3yot33at3r
|
|
September 27, 2015, 02:20:40 PM |
|
Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon
|
|
|
|
Meuh6879
Legendary
Offline
Activity: 1512
Merit: 1012
|
|
September 27, 2015, 03:32:41 PM |
|
Yep ... it's good.
|
|
|
|
jedimstr
|
|
September 27, 2015, 03:56:06 PM |
|
Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon More than likely a big player has us as a backup pool and they just rebooted/restarted their primary pool instance. Definitely looks more like a failover spike to me.
|
|
|
|
jtoomim
|
|
September 27, 2015, 11:49:21 PM |
|
Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks. http://xtnodes.com/bip101_mining_pools.phphttp://74.82.233.205:9334/static/graphs.html?WeekSpeaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
p3yot33at3r
|
|
September 28, 2015, 12:15:43 AM Last edit: September 28, 2015, 12:31:59 AM by p3yot33at3r |
|
Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.
Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.
I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem? What OS are you using for your nodes? I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go
|
|
|
|
jtoomim
|
|
September 28, 2015, 12:55:37 AM Last edit: September 28, 2015, 01:20:37 AM by jtoomim |
|
I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem? What OS are you using for your nodes? I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go Keep in mind that I have 350 TH/s on my node 24/7, and spikes up to 3.0 PH/s. We probably have about two hundred miners (i.e. computers, not owners) that use our node. I think that might have something to do with the CPU load. Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz I think the problem might be related to the use of the poll()-based reactor in twisted. The poll() function checks a bunch of file descriptors (e.g. network ports) to see if any have data available. When you have a lot of http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.htmlHere are the first few lines of my cProfile stats results for my non-XT node (slower CPU), sorted by the total time each function took (not including subroutines): >>> p.print_stats() Sun Sep 27 17:25:26 2015 cprofile-001.log
1042548800 function calls (1022440006 primitive calls) in 9605.998 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function) 1828987 7667.676 0.004 7667.676 0.004 {built-in method poll} 133405 631.073 0.005 701.931 0.005 p2pool/p2p.py:189(update_remote_view_of_my_known_txs) 2025600 55.807 0.000 91.726 0.000 p2pool/util/math.py:64(add_dicts) 6083060/521034 42.351 0.000 204.197 0.000 p2pool/util/pack.py:292(read) 8211943/2317050 41.329 0.000 187.231 0.000 p2pool/util/pack.py:298(write) 50788863 38.770 0.000 45.809 0.000 p2pool/util/pack.py:13(read) 1748230 33.862 0.000 35.874 0.000 p2pool/util/graph.py:14(_shift) 3454845/2954699 31.683 0.000 240.778 0.000 p2pool/util/pack.py:51(_pack) 33374468 31.063 0.000 63.310 0.000 p2pool/util/pack.py:96(read) 39492368 30.537 0.000 42.449 0.000 p2pool/util/pack.py:115(write) 17576 29.261 0.002 36.366 0.002 p2pool/node.py:252(_) 2559639/2225541 27.652 0.000 1004.009 0.000 p2pool/util/variable.py:68(set) 12883 25.897 0.002 298.297 0.023 p2pool/p2p.py:396(handle_remember_tx) 925154/859232 25.618 0.000 191.821 0.000 p2pool/util/pack.py:166(read) 1748230 25.587 0.000 128.783 0.000 p2pool/util/graph.py:46(_add_datum) 7155588 23.545 0.000 32.801 0.000 p2pool/util/pack.py:221(write) 1432678/1382638 22.485 0.000 153.893 0.000 p2pool/util/pack.py:174(write) 4503389/2916784 21.317 0.000 29.991 0.000 p2pool/util/pack.py:270(__eq__) 6426883 21.155 0.000 33.183 0.000 p2pool/util/pack.py:215(read) 515095 20.821 0.000 34.674 0.000 p2pool/util/math.py:186(natural_to_string) 33493018 17.543 0.000 26.334 0.000 p2pool/util/pack.py:259(__getitem__) 101548128/101547801 17.280 0.000 18.491 0.000 {len} 2198269/2198243 16.876 0.000 101.744 0.000 {sorted} 47813834 16.131 0.000 16.131 0.000 {_struct.pack} 941 15.535 0.017 35.789 0.038 p2pool/data.py:119(generate_transaction) 5306039 14.821 0.000 14.821 0.000 {_hashlib.openssl_sha256} 5401 14.798 0.003 14.803 0.003 {method 'items' of 'dict' objects} 44406 14.483 0.000 24.064 0.001 p2pool/bitcoin/sha256.py:17(process) 49754459 14.364 0.000 15.180 0.000 {getattr} 82295913 13.969 0.000 13.969 0.000 {method 'append' of 'list' objects} 7485573 13.270 0.000 13.270 0.000 {binascii.a2b_hex} 34547735 13.172 0.000 13.172 0.000 p2pool/util/math.py:69(<genexpr>) 5850 11.564 0.002 407.450 0.070 p2pool/node.py:246(_) 2712960 10.420 0.000 13.272 0.000 p2pool/data.py:1011(add_share) 7433713 10.328 0.000 16.661 0.000 p2pool/util/forest.py:332(get_chain) 2534168/285416 10.066 0.000 1790.124 0.006 {method 'send' of 'generator' objects} 16053 9.917 0.001 9.917 0.001 /usr/lib/python2.7/json/encoder.py:205(iterencode) 43314052 9.827 0.000 9.827 0.000 {method 'get' of 'dict' objects} 11226 9.816 0.001 9.816 0.001 /usr/lib/python2.7/json/decoder.py:371(raw_decode) 6337858 9.336 0.000 17.570 0.000 p2pool/util/pack.py:188(read) 35144317 9.291 0.000 9.291 0.000 p2pool/util/math.py:63(<lambda>) 2712960 9.236 0.000 11.115 0.000 p2pool/data.py:1022(add_verified_hash) 5103 8.790 0.002 175.572 0.034 p2pool/node.py:23(handle_shares) 25577856 8.662 0.000 8.662 0.000 p2pool/bitcoin/sha256.py:18(rightrotate) 5306038 8.635 0.000 8.635 0.000 {method 'digest' of '_hashlib.HASH' objects} 1748230 8.348 0.000 27.323 0.000 p2pool/util/graph.py:23(_) 185065 8.338 0.000 1097.603 0.006 p2pool/util/p2protocol.py:63(packetReceived) 2236393 8.072 0.000 10.352 0.000 p2pool/util/datachunker.py:18(get) 15805 7.840 0.000 23.109 0.001 p2pool/data.py:338(get_other_tx_hashes) 16472083 7.647 0.000 7.647 0.000 {divmod} 13992152 7.534 0.000 11.242 0.000 p2pool/util/pack.py:261(__setitem__) 1748230 7.523 0.000 45.516 0.000 p2pool/util/graph.py:32(_shift_bins_so_t_is_not_past_end) 2526120 7.308 0.000 12.750 0.000 p2pool/bitcoin/height_tracker.py:106(get_height_rel_highest) 6389979 6.975 0.000 7.048 0.000 {method 'join' of 'str' objects} 138138 6.974 0.000 75.463 0.001 p2pool/util/skiplist.py:12(__call__) 1742141 6.708 0.000 37.944 0.000 p2pool/bitcoin/data.py:10(hash256) 3859910 6.190 0.000 43.954 0.000 p2pool/util/pack.py:313(write) 8056916 5.986 0.000 9.912 0.000 p2pool/util/pack.py:192(write) 157 5.716 0.036 35.242 0.224 p2pool/main.py:199(save_shares) 6809454/4283334 5.614 0.000 29.820 0.000 {max} 6911540 5.597 0.000 5.597 0.000 {binascii.b2a_hex} 969 5.518 0.006 28.824 0.030 p2pool/bitcoin/data.py:180(calculate_merkle_link) 3556045 5.510 0.000 6.936 0.000 p2pool/util/graph.py:24(<lambda>) 1828987 5.438 0.000 9135.571 0.005 /usr/lib/python2.7/dist-packages/twisted/internet/pollreactor.py:147(doPoll) 3454845/2954699 5.414 0.000 245.595 0.000 p2pool/util/pack.py:73(pack) 4553432 5.247 0.000 20.618 0.000 p2pool/util/pack.py:130(read) 17540412 5.109 0.000 5.109 0.000 {method 'iteritems' of 'dict' objects} 95 5.102 0.054 5.102 0.054 {posix.fsync} 2236477/2236471 5.097 0.000 1135.022 0.001 p2pool/util/p2protocol.py:29(dataReceiver) 6729729 4.927 0.000 33.823 0.000 p2pool/util/pack.py:82(packed_size) 33380877 4.815 0.000 4.815 0.000 {ord} 1828987 4.779 0.000 459.169 0.000 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:762(runUntilCurrent) 17987582 4.687 0.000 4.687 0.000 {setattr} 2807093 4.630 0.000 223.412 0.000 p2pool/util/pack.py:41(_unpack) 438995/222789 4.491 0.000 1064.895 0.005 p2pool/util/variable.py:35(happened) 3928877 4.374 0.000 9.756 0.000 p2pool/util/pack.py:134(write) 1 4.369 4.369 9605.102 9605.102 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:1172(mainLoop) 336129 4.218 0.000 37.090 0.000 {method 'update' of 'dict' objects} 11691309 4.045 0.000 4.045 0.000 {isinstance} 3042144 4.038 0.000 5.191 0.000 p2pool/bitcoin/data.py:217(target_to_average_attempts) 5804151 3.976 0.000 23.636 0.000 p2pool/data.py:367(<genexpr>)
That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program. I think I might try a try: from twisted.internet import epollreactor epollreactor.install() except: print "twisted.internet.epollreactor could not be loaded. " print "Falling back to pollreactor. Performance may suffer."
from twisted.internet import reactor
or something like that. It will probably take a little more than that, as I don't know if the APIs are compatible. The biggest performance improvements are usually algorithmic, not language-based. You shouldn't change languages until you are pretty sure that you have the algorithm down. There are some big algorithmic improvements still to be had in p2pool, so I don't think the time is yet ripe for a rewrite in C. (However, there are quite a few modules (mostly in the utils directory, like utils/math.py and utils/pack.py) that would make for straightforward C modules that could be called by python. But more on that later, maybe.) edit: changed the code above. It seems to be working now. I don't know if the performance is improved yet. I'll check back in a few hours. This epollreactor thing will only run on Linux >= 2.6, by the way.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
geneda
Newbie
Offline
Activity: 7
Merit: 0
|
|
September 28, 2015, 10:55:58 AM |
|
This error is normal when you have no data to display somewhere... it was told like 200 times already.
Wow, I can do without the snide remark. If you you are upset, just don't post...geez. Anyway, your "answer" doesn't appear to be valid. I also get those error messages when the miner attempts to connect. Anyone else (besides rav3n) have any advice? Thanks for the time.
|
|
|
|
p3yot33at3r
|
|
September 28, 2015, 11:07:45 AM |
|
Anyway, your "answer" doesn't appear to be valid. I also get those error messages when the miner attempts to connect.
Thanks for the time.
What credentials are you using on your miner?
|
|
|
|
p3yot33at3r
|
|
September 28, 2015, 11:50:49 AM |
|
That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.
I take it you mean milliseconds? Most of what you say goes straight over my head I'm afraid - far beyond my pay grade Have you tried contacting forrestv at all?
|
|
|
|
geneda
Newbie
Offline
Activity: 7
Merit: 0
|
|
September 28, 2015, 12:17:16 PM |
|
Anyway, your "answer" doesn't appear to be valid. I also get those error messages when the miner attempts to connect.
Thanks for the time.
What credentials are you using on your miner? I wasn't sure what to put in there, but I have it running on my local machine with the credentials that are in my bitcoin.conf. All I did was copy bitcoin-qt and the blockchain up to my internet server, start bitcoin and p2pool (changing the rcpip and rpcallowip to the server address) then I just changed the URL in my miner. But it won't connect. thanks again for the time...
|
|
|
|
btcscot
Newbie
Offline
Activity: 30
Merit: 0
|
|
September 28, 2015, 02:40:02 PM |
|
Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help
If you want to connect then its 62.75.188.90:9332
I will be attaching my domain at some point soon
|
|
|
|
p3yot33at3r
|
|
September 28, 2015, 02:42:04 PM |
|
Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help
If you want to connect then its 62.75.188.90:9332
I will be attaching my domain at some point soon
I can see it's working on the IP you posted, well done! Glad you got it up & running - welcome to p2pool!
|
|
|
|
btcscot
Newbie
Offline
Activity: 30
Merit: 0
|
|
September 28, 2015, 02:47:39 PM |
|
Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help
If you want to connect then its 62.75.188.90:9332
I will be attaching my domain at some point soon
I can see it's working on the IP you posted, well done! Glad you got it up & running - welcome to p2pool! Yeah thanks now just need my miner to move onto it
|
|
|
|
Meuh6879
Legendary
Offline
Activity: 1512
Merit: 1012
|
|
September 28, 2015, 05:07:24 PM |
|
After the 5Ph/s peak ... we have waves ... so, P2Pool is the best pool ? Sure and it can deserve more than that ! COME HERE !!!
|
|
|
|
jtoomim
|
|
September 29, 2015, 01:15:23 AM |
|
I noticed on the p2pool scroll every time I hit the the stat page, I get the following errors:
...
2015-09-26 06:34:32.559000 > 2015-09-26 06:34:32.559000 > File "p2pool\web.pyc", line 338, in <lambda> 2015-09-26 06:34:32.559000 > 2015-09-26 06:34:32.559000 > exceptions.TypeError: %x format: a number is requir ed, not NoneType
Here's line 338 of web.py: new_root.putChild('best_share_hash', WebInterface(lambda: '%064x' % node.best_share_var.value)) I think what it's saying is that you don't have any shares yet, so you don't have a best_share_var. You should know that Antminer S5s have some significant problems with p2pool that causes (a) lost hashrate unless you replace the default cgminer binary, and (b) instability, which can result in (c) hardware damage due to the S5 shutting down the fan immediately when cgminer stops running but heat continuing to be generated by the ASICs for several minutes afterward.
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
jtoomim
|
|
September 29, 2015, 01:21:54 AM |
|
That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.
I take it you mean milliseconds? Most of what you say goes straight over my head I'm afraid - far beyond my pay grade Have you tried contacting forrestv at all? No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call. On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading).
|
Hosting bitcoin miners for $65 to $80/kW/month on clean, cheap hydro power. http://Toom.im
|
|
|
idonothave
|
|
September 29, 2015, 08:25:31 AM |
|
thread should be renamed, I would recommend "p2pool: Decentralized, DoS-resistant, Hop-Proof pool"
|
|
|
|
jonnybravo0311
Legendary
Offline
Activity: 1344
Merit: 1024
Mine at Jonny's Pool
|
|
September 29, 2015, 02:25:46 PM |
|
That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.
I take it you mean milliseconds? Most of what you say goes straight over my head I'm afraid - far beyond my pay grade Have you tried contacting forrestv at all? No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call. On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading). You're spot on jtoomin. The fact is that the code is single threaded, so even with a blazing fast CPU, you're going to see huge spikes in CPU load when you've got significant traffic (either because of transaction spamming of the network, or from a large number of connected miners). While not an ideal solution, if you've got a large number of miners on your node, you might want to setup multiple nodes and spread the traffic load across them. Not sure if that's an option for you, though... it's just a suggestion.
|
Jonny's Pool - Mine with us and help us grow! Support a pool that supports Bitcoin, not a hardware manufacturer's pockets! No SPV cheats. No empty blocks.
|
|
|
Meuh6879
Legendary
Offline
Activity: 1512
Merit: 1012
|
|
September 29, 2015, 02:39:44 PM |
|
|
|
|
|
p3yot33at3r
|
|
September 29, 2015, 08:16:53 PM |
|
Is anyone here using the S7 on p2pool? If so, how are they performing? I'm interested to know if they are still using the crazy queue settings with their cgminer fork & if anyone has replaced it with the mainline cgminer yet. If bitmain lower their prices again soon, I might be tempted.....if they work OK with p2pool that is.
|
|
|
|
|