Bitcoin Forum
December 07, 2016, 09:01:39 AM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 ... 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 [671] 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 ... 744 »
  Print  
Author Topic: [1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool  (Read 2032336 times)
jedimstr
Hero Member
*****
Offline Offline

Activity: 784



View Profile
September 27, 2015, 03:56:06 PM
 #13401

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon  Grin

More than likely a big player has us as a backup pool and they just rebooted/restarted their primary pool instance.  Definitely looks more like a failover spike to me.

1481101299
Hero Member
*
Offline Offline

Posts: 1481101299

View Profile Personal Message (Offline)

Ignore
1481101299
Reply with quote  #2

1481101299
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481101299
Hero Member
*
Offline Offline

Posts: 1481101299

View Profile Personal Message (Offline)

Ignore
1481101299
Reply with quote  #2

1481101299
Report to moderator
jtoomim
Hero Member
*****
Offline Offline

Activity: 555


View Profile WWW
September 27, 2015, 11:49:21 PM
 #13402

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon  Grin

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

http://xtnodes.com/bip101_mining_pools.php

http://74.82.233.205:9334/static/graphs.html?Week

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

Hosting bitcoin miners for $75 to $90/kW/month on clean, cheap hydro power.
http://Toom.im
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 28, 2015, 12:15:43 AM
 #13403

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  Cheesy

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  Wink
jtoomim
Hero Member
*****
Offline Offline

Activity: 555


View Profile WWW
September 28, 2015, 12:55:37 AM
 #13404

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  Cheesy

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  Wink

Keep in mind that I have 350 TH/s on my node 24/7, and spikes up to 3.0 PH/s. We probably have about two hundred miners (i.e. computers, not owners) that use our node. I think that might have something to do with the CPU load.

Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz

I think the problem might be related to the use of the poll()-based reactor in twisted. The poll() function checks a bunch of file descriptors (e.g. network ports) to see if any have data available. When you have a lot of

http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.html

Here are the first few lines of my cProfile stats results for my non-XT node (slower CPU), sorted by the total time each function took (not including subroutines):

Code:
>>> p.print_stats()
Sun Sep 27 17:25:26 2015    cprofile-001.log

         1042548800 function calls (1022440006 primitive calls) in 9605.998 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1828987 7667.676    0.004 7667.676    0.004 {built-in method poll}
   133405  631.073    0.005  701.931    0.005 p2pool/p2p.py:189(update_remote_view_of_my_known_txs)
  2025600   55.807    0.000   91.726    0.000 p2pool/util/math.py:64(add_dicts)
6083060/521034   42.351    0.000  204.197    0.000 p2pool/util/pack.py:292(read)
8211943/2317050   41.329    0.000  187.231    0.000 p2pool/util/pack.py:298(write)
 50788863   38.770    0.000   45.809    0.000 p2pool/util/pack.py:13(read)
  1748230   33.862    0.000   35.874    0.000 p2pool/util/graph.py:14(_shift)
3454845/2954699   31.683    0.000  240.778    0.000 p2pool/util/pack.py:51(_pack)
 33374468   31.063    0.000   63.310    0.000 p2pool/util/pack.py:96(read)
 39492368   30.537    0.000   42.449    0.000 p2pool/util/pack.py:115(write)
    17576   29.261    0.002   36.366    0.002 p2pool/node.py:252(_)
2559639/2225541   27.652    0.000 1004.009    0.000 p2pool/util/variable.py:68(set)
    12883   25.897    0.002  298.297    0.023 p2pool/p2p.py:396(handle_remember_tx)
925154/859232   25.618    0.000  191.821    0.000 p2pool/util/pack.py:166(read)
  1748230   25.587    0.000  128.783    0.000 p2pool/util/graph.py:46(_add_datum)
  7155588   23.545    0.000   32.801    0.000 p2pool/util/pack.py:221(write)
1432678/1382638   22.485    0.000  153.893    0.000 p2pool/util/pack.py:174(write)
4503389/2916784   21.317    0.000   29.991    0.000 p2pool/util/pack.py:270(__eq__)
  6426883   21.155    0.000   33.183    0.000 p2pool/util/pack.py:215(read)
   515095   20.821    0.000   34.674    0.000 p2pool/util/math.py:186(natural_to_string)
 33493018   17.543    0.000   26.334    0.000 p2pool/util/pack.py:259(__getitem__)
101548128/101547801   17.280    0.000   18.491    0.000 {len}
2198269/2198243   16.876    0.000  101.744    0.000 {sorted}
 47813834   16.131    0.000   16.131    0.000 {_struct.pack}
      941   15.535    0.017   35.789    0.038 p2pool/data.py:119(generate_transaction)
  5306039   14.821    0.000   14.821    0.000 {_hashlib.openssl_sha256}
     5401   14.798    0.003   14.803    0.003 {method 'items' of 'dict' objects}
    44406   14.483    0.000   24.064    0.001 p2pool/bitcoin/sha256.py:17(process)
 49754459   14.364    0.000   15.180    0.000 {getattr}
 82295913   13.969    0.000   13.969    0.000 {method 'append' of 'list' objects}
  7485573   13.270    0.000   13.270    0.000 {binascii.a2b_hex}
 34547735   13.172    0.000   13.172    0.000 p2pool/util/math.py:69(<genexpr>)
     5850   11.564    0.002  407.450    0.070 p2pool/node.py:246(_)
  2712960   10.420    0.000   13.272    0.000 p2pool/data.py:1011(add_share)
  7433713   10.328    0.000   16.661    0.000 p2pool/util/forest.py:332(get_chain)
2534168/285416   10.066    0.000 1790.124    0.006 {method 'send' of 'generator' objects}
    16053    9.917    0.001    9.917    0.001 /usr/lib/python2.7/json/encoder.py:205(iterencode)
 43314052    9.827    0.000    9.827    0.000 {method 'get' of 'dict' objects}
    11226    9.816    0.001    9.816    0.001 /usr/lib/python2.7/json/decoder.py:371(raw_decode)
  6337858    9.336    0.000   17.570    0.000 p2pool/util/pack.py:188(read)
 35144317    9.291    0.000    9.291    0.000 p2pool/util/math.py:63(<lambda>)
  2712960    9.236    0.000   11.115    0.000 p2pool/data.py:1022(add_verified_hash)
     5103    8.790    0.002  175.572    0.034 p2pool/node.py:23(handle_shares)
 25577856    8.662    0.000    8.662    0.000 p2pool/bitcoin/sha256.py:18(rightrotate)
  5306038    8.635    0.000    8.635    0.000 {method 'digest' of '_hashlib.HASH' objects}
  1748230    8.348    0.000   27.323    0.000 p2pool/util/graph.py:23(_)
   185065    8.338    0.000 1097.603    0.006 p2pool/util/p2protocol.py:63(packetReceived)
  2236393    8.072    0.000   10.352    0.000 p2pool/util/datachunker.py:18(get)
    15805    7.840    0.000   23.109    0.001 p2pool/data.py:338(get_other_tx_hashes)
 16472083    7.647    0.000    7.647    0.000 {divmod}
 13992152    7.534    0.000   11.242    0.000 p2pool/util/pack.py:261(__setitem__)
  1748230    7.523    0.000   45.516    0.000 p2pool/util/graph.py:32(_shift_bins_so_t_is_not_past_end)
  2526120    7.308    0.000   12.750    0.000 p2pool/bitcoin/height_tracker.py:106(get_height_rel_highest)
  6389979    6.975    0.000    7.048    0.000 {method 'join' of 'str' objects}
   138138    6.974    0.000   75.463    0.001 p2pool/util/skiplist.py:12(__call__)
  1742141    6.708    0.000   37.944    0.000 p2pool/bitcoin/data.py:10(hash256)
  3859910    6.190    0.000   43.954    0.000 p2pool/util/pack.py:313(write)
  8056916    5.986    0.000    9.912    0.000 p2pool/util/pack.py:192(write)
      157    5.716    0.036   35.242    0.224 p2pool/main.py:199(save_shares)
6809454/4283334    5.614    0.000   29.820    0.000 {max}
  6911540    5.597    0.000    5.597    0.000 {binascii.b2a_hex}
      969    5.518    0.006   28.824    0.030 p2pool/bitcoin/data.py:180(calculate_merkle_link)
  3556045    5.510    0.000    6.936    0.000 p2pool/util/graph.py:24(<lambda>)
  1828987    5.438    0.000 9135.571    0.005 /usr/lib/python2.7/dist-packages/twisted/internet/pollreactor.py:147(doPoll)
3454845/2954699    5.414    0.000  245.595    0.000 p2pool/util/pack.py:73(pack)
  4553432    5.247    0.000   20.618    0.000 p2pool/util/pack.py:130(read)
 17540412    5.109    0.000    5.109    0.000 {method 'iteritems' of 'dict' objects}
       95    5.102    0.054    5.102    0.054 {posix.fsync}
2236477/2236471    5.097    0.000 1135.022    0.001 p2pool/util/p2protocol.py:29(dataReceiver)
  6729729    4.927    0.000   33.823    0.000 p2pool/util/pack.py:82(packed_size)
 33380877    4.815    0.000    4.815    0.000 {ord}
  1828987    4.779    0.000  459.169    0.000 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:762(runUntilCurrent)
 17987582    4.687    0.000    4.687    0.000 {setattr}
  2807093    4.630    0.000  223.412    0.000 p2pool/util/pack.py:41(_unpack)
438995/222789    4.491    0.000 1064.895    0.005 p2pool/util/variable.py:35(happened)
  3928877    4.374    0.000    9.756    0.000 p2pool/util/pack.py:134(write)
        1    4.369    4.369 9605.102 9605.102 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:1172(mainLoop)
   336129    4.218    0.000   37.090    0.000 {method 'update' of 'dict' objects}
 11691309    4.045    0.000    4.045    0.000 {isinstance}
  3042144    4.038    0.000    5.191    0.000 p2pool/bitcoin/data.py:217(target_to_average_attempts)
  5804151    3.976    0.000   23.636    0.000 p2pool/data.py:367(<genexpr>)

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I think I might try a
Code:
try:
    from twisted.internet import epollreactor
    epollreactor.install()
except:
    print "twisted.internet.epollreactor could not be loaded. "
    print "Falling back to pollreactor. Performance may suffer."

from twisted.internet import reactor
or something like that. It will probably take a little more than that, as I don't know if the APIs are compatible.

The biggest performance improvements are usually algorithmic, not language-based. You shouldn't change languages until you are pretty sure that you have the algorithm down. There are some big algorithmic improvements still to be had in p2pool, so I don't think the time is yet ripe for a rewrite in C.

(However, there are quite a few modules (mostly in the utils directory, like utils/math.py and utils/pack.py) that would make for straightforward C modules that could be called by python. But more on that later, maybe.)

edit: changed the code above. It seems to be working now. I don't know if the performance is improved yet. I'll check back in a few hours. This epollreactor thing will only run on Linux >= 2.6, by the way.

Hosting bitcoin miners for $75 to $90/kW/month on clean, cheap hydro power.
http://Toom.im
geneda
Newbie
*
Offline Offline

Activity: 7


View Profile
September 28, 2015, 10:55:58 AM
 #13405

This error is normal when you have no data to display somewhere... it was told like 200 times already.

Wow, I can do without the snide remark.  If you you are upset, just don't post...geez.

Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Anyone else (besides rav3n) have any advice?

Thanks for the time.
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 28, 2015, 11:07:45 AM
 #13406


Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Thanks for the time.

What credentials are you using on your miner?
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 28, 2015, 11:50:49 AM
 #13407


That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.


I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Wink Have you tried contacting forrestv at all?
geneda
Newbie
*
Offline Offline

Activity: 7


View Profile
September 28, 2015, 12:17:16 PM
 #13408


Anyway, your "answer" doesn't appear to be valid.  I also get those error messages when the miner attempts to connect.

Thanks for the time.

What credentials are you using on your miner?

I wasn't sure what to put in there, but I have it running on my local machine with the credentials that are in my bitcoin.conf.
All I did was copy bitcoin-qt and the blockchain up to my internet server, start bitcoin and p2pool (changing the rcpip and rpcallowip to the server address) then I just changed the
URL in my miner.  But it won't connect.

thanks again for the time...
btcscot
Jr. Member
*
Offline Offline

Activity: 30


View Profile
September 28, 2015, 02:40:02 PM
 #13409

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 28, 2015, 02:42:04 PM
 #13410

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon

I can see it's working on the IP you posted, well done!

Glad you got it up & running - welcome to p2pool!  Smiley
btcscot
Jr. Member
*
Offline Offline

Activity: 30


View Profile
September 28, 2015, 02:47:39 PM
 #13411

Wohooooo my node is live haven't connected a miner to it yet but its live Big thanks to p3yot33at3r for all the help

If you want to connect then its
62.75.188.90:9332

I will be attaching my domain at some point soon

I can see it's working on the IP you posted, well done!

Glad you got it up & running - welcome to p2pool!  Smiley

Yeah thanks now just need my miner to move onto it
Meuh6879
Legendary
*
Offline Offline

Activity: 1092



View Profile
September 28, 2015, 05:07:24 PM
 #13412

After the 5Ph/s peak ... we have waves ...  Grin so, P2Pool is the best pool ?  Cheesy

Sure and it can deserve more than that !

COME HERE !!!


French ... but not so much   ---===---   P2P ... it's people at the end   ---===---   P2Pool (10,9 GH/s).
Comment miner des bitcoins ? Un tutoriel est là : https://bitcointalk.org/index.php?topic=1114415.0
Bitcoin change everything ... an explain of this fact : https://www.youtube.com/watch?v=joITmEr4SjY
jtoomim
Hero Member
*****
Offline Offline

Activity: 555


View Profile WWW
September 29, 2015, 01:15:23 AM
 #13413

I noticed on the p2pool scroll every time I hit the the stat page, I get the following errors:

...

2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "p2pool\web.pyc", line 338, in <lambda>
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 > exceptions.TypeError: %x format: a number is requir
ed, not NoneType

Here's line 338 of web.py:


    new_root.putChild('best_share_hash', WebInterface(lambda: '%064x' % node.best_share_var.value))

I think what it's saying is that you don't have any shares yet, so you don't have a best_share_var.

You should know that Antminer S5s have some significant problems with p2pool that causes (a) lost hashrate unless you replace the default cgminer binary, and (b) instability, which can result in (c) hardware damage due to the S5 shutting down the fan immediately when cgminer stops running but heat continuing to be generated by the ASICs for several minutes afterward.

Hosting bitcoin miners for $75 to $90/kW/month on clean, cheap hydro power.
http://Toom.im
jtoomim
Hero Member
*****
Offline Offline

Activity: 555


View Profile WWW
September 29, 2015, 01:21:54 AM
 #13414


That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.


I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Wink Have you tried contacting forrestv at all?

No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call.

On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading).

Hosting bitcoin miners for $75 to $90/kW/month on clean, cheap hydro power.
http://Toom.im
idonothave
Full Member
***
Offline Offline

Activity: 234


View Profile
September 29, 2015, 08:25:31 AM
 #13415

thread should be renamed, I would recommend "p2pool: Decentralized, DoS-resistant, Hop-Proof pool"
jonnybravo0311
Legendary
*
Offline Offline

Activity: 1008


Mine at Jonny's Pool


View Profile WWW
September 29, 2015, 02:25:46 PM
 #13416


That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.


I take it you mean milliseconds?

Most of what you say goes straight over my head I'm afraid - far beyond my pay grade  Wink Have you tried contacting forrestv at all?

No, seconds. That's the cumulative use of all of the 1828987 calls to poll(), or an average of 0.004 seconds per call.

On second thought, those poll() calls might be including time that the CPU isn't actually busy, and yielding to other threads and processes. I'll have to take a closer look, and maybe compare the results with my other node (which has much higher loading).
You're spot on jtoomin.  The fact is that the code is single threaded, so even with a blazing fast CPU, you're going to see huge spikes in CPU load when you've got significant traffic (either because of transaction spamming of the network, or from a large number of connected miners).  While not an ideal solution, if you've got a large number of miners on your node, you might want to setup multiple nodes and spread the traffic load across them.  Not sure if that's an option for you, though... it's just a suggestion.

Jonny's Pool - Mine with us and help us grow!  Support a pool that supports Bitcoin, not a hardware manufacturer's pockets!  No SPV cheats.  No empty blocks.
Meuh6879
Legendary
*
Offline Offline

Activity: 1092



View Profile
September 29, 2015, 02:39:44 PM
 #13417


French ... but not so much   ---===---   P2P ... it's people at the end   ---===---   P2Pool (10,9 GH/s).
Comment miner des bitcoins ? Un tutoriel est là : https://bitcointalk.org/index.php?topic=1114415.0
Bitcoin change everything ... an explain of this fact : https://www.youtube.com/watch?v=joITmEr4SjY
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 29, 2015, 08:16:53 PM
 #13418

Is anyone here using the S7 on p2pool? If so, how are they performing? I'm interested to know if they are still using the crazy queue settings with their cgminer fork & if anyone has replaced it with the mainline cgminer yet. If bitmain lower their prices again soon, I might be tempted.....if they work OK with p2pool that is.
jonnybravo0311
Legendary
*
Offline Offline

Activity: 1008


Mine at Jonny's Pool


View Profile WWW
September 29, 2015, 09:03:51 PM
 #13419

Woot!  Back to back blocks for p2pool Grin

Jonny's Pool - Mine with us and help us grow!  Support a pool that supports Bitcoin, not a hardware manufacturer's pockets!  No SPV cheats.  No empty blocks.
p3yot33at3r
Sr. Member
****
Offline Offline

Activity: 266



View Profile
September 29, 2015, 09:08:46 PM
 #13420

Woot!  Back to back blocks for p2pool Grin

BOOM BOOM!!   Grin Grin
Pages: « 1 ... 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 [671] 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 ... 744 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!