[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[btcscot]

hey guys hit a problem and cant get past it

ERROR: problem running ufw-init
modprobe: ERROR: ../libkmod/libkmod.c:507 kmod_lookup_alias_from_builtin_file() could not open builtin file '/lib/modules/3.13.0-042stab108.8/modules.builtin.bin'
modprobe: FATAL: Module nf_conntrack_ftp not found.
modprobe: ERROR: ../libkmod/libkmod.c:507 kmod_lookup_alias_from_builtin_file() could not open builtin file '/lib/modules/3.13.0-042stab108.8/modules.builtin.bin'
modprobe: FATAL: Module nf_nat_ftp not found.
modprobe: ERROR: ../libkmod/libkmod.c:507 kmod_lookup_alias_from_builtin_file() could not open builtin file '/lib/modules/3.13.0-042stab108.8/modules.builtin.bin'
modprobe: FATAL: Module nf_conntrack_netbios_ns not found.
iptables-restore: line 30 failed
iptables-restore: line 10 failed
iptables-restore: line 65 failed
iptables-restore v1.4.21: Couldn't load target `ufw-user-input':No such file or directory

Error occurred at line: 2
Try `iptables-restore -h' or 'iptables-restore --help' for more information.
ip6tables-restore: line 42 failed
ip6tables-restore: line 2 failed
ip6tables-restore: line 8 failed
ip6tables-restore: line 73 failed
ip6tables-restore v1.4.21: Couldn't load target `ufw6-skip-to-policy-input':No such file or directory

Error occurred at line: 19
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
ip6tables-restore: line 10 failed
ip6tables-restore: line 65 failed
ip6tables-restore v1.4.21: Couldn't load target `ufw6-user-input':No such file or directory

Error occurred at line: 2
Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
sysctl: permission denied on key 'net.ipv4.tcp_sack'

Problem running '/etc/ufw/after.rules'
Problem running '/lib/ufw/user.rules'
Problem running '/etc/ufw/before6.rules'
Problem running '/etc/ufw/after6.rules'
Problem running '/lib/ufw/user6.rules'

Cloning into 'p2pool'...
remote: Counting objects: 9392, done.
remote: Total 9392 (delta 0), reused 0 (delta 0), pack-reused 9392
Receiving objects: 100% (9392/9392), 3.72 MiB | 1.78 MiB/s, done.
Resolving deltas: 100% (5910/5910), done.
Checking connectivity... done.
--2015-09-24 14:41:22--  http://pool.nitro.gr/p2pool-files.tar.gz
Resolving pool.nitro.gr (pool.nitro.gr)... 72.52.4.91
Connecting to pool.nitro.gr (pool.nitro.gr)|72.52.4.91|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘p2pool-files.tar.gz’

    [ <=>                                                                                                                ] 20.645      --.-K/s   in 0,01s

2015-09-24 14:41:23 (1,67 MB/s) - ‘p2pool-files.tar.gz’ saved [20645]


gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
install-merged-mining.sh: 337: install-merged-mining.sh: Bad substitution
root@euve82885:~#


any ideas?

[p3yot33at3r]

Looks like your IP tables is screwed - your firewall also, have you been adjusting it's settings?

Also, what's this:

--2015-09-24 14:41:22--  http://pool.nitro.gr/p2pool-files.tar.gz
Resolving pool.nitro.gr (pool.nitro.gr)... 72.52.4.91
Connecting to pool.nitro.gr (pool.nitro.gr)|72.52.4.91|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘p2pool-files.tar.gz’

    [ <=>                                                                                                                ] 20.645      --.-K/s   in 0,01s

2015-09-24 14:41:23 (1,67 MB/s) - ‘p2pool-files.tar.gz’ saved [20645]


gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
install-merged-mining.sh: 337: install-merged-mining.sh: Bad substitution
root@euve82885:~#

Did you download p2pool from the official repo? That bold text looks suspect.......

[btcscot]

Looks like your IP tables is screwed - your firewall also, have you been adjusting it's settings?

Also, what's this:

--2015-09-24 14:41:22--  http://pool.nitro.gr/p2pool-files.tar.gz
Resolving pool.nitro.gr (pool.nitro.gr)... 72.52.4.91
Connecting to pool.nitro.gr (pool.nitro.gr)|72.52.4.91|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘p2pool-files.tar.gz’

    [ <=>                                                                                                                ] 20.645      --.-K/s   in 0,01s

2015-09-24 14:41:23 (1,67 MB/s) - ‘p2pool-files.tar.gz’ saved [20645]


gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
install-merged-mining.sh: 337: install-merged-mining.sh: Bad substitution
root@euve82885:~#

Did you download p2pool from the official repo? That bold text looks suspect.......

I belive so it was the one from the auto config from the start of this topic where you run the wget and it does the rest apparently

[btcscot]

Good stuff. There's a guide at the very beginning of this thread, or you can use this:

https://bitcointalk.org/index.php?topic=651819.0;topicseen

..& just use what you need to get up & running. Some of the links are a bit outdated - I keep meaning to write an updated install guide, but keep getting side-tracked, etc......

If you have any questions, just ask here - someone will point you in the right direction.....eventually.

i used this link

[p3yot33at3r]

That guide is not maintained & very out of date, so are many of the links it uses. You should remove it all & download from the official repo:

https://github.com/p2pool/p2pool

From terminal:

Code:

git clone https://github.com/p2pool/p2pool.git

Then punch this into a terminal:

Code:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install software-properties-common
sudo add-apt-repository ppa:bitcoin/bitcoin
sudo apt-get update
sudo apt-get install libdb4.8-dev libdb4.8++-dev
sudo apt-get --no-install-recommends install python-software-properties screen git python-rrdtool python-pygame python-scipy python-twisted python-twisted-web python-imaging build-essential libglib2.0-dev libglibmm-2.4-dev python-dev libssl-dev libboost-all-dev dh-autoreconf libcurl4-openssl-dev libminiupnpc-dev ufw libtool autotools-dev autoconf automake pkg-config libprotoc8 libgtk2.0-dev libgmp-dev libmpfr-dev yasm libevent-dev

This will install all the needed dependencies for p2pool & your merge mined coins. If you want to have a GUI for your wallets, add:

Code:

sudo apt-get install libqt4-dev libprotobuf-dev protobuf-compiler

For your Bitcoin wallet, type:

Code:

sudo apt-get install bitcoind

Or, for the GUI version:

Code:

sudo apt-get install bitcoin-qt

This should get you started  

[btcscot]

That guide is not maintained & very out of date, so are many of the links it uses. You should remove it all & download from the official repo:

https://github.com/p2pool/p2pool

From terminal:

Code:

git clone https://github.com/p2pool/p2pool.git

Then punch this into a terminal:

Code:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install software-properties-common
sudo add-apt-repository ppa:bitcoin/bitcoin
sudo apt-get update
sudo apt-get install libdb4.8-dev libdb4.8++-dev
sudo apt-get --no-install-recommends install python-software-properties screen git python-rrdtool python-pygame python-scipy python-twisted python-twisted-web python-imaging build-essential libglib2.0-dev libglibmm-2.4-dev python-dev libssl-dev libboost-all-dev dh-autoreconf libcurl4-openssl-dev libminiupnpc-dev ufw libtool autotools-dev autoconf automake pkg-config libprotoc8 libgtk2.0-dev libgmp-dev libmpfr-dev samba nautilus gdebi yasm libevent-dev

This will install all the needed dependencies for p2pool & your merge mined coins. If you want to have a GUI for your wallets, add:

Code:

sudo apt-get install libqt4-dev libprotobuf-dev protobuf-compiler

For your Bitcoin wallet, type:

Code:

sudo apt-get install bitcoind

Or, for the GUI version:

Code:

sudo apt-get install bitcoin-qt

This should get you started 

thanks going to try this once the vps gets rebooted and hopefully it will get sorted to uninstall all the crap from before easy ?

[geneda]

Is anyone using p2pool on a windows machine?  Is it not recommend?  I followed the instruction on to separate machines and get the same error on both machines.


C:\Python27\Lib\site-packages>run_p2pool.exe

2015-09-25 06:51:43.484000 p2pool (version 14.0)
2015-09-25 06:51:43.484000
2015-09-25 06:51:43.484000 Testing bitcoind RPC connection to 'http://127.0.0.1:
8332/' with username 'user'...
2015-09-25 06:51:43.937000 > Error getting work from bitcoind:
2015-09-25 06:51:43.937000 > Traceback (most recent call last):
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 551, in _
runCallbacks
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 1101, in
gotResult
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 > --- <exception caught here> ---
2015-09-25 06:51:43.939000 >   File "p2pool\util\deferral.pyc", line 41, in f
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "p2pool\bitcoin\helper.pyc", line 39, in get
work
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.941000 >   File "p2pool\util\jsonrpc.pyc", line 133, in _htt
p_do
2015-09-25 06:51:43.941000 >
2015-09-25 06:51:43.941000 > p2pool.util.jsonrpc.NarrowError: -10 Bitcoin is dow
nloading blocks...

[btcscot]

Is anyone using p2pool on a windows machine?  Is it not recommend?  I followed the instruction on to separate machines and get the same error on both machines.


C:\Python27\Lib\site-packages>run_p2pool.exe

2015-09-25 06:51:43.484000 p2pool (version 14.0)
2015-09-25 06:51:43.484000
2015-09-25 06:51:43.484000 Testing bitcoind RPC connection to 'http://127.0.0.1:
8332/' with username 'user'...
2015-09-25 06:51:43.937000 > Error getting work from bitcoind:
2015-09-25 06:51:43.937000 > Traceback (most recent call last):
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 551, in _
runCallbacks
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 1101, in
gotResult
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.938000 >
2015-09-25 06:51:43.938000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 > --- <exception caught here> ---
2015-09-25 06:51:43.939000 >   File "p2pool\util\deferral.pyc", line 41, in f
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.939000 >
2015-09-25 06:51:43.939000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "p2pool\bitcoin\helper.pyc", line 39, in get
work
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "twisted\internet\defer.pyc", line 1043, in
_inlineCallbacks
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.940000 >   File "twisted\python\failure.pyc", line 382, in t
hrowExceptionIntoGenerator
2015-09-25 06:51:43.940000 >
2015-09-25 06:51:43.941000 >   File "p2pool\util\jsonrpc.pyc", line 133, in _htt
p_do
2015-09-25 06:51:43.941000 >
2015-09-25 06:51:43.941000 > p2pool.util.jsonrpc.NarrowError: -10 Bitcoin is dow
nloading blocks...

i think you have just hit the same point as i have you are compiling the btc blocks if you go to

cd ~/.bitcoin

then run

tail -f debug.log

if its still saying dates from like 2013 you cant run the pool till it downloads all the files

[p3yot33at3r]

Is anyone using p2pool on a windows machine?  Is it not recommend?  I followed the instruction on to separate machines and get the same error on both machines.

2015-09-25 06:51:43.941000 > p2pool.util.jsonrpc.NarrowError: -10 Bitcoin is dow
nloading blocks...

Exactly - you must wait for Bitcoin wallet to finish sync. It will take a while, it's a large file, just wait until it's finished then restart p2pool.

[geneda]

Thanks guys!  I though the daemon used the same path as bitcoin-qt, I guess not.  I have the whole blockchain, I just need to point the daemon to it.  Looking for path in registry now..

Thanks agian!

Edit:  found it -datadir=<dir>

[geneda]

One other quick question, I see this in the p2pool scroll:

2015-09-25 08:41:34.203000
2015-09-25 08:41:34.204000 Determining payout address...
2015-09-25 08:41:34.205000     Loaded cached address: <some address>
<some address cont>...
2015-09-25 08:41:34.209000     ...success! Payout address: <some address>
<some address cont>

I don't recognize the address.  It's not any that I see in bitcoin-qt

[p3yot33at3r]

One other quick question, I see this in the p2pool scroll:

2015-09-25 08:41:34.203000
2015-09-25 08:41:34.204000 Determining payout address...
2015-09-25 08:41:34.205000     Loaded cached address: <some address>
<some address cont>...
2015-09-25 08:41:34.209000     ...success! Payout address: <some address>
<some address cont>

I don't recognize the address.  It's not any that I see in bitcoin-qt

P2pool creates an address to payout to in your wallet when it first connects. In your wallet menu, click file>receiving addresses & it should be listed there. For Bitcoind, in terminal type:

Code:

bitcoind listreceivedbyaddress 0 true

..it will display the address 

[geneda]

OK, I got my windows p2pool running nicely, I have my S5 pointing to it and life is good.. Well, I want to move the config to a dedicated server I have on the internet.
So I got it all configured on the server and things appear great until I go to the stat page on :9332.  I don't see an error on the web page, but it doesn't list all the address payouts etc.

I noticed on the p2pool scroll every time I hit the the stat page, I get the following errors:

2015-09-26 06:34:32.543000 > Error in DeferredResource handler:
2015-09-26 06:34:32.543000 > Traceback (most recent call last):
2015-09-26 06:34:32.543000 >   File "p2pool\util\deferred_resource.pyc", line 24
, in render
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "twisted\internet\defer.pyc", line 134, in m
aybeDeferred
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "twisted\web\resource.pyc", line 216, in ren
der
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "twisted\internet\defer.pyc", line 1187, in
unwindGenerator
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 > --- <exception caught here> ---
2015-09-26 06:34:32.559000 >   File "twisted\internet\defer.pyc", line 1045, in
_inlineCallbacks
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "p2pool\web.pyc", line 198, in render_GET
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 >   File "p2pool\web.pyc", line 338, in <lambda>
2015-09-26 06:34:32.559000 >
2015-09-26 06:34:32.559000 > exceptions.TypeError: %x format: a number is requir
ed, not NoneType



I have ports 8333 and 9332 opened.  Like I said, I can hit the page fine from my house, but I get those errors on the scroll every time.

[rav3n_pl]

This error is normal when you have no data to display somewhere... it was told like 200 times already.

[p3yot33at3r]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

[Meuh6879]

  Yep ... it's good.

[jedimstr]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

More than likely a big player has us as a backup pool and they just rebooted/restarted their primary pool instance.  Definitely looks more like a failover spike to me.

[jtoomim]

Nice to see some blocks rolling in again - makes up for the baron spell earlier. I'm seeing some big spikes in hash rate lately, up to 5Ph+ - hopefully a big player will be joining us soon 

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

http://xtnodes.com/bip101_mining_pools.php

http://74.82.233.205:9334/static/graphs.html?Week

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

[p3yot33at3r]

Both of the two p2pool blocks found yesterday were mined at our node. We've been getting a few people renting hashrate and pointing it at our node or a couple of other nodes that run on BitcoinXT. Yesterday, we had someone point about 2.5 PH/s at our node for a few hours. That miner found one of the blocks.

Speaking of > 2 PH/s on a single node, is anybody working on (or interested in collaborating on) performance enhancements for p2pool? I'm seeing a lot of load spikes up to 100% CPU cure utilization for up to a few seconds by the p2pool process, and switching to pypy doesn't solve the issue. Right now I'm using python -m cProfile -o cprofile-001.log run_p2pool.py to collect and analyze data on what functions in p2pool are hosing my CPU.

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  

[jtoomim]

I've never seen cpu usage go higher than a 35% spike running p2pool, it usually sits around 8% - & that's merge mining 12 other alts. Maybe using XT is the problem?  

What OS are you using for your nodes?

I think the biggest performance improvement could be achieved by using C instead of python, but this has been discussed many times previously in this thread to no avail. I never could get pypy to work with p2pool, could you describe how you installed/ran it - I'd like to have another go  

Keep in mind that I have 350 TH/s on my node 24/7, and spikes up to 3.0 PH/s. We probably have about two hundred miners (i.e. computers, not owners) that use our node. I think that might have something to do with the CPU load.

Linux prime 3.2.0-4-amd64 #1 SMP Debian 3.2.63-2+deb7u2 x86_64 GNU/Linux

Intel(R) Core(TM) i3-2120 CPU @ 3.30GHz

I think the problem might be related to the use of the poll()-based reactor in twisted. The poll() function checks a bunch of file descriptors (e.g. network ports) to see if any have data available. When you have a lot of

http://pipeit.blogspot.com/2011/07/select-poll-and-epoll-twisted-story.html

Here are the first few lines of my cProfile stats results for my non-XT node (slower CPU), sorted by the total time each function took (not including subroutines):

Code:

>>> p.print_stats()
Sun Sep 27 17:25:26 2015    cprofile-001.log

         1042548800 function calls (1022440006 primitive calls) in 9605.998 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1828987 7667.676    0.004 7667.676    0.004 {built-in method poll}
   133405  631.073    0.005  701.931    0.005 p2pool/p2p.py:189(update_remote_view_of_my_known_txs)
  2025600   55.807    0.000   91.726    0.000 p2pool/util/math.py:64(add_dicts)
6083060/521034   42.351    0.000  204.197    0.000 p2pool/util/pack.py:292(read)
8211943/2317050   41.329    0.000  187.231    0.000 p2pool/util/pack.py:298(write)
 50788863   38.770    0.000   45.809    0.000 p2pool/util/pack.py:13(read)
  1748230   33.862    0.000   35.874    0.000 p2pool/util/graph.py:14(_shift)
3454845/2954699   31.683    0.000  240.778    0.000 p2pool/util/pack.py:51(_pack)
 33374468   31.063    0.000   63.310    0.000 p2pool/util/pack.py:96(read)
 39492368   30.537    0.000   42.449    0.000 p2pool/util/pack.py:115(write)
    17576   29.261    0.002   36.366    0.002 p2pool/node.py:252(_)
2559639/2225541   27.652    0.000 1004.009    0.000 p2pool/util/variable.py:68(set)
    12883   25.897    0.002  298.297    0.023 p2pool/p2p.py:396(handle_remember_tx)
925154/859232   25.618    0.000  191.821    0.000 p2pool/util/pack.py:166(read)
  1748230   25.587    0.000  128.783    0.000 p2pool/util/graph.py:46(_add_datum)
  7155588   23.545    0.000   32.801    0.000 p2pool/util/pack.py:221(write)
1432678/1382638   22.485    0.000  153.893    0.000 p2pool/util/pack.py:174(write)
4503389/2916784   21.317    0.000   29.991    0.000 p2pool/util/pack.py:270(__eq__)
  6426883   21.155    0.000   33.183    0.000 p2pool/util/pack.py:215(read)
   515095   20.821    0.000   34.674    0.000 p2pool/util/math.py:186(natural_to_string)
 33493018   17.543    0.000   26.334    0.000 p2pool/util/pack.py:259(__getitem__)
101548128/101547801   17.280    0.000   18.491    0.000 {len}
2198269/2198243   16.876    0.000  101.744    0.000 {sorted}
 47813834   16.131    0.000   16.131    0.000 {_struct.pack}
      941   15.535    0.017   35.789    0.038 p2pool/data.py:119(generate_transaction)
  5306039   14.821    0.000   14.821    0.000 {_hashlib.openssl_sha256}
     5401   14.798    0.003   14.803    0.003 {method 'items' of 'dict' objects}
    44406   14.483    0.000   24.064    0.001 p2pool/bitcoin/sha256.py:17(process)
 49754459   14.364    0.000   15.180    0.000 {getattr}
 82295913   13.969    0.000   13.969    0.000 {method 'append' of 'list' objects}
  7485573   13.270    0.000   13.270    0.000 {binascii.a2b_hex}
 34547735   13.172    0.000   13.172    0.000 p2pool/util/math.py:69(<genexpr>)
     5850   11.564    0.002  407.450    0.070 p2pool/node.py:246(_)
  2712960   10.420    0.000   13.272    0.000 p2pool/data.py:1011(add_share)
  7433713   10.328    0.000   16.661    0.000 p2pool/util/forest.py:332(get_chain)
2534168/285416   10.066    0.000 1790.124    0.006 {method 'send' of 'generator' objects}
    16053    9.917    0.001    9.917    0.001 /usr/lib/python2.7/json/encoder.py:205(iterencode)
 43314052    9.827    0.000    9.827    0.000 {method 'get' of 'dict' objects}
    11226    9.816    0.001    9.816    0.001 /usr/lib/python2.7/json/decoder.py:371(raw_decode)
  6337858    9.336    0.000   17.570    0.000 p2pool/util/pack.py:188(read)
 35144317    9.291    0.000    9.291    0.000 p2pool/util/math.py:63(<lambda>)
  2712960    9.236    0.000   11.115    0.000 p2pool/data.py:1022(add_verified_hash)
     5103    8.790    0.002  175.572    0.034 p2pool/node.py:23(handle_shares)
 25577856    8.662    0.000    8.662    0.000 p2pool/bitcoin/sha256.py:18(rightrotate)
  5306038    8.635    0.000    8.635    0.000 {method 'digest' of '_hashlib.HASH' objects}
  1748230    8.348    0.000   27.323    0.000 p2pool/util/graph.py:23(_)
   185065    8.338    0.000 1097.603    0.006 p2pool/util/p2protocol.py:63(packetReceived)
  2236393    8.072    0.000   10.352    0.000 p2pool/util/datachunker.py:18(get)
    15805    7.840    0.000   23.109    0.001 p2pool/data.py:338(get_other_tx_hashes)
 16472083    7.647    0.000    7.647    0.000 {divmod}
 13992152    7.534    0.000   11.242    0.000 p2pool/util/pack.py:261(__setitem__)
  1748230    7.523    0.000   45.516    0.000 p2pool/util/graph.py:32(_shift_bins_so_t_is_not_past_end)
  2526120    7.308    0.000   12.750    0.000 p2pool/bitcoin/height_tracker.py:106(get_height_rel_highest)
  6389979    6.975    0.000    7.048    0.000 {method 'join' of 'str' objects}
   138138    6.974    0.000   75.463    0.001 p2pool/util/skiplist.py:12(__call__)
  1742141    6.708    0.000   37.944    0.000 p2pool/bitcoin/data.py:10(hash256)
  3859910    6.190    0.000   43.954    0.000 p2pool/util/pack.py:313(write)
  8056916    5.986    0.000    9.912    0.000 p2pool/util/pack.py:192(write)
      157    5.716    0.036   35.242    0.224 p2pool/main.py:199(save_shares)
6809454/4283334    5.614    0.000   29.820    0.000 {max}
  6911540    5.597    0.000    5.597    0.000 {binascii.b2a_hex}
      969    5.518    0.006   28.824    0.030 p2pool/bitcoin/data.py:180(calculate_merkle_link)
  3556045    5.510    0.000    6.936    0.000 p2pool/util/graph.py:24(<lambda>)
  1828987    5.438    0.000 9135.571    0.005 /usr/lib/python2.7/dist-packages/twisted/internet/pollreactor.py:147(doPoll)
3454845/2954699    5.414    0.000  245.595    0.000 p2pool/util/pack.py:73(pack)
  4553432    5.247    0.000   20.618    0.000 p2pool/util/pack.py:130(read)
 17540412    5.109    0.000    5.109    0.000 {method 'iteritems' of 'dict' objects}
       95    5.102    0.054    5.102    0.054 {posix.fsync}
2236477/2236471    5.097    0.000 1135.022    0.001 p2pool/util/p2protocol.py:29(dataReceiver)
  6729729    4.927    0.000   33.823    0.000 p2pool/util/pack.py:82(packed_size)
 33380877    4.815    0.000    4.815    0.000 {ord}
  1828987    4.779    0.000  459.169    0.000 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:762(runUntilCurrent)
 17987582    4.687    0.000    4.687    0.000 {setattr}
  2807093    4.630    0.000  223.412    0.000 p2pool/util/pack.py:41(_unpack)
438995/222789    4.491    0.000 1064.895    0.005 p2pool/util/variable.py:35(happened)
  3928877    4.374    0.000    9.756    0.000 p2pool/util/pack.py:134(write)
        1    4.369    4.369 9605.102 9605.102 /usr/lib/python2.7/dist-packages/twisted/internet/base.py:1172(mainLoop)
   336129    4.218    0.000   37.090    0.000 {method 'update' of 'dict' objects}
 11691309    4.045    0.000    4.045    0.000 {isinstance}
  3042144    4.038    0.000    5.191    0.000 p2pool/bitcoin/data.py:217(target_to_average_attempts)
  5804151    3.976    0.000   23.636    0.000 p2pool/data.py:367(<genexpr>)

That indicates 7667 seconds used by the poll() function inside twisted, of about 9606 seconds total for the whole program.

I think I might try a

Code:

try:
    from twisted.internet import epollreactor
    epollreactor.install()
except:
    print "twisted.internet.epollreactor could not be loaded. "
    print "Falling back to pollreactor. Performance may suffer."

from twisted.internet import reactor

or something like that. It will probably take a little more than that, as I don't know if the APIs are compatible.

The biggest performance improvements are usually algorithmic, not language-based. You shouldn't change languages until you are pretty sure that you have the algorithm down. There are some big algorithmic improvements still to be had in p2pool, so I don't think the time is yet ripe for a rewrite in C.

(However, there are quite a few modules (mostly in the utils directory, like utils/math.py and utils/pack.py) that would make for straightforward C modules that could be called by python. But more on that later, maybe.)

edit: changed the code above. It seems to be working now. I don't know if the performance is improved yet. I'll check back in a few hours. This epollreactor thing will only run on Linux >= 2.6, by the way.