[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[PatMan]

I just noticed cgminer is getting 10-15% rejects from p2pool now as well.

What seems to be happening is p2pool freezes as it processes a huge amount of transactions from some node.  cgminer complains the pool isn't providing work fast enough, p2pool starts responding again, a bunch of work gets rejected and p2pool drops a bunch of connections because it timed out.

M

I noticed that on my cgminer too, I downloaded the newest release and was going to install it but thought I'd wait until this settles down again....

[mdude77]

I just noticed cgminer is getting 10-15% rejects from p2pool now as well.

What seems to be happening is p2pool freezes as it processes a huge amount of transactions from some node.  cgminer complains the pool isn't providing work fast enough, p2pool starts responding again, a bunch of work gets rejected and p2pool drops a bunch of connections because it timed out.

M

I noticed that on my cgminer too, I downloaded the newest release and was going to install it but thought I'd wait until this settles down again....

I also noticed my unit value decreased again.  p2pool is usually 1 integer lower than non p2pools per GPU, now it's 2 or much greater.  ie, I usually get about 9.1unit per 7970, so a 3x7970 rigs gets my around 27.  on p2p, I expect around 24.  right now my 3x7970 rigs are at 17 and 18.  there's a huge amount of work going to waste here.

I'm switching over to EMC and Oz until I see things settle down.

M

[lenny_]

Memory leak found. p2pool process eating +600 MB of memory (+1GB virtual). Node uptime was about 12 hours?
From /proc:

Code:

$ cat status
Name:   python
State:  D (disk sleep)
Tgid:   9064
Pid:    9064
PPid:   9063
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
FDSize: 128
Groups: 24 25 29 30 44 46 1000
VmPeak:  1105160 kB
VmSize:  1105156 kB
VmLck:         0 kB
VmHWM:    852568 kB
VmRSS:    643744 kB
VmData:  1038240 kB
VmStk:       204 kB
VmExe:      2164 kB
VmLib:      4976 kB
VmPTE:      2048 kB
Threads:        4
SigQ:   0/16382
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000001001000
SigCgt: 0000000180016202
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: ffffffffffffffff
Cpus_allowed:   f
Cpus_allowed_list:      0-3
Mems_allowed:   00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        803930
nonvoluntary_ctxt_switches:     272976

From htop:

Code:

PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
 9064 pioruns   20   0 1079M  620M   616 D  1.0 62.6  8h25:41 python /home/pioruns/p2pool/run_p2pool.py --fee=0.1 --give-author=0.1 --disable-upnp --outgoing-conns 5 --merged=xxx

EDIT: I just updated p2pool to latest version from git. Maybe will be better this time.

Newest version from git and first errors founded:

Code:

2012-11-26 10:31:28.835087 Punishing share for 'not all txs present'! Jumping from ecbce9f3 to 7b2cc82d!
2012-11-26 10:31:29.650609 Punishing share for 'not all txs present'! Jumping from ecbce9f3 to 7b2cc82d!
2012-11-26 10:31:59.581719 Processing 501 shares from 83.169.39.90:9333...
2012-11-26 10:32:00.210566 Punishing share for 'not all txs present'! Jumping from ecbce9f3 to 7b2cc82d!
2012-11-26 10:32:00.211285 ... done processing 501 shares. New: 501 Have: 19226/~17280
2012-11-26 10:32:00.211971 Requesting parent share 69de4908 from 24.211.170.224:9333
2012-11-26 10:32:00.215959 > Watchdog timer went off at:
2012-11-26 10:32:00.216313 >   File "/home/pioruns/p2pool/run_p2pool.py", line 5, in <module>
2012-11-26 10:32:00.216520 >     main.run()
2012-11-26 10:32:00.216704 >   File "/home/pioruns/p2pool/p2pool/main.py", line 561, in run
2012-11-26 10:32:00.216889 >     reactor.run()
2012-11-26 10:32:00.217071 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1165, in run
2012-11-26 10:32:00.217257 >     self.mainLoop()
2012-11-26 10:32:00.217434 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1177, in mainLoop
2012-11-26 10:32:00.217615 >     self.doIteration(t)
2012-11-26 10:32:00.217789 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py", line 140, in doSelect
2012-11-26 10:32:00.217972 >     _logrun(selectable, _drdw, selectable, method, dict)
2012-11-26 10:32:00.218150 >   File "/usr/lib/python2.6/dist-packages/twisted/python/log.py", line 84, in callWithLogger
2012-11-26 10:32:00.218330 >     return callWithContext({"system": lp}, func, *args, **kw)
2012-11-26 10:32:00.218505 >   File "/usr/lib/python2.6/dist-packages/twisted/python/log.py", line 69, in callWithContext
2012-11-26 10:32:00.218682 >     return context.call({ILogContext: newCtx}, func, *args, **kw)
2012-11-26 10:32:00.218858 >   File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 59, in callWithContext
2012-11-26 10:32:00.219210 >     return self.currentContext().callWithContext(ctx, func, *args, **kw)
2012-11-26 10:32:00.219488 >   File "/usr/lib/python2.6/dist-packages/twisted/python/context.py", line 37, in callWithContext
2012-11-26 10:32:00.219753 >     return func(*args,**kw)
2012-11-26 10:32:00.220006 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
2012-11-26 10:32:00.220273 >     why = getattr(selectable, method)()
2012-11-26 10:32:00.220537 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/tcp.py", line 460, in doRead
2012-11-26 10:32:00.220827 >     return self.protocol.dataReceived(data)
2012-11-26 10:32:00.221105 >   File "/home/pioruns/p2pool/p2pool/p2p.py", line 146, in new_dataReceived
2012-11-26 10:32:00.221366 >     old_dataReceived(data)
2012-11-26 10:32:00.221618 >   File "/home/pioruns/p2pool/p2pool/util/p2protocol.py", line 39, in dataReceived
2012-11-26 10:32:00.221893 >     self.dataReceived2(data)
2012-11-26 10:32:00.222179 >   File "/home/pioruns/p2pool/p2pool/util/datachunker.py", line 40, in _DataChunker
2012-11-26 10:32:00.222467 >     wants = receiver.send(buf.get(wants))
2012-11-26 10:32:00.222760 >   File "/home/pioruns/p2pool/p2pool/util/p2protocol.py", line 66, in dataReceiver
2012-11-26 10:32:00.223102 >     self.packetReceived(command, type_.unpack(payload))
2012-11-26 10:32:00.223384 >   File "/home/pioruns/p2pool/p2pool/p2p.py", line 91, in packetReceived
2012-11-26 10:32:00.223616 >     p2protocol.Protocol.packetReceived(self, command, payload2)
2012-11-26 10:32:00.223809 >   File "/home/pioruns/p2pool/p2pool/util/p2protocol.py", line 80, in packetReceived
2012-11-26 10:32:00.223989 >     handler(**payload2)
2012-11-26 10:32:00.224166 >   File "/home/pioruns/p2pool/p2pool/p2p.py", line 316, in handle_sharereply
2012-11-26 10:32:00.224348 >     self.get_shares.got_response(id, res)
2012-11-26 10:32:00.224525 >   File "/home/pioruns/p2pool/p2pool/util/deferral.py", line 124, in got_response
2012-11-26 10:32:00.224704 >     df.callback(resp)
2012-11-26 10:32:00.224880 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 318, in callback
2012-11-26 10:32:00.225058 >     self._startRunCallbacks(result)
2012-11-26 10:32:00.225285 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 424, in _startRunCallbacks
2012-11-26 10:32:00.225476 >     self._runCallbacks()
2012-11-26 10:32:00.225655 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 441, in _runCallbacks
2012-11-26 10:32:00.225834 >     self.result = callback(self.result, *args, **kw)
2012-11-26 10:32:00.226008 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 949, in gotResult
2012-11-26 10:32:00.226185 >     _inlineCallbacks(r, g, deferred)
2012-11-26 10:32:00.226361 >   File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 893, in _inlineCallbacks
2012-11-26 10:32:00.226539 >     result = g.send(result)
2012-11-26 10:32:00.226713 >   File "/home/pioruns/p2pool/p2pool/node.py", line 124, in download_shares
2012-11-26 10:32:00.226892 >     self.handle_shares(shares, peer)
2012-11-26 10:32:00.227306 >   File "/home/pioruns/p2pool/p2pool/node.py", line 40, in handle_shares
2012-11-26 10:32:00.227593 >     self.node.set_best_share()
2012-11-26 10:32:00.227851 >   File "/home/pioruns/p2pool/p2pool/node.py", line 284, in set_best_share
2012-11-26 10:32:00.228106 >     best, desired, decorated_heads = self.tracker.think(self.get_height_rel_highest, self.bitcoind_work.value['previous_block'], self.bitcoind_work.value['bits'], self.known_txs_var.value)
2012-11-26 10:32:00.228406 >   File "/home/pioruns/p2pool/p2pool/data.py", line 1030, in think
2012-11-26 10:32:00.228670 >     ), h) for h in self.verified.tails.get(best_tail, []))
2012-11-26 10:32:00.228924 >   File "/home/pioruns/p2pool/p2pool/data.py", line 1030, in <genexpr>
2012-11-26 10:32:00.229177 >     ), h) for h in self.verified.tails.get(best_tail, []))
2012-11-26 10:32:00.229460 >   File "/home/pioruns/p2pool/p2pool/data.py", line 336, in should_punish_reason
2012-11-26 10:32:00.229750 >     other_txs = self._get_other_txs(tracker, known_txs)
2012-11-26 10:32:00.230013 >   File "/home/pioruns/p2pool/p2pool/data.py", line 320, in _get_other_txs
2012-11-26 10:32:00.230266 >     other_tx_hashes = self.get_other_tx_hashes(tracker)
2012-11-26 10:32:00.230516 >   File "/home/pioruns/p2pool/p2pool/data.py", line 317, in get_other_tx_hashes
2012-11-26 10:32:00.230779 >     return [tracker.items[tracker.get_nth_parent_hash(self.hash, x['share_count'])].share_info['new_transaction_hashes'][x['tx_count']] for x in self.share_info['transaction_hash_refs']]
2012-11-26 10:32:00.231127 >   File "/home/pioruns/p2pool/p2pool/util/memoize.py", line 30, in b
2012-11-26 10:32:00.231403 >     backing[(f, args)] = res
2012-11-26 10:32:00.231708 >   File "/home/pioruns/p2pool/p2pool/util/memoize.py", line 16, in __setitem__
2012-11-26 10:32:00.231989 >     while len(self.inner) > self.n:
2012-11-26 10:32:00.232262 >   File "/home/pioruns/p2pool/p2pool/main.py", line 242, in <lambda>
2012-11-26 10:32:00.232547 >     sys.stderr.write, 'Watchdog timer went off at:\n' + ''.join(traceback.format_stack())
2012-11-26 10:32:00.361499 P2Pool: 18719 shares in chain (18724 verified/19226 total) Peers: 3 (0 incoming)

EDIT:
After 2 hours, it doesn't look nice. Actual memory usage 457MB, virtual 607MB. Pool is hanging for 2-3 minutes when processing packs of "501 shares" form different peers. Sometime processing takes 2 seconds, sometime up to 7 minutes. During processing, p2pool is completely unresponsible, mining doesn't work and webpage does not responding. Also p2pool reporting that it lost communication with bitcoind, while it is working fine all times.
Looks like with this new features, p2pool become completely unavailable for laptops and other low-end machines. You can check my daily stats here: http://lenny.dnsd.me:9332/static/graphs.html?Day looks horrible

[EpicFail]

Is anyone currently NOT having problems?

[iongchun]

I don't know if it is a reasonable move... but I just add about twenty IP address to firewall blocking table.
It looks like p2pool are spamming itself.

[spiccioli]

EDIT:
After 2 hours, it doesn't look nice. Actual memory usage 457MB, virtual 607MB. Pool is hanging for 2-3 minutes when processing packs of "501 shares" form different peers. Sometime processing takes 2 seconds, sometime up to 7 minutes. During processing, p2pool is completely unresponsible, mining doesn't work and webpage does not responding. Also p2pool reporting that it lost communication with bitcoind, while it is working fine all times.
Looks like with this new features, p2pool become completely unavailable for laptops and other low-end machines. You can check my daily stats here: http://lenny.dnsd.me:9332/static/graphs.html?Day looks horrible

same here, PIV 3GHz, an old pc, but not so slow, it stops when it receives those 501 shares packets...



spiccioli

[Subo1977]

Here is the same, with newest P2Pool and Bitcoind from Git on Linux

[BR0KK]

Incredible high stale rates and  memory issue

Code:

Version: 9.0

Pool rate: 401GH/s (28% stale) Share difficulty: 761

Node uptime: 0.373 days Peers: 10 out, 2 in

Local rate: 3.79GH/s (46% DOA) Expected time to share: 0.240 hours

Shares: 56 total (7 orphaned, 21 dead) Efficiency: 69.72%

[K1773R]

just wait until forrestv accepts the pull request or do it yourself!

[cabin]

so I guess we should all be ready for a new version soon to fix all these issues.. hopefully we can get the whole pool upgraded fast if that is needed.

[abbeytim]

that 28% stale share isnt good either. i understand that there are more stales in p2pool but 28% ??

[Aseras]

the same thing happened last fork. it always seems like just when p2pool is working good someone throws a wrench in it and it goes to crap.

[Prattler]

After the switchover I'm seeing high memory usage (1.1 GB), high CPU usage and high DoA rate (increased from 2% to 5%). I imagine the increased DoA is due to the CPU usage (lockups).

Using recent git, so the new fixes didn't help.

[TAiS46]

Hey,

I have now run p2pool + miner for about 10 minutes.
Are this stats normal?
I have a lot of rejected shares.

[forrestv]

Things should get better tomorrow when the last of the old shares fall off the end of the sharechain and I make a new release that fixes several critical issues.

[Krak]

Hey,

I have now run p2pool + miner for about 10 minutes.
Are this stats normal?
I have a lot of rejected shares.

http://i.imagebanana.com/img/69p4jvda/Bildschirmfoto20121127um02.31.22.png

http://i.imagebanana.com/img/54xmi7v6/Bildschirmfoto20121127um02.31.36.png

BFL Singles don't really work with p2pool.

[TAiS46]

Things should get better tomorrow when the last of the old shares fall off the end of the sharechain and I make a new release that fixes several critical issues.

ok lets see
hope it will work better tomorrow.
if yes, I will set up a new pool for bitcoin-24.com

[bitpop]

DO NOT use bfl singles with p2pool

[TAiS46]

ok, so I will remove it.
are there any other pool server I can set up to work on own servers?

[bitpop]

There is pool software you could setup but theres no point and its very difficult. Its just like solo mining which you could do just with regular bitcoin client. Depending on how many singles you have, it could be a good choice. But until we get the new SC singles which should work with p2pool, just use a regular pool like slushs.