[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[Duce]

Thanks guys I was looking at the protocol of 1400 and thought the pull had updated my version, dumb on my part. I now have 14.0-5-gb07b1e3 running. I did keep my data and web-static files after performing a clone. All looks good with the version now just a wait and let it run to see if the latency and traffic rate get back to normal.

So do I really need to move up to 10.2 or can I just continue at this point with 10.1?

EDIT: Updated to 10.2 as well.

[Duce]

It appears that if you are updated to version 14 you will not show up here http://p2pool.jir.dk/bitcoin/ anymore. Not sure if you guys have taken a look here before but if this is the case we have more that will be left behind.

[jedimstr]

It appears that if you are updated to version 14 you will not show up here http://p2pool.jir.dk/bitcoin/ anymore. Not sure if you guys have taken a look here before but if this is the case we have more that will be have already been left behind.

FTFY - Hardfork already occured so these people won't be paid anything when p2pool gets a block.

Owner of the site is available on twitter: https://twitter.com/mclemmensen

Can someone ask this guy to switch to only showing 14.0+ nodes and tell him/her why?

[squidicuz]

I just checked my node, as it was showing down.  It looks like it's hyper trying to retrieve shares from multiple places:

Code:

2015-07-05 19:06:16.252636 Requesting parent share 66033bd7 from 74.82.233.205:9333
2015-07-05 19:06:17.357663 Requesting parent share 66033bd7 from 149.210.228.165:9333
2015-07-05 19:06:18.809206 Requesting parent share 66033bd7 from 80.60.85.169:9333
2015-07-05 19:06:19.088368 Requesting parent share 37dddcee from 74.82.233.205:9333

Notice it requested the same parent share from four nodes within 4 seconds.  That's not very efficient, and I assume that's why it's having a hard time keeping up with the shares.  It's doing this regularly.

M

I've been seeing this the last few days too, but put it down to the forking & old nodes still not being updated - but maybe it's something else.....?

I have been investigating this all day.  Sometimes the entire instance times out, web interface won't even respond.  Not sure what is going on, but it can be seen in the charts on several other nodes as well. Traffic is higher than usual.  In the log there are many requests for parent shares and the sending of shares.

Below, the entire instance is locked for 2 minutes while 'Processing 164 shares from ...' then it crashes.

Code:

2015-07-06 02:17:58.607805 Processing 164 shares from 217.12.34.157:56040...

2015-07-06 02:20:06.389604 ... done processing 164 shares. New: 87 Have: 22084/~17280

2015-07-06 02:20:06.389918 Requesting parent share 51689bb2 from 149.210.228.165:41644
2015-07-06 02:20:06.395188 > Watchdog timer went off at:
2015-07-06 02:20:06.395285 >   File "/home/pool/p2pool_latest/run_p2pool.py", line 5, in <module>
2015-07-06 02:20:06.395344 >     main.run()
2015-07-06 02:20:06.395395 >   File "/home/pool/p2pool_latest/p2pool/main.py", line 666, in run
2015-07-06 02:20:06.395444 >     reactor.run()
2015-07-06 02:20:06.395492 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 02:20:06.395541 >     self.mainLoop()
2015-07-06 02:20:06.395589 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1204, in mainLoop
2015-07-06 02:20:06.395637 >     self.doIteration(t)
2015-07-06 02:20:06.395685 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/epollreactor.py", line 396, in doPoll
2015-07-06 02:20:06.395733 >     log.callWithLogger(selectable, _drdw, selectable, fd, event)
2015-07-06 02:20:06.395781 >   File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 88, in callWithLogger
2015-07-06 02:20:06.395829 >     return callWithContext({"system": lp}, func, *args, **kw)
2015-07-06 02:20:06.395877 >   File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 73, in callWithContext
2015-07-06 02:20:06.395924 >     return context.call({ILogContext: newCtx}, func, *args, **kw)
2015-07-06 02:20:06.395972 >   File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
2015-07-06 02:20:06.396019 >     return self.currentContext().callWithContext(ctx, func, *args, **kw)
2015-07-06 02:20:06.396067 >   File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
2015-07-06 02:20:06.396132 >     return func(*args,**kw)
2015-07-06 02:20:06.396181 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/posixbase.py", line 614, in _doReadOrWrite
2015-07-06 02:20:06.396240 >     why = selectable.doRead()
2015-07-06 02:20:06.396288 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 215, in doRead
2015-07-06 02:20:06.396336 >     return self._dataReceived(data)
2015-07-06 02:20:06.396382 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 221, in _dataReceived
2015-07-06 02:20:06.396428 >     rval = self.protocol.dataReceived(data)
2015-07-06 02:20:06.396475 >   File "/home/pool/p2pool_latest/p2pool/p2p.py", line 172, in new_dataReceived
2015-07-06 02:20:06.396521 >     old_dataReceived(data)
2015-07-06 02:20:06.396568 >   File "/home/pool/p2pool_latest/p2pool/util/p2protocol.py", line 27, in dataReceived
2015-07-06 02:20:06.396613 >     self.dataReceived2(data)
2015-07-06 02:20:06.396659 >   File "/home/pool/p2pool_latest/p2pool/util/datachunker.py", line 40, in _DataChunker
2015-07-06 02:20:06.396704 >     wants = receiver.send(buf.get(wants))
2015-07-06 02:20:06.396750 >   File "/home/pool/p2pool_latest/p2pool/util/p2protocol.py", line 57, in dataReceiver
2015-07-06 02:20:06.396796 >     self.packetReceived(command, type_.unpack(payload, self.ignore_trailing_payload))
2015-07-06 02:20:06.396842 >   File "/home/pool/p2pool_latest/p2pool/p2p.py", line 91, in packetReceived
2015-07-06 02:20:06.396887 >     p2protocol.Protocol.packetReceived(self, command, payload2)
2015-07-06 02:20:06.396933 >   File "/home/pool/p2pool_latest/p2pool/util/p2protocol.py", line 71, in packetReceived
2015-07-06 02:20:06.396979 >     handler(**payload2)
2015-07-06 02:20:06.397025 >   File "/home/pool/p2pool_latest/p2pool/p2p.py", line 366, in handle_sharereply
2015-07-06 02:20:06.397070 >     self.get_shares.got_response(id, res)
2015-07-06 02:20:06.397116 >   File "/home/pool/p2pool_latest/p2pool/util/deferral.py", line 124, in got_response
2015-07-06 02:20:06.397161 >     df.callback(resp)
2015-07-06 02:20:06.397207 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 382, in callback
2015-07-06 02:20:06.397253 >     self._startRunCallbacks(result)
2015-07-06 02:20:06.397300 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
2015-07-06 02:20:06.397346 >     self._runCallbacks()
2015-07-06 02:20:06.397392 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
2015-07-06 02:20:06.397438 >     current.result = callback(current.result, *args, **kw)
2015-07-06 02:20:06.397484 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
2015-07-06 02:20:06.397530 >     _inlineCallbacks(r, g, deferred)
2015-07-06 02:20:06.397576 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
2015-07-06 02:20:06.397623 >     result = g.send(result)
2015-07-06 02:20:06.397669 >   File "/home/pool/p2pool_latest/p2pool/node.py", line 135, in download_shares
2015-07-06 02:20:06.397716 >     self.handle_shares([(share, []) for share in shares], peer)
2015-07-06 02:20:06.397762 >   File "/home/pool/p2pool_latest/p2pool/node.py", line 48, in handle_shares
2015-07-06 02:20:06.397807 >     self.node.set_best_share()
2015-07-06 02:20:06.397852 >   File "/home/pool/p2pool_latest/p2pool/node.py", line 295, in set_best_share
2015-07-06 02:20:06.397899 >     best, desired, decorated_heads, bad_peer_addresses = self.tracker.think(self.get_height_rel_highest, self.bitcoind_work.value['previous_block'], self.bitcoind_work.value['bits'], self.known_txs_var.value)

2015-07-06 02:20:06.397950 >   File "/home/pool/p2pool_latest/p2pool/data.py", line 816, in think
2015-07-06 02:20:06.397996 >     if not self.attempt_verify(share):
2015-07-06 02:20:06.398043 >   File "/home/pool/p2pool_latest/p2pool/data.py", line 760, in attempt_verify
2015-07-06 02:20:06.398089 >     share.check(self)
2015-07-06 02:20:06.398134 >   File "/home/pool/p2pool_latest/p2pool/data.py", line 644, in check
2015-07-06 02:20:06.398191 >     other_tx_hashes = [tracker.items[tracker.get_nth_parent_hash(self.hash, share_count)].share_info['new_transaction_hashes'][tx_count] for share_count, tx_count in self.iter_transaction_hash_refs()]
2015-07-06 02:20:06.398250 >   File "/home/pool/p2pool_latest/p2pool/util/memoize.py", line 28, in b
2015-07-06 02:20:06.398299 >     res = f(*args)
2015-07-06 02:20:06.398346 >   File "/home/pool/p2pool_latest/p2pool/util/skiplist.py", line 25, in __call__
2015-07-06 02:20:06.398391 >     for i in xrange(skip_length):
2015-07-06 02:20:06.398437 >   File "/home/pool/p2pool_latest/p2pool/main.py", line 313, in <lambda>
2015-07-06 02:20:06.398483 >     sys.stderr.write, 'Watchdog timer went off at:\n' + ''.join(traceback.format_stack())
2015-07-06 02:20:06.416132 > ########################################
2015-07-06 02:20:06.416179 > >>> Warning: LOST CONTACT WITH BITCOIND for 2.4 minutes! Check that it isn't frozen or dead!
2015-07-06 02:20:06.416209 > ########################################
2015-07-06 02:20:06.416237 P2Pool: 17371 shares in chain (22084 verified/22084 total) Peers: 23 (13 incoming)
2015-07-06 02:20:06.416257  Local: 2690GH/s in last 10.0 minutes Local dead on arrival: ~1.0% (0-3%) Expected time to share: 1.5 hours
2015-07-06 02:20:06.416277  Shares: 3 (0 orphan, 0 dead) Stale rate: ~0.0% (0-57%) Efficiency: ~111.7% (48-112%) Current payout: (0.0035)=0.0035 BTC
2015-07-06 02:20:06.416301  Pool: 893TH/s Stale rate: 10.4% Expected time to block: 2.7 days
2015-07-06 02:20:06.632624 Connection timed out, disconnecting from 73.207.65.7:54652
2015-07-06 02:20:06.633329 Connection timed out, disconnecting from 85.172.106.54:42034
2015-07-06 02:20:06.633408 Connection timed out, disconnecting from 149.210.228.165:41644
2015-07-06 02:20:06.633623 Connection timed out, disconnecting from 101.100.174.138:56623
2015-07-06 02:20:06.633696 Connection timed out, disconnecting from 188.0.188.220:57817
2015-07-06 02:20:06.633768 Connection timed out, disconnecting from 94.42.115.50:9333
2015-07-06 02:20:06.633841 Connection timed out, disconnecting from 32.213.157.64:9333
2015-07-06 02:20:06.633909 Connection timed out, disconnecting from 219.138.161.162:9333
2015-07-06 02:20:06.633977 Connection timed out, disconnecting from 174.54.199.218:9333
2015-07-06 02:20:06.634043 Connection timed out, disconnecting from 114.44.155.126:64555
2015-07-06 02:20:06.634109 Connection timed out, disconnecting from 71.91.100.26:57844
2015-07-06 02:20:06.634175 Connection timed out, disconnecting from 52.24.128.249:33770
2015-07-06 02:20:06.634239 Connection timed out, disconnecting from 61.219.119.41:9333
2015-07-06 02:20:06.634305 Connection timed out, disconnecting from 74.82.244.100:9333
2015-07-06 02:20:06.634370 Connection timed out, disconnecting from 118.112.143.28:22493
2015-07-06 02:20:06.634436 Connection timed out, disconnecting from 209.195.4.74:9333
2015-07-06 02:20:06.634502 Connection timed out, disconnecting from 213.184.8.22:9333
2015-07-06 02:20:06.634566 Connection timed out, disconnecting from 80.60.85.169:9333
2015-07-06 02:20:06.634629 Connection timed out, disconnecting from 183.203.214.203:58445
2015-07-06 02:20:06.634694 Connection timed out, disconnecting from 183.203.214.203:38900
2015-07-06 02:20:06.634791 Connection timed out, disconnecting from 78.46.88.136:9333
2015-07-06 02:20:06.634857 Connection timed out, disconnecting from 176.28.50.57:32932
2015-07-06 02:20:06.634923 Connection timed out, disconnecting from 217.12.34.157:56040
2015-07-06 02:20:06.762316 > Error getting work from bitcoind:
2015-07-06 02:20:06.762371 > Traceback (most recent call last):
2015-07-06 02:20:06.762396 > Failure: twisted.internet.defer.TimeoutError: Getting http://127.0.0.1:8332/ took longer than 30 seconds.
2015-07-06 02:20:06.762585 Lost peer 74.82.244.100:9333 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.762737 Lost peer 209.195.4.74:9333 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.764194 Lost peer 73.207.65.7:54652 - Connection was aborted locally, using.
2015-07-06 02:20:06.764312 Lost peer 85.172.106.54:42034 - Connection was aborted locally, using.
2015-07-06 02:20:06.764429 Lost peer 149.210.228.165:41644 - Connection was aborted locally, using.
2015-07-06 02:20:06.764522 > in download_shares:
2015-07-06 02:20:06.764552 > Traceback (most recent call last):
2015-07-06 02:20:06.764576 > Failure: twisted.internet.error.ConnectionAborted: Connection was aborted locally, using.
2015-07-06 02:20:06.764695 Lost peer 101.100.174.138:56623 - Connection was aborted locally, using.
2015-07-06 02:20:06.764798 Lost peer 188.0.188.220:57817 - Connection was aborted locally, using.
2015-07-06 02:20:06.764911 Lost peer 94.42.115.50:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.765021 Lost peer 32.213.157.64:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.765130 Lost peer 219.138.161.162:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.765238 Lost peer 174.54.199.218:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.765339 Lost peer 114.44.155.126:64555 - Connection was aborted locally, using.
2015-07-06 02:20:06.765437 Lost peer 71.91.100.26:57844 - Connection was aborted locally, using.
2015-07-06 02:20:06.765531 Lost peer 52.24.128.249:33770 - Connection was aborted locally, using.
2015-07-06 02:20:06.765635 Lost peer 61.219.119.41:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.765746 Lost peer 118.112.143.28:22493 - Connection was aborted locally, using.
2015-07-06 02:20:06.765865 Lost peer 213.184.8.22:9333 - Connection was aborted locally, using.

2015-07-06 02:20:06.765971 Lost peer 80.60.85.169:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.766070 Lost peer 183.203.214.203:58445 - Connection was aborted locally, using.
2015-07-06 02:20:06.766174 Lost peer 183.203.214.203:38900 - Connection was aborted locally, using.
2015-07-06 02:20:06.766285 Lost peer 78.46.88.136:9333 - Connection was aborted locally, using.
2015-07-06 02:20:06.766401 Lost peer 176.28.50.57:32932 - Connection was aborted locally, using.
2015-07-06 02:20:06.766504 Lost peer 217.12.34.157:56040 - Connection was aborted locally, using.
2015-07-06 02:20:06.776051 Incoming connection to peer 61.161.171.210:54945 established. p2pool version: 1300 '13.4'
2015-07-06 02:20:06.834950 Lost peer 61.161.171.210:54945 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.835130 > in handle_share_hashes:
2015-07-06 02:20:06.835163 > Traceback (most recent call last):
2015-07-06 02:20:06.835187 > Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.835786 > Unhandled Error
2015-07-06 02:20:06.835816 > Traceback (most recent call last):
2015-07-06 02:20:06.835839 >   File "/home/pool/p2pool_latest/p2pool/main.py", line 666, in run
2015-07-06 02:20:06.835862 >     reactor.run()
2015-07-06 02:20:06.835883 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 02:20:06.835905 >     self.mainLoop()
2015-07-06 02:20:06.835926 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 02:20:06.835947 >     self.runUntilCurrent()
2015-07-06 02:20:06.835968 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 02:20:06.835989 >     call.func(*call.args, **call.kw)
2015-07-06 02:20:06.836009 > --- <exception caught here> ---
2015-07-06 02:20:06.836030 >   File "/home/pool/p2pool_latest/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 02:20:06.836051 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 02:20:06.836073 >   File "/home/pool/p2pool_latest/p2pool/work.py", line 209, in preprocess_request
2015-07-06 02:20:06.836094 >     raise jsonrpc.Error_for_code(-12345)(u'p2pool is not connected to any peers')
2015-07-06 02:20:06.836129 > p2pool.util.jsonrpc.NarrowError: -12345 p2pool is not connected to any peers
2015-07-06 02:20:06.911461 Incoming connection to peer 108.61.173.129:36676 established. p2pool version: 1400 '13.4-67-gbcd9a50-dirty'
2015-07-06 02:20:06.991277 Lost peer 108.61.173.129:36676 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.991444 > in handle_share_hashes:
2015-07-06 02:20:06.991476 > Traceback (most recent call last):
2015-07-06 02:20:06.991500 > Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:06.991820 Incoming connection to peer 108.24.83.254:60387 established. p2pool version: 1400 '14.0'
2015-07-06 02:20:07.042605 Lost peer 108.24.83.254:60387 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.043195 Incoming connection to peer 92.3.46.0:48284 established. p2pool version: 1400 '14.0-5-gb07b1e3'
2015-07-06 02:20:07.094066 Lost peer 92.3.46.0:48284 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.094234 > in handle_share_hashes:
2015-07-06 02:20:07.094265 > Traceback (most recent call last):
2015-07-06 02:20:07.094289 > Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.094607 Incoming connection to peer 212.74.236.210:58625 established. p2pool version: 1400 '14.0-5-gb07b1e3-dirty'
2015-07-06 02:20:07.145878 Lost peer 212.74.236.210:58625 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.146046 > in handle_share_hashes:
2015-07-06 02:20:07.146078 > Traceback (most recent call last):
2015-07-06 02:20:07.146102 > Failure: twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.146438 Incoming connection to peer 218.174.205.110:58599 established. p2pool version: 1400 '14.0'
2015-07-06 02:20:07.197178 Lost peer 218.174.205.110:58599 - Connection to the other side was lost in a non-clean fashion.
2015-07-06 02:20:07.223252 > Unhandled Error
2015-07-06 02:20:07.223324 > Traceback (most recent call last):
2015-07-06 02:20:07.223350 >   File "/home/pool/p2pool_latest/p2pool/main.py", line 666, in run
2015-07-06 02:20:07.223373 >     reactor.run()
2015-07-06 02:20:07.223394 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 02:20:07.223416 >     self.mainLoop()
2015-07-06 02:20:07.223438 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 02:20:07.223459 >     self.runUntilCurrent()
2015-07-06 02:20:07.223481 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 02:20:07.223504 >     call.func(*call.args, **call.kw)
2015-07-06 02:20:07.223525 > --- <exception caught here> ---
2015-07-06 02:20:07.223546 >   File "/home/pool/p2pool_latest/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 02:20:07.223568 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 02:20:07.223590 >   File "/home/pool/p2pool_latest/p2pool/work.py", line 209, in preprocess_request
2015-07-06 02:20:07.223611 >     raise jsonrpc.Error_for_code(-12345)(u'p2pool is not connected to any peers')
2015-07-06 02:20:07.223632 > p2pool.util.jsonrpc.NarrowError: -12345 p2pool is not connected to any peers
2015-07-06 02:20:09.420178 > ########################################
2015-07-06 02:20:09.420308 > >>> Warning: LOST CONTACT WITH BITCOIND for 2.4 minutes! Check that it isn't frozen or dead!
2015-07-06 02:20:09.420379 > ########################################

P2Pool Version: 14.0-5-gb07b1e3

[forrestv]

Please try the latest commit. It doesn't allow connections to v13 nodes.

[iSOcH]

edit: ok, i just updated to 14.0-6-g18b5ad5 (was using an early 14.0 version before)

i'm seeing the same problem, and think its getting worse

Code:

2015-07-06 09:10:52.693091 P2Pool: 17361 shares in chain (17365 verified/22000 total) Peers: 15 (9 incoming)

there are a lot of "requesting parent shares", many of them for the same share

Code:

grep -i "requesting parent" /var/cache/bitcoin/p2pool/bitcoin/log | grep 2015-07-06 | wc -l
3136

comparision to yesterday:

Code:

grep -i "requesting parent" /var/cache/bitcoin/p2pool/bitcoin/log | grep 2015-07-05 | wc -l
237

my node is sending and receiving a lot of shares:

Code:

2015-07-06 08:57:43.262967 Sending 432 shares to 217.12.34.158:46568
2015-07-06 08:57:44.981311 Sending 239 shares to 66.75.110.131:52309
2015-07-06 08:57:46.475401 Sending 357 shares to 84.10.54.202:60030
2015-07-06 08:57:47.830332 Sending 354 shares to 81.169.205.215:52083
2015-07-06 08:57:48.235190 Sending 390 shares to 81.169.205.215:52083
2015-07-06 08:57:53.401643 Sending 194 shares to 217.12.34.158:46568
2015-07-06 08:57:54.751672 Sending 320 shares to 81.169.205.215:52083
2015-07-06 08:57:57.688322 Sending 99 shares to 81.169.205.215:52083
2015-07-06 08:57:59.785758 Sending 39 shares to 207.105.189.40:9333
2015-07-06 08:58:02.377113 Sending 1 shares to 167.88.213.138:61696
2015-07-06 08:58:03.077039 Sending 87 shares to 87.78.137.229:2623
2015-07-06 08:58:04.886128 Sending 56 shares to 217.12.34.158:46568
2015-07-06 08:58:07.186807 Sending 125 shares to 66.75.110.131:52309
2015-07-06 08:58:07.707483 Sending 473 shares to 217.12.34.158:46568
2015-07-06 08:58:13.227003 Sending 141 shares to 81.169.205.215:52083
2015-07-06 08:58:22.045891 Sending 148 shares to 207.105.189.40:9333
2015-07-06 08:58:23.822276 Sending 103 shares to 217.12.34.158:46568
2015-07-06 08:58:24.533633 Sending 249 shares to 217.12.34.158:46568
2015-07-06 08:58:27.799458 Sending 284 shares to 81.169.205.215:52083
2015-07-06 08:58:28.456767 Sending 208 shares to 87.78.137.229:2623
2015-07-06 08:58:33.534560 Sending 384 shares to 217.12.34.158:46568
2015-07-06 08:58:36.042705 Sending 238 shares to 207.105.189.40:9333
2015-07-06 08:58:38.699096 Sending 32 shares to 207.105.189.40:9333
2015-07-06 08:58:45.645699 Sending 371 shares to 217.12.34.158:46568
2015-07-06 08:58:46.294769 Sending 23 shares to 87.78.137.229:2623
2015-07-06 08:58:50.296814 Sending 453 shares to 81.169.205.215:52083
2015-07-06 08:59:05.330462 Sending 358 shares to 81.169.205.215:52083
2015-07-06 08:59:13.372054 Sending 53 shares to 84.10.54.202:60030
2015-07-06 08:59:15.469112 Sending 111 shares to 81.169.205.215:52083
2015-07-06 09:02:13.983653 Sending 1 shares to 71.12.171.6:9333
2015-07-06 09:02:22.237668 Sending 1 shares to 114.32.105.215:9333
2015-07-06 09:02:54.547177 Sending 8 shares to 86.121.82.232:48978
2015-07-06 09:03:04.895471 Sending 442 shares to 86.121.82.232:48978
2015-07-06 09:03:59.491288 Sending 384 shares to 121.121.100.90:60738
2015-07-06 09:04:11.688657 Sending 161 shares to 121.121.100.90:60738
2015-07-06 09:04:15.047931 Sending 74 shares to 76.182.12.77:9333
2015-07-06 09:04:26.285530 Sending 337 shares to 91.220.223.6:24686
2015-07-06 09:04:35.717035 Sending 175 shares to 91.220.223.6:24686
2015-07-06 09:04:35.877523 Sending 50 shares to 121.121.100.90:60738
2015-07-06 09:04:44.011146 Sending 457 shares to 121.121.100.90:60738
2015-07-06 09:05:08.689059 Sending 20 shares to 91.220.223.6:24686
2015-07-06 09:05:17.471923 Sending 311 shares to 121.121.100.90:60738
2015-07-06 09:05:18.272154 Sending 1 shares to 46.255.7.139:9333
2015-07-06 09:05:21.415614 Sending 318 shares to 91.220.223.6:24686
2015-07-06 09:05:23.649835 Sending 152 shares to 68.0.31.35:53198
2015-07-06 09:05:24.411105 Sending 1 shares to 205.206.218.175:45602
2015-07-06 09:05:24.753467 Sending 164 shares to 121.121.100.90:60738
2015-07-06 09:05:26.774989 Sending 6 shares to 205.206.218.175:45602
2015-07-06 09:05:29.430412 Sending 388 shares to 121.121.100.90:60738
2015-07-06 09:05:39.312566 Sending 462 shares to 91.220.223.6:24686
2015-07-06 09:05:39.918280 Sending 109 shares to 91.220.223.6:24686
2015-07-06 09:05:42.200108 Sending 144 shares to 91.220.223.6:24686
2015-07-06 09:05:45.093282 Sending 1 shares to 205.206.218.175:45602
2015-07-06 09:05:46.982847 Sending 229 shares to 91.220.223.6:24686
2015-07-06 09:06:00.151759 Sending 103 shares to 91.220.223.6:24686
2015-07-06 09:06:00.396437 Sending 221 shares to 205.206.218.175:45602
2015-07-06 09:06:04.071787 Sending 260 shares to 205.206.218.175:45602
2015-07-06 09:06:13.482495 Sending 278 shares to 210.54.35.194:60303
2015-07-06 09:06:14.652301 Sending 380 shares to 97.124.189.139:9333
2015-07-06 09:06:15.706650 Sending 196 shares to 91.220.223.6:24686
2015-07-06 09:06:18.408067 Sending 49 shares to 210.54.35.194:60303
2015-07-06 09:06:20.579338 Sending 330 shares to 205.206.218.175:45602
2015-07-06 09:06:26.114314 Sending 321 shares to 68.0.31.35:53198
2015-07-06 09:06:36.371023 Sending 200 shares to 97.124.189.139:9333
2015-07-06 09:06:44.308073 Sending 109 shares to 68.0.31.35:53198
2015-07-06 09:06:44.515419 Sending 247 shares to 205.206.218.175:45602
2015-07-06 09:06:48.962999 Sending 104 shares to 68.0.31.35:53198
2015-07-06 09:06:50.953352 Sending 446 shares to 210.54.35.194:60303
2015-07-06 09:06:56.667450 Sending 384 shares to 205.206.218.175:45602
2015-07-06 09:06:58.010280 Sending 238 shares to 205.206.218.175:45602
2015-07-06 09:06:59.215586 Sending 75 shares to 210.54.35.194:60303
2015-07-06 09:06:59.574578 Sending 244 shares to 97.124.189.139:9333
2015-07-06 09:07:19.492058 Sending 396 shares to 68.0.31.35:53198
2015-07-06 09:07:43.266797 Sending 431 shares to 210.54.35.194:60303
2015-07-06 09:08:21.063731 Sending 277 shares to 210.54.35.194:60303
2015-07-06 09:09:00.669260 Sending 307 shares to 68.0.31.35:53198
2015-07-06 09:09:03.103442 Sending 133 shares to 77.120.40.53:34304
2015-07-06 09:09:05.315031 Sending 196 shares to 77.120.40.53:34304
2015-07-06 09:09:14.829781 Sending 79 shares to 205.206.218.175:45602
2015-07-06 09:09:16.468476 Sending 106 shares to 205.206.218.175:45602
2015-07-06 09:09:21.670862 Sending 209 shares to 68.0.31.35:53198
2015-07-06 09:09:26.785464 Sending 229 shares to 77.120.40.53:34304
2015-07-06 09:09:31.660826 Sending 164 shares to 77.120.40.53:34304
2015-07-06 09:09:33.344424 Sending 118 shares to 77.120.40.53:34304
2015-07-06 09:09:41.977894 Sending 181 shares to 72.74.133.29:9333
2015-07-06 09:09:50.659425 Sending 84 shares to 158.255.89.244:9333
2015-07-06 09:09:53.836714 Sending 5 shares to 85.23.230.255:56026
2015-07-06 09:10:03.962227 Sending 442 shares to 85.23.230.255:56026
2015-07-06 09:10:05.703970 Sending 21 shares to 96.49.37.162:59982
2015-07-06 09:10:12.832631 Sending 275 shares to 96.49.37.162:59982
2015-07-06 09:10:19.013148 Sending 77 shares to 96.49.37.162:59982
2015-07-06 09:10:44.225691 Sending 237 shares to 210.54.35.194:60303
2015-07-06 09:10:51.302060 Sending 116 shares to 158.255.89.244:9333
2015-07-06 09:10:56.587289 Sending 56 shares to 68.0.31.35:53198
2015-07-06 09:11:02.556936 Sending 174 shares to 77.120.40.53:34304
2015-07-06 09:11:05.365188 Sending 60 shares to 77.120.40.53:34304
2015-07-06 09:11:07.470353 Sending 461 shares to 72.74.133.29:9333
2015-07-06 09:11:28.388102 Sending 10 shares to 85.23.230.255:56026
2015-07-06 09:11:39.291442 Sending 106 shares to 97.124.189.139:9333
2015-07-06 09:11:50.207824 Sending 95 shares to 85.23.230.255:56026
2015-07-06 09:12:06.716451 Sending 114 shares to 97.124.189.139:9333
2015-07-06 09:12:33.315635 Sending 74 shares to 72.74.133.29:9333

Code:

2015-07-06 07:57:42.506601 Processing 459 shares from 91.220.223.6:4058...
2015-07-06 07:57:48.218828 Processing 102 shares from 91.220.223.6:4058...
2015-07-06 07:58:04.966723 Processing 134 shares from 70.117.102.125:17868...
2015-07-06 07:58:09.559901 Processing 307 shares from 70.117.102.125:17868...
2015-07-06 07:58:14.545549 Processing 113 shares from 92.13.96.93:9333...
2015-07-06 07:58:35.650643 Processing 211 shares from 70.162.133.247:52496...
2015-07-06 07:58:39.554653 Processing 53 shares from 85.23.230.255:49751...
2015-07-06 07:58:40.493792 Processing 125 shares from 91.220.223.6:4058...
2015-07-06 07:58:42.599736 Processing 172 shares from 108.24.83.254:9333...
2015-07-06 07:58:58.971888 Processing 92 shares from 70.162.133.247:52496...
2015-07-06 07:59:44.083968 Processing 185 shares from 70.162.133.247:52496...
2015-07-06 07:59:48.727710 Processing 364 shares from 54.235.94.19:50976...
2015-07-06 08:02:44.458872 Processing 6 shares from 65.78.62.173:53147...
2015-07-06 08:03:23.763359 Processing 225 shares from 220.233.178.199:9333...
2015-07-06 08:03:25.056965 Processing 351 shares from 108.61.173.129:9333...
2015-07-06 08:03:32.789686 Processing 92 shares from 158.255.89.244:9333...
2015-07-06 08:03:33.192410 Processing 22 shares from 174.54.199.218:9333...
2015-07-06 08:03:36.071631 Processing 148 shares from 108.61.173.129:9333...
2015-07-06 08:03:37.450641 Processing 55 shares from 174.54.199.218:9333...
2015-07-06 08:03:37.595077 Processing 54 shares from 84.200.24.186:9333...
2015-07-06 08:03:42.535761 Processing 132 shares from 84.200.24.186:9333...
2015-07-06 08:03:50.493665 Processing 262 shares from 65.78.62.173:53147...
2015-07-06 08:03:51.128255 Processing 15 shares from 65.78.62.173:53147...
2015-07-06 08:03:52.541389 Processing 199 shares from 84.200.24.186:9333...
2015-07-06 08:03:53.356317 Processing 123 shares from 84.200.24.186:9333...
2015-07-06 08:04:05.079732 Processing 64 shares from 174.54.199.218:9333...
2015-07-06 08:04:17.858134 Processing 126 shares from 158.255.89.244:9333...
2015-07-06 08:04:23.291073 Processing 42 shares from 174.54.199.218:9333...
2015-07-06 08:04:36.733671 Processing 179 shares from 174.54.199.218:9333...
2015-07-06 08:04:39.076905 Processing 20 shares from 178.252.84.209:61515...
2015-07-06 08:04:47.629434 Processing 183 shares from 65.78.62.173:53147...
2015-07-06 08:05:01.093402 Processing 183 shares from 84.200.24.186:9333...
2015-07-06 08:05:11.489603 Processing 198 shares from 174.54.199.218:9333...
2015-07-06 08:05:15.244884 Processing 262 shares from 65.78.62.173:53147...
2015-07-06 08:05:16.122226 Processing 194 shares from 174.54.199.218:9333...
2015-07-06 08:05:17.559818 Processing 139 shares from 158.255.89.244:9333...
2015-07-06 08:05:21.823694 Processing 248 shares from 158.255.89.244:9333...
2015-07-06 08:05:24.904440 Processing 318 shares from 65.78.62.173:53147...
2015-07-06 08:05:26.918100 Processing 329 shares from 84.200.24.186:9333...
2015-07-06 08:05:44.829798 Processing 195 shares from 84.200.24.186:9333...
2015-07-06 08:05:59.164161 Processing 23 shares from 65.78.62.173:53147...
2015-07-06 08:06:41.127412 Processing 32 shares from 176.31.181.71:48567...
2015-07-06 08:07:15.135155 Processing 148 shares from 176.31.181.71:48567...
2015-07-06 08:07:48.037788 Processing 170 shares from 174.54.199.218:9333...
2015-07-06 08:11:48.094273 Processing 6 shares from 64.232.216.213:9333...
2015-07-06 08:11:51.416881 Processing 122 shares from 64.232.216.213:9333...
2015-07-06 08:11:52.668809 Processing 34 shares from 64.232.216.213:9333...
2015-07-06 08:11:58.094806 Processing 309 shares from 64.232.216.213:9333...
2015-07-06 08:12:09.200667 Processing 103 shares from 54.235.94.19:51332...
2015-07-06 08:12:11.373251 Processing 347 shares from 64.232.216.213:9333...
2015-07-06 08:12:20.238103 Processing 212 shares from 54.235.94.19:51332...
2015-07-06 08:12:21.503887 Processing 70 shares from 64.232.216.213:9333...
2015-07-06 08:13:17.269496 Processing 85 shares from 219.138.161.162:9333...
2015-07-06 08:13:41.564108 Processing 120 shares from 82.200.205.30:45864...
2015-07-06 08:14:22.877583 Processing 196 shares from 82.200.205.30:45864...
2015-07-06 08:14:24.591300 Processing 156 shares from 64.232.216.213:9333...
2015-07-06 08:14:35.614936 Processing 130 shares from 82.200.205.30:45864...
2015-07-06 08:14:42.257222 Processing 304 shares from 64.232.216.213:9333...
2015-07-06 08:15:01.130177 Processing 33 shares from 64.232.216.213:9333...
2015-07-06 08:15:07.632273 Processing 151 shares from 82.200.205.30:45864...
2015-07-06 08:15:10.787820 Processing 210 shares from 64.232.216.213:9333...
2015-07-06 08:15:16.095788 Processing 52 shares from 82.200.205.30:45864...
2015-07-06 08:15:19.023708 Processing 42 shares from 64.232.216.213:9333...
2015-07-06 08:16:14.770233 Processing 104 shares from 64.232.216.213:9333...
2015-07-06 08:16:16.571101 Processing 235 shares from 173.171.178.225:53628...
2015-07-06 08:16:22.463147 Processing 93 shares from 5.189.148.57:58544...
2015-07-06 08:16:28.740375 Processing 164 shares from 64.232.216.213:9333...
2015-07-06 08:16:41.650375 Processing 352 shares from 64.232.216.213:9333...
2015-07-06 08:16:42.710769 Processing 113 shares from 173.171.178.225:53628...
2015-07-06 08:16:43.425153 Processing 228 shares from 5.189.148.57:58544...
2015-07-06 08:16:43.790136 Processing 113 shares from 5.189.148.57:58544...
2015-07-06 08:16:44.962483 Processing 103 shares from 64.232.216.213:9333...
2015-07-06 08:17:16.807264 Processing 19 shares from 176.28.50.57:39331...
2015-07-06 08:17:23.856513 Processing 52 shares from 121.44.22.78:17729...
2015-07-06 08:17:26.417531 Processing 226 shares from 176.28.50.57:39331...
2015-07-06 08:17:29.118916 Processing 51 shares from 176.28.50.57:39331...
2015-07-06 08:17:49.659536 Processing 340 shares from 64.232.216.213:9333...
2015-07-06 08:20:34.687452 Processing 8 shares from 212.74.236.210:29333...
2015-07-06 08:20:40.157627 Processing 65 shares from 212.74.236.210:29333...
2015-07-06 08:20:45.098282 Processing 194 shares from 121.121.100.90:9333...
2015-07-06 08:20:46.571868 Processing 113 shares from 64.232.216.213:9333...
2015-07-06 08:20:48.898717 Processing 75 shares from 121.121.100.90:9333...
2015-07-06 08:20:50.384662 Processing 296 shares from 108.24.83.254:9333...
2015-07-06 08:20:51.813328 Processing 52 shares from 121.121.100.90:9333...
2015-07-06 08:20:56.950547 Processing 214 shares from 64.232.216.213:9333...
2015-07-06 08:20:59.391263 Processing 16 shares from 108.24.83.254:9333...
2015-07-06 08:21:06.848067 Processing 100 shares from 108.24.83.254:9333...
2015-07-06 08:21:14.868228 Processing 140 shares from 121.121.100.90:9333...
2015-07-06 08:21:16.969600 Processing 247 shares from 64.232.216.213:9333...
2015-07-06 08:21:17.479374 Processing 47 shares from 108.24.83.254:9333...
2015-07-06 08:21:33.340166 Processing 276 shares from 72.241.129.31:9333...
2015-07-06 08:21:36.561397 Processing 41 shares from 197.231.221.211:21910...
2015-07-06 08:21:39.907075 Processing 57 shares from 72.241.129.31:9333...
2015-07-06 08:21:47.622789 Processing 67 shares from 64.232.216.213:9333...
2015-07-06 08:21:53.802754 Processing 187 shares from 108.24.83.254:9333...
2015-07-06 08:22:12.216464 Processing 268 shares from 121.121.100.90:9333...
2015-07-06 08:22:18.944815 Processing 272 shares from 197.231.221.211:21910...
2015-07-06 08:22:28.880690 Processing 413 shares from 72.241.129.31:9333...
2015-07-06 08:22:49.089047 Processing 326 shares from 72.241.129.31:9333...
2015-07-06 08:23:12.468568 Processing 221 shares from 108.24.83.254:9333...
2015-07-06 08:23:19.223551 Processing 354 shares from 72.241.129.31:9333...
2015-07-06 08:23:22.191334 Processing 327 shares from 64.232.216.213:9333...
2015-07-06 08:23:27.721243 Processing 201 shares from 197.231.221.211:21910...
2015-07-06 08:23:34.096120 Processing 62 shares from 197.231.221.211:21910...
2015-07-06 08:23:40.648275 Processing 217 shares from 64.232.216.213:9333...
2015-07-06 08:26:31.745862 Processing 140 shares from 212.74.236.210:9333...
2015-07-06 08:27:47.266766 Processing 106 shares from 212.74.236.210:6146...
2015-07-06 08:27:53.039278 Processing 41 shares from 212.74.236.210:6146...
2015-07-06 08:28:01.028543 Processing 49 shares from 61.219.120.109:59419...
2015-07-06 08:28:04.134441 Processing 18 shares from 61.219.120.109:59419...
2015-07-06 08:28:05.050910 Processing 144 shares from 212.74.236.210:6146...
2015-07-06 08:28:09.214933 Processing 92 shares from 61.219.120.109:59419...
2015-07-06 08:28:10.445878 Processing 38 shares from 212.74.236.210:6146...
2015-07-06 08:28:11.302566 Processing 127 shares from 212.74.236.210:6146...
2015-07-06 08:28:27.035481 Processing 272 shares from 212.74.236.210:6146...
2015-07-06 08:28:28.282489 Processing 58 shares from 24.155.164.78:3334...
2015-07-06 08:28:29.345884 Processing 91 shares from 24.155.164.78:3334...
2015-07-06 08:28:32.716613 Processing 112 shares from 193.24.30.38:55836...
2015-07-06 08:29:50.536581 Processing 138 shares from 193.24.30.38:55836...
2015-07-06 08:30:03.985364 Processing 160 shares from 87.173.17.22:2010...
2015-07-06 08:30:16.953490 Processing 34 shares from 193.24.30.38:55836...
2015-07-06 08:30:29.904574 Processing 197 shares from 87.173.17.22:2010...
2015-07-06 08:30:39.899862 Processing 180 shares from 87.173.17.22:2010...
2015-07-06 08:31:04.155039 Processing 272 shares from 87.173.17.22:2010...
2015-07-06 08:31:13.691855 Processing 59 shares from 87.173.17.22:2010...
2015-07-06 08:31:32.469704 Processing 101 shares from 87.173.17.22:2010...
2015-07-06 08:31:45.141423 Processing 233 shares from 96.44.189.102:19199...
2015-07-06 08:31:50.906108 Processing 150 shares from 96.44.189.102:19199...
2015-07-06 08:32:05.607790 Processing 16 shares from 87.173.17.22:2010...
2015-07-06 08:32:17.327780 Processing 192 shares from 87.173.17.22:2010...
2015-07-06 08:32:27.330987 Processing 460 shares from 87.173.17.22:2010...
2015-07-06 08:32:37.655321 Processing 97 shares from 87.173.17.22:2010...
2015-07-06 08:32:42.587409 Processing 436 shares from 96.44.189.102:19199...
2015-07-06 08:32:45.437810 Processing 25 shares from 87.173.17.22:2010...
2015-07-06 08:33:04.604113 Processing 303 shares from 87.173.17.22:2010...
2015-07-06 08:33:07.563523 Processing 112 shares from 87.173.17.22:2010...
2015-07-06 08:33:57.170538 Processing 72 shares from 187.190.232.123:9333...
2015-07-06 08:34:03.104248 Processing 86 shares from 95.170.95.133:9333...
2015-07-06 08:34:36.459475 Processing 141 shares from 96.44.189.102:19199...
2015-07-06 08:50:50.438367 Processing 137 shares from 207.105.189.40:9333...
2015-07-06 08:50:59.074901 Processing 172 shares from 124.149.66.255:59002...
2015-07-06 08:51:22.968728 Processing 273 shares from 176.28.50.57:10333...
2015-07-06 08:51:25.579870 Processing 242 shares from 82.200.205.30:54028...
2015-07-06 08:51:27.616058 Processing 88 shares from 108.31.56.9:45095...
2015-07-06 08:51:27.822882 Processing 17 shares from 108.31.56.9:45095...
2015-07-06 08:51:31.588282 Processing 82 shares from 108.31.56.9:45095...
2015-07-06 08:52:27.691674 Processing 112 shares from 108.31.56.9:45095...
2015-07-06 08:52:50.961663 Processing 94 shares from 62.153.208.26:58558...
2015-07-06 08:52:58.783569 Processing 218 shares from 207.105.189.40:9333...
2015-07-06 08:53:17.876460 Processing 13 shares from 82.200.205.30:54028...
2015-07-06 08:53:23.630192 Processing 109 shares from 98.225.44.138:60123...
2015-07-06 08:53:25.226613 Processing 83 shares from 98.225.44.138:60123...
2015-07-06 08:53:25.640869 Processing 74 shares from 176.28.50.57:10333...
2015-07-06 08:53:28.766297 Processing 130 shares from 62.153.208.26:58558...
2015-07-06 08:53:32.820625 Processing 136 shares from 108.31.56.9:45095...
2015-07-06 08:53:49.677174 Processing 90 shares from 108.31.56.9:45095...
2015-07-06 08:53:53.416617 Processing 220 shares from 207.105.189.40:9333...
2015-07-06 08:53:59.879027 Processing 204 shares from 176.28.50.57:10333...
2015-07-06 08:54:04.096971 Processing 142 shares from 82.200.205.30:54028...
2015-07-06 08:54:09.045677 Processing 196 shares from 82.200.205.30:54028...
2015-07-06 08:54:13.271712 Processing 241 shares from 108.31.56.9:45095...
2015-07-06 08:54:14.709749 Processing 12 shares from 176.28.50.57:10333...
2015-07-06 08:54:18.432235 Processing 32 shares from 82.200.205.30:54028...
2015-07-06 08:55:13.442662 Processing 33 shares from 173.220.42.2:9333...
2015-07-06 08:55:25.198630 Processing 156 shares from 173.220.42.2:9333...
2015-07-06 08:55:48.831844 Processing 403 shares from 71.91.100.26:48014...
2015-07-06 08:56:10.149369 Processing 49 shares from 87.78.137.229:2623...
2015-07-06 08:56:22.522882 Processing 191 shares from 71.91.100.26:48014...
2015-07-06 08:56:39.266115 Processing 239 shares from 71.91.100.26:48014...
2015-07-06 08:56:58.239486 Processing 75 shares from 96.49.37.162:59606...
2015-07-06 08:57:24.251357 Processing 11 shares from 173.220.42.2:9333...
2015-07-06 08:57:42.829246 Processing 151 shares from 173.220.42.2:9333...
2015-07-06 08:58:35.670248 Processing 23 shares from 173.220.42.2:9333...
2015-07-06 08:58:54.793671 Processing 128 shares from 84.10.54.202:60030...
2015-07-06 08:59:03.893737 Processing 53 shares from 173.220.42.2:9333...
2015-07-06 08:59:16.297736 Processing 321 shares from 173.220.42.2:9333...
2015-07-06 09:02:49.146485 Processing 202 shares from 69.164.211.28:9333...
2015-07-06 09:02:55.644752 Processing 239 shares from 76.182.12.77:9333...
2015-07-06 09:02:58.478596 Processing 61 shares from 76.182.12.77:9333...
2015-07-06 09:02:59.477199 Processing 144 shares from 69.164.211.28:9333...
2015-07-06 09:03:01.435703 Processing 150 shares from 76.182.12.77:9333...
2015-07-06 09:03:05.698778 Processing 62 shares from 174.54.199.218:9333...
2015-07-06 09:03:17.599615 Processing 274 shares from 193.24.30.38:9333...
2015-07-06 09:03:23.464968 Processing 27 shares from 69.164.211.28:9333...
2015-07-06 09:03:27.512256 Processing 176 shares from 69.164.211.28:9333...
2015-07-06 09:03:38.755387 Processing 252 shares from 76.182.12.77:9333...
2015-07-06 09:03:42.077654 Processing 132 shares from 69.164.211.28:9333...
2015-07-06 09:03:45.897913 Processing 64 shares from 174.54.199.218:9333...
2015-07-06 09:03:51.784305 Processing 355 shares from 193.24.30.38:9333...
2015-07-06 09:04:13.053052 Processing 64 shares from 174.54.199.218:9333...
2015-07-06 09:04:29.121729 Processing 187 shares from 193.24.30.38:9333...
2015-07-06 09:04:31.405737 Processing 15 shares from 193.24.30.38:9333...
2015-07-06 09:05:00.921468 Processing 86 shares from 76.182.12.77:9333...
2015-07-06 09:06:02.286723 Processing 139 shares from 76.182.12.77:9333...
2015-07-06 09:06:09.275726 Processing 309 shares from 76.182.12.77:9333...
2015-07-06 09:07:03.459177 Processing 459 shares from 181.45.147.193:60891...
2015-07-06 09:07:19.240933 Processing 319 shares from 181.45.147.193:60891...
2015-07-06 09:07:25.668482 Processing 99 shares from 205.206.218.175:45602...
2015-07-06 09:07:43.826698 Processing 26 shares from 121.121.100.90:60738...
2015-07-06 09:08:14.128023 Processing 224 shares from 205.206.218.175:45602...
2015-07-06 09:08:20.827825 Processing 323 shares from 121.121.100.90:60738...
2015-07-06 09:08:52.279143 Processing 236 shares from 121.121.100.90:60738...

[squidicuz]

Please try the latest commit. It doesn't allow connections to v13 nodes.

Looks like that fixes it. Right out of boot it banned the version 1300 nodes it connected to first, then connected to the version 1400 nodes, and is now no longer getting stuck.

Thanks!

[p3yot33at3r]

Please try the latest commit. It doesn't allow connections to v13 nodes.

Looks like that fixes it. Right out of boot it banned the version 1300 nodes it connected to first, then connected to the version 1400 nodes, and is now no longer getting stuck.

Thanks!

Yup - that's seems to have done it for me too. Looking at my graphs & logs, it looks like my node has been trying to download & process invalid shares all night:



..causing this:



Recommend everyone update again asap so as to avoid this happening to them.

Thanks forrestv for the quick response & fix! 

[jtoomim]

Does anybody know what is going on with P2P today with the high latency? It is causing my bitcoind to drop connectivity and my SP's to drop as well (the S5 seems to drop as well looking at my node). It has been up/down for the last six hours. I looked at other pools and some seem to have the same problem as well. I am running the current 13.4-83-gb07b1e3 P2P version and 0.10.1 core.

There was a hard fork about a week ago. v13 nodes are no longer making any revenue. You need to update p2pool immediately.

Furthermore, there was a second bug that began over the last day that causes shares submitted by v13 nodes to be sent and reprocessed repeatedly due to the hard fork. The newest version solves the issue.

[windpath]

My node is running the previous version 14 (before version 13 shares were excluded) and performance is suffering, I will be able to update this afternoon...

[jonnybravo0311]

Please try the latest commit. It doesn't allow connections to v13 nodes.

Thanks forrestv.  That appears to have fixed the issue for me as well.  I was noticing a ton of disconnects and my miners failing over to backup pools.  So far with the update, my miners seem to be sticking on my node, and I'm not seeing the constant share traffic in the logs.

[p3yot33at3r]

Has anyone else started getting bitcoind disconnects since upgrading to the latest version?

[Duce]

Has anyone else started getting bitcoind disconnects since upgrading to the latest version?

Yes, and I went from 10.1 to 10.2 to ensure it was not a Bitcoind version issue. I will not be able to update my node until later today though to see if this solves this issue as well. I hope it does.

[p3yot33at3r]

Has anyone else started getting bitcoind disconnects since upgrading to the latest version?

Yes, and I went from 10.1 to 10.2 to ensure it was not a Bitcoind version issue. I will not be able to update my node until later today though to see if this solves this issue as well. I hope it does.

Right. I also just noticed the eu relaynetwork was down - so I've disabled it (the latest version didn't work on 14.04 anyway) just in case that had something to do with it.

[jonnybravo0311]

I just started noticing this behavior on my node as well.  I was about to reply that I wasn't seeing it, then as soon as I started typing I got the "lost contact with bitcoind" errors flying through my p2pool logs.  Also, I'm noticing the same with the auxblock calls for NMC merge mining.  Not sure what's going on here.

Log snippet from NMC error:

Code:

2015-07-06 12:22:36.459226 > Error while calling merged getauxblock on http://127.0.0.1:7333:
2015-07-06 12:22:36.459364 > Traceback (most recent call last):
2015-07-06 12:22:36.459425 > Failure: twisted.internet.defer.TimeoutError: Getting http://127.0.0.1:7333 took longer than 5 seconds.

Log snippet showing lost contact with bitcoind:

Code:

2015-07-06 12:23:39.257837 Error when requesting noncached value:
2015-07-06 12:23:39.258283 > Traceback (most recent call last):
2015-07-06 12:23:39.258417 > Failure: twisted.internet.defer.TimeoutError: Getting http://127.0.0.1:8332/ took longer than 30 seconds.
2015-07-06 12:23:39.258510 
2015-07-06 12:23:39.259131 > Error getting work from bitcoind:
2015-07-06 12:23:39.259212 > Traceback (most recent call last):
2015-07-06 12:23:39.259269 > Failure: twisted.internet.defer.TimeoutError: Getting http://127.0.0.1:8332/ took longer than 30 seconds.
2015-07-06 12:23:39.438495 P2Pool: 17354 shares in chain (17358 verified/17358 total) Peers: 11 (7 incoming)
2015-07-06 12:23:39.438650  Local: 760GH/s in last 10.0 minutes Local dead on arrival: ~3.3% (2-6%) Expected time to share: 6.7 hours
2015-07-06 12:23:39.438709  Shares: 1 (0 orphan, 0 dead) Stale rate: ~0.0% (0-80%) Efficiency: ~117.5% (24-118%) Current payout: (0.0000)=0.0000 BTC
2015-07-06 12:23:39.438780  Pool: 997TH/s Stale rate: 14.9% Expected time to block: 2.5 days
2015-07-06 12:23:40.143664 Lost peer 101.100.174.138:46941 - Connection was aborted locally, using.
2015-07-06 12:23:40.342505 New work for worker! Difficulty: 419.298668 Share difficulty: 5782371.706794 Total block value: 25.630668 BTC including 2714 transactions
2015-07-06 12:23:51.731872 Peer sent entire transaction 02c8b5f4fb93ccb26cfde2d76c857f2915b42341bee558c2f5743442c51c9163 that was already received
2015-07-06 12:23:51.762187 Peer sent entire transaction 3a4e6b7d4c33c6a5b3622ce072806806bbba6fb244d6a1da390384b39da5b842 that was already received
2015-07-06 12:23:51.864129 Peer sent entire transaction 02c8b5f4fb93ccb26cfde2d76c857f2915b42341bee558c2f5743442c51c9163 that was already received
2015-07-06 12:23:52.317146 > ########################################
2015-07-06 12:23:52.317516 > >>> Warning: LOST CONTACT WITH BITCOIND for 1.2 minutes! Check that it isn't frozen or dead!
2015-07-06 12:23:52.317690 > ########################################
2015-07-06 12:23:52.317892 P2Pool: 17354 shares in chain (17358 verified/17358 total) Peers: 10 (6 incoming)
2015-07-06 12:23:52.318044  Local: 567GH/s in last 10.0 minutes Local dead on arrival: ~1.4% (0-3%) Expected time to share: 9.0 hours
2015-07-06 12:23:52.318190  Shares: 1 (0 orphan, 0 dead) Stale rate: ~0.0% (0-80%) Efficiency: ~117.5% (24-118%) Current payout: (0.0000)=0.0000 BTC
2015-07-06 12:23:52.318347  Pool: 997TH/s Stale rate: 14.9% Expected time to block: 2.5 days
2015-07-06 12:23:52.984908 > Unhandled Error
2015-07-06 12:23:52.985308 > Traceback (most recent call last):
2015-07-06 12:23:52.985485 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:52.985652 >     reactor.run()
2015-07-06 12:23:52.985854 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:52.986325 >     self.mainLoop()
2015-07-06 12:23:52.986489 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:52.986682 >     self.runUntilCurrent()
2015-07-06 12:23:52.986848 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:52.987004 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:52.987165 > --- <exception caught here> ---
2015-07-06 12:23:52.987294 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:52.987489 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:52.987621 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:52.987745 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:52.987868 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:23:52.988240 > Unhandled Error
2015-07-06 12:23:52.988373 > Traceback (most recent call last):
2015-07-06 12:23:52.988495 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:52.988683 >     reactor.run()
2015-07-06 12:23:52.988807 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:52.988934 >     self.mainLoop()
2015-07-06 12:23:52.989056 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:52.989191 >     self.runUntilCurrent()
2015-07-06 12:23:52.989313 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:52.989436 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:52.989556 > --- <exception caught here> ---
2015-07-06 12:23:52.989677 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:52.989909 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:52.990046 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:52.990170 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:52.990291 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:23:53.006595 > Unhandled Error
2015-07-06 12:23:53.006974 > Traceback (most recent call last):
2015-07-06 12:23:53.007146 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:53.007307 >     reactor.run()
2015-07-06 12:23:53.007470 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:53.007627 >     self.mainLoop()
2015-07-06 12:23:53.007789 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:53.007917 >     self.runUntilCurrent()
2015-07-06 12:23:53.008041 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:53.008164 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:53.008285 > --- <exception caught here> ---
2015-07-06 12:23:53.008506 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:53.008669 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:53.008810 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:53.008946 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:53.009139 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:23:53.010013 > Unhandled Error
2015-07-06 12:23:53.010236 > Traceback (most recent call last):
2015-07-06 12:23:53.010424 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:53.010634 >     reactor.run()
2015-07-06 12:23:53.010839 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:53.011032 >     self.mainLoop()
2015-07-06 12:23:53.011250 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:53.011415 >     self.runUntilCurrent()
2015-07-06 12:23:53.011552 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:53.011737 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:53.011883 > --- <exception caught here> ---
2015-07-06 12:23:53.012011 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:53.012178 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:53.012357 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:53.012494 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:53.012620 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:23:53.013144 > Unhandled Error
2015-07-06 12:23:53.013335 > Traceback (most recent call last):
2015-07-06 12:23:53.013467 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:53.013593 >     reactor.run()
2015-07-06 12:23:53.013717 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:53.013888 >     self.mainLoop()
2015-07-06 12:23:53.014024 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:53.014147 >     self.runUntilCurrent()
2015-07-06 12:23:53.014269 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:53.014391 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:53.014510 > --- <exception caught here> ---
2015-07-06 12:23:53.014674 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:53.014830 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:53.014958 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:53.015087 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:53.015217 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:23:53.015589 > Unhandled Error
2015-07-06 12:23:53.015722 > Traceback (most recent call last):
2015-07-06 12:23:53.015844 >   File "/home/miner/p2pool/p2pool/main.py", line 666, in run
2015-07-06 12:23:53.015966 >     reactor.run()
2015-07-06 12:23:53.016095 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1192, in run
2015-07-06 12:23:53.016222 >     self.mainLoop()
2015-07-06 12:23:53.016342 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1201, in mainLoop
2015-07-06 12:23:53.016463 >     self.runUntilCurrent()
2015-07-06 12:23:53.016583 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 824, in runUntilCurrent
2015-07-06 12:23:53.016713 >     call.func(*call.args, **call.kw)
2015-07-06 12:23:53.016849 > --- <exception caught here> ---
2015-07-06 12:23:53.016969 >   File "/home/miner/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 12:23:53.017090 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 12:23:53.017254 >   File "/home/miner/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 12:23:53.017387 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 12:23:53.017510 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind
2015-07-06 12:24:06.272733 > Peer referenced unknown transaction 61cbf98adbeb8f843ed300add451e33643109e7ad9097e12872a066b4c55400c, disconnecting

EDIT: it's definitely an intermittent thing... because now my node is back to running smoothly again.

[p3yot33at3r]

I've not had any issues with the NMC client (using latest namecore), only the bitcoind problem. The pool hash rate has dropped massively - there's obviously a stability problem somewhere - but I haven't seen another disconnect since disabling the relayclient.......yet  

This forum constantly dropping isn't helping much either 

[jonnybravo0311]

I've not had any issues with the NMC client (using latest namecore), only the bitcoind problem. The pool hash rate has dropped massively - there's obviously a stability problem somewhere - but I haven't seen another disconnect since disabling the relayclient.......yet  

This forum constantly dropping isn't helping much either 

LOL... typing a reply, hitting submit only to see, "Bad Gateway" or some other error is a pain .

I've still got the relay client running and not experiencing any issues with it... at least as far as I can tell.  Node seems to have stabilized for the time being, so not sure what was going on earlier...

[p3yot33at3r]

Yeah, frustrating isn't it?  

I'm seeing some huge cpu usage by both bitcoind & p2pool - over 80% sometimes, my latency has also gone through the roof, even after restarting.

Is that plonker doing another "stress test" again?  

Edit:

OK, this is getting silly. 99% cpu, & often.  :



Never seen it jumping up to these levels before - I'm beginning to think that the problem is not with p2pool or bitcoind, but with the network itself. I'm seeing some crazy amounts of traffic with super high latency, it seems to be working my node to death despite limiting connections, disabling the relaynetworkclient, prioritizing python & bitcoind, QOS & every other trick I can think of.......

I can see by the total p2pool hash rate & the opinions of others in other threads that something is wrong somewhere, but I'm buggered if I know what it is. I'm open to any & all suggestions people.......just hoping it's not the "usual suspects" again, messing with their pools code & screwing up the network somehow........

[Meuh6879]

Well, we have a "massive blacklist" of 0.9.4 and less ... now.

Here is a debug.log of the 0.11.0 RC3.


perhaps, it's liked to the network activity (overhead !)

[nonnakip]

NastyPool is running latest (14.0-6-g18b5ad5) and I also see large "LOST CONTACT WITH BITCOIND" blocks. I typed a rather large report of what I could find. But then I check the log again and it is happening right now! When I run bitcoin-cli I have no problems connecting to bitcoind. P2Pool only has 36 file descriptors open (10240 allowed).

Code:

2015-07-06 19:36:05.119752 > Unhandled Error
2015-07-06 19:36:05.119846 > Traceback (most recent call last):
2015-07-06 19:36:05.119973 >   File "/home/p2pool/p2pool/p2pool/main.py", line 666, in run
2015-07-06 19:36:05.120065 >     reactor.run()
2015-07-06 19:36:05.120160 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1169, in run
2015-07-06 19:36:05.120246 >     
2015-07-06 19:36:05.120328 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1178, in mainLoop
2015-07-06 19:36:05.120427 >     
2015-07-06 19:36:05.120511 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 800, in runUntilCurrent
2015-07-06 19:36:05.120597 >     
2015-07-06 19:36:05.120687 > --- <exception caught here> ---
2015-07-06 19:36:05.120773 >   File "/home/p2pool/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 19:36:05.120858 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 19:36:05.120990 >   File "/home/p2pool/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 19:36:05.121082 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 19:36:05.121180 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind

EDIT: bitcoind 0.10.2 debug.log shows nothing unusual.