[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[p3yot33at3r]

Hmmm.......has anyone made contact with forrestv? It would be good to get his take of what's happening........

[squidicuz]

NastyPool is running latest (14.0-6-g18b5ad5) and I also see large "LOST CONTACT WITH BITCOIND" blocks. I typed a rather large report of what I could find. But then I check the log again and it is happening right now! When I run bitcoin-cli I have no problems connecting to bitcoind. P2Pool only has 36 file descriptors open (10240 allowed).

Code:

2015-07-06 19:36:05.119752 > Unhandled Error
2015-07-06 19:36:05.119846 > Traceback (most recent call last):
2015-07-06 19:36:05.119973 >   File "/home/p2pool/p2pool/p2pool/main.py", line 666, in run
2015-07-06 19:36:05.120065 >     reactor.run()
2015-07-06 19:36:05.120160 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1169, in run
2015-07-06 19:36:05.120246 >     
2015-07-06 19:36:05.120328 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 1178, in mainLoop
2015-07-06 19:36:05.120427 >     
2015-07-06 19:36:05.120511 >   File "/usr/lib/python2.7/dist-packages/twisted/internet/base.py", line 800, in runUntilCurrent
2015-07-06 19:36:05.120597 >     
2015-07-06 19:36:05.120687 > --- <exception caught here> ---
2015-07-06 19:36:05.120773 >   File "/home/p2pool/p2pool/p2pool/bitcoin/stratum.py", line 38, in _send_work
2015-07-06 19:36:05.120858 >     x, got_response = self.wb.get_work(*self.wb.preprocess_request('' if self.username is None else self.username))
2015-07-06 19:36:05.120990 >   File "/home/p2pool/p2pool/p2pool/work.py", line 211, in preprocess_request
2015-07-06 19:36:05.121082 >     raise jsonrpc.Error_for_code(-12345)(u'lost contact with bitcoind')
2015-07-06 19:36:05.121180 > p2pool.util.jsonrpc.NarrowError: -12345 lost contact with bitcoind

EDIT: bitcoind 0.10.2 debug.log shows nothing unusual.

Give it some time, it will stabilize.  My node timed out a few more times after I updated, then settled down and hasn't all day.  The version 13 peers are creating an issue with the entire pools network, causing higher than normal traffic rates and timeouts.

[forrestv]

Hmmm.......has anyone made contact with forrestv? It would be good to get his take of what's happening........

Hey, I've been watching. You are trying the recent P2Pool commit that blocks pre-v14 peers, right? If that's confirmed to help, I'll make a P2Pool release with it.

Some P2P network thrashing is normal after a fork since you have the unupgraded nodes madly trying to find peers and download the (invalid to them) sharechain. Normally, there are less unupgraded nodes, but this fork was rushed, so I think that that contributed to it.

However, some are reporting high bitcoind CPU usage - P2Pool P2P network activity shouldn't be affecting bitcoind, so I'm not sure about that. Is that happening for Bitcoin users without P2Pool? It might be some similar post-BIP66 effect.

[jonnybravo0311]

Looks like the network is getting hammered with transactions... I turned off Matt's relay network and restarted everything.  Looking at blockchain.info, there's ver 11k unconfirmed transactions currently.

My node is currently trying to put together blocks with 2k transactions...

Code:

2015-07-06 16:11:19.107855 New work for worker! Difficulty: 1217.831707 Share difficulty: 5838904.174398 Total block value: 25.382547 BTC including 1995 transactions
2015-07-06 16:11:52.966132 New work for worker! Difficulty: 1089.745702 Share difficulty: 5838098.085605 Total block value: 25.386081 BTC including 2018 transactions

[nonnakip]

However, some are reporting high bitcoind CPU usage - P2Pool P2P network activity shouldn't be affecting bitcoind, so I'm not sure about that. Is that happening for Bitcoin users without P2Pool? It might be some similar post-BIP66 effect.

I noticed when I started P2Pool that my node is carrying about 6000 unverified shares (17354 verified but on start it loaded 23000). I assume these are shares from <14.0. Perhaps all these unverifiable shares are causing the 100% CPU usage and bitcoind work_poller delays?

Would it be better to delete my data directory and redownload share chain? Or just wait until we solve a block and maybe the problem goes away by itself?

[p3yot33at3r]

Hmmm.......has anyone made contact with forrestv? It would be good to get his take of what's happening........

Hey, I've been watching. You are trying the recent P2Pool commit that blocks pre-v14 peers, right? If that's confirmed to help, I'll make a P2Pool release with it.

Some P2P network thrashing is normal after a fork since you have the unupgraded nodes madly trying to find peers and download the (invalid to them) sharechain. Normally, there are less unupgraded nodes, but this fork was rushed, so I think that that contributed to it.

However, some are reporting high bitcoind CPU usage - P2Pool P2P network activity shouldn't be affecting bitcoind, so I'm not sure about that. Is that happening for Bitcoin users without P2Pool? It might be some similar post-BIP66 effect.

Hi forrestv,

Thanks for chiming in buddy. Yes, using the latest p2pool repo. I'm seeing high cpu usage on both p2pool & bitcoind, but the cpu usage on p2pool is much more prolonged & constantly higher than bitcoind, which seems to have sudden peaks & bursts of high usage - resulting in very high latency issues.

It's a tricky one eh?  

I even tried re-downloading the p2pool sharechain......

[xyzzy099]

Looks like someone is pounding the network with transactions:

https://bitcointalk.org/index.php?topic=1111811.msg11807542#msg11807542


Maybe this accounts for the problems you guys are seeing.

[p3yot33at3r]

That would explain a lot, the thought did occur to me earlier:

Is that plonker doing another "stress test" again?  

If this is the case, it happening at around the same time as a forking would have to be intentional I reckon........is it from the same origin as the last time?

[mdude77]

I upgraded here as well.  Here's a message I don't recall seeing before:

Code:

2015-07-06 17:10:22.969017 Peer 104.193.40.248:9333 misbehaving, will drop and ban. Reason: first message was not version message
2015-07-06 17:10:22.969112 Bad peer banned: (u'104.193.40.248', 9333)

Otherwise I'm getting lots of bad peers.  For being at 95% on the new version, I sure do get connected to the 5% pretty easily.

M

[squidicuz]

I upgraded here as well.  Here's a message I don't recall seeing before:

Code:

2015-07-06 17:10:22.969017 Peer 104.193.40.248:9333 misbehaving, will drop and ban. Reason: first message was not version message
2015-07-06 17:10:22.969112 Bad peer banned: (u'104.193.40.248', 9333)

Otherwise I'm getting lots of bad peers.  For being at 95% on the new version, I sure do get connected to the 5% pretty easily.

M

The bad peers are the remaining forked version 13 peers being removed.  Once your node ignores them all, stability will improve.  The high transaction volume on the bitcoin network combined with the persistent p2pool share fetching of version 13 nodes, along with some nodes timing out and further adding peer stress, is really hammering everyone.
Updating to the latest version of p2pool from github is needed in order to block the connection spam and stabilize the pool.

[windpath]

Things seem to be running smoothly again after the final update at around 3PM EDT today.

There has been a flood of TX's on the bitcoin network, and GBT latency is currently higher than I'd like to see it, but I believe P2Pool is functioning as it should with the latest version.

As Forrest mentioned, on the newest version it appears we are simply rejecting anything from a node with lower than p2pool version 14, I show 100% V 14 peers on my node...

Anyone monitoring how much of the p2pool network we left behind?

Here is the bitcoin transaction flood from today (source linked, click on "txInMemoryPool" on lower left):

[jonnybravo0311]

You are welcome... and welcome to p2pool .

[jonnybravo0311]

So it appears the problems continue... my node has been effectively offline for the majority of the day today.  P2Pool is pegging my CPU at 100% and I'm consistently seeing the "lost contact with bitcoind" message in the logs.  Is this solely because of the ridiculous number of unconfirmed transactions that are still flooding the network (nearly 13k at this point)?  Granted, my VPS isn't exactly a monster (but not horrible as a dual core droplet).

I'm on the latest version of p2pool and bitcoin core 0.10.2 compiled from source.

To the miners who are on my node, I apologize for the inconvenience this is likely causing you.  On the bright side, my miners fall back to ck.'s solo pool, so maybe I'll be lucky and they'll solve a block for me .

EDIT: when my node does appear online, and the miners are connected, they're reporting huge rejected hash percentages (25% or more).

[luthermarcus]

I had the problem of bitcoin core freezing (I would only notice it on p2pool log "lost connection" or through email saying my node has stalled) since i upgraded from 10.0 to 10.2 i thought it was my linksys router so i bought a switch should be arriving any day now. Could still be a coincidence but more and more people are speaking out about it. Bitcoin has been more stable now but does drop at peeks. Bitcoin uses a about 4 g ram which is 51% of a little under 8 g's and a quater to half my cpu on my intel 2 Quad CPU (Q6700 @ 2.66GHz). That's bitcoin alone. Once the switch comes in ill put p2pool back online and put out stats I should be getting 100mb/s but running a speedtest i was getting 7.5 which is slow so it could have been the connection timing out.

[p3yot33at3r]

I dunno. It might be nothing, but I was going through my p2pool logs earlier & noticed the same IP over & over again that was connecting, submitting dodgy hash, getting banned for it then reconnecting again on a different port & doing the same thing all over again, constantly & repeatedly using a different port every time. My logs are full of the same IP, namely 219.239.31.11 - it's in China:

Code:

2015-07-06 06:14:57.151757 invalid hash for 219.239.31.11 'remember_tx' 748990 a9117513
2015-07-06 06:14:57.151874 Bad peer banned: ('219.239.31.11', 64888)
2015-07-06 06:14:57.155612 Lost peer 219.239.31.11:64888 - Connection was aborted locally, using.
2015-07-06 06:14:57.156126 > in download_shares:
2015-07-06 06:14:57.156218 > Traceback (most recent call last):
2015-07-06 06:14:57.156259 > Failure: twisted.internet.error.ConnectionAborted: Connection was aborted locally, using.

Then reconnect on different port, request shares, submit bad hash, get banned again:

Code:

2015-07-06 05:13:49.929600 Incoming connection to peer 219.239.31.11:33919 established. p2pool version: 1400 '14.0'
2015-07-06 05:13:54.675766 invalid hash for 219.239.31.11 'remember_tx' 292502 69c836d4
2015-07-06 05:13:54.675885 Bad peer banned: ('219.239.31.11', 33919)
2015-07-06 05:13:54.676322 Lost peer 219.239.31.11:33919 - Connection was aborted locally, using.

Over & over, closely followed by a bitcoind disconnect.

Note that whoever this is is using v14, so isn't getting banned by p2pool straight away. My logs are full of this IP using a different port every connection, submitting bad hash, requesting good then getting banned - so instead of waiting for 65535 banned connections to get through, I figured I'd just get rid, so I banned it at the router level firewall too.

Since I did that a few hours ago I haven't had a single disconnect. Yet. Maybe I'm clutching at straws, but it just seems a bit weird that some China IP keeps flooding my node with shit so close to losing contact with bitcoind. I'd be interested to hear if anyone else has this IP in their logs........

Or any other ideas of course, but check your logs anyway - see if you have the same IP as in my logs.......

EDIT: Still getting this though  :

[luthermarcus]

I dunno. It might be nothing, but I was going through my p2pool logs earlier & noticed the same IP over & over again that was connecting, submitting dodgy hash, getting banned for it then reconnecting again on a different port & doing the same thing all over again, constantly & repeatedly using a different port every time. My logs are full of the same IP, namely 219.239.31.11 - it's in China:

Code:

2015-07-06 06:14:57.151757 invalid hash for 219.239.31.11 'remember_tx' 748990 a9117513
2015-07-06 06:14:57.151874 Bad peer banned: ('219.239.31.11', 64888)
2015-07-06 06:14:57.155612 Lost peer 219.239.31.11:64888 - Connection was aborted locally, using.
2015-07-06 06:14:57.156126 > in download_shares:
2015-07-06 06:14:57.156218 > Traceback (most recent call last):
2015-07-06 06:14:57.156259 > Failure: twisted.internet.error.ConnectionAborted: Connection was aborted locally, using.

Then reconnect on different port, request shares, submit bad hash, get banned again:

Code:

2015-07-06 05:13:49.929600 Incoming connection to peer 219.239.31.11:33919 established. p2pool version: 1400 '14.0'
2015-07-06 05:13:54.675766 invalid hash for 219.239.31.11 'remember_tx' 292502 69c836d4
2015-07-06 05:13:54.675885 Bad peer banned: ('219.239.31.11', 33919)
2015-07-06 05:13:54.676322 Lost peer 219.239.31.11:33919 - Connection was aborted locally, using.

Over & over, closely followed by a bitcoind disconnect.

Note that whoever this is is using v14, so isn't getting banned by p2pool straight away. My logs are full of this IP using a different port every connection, submitting bad hash, requesting good then getting banned - so instead of waiting for 65535 banned connections to get through, I figured I'd just get rid, so I banned it at the router level firewall too.

Since I did that a few hours ago I haven't had a single disconnect. Yet. Maybe I'm clutching at straws, but it just seems a bit weird that some China IP keeps flooding my node with shit so close to losing contact with bitcoind. I'd be interested to hear if anyone else has this IP in their logs........

Or any other ideas of course, but check your logs anyway - see if you have the same IP as in my logs.......

EDIT: Still getting this though  :

Here is an article which could explain what your seeing
https://bitcoinmagazine.com/21125/bitcoin-network-still-backlogged-tens-thousands-unconfirmed-transactions-causing-delays/

[p3yot33at3r]

Yeah, I'm aware of the attack - been following it:

https://www.reddit.com/r/Bitcoin/comments/3cfext/bitcoin_under_attack/

&

https://www.reddit.com/r/Bitcoin/comments/3cg95d/bitcoin_is_resilient/

But I don't think that has anything to do with the IP flooding my node.

BTW - love this quote:

Now the interesting thing is that some core devs (paging Peter Todd, Luke-Jr, Gregory Maxwell) (some with notable direct conflicts of interest by working at Blockstream) do not wish to allow bitcoin to grow......

QFT. 

[p3yot33at3r]

BLOCK!!

Phew.......

[squidicuz]

I dunno. It might be nothing, but I was going through my p2pool logs earlier & noticed the same IP over & over again that was connecting, submitting dodgy hash, getting banned for it then reconnecting again on a different port & doing the same thing all over again, constantly & repeatedly using a different port every time. My logs are full of the same IP, namely 219.239.31.11 - it's in China:

Code:

2015-07-06 06:14:57.151757 invalid hash for 219.239.31.11 'remember_tx' 748990 a9117513
2015-07-06 06:14:57.151874 Bad peer banned: ('219.239.31.11', 64888)
2015-07-06 06:14:57.155612 Lost peer 219.239.31.11:64888 - Connection was aborted locally, using.
2015-07-06 06:14:57.156126 > in download_shares:
2015-07-06 06:14:57.156218 > Traceback (most recent call last):
2015-07-06 06:14:57.156259 > Failure: twisted.internet.error.ConnectionAborted: Connection was aborted locally, using.

Then reconnect on different port, request shares, submit bad hash, get banned again:

Code:

2015-07-06 05:13:49.929600 Incoming connection to peer 219.239.31.11:33919 established. p2pool version: 1400 '14.0'
2015-07-06 05:13:54.675766 invalid hash for 219.239.31.11 'remember_tx' 292502 69c836d4
2015-07-06 05:13:54.675885 Bad peer banned: ('219.239.31.11', 33919)
2015-07-06 05:13:54.676322 Lost peer 219.239.31.11:33919 - Connection was aborted locally, using.

Over & over, closely followed by a bitcoind disconnect.

Note that whoever this is is using v14, so isn't getting banned by p2pool straight away. My logs are full of this IP using a different port every connection, submitting bad hash, requesting good then getting banned - so instead of waiting for 65535 banned connections to get through, I figured I'd just get rid, so I banned it at the router level firewall too.

Since I did that a few hours ago I haven't had a single disconnect. Yet. Maybe I'm clutching at straws, but it just seems a bit weird that some China IP keeps flooding my node with shit so close to losing contact with bitcoind. I'd be interested to hear if anyone else has this IP in their logs........

Or any other ideas of course, but check your logs anyway - see if you have the same IP as in my logs.......

EDIT: Still getting this though  :

https://i.imgur.com/4r7ElJH.png

That node likely does not have the most recent update to block v13 nodes. So it will get bad shares from its bad peers and then try to pass them on.  For some reason this is creating a timeout within p2pool at times, which then causes a disconnect to bitcoind and the other [p2pool] peers.

The best course of action is to manually ban it, like you did.

With a bunch of nodes now updated to 14.0-6, the pool seems to be recovering.  Just a few hiccups here and there.  Even still, we managed to score a block tonight!

[Meuh6879]

Yes ... above 0.9.1 bitcoin core, RPC of Bitcoin Core (or bitcoind) crash because of the P2Pool server ...
It's not new.

I have an automated routine to restart Bitcoin Core every 30min~1h for this.