I came to the conclusion that is was adversely affecting me when using p2pool - but that works slightly differently to most pools and is more latency sensitive.
But you're connecting to a local p2pool node, so there's essentially 0 ping.
Yes p2pool is running locally, but approximately every 10 secs the work you are doing is invalidated by long polling (which is delayed by your latency). So, my miner spends my latency time, working on invalid work.
Likewise, every time you get a share that p2pool reports to the rest of the network, it is delayed by your latency. So high latency increases the chances someone elses share will turn yours into an orphan/stale.
Someone with better connectivity and latency on p2pool can achieve better efficiency than me.