On p2pool it will.
P2Pool is the only place your latency will affect anything, due to the extremely short rounds that P2Pool runs (10 seconds). For normal mining, it won't affect you one bit.
Now disconnects might affect you, esp if you're using a stratum pool, but a lot of those disconnect issues have been worked out.