Are you doing PPS or proportional?
Proportional can introduce a lot of variance. As the total number of shares increases your proportion is smaller and smaller, especially if the pool has been "unlucky".
But the more plausible cause is this: http://bitcoin.sipa.be/speed-lin-10k.png