You shouldnt be too worried as the person above me mentioned. But you should be worried on if the miner starts getting more errors in work be counted for, meaning people who already solved the block way ahead of you.
Yeah, what you should be watching out for are rejected or stale shares, and high rate of hardware errors. Otherwise, you should be fine.