Mining speed on BTC Guild was 5,671 Th/s when BTC was above 580, now it's at 4845 TH/s and price is below 535. I keep seeing a pattern of it... just saying, I'm thinking some people stop mining when the price drops.
The price is high enough where just about all miners are just profitable in fiat, even if their margins are rapidly slimming. An 800TH/s fluctuation on a single pool can be attributed to a number of factors, including typical mining speed variability, pool switching or load balancing.
isn't difficulty supposed to be lower as well when people stop mining?
If enough miners stop mining, yes, but difficulty readjustment only happens every 2016 blocks. All but one miner could stop mining right now and the difficulty would stay the same until that single miner was able to finish out the rest of the round (which would take a loonnngg time).