A solution to this is to apply control theory to the adjustment.
idk much about control theory but i can think of a crude solution pretty easily. Retarget every block and compare the last 10 blocks to the 10 blocks before that. If the average hashing power for the last 10 blocks was lower than the average hashing power for the 10 blocks before that adjust down, if the average hashing power for the last 10 blocks was higher than the average hashing power for the 10 blocks before that adjust up.
but yea im sure thats a pathetic attempt compared to what a "control theorist" could devise