Oh, and one more thing: what happens when the value of a new block goes from 50 BTC to 25 BTC?  Will the difficulty automatically go down then?  If not, then same question.

I am trying to understand the way in which the block generation difficulty is adjusted.  If people stop generating blocks, is it possible for the difficulty to decrease without 2016 blocks going by?  If not, then from what I understand, it seems like the following undesirable situation could arise:

0. The value of one BTC increases dramatically (say to 100 USD).
1. More people start mining, and so the difficulty increases a lot, and the electricity cost to generate a block settles at a bit less than 5000 USD as expected.
2. The value of one BTC decreases (for unrelated reasons) to 10 USD.
3. Almost everyone stops mining, because it's not worth spending 5000 USD to generate a block.  Now we only get (say) one block per day from a few people who are mining just for fun.
4. If the value of a BTC doesn't rise again, it takes about 5 years (in this case) for 2016 blocks to go by so that the difficulty can go down.

Am I missing something?  Thanks for reading.