If difficulty goes to high then marginal miners will be unprofitable ... they quit and difficulty falls.
Difficulty will not fall at first. You assume people are intelligent. I think it will slow and flat line.
A lot of variables to consider
Miners with and without electric bills
Miners that are good and bad at math
Miners that speculate the appreciation of coins
What we need is hedging. Miners need futures contracts so we can offset the difficulty of mining in the hole. If hedging instruments appear then I'm sure mining will continue even as difficulty accelerates.