Either the price of bitcoin has to go up a bunch, or miners have to shut down because they can no longer pay their electric bills on a 1 coin/block. Then since they shut down their miners, the network shrinks and because unsecure and open to attack by anyone who decides to flip on their asic farm that they had previously shut down.
The difficulty of mining should adjust more smoothly than that. I doubt we'll see a sudden drop in difficulty due to a lot of miners giving up. New miners will stop joining the system because it won't be profitable enough to invest in new hardware. Old miners will tend to keep going, because they've already paid for their hardware. The ones who give up, because they can't even afford their electricity bills, will be the ones who have the least efficient hardware, so they will tend to be the ones who were making least contribution to the network, so their quitting won't have much impact. And of course, as each one quits, the remainder become more profitable.
Also, as transaction volume increases, the transaction fees per block will increase. This is for two reasons. First, there will be more transactions in each block. Second, there will be more users competing for the space in each block, so there will be more incentive for users to offer higher fees so their own transactions get processed quicker. So as long as Bitcoin is successful, there should be enough reward for mining to keep enough miners at it to discourage 51% attacks.
(Personally, I think governments and large companies will eventually start mining themselves, not for block rewards but as a kind of public good. America will want to be sure Russia can't mount a 51% attack. Microsoft will want to be sure Google can't mount one. One day, they will all be dependant on Bitcoin remaining healthy.)
