So this gets me wondering, What happens when the block rewards are too low to sustain mining?
There are several topics on this, in case you're interested. Short version: I don't worry about it
1. Won't this lead to problems on the bitcoin network? Like slow transactions and high fee?
Transaction speed is defined by the protocol, it's the same (on average) with 1 miner as it will be with a million miners.
Fees depend on block scarcity, at the end of 2017 fees often got half as high as the block reward already.
2. Is there a contingency plan for this kind of scenario?
Yes: fees.
If anything, lower block earnings will lead to a lower energy consumption and a lower drain of resources from the Bitcoin ecosystem to hardware manufacturers and power companies. But high block earnings lead to a high difficulty, and thus secure network.
