What does that difficulty graph you posted to do with the speed transactions are confirmed?
Here is what difficulty is:
https://en.bitcoin.it/wiki/DifficultyThe speed on which transactions are confirmed is not related to the numbers of miners/difficulty/hashrate.
But of course, since there is limited blockspace and more and more people want to transfer coins but they don't want or can't use segwit transactions this will just get worse.