How would you feel if your transaction is stuck for more than an hour. I think that 5-30 minutes is a decent time to process your withdrawal. Unlike the experience that I have before, that waiting for me to be honest is acceptable.
Whenever my transactions take longer than usual, I always feel like I clicked or types something wrong during the transaction. And it happened to me ones. But other than that, I always received mine most of the time on time.
That incident that made you suspicious about longer transactions puts that idea on your mind that something must be wrong.
Well, the way to solve it is by just paying the right fee and if you want to accelerate it there's a way but much better to pay with high priority so that you will receive it earlier on time, this is an option though.