These charges are cosmic. If I reduce the gas to e.g. 20 and the fee is accepted, do I risk losing the money or will I just wait 2 days for it?
I think that there are two possibilities, the first should be your transaction will not be confirmed until the average fees will be going down to touch it. I don't even think if your transaction will be confirmed in two days and it will be more than a month.
The only possibility when your transaction is not yet confirmed by the block and then the average gas fees will increase, your tx will be rejected by the network and you will be losing your fees.
It's better if you can use the reliable fees.