The big payout isn't necessarily the problem. It's the **probability behind that payout**.
Speaking of me, I personally try to ignore side bet completely.
To me, placing any side bet is like giving the casino free tax.

But yes, sometime when I sit down for casual table session with friends, I throw in couple of small chips just for fun. But I know that money is just a waste!
Nowadays, there are many new side bets and progressive jackpot option coming online, which are even more aggressive.
That last point you made is the real essence.
Problem is not large size of payout, but the probability behind it.