Rule of thumb:
For normal transactions, 0.1 mBTC is more than enough to insure quick confirmation. But not enough to ruin you.
For really large transactions (large as in many inputs and/or outputs, not many BTC): Increase to 0.5 mBTC just to be safe.
For large amounts (several BTC) and "old" coins (inputs from last month or older), you rarely need a fee, but add 0.1 mBTC anyway since it is vanishing in this context, and since somebody could be running a "stress test" giving longer confirmation times.
Some wallets try to estimate a reasonable fee (breadwallet, for example). I do not have good experience with that, sometimes they manage to use so small a fee that confirmation is unnecessarily slow.
TL;DR: Pay 0.1 mBTC. It is small enough to not matter to you, but large enough to matter to the network