It's a risk I'm willing to take.
That's great, but certainly not for the average Joe.
Don't tell people you have money in your head. I already said too much.
That is a given, if you want to keep your money safe. Running your mouth and telling everyone about your wealth is possibly the worst thing that you can do.
With that out of the way, we have already established that the difficulty will be determined by how complex your passphrase is. What would the downsides be, if you were to encrypt your seeds with AES and a less complex but sufficiently secure password?
The attacker cannot do anything without having access to your password and the added benefit is that the attacker would have to execute a targeted attack against you to crack the password which would exhaust quite a bit of resources.