Everyone says to not use brainwallets but can't they be secure with a good enough passphrase?

Say you memorized 5-6 diceware words. You have 64.6 and 77.5 bits of entropy respectively. Realistically would these ever be cracked?

Or a good mnemonic pattern, thinking back to your childhood for example.

Maybe I'm just a noob for I don't see why this is a bad idea. I mean wouldn't someone breaking into your home and stealing your seed phrase be more likely?