Well, suppose a computer could test 1 million per second. It could crack all 5-word diceware brain wallets in 8 million seconds, or about 3 months.
There are 7776 possible diceware words, meaning 7776
5 combinations. This would still take over 900,000 years at 1 million per second.
There are two things to consider when deciding if a wallet is secure. Firstly, how likely it is that someone else will gain access, and secondly, how likely it is that you will lose access. Standard brain wallet fail on both those cases.
If you are dead set on memorizing a wallet, then far safer to generate a 12 word seed phrase and memorize that. Then at least your wallet has 128 bits of entropy and not ~64 bits of entropy. Although it might seem that 64 bits is only half as strong as 128 bits, it's actually about 18 billion billion times less strong. Do you really want your wallet to be 18 billion billion times easier to crack than everyone else's? I don't think so.
Also, even if you memorize a wallet, you should
absolutely also have a written back up somewhere safe. Your scenario of someone breaking in to your home to steal your seed phrase: If you are very concerned about this, then there are 100 places in your house you could hide a small piece of paper that a thief would never find even with a week of looking specifically for it. Think of places like under floorboards, inside the lining of your sofa, etc. Or go an place your seed phrase in a safe deposit box at your bank if you want. There are a thousand things that could happen to you tomorrow with no warning which could result in memory loss and complete loss of your wallet if you don't have a back up.