I think you misunderstand it doesn't require someone to target you. Hackers can simply target all possible addresses simultaneously. Passphrases are relatively low entropy and without salt and key stretching hackers can just continually try all probable passphrases and steal funds from the addresses they find.
i.e. try a particular pass-phrase, generate the private key, compute the corresponding address, see if the address has value, if it does steal it, if it doesn't move on.
Likely whoever brute forces the password doesn't even know who you are, nor do they care. They are just looking for an address with value to steal.
Say this address (just grabbed it randomly from blockchain.info) is derived from a private key which is produced from the passphrase "password".
http://blockchain.info/address/14dnpgFWkqxsxb8ZbMVSTKVWofuUAD77rBWho owns this address? Who knows? Who cares. As soon as the hacker determines they have a private key for an address with value they will spend it. It is important to realize that with a brain wallet the funds can be stolen simply by discovering the passphrase. With a random key, or even an encrypted bitcoin wallet that isn't the case. It requires some physical or electronic access which is a significantly harder thing to achieve.
This is why I chuckle at those who think that bitcoins cannot be confiscated or stolen. If there is enough money at stake, thugs (including governments) find it cheaper and more effective to use violence than code-cracking.
What Bitcoins? Prove how many Bitcoins I have.