And if you're thinking what I think you're thinking, remember that you don't actually have to find their private key to claim someone else's bitcoins. All you have to do is find any one of the roughly 2^96 private keys whose corresponding public key hashes to that address.

Ok so for somebody to be lucky enough to steal the bitcoins out of my main backup address...

They first need to generate my address which could be any one of 2^160 possibilities. And then they must generate the corresponding public key that gives them access to the bitcoins in my address? Of where there at 2^96 possibilities?

Generating the public key is trivial.

Or must they simple generate 2^96 bitcoin keys... at which point they would control every single bitcoin address in the world?

The most sensible way to attempt the attack (which is still insane) is to generate random private keys, calculate the corresponding addresses, and then see if that address has a non-zero balance. I believe there are 2^160 possible addresses. So even if there are 1,000,000,000 addresses with non-zero balances, your odds of getting a non-zero balance on a single key are 1 in 2^128.

So brute-forcing a single bitcoin address with a non-zero balance (assuming there are a billion of them, which is generous), is as hard as, say, brute-forcing a given 128-bit AES key.