So each private key can ultimately produce two different addresses - one compressed and one uncompressed?
There are no addresses at the technical level. Addresses are something the user interface provides to us humans to make it easier for us to talk about transferring control over value.
I assume you are asking about P2PKH addresses (the addresses that start with a '1'). In that case, yes, each private key can ultimately produce two different addresses.
However, recall that there are also P2SH addresses (the addresses that start with a '3').
If you include P2SH addresses, then for so long as we choose to compute an address as a RIPEMD160 hash of some information encoded with base58check encoding, there will be 2
160 possible P2SH addresses for every private key.
And different private keys can produce the same address (i.e. there are 2^96 private keys for each address)?
Correct. Although, realistically, only one of those private keys will ever be known.
From the web:
"A compressed key is just a way of storing a public key in fewer bytes (33 instead of 65). There are no compatibility or security issues because they are precisely the same keys, just stored in a different way. "
So they're the same keys but when hashed produce different addresses?
Correct, because they are "just stored in a different way", and what is being hashed is that way of storing it.