I can only assume that you mean blockchain.info's My Wallet. If so, the passphrase was the password to your web wallet, not to the address. You can export your private key by logging back into My Wallet with that password and your username (looks something like 67d16919-7f0e-473e-a8e8-de3c20df5a78).
If you can't remember your username check in your history, there should be a url like:
https://blockchain.info/wallet/67d16919-7f0e-473e-a8e8-de3c20df5a78Blockchain.info's My Wallet is a web wallet (a highly secure, client side one at that). It is a service that stores your private keys in a manner that you can retrieve them but they can't. You can also make transactions with this but it all happens client side. I would recommend using My Wallet, but remember, the password is to encrypt the web wallet, not the private key's passcode.
If you need any help recovering your money, send me a PM. =)