People are tossing out technical concepts with little details. Some of these concepts aren't necessary to use Bitcoin, but now that they've been mentioned, it would be a bad idea to at least explain what people are saying.
A bitcoin address is a unique identifier that you can give to someone so they can send you bitcoins. You can't "choose" a specific bitcoin address, it has to be generated with software. As such it is pretty much a random string of 27 to 34 characters starting with a 1 or 3. There are "vanity address generators" that simply continuously generate random addresses until they stumble across one that matches a string of characters that you want in your address. However, the ore characters you want, the more difficult it is (and longer it takes) for the vanity generator to stumble across a matching address. You can have multiple addresses. You can give out a different address to each person/entity that might send you bitcoins so you can keep track of where the bitcoins came from. You can even give out a different bitcoin address for every transaction you receive.
A private key is a secret string of characters (typically starting with 5, L, or K) that is mathematically linked to your bitcoin address. Each bitcoin address will have its own private key. Anyone with access to the private key and a bit of technical knowledge can spend all the bitcoins that are sent to the address that is associated with the private key. This makes the private key extremely important. If you loose your private keys, then you can never again spend your bitcoins. They are lost permanently. If someone else gains access to your private keys, they can steal/spend your bitcoins. So you need to protect your private keys both from loss and theft. If you have a private key it is always possible to regenerate the bitcoin address that is associated with that key, however the reverse is impossible. If you have a bitcoin address, it is not possible to determine what the private key is. This means it is safe to give out your bitcoin addresses (so others can send you money), but knowing your bitcoin address is not enough to recover bitcoins if your private keys are lost.
A wallet is software (or a website) that allows you to receive, keep track of, and send bitcoins. Most wallets will keep track of multiple addresses for you and will provide a method for generating a new random address whenever you need one. Wallets keep track of the private keys for you, and use those private keys to create transactions for you when you want to send bitcoins somewhere. Some wallets will create additional bitcoin addresses without telling you about them, and send some of your bitcoins to those hidden addresses for safe keeping, so your wallet may be keeping track of more addresses than you are aware of. Most wallets provide a method of backing up the private keys that the wallet is keeping track of so that you can recover your wallet if your computer crashes. Many wallets also provide a way to use a password to encrypt the private keys that the wallet is keeping track of so that if someone else (or some malicious software) gains access to the computer, the private keys can't be used without the password. Some wallets also provide a method of printing out the private keys on paper to protect them from loss.
A public key is used to verify that you have the right to spend bitcoins that were sent to an address that you have control over. Like the bitcoin address, the public key is also mathematically linked to the private key. In general you will never see your public keys, and you will have no direct need to use or know them. As long as the wallet has a good copy of the private key, it can always regenerate the public key as needed when sending transactions. While a bitcoin address is similar to a public key, in that it is mathematically linked to the private key and that it is safe to provide it to others, they are not technically the same. Technically, a bitcoin address is a hash of a public key with an added checksum to prevent typos from accidentally sending bitcoins to the wrong address.
If you don't have control over the private key or have a method available to access a private key if/when needed, then you don't really control the bitcoins that are associated with the address. If there is a chance that someone else has access to the private key of an address, then for the sake of protecting your bitcoins from theft you must assume that person (or those people) have the ability to spend "your" bitcoins without your permission. As an example, any bitcoins stored with MtGox. MtGox may present you with a bitcoin address so that you can send bitcoins to (or receive bitcoins at) your "account" with them, however since they have the private key to the address, they have sole control over all bitcoins received at that address. You are trusting them to send those bitcoins elsewhere at your request, and trusting that they will still be there to do so when you make such a request. Essentially, you have handed your money over to a complete stranger, and in return that stranger has offered you a promise to honor any requests you might make as to what to do with that money. If you have more faith in the abilities of MtGox to secure the private keys from loss or theft than you have in your own abilities, then this can be a good thing. However, there is no insurance policy that will guarantee the safety of your bitcoins (unlike a bank in the U.S. that is insured by the FDIC).
If your bitcoins are stored at MtGox, you can send them directly from MtGox to any address you like (such as to loan them out or purchase something). However, some services (such as SatoshiDice) and some people make the false assumption that bitcoins can be returned to the sender by sending them back to whatever address was last in control of the bitcoins. In the case of MtGox, when you request that bitcoins be sent somewhere, MtGox just chooses some random bitcoins that they have control over and sends them. Then they update the balance that they display to you to reflect less bitcoins in your account. When bitcoins are sent back to the address that MtGox controls, they have no way of knowing whose "account" those bitcoins should be attributed to. As such, you should avoid using services (or transacting with people) who expect to be able to send coins back to the address they "came from". If you wish to engage in a transaction with such a service or person, you'll need to first transfer the bitcoins from MtGox to your own wallet before sending them on.
If you feel confident enough in your own ability to maintain good backups of your wallet and to protect your wallet from hackers and thiefs, then you'll want to consider keeping all your bitcoins under your own control whenever possible. This means choosing a wallet and only storing bitcoins at MtGox if/when you intend to exchange them. If you choose to use your own wallet, here are some to look into:
Armory
Bitcoin-Qt
Electrum
MultiBit
https://blockchain.info/walletEach has its own "quirks". Some are intrinsically a bit more secure than others. Some are more convenient to use than others. I'll leave it up to you to do the research and choose the best wallet for your own particular needs.
