I'm assuming you are using the localbitcoins.com escrow service?
If so, the FAQ at localbitcoins.com says the following:
How safe is the escrow if I'm buying bitcoins?
Very safe. The bitcoins are locked from the seller when they are in the escrow. The seller can only release them to buyer, or after 24 hours, file a dispute. Usually sellers strive to provide speedy service, and you will get your bitcoins fast. Sometimes delays happen, depending on transfer method, and patience is required. However there isn't a risk that the seller can run with the bitcoins. The seller can file a dispute, and then the localbitcoins staff researches the case manually.
So if you trust localbitcoins.com, then you can trust that either the bitcoins will be released, or the staff at localbitcoins.com will manually research the case and then you have to trust them to make a fair and reasonable determination.
If you do not trust localbitcoins.com, then you should not be using their escrow service.
You should not have to ever give your bitcoin address to the seller at all. Once the payment is made, the seller should release the escrow. The localbitcoins.com will instantly transfer your bitcoins to your localbitcoins.com wallet. Then you can send the bitcoins from the wallet localbitcoins.com provided you with on their website to any address you like. Note that localbitcoins.com will withhold 0.003 BTC from you and will force you to pay a 0.0005 BTC transaction fee when you send the bitcoins from your wallet at their site to any other address.