The origin of a payment

In your wallet, you have a collection of previous payments you have received. If you have sent out other payments (creating change), these individual balances may be stored in differing amounts in various addresses not shown to you.

When you create a transaction sending Buddy a payment, one or several of the previous payments you received will be used to fund the transaction paying Buddy.

When Buddy creates a transaction sending you a payment, one or several of the previous payments he received will be used to fund the transaction paying you back. There is no guarantee that he will be sending the same "input" payment back to you as an "output". Since transaction fees will alter the amount, the Bitcoin client may combine different input amounts to find an ideal amount to send, or the particular payment may have already spent, so you will get back some other coins from different funding sources.

An outside observer may not be able to readily observe what's going on. When you send a payment, it may come from other addresses than your published addresses, and when you get your refund, you can also create a one-time-use address just for that. Never using the same address twice makes it very hard for someone to have any idea what's going on between two individuals.