There is also a project called
CoinJoin in the works to help out with anonymity. If this ever gets widely adopted you
really won't want to assume the inputs to a transaction come from the same owner, as the purpose of CoinJoin is to make sure that doesn't happen.
The correct answer to your question is to
ask your user what address they want you to send refunds/withdrawals to. Thank you for asking instead of making an assumption. By the way, transaction IDs are
not reliable without confirmations so don't have our app rely on those either.
