That is a terrible idea. It circumvents behavior that is designed to make bitcoin safer for users, and does so in a way which may not be safe when interfacing with other transaction construction protocols. Please don't do it.
But so as to not be a complete jerk, I will answer your specific question: you need the input's referenced output. The transaction intput has a hash and vout index which you can use to lookup the output from the UTXO set (if it exists and is unspent). The scriptPubKey in that output is the "address" you are looking for.
Thanks for the hints
As for the actual idea, change is usually pretty obvious anyway, since the actual amount sent is commonly a round number, and change is not. I'm curious what construction protocols it might break though, the code itself would only be used for building standard transactions in the wallet