1. What is the formula to calculate segwit transaction fee in which the weight will be included.
There's no fancy formula, all you have to do is to multiply non-Witness data by 4 and Witness data by 1 and that's the Weight of the transaction.
Then divide the weight by four to get the 'vBytes', that's basically the same as what your client (
eg. Electrum) is displaying as 'Bytes' if it supports SegWit.
Lastly, compute based from the usual 'n sat/B' fee rate.
Read
https://en.bitcoinwiki.org/wiki/Block_weight#Detailed_example to know which parts of the transaction are 'Witness' data.
2. If sending to another type of address like the compatible and legacy, will the fee still be reduce? Also, if sending from another type of address like the compatible and legacy to segwit, will the fee be like normal transaction fee?
Outputs wont count with the virtual size reduction since those are non-Witness data.
3. Why is P2SH (compatible) address fee also low compared to legacy address when both use transaction size to calculate the fee.
You mean P2SH-SegWit? It's not the "
raw size", but "
virtual size".
Because some parts of it is Witness data and as I said, those are counted as '1' Weight unit compared to '4' for non-Witness data.
See it for yourself by checking your Electrum transaction history and check it on blockstream.info, the weight should be four times the 'bytes' displayed in Electrum whether it's SegWit or not.
If you want to see the raw Bytes, use blockchain.com's explorer.