In order to benefit from the reduced fee from segwit, all the addresses in which you are sending the money must be using bech32 addresses to receive the bitcoins, or it only depends on your end? (using bech32 to send the bitcoin, then even if you are sending it to a legacy address, you still get the same reduction of fees)
No. Only the address sent from needs to be Segwit (and bech32 is only 1 option, you can used nested Segwit if you need to pay someone not using a Segwit address)
I assume that both ends must be bech32 to reduce the size of the transaction?
There is no reduction in size, more space is allowed in blocks for segwit txs. For the nested (i.e. backwards compatible) address type, the transactions are slightly bigger than regular compressed key txs.