So, I set up private electrum server for me and point client to connect to my server in network options. Am I safe from that privacy issue "server will know all my addresses"? I may be wrong, but electrum client is connecting to other servers (maybe for different purposes like headers download), right? So what server gets my addresses, the one in 'server' setting only or any other it is connecting to?
only your server will see your addresses.
the other servers are used in order to get block headers for SPV.