When using listen=1 you are likely exposing your IP to someone who wants to connect to your node and there is a potential risk on listen=1 without using TOR or hidden service malicious nodes/attackers can attempt to exploit something in your node.
Since it reveals your IP to someone you already compromise your privacy and it consumes much CPU and memory.
If you set
listen=0 then your node simply rejects incoming connections. It will still make outgoing connections, because otherwise how would it function as a node at all? Each peer you are connected to will be able to see your IP, regardless of what setting you use for
listen.
If privacy is your goal, then the only solution is running via Tor (or maybe a VPN, depending on your particular needs).