You can connect your ledger nano x wallet with full node if you follow this GUIDE from ledger website.
Since OP explicitly mention about privacy and TOR usage, it's not good idea since Ledger is known for blocking Tor due to technical problem. I don't know if the wallet functionally works if you use your own full node.
You can also use Electrum and your own Electrum personal server.
I also recommend this. Just make sure Electrum only connected only to your server before you connect Electrum with your hardware wallet.
Thank you all ! Grateful to hear your feedback! I'll try the Electrum setup to the nano X once my Raspiblitz is here. Will let you know how it goes!