It's actually pretty easy to recover BTC sent to BCH address (or BCH sent to BTC address), assuming you control the private keys for the address that the coins have been sent to...
Unfortunately, with regards to the Ledger, the methods required for recovery will require that you use your wallet seed (there is not other way to get the private keys out)... this could potentially expose all the wallets you have created with that seed.
Basically:
1. Goto:
https://iancoleman.io/bip39/ and create an offline copy of it ("Offline Usage" instructions at bottom of page)
2. Using offline copy on offline computer, enter your seed as BIP39 Mnemonic
3. Select "BCH - Bitcoin Cash" as the "Coin"
4. All you addresses/private keys will be displayed at the bottom
5. Find the address you sent to, copy the matching private key
6. Either import or sweep that private key into the
BTC wallet of your choice.
Optional, but recommended: Create a completely new seed for your Ledger and send all your coins to the new wallets.