It's possible... but involves effectively compromising your entire Ledger wallet... as you would need to input your hardware wallet 24 word seed mnemonic into a tool like:
https://iancoleman.io/bip39/ (NOTE: click "BIP49" to see your SegWit addresses).
Theoretically, if you can find a VTC wallet that supports SegWit and will let you import the private key, it *might* work. But I'm not sure if any given VTC wallet will generate the same "3" address from the same private key... I tried a quick test with Vertcoin Core... generated a random 24 word seed mnemonic on the BIP39 mnemonic code convertor... clicked BIP49, 1st address given was:
m/49'/0'/0'/0/0 3M81vpyFKD2Wz2vySnuZZANNcqkk6goUUp L18efys5GwdKs8KwMufC3ukMms5yUiuqtFpw2DBsFW5pLgewrpMV
used "importprivkey L18efys5GwdKs8KwMufC3ukMms5yUiuqtFpw2DBsFW5pLgewrpMV" command in the debug console in Vertcoin Core... got address: VufLUihJAMZ3u6ytndSFpZXPDQrV16ePpW
Then used "addwitnessaddress VufLUihJAMZ3u6ytndSFpZXPDQrV16ePpW" command in the debug console and got SegWit address: 3M81vpyFKD2Wz2vySnuZZANNcqkk6goUUp
So, it would appear you should be able to generate the same "3" address as your Ledger SegWit BTC address... which means, in theory anyway, you should be able to rescue the VTC.
NOTE: Even if done "offline", you should really consider your seed mnemonic to be compromised and take appropriate steps to ensure the security of your coins. (ie. move all your coins to new wallets, reset the ledger and then move all coins back etc)