BIP39 Passphrase is also known as the 13th/25th word, which is encrypted inside the device, Ledger confirmed that they do not copy the passphrase, but
Ledger team lied several times, so since it is encrypted alongside with 12th word, it is best to assume that it is will be sent to the 3rd party servers.
Using a memorable Passphrase does not give you a good entropy and therefore you should not feel safe if your wallet is hacked.