What do you mean that you are trying to retrieve them? Are you trying to send them from your address on your Nano S to somewhere else?
I managed to find this on Ledger's reddit page from last year >
https://www.reddit.com/r/ledgerwallet/comments/it5f4r/bug_report_when_i_send_xrp_held_until_escalated/.
That user reported seeing the same error. However, his XRP was still sent. Can you check your address on a block explorer to see if the transaction was performed despite the error?
One more thing. Are you aware that XRP addresses have a minimum reserve requirement? This reserve was 25 XRP in the past. I think they lowered it to 10 XRP in the meantime. What that means is that those 10 XRP will always have to stay on your address and you can never spend them. They don't belong to you. Ripple says it's a measure against spam.
So if you have 20 XRP on an address, only 10 of those are yours to spend. If that address of yours holds only 10, you wont be able to spend those. It has got nothing to do with Ledger. It's the way the Ripple network was set up.
