I think I have solved it, but it still confuses me a lot.
When I first got this ledger, I have now remembered that I saved a photo of the seeds to my secure folder(mistake #1). I think there must be some malware on the phone, because I have never entered my seeds onto my computer or anywhere else and I don't ever remember opening the image. They must have gotten the 4 digit passcode from monitoring my phone when I used it with other apps, for example some apps are unlocked with the same 4 digit code I use to unlock my ledger(mistake #2). This is the only way I can see that they have accessed it. I've run anti virus etc on there but it's found nothing. I also checked all the installed apps and there is nothing strange here. In fact this phone is rarely used and only really for whatsapp and google authenticator. For this to be true, it would mean someone has had access to my phone for over a month which is very worrying
Physically, the seeds are hidden in my house. I live alone and no one else has access, certainly no one I know knows my passcode or understands bitcoin, or even knows I own a ledger.
$5000 is a nice expensive lesson in security I guess. Thanks all for any replies and help
I'm off to buy a new phone