You could at least use some other addresses for payments and can-do alibi that it's from DEX, any platform or from P2P. I'm for sure he's just jealous of you and now you're making money, that kind of people should be cut off for your peace of mind, it's not healthy in the long run too. There are many ways to use BTC too, it's not about using your main only, that's why we can have more wallets for different purposes, maybe it's about you who forgot there are things exist and can escape the situation easily but you forgot that crypto is very deep, so you can explain it to him without showing your main balance. You're the one who used that address and expecting to have privacy, at least you've learned your lesson and use some burner wallets for P2P.
Story of my life, I have learned a lesson, i do wish this happens from someone's end and not this particular person because he has a very loose mouth, I moved asset already but it's not going to be the end as few more people will know, like I've said in my OP I wasn't expecting this to happen or I should say I let down my guard, forgetful about the person I was dealing with. Thanks