Do you run your own stake pool or have you delegated your stake in someone's pool? Delegation can be carried out using ADA's 2 official wallets : Daedalus and Yoroi so you are safe to use Yoroi provided you have downloaded them from the cardano.org link or from the Play Store. Please note at the time of this writing google provides links to various phishing sites of Yoroi but the official website is
https://yoroi-wallet.com/ so kindly double check while using it.
You need to understand while the delegation process your unspent ADA will move from your address to your change address to increase the privacy. You are seeing the ADA in your wallet because they stay in change address and are present in your wallet and you are safe here. You can read more about ADA's Change address here :
Understanding Unspent Transaction Outputs in Cardano. AFAIK if you see your funds in your wallet, it's safely sitting in Change Address so you don't need to worry about it.