Well ...
if someone has access to your teamviewer-logindata, that person could load files on your pc and execute them.
If there were known exploits for random teamviewer-takeovers, you would have read about it.
There are tons of ways that a stealer / keylogger could do damage.
Even if someone changed the receive-address you are mining to it would make you mine for nothing until you notice.
Unlikely but well ... you never know who is on the other end.