Yea I'd say nano bots are a legitimate cause for concern. I recently seen a vid discussing governments making bugs spy on others buy controlling their flight and using tiny cameras. Also add in the advancement of the robotic technology and quantum computing I could imagine such a thing as nano bots but I'd suspect they would only be able to do limited tasks in my lifetime.
there is worst than that, implementing chips on bugs and controlling them (if this is left without constraint and legislation bugs are going to be only the begining, I see that implemented in animals next and then to humans )
Some good points you've brought up for sure.
I'd also like to mention that they will take over by doing the jobs that people do for example factory work and transportation comes to mind.