Quantum computer is expected to have the capability to break a 256 bit encryption within seconds. However, it has not yet become a reality, only a certain degree of it have been achieved!
Now, I would agree to the point you are trying to establish! Having an weapon doesn't always mean it will be misfired and used against the humanity! So even if the quantum computer is built so capable enough to break the encryption of bitcoin network, it is least likely to be used for this purpose!
Although it was expected to crash bitcoin's privacy and security, Until now, not a single move was done against bitcoin, IBM's latest quantum computers are just about to be launched around October and it was said that this isn't the fastest computer that will ever be made. The fastest qunatum computer that is existing so far is the one that Google has developed and is said that has the quantum supremacy. despite of it's existence, not a singke damage has been done so there's nothing to be worried about.