Isn't the point that as long as we use sufficiently large key sizes, it doesn't matter?

No, because ECDSA and RSA are based on problems that are considered hard in today's mathematics. That does not preclude them from being easy in future mathematics. The underlying assumptions of discrete logarithms and integer factorizations are that they will remain hard, but there is no guarantee.

And then of course the whole quantum computing thing.

My point was that a successfully retrieving a private key from a public key isn't a problem if it's done by brute force (and not some novel new way that reduces the hardness of that operation), and the key sizes involved are significantly smaller than what we use.