But isn't that public now, and can't that be reused by someone else as proof they can spend my account's remaining bitcoins?
One or more public keys are made public at the point of "spending" funds - this is still no real risk as the private key is never seen (except by yourself) although if any weakness is found with ECDSA (or some future quantum computer is built to discover private keys from public ones) then potentially any remaining funds at that address could be at risk.
It is for this reason that you are advised never to re-use an address.