Correct my misunderstanding then:
Private keys are intended to be one-to-one with public keys, so that would certainly be a flaw in ECDSA if two private keys correspond to one public key, but since you turn the 256 bit public key into a 160 bit digest, it would just be incredibly unlikely, not impossible, for an ideal hash function to map two different inputs 256 bit to a single 160 bit output.
What's wrong with my understanding?
There is nothing wrong with your understanding.
When I say:
the only way you "
could" do something
I mean it in the same way as when I say:
"The only way all the air molecules in the room
could spontaneously collect in one corner instantly suffocating everyone in the room is if there is a flaw in our current understanding of physics."
Sure, there is an "incredibly unlikely, not impossible" probability of it happening, but in terms that the typical human being understands it can't happen.
Meanwhile when you say "not impossible", you are simply using a non-zero probability number (regardless of how extremely small that number is) as an indication of "possibility".