So is it JUST down to possibility? Everyday that possibility gets less as more people join and more keys are made, I have bigger chance of Hitting a wallet with a random key?
Yes. The only security is the extremely unlikely possibility that you can guess a private key. And yes, that possibility gets slightly lower as more and more Bitcoin addresses are used (since you only have to guess one of them instead of one particular key). But there are about 10^77 possible private keys. For comparison there is something like 10^78 to 10^82 atoms in the visible universe. The security comes from the very large size of the number.
But to be clear, if someone can guess a private key, they can access the Bitcoin associated with the Bitcoin address generated from that private key. They don't need anything else.