There are 2160 possible addresses, which is a ridiculous number. If every person on Earth makes ten addresses per second for 20 years (2x1018 total addresses), then the probability that at least two of these addresses collide is about 1.57x10-12.
There are a lot of address, it's true. But difficulty of generating one address is the same as calculating one hash. What if somebody starts to generate address using OpenCL with speed of 4 Ghashes? This guy can get 345600000000000 of addresses every day. 63072*1012
This is relatively slow, but remember - address is a limited resource. What if there are 1000000 people? What if in 10 years we can calculate 10 Thashes/sec easily?