Current implementations of the bitcoin client download the entire block chain, but future versions will not have to. There is a technique called Merkle tree (
http://en.wikipedia.org/wiki/Hash_tree) that is used in the system which allows you to delete old and not-relevant-to-you transactions. It has the potential to scale extremely well.
I don't think any implementation does this yet, but I might be wrong.