Someone could create Electrum-like protocol, server and client so you don't need to download whole blockchain. But it'll be terrible for privacy even if they use VPN/Tor.
I can think of more problems: centralization with just a few nodes, different nodes will have different "mempools", some nodes might apply censorship and spam will be rampant. Although I remember a post (probably from theymos) explaining that spam could be handled in a similar way as the Trust system, where each user has a list of users who's posts are considered spam.
Let me put it this way: there's probably a reason why nobody uses it
