Your wallet may not be aware of every 0-confirm transaction is the network is large enough but every confirmed transaction is part of the block chain. The main wallet needs the entire block chain to validate transactions.
There are lightweight wallets which rely on a trusted server to handle the work of parsing the blockchain. As the blockchain grows that may be an option for more users. There are also discussion on protocols which would allow lightweight clients to get "assistance" from the network of full clients in order to eliminate the trust issue.