The RPC interface certainly is not the fastest. There is a C++ blockchain parser that someone wrote that you could base your system on. If you want a really simple and quick solution that depends on a central server: use blockchain.info's api. It can be queried for all txs regarding a certain address.
On another node I wrote a python script to parse through all txs in all blocks, does some calculations and stores everything back into a database - it takes a while but works fast enough. The biggest hurdle for me was that I ran out of network sockets since somehow bitcoin-qt or my python rpc interface never closed them - so I had to tweak some parameters of linux network stack. Other than that look ups and writes to my database were much slower than the RPC interface.