Reading over the bitcoind API, the only way I can see how to sync new transactions in the wallet with the state of my bitcoin-oriented web app is to read the entire wallet transaction history, then compare each transaction with a database I maintain in the app to see whether it has already been processed or not.

This seems ugly, difficult to code and also prone to scaling issues (going through each and every transaction is fine when it's 50, but what about 1000, 250,000 )

Maybe I'm just really clueless and failing to see an elegant solution.

Well, then the bitcoin client might have to mirror all the functionality that's available when doing SQL queries against a full fledged RDBMS.

I don't really see how, if you're talking about scalability issues that would get solved by nice bitcoin callbacks.

Okay, so here's how bitcoin central handles it.
It polls the client at regular intervals and does exactly what lucky said, it INSERTs unknown transactions, and updates the confirmations of the known ones (and no, the tx_id is not defined as unique )

It polls the client for each account, not for a global listtransactions (which doesn't really seem to work anyway AFAIK).

Yes, this approach is sub-optimal, but IMO it's the only one that allows enough flexibility in the ways you want to deal with the data.

I'm completely conscious of the limitations and possible security flaws of this approach, and that's why the bitcoin central code *also* takes advantage of the accounts functionality, providing a second security layer.

That, IMHO is the best possible approach for a web app.