I've worked out the following scheme:
1. Listen for rawtx topic using ZeroMQ in order to receive raw transaction body.
2. Parse binary body and extract addresses from output scripts.
3. Compare addresses with local database in order to find matches.
Of course, list of pending transactions could be loaded into memory entirely in order to prevent too much requests to the database. Each address takes 20 bytes of memory, so you could store ~50K of them in 1MB of RAM.
Is this an optimal way to monitor specific addresses?