Advice on service architecture

Hello Bitcointalk!

I'd like to ask advice on how best to monitor and keep track of wallet transactions. I'm sure this has been talked about a lot before so feel free to point me to other threads, too.

We're designing a centralized, "pooled", web-wallet-esque service. The centralized, pooled nature of this system is critical as it will enable off-blockchain transactions and thus can support micro-transactions (by not having to deal with the relatively high miner fees for individual transactions).

We'll need to track incoming Bitcoin transactions on a per-address basis (to credit the appropriate account/user). Also need to be able to track all historical transactions (in case of a need to reconstruct or verify past Bitcoin transactions.

Right now we're using a stock 0.9.x bitcoind as a hot wallet using simple JSON RPC calls to implement this. Specifically we're regularly polling the listtransactions RPC in a "paged" manner to watch for new incoming and confirmed transactions. Those Bitcoin transactions are recorded in another database whereby accounts are credited and debited appropriately based on the blockchain changes. The polling is "smart" in that it only takes action on new transactions it hasn't seen before and will only advance to the next listtransactions "page" if it's necessary. This is an attempt to keep the polling overhead as low as possible and to be quick and speedy. So far it seems to be working well enough and the code is small and easy to follow.

We have plans to integrate with the walletnotify command to further lessen our dependance on polling, but the core listtransactions monitoring is what would still be used, even if walletnotify is used to trigger it.

Our initial approach was to do everything manually. Essentially re-implement a wallet by using getrawmempool, getrawtransaction, getblock, etc. and manually tracking relevant UTXOs, private key storage and escrow, transaction generation, etc. While we got pretty close we found that there's so many edge cases around getting a full wallet implementation correct (valid transaction/txn scripts, malleated txns, forking/reorgs, confirmations, etc.) and we ran into performance issues having to parse the whole blockchain over RPC calls. The stakes are high (real people's hard earned money) and didn't want our inexperience to put that in jeopardy. We now think it's quicker and much, much safer for us to rely on the reference client's wallet since it already handles all of that and is well tested, etc.

So I'd like to ask: is this a reasonable approach? Are there big no no's with this what we're doing here? Any scalability issues with any of this? We hope to scale to somewhere between 10,000 to 100,000 users or so using a single bitcoind instance which implies 50,000 to 500,000 Bitcoin addresses, perhaps multiple hundreds of thousands of transactions, etc. Are those numbers doable with a standard Bitcoind? We assume we'll need to engineer a more custom, scalable architecture (or perhaps scale horizontally using bitcoind) in the future. Perhaps based on Bitpay Insight or something like it. But to get our idea functioning and off the ground this does this seem like a workable, reliable, reasonable, and secure approach?

We're very concerned about security. We'll use an encrypted wallet for all of this but it seems difficult to protect the passphrase well enough in the system so as not to make it trivial to hack (assuming a malicious user had access to the server(s)) by doing something like proxying bitcoind RPC traffic and gobbling up the unlock passphrase as just one example. We'll be using hot/cold wallet proportioned reserves to mitigate the hot-wallet attack vector. But is there anything else that can be done? Does the wallet encryption support OTP or any other challenge-based authentication? Any other tips in general on being as secure as possible with the walletpassphrase RPC? One thought we had was to have a transaction-sending bitcoind only. I.e. the txn monitoring bitcoind has a syncrhonized wallet (somehow) but it never unlocks it - the system for sending Bitcoin is entirely different, purpose built, and because of that it's attack vectors can be better controlled. But not sure.

We don't have a ton of experience with Bitcoin or Bitcoind so wanted to engage you fine folks with our thought process here.

Thanks for any help!