|
Title: Faster rpc calls to bitcoin client? Post by: johnjacksonbtc on September 17, 2016, 03:33:27 PM I have built my own blockchain explorer that does not have any crypto and does a lot of rpc calls. When it comes to 1MB block full analysis, each txin value is resolved and it does thousands of rpc calls. Any tips to get it faster than < 10 calls per sec?
Title: Re: Faster rpc calls to bitcoin client? Post by: achow101 on September 17, 2016, 03:37:25 PM How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704.
Title: Re: Faster rpc calls to bitcoin client? Post by: johnjacksonbtc on September 17, 2016, 05:41:01 PM How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704. Currently two getblock calls are made for verbose param false and true, then getrawtransaction for each transaction, and then getrawtransaction for each transaction input txid. If getrawtransaction fails (for example block 0), then decoderawtransacion call. Most waiting goes if getrawtransaction is called recursively till generation transactions remains. Title: Re: Faster rpc calls to bitcoin client? Post by: achow101 on September 17, 2016, 06:11:22 PM How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704. Currently two getblock calls are made for verbose param false and true, then getrawtransaction for each transaction, and then getrawtransaction for each transaction input txid. If getrawtransaction fails (for example block 0), then decoderawtransacion call. Most waiting goes if getrawtransaction is called recursively till generation transactions remains. Also, you can try apply the patch in the PR I linked and use the extraVerbose option it provides to get you all of the transaction data in a getblock call. Title: Re: Faster rpc calls to bitcoin client? Post by: johnjacksonbtc on September 17, 2016, 07:45:33 PM That is horribly inefficient. You should not be relying on Bitcoin Core for all of that. Rather, you should be calling getblock once and then getrawtransaction for the each transaction in the block. Then you should be storing data in a separate database which you can access much more quickly. Relying on Bitcoin Core's RPC to get you a lot of data quickly is a terrible idea. You should also batch the getrawtransaction calls so that you aren't sending it thousands of times but just once. Also, you can try apply the patch in the PR I linked and use the extraVerbose option it provides to get you all of the transaction data in a getblock call. I haven't found more efficient database than bitcoin-qt client itself, separate database solution was even more horrible. I am currently sitting on 0.12.1 addrindex patched version to avoid any more separate database solutions. No offense but making uncommon features mandatory is even more terrible idea when plan is to release software that is maximally compatible with anything (even with pruned nodes) to make it useful for common users. During testing I discovered node rpc endpoint does not take advantage from http pipelining, best shot is http keep alive feature that speeds things a bit up. PS. Simply preparing to steal interest from centralized bitcoin explorer services. |