Bitcoin Forum
May 01, 2024, 03:01:33 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Faster rpc calls to bitcoin client?  (Read 568 times)
johnjacksonbtc (OP)
Newbie
*
Offline Offline

Activity: 20
Merit: 0


View Profile
September 17, 2016, 03:33:27 PM
 #1

I have built my own blockchain explorer that does not have any crypto and does a lot of rpc calls. When it comes to 1MB block full analysis, each txin value is resolved and it does thousands of rpc calls. Any tips to get it faster than < 10 calls per sec?
1714575693
Hero Member
*
Offline Offline

Posts: 1714575693

View Profile Personal Message (Offline)

Ignore
1714575693
Reply with quote  #2

1714575693
Report to moderator
1714575693
Hero Member
*
Offline Offline

Posts: 1714575693

View Profile Personal Message (Offline)

Ignore
1714575693
Reply with quote  #2

1714575693
Report to moderator
1714575693
Hero Member
*
Offline Offline

Posts: 1714575693

View Profile Personal Message (Offline)

Ignore
1714575693
Reply with quote  #2

1714575693
Report to moderator
Each block is stacked on top of the previous one. Adding another block to the top makes all lower blocks more difficult to remove: there is more "weight" above each block. A transaction in a block 6 blocks deep (6 confirmations) will be very difficult to remove.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1714575693
Hero Member
*
Offline Offline

Posts: 1714575693

View Profile Personal Message (Offline)

Ignore
1714575693
Reply with quote  #2

1714575693
Report to moderator
1714575693
Hero Member
*
Offline Offline

Posts: 1714575693

View Profile Personal Message (Offline)

Ignore
1714575693
Reply with quote  #2

1714575693
Report to moderator
achow101
Moderator
Legendary
*
Offline Offline

Activity: 3374
Merit: 6571


Just writing some code


View Profile WWW
September 17, 2016, 03:37:25 PM
 #2

How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704.

johnjacksonbtc (OP)
Newbie
*
Offline Offline

Activity: 20
Merit: 0


View Profile
September 17, 2016, 05:41:01 PM
 #3

How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704.

Currently two getblock calls are made for verbose param false and true, then getrawtransaction for each transaction, and then getrawtransaction for each transaction input txid. If getrawtransaction fails (for example block 0), then decoderawtransacion call. Most waiting goes if getrawtransaction is called recursively till generation transactions remains.
achow101
Moderator
Legendary
*
Offline Offline

Activity: 3374
Merit: 6571


Just writing some code


View Profile WWW
September 17, 2016, 06:11:22 PM
 #4

How are you getting the details? Are you calling getblock and then getrawtransaction for each transaction? If so, see https://github.com/bitcoin/bitcoin/pull/8704.

Currently two getblock calls are made for verbose param false and true, then getrawtransaction for each transaction, and then getrawtransaction for each transaction input txid. If getrawtransaction fails (for example block 0), then decoderawtransacion call. Most waiting goes if getrawtransaction is called recursively till generation transactions remains.
That is horribly inefficient. You should not be relying on Bitcoin Core for all of that. Rather, you should be calling getblock once and then getrawtransaction for the each transaction in the block. Then you should be storing data in a separate database which you can access much more quickly. Relying on Bitcoin Core's RPC to get you a lot of data quickly is a terrible idea. You should also batch the getrawtransaction calls so that you aren't sending it thousands of times but just once.

Also, you can try apply the patch in the PR I linked and use the extraVerbose option it provides to get you all of the transaction data in a getblock call.

johnjacksonbtc (OP)
Newbie
*
Offline Offline

Activity: 20
Merit: 0


View Profile
September 17, 2016, 07:45:33 PM
 #5

That is horribly inefficient. You should not be relying on Bitcoin Core for all of that. Rather, you should be calling getblock once and then getrawtransaction for the each transaction in the block. Then you should be storing data in a separate database which you can access much more quickly. Relying on Bitcoin Core's RPC to get you a lot of data quickly is a terrible idea. You should also batch the getrawtransaction calls so that you aren't sending it thousands of times but just once.

Also, you can try apply the patch in the PR I linked and use the extraVerbose option it provides to get you all of the transaction data in a getblock call.

I haven't found more efficient database than bitcoin-qt client itself, separate database solution was even more horrible. I am currently sitting on 0.12.1 addrindex patched version to avoid any more separate database solutions. No offense but making uncommon features mandatory is even more terrible idea when plan is to release software that is maximally compatible with anything (even with pruned nodes) to make it useful for common users. During testing I discovered node rpc endpoint does not take advantage from http pipelining, best shot is http keep alive feature that speeds things a bit up.

PS. Simply preparing to steal interest from centralized bitcoin explorer services.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!