How to speed up json/rpc calls ? to bitcoin core

[btctousd81]

hi,

i am trying to load bitcoin blockchain in mysql.,
and going through the simple json/rpc way.

at current speed , it will take about 1 year+ to process all blocks till today and all transactions.

i am using php to query data.

is there way to speed up the process ?
by using multiple threads ? i can do that.

how many parallel/multiple threads the bitcoin-core json/rpc can handle ?

anyone tried this approach before ?


Thanks for your time.

[CrazyCraig]

hi,

i am trying to load bitcoin blockchain in mysql.,
and going through the simple json/rpc way.

at current speed , it will take about 1 year+ to process all blocks till today and all transactions.

i am using php to query data.

is there way to speed up the process ?
by using multiple threads ? i can do that.

how many parallel/multiple threads the bitcoin-core json/rpc can handle ?

anyone tried this approach before ?


Thanks for your time.

If you don't mind me asking why are you trying to store everything in a MySQL database? One of the best things about bitcoin is the RPC itself. You can get all the data you need on demand.

Regarding you question, there are somethings that you can do to help. Try storing data in a message queue such as redis and use a cron to migrate the information to the database. That might help speed up as the driver isnt waiting on the response of the rpc itself.

[btctousd81]

hi,

i am trying to load bitcoin blockchain in mysql.,
and going through the simple json/rpc way.

at current speed , it will take about 1 year+ to process all blocks till today and all transactions.

i am using php to query data.

is there way to speed up the process ?
by using multiple threads ? i can do that.

how many parallel/multiple threads the bitcoin-core json/rpc can handle ?

anyone tried this approach before ?


Thanks for your time.

If you don't mind me asking why are you trying to store everything in a MySQL database? One of the best things about bitcoin is the RPC itself. You can get all the data you need on demand.

Regarding you question, there are somethings that you can do to help. Try storing data in a message queue such as redis and use a cron to migrate the information to the database. That might help speed up as the driver isnt waiting on the response of the rpc itself.

i know rpc is great, but i want to analize the data , i cant ask rpc to give me address having highest balance.
or lowest balance., or transactions sending 0 btc .

i am more fammiliar with mysql i can see data directly.,

i am working on site like blockexplorer so its not very feasible to send all requests to rpc , as rpc has some limitations on how many threads it can handle., while in mysql type relational dbs, its much easier to scale.

i have loaded balance data in mysql, but it took me 3-4 days , to try different libraries and benchmark which one is faster to process block.

current database is not 100% accurate so i want to parse blockchain again.

i have tried running rpc queries without inserting data in database and its quite slow. like avarage block takes 100 seconds to process all transactions and get all vouts and vin

so i am looking for something much faster. i can do parallel queries to bitcoind using json/rpc, but how many parallel queriees can bitcoind handle.,
i tried sending 100 queries at a time and debug.log got filled with

Code:

WARNING: request rejected because http work queue depth exceeded, it can be increased with the -rpcworkqueue= setting

[CrazyCraig]

The debug log told you how to allow more requests. Bump that setting up. You could also have more than one rpc client.

The task is repetitive. For you to find out if an address has a zero balance in Mysql you will need to parse the whole blockchain inside of your database. Essentially, processing through the whole dataset. Your essentially recreating a slower version of the rpc.

It might be more feasible to  query the rpc itself and avoid all the wasted processing power.

There might also be software out there that does this for you. I would seearch for that.

I can’t speak to how the block explorers do it but I believe they just query the rpc.

[btctousd81]

The debug log told you how to allow more requests. Bump that setting up. You could also have more than one rpc client.

The task is repetitive. For you to find out if an address has a zero balance in Mysql you will need to parse the whole blockchain inside of your database. Essentially, processing through the whole dataset. Your essentially recreating a slower version of the rpc.

It might be more feasible to  query the rpc itself and avoid all the wasted processing power.

There might also be software out there that does this for you. I would seearch for that.

I can’t speak to how the block explorers do it but I believe they just query the rpc.

nobody queries rpc for running site, mysql can handle 1k+ queries per second, and rpc cant even handle 100 queries per second.

i have already have

rpcworkqueue=128
rpcthreads=128
rpctimeout=220

this in my bitcoin.conf

and still got that

WARNING

You could also have more than one rpc client.

what do you mean by this ?
i tried running 100 threads simultaneously.


Thanks for your input.

[CrazyCraig]

nobody queries rpc for running site, mysql can handle 1k+ queries per second, and rpc cant even handle 100 queries per second.

i have already have

rpcworkqueue=128
rpcthreads=128
rpctimeout=220

this in my bitcoin.conf

and still got that

WARNING

You could also have more than one rpc client.

what do you mean by this ?
i tried running 100 threads simultaneously.


Thanks for your input.

just curious what are specs of the machine you are running this on?

Btw I am aware of the capabilities of MySQL. I'm trying to help you find an alternative method because of your issues.

[btctousd81]

nobody queries rpc for running site, mysql can handle 1k+ queries per second, and rpc cant even handle 100 queries per second.

i have already have

rpcworkqueue=128
rpcthreads=128
rpctimeout=220

this in my bitcoin.conf

and still got that

WARNING

You could also have more than one rpc client.

what do you mean by this ?
i tried running 100 threads simultaneously.


Thanks for your input.

just curious what are specs of the machine you are running this on?

Btw I am aware of the capabilities of MySQL. I'm trying to help you find an alternative method because of your issues.

sorry if my last reply was mean.

i was just trying to say, that mysql is very powerful vs., bitcoind queriying the blk files or chainstate.


my machine is
fx 8350
32 gigs ddr3 1600
asus gaming motherboard.
latest centos 7 x64 , minimal + bitcoind

thanks for your time.

[CrazyCraig]

sorry if my last reply was mean.

i was just trying to say, that mysql is very powerful vs., bitcoind queriying the blk files or chainstate.


my machine is
fx 8350
32 gigs ddr3 1600
asus gaming motherboard.
latest centos 7 x64 , minimal + bitcoind

thanks for your time.

I didn't realize you were running this from your local computer. You may have a connection limit / bandwidth limit set in your operating system or from your ISP. The RPC and DB might also be battling for processing power.  I suggest, if you can, using a headless node on a cloud provider.

You could also try fine tuning the RPC.
- Set max connections & max buffer
- Up the threshold
- Increase the memory requirements.

Optimize MySQL
- Configure max threads / memory / etc.
- Verify storage engine meets your needs.

MySQL is very powerful. Every application is different and while I do not know the exact purpose of yours I was recommending you avoid storing data where you can and directly interface with the blockchain.

Perhaps you could do the following:
- Start X blocks back and store from that point forward.
- Store data and chain state on-demand. Query your database and if the data doesn't exist create it from RPC calls.

Take a look at the following block explorers. One uses a database and the other doesnt.
- Bitpay Insight API (Bitpay Bitcore)
- Iquidus Explorer

[pebwindkraft]

assuming you have a fully sync'd client, you can read-in directly the blk*.dat files into mysql, without the need for the JSON API:

https://github.com/bitcoin-abe/bitcoin-abe/blob/master/README-MYSQL.txt

[btctousd81]

assuming you have a fully sync'd client, you can read-in directly the blk*.dat files into mysql, without the need for the JSON API:

https://github.com/bitcoin-abe/bitcoin-abe/blob/master/README-MYSQL.txt

i read initial bitcoin-abe sync takes month+

[Anti-Cen]

assuming you have a fully sync'd client, you can read-in directly the blk*.dat files into mysql, without the need for the JSON API:
https://github.com/bitcoin-abe/bitcoin-abe/blob/master/README-MYSQL.txt

Getting direct access to blk0001.dat is all very easy until you try doing from windows without running
a full node and doing RPC that is real slow from what I am picking up.

I think the file format needed is LevelDB but i cannot even get a LeveDB programs in C++ to compile in Visual Studio
that I can then use from C# or even find a windows based copy of LevelDB.dll to Pin.Invoke from C#

Hopefully i can hit the ground running once I can read the block .dat files but for now i am stuck in the mud
and much of the code hanging around the internet that works in windows is well out of date anyway.

Just reading Peers.dat for now would be nice 

[btctousd81]

assuming you have a fully sync'd client, you can read-in directly the blk*.dat files into mysql, without the need for the JSON API:
https://github.com/bitcoin-abe/bitcoin-abe/blob/master/README-MYSQL.txt

Getting direct access to blk0001.dat is all very easy until you try doing from windows without running
a full node and doing RPC that is real slow from what I am picking up.

I think the file format needed is LevelDB but i cannot even get a LeveDB programs in C++ to compile in Visual Studio
that I can then use from C# or even find a windows based copy of LevelDB.dll to Pin.Invoke from C#

Hopefully i can hit the ground running once I can read the block .dat files but for now i am stuck in the mud
and much of the code hanging around the internet that works in windows is well out of date anyway.

Just reading Peers.dat for now would be nice  

the trouble in reading blockfiles or chainstate db is , core developers keep changing the internal format., so you must be pro in c,c++ to read and understand from core source files.

[Anti-Cen]

the trouble in reading blockfiles or chainstate db is , core developers keep changing the internal format., so you must be pro in c,c++ to read and understand from core source files.

this is true and most of what I have seen with Bitcoin core sure does look over complicated and has lead
to lots of programs having to use REST API's on-line which is not really the way to go.

I will get to were I want to go and do it in a way that I want to do it

[btctousd81]

the trouble in reading blockfiles or chainstate db is , core developers keep changing the internal format., so you must be pro in c,c++ to read and understand from core source files.

this is true and most of what I have seen with Bitcoin core sure does look over complicated and has lead
to lots of programs having to use REST API's on-line which is not really the way to go.

I will get to were I want to go and do it in a way that I want to do it

if you dont mind spending some bucks ,then i heard user amaclin had some cpp code to read the blk files and he is pro too in terms fo bitcoin protocol.

[tulerpeton]

Any news about running a fast blockchain dump using json/rpc ?
It's taking month, like something is locked and calls serialized.

[btctousd81]

Any news about running a fast blockchain dump using json/rpc ?
It's taking month, like something is locked and calls serialized.

there is method called "batch json" in in, you can combine multiple json requests in single query., it should speed up the reply.

[copyleft]

So, I'm running bitcoind locally.

I'm finding it to be veeery slow to work with on my local machine. After doing soem googling, I see others have had the same issues...

The REST interface is faster, but there's no interface for retrieving UTXOs from a given address... which would mean querying the slow JSON RPC client to run getaddress or hitting an external API (which defeats the purpose of running bitcoind myself).


Is there a faster alternative to querying bitcoin without querying any external APIs?

I don't mind the long waittimes for broadcasts, I guess, but for address balance checking and constructing transactions from UTXOs, I need milliseconds. I tried putting up the settings in my bitcoin.conf, and the lag still continues. I put in timeouts in my app, but it's its lagging on one request... imagine thousands per second.

I reaaaaallly don't want any external API's in this. Perhaps there's a way to consume the downloaded blockchain data and construct my own database from it, tuning for more performant SQL queries?

in the meantime, I'm going to try bitcore from bitpay.

[DaveF]

Short version: RPC is slow.

Longer Version: Take a look at this thread, it's been discussed many times before: https://bitcointalk.org/index.php?topic=5159374

If you don't want to read the other thread: You are better off pulling the data into a DB and pulling the data from a DB. RPC calls lock the bitcoind database so they are done sequentially and slowly.

-Dave

[btctousd81]

for those, who are really looking to speed up data.

for my another project , what i did is , i modified bitcoin-core files and added my custom code., it was fucking pain in the ass for me, as i was noob in c,c++ but with the help of friend and some online tuts, i managed to do what i wanted.

you need to do the same.

its performance is way better than json rpc calls.

[Didldak]

So I am guessing that you are syncing chainstate to mysql directly from bitcoind code?