How to improve bitcoind signing performance?

[dave111223]

Currently using bitcoind 0.10 headless on Debian (although had the same issue with all previously bitcoind versions).

The wallet info:
Wallet.dat size: 160MB
Number of addresses: ~16,000
Unspent outputs: (varies but around: 20-40)

The signing function is quite sluggish.   Even for simple 1 or 2 UTXO the signing takes around 2-5 seconds.
With more complex transactions the signing can take more than 30 seconds!

Is there any way to speed up this process (preferably without having to do UTXO consolidation transactions, and just via a setting change)

Load on the server seems fairly low, and CPU usage is very low.
Memory: 4GB
load average: 0.39, 0.43, 0.35

[1713860009]

1713860009

[1713860009]

1713860009

[1713860009]

1713860009

[1713860009]

1713860009

[1713860009]

1713860009

[dave111223]

Here are some specific examples of this sluggishness:

Code:

$ time /path/to/my/bitcoin/bitcoin-cli getbalance "" 3

real    0m3.398s
user    0m0.008s
sys     0m0.004s

Code:

$ time /path/to/my/bitcoin/bitcoin-cli listunspent

real    0m4.213s
user    0m0.008s
sys     0m0.004s

The times are consistently 3-4 seconds every time

But then some commands are much quicker:

Code:

$ time /path/to/my/bitcoin/bitcoin-cli getinfo

real    0m0.039s
user    0m0.004s
sys     0m0.004s

[achow101_alt]

Here are some specific examples of this sluggishness:

Code:

$ time /path/to/my/bitcoin/bitcoin-cli getbalance "" 3

real    0m3.398s
user    0m0.008s
sys     0m0.004s

Code:

$ time /path/to/my/bitcoin/bitcoin-cli listunspent

real    0m4.213s
user    0m0.008s
sys     0m0.004s

The times are consistently 3-4 seconds every time

But then some commands are much quicker:

Code:

$ time /path/to/my/bitcoin/bitcoin-cli getinfo

real    0m0.039s
user    0m0.004s
sys     0m0.004s

This probably happens because you have about 16000 addresses, and listunspent and getbalance need to scan for all of the unspent outputs and calculate the balance of all 16000 addresses. This is why it takes a few seconds. getinfo on the otherhand, does not need to scan through all of your addresses, and already has the data that it needs for the output.

[dave111223]

But why is it so inefficient?

A Mysql sum on 16,000 records takes fractions of a second on very low spec hardware....
So why does bitcoind take 3-4 seconds? (on very nice hardware)

[achow101_alt]

But why is it so inefficient?

A Mysql sum on 16,000 records takes fractions of a second on very low spec hardware....
So why does bitcoind take 3-4 seconds? (on very nice hardware)

bitcoind is slower because there are not just 16000 things to look through. It must search through all 34 gigs of blockchain and find all of one addresses transactions, add up the inputs and outputs, and find which transactions have yet to be spent. Then it must repeat this for all 16000 of your addresses in order to return the data you asked for. This is why it takes so long.

[dave111223]

bitcoind is slower because there are not just 16000 things to look through. It must search through all 34 gigs of blockchain and find all of one addresses transactions, add up the inputs and outputs, and find which transactions have yet to be spent. Then it must repeat this for all 16000 of your addresses in order to return the data you asked for. This is why it takes so long.

Why would it need to search from all 34GB?  Caches and indexes are not used?

[achow101_alt]

bitcoind is slower because there are not just 16000 things to look through. It must search through all 34 gigs of blockchain and find all of one addresses transactions, add up the inputs and outputs, and find which transactions have yet to be spent. Then it must repeat this for all 16000 of your addresses in order to return the data you asked for. This is why it takes so long.

Why would it need to search from all 34GB?  Caches and indexes are not used?

AFAIK Bitcoin Core keeps an index (or something like that) of all of the transactions for the addresses so it may not look through the entire blockchain. However, getbalance and listunspent must still iterate through that list for each of the 16000 addresses in order to find which ones are still unspent and what the total balance is.

[dave111223]

AFAIK Bitcoin Core keeps an index (or something like that) of all of the transactions for the addresses so it may not look through the entire blockchain. However, getbalance and listunspent must still iterate through that list for each of the 16000 addresses in order to find which ones are still unspent and what the total balance is.

So what I'm saying is, with a mysql database, even of size 1,000,000 records you can SUM the balances of 16,000 records in fractions of a second.
I don't see why bitcoind cannot achieve the same performance?

I mean literally if I imported the data from bitcoind into mysql (a list of all my addresses and transactions), it would take me millionseconds to do these same queries?

[achow101_alt]

AFAIK Bitcoin Core keeps an index (or something like that) of all of the transactions for the addresses so it may not look through the entire blockchain. However, getbalance and listunspent must still iterate through that list for each of the 16000 addresses in order to find which ones are still unspent and what the total balance is.

So what I'm saying is, with a mysql database, even of size 1,000,000 records you can SUM the balances of 16,000 records in fractions of a second.
I don't see why bitcoind cannot achieve the same performance?

I mean literally if I imported the data from bitcoind into mysql (a list of all my addresses and transactions), it would take me millionseconds to do these same queries?

At this point, I am not entirely sure. But, I believe that it has something to do with verifying each transaction e.g. checking the signature and scripts, however, as I said, I am not sure. I recommend that you take a look at the code and see if you can find the issue yourself. You can find the source at https://www.github.com/bitcoin/bitcoin and there are developer docs, which are easier to follow, at https://dev.visucore.com/bitcoin/doxygen/. The specific methods you are looking for are located in src/wallet/rpcwallet.cpp. If you do find that there is an issue with performance with no reasonable explanation, please open an issue with the developers, and perhaps write a patch and submit that to the core devs.