Bitcoind SLOOOOW

[Kazu]

My Bitcoind is really slow. I'm trying to to start a service using it a bit (by a bit, I mean really not that much) and for some reason its using up 40%+ of my CPU all the time. I suspect it is slowing down my page loads as they are really quite slow. Has anybody else ran into this problem, and if so, do they have any workarounds/alternate software (I still need account management & all other bitcoind features but mining)/other fixes?

Thanks tons!

[Kazu]

I have no problem with having the first starting time being slow - I can wait. The part that is annoying is that functions that must be repeated on nearly every pageload, like getbalance, seem to be absurdly slow. Is there any way of speeding those up?

[btcsql]

The solution is simple: don't run Bitcoind on the same server as your webserver. It's much faster to have it dedicated, and simply connect/issue commands/grab data via API format.

[niko]

The solution is simple: don't run Bitcoind on the same server as your webserver. It's much faster to have it dedicated, and simply connect/issue commands/grab data via API format.

This should provide for more security, too. Sorry, I know these are not answers to your question, it's a workaround. However, it's a good workaround.

[Belkaar]

You should also consider getting the needed info via a cron job an saving it to a database that the frontend uses. This decouples page loading times from API-processing.

[Kazu]

I sort of need the pageload connected with the API because the balance can't be on a delay for what I'm doing. Thanks for the idea, though.

I'm considering running bitcoind on a dedicated server, however that seems like a pretty big waste of resources. Is there any way of viewing what Bitcoind is doing all this time, like viewing some sort of output? I just want to see why its taking so incredibly long.

[Kazu]

So I tried using a  microtime() difference to calculate the time it takes to execute a bitcoind getbalance on an account with a couple hundred transactions. I have tried two methds:

php system() to call bitcoind
Bitcoind JSON rpc connection

And...

EXEC: 222.71010398865
EXEC: 12.618118047714
EXEC: 3.6932117938995
EXEC: 1.6202571392059
EXEC: 1.7495849132538
EXEC: 3.5270199775696
EXEC: 3.0422019958496
EXEC: 2.6602010726929
EXEC: 1.5538840293884
EXEC: 0.58772110939026
EXEC: 0.97496891021729
EXEC: 0.99640703201294
EXEC: 2.7791039943695
EXEC: 1.8489720821381
EXEC: 0.82930707931519
EXEC: 0.25357699394226
EXEC: 0.62351679801941
EXEC: 1.3590130805969
EXEC: 4.4388699531555
EXEC: 0.70736193656921
EXEC: 1.473837852478
EXEC: 0.8196861743927
EXEC: 0.49087500572205
EXEC: 0.31272411346436
EXEC: 0.3066029548645
EXEC: 0.93807220458984
EXEC: 2.9650371074677
EXEC: 4.9273610115051
EXEC: 2.4882531166077
EXEC: 1.2758350372314
EXEC: 0.81832599639893
EXEC: 1.2282900810242

JSON: 0.70425081253052
JSON: 0.31246995925903
JSON: 6.2438280582428
JSON: 0.75746512413025
JSON: 4.1969089508057
JSON: 0.15843486785889
JSON: 1.0560338497162
JSON: 4.0856080055237
JSON: 0.34556603431702
JSON: 9.2187669277191

I've heard of inefficient processing but this is just ridiculous. I've got to be doing something wrong, there is no way this can be normal.

Here is my PHP:

JSON:
require_once 'jsonRPCClient.php';
$bitcoin = new jsonRPCClient('http://bitcoinrpc:reallylongrpcpassword@127.0.0.1:8332/');
echo $bitcoin->getbalance($_REQUEST['user'], 4);

EXEC:
echo system('sudo -H -u ubuntu bitcoind getbalance ' . $_REQUEST['user'] . '');

[gmaxwell]

what the heck are you doing that for?  It's a JSON rpc specifically so you can talk to it via JSON!   Using bitcoind via system gives you huge forking overhead for a big library heavy binary (not to mention the easy of getting yourself exploited of system()...).

[Kazu]

I tried both, just to see if my json thing got messed up. Look at the JSON api, its still screwed up. It was just for testing purposes.

[btc4ever]

I have noticed slowness and high CPU usage also.

1) bitcoind CPU usage is sometimes as high as 99% over a 3 second period (top) when bitcoind is fully caught up with blockchain and just sitting there with no local transactions or JSON requests.  This is on testnet.  It is consistently high enough usage that stopping bitcoind means the difference between choppy and smooth video playback on my older laptop.

2) sometimes jsonrpc requests block for quite a while.  many seconds.  For this reason (and others), I do almost all bitcoind calls in an offline process that syncs to my DB.  Still the blocking concerns me, especially for readonly calls like getinfo() or getbalance().  

[btc4ever]

Look at the JSON times.   Does 9 seconds seem right to you?

Or do you think that system() accounts for 222 seconds of overhead?  12 seconds?    3 seconds?

miss the point much?

what the heck are you doing that for?  It's a JSON rpc specifically so you can talk to it via JSON!   Using bitcoind via system gives you huge forking overhead for a big library heavy binary (not to mention the easy of getting yourself exploited of system()...).

[Kazu]

It seems dubious and boring to have to do all the transactions in a DB. That is very sad. Are you sure there is no other way of doing this? At that point I may as well store them in a $_SESSION variable or something (just for balance & transaction history etc... read only things that are displayed to the user and that there would be no reason to manipulate), at least then you only have to check for users that are logged in.

[Kazu]

A bit more information: Bitcoind is nowhere near as slow when doing it through shell? WTF?

Script started on Mon 22 Jul 2013 08:06:52 PM UTC
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.376s
user   0m0.004s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.226s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.198s
user   0m0.016s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.041s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.073s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.042s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.041s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.045s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.039s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.004s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.004s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.050s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.008s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.049s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.004s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.061s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
^[[A-101115.00029185

real   0m0.600s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.060s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.044s
user   0m0.012s
sys   0m0.000s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.039s
user   0m0.008s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.036s
user   0m0.008s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.047s
user   0m0.004s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.036s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.040s
user   0m0.000s
sys   0m0.012s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.038s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.036s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.036s
user   0m0.004s
sys   0m0.008s
]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance ""
-101115.00029185

real   0m0.037s
user   0m0.008s
sys   0m0.004s
]0;ubuntu: ~ubuntu:~$ exit
exit

Script done on Mon 22 Jul 2013 08:07:48 PM UTC

Any logical reason why this is like this???

Maybe it would be way faster to just use a shell script to write it somewhere, then read it from there whenever I need it?

[btc4ever]

as I understand it, "bitcoind getbalance" is just calling the daemon with jsonrpc.  It shouldn't be significantly different from your php script calling it with jsonrpc.

I have noticed that sometimes it is slower responding when downloading the blockchain.  Were you fully up-to-date in both instances?

What happens if you test from your php script and bitcoind getbalance back-to-back?   The idea being to test them both at nearly the same moment in time, in case something on the network was slowing bitcoind previously.

[jgarzik]

A bit more information: Bitcoind is nowhere near as slow when doing it through shell? WTF?

Not sure I understand what this means.

as I understand it, "bitcoind getbalance" is just calling the daemon with jsonrpc.  It shouldn't be significantly different from your php script calling it with jsonrpc.

That is correct.  "bitcoind getbalance" executes a new copy of bitcoind, which connects via TCP to an existing bitcoind, sends and receives JSON-RPC, then disconnects and exits.  Not much different than using PHP or cURL to make JSON-RPC calls.

[Kazu]

Not sure I understand what this means

When I ssh into the same server, go bitcoind getbalance "", I get the balance many times faster than when I go to that same server, make a php file, include the JSON library, and go all $bitcoin->getbalance("");

Now look at this:

$time = microtime(true);
$bitcoin = new jsonRPCClient('http://bitcoinrpc:hugepassword@127.0.0.1:8332/');
$write = microtime(true) - $time;
file_put_contents('log.txt', '<br>inittime: ' . $write, FILE_APPEND);

And guess what the output is? (There are lines of code before and after this, but those shouldn't matter... right?)

inittime: 1.7881393432617E-5
inittime: 1.6927719116211E-5
inittime: 1.9073486328125E-5
inittime: 1.8119812011719E-5
inittime: 2.0980834960938E-5
inittime: 2.1934509277344E-5
inittime: 2.0027160644531E-5
inittime: 1.8119812011719E-5
inittime: 1.5974044799805E-5
inittime: 4.0054321289062E-5
inittime: 1.6927719116211E-5
inittime: 1.1920928955078E-5
inittime: 1.8119812011719E-5
inittime: 1.7166137695312E-5
inittime: 1.8119812011719E-5
inittime: 3.9100646972656E-5

So?? This means that the actual act of getbalance() is taking a long time or something?

[gmaxwell]

Are you actually checking the result?  Perhaps you're getting an authentication error or a failure connecting, your example has 127.0.0.1 which should work, though if your real test doesn't have that: bitcoind listens only on localhost for rpc by default.

[jgarzik]

So?? This means that the actual act of getbalance() is taking a long time or something?

They are doing the same thing under the hood: making a TCP connection to bitcoind, sending/receiving JSON-RPC data.

[Kazu]

Okay I think I found the issue, and it also explains why I havent' been having this problem till now.

Transaction history. Through the process of debugging my site, I completed several thousand transactions (no, I didnt sit there clicking all day, its a micropayment gaming website so there are tons of automatic move() commands going on when I set my debugger going). Apparently, in order to get the balance of any individual account, it has to manually calculate out ALL of the transactions. You'd think they'd save each account's balance in some place and keep a separate transaction history but nooooo, they had to do it the random and inexplicably difficult way.

Is there any easy way of getting around this? As far as I can see I must either:

A) Constantly flush the transaction history occasionally, then re-load each account with what they had with a single move() command to each account, which is not only annoying but also means that people can't view their transaction history after a while.
B) Every once and a while re-load each account on a single address associated with that account, then flush transaction history.
OR
C) Not use Bitcoind's accounts at all, keep a single wallet and then have a database and store things in there including the transaction. Which is majorly re-inventing the wheel.

Is there any simpler way you can think of to do this? This seems like it basically renders bitcoind's account management ultra-inefficient and sorta is a major buzzkill with regard to my release schedule.

[grue]

use walletnotify instead of polling for transactions.