Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 06:00:40 AM |
|
My Bitcoind is really slow. I'm trying to to start a service using it a bit (by a bit, I mean really not that much) and for some reason its using up 40%+ of my CPU all the time. I suspect it is slowing down my page loads as they are really quite slow. Has anybody else ran into this problem, and if so, do they have any workarounds/alternate software (I still need account management & all other bitcoind features but mining)/other fixes?
Thanks tons!
|
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 06:35:07 AM |
|
I have no problem with having the first starting time being slow - I can wait. The part that is annoying is that functions that must be repeated on nearly every pageload, like getbalance, seem to be absurdly slow. Is there any way of speeding those up?
|
|
|
|
btcsql
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:28:11 AM |
|
The solution is simple: don't run Bitcoind on the same server as your webserver. It's much faster to have it dedicated, and simply connect/issue commands/grab data via API format.
|
|
|
|
niko
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:38:30 AM |
|
The solution is simple: don't run Bitcoind on the same server as your webserver. It's much faster to have it dedicated, and simply connect/issue commands/grab data via API format.
This should provide for more security, too. Sorry, I know these are not answers to your question, it's a workaround. However, it's a good workaround.
|
They're there, in their room. Your mining rig is on fire, yet you're very calm.
|
|
|
Belkaar
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 08:34:49 AM |
|
You should also consider getting the needed info via a cron job an saving it to a database that the frontend uses. This decouples page loading times from API-processing.
|
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 05:40:30 PM |
|
I sort of need the pageload connected with the API because the balance can't be on a delay for what I'm doing. Thanks for the idea, though.
I'm considering running bitcoind on a dedicated server, however that seems like a pretty big waste of resources. Is there any way of viewing what Bitcoind is doing all this time, like viewing some sort of output? I just want to see why its taking so incredibly long.
|
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:02:22 PM |
|
So I tried using a microtime() difference to calculate the time it takes to execute a bitcoind getbalance on an account with a couple hundred transactions. I have tried two methds: php system() to call bitcoind Bitcoind JSON rpc connection And... EXEC: 222.71010398865 EXEC: 12.618118047714 EXEC: 3.6932117938995 EXEC: 1.6202571392059 EXEC: 1.7495849132538 EXEC: 3.5270199775696 EXEC: 3.0422019958496 EXEC: 2.6602010726929 EXEC: 1.5538840293884 EXEC: 0.58772110939026 EXEC: 0.97496891021729 EXEC: 0.99640703201294 EXEC: 2.7791039943695 EXEC: 1.8489720821381 EXEC: 0.82930707931519 EXEC: 0.25357699394226 EXEC: 0.62351679801941 EXEC: 1.3590130805969 EXEC: 4.4388699531555 EXEC: 0.70736193656921 EXEC: 1.473837852478 EXEC: 0.8196861743927 EXEC: 0.49087500572205 EXEC: 0.31272411346436 EXEC: 0.3066029548645 EXEC: 0.93807220458984 EXEC: 2.9650371074677 EXEC: 4.9273610115051 EXEC: 2.4882531166077 EXEC: 1.2758350372314 EXEC: 0.81832599639893 EXEC: 1.2282900810242 JSON: 0.70425081253052 JSON: 0.31246995925903 JSON: 6.2438280582428 JSON: 0.75746512413025 JSON: 4.1969089508057 JSON: 0.15843486785889 JSON: 1.0560338497162 JSON: 4.0856080055237 JSON: 0.34556603431702 JSON: 9.2187669277191 I've heard of inefficient processing but this is just ridiculous. I've got to be doing something wrong, there is no way this can be normal. Here is my PHP: JSON: require_once 'jsonRPCClient.php'; $bitcoin = new jsonRPCClient(' http://bitcoinrpc:reallylongrpcpassword@127.0.0.1:8332/');echo $bitcoin->getbalance($_REQUEST['user'], 4); EXEC: echo system('sudo -H -u ubuntu bitcoind getbalance ' . $_REQUEST['user'] . '');
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8646
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:08:33 PM |
|
what the heck are you doing that for? It's a JSON rpc specifically so you can talk to it via JSON! Using bitcoind via system gives you huge forking overhead for a big library heavy binary (not to mention the easy of getting yourself exploited of system()...).
|
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:10:02 PM |
|
I tried both, just to see if my json thing got messed up. Look at the JSON api, its still screwed up. It was just for testing purposes.
|
|
|
|
btc4ever
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:20:59 PM |
|
I have noticed slowness and high CPU usage also.
1) bitcoind CPU usage is sometimes as high as 99% over a 3 second period (top) when bitcoind is fully caught up with blockchain and just sitting there with no local transactions or JSON requests. This is on testnet. It is consistently high enough usage that stopping bitcoind means the difference between choppy and smooth video playback on my older laptop.
2) sometimes jsonrpc requests block for quite a while. many seconds. For this reason (and others), I do almost all bitcoind calls in an offline process that syncs to my DB. Still the blocking concerns me, especially for readonly calls like getinfo() or getbalance().
|
Psst!! Wanna make bitcoin unstoppable? Why the Only Real Way to Buy Bitcoins Is on the Streets. Avoid banks and centralized exchanges. Buy/Sell coins locally. Meet other bitcoiners and develop your network. Try localbitcoins.com or find or start a buttonwood / satoshi square in your area. Pass it on!
|
|
|
btc4ever
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:23:13 PM |
|
Look at the JSON times. Does 9 seconds seem right to you? Or do you think that system() accounts for 222 seconds of overhead? 12 seconds? 3 seconds? miss the point much? what the heck are you doing that for? It's a JSON rpc specifically so you can talk to it via JSON! Using bitcoind via system gives you huge forking overhead for a big library heavy binary (not to mention the easy of getting yourself exploited of system()...).
|
Psst!! Wanna make bitcoin unstoppable? Why the Only Real Way to Buy Bitcoins Is on the Streets. Avoid banks and centralized exchanges. Buy/Sell coins locally. Meet other bitcoiners and develop your network. Try localbitcoins.com or find or start a buttonwood / satoshi square in your area. Pass it on!
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 07:31:41 PM |
|
It seems dubious and boring to have to do all the transactions in a DB. That is very sad. Are you sure there is no other way of doing this? At that point I may as well store them in a $_SESSION variable or something (just for balance & transaction history etc... read only things that are displayed to the user and that there would be no reason to manipulate), at least then you only have to check for users that are logged in.
|
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 08:09:43 PM |
|
A bit more information: Bitcoind is nowhere near as slow when doing it through shell? WTF?
Script started on Mon 22 Jul 2013 08:06:52 PM UTC ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.376s user 0m0.004s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.226s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.198s user 0m0.016s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.041s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.073s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.042s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.041s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.045s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.039s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.004s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.004s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.050s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.008s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.049s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.004s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.061s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" ^[[A-101115.00029185
real 0m0.600s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.060s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.044s user 0m0.012s sys 0m0.000s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.039s user 0m0.008s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.036s user 0m0.008s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.047s user 0m0.004s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.036s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.040s user 0m0.000s sys 0m0.012s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.038s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.036s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.036s user 0m0.004s sys 0m0.008s ]0;ubuntu: ~ubuntu:~$ time bitcoind getbalance "" -101115.00029185
real 0m0.037s user 0m0.008s sys 0m0.004s ]0;ubuntu: ~ubuntu:~$ exit exit
Script done on Mon 22 Jul 2013 08:07:48 PM UTC
Any logical reason why this is like this???
Maybe it would be way faster to just use a shell script to write it somewhere, then read it from there whenever I need it?
|
|
|
|
btc4ever
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 08:31:59 PM |
|
as I understand it, "bitcoind getbalance" is just calling the daemon with jsonrpc. It shouldn't be significantly different from your php script calling it with jsonrpc.
I have noticed that sometimes it is slower responding when downloading the blockchain. Were you fully up-to-date in both instances?
What happens if you test from your php script and bitcoind getbalance back-to-back? The idea being to test them both at nearly the same moment in time, in case something on the network was slowing bitcoind previously.
|
Psst!! Wanna make bitcoin unstoppable? Why the Only Real Way to Buy Bitcoins Is on the Streets. Avoid banks and centralized exchanges. Buy/Sell coins locally. Meet other bitcoiners and develop your network. Try localbitcoins.com or find or start a buttonwood / satoshi square in your area. Pass it on!
|
|
|
jgarzik
Legendary
Offline
Activity: 1596
Merit: 1099
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 08:38:51 PM |
|
A bit more information: Bitcoind is nowhere near as slow when doing it through shell? WTF?
Not sure I understand what this means. as I understand it, "bitcoind getbalance" is just calling the daemon with jsonrpc. It shouldn't be significantly different from your php script calling it with jsonrpc.
That is correct. "bitcoind getbalance" executes a new copy of bitcoind, which connects via TCP to an existing bitcoind, sends and receives JSON-RPC, then disconnects and exits. Not much different than using PHP or cURL to make JSON-RPC calls.
|
Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own. Visit bloq.com / metronome.io Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 22, 2013, 08:46:02 PM |
|
Not sure I understand what this means When I ssh into the same server, go bitcoind getbalance "", I get the balance many times faster than when I go to that same server, make a php file, include the JSON library, and go all $bitcoin->getbalance(""); Now look at this: $time = microtime(true); $bitcoin = new jsonRPCClient(' http://bitcoinrpc:hugepassword@127.0.0.1:8332/');$write = microtime(true) - $time; file_put_contents('log.txt', '<br>inittime: ' . $write, FILE_APPEND); And guess what the output is? (There are lines of code before and after this, but those shouldn't matter... right?) inittime: 1.7881393432617E-5 inittime: 1.6927719116211E-5 inittime: 1.9073486328125E-5 inittime: 1.8119812011719E-5 inittime: 2.0980834960938E-5 inittime: 2.1934509277344E-5 inittime: 2.0027160644531E-5 inittime: 1.8119812011719E-5 inittime: 1.5974044799805E-5 inittime: 4.0054321289062E-5 inittime: 1.6927719116211E-5 inittime: 1.1920928955078E-5 inittime: 1.8119812011719E-5 inittime: 1.7166137695312E-5 inittime: 1.8119812011719E-5 inittime: 3.9100646972656E-5 So?? This means that the actual act of getbalance() is taking a long time or something?
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8646
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 23, 2013, 01:11:23 AM |
|
Are you actually checking the result? Perhaps you're getting an authentication error or a failure connecting, your example has 127.0.0.1 which should work, though if your real test doesn't have that: bitcoind listens only on localhost for rpc by default.
|
|
|
|
jgarzik
Legendary
Offline
Activity: 1596
Merit: 1099
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 23, 2013, 01:34:15 AM |
|
So?? This means that the actual act of getbalance() is taking a long time or something?
They are doing the same thing under the hood: making a TCP connection to bitcoind, sending/receiving JSON-RPC data.
|
Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own. Visit bloq.com / metronome.io Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
|
|
|
Kazu (OP)
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 23, 2013, 05:30:23 AM |
|
Okay I think I found the issue, and it also explains why I havent' been having this problem till now.
Transaction history. Through the process of debugging my site, I completed several thousand transactions (no, I didnt sit there clicking all day, its a micropayment gaming website so there are tons of automatic move() commands going on when I set my debugger going). Apparently, in order to get the balance of any individual account, it has to manually calculate out ALL of the transactions. You'd think they'd save each account's balance in some place and keep a separate transaction history but nooooo, they had to do it the random and inexplicably difficult way.
Is there any easy way of getting around this? As far as I can see I must either:
A) Constantly flush the transaction history occasionally, then re-load each account with what they had with a single move() command to each account, which is not only annoying but also means that people can't view their transaction history after a while. B) Every once and a while re-load each account on a single address associated with that account, then flush transaction history. OR C) Not use Bitcoind's accounts at all, keep a single wallet and then have a database and store things in there including the transaction. Which is majorly re-inventing the wheel.
Is there any simpler way you can think of to do this? This seems like it basically renders bitcoind's account management ultra-inefficient and sorta is a major buzzkill with regard to my release schedule.
|
|
|
|
grue
Legendary
Offline
Activity: 2058
Merit: 1434
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 23, 2013, 10:36:39 PM |
|
use walletnotify instead of polling for transactions.
|
|
|
|
|