maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
November 28, 2013, 08:14:58 AM |
|
We have ubuntu 12.04 and bitcoin 0.8.4. We have used these same version previously for setting up many bitcoin based servers and websites. But recently when creating new server with these versions of ubuntu and bitcoin, I am seeing huge memory consumption by bitcoind. I see virtual memory is approaching around 1.6 G before crashing the daemon. And it happens quite fast within 1/2 hr or so. There is definitely some memory leak, but I am not running anything at the on server and not even interacting with bitcoind. bitcoind is being started on a freshly build ubuntu server. Is anyone else seeing this issue?
I have tried setting up maxCLient with no effect. Have tried increasing the swap space, which made bicoind run longer, but still crashed later on as memory leak is continuously growing.
Any help or clue appreciated.
Thanks, -Arvind
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8684
|
|
November 28, 2013, 08:20:48 AM |
|
Is this a 32 bit or a 64 bit binary?
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
November 28, 2013, 08:54:31 AM |
|
Its 64 bit.
Thanks, -Arvind
|
|
|
|
zvs
Legendary
Offline
Activity: 1680
Merit: 1000
https://web.archive.org/web/*/nogleg.com
|
|
November 29, 2013, 12:31:11 AM |
|
Its 64 bit.
Thanks, -Arvind
this happens to me too, but it doesn't crash. i've never seen it higher than 6GB it takes about an hour to get up to 1.5GB or so, with 600+ connections ed: oh, it's related to transactions for sure. when i remove all priority transactions and set tx fee to 0.001, it goes up much, much slower
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
November 29, 2013, 06:53:33 PM |
|
For some reason, now it started behaving very well. Virtual memory used is stable at 860m. Only thing I did is I aded 2 G swap space, but I dont see how that will help bitcoind to keep its footprint low!! Anything to do with bitcoin network?
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8684
|
|
November 29, 2013, 08:01:16 PM |
|
The virt counter is not useful— it's measuring address space used, and there is normally quite a bit more address space used than pages due to threads, mmap, and other things that use address space. The useful memory metric is res (resident). On x86_64 with about 100 connections I'm used to seeing about 488 resident.
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
November 30, 2013, 05:15:34 PM |
|
Hi gmaxwell,
this time after running longer, still bitcoind crashed. Here is crash message:
Nov 29 23:58:59 localhost kernel: [228032.815365] Out of memory: Kill process 25217 (bitcoind) score 889 or sacrifice child Nov 29 23:58:59 localhost kernel: [228032.815382] Killed process 25217 (bitcoind) total-vm:3629112kB, anon-rss:777192kB, file-rss:0kB
This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?
Thanks,
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8684
|
|
November 30, 2013, 07:38:35 PM |
|
This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?
3.6GBytes VM isn't huge for a process which uses mmap, note the res is just 777MBytes. Can you run uname -a on that system as well as file /path/to/your/bitcoind and show me the results?
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
December 03, 2013, 08:15:34 AM |
|
Can you run uname -a on that system as well as file /path/to/your/bitcoind and show me the results?
Hi gmaxwell, thanks again for your response. Here is the output of those two cmds: root@coinfaucet1:~# uname -a Linux coinfaucet1 3.2.0-55-virtual #85-Ubuntu SMP Wed Oct 2 12:49:31 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
root@coinfaucet1:~# file /usr/lib/bitcoin/bitcoind /usr/lib/bitcoin/bitcoind: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, BuildID[sha1]=0x034faf497f96330f75906d0d74e72edf2f91b16c, stripped
Thanks,
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8684
|
|
December 03, 2013, 08:46:59 AM |
|
Is this on a VPS? Can you run ulimit -a and show me the output?
|
|
|
|
zvs
Legendary
Offline
Activity: 1680
Merit: 1000
https://web.archive.org/web/*/nogleg.com
|
|
December 03, 2013, 03:33:23 PM |
|
This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?
3.6GBytes VM isn't huge for a process which uses mmap, note the res is just 777MBytes. Can you run uname -a on that system as well as file /path/to/your/bitcoind and show me the results? Eh. Doesn't everyone's bitcoind gradually bloat? I've seen reserved memory of over 4GB frequently & as recently as the git pull from several weeks ago this is running with 600 initial connections which usually grows to around 850 i used to restart it once or twice a day
|
|
|
|
gmaxwell
Moderator
Legendary
Offline
Activity: 4242
Merit: 8684
|
|
December 03, 2013, 04:30:31 PM |
|
Eh. Doesn't everyone's bitcoind gradually bloat?
Absolutely not. I've seen reserved memory of over 4GB frequently & as recently as the git pull from several weeks ago
Can you tell me more about what configuration you're running? this is running with 600 initial connections which usually grows to around 850
Not to be too blunt, I hope, but are you making up figures? You shouldn't be able to obtain that many connections (and at that level you're close to causing memory corruption from >1024 file desc in select).
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
December 04, 2013, 03:51:39 AM |
|
Is this on a VPS? Can you run ulimit -a and show me the output?
Hi gmaxwell, Here is the output: root@coinfaucet1:~# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7803 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7803 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
|
|
|
|
maayainfotech (OP)
Newbie
Offline
Activity: 8
Merit: 0
|
|
December 05, 2013, 04:24:41 PM |
|
Tried even with ubuntu version was 13.10 and bitcoin version 0.8.5. But still seeing it getting crashed similar fashion. Out of memory: Kill process 2716 (bitcoind) score 907 or sacrifice child Killed process 2716 (bitcoind) total-vm:1704204kB, anon-rss:948376kB, file-rss:0kB I am not sure why I am the only one seeing this problem. Its fresh installation, so if I am seeing this error others should also face similar issues. Whats wrong with it!!!
|
|
|
|
jlp
|
|
April 01, 2014, 02:37:28 AM Last edit: April 01, 2014, 07:43:04 PM by jlp |
|
I have essentially the same problem that you have. I have the same output from "uname -a", "file /path/to/bitcoind", "ulimit -a", etc. The only difference from you is that I'm running bitcoind 0.9.0. After I start bitcoind, I can see in "top" that its memory usage just keeps rising. My bitcoind crashes all the time and within less than 30 minutes. I have the following in bitcoin.conf to limit the connections, but they have no effect on slowing or stopping the crashes: connections=1 maxconnections=1
I was thinking of adding memory to the server, but what's the point if we don't know the maximum amount of memory that bitcoind will use? I tried the following to limit the amount of memory and virtual memory, but this only causes bitcoind to crash faster (because bitcoind hits its limit faster?). ulimit -m 500000 -v 1000000 There must be lots of other people having this problem. Did you ever find the fix? Edit: I think I found the solution. I added swap file and used the following: ulimit -m 200000 -v 1000000 So far, so good. Does anyone know how much physical RAM and swap file the server should have for running bitcoind 0.9.0, and what the "ulimit -m -v", "connections" and "maxconnections" settings should be?
|
|
|
|
|