bitcoind 0.8.4 memory leak causing crash quite often

[maayainfotech]

We have ubuntu 12.04 and bitcoin 0.8.4. We have used these same version previously for setting up many bitcoin based servers and websites. But recently when creating new server with these versions of ubuntu and bitcoin, I am seeing huge memory consumption by bitcoind. I see virtual memory is approaching around 1.6 G before crashing the daemon. And it happens quite fast within 1/2 hr or so.  There is definitely some memory leak, but I am not running anything at the on server and not even interacting with bitcoind. bitcoind is being started on a freshly build ubuntu server. Is anyone else seeing this issue?

I have tried setting up maxCLient with no effect. Have tried increasing the swap space, which made bicoind run longer, but still crashed later on as memory leak is continuously growing.

Any help or clue appreciated.

Thanks,
-Arvind

[1715022492]

1715022492

[1715022492]

1715022492

[1715022492]

1715022492

[1715022492]

1715022492

[1715022492]

1715022492

[1715022492]

1715022492

[gmaxwell]

Is this a 32 bit or a 64 bit binary?

[maayainfotech]

Its 64 bit.

Thanks,
-Arvind

[zvs]

Its 64 bit.

Thanks,
-Arvind

this happens to me too, but it doesn't crash.   i've never seen it higher than 6GB

it takes about an hour to get up to 1.5GB or so, with 600+ connections

ed:  oh, it's related to transactions for sure.  when i remove all priority transactions and set tx fee to 0.001, it goes up much, much slower

[maayainfotech]

For some reason, now it started behaving very well. Virtual memory used is stable at 860m. Only thing I did is I aded 2 G swap space, but I dont see how that will help bitcoind to keep its footprint low!! Anything to do with bitcoin network?

[gmaxwell]

The virt counter is not useful— it's measuring address space used, and there is normally quite a bit more address space used than pages due to threads, mmap, and other things that use address space.  The useful memory metric is res (resident).  On x86_64 with about 100 connections I'm used to seeing about 488 resident.

[maayainfotech]

Hi gmaxwell,

this time after running longer, still bitcoind crashed. Here is crash message:

Nov 29 23:58:59 localhost kernel: [228032.815365] Out of memory: Kill process 25217 (bitcoind) score 889 or sacrifice child
Nov 29 23:58:59 localhost kernel: [228032.815382] Killed process 25217 (bitcoind) total-vm:3629112kB, anon-rss:777192kB, file-rss:0kB

This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?

Thanks,

[gmaxwell]

This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?

3.6GBytes VM isn't huge for a process which uses mmap, note the res is just 777MBytes.

Can you run uname -a on that system as well as file /path/to/your/bitcoind  and show me the results?

[maayainfotech]

Can you run uname -a on that system as well as file /path/to/your/bitcoind  and show me the results?

Hi gmaxwell, thanks again for your response. Here is the output of those two cmds:

Code:

root@coinfaucet1:~# uname -a
Linux coinfaucet1 3.2.0-55-virtual #85-Ubuntu SMP Wed Oct 2 12:49:31 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

root@coinfaucet1:~# file /usr/lib/bitcoin/bitcoind 
/usr/lib/bitcoin/bitcoind: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, BuildID[sha1]=0x034faf497f96330f75906d0d74e72edf2f91b16c, stripped

Thanks,

[gmaxwell]

Is this on a VPS?   Can you run ulimit -a and show me the output?

[zvs]

This shows total VM used was 3.6 G which is huge. Why it could be using this much of memory? How I can reduce this? why this problem is showing up now?

3.6GBytes VM isn't huge for a process which uses mmap, note the res is just 777MBytes.

Can you run uname -a on that system as well as file /path/to/your/bitcoind  and show me the results?

Eh.  Doesn't everyone's bitcoind gradually bloat?

I've seen reserved memory of over 4GB frequently & as recently as the git pull from several weeks ago

this is running with 600 initial connections which usually grows to around 850

i used to restart it once or twice a day

[gmaxwell]

Eh.  Doesn't everyone's bitcoind gradually bloat?

Absolutely not.

I've seen reserved memory of over 4GB frequently & as recently as the git pull from several weeks ago

Can you tell me more about what configuration you're running?

this is running with 600 initial connections which usually grows to around 850

Not to be too blunt, I hope, but are you making up figures? You shouldn't be able to obtain that many connections (and at that level you're close to causing memory corruption from >1024 file desc in select).

[maayainfotech]

Is this on a VPS?   Can you run ulimit -a and show me the output?

Hi gmaxwell, Here is the output:

Code:

root@coinfaucet1:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7803
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7803
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

[maayainfotech]

Tried even with ubuntu version was 13.10 and bitcoin version 0.8.5. But still seeing it getting crashed similar fashion.

 Out of memory: Kill process 2716 (bitcoind) score 907 or sacrifice child
Killed process 2716 (bitcoind) total-vm:1704204kB, anon-rss:948376kB, file-rss:0kB

I am not sure why I am the only one seeing this problem. Its fresh installation, so if I am seeing this error others should also face similar issues. Whats wrong with it!!!

 

[jlp]

I am not sure why I am the only one seeing this problem. Its fresh installation, so if I am seeing this error others should also face similar issues. Whats wrong with it!!!

 

I have essentially the same problem that you have.  I have the same output from "uname -a", "file /path/to/bitcoind", "ulimit -a", etc.  The only difference from you is that I'm running bitcoind 0.9.0.

After I start bitcoind, I can see in "top" that its memory usage just keeps rising.  My bitcoind crashes all the time and within less than 30 minutes.  I have the following in bitcoin.conf to limit the connections, but they have no effect on slowing or stopping the crashes:

Code:

connections=1
maxconnections=1

I was thinking of adding memory to the server, but what's the point if we don't know the maximum amount of memory that bitcoind will use?

I tried the following to limit the amount of memory and virtual memory, but this only causes bitcoind to crash faster (because bitcoind hits its limit faster?).

Code:

ulimit -m 500000 -v 1000000

There must be lots of other people having this problem.  Did you ever find the fix?

Edit:  I think I found the solution.  I added swap file and used the following:

Code:

ulimit -m 200000 -v 1000000

So far, so good.

Does anyone know how much physical RAM and swap file the server should have for running bitcoind 0.9.0, and what the "ulimit -m -v", "connections" and "maxconnections" settings should be?