bitcoind invoked oom-killer

[jlp]

My site crashed because MySQL stopped running.  According to the syslog, the following happened:

Code:

...
bitcoind invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000
bitcoind cpuset=/ mems_allowed=0
...
...
2299 total pagecache pages
1911 pages in swap cache
Swap cache stats: add 855411, delete 853500, find 7256250/7310330
Free swap  = 0kB
Total swap = 1048572kB
524270 pages RAM
11529 pages reserved
61072 pages shared
497066 pages non-shared
...
...
Out of memory: Kill process 5919 (mysqld) score 25 or sacrifice child
Killed process 5919 (mysqld) total-vm:2126812kB, anon-rss:29312kB, file-rss:0kB
init: mysql main process (5919) killed by KILL signal
init: mysql main process ended, respawning
apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
apache2 cpuset=/ mems_allowed=0
...

I'm guessing that "bitcoind invoked oom-killer" caused the server to run out of memory which caused the server to kill mysqld.  Am I correct?

I have 2 GB of memory.  This is what shows on "top":

Code:

Mem:   2050964k total,  1976288k used,    74676k free,    18068k buffers
Swap:  1048572k total,   948164k used,   100408k free,   742556k cached

According to "top", bitcoind usually uses approximately 30% of memory (which I think is WAY TOO much).  My server is mainly for my site that runs bitcoind.  There is little traffic, so Apache cannot be using much memory.  I would've assume that 2 GB is overkill.

What is causing "bitcoind invoked oom-killer"?  How can I prevent this from happening?

[1714946253]

1714946253

[1714946253]

1714946253

[-ck]

Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher.

Translation: You need more ram to use bitcoind, and use the official binary if you can.

[jlp]

Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher.

Translation: You need more ram to use bitcoind, and use the official binary if you can.

Thanks for your insight.

I used the official binary.

My server has 2 GB of RAM.  "top" tells me that bitcoind uses 30% of of my server's memory, which equates to 600 MB.  Are you saying that bitcoind will request 2 GB (or more) every once in a while?  What causes this?  Will the memory usage go down if I reduce the number of connections?

[innocent93]

Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred.

[emrebey]

it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally.

[sikaxchange]

its true its either your memory is full or there is a problem with your swap i suggest that you occassionally stop

[jlp]

Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred.

Yes, it's a VPS.  4-8 gb?  Wow, bitcoind is a hog.

it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally.

Can you elaborate on how I can track down with / top?  What am I supposed to be looking for when I stop bitcoind occasionally?

[emrebey]

just check randomly what process is using ram resources with top or htop.

some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.

there's probably is a better way to monitor this but I am not a system administrator :-)

[bronan]

Yes lol its a mem hog

I agree that 4 Gb mem is probably even on the edge, i bought extra ram because i observed several out of memory messages in other software as well.
Ofcourse it depends on what other threads and programs you have running.

Runs kinda smooth on my little server with 16 Gb mem, you need to spot the spikes it uses the 16 is a bit overkill but the price was low so another 8 Gb does not hurt anything

[-ck]

just check randomly what process is using ram resources with top or htop.

some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.

there's probably is a better way to monitor this but I am not a system administrator :-)

This is unnecessary. The ram usage is all due to bitcoind. Monitoring it will just tell you "yes it's bitcoind, it spikes to high levels" which we already know.

You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it.

[jlp]

just check randomly what process is using ram resources with top or htop.

some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.

there's probably is a better way to monitor this but I am not a system administrator :-)

Most of the memory is used by bitcoind.  It usually shows 30%.  Below is some of the output from top:

Code:

  PID USER        PR  NI  VIRT      RES  SHR S %CPU %MEM        TIME+  COMMAND                                                 
  652 bitcoin     35  15 2689m     612m 4048 S    1 30.6     844:08.78 bitcoind                                                
  20177 www-data  20   0  358m      15m 3784 S    0  0.7       0:00.29 apache2                                                
  20463 www-data  20   0  357m      14m 3728 S    0  0.7       0:00.20 apache2                                                
  20136 www-data  20   0  358m      14m 3760 S    0  0.7       0:00.20 apache2                                                  
  20474 www-data  20   0  357m      14m 3572 S    0  0.7       0:00.11 apache2                                                
  19096 www-data  20   0  356m      13m 3972 S    0  0.7       0:01.09 apache2                                                
  19098 www-data  20   0  355m      13m 3908 S    0  0.7       0:00.98 apache2                                                
  20135 www-data  20   0  352m     9092 3812 S    0  0.4       0:00.28 apache2 

Not only is bitcoind a memory hog, it is also a disk hog.

You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it.

What do you think is the lowest that I can set cache and maximum connections to?  According to https://en.bitcoin.it/wiki/Running_Bitcoin , there is a dbcache setting, but no cache setting.  Are you referring to dbcache?