Title: bitcoind invoked oom-killer Post by: jlp on April 19, 2015, 03:58:55 PM My site crashed because MySQL stopped running. According to the syslog, the following happened:
Code: ... I'm guessing that "bitcoind invoked oom-killer" caused the server to run out of memory which caused the server to kill mysqld. Am I correct? I have 2 GB of memory. This is what shows on "top": Code: Mem: 2050964k total, 1976288k used, 74676k free, 18068k buffers According to "top", bitcoind usually uses approximately 30% of memory (which I think is WAY TOO much). My server is mainly for my site that runs bitcoind. There is little traffic, so Apache cannot be using much memory. I would've assume that 2 GB is overkill. What is causing "bitcoind invoked oom-killer"? How can I prevent this from happening? Title: Re: bitcoind invoked oom-killer Post by: -ck on April 21, 2015, 01:36:27 AM Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher.
Translation: You need more ram to use bitcoind, and use the official binary if you can. Title: Re: bitcoind invoked oom-killer Post by: jlp on April 21, 2015, 01:45:23 AM Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher. Translation: You need more ram to use bitcoind, and use the official binary if you can. Thanks for your insight. I used the official binary. My server has 2 GB of RAM. "top" tells me that bitcoind uses 30% of of my server's memory, which equates to 600 MB. Are you saying that bitcoind will request 2 GB (or more) every once in a while? What causes this? Will the memory usage go down if I reduce the number of connections? Title: Re: bitcoind invoked oom-killer Post by: innocent93 on April 21, 2015, 06:54:20 AM Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred.
Title: Re: bitcoind invoked oom-killer Post by: emrebey on April 21, 2015, 08:03:45 AM it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally.
Title: Re: bitcoind invoked oom-killer Post by: sikaxchange on April 21, 2015, 12:50:40 PM its true its either your memory is full or there is a problem with your swap i suggest that you occassionally stop
Title: Re: bitcoind invoked oom-killer Post by: jlp on April 21, 2015, 07:37:15 PM Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred. Yes, it's a VPS. 4-8 gb? Wow, bitcoind is a hog. it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally. Can you elaborate on how I can track down with / top? What am I supposed to be looking for when I stop bitcoind occasionally? Title: Re: bitcoind invoked oom-killer Post by: emrebey on April 22, 2015, 09:16:15 AM just check randomly what process is using ram resources with top or htop.
some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage. there's probably is a better way to monitor this but I am not a system administrator :-) Title: Re: bitcoind invoked oom-killer Post by: bronan on April 22, 2015, 09:24:43 AM Yes lol its a mem hog
I agree that 4 Gb mem is probably even on the edge, i bought extra ram because i observed several out of memory messages in other software as well. Ofcourse it depends on what other threads and programs you have running. Runs kinda smooth on my little server with 16 Gb mem, you need to spot the spikes it uses the 16 is a bit overkill but the price was low so another 8 Gb does not hurt anything ;) Title: Re: bitcoind invoked oom-killer Post by: -ck on April 22, 2015, 10:26:40 AM just check randomly what process is using ram resources with top or htop. This is unnecessary. The ram usage is all due to bitcoind. Monitoring it will just tell you "yes it's bitcoind, it spikes to high levels" which we already know.some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage. there's probably is a better way to monitor this but I am not a system administrator :-) You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it. Title: Re: bitcoind invoked oom-killer Post by: jlp on April 22, 2015, 07:33:47 PM just check randomly what process is using ram resources with top or htop. some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage. there's probably is a better way to monitor this but I am not a system administrator :-) Most of the memory is used by bitcoind. It usually shows 30%. Below is some of the output from top: Code: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND Not only is bitcoind a memory hog, it is also a disk hog. You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it. What do you think is the lowest that I can set cache and maximum connections to? According to https://en.bitcoin.it/wiki/Running_Bitcoin , there is a dbcache setting, but no cache setting. Are you referring to dbcache? |