jlp (OP)
|
|
April 19, 2015, 03:58:55 PM |
|
My site crashed because MySQL stopped running. According to the syslog, the following happened: ... bitcoind invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=-17, oom_score_adj=-1000 bitcoind cpuset=/ mems_allowed=0 ... ... 2299 total pagecache pages 1911 pages in swap cache Swap cache stats: add 855411, delete 853500, find 7256250/7310330 Free swap = 0kB Total swap = 1048572kB 524270 pages RAM 11529 pages reserved 61072 pages shared 497066 pages non-shared ... ... Out of memory: Kill process 5919 (mysqld) score 25 or sacrifice child Killed process 5919 (mysqld) total-vm:2126812kB, anon-rss:29312kB, file-rss:0kB init: mysql main process (5919) killed by KILL signal init: mysql main process ended, respawning apache2 invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0 apache2 cpuset=/ mems_allowed=0 ... I'm guessing that "bitcoind invoked oom-killer" caused the server to run out of memory which caused the server to kill mysqld. Am I correct? I have 2 GB of memory. This is what shows on "top": Mem: 2050964k total, 1976288k used, 74676k free, 18068k buffers Swap: 1048572k total, 948164k used, 100408k free, 742556k cached According to "top", bitcoind usually uses approximately 30% of memory (which I think is WAY TOO much). My server is mainly for my site that runs bitcoind. There is little traffic, so Apache cannot be using much memory. I would've assume that 2 GB is overkill. What is causing "bitcoind invoked oom-killer"? How can I prevent this from happening?
|
|
|
|
|
|
|
|
|
"In a nutshell, the network works like a distributed
timestamp server, stamping the first transaction to spend a coin. It
takes advantage of the nature of information being easy to spread but
hard to stifle." -- Satoshi
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
|
-ck
Legendary
Offline
Activity: 4102
Merit: 1633
Ruu \o/
|
|
April 21, 2015, 01:36:27 AM |
|
Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher.
Translation: You need more ram to use bitcoind, and use the official binary if you can.
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
jlp (OP)
|
|
April 21, 2015, 01:45:23 AM |
|
Bitcoind uses a LOT of ram, and even more so (probably with leaks) if you compile it yourself with anything but the exact set of libraries that the official binary is compiled with. I struggled with even 4GB of ram on my server until I gave up compiling it myself and ended up using the official binary. However even that hovers around 1.2GB mem usage when well connected and often spikes much higher.
Translation: You need more ram to use bitcoind, and use the official binary if you can.
Thanks for your insight. I used the official binary. My server has 2 GB of RAM. "top" tells me that bitcoind uses 30% of of my server's memory, which equates to 600 MB. Are you saying that bitcoind will request 2 GB (or more) every once in a while? What causes this? Will the memory usage go down if I reduce the number of connections?
|
|
|
|
innocent93
Legendary
Offline
Activity: 896
Merit: 1000
|
|
April 21, 2015, 06:54:20 AM |
|
Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred.
|
|
|
|
emrebey
|
|
April 21, 2015, 08:03:45 AM |
|
it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally.
|
|
|
|
sikaxchange
Newbie
Offline
Activity: 14
Merit: 0
|
|
April 21, 2015, 12:50:40 PM |
|
its true its either your memory is full or there is a problem with your swap i suggest that you occassionally stop
|
|
|
|
jlp (OP)
|
|
April 21, 2015, 07:37:15 PM |
|
Is this on a VPS? VPS host systems are well known for killing mysqld and bitcoind with the OOM killer, so you definitely need more RAM. For a website and mysql and bitcoind you can't get away with less than 4gb although 8gb is preferred.
Yes, it's a VPS. 4-8 gb? Wow, bitcoind is a hog. it seems you don't have space on your memory including swap. you need to understand the reason of the peak. track down with / top or stop occasionally.
Can you elaborate on how I can track down with / top? What am I supposed to be looking for when I stop bitcoind occasionally?
|
|
|
|
emrebey
|
|
April 22, 2015, 09:16:15 AM |
|
just check randomly what process is using ram resources with top or htop.
some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.
there's probably is a better way to monitor this but I am not a system administrator :-)
|
|
|
|
bronan
|
|
April 22, 2015, 09:24:43 AM |
|
Yes lol its a mem hog I agree that 4 Gb mem is probably even on the edge, i bought extra ram because i observed several out of memory messages in other software as well. Ofcourse it depends on what other threads and programs you have running. Runs kinda smooth on my little server with 16 Gb mem, you need to spot the spikes it uses the 16 is a bit overkill but the price was low so another 8 Gb does not hurt anything
|
|
|
|
-ck
Legendary
Offline
Activity: 4102
Merit: 1633
Ruu \o/
|
|
April 22, 2015, 10:26:40 AM |
|
just check randomly what process is using ram resources with top or htop.
some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.
there's probably is a better way to monitor this but I am not a system administrator :-)
This is unnecessary. The ram usage is all due to bitcoind. Monitoring it will just tell you "yes it's bitcoind, it spikes to high levels" which we already know. You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it.
|
Developer/maintainer for cgminer, ckpool/ckproxy, and the -ck kernel 2% Fee Solo mining at solo.ckpool.org -ck
|
|
|
jlp (OP)
|
|
April 22, 2015, 07:33:47 PM |
|
just check randomly what process is using ram resources with top or htop.
some process is making a peak on ram usage therefore kernel starts killing processes. I suggest htop - you can order processes by ram usage.
there's probably is a better way to monitor this but I am not a system administrator :-)
Most of the memory is used by bitcoind. It usually shows 30%. Below is some of the output from top: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 652 bitcoin 35 15 2689m 612m 4048 S 1 30.6 844:08.78 bitcoind 20177 www-data 20 0 358m 15m 3784 S 0 0.7 0:00.29 apache2 20463 www-data 20 0 357m 14m 3728 S 0 0.7 0:00.20 apache2 20136 www-data 20 0 358m 14m 3760 S 0 0.7 0:00.20 apache2 20474 www-data 20 0 357m 14m 3572 S 0 0.7 0:00.11 apache2 19096 www-data 20 0 356m 13m 3972 S 0 0.7 0:01.09 apache2 19098 www-data 20 0 355m 13m 3908 S 0 0.7 0:00.98 apache2 20135 www-data 20 0 352m 9092 3812 S 0 0.4 0:00.28 apache2 Not only is bitcoind a memory hog, it is also a disk hog. You can disable the wallet, set the cache lower and decrease the maximum connections but that's about all you can do to minimise its ram usage. It will still use heaps of ram. Short of a rewrite, or using different software, there's no way around it.
What do you think is the lowest that I can set cache and maximum connections to? According to https://en.bitcoin.it/wiki/Running_Bitcoin , there is a dbcache setting, but no cache setting. Are you referring to dbcache?
|
|
|
|
|