Full Node: Out of Memory Error

[Arthur38]

Hello,

I have a Full Node running on 0.13.2. However, every 24 hours it crashes with this message:

Code:

Feb 19 20:32:37 kernel: [258198.535771] [16697]  1000 16697   413383   104742     350       5       60             0 bitcoind
Feb 19 20:32:37 kernel: [258198.535773] Out of memory: Kill process 16697 (bitcoind) score 33 or sacrifice child
Feb 19 20:32:37 kernel: [258198.540063] Killed process 16697 (bitcoind) total-vm:1653532kB, anon-rss:402216kB, file-rss:16752kB

I have 4 GB of ram and 8 GB of swap.

Thank you for your help.

[VelhoDoRio]

Can you provide more data?
Like:
Your system;
Can you replicate the error in some way?
Your bitcoin.conf settings.
What is the load of your system when running the node.

[cr1776]

I am curious what else is running on the system. In my experience that error is from the Linux kernel killing a process because the system is low on memory as a whole. Then the out of memory killer picks a process to kill based on various variables rolled into a score - hence your score 33.

Is something else on the machine memory and swap intensive?  If so, it may be that process that is causing the issue and bitcoin just happens to be the process that the OOM killer is picking based on its heuristics to kill.

Obviously much of this is a guess without more info.

[nemgun]

You can paste this command in a terminal : df -h --total
you will have the amount of free disk space.
Now you can monitore the computer/VPS using htop: sudo apt-get install -y htop && htop
You will have several informations displayed, like the CPU usage, mommory usage, the ressources used by each process ...

You have some solutions to check what is going on, could you try to add "prune = 50000" to your bitcoin.conf file after deleting the blockchain ?
This command will let you download the last 50k blocks instead of downloading the full blockchain, this way you will spare some space for the next blocks.

As you have a 8gb swap, i don't think the problem is comming from there, but you will for sure notice a big usage of disc space, so your best chance is to prune the blockchain.

[Arthur38]

Here is the list of process running at the time of the crash.

Code:

Feb 19 20:32:37 kernel: [258198.535703] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Feb 19 20:32:37 kernel: [258198.535708] [  406]     0   406    10180     1229      21       4      558             0 systemd-journal
Feb 19 20:32:37 kernel: [258198.535710] [  432]     0   432    23693      245      16       3        3             0 lvmetad
Feb 19 20:32:37 kernel: [258198.535711] [  468]     0   468    10748      800      23       3        2         -1000 systemd-udevd
Feb 19 20:32:37 kernel: [258198.535713] [  550]   100   550    25081      539      20       4       13             0 systemd-timesyn
Feb 19 20:32:37 kernel: [258198.535715] [  965]     0   965     4030      622      12       3       22             0 dhclient
Feb 19 20:32:37 kernel: [258198.535716] [ 1085]     0  1085     1306       29       8       3        0             0 iscsid
Feb 19 20:32:37 kernel: [258198.535718] [ 1086]     0  1086     1431      879       9       3        0           -17 iscsid
Feb 19 20:32:37 kernel: [258198.535720] [ 1090]     0  1090    68652     1038      36       3       23             0 accounts-daemon
Feb 19 20:32:37 kernel: [258198.535721] [ 1092]     0  1092     6511      452      19       3        1             0 atd
Feb 19 20:32:37 kernel: [258198.535723] [ 1094]     0  1094     6932      589      18       3       46             0 cron
Feb 19 20:32:37 kernel: [258198.535724] [ 1096]   104  1096    65158      738      29       3       18             0 rsyslogd
Feb 19 20:32:37 kernel: [258198.535726] [ 1098]     0  1098    77216      352      20       3        7             0 lxcfs
Feb 19 20:32:37 kernel: [258198.535728] [ 1102]     0  1102   141470     3735      66       6     1492          -500 dockerd
Feb 19 20:32:37 kernel: [258198.535729] [ 1104]   107  1104    10758      732      26       3       27          -900 dbus-daemon
Feb 19 20:32:37 kernel: [258198.535731] [ 1115]     0  1115     1100      298       7       3        1             0 acpid
Feb 19 20:32:37 kernel: [258198.535732] [ 1143]     0  1143     7545     1059      18       3       21             0 systemd-logind
Feb 19 20:32:37 kernel: [258198.535734] [ 1150]     0  1150    16380      901      36       3       30         -1000 sshd
Feb 19 20:32:37 kernel: [258198.535736] [ 1161]     0  1161     3344       35      11       3        2             0 mdadm
Feb 19 20:32:37 kernel: [258198.535737] [ 1165]     0  1165    69278      843      39       3        8             0 polkitd
Feb 19 20:32:37 kernel: [258198.535739] [ 1242]     0  1242     4868       51      14       3       13             0 irqbalance
Feb 19 20:32:37 kernel: [258198.535741] [ 1256]     0  1256     3619      460      12       3        5             0 agetty
Feb 19 20:32:37 kernel: [258198.535742] [ 1257]     0  1257     3665      365      11       3        4             0 agetty
Feb 19 20:32:37 kernel: [258198.535744] [ 1259]     0  1259    72894     1833      28       5      318          -500 docker-containe
Feb 19 20:32:37 kernel: [258198.535745] [ 1680]     0  1680    49036      955      23       5       82          -500 docker-proxy
Feb 19 20:32:37 kernel: [258198.535747] [ 1686]     0  1686    51736      229      21       5      147          -500 docker-containe
Feb 19 20:32:37 kernel: [258198.535748] [ 1705]     0  1705     4544       88      15       3      119             0 bash
Feb 19 20:32:37 kernel: [258198.535750] [ 1803]     0  1803    63847      691      98       3     1065             0 apache2
Feb 19 20:32:37 kernel: [258198.535752] [ 1809]    33  1809    83011     1473     101       3     1011             0 apache2
Feb 19 20:32:37 kernel: [258198.535753] [ 1813]    33  1813    82994     1242     101       3     1048             0 apache2
Feb 19 20:32:37 kernel: [258198.535755] [ 1814]    33  1814    82995     1214     101       3     1044             0 apache2
Feb 19 20:32:37 kernel: [258198.535756] [ 1815]    33  1815    82976     1216     101       3     1023             0 apache2
Feb 19 20:32:37 kernel: [258198.535758] [ 1816]    33  1816    82995     1401     101       3     1020             0 apache2
Feb 19 20:32:37 kernel: [258198.535759] [ 1817]    33  1817    83058     1314     101       3     1047             0 apache2
Feb 19 20:32:37 kernel: [258198.535761] [ 1818]    33  1818    82988     1208     101       3     1044             0 apache2
Feb 19 20:32:37 kernel: [258198.535762] [ 6992]     0  6992    35607     1924      26       5      146             0 snapd
Feb 19 20:32:37 kernel: [258198.535764] [ 8550]    33  8550    82968     1439     101       3      993             0 apache2
Feb 19 20:32:37 kernel: [258198.535765] [ 8552]    33  8552    82962     1286     101       3     1021             0 apache2
Feb 19 20:32:37 kernel: [258198.535767] [ 8553]    33  8553    82968     1247     101       3     1023             0 apache2
Feb 19 20:32:37 kernel: [258198.535768] [ 8554]    33  8554    82960     1244     101       3     1020             0 apache2
Feb 19 20:32:37 kernel: [258198.535769] [ 8555]    33  8555    63973     1187      94       3     1008             0 apache2
Feb 19 20:32:37 kernel: [258198.535771] [16697]  1000 16697   413383   104742     350       5       60             0 bitcoind

I have an apache2 server running in a docker (looks like they use a lot of memory?).
Blockchain is 100% downloaded.

[nemgun]

you should have a look at the mommory usage bars in htop, if it goes over 100% or not, and you should also check the swap usage which is bellow memory usage.
Have you checked debug.log ?
Could you try to find the lines writen there before the crash happens ?

[Arthur38]

In htop I see that the node is currently using 579M of memory.

Line written before bitcoind is killed in /var/lol/syslog:

Code:

Feb 19 20:32:37 kernel: [258198.535578] dhclient invoked oom-killer: gfp_mask=0x26000c0, order=2, oom_score_adj=0
Feb 19 20:32:37 kernel: [258198.535580] dhclient cpuset=/ mems_allowed=0
Feb 19 20:32:37 kernel: [258198.535584] CPU: 0 PID: 965 Comm: dhclient Not tainted 4.4.0-62-generic #83-Ubuntu
Feb 19 20:32:37 kernel: [258198.535586] Hardware name: Xen HVM domU, BIOS 4.2.amazon 12/09/2016
Feb 19 20:32:37 kernel: [258198.535587]  0000000000000286 000000006830faae ffff8800e850faf0 ffffffff813f7c63
Feb 19 20:32:37 kernel: [258198.535590]  ffff8800e850fcc8 ffff880015006200 ffff8800e850fb60 ffffffff8120ad4e
Feb 19 20:32:37 kernel: [258198.535592]  ffffffff81cd2d7f 0000000000000000 ffffffff81e67760 0000000000000206
Feb 19 20:32:37 kernel: [258198.535594] Call Trace:
Feb 19 20:32:37 kernel: [258198.535600]  [<ffffffff813f7c63>] dump_stack+0x63/0x90
Feb 19 20:32:37 kernel: [258198.535603]  [<ffffffff8120ad4e>] dump_header+0x5a/0x1c5
Feb 19 20:32:37 kernel: [258198.535607]  [<ffffffff811926c2>] oom_kill_process+0x202/0x3c0
Feb 19 20:32:37 kernel: [258198.535608]  [<ffffffff81192ae9>] out_of_memory+0x219/0x460
Feb 19 20:32:37 kernel: [258198.535611]  [<ffffffff81198a5d>] __alloc_pages_slowpath.constprop.88+0x8fd/0xa70
Feb 19 20:32:37 kernel: [258198.535614]  [<ffffffff81198e56>] __alloc_pages_nodemask+0x286/0x2a0
Feb 19 20:32:37 kernel: [258198.535616]  [<ffffffff81198f0b>] alloc_kmem_pages_node+0x4b/0xc0
Feb 19 20:32:37 kernel: [258198.535619]  [<ffffffff8107ea5e>] copy_process+0x1be/0x1b70
Feb 19 20:32:37 kernel: [258198.535623]  [<ffffffff8170fae8>] ? sock_sendmsg+0x38/0x50
Feb 19 20:32:37 kernel: [258198.535625]  [<ffffffff817100d1>] ? SYSC_sendto+0x101/0x190
Feb 19 20:32:37 kernel: [258198.535627]  [<ffffffff810805a0>] _do_fork+0x80/0x360
Feb 19 20:32:37 kernel: [258198.535629]  [<ffffffff81080929>] SyS_clone+0x19/0x20
Feb 19 20:32:37 kernel: [258198.535632]  [<ffffffff818385f2>] entry_SYSCALL_64_fastpath+0x16/0x71
Feb 19 20:32:37 kernel: [258198.535634] Mem-Info:
Feb 19 20:32:37 kernel: [258198.535637] active_anon:55728 inactive_anon:62585 isolated_anon:0
Feb 19 20:32:37 kernel: [258198.535637]  active_file:417178 inactive_file:416336 isolated_file:0
Feb 19 20:32:37 kernel: [258198.535637]  unevictable:929 dirty:6 writeback:0 unstable:0
Feb 19 20:32:37 kernel: [258198.535637]  slab_reclaimable:34325 slab_unreclaimable:5712
Feb 19 20:32:37 kernel: [258198.535637]  mapped:8525 shmem:3376 pagetables:2259 bounce:0
Feb 19 20:32:37 kernel: [258198.535637]  free:8451 free_pcp:0 free_cma:0
Feb 19 20:32:37 kernel: [258198.535640] Node 0 DMA free:15788kB min:28kB low:32kB high:40kB active_anon:0kB inactive_anon:8kB active_file:12kB inactive_file:4kB unevictable:32kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:32kB dirty:0kB writeback:0kB mapped:12kB shmem:0kB slab_reclaimable:20kB slab_unreclaimable:40kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535644] lowmem_reserve[]: 0 3745 3934 3934 3934
Feb 19 20:32:37 kernel: [258198.535647] Node 0 DMA32 free:16888kB min:7528kB low:9408kB high:11292kB active_anon:211756kB inactive_anon:227384kB active_file:1603828kB inactive_file:1599912kB unevictable:3376kB isolated(anon):0kB isolated(file):0kB present:3915776kB managed:3835152kB mlocked:3376kB dirty:24kB writeback:0kB mapped:32608kB shmem:8864kB slab_reclaimable:125296kB slab_unreclaimable:20152kB kernel_stack:2656kB pagetables:8312kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535651] lowmem_reserve[]: 0 0 189 189 189
Feb 19 20:32:37 kernel: [258198.535654] Node 0 Normal free:1128kB min:380kB low:472kB high:568kB active_anon:11156kB inactive_anon:22948kB active_file:64872kB inactive_file:65428kB unevictable:308kB isolated(anon):0kB isolated(file):0kB present:393216kB managed:193908kB mlocked:308kB dirty:0kB writeback:0kB mapped:1480kB shmem:4640kB slab_reclaimable:11984kB slab_unreclaimable:2656kB kernel_stack:784kB pagetables:724kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Feb 19 20:32:37 kernel: [258198.535657] lowmem_reserve[]: 0 0 0 0 0
Feb 19 20:32:37 kernel: [258198.535660] Node 0 DMA: 5*4kB (ME) 5*8kB (MEH) 3*16kB (MH) 2*32kB (EH) 2*64kB (UE) 3*128kB (UME) 3*256kB (UME) 2*512kB (ME) 3*1024kB (MEH) 1*2048kB (E) 2*4096kB (M) = 15788kB
Feb 19 20:32:37 kernel: [258198.535670] Node 0 DMA32: 625*4kB (UME) 1553*8kB (UME) 3*16kB (H) 1*32kB (H) 0*64kB 1*128kB (H) 1*256kB (H) 1*512kB (H) 1*1024kB (H) 0*2048kB 0*4096kB = 16924kB
Feb 19 20:32:37 kernel: [258198.535679] Node 0 Normal: 142*4kB (UME) 76*8kB (UME) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1176kB
Feb 19 20:32:37 kernel: [258198.535686] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Feb 19 20:32:37 kernel: [258198.535687] 838165 total pagecache pages
Feb 19 20:32:37 kernel: [258198.535696] 627 pages in swap cache
Feb 19 20:32:37 kernel: [258198.535697] Swap cache stats: add 87692, delete 87065, find 64623/66921
Feb 19 20:32:37 kernel: [258198.535698] Free swap  = 8350600kB
Feb 19 20:32:37 kernel: [258198.535699] Total swap = 8388604kB
Feb 19 20:32:37 kernel: [258198.535700] 1081245 pages RAM
Feb 19 20:32:37 kernel: [258198.535700] 0 pages HighMem/MovableOnly
Feb 19 20:32:37 kernel: [258198.535701] 70004 pages reserved
Feb 19 20:32:37 kernel: [258198.535702] 0 pages cma reserved
Feb 19 20:32:37 kernel: [258198.535703] 0 pages hwpoisoned

And debug.log before and after crash (20:32):

Code:

2017-02-19 20:32:17 receive version message: /BitCoinJ:0.11.2/MultiBit:0.5.18/: version 70001, blocks=412192, us=127.0.0.1:8333, peer=321
2017-02-19 20:32:18 connection from 138.197.197.152:31044 dropped (banned)
2017-02-19 20:32:20 connection from 138.197.197.164:60006 dropped (banned)
2017-02-19 20:32:26 connection from 138.68.10.138:57854 dropped (banned)
2017-02-19 20:32:26 connection from 138.197.197.108:15696 dropped (banned)
2017-02-19 20:32:28 connection from 138.197.197.152:33434 dropped (banned)
2017-02-19 20:32:29 connection from 138.197.195.32:38820 dropped (banned)
2017-02-19 20:32:32 connection from 138.197.194.32:8144 dropped (banned)
2017-02-19 20:32:33 connection from 138.197.197.174:43606 dropped (banned)
2017-02-19 20:32:35 connection from 138.197.197.179:4758 dropped (banned)
2017-02-19 20:32:36 connection from 138.197.197.108:18216 dropped (banned)
2017-02-19 20:35:01 
2017-02-19 20:35:01 Bitcoin version v0.13.2.0-g0d71914
2017-02-19 20:35:01 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2017-02-19 20:35:01 Default data directory /home/ubuntu/.bitcoin
2017-02-19 20:35:01 Using data directory /home/ubuntu/.bitcoin
2017-02-19 20:35:01 Using config file /home/ubuntu/.bitcoin/bitcoin.conf
2017-02-19 20:35:01 Using at most 125 connections (1024 file descriptors available)
2017-02-19 20:35:01 Using 2 threads for script verification
2017-02-19 20:35:01 scheduler thread start

Thank you for your help.

[nemgun]

I am sorry but i don't see any relevant information here, maybe you should ask for moderators as some of them are part of the development team, at the same time. Or you can send these logs in a github issue and post the link here so we can follow what they are saying, i am interested to see what is the cause of this problem as i run a bitcoin full node, it may be usefull in the future.

[Arthur38]

Okay thank you for your help.
Problem does not seem to be straightforward.

Will open a GitHub issue.