Bitcoind stops while syncing - "Failed to open mempool file from disk"

[merc1er]

I installed Bitcoin Core on a Raspberry Pi and tried to sync the block chain using bitcoind.
The data is written on an external hard drive (1TB) mounted on the machine.

After I run the `bitcoind -daemon` command, the block chain starts syncing, but after a few hours it stops.
This is what I have at the end of the debug.log file:

----
2019-02-10T05:41:04Z UpdateTip: new best=00000000727339b1f121c82ad73e522d8e83d7ed4b1b3473aa10f26029f2b046 height=12774 version=0x00000001 log2_work=45.641058 tx=12882 date='2009-04-30T12:58:37Z' progress=0.000035 cache=2.4MiB(12863txo)
2019-02-10T05:41:04Z Failed to open mempool file from disk. Continuing anyway.
2019-02-10T05:41:04Z tor: Thread interrupt
2019-02-10T05:41:04Z Shutdown: In progress...
2019-02-10T05:41:04Z addcon thread exit
2019-02-10T05:41:04Z opencon thread exit
2019-02-10T05:41:04Z torcontrol thread exit
2019-02-10T05:41:04Z net thread exit
2019-02-10T05:41:04Z msghand thread exit
2019-02-10T05:41:04Z scheduler thread interrupt
2019-02-10T05:41:06Z [default wallet] Releasing wallet
2019-02-10T05:41:06Z Shutdown: done
----

Does anyone have an idea? Thank you!

[Pmalek]

It could be that you have a corrupted block. Have you tried deleting the entire blockchain and letting Bitcoin Core download it again from the beginning?
Just make sure you have a backup of your wallet.dat file to be on the safe side.

[KingZee]

Since it's just starting to sync there's no harm in resyncing.

It does look like it's corrupt, delete the blocks and chainstate folders and restart bitcoind.

[aliashraf]

OP, It looks to me as a hardware failure, you need to check your hardware anyway:

• Test memory integrity of your raspberry device for this you want memtester. You would simply follow this thread in raspberry forum: https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=208821&hilit=memtester&start=75#p1299726

• Run a stress test on your device. This would be a useful resource: https://core-electronics.com.au/tutorials/stress-testing-your-raspberry-pi.html

• Check your external hdd using  an open source tool like https://www.smartmontools.org/

[darosior]

I installed Bitcoin Core on a Raspberry Pi and tried to sync the block chain using bitcoind.
The data is written on an external hard drive (1TB) mounted on the machine.

After I run the `bitcoind -daemon` command, the block chain starts syncing, but after a few hours it stops.
This is what I have at the end of the debug.log file:

----
2019-02-10T05:41:04Z UpdateTip: new best=00000000727339b1f121c82ad73e522d8e83d7ed4b1b3473aa10f26029f2b046 height=12774 version=0x00000001 log2_work=45.641058 tx=12882 date='2009-04-30T12:58:37Z' progress=0.000035 cache=2.4MiB(12863txo)
2019-02-10T05:41:04Z Failed to open mempool file from disk. Continuing anyway.
2019-02-10T05:41:04Z tor: Thread interrupt
2019-02-10T05:41:04Z Shutdown: In progress...
2019-02-10T05:41:04Z addcon thread exit
2019-02-10T05:41:04Z opencon thread exit
2019-02-10T05:41:04Z torcontrol thread exit
2019-02-10T05:41:04Z net thread exit
2019-02-10T05:41:04Z msghand thread exit
2019-02-10T05:41:04Z scheduler thread interrupt
2019-02-10T05:41:06Z [default wallet] Releasing wallet
2019-02-10T05:41:06Z Shutdown: done
----

Does anyone have an idea? Thank you!

Hi,

Have you touched you external drive ? It mights just be a partition problem. Try to shutdown, remount your drive (specifying the right path to bitcoind). Don't forget to check the permissions, you'll surely mount the drive as root while bitcoind runs as another user. If it works then you should consider using fstab so the drive is always mounted in the same dir with the same permissions.

[merc1er]

delete the blocks and chainstate folders and restart bitcoind.

It restarted from the first block already, but i'm trying this one more time.

@darosior I changed ownership to `pi` and chmod-ed the hard drive mount. Does bitcoind use a specific user that I need to change to?

Also, note that I didn't specify a path to save the block chain to bitcoind - instead, the folder .bitcoin links to my drive.

[darosior]

delete the blocks and chainstate folders and restart bitcoind.

It restarted from the first block already, but i'm trying this one more time.

@darosior I changed ownership to `pi` and chmod-ed the hard drive mount. Does bitcoind use a specific user that I need to change to?

Also, note that I didn't specify a path to save the block chain to bitcoind - instead, the folder .bitcoin links to my drive.

`bitcoind` uses a specific user only if you specify it (for instance in a systemd service).

[merc1er]

Thank you! I didn't specify such thing.

It's now re-syncing (about 2%). Let's see if it goes through.

[merc1er]

It stopped with the following at the end of `debug.log`:

---
2019-02-10T15:18:11Z UpdateTip: new best=00000000000000249597632a2fe08578cd1d2c0f620342ca91c3611786fe464f height=232591 version=0x00000002 log2_work=69.838772 tx=16550595 date='2013-04-22T14:31:02Z' progress=0.044502 cache=597.4MiB(5216340txo)
2019-02-10T15:18:12Z Error: Out of memory. Terminating.
---

I rebooted the raspi and it went straight to recovery mode... I will start from scratch again and do the hardware checks mentioned above.

[aliashraf]

It stopped with the following at the end of `debug.log`:

---
2019-02-10T15:18:11Z UpdateTip: new best=00000000000000249597632a2fe08578cd1d2c0f620342ca91c3611786fe464f height=232591 version=0x00000002 log2_work=69.838772 tx=16550595 date='2013-04-22T14:31:02Z' progress=0.044502 cache=597.4MiB(5216340txo)
2019-02-10T15:18:12Z Error: Out of memory. Terminating.
---

I rebooted the raspi and it went straight to recovery mode... I will start from scratch again and do the hardware checks mentioned above.

And please keep us informed, as mentioned above 

[darosior]

It stopped with the following at the end of `debug.log`:

---
2019-02-10T15:18:11Z UpdateTip: new best=00000000000000249597632a2fe08578cd1d2c0f620342ca91c3611786fe464f height=232591 version=0x00000002 log2_work=69.838772 tx=16550595 date='2013-04-22T14:31:02Z' progress=0.044502 cache=597.4MiB(5216340txo)
2019-02-10T15:18:12Z Error: Out of memory. Terminating.
---

I rebooted the raspi and it went straight to recovery mode... I will start from scratch again and do the hardware checks mentioned above.

The synchronization seems to be too resource-incentive for a rpi. You might want to take a look to https://bitcoin.org/en/full-node#reduce-storage (there are options to reduce traffic too).

[merc1er]

You might want to take a look to https://bitcoin.org/en/full-node#reduce-storage (there are options to reduce traffic too).

I do need to run an unpruned node as my goal is to run a lightning node 

[darosior]

You might want to take a look to https://bitcoin.org/en/full-node#reduce-storage (there are options to reduce traffic too).

I do need to run an unpruned node as my goal is to run a lightning node 

Since v0.6, C-Lightning does support running on top of pruned nodes, and the doc I provided is not only for pruned nodes (for instance just lower the `dbcache` would reduce the RAM usage).

[merc1er]

Since v0.6, C-Lightning does support running on top of pruned nodes, and the doc I provided is not only for pruned nodes (for instance just lower the `dbcache` would reduce the RAM usage).

Oh that’s interesting! I will look into it, thank you.

[Lauda]

It stopped with the following at the end of `debug.log`:

---
2019-02-10T15:18:11Z UpdateTip: new best=00000000000000249597632a2fe08578cd1d2c0f620342ca91c3611786fe464f height=232591 version=0x00000002 log2_work=69.838772 tx=16550595 date='2013-04-22T14:31:02Z' progress=0.044502 cache=597.4MiB(5216340txo)
2019-02-10T15:18:12Z Error: Out of memory. Terminating.
---
I rebooted the raspi and it went straight to recovery mode... I will start from scratch again and do the hardware checks mentioned above.

Have you tried reducing the size of the cache?

[merc1er]

Okay so I am giving it another try with the following parameters:

Code:

bitcoind -daemon -maxmempool=100 -dbcache=200 -maxsigcachesize=4

I tried with -dbcache set to 50 but the verification progress (syncing) was way too slow!

It seems to be syncing a bit slower than before with the current parameters but let's hope it keeps the RAM usage below 80%.

[merc1er]

So it has been syncing overnight and I am now at 11%.

Bitcoind RAM usage remains at about 50% (500MB). Looks like it's working! Thank you for the support. 

[merc1er]

It's very slow IMO, but i guess it's better than can't sync properly.

Depending on percentage free RAM, you could increase dbcache or/and maxsigcachesize number slightly (between 5-25%) if you want faster sync and prepare to take risks. If you have PC/notebook, you could sync from it and on Pi you just need to wait for block verification.

Indeed, 17% after 2 days. I increased -dbcache by 25% but RAM went over 95%.
I will be playing with the arguments to reach a faster stable syncing.

[darosior]

It's very slow IMO, but i guess it's better than can't sync properly.

Depending on percentage free RAM, you could increase dbcache or/and maxsigcachesize number slightly (between 5-25%) if you want faster sync and prepare to take risks. If you have PC/notebook, you could sync from it and on Pi you just need to wait for block verification.

Indeed, 17% after 2 days. I increased -dbcache by 25% but RAM went over 95%.
I will be playing with the arguments to reach a faster stable syncing.

It is indeed slow and the higher the blockheight, the slower it will become (larger blocks). You should use `-blocksonly` and `banscore=1` as mentionned in the link @ETFBitcoin provided.