bitcoin core full node keeps crashing after power failure

[callmejoe]

been running core for about a month on my raspberry pi2 without a problem.  had a power failure last week and ever since it keeps crashing. i was seeing corruption errors in my debug.log so i copied over new copies of the block and chainstate folders from another machine.  now runs fine for a few hours then crashes.  i've checked ram with memtest and all seems fine with hardware. lots of error in the debug.log. 

can anyone help?

Code:

2019-04-30T21:42:57Z Bitcoin Core version v0.17.1 (release build)
2019-04-30T21:42:57Z InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2019-04-30T21:42:57Z Assuming ancestors of block 0000000000000000002e63058c023a9a1de233554f28c7b21380b6c9003f36a8 have valid signatures.
2019-04-30T21:42:57Z Setting nMinimumChainWork=0000000000000000000000000000000000000000028822fef1c230963535a90d
2019-04-30T21:42:57Z Using the 'standard' SHA256 implementation
2019-04-30T21:42:57Z Default data directory /home/pi/.bitcoin
2019-04-30T21:42:57Z Using data directory /media/pi/crypto
2019-04-30T21:42:57Z Using config file /media/pi/crypto/bitcoin.conf
2019-04-30T21:42:57Z Using at most 125 automatic connections (65536 file descriptors available)
2019-04-30T21:42:57Z Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2019-04-30T21:42:57Z Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2019-04-30T21:42:57Z Using 4 threads for script verification
2019-04-30T21:42:57Z scheduler thread start
2019-04-30T21:42:57Z Binding RPC on address 0.0.0.0 port 8332 failed.
2019-04-30T21:42:57Z HTTP: creating work queue of depth 16
2019-04-30T21:42:57Z Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcauth for rpcauth auth generation.
2019-04-30T21:42:57Z HTTP: starting 4 worker threads
2019-04-30T21:42:57Z Using wallet directory /media/pi/crypto
2019-04-30T21:42:57Z init message: Verifying wallet(s)...
2019-04-30T21:42:57Z Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2019-04-30T21:42:57Z Using wallet wallet.dat
2019-04-30T21:42:57Z BerkeleyEnvironment::Open: LogDir=/media/pi/crypto/database ErrorFile=/media/pi/crypto/db.log
2019-04-30T21:42:57Z Cache configuration:
2019-04-30T21:42:57Z * Using 2.0MiB for block index database
2019-04-30T21:42:57Z * Using 8.0MiB for chain state database
2019-04-30T21:42:57Z * Using 440.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2019-04-30T21:42:57Z init message: Loading block index...
2019-04-30T21:42:57Z Opening LevelDB in /media/pi/crypto/blocks/index
2019-04-30T21:42:57Z Opened LevelDB successfully
2019-04-30T21:42:57Z Using obfuscation key for /media/pi/crypto/blocks/index: 0000000000000000
2019-04-30T21:43:32Z LoadBlockIndexDB: last block file = 1619
2019-04-30T21:43:32Z LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=27, size=36640374, heights=573953...573979, time=2019-04-30...2019-04-30)
2019-04-30T21:43:32Z Checking all blk files are present...
2019-04-30T21:43:33Z Opening LevelDB in /media/pi/crypto/chainstate
2019-04-30T21:43:35Z Opened LevelDB successfully
2019-04-30T21:43:35Z Using obfuscation key for /media/pi/crypto/chainstate: 98e878d5582f791d
2019-04-30T21:43:36Z Loaded best chain: hashBestChain=00000000000000000003919fc59de4197931af8a04b4bdf583c5f4138ffb0963 height=573979 date=2019-04-30T21:37:54Z progress=0.999998
2019-04-30T21:43:36Z init message: Rewinding blocks...
2019-04-30T21:43:45Z init message: Verifying blocks...
2019-04-30T21:43:45Z Verifying last 6 blocks at level 3
2019-04-30T21:43:45Z [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2019-04-30T21:44:12Z No coin database inconsistencies in last 6 blocks (17580 transactions)
2019-04-30T21:44:12Z  block index           74625ms
2019-04-30T21:44:12Z init message: Loading wallet...
2019-04-30T21:44:13Z [default wallet] nFileVersion = 170100
2019-04-30T21:44:13Z [default wallet] Keys: 2003 plaintext, 0 encrypted, 2003 w/ metadata, 2003 total. Unknown wallet records: 1
2019-04-30T21:44:13Z [default wallet] Wallet completed loading in            1053ms
2019-04-30T21:44:13Z [default wallet] setKeyPool.size() = 2000
2019-04-30T21:44:13Z [default wallet] mapWallet.size() = 1
2019-04-30T21:44:13Z [default wallet] mapAddressBook.size() = 2
2019-04-30T21:44:13Z mapBlockIndex.size() = 573981
2019-04-30T21:44:13Z nBestHeight = 573979
2019-04-30T21:44:13Z torcontrol thread start
2019-04-30T21:44:13Z Bound to [::]:8333
2019-04-30T21:44:13Z Bound to 0.0.0.0:8333
2019-04-30T21:44:13Z init message: Loading P2P addresses...
2019-04-30T21:44:13Z Leaving InitialBlockDownload (latching to false)
2019-04-30T21:44:13Z Loaded 8921 addresses from peers.dat  254ms
2019-04-30T21:44:13Z init message: Loading banlist...
2019-04-30T21:44:13Z init message: Starting network threads...
2019-04-30T21:44:13Z net thread start
2019-04-30T21:44:13Z dnsseed thread start
2019-04-30T21:44:13Z addcon thread start
2019-04-30T21:44:13Z opencon thread start
2019-04-30T21:44:13Z init message: Done loading
2019-04-30T21:44:13Z msghand thread start
2019-04-30T21:44:14Z New outbound peer connected: version: 70015, blocks=573979, peer=0
2019-04-30T21:44:15Z New outbound peer connected: version: 70015, blocks=573979, peer=1
2019-04-30T21:44:16Z New outbound peer connected: version: 70015, blocks=573979, peer=3
2019-04-30T21:44:16Z New outbound peer connected: version: 70015, blocks=573979, peer=4
2019-04-30T21:44:23Z New outbound peer connected: version: 70015, blocks=573979, peer=6
2019-04-30T21:44:24Z P2P peers available. Skipped DNS seeding.
2019-04-30T21:44:24Z dnsseed thread exit
2019-04-30T21:44:29Z New outbound peer connected: version: 70015, blocks=573979, peer=7
2019-04-30T21:44:30Z New outbound peer connected: version: 70015, blocks=573979, peer=9
2019-04-30T21:44:31Z New outbound peer connected: version: 70015, blocks=573979, peer=10
2019-04-30T21:45:23Z UpdateTip: new best=00000000000000000017d6c79a1731823500259a440447c93e56704d0369c565 height=573980 version=0x2000e000 log2_work=90.592798 tx=408151121 date='2019-04-30T21:45:00Z' progress=1.000000 cache=1.9MiB(17206txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T21:47:00Z Imported mempool transactions from disk: 8775 succeeded, 656 failed, 0 expired, 6 already there
2019-04-30T21:54:22Z UpdateTip: new best=00000000000000000019767c51510a1f76155603e64aedcc4cca243cc2947bd7 height=573981 version=0x20000000 log2_work=90.592819 tx=408154049 date='2019-04-30T21:53:51Z' progress=1.000000 cache=4.2MiB(37050txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T22:12:11Z UpdateTip: new best=000000000000000000040175210ba17f8626f22a6228f6faf8d0934d036a7a62 height=573982 version=0x20000000 log2_work=90.59284 tx=408157446 date='2019-04-30T22:11:42Z' progress=1.000000 cache=5.3MiB(47272txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T22:13:53Z UpdateTip: new best=000000000000000000072a0aee53b339ade933011060b47c870452cb6de286a3 height=573983 version=0x20000000 log2_work=90.592861 tx=408160279 date='2019-04-30T22:13:38Z' progress=1.000000 cache=6.1MiB(53386txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T22:15:12Z UpdateTip: new best=00000000000000000029bb2f3d59de60d16218e1ec4ed80bb7b291f8c6c227fb height=573984 version=0x20000000 log2_work=90.592883 tx=408162627 date='2019-04-30T22:14:58Z' progress=1.000000 cache=6.8MiB(59268txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T22:15:23Z ERROR: AcceptBlockHeader: Consensus::ContextualCheckBlockHeader: 000000000000001daed8bac77bdf571441649e51732e91bec984cf97a39005a1, bad-diffbits, incorrect proof of work (code 16)
2019-04-30T22:27:28Z socket send error Connection reset by peer (104)
2019-04-30T22:27:30Z UpdateTip: new best=0000000000000000002265e5cec31906641c6141089871376b1ee1e768f9cbac height=573985 version=0x20000000 log2_work=90.592904 tx=408164786 date='2019-04-30T22:26:51Z' progress=1.000000 cache=8.0MiB(70716txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T22:27:47Z Pre-allocating up to position 0x700000 in rev01619.dat
2019-04-30T22:27:47Z UpdateTip: new best=000000000000000000166bc190f9a844b037d19b94588d9f51de94dab0020501 height=573986 version=0x20c00000 log2_work=90.592925 tx=408168155 date='2019-04-30T22:27:30Z' progress=1.000000 cache=8.8MiB(78339txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T22:35:48Z ping timeout: 1200.047054s
2019-04-30T22:38:19Z UpdateTip: new best=00000000000000000015d3ea9afb425fbbf5bcb6a24f8e51d0b2997d903a93ee height=573987 version=0x20000000 log2_work=90.592946 tx=408170370 date='2019-04-30T22:38:04Z' progress=1.000000 cache=9.7MiB(86604txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T22:39:43Z socket recv error Connection reset by peer (104)
2019-04-30T22:52:41Z UpdateTip: new best=00000000000000000018c47cbdcba8db7eb85ae3f064c8b048859c2e118da32a height=573988 version=0x20000000 log2_work=90.592967 tx=408172669 date='2019-04-30T22:52:35Z' progress=1.000000 cache=10.8MiB(96703txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T22:57:43Z UpdateTip: new best=000000000000000000197624ca64a603f4dce4bde2a746d2b03defe6f9512e2e height=573989 version=0x20400000 log2_work=90.592988 tx=408174486 date='2019-04-30T22:56:51Z' progress=1.000000 cache=11.4MiB(102239txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T23:00:17Z Pre-allocating up to position 0x4000000 in blk01619.dat
2019-04-30T23:00:21Z UpdateTip: new best=0000000000000000000dbc39c8c51d0a82e2eecdcf069e88f75950b972f3a11d height=573990 version=0x20800000 log2_work=90.593009 tx=408176944 date='2019-04-30T23:00:11Z' progress=1.000000 cache=12.7MiB(110025txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T23:03:28Z UpdateTip: new best=000000000000000000163f792be5ff2cdea5ecf71f129ee4f9446576efb7d7bc height=573991 version=0x20000000 log2_work=90.59303 tx=408179185 date='2019-04-30T23:03:29Z' progress=1.000000 cache=13.5MiB(117411txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T23:05:50Z Pre-allocating up to position 0x800000 in rev01619.dat
2019-04-30T23:05:50Z UpdateTip: new best=0000000000000000000052bedba40be6984eaf74c3998c5770dda90fc868b12b height=573992 version=0x20c00000 log2_work=90.593051 tx=408180797 date='2019-04-30T23:05:37Z' progress=1.000000 cache=14.3MiB(125143txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T23:09:53Z socket recv error Connection reset by peer (104)
2019-04-30T23:10:30Z UpdateTip: new best=0000000000000000000e0def4bde69312d45ae7b9066a8c50714f3a4e32847cf height=573993 version=0x20c00000 log2_work=90.593072 tx=408183766 date='2019-04-30T23:09:50Z' progress=1.000000 cache=15.0MiB(131418txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T23:15:21Z socket send error Connection reset by peer (104)
2019-04-30T23:24:38Z socket recv error Connection reset by peer (104)
2019-04-30T23:27:53Z socket recv error Connection reset by peer (104)
2019-04-30T23:34:35Z socket recv error Connection reset by peer (104)
2019-04-30T23:38:23Z socket recv error Connection reset by peer (104)
2019-04-30T23:40:16Z socket recv error Connection reset by peer (104)
2019-04-30T23:40:30Z UpdateTip: new best=0000000000000000001ea792eb1a0b5fada2afc2f6bc45398eee19ef7b6e3932 height=573994 version=0x20000000 log2_work=90.593093 tx=408186718 date='2019-04-30T23:40:22Z' progress=1.000000 cache=16.0MiB(141622txo) warning='38 of last 100 blocks have unexpected version'
2019-04-30T23:41:11Z UpdateTip: new best=00000000000000000020ed3faef1fca2a94d3ed2db2df18f1ea9deda53059e8b height=573995 version=0x20000000 log2_work=90.593114 tx=408188268 date='2019-04-30T23:40:56Z' progress=1.000000 cache=16.8MiB(148255txo) warning='37 of last 100 blocks have unexpected version'
2019-04-30T23:42:32Z socket send error Broken pipe (32)
2019-04-30T23:45:14Z socket recv error Connection reset by peer (104)
2019-04-30T23:47:02Z socket recv error Connection reset by peer (104)
2019-04-30T23:51:00Z socket recv error Connection reset by peer (104)
2019-04-30T23:57:30Z UpdateTip: new best=0000000000000000002188e2d7beaaabe9a01e8212b8f75ba866b210a8bbf9c6 height=573996 version=0x20800000 log2_work=90.593136 tx=408191222 date='2019-04-30T23:57:00Z' progress=1.000000 cache=17.2MiB(152867txo) warning='37 of last 100 blocks have unexpected version'
2019-05-01T00:00:39Z socket recv error Connection reset by peer (104)
2019-05-01T00:00:48Z socket send error Broken pipe (32)
2019-05-01T00:04:19Z socket recv error Connection reset by peer (104)
2019-05-01T00:22:54Z UpdateTip: new best=0000000000000000001f4cbb286e91ac92dcc07903c6b4c8230eb54dd00e614e height=573997 version=0x2000e000 log2_work=90.593157 tx=408194316 date='2019-05-01T00:22:57Z' progress=1.000000 cache=18.2MiB(161582txo) warning='37 of last 100 blocks have unexpected version'
2019-05-01T00:24:41Z socket send error Broken pipe (32)
2019-05-01T00:26:13Z socket recv error Connection reset by peer (104)
2019-05-01T00:27:09Z socket send error Broken pipe (32)
2019-05-01T00:31:09Z socket send error Broken pipe (32)
2019-05-01T00:31:54Z socket send error Broken pipe (32)
2019-05-01T00:32:01Z ping timeout: 1200.034540s
2019-05-01T00:36:16Z socket send error Broken pipe (32)
2019-05-01T00:37:39Z Pre-allocating up to position 0x900000 in rev01619.dat
2019-05-01T00:37:40Z UpdateTip: new best=0000000000000000000d59369ed30e889720ebe2d207d49f592b36af186f0feb height=573998 version=0x20400000 log2_work=90.593178 tx=408196658 date='2019-05-01T00:37:05Z' progress=1.000000 cache=18.7MiB(166026txo) warning='38 of last 100 blocks have unexpected version'
2019-05-01T00:40:38Z socket send error Broken pipe (32)
2019-05-01T00:46:04Z UpdateTip: new best=00000000000000000011e3146b8239b2b9d8d62fe7dbbcb627a348bc5572c482 height=573999 version=0x20000000 log2_work=90.593199 tx=408199257 date='2019-05-01T00:46:21Z' progress=1.000000 cache=19.1MiB(169988txo) warning='38 of last 100 blocks have unexpected version'
2019-05-01T00:46:10Z socket send error Broken pipe (32)
2019-05-01T00:46:35Z socket send error Broken pipe (32)
2019-05-01T00:48:02Z LevelDB read failure: Corruption: not an sstable (bad magic number)
2019-05-01T00:48:02Z Fatal LevelDB error: Corruption: not an sstable (bad magic number)
2019-05-01T00:48:02Z You can use -debug=leveldb to get more complete diagnostic messages
2019-05-01T00:48:02Z Error: Error reading from database, shutting down.
2019-05-01T00:48:02Z Error reading from database: Fatal LevelDB error: Corruption: not an sstable (bad magic number)

[1713565099]

1713565099

[1713565099]

1713565099

[DaveF]

Did you try a reindex?
On a RPi it's going to take a while, but that should fix it.

-Dave

[callmejoe]

Did you try a reindex?
On a RPi it's going to take a while, but that should fix it.

-Dave

i thought i could avoid the reindex if i copied over fresh working copies of the blocks and chainstate.  but i guess that didnt work.

i'll give reindex a try.

[HCP]

If you still keep getting those errors even after a reindex, then the most likely culprit after that will be your storage drive (given that the RAM tested OK with MemTest)... I assume it's on a USB external HDD?

[NeuroticFish]

2019-05-01T00:48:02Z Error: Error reading from database, shutting down.
2019-05-01T00:48:02Z Error reading from database: Fatal LevelDB error: Corruption: not an sstable (bad magic number)

Since it runs for hours then it fails at reading the DB, an idea that came into my mind is that maybe your HDD got into trouble - from a cross link to a bad sector.
I would also check thoroughly the HDD. I mean some *nix equivalent for chkdsk /r

[klaaas]

i'll give reindex a try.

Reading here that could take up days/weeks.

Was the transaction index switched on (txindex=1) where you took the blockchain backup from ?

[Jet Cash]

If you had a power failure, you could have created a transient and knocked out a component. Do you get any failures when you run anything else?

[DaveF]

2019-05-01T00:48:02Z Error: Error reading from database, shutting down.
2019-05-01T00:48:02Z Error reading from database: Fatal LevelDB error: Corruption: not an sstable (bad magic number)

Since it runs for hours then it fails at reading the DB, an idea that came into my mind is that maybe your HDD got into trouble - from a cross link to a bad sector.
I would also check thoroughly the HDD. I mean some *nix equivalent for chkdsk /r

*nix command is fcsk
It might not run till the next reboot for repair.

i'll give reindex a try.

Reading here that could take up days/weeks.

Was the transaction index switched on (txindex=1) where you took the blockchain backup from ?

IIRC it should not matter, the error he gave in the 1st post points to the main DB index.
It's been a while since I saw it.

If you had a power failure, you could have created a transient and knocked out a component. Do you get any failures when you run anything else?

It's an RPi, they *tend*to be tough. (The SD card is another matter)

-Dave

[callmejoe]

i actually have a 500GB sdcard in the pi not a HDD.  tried an HDD but kept having problems.  paid a ton for that sd card so i hope its not trashed.

i did run fsck on all the partitions after the power failure and no errors were found.

and if it is some storage corruption i would expect the OS to crash all the time, but i have no other issues other than running bitcoin. when bitcoin core stops the OS does not.  so i am going on the assumption the sdcard is still good. for now anyway, at least until this reindex is done.  8 hours running and up to blk803.dat.....

[DaveF]

i actually have a 500GB sdcard in the pi not a HDD.  tried an HDD but kept having problems.  paid a ton for that sd card so i hope its not trashed.

i did run fsck on all the partitions after the power failure and no errors were found.

and if it is some storage corruption i would expect the OS to crash all the time, but i have no other issues other than running bitcoin. when bitcoin core stops the OS does not.  so i am going on the assumption the sdcard is still good. for now anyway, at least until this reindex is done.  8 hours running and up to blk803.dat.....

Any luck yet or still re-indexing?
What issues were you having with the HDD? I was never happy with the performance with spinning drives and the RPi. In theory it should not matter as the USB is the limiting factor but it just "felt" slow to me. An SSD with a good adapter worked for me for a few years.

-Dave

[callmejoe]

Any luck yet or still re-indexing?
What issues were you having with the HDD? I was never happy with the performance with spinning drives and the RPi. In theory it should not matter as the USB is the limiting factor but it just "felt" slow to me. An SSD with a good adapter worked for me for a few years.

-Dave

well after 20 or so hours it finished reindexing then it started reindexing the chainstate or maybe it was rescanning TX (not sure).  got stuck on height=366211. lost all my connections, then the server stopped.

i am going to reformat the partition i had set up on the sdcard for the blockchain files and try again to copy the blockchain from another machine.

the external HDD kept crashing on me. im not sure the pi was powerful enough to handle it so i just switched to the sdcard.

[callmejoe]

well i updated the copy of the blockchain and it was looking good for a few hours then crashed again.

what is weird is that there is no shutdown errors at the end of debug.log.  last entry is just normal update

Code:

2019-05-02T22:03:32Z UpdateTip: new best=00000000000000000026372f62915c43c04d54d50e79370e825fe0cd9a60c56d height=574335 version=0x20000000 log2_work=90.600264 tx=409043083 date='2019-05-02T22:03:03Z' progress=1.000000 cache=19.5MiB(174981txo) warning='41 of last 100 blocks have unexpected version'

maybe i will check the OS logs...

[DaveF]

1) Just to confirm it's a RPi 2 with 1GB of RAM. Not one of the older ones with only 512MB
2) Were you able to restart bigcoind?

Just bouncing some other ideas, since you had an issue running with an external drive, have you run something to really stress the RPi to see if it's a hardware issue?
A lot of people run external drives, so between that problem and this one it's not unreasonable to consider that it might be hardware. (Possible wall power adapter issue?)

-Dave

[callmejoe]

1) Just to confirm it's a RPi 2 with 1GB of RAM. Not one of the older ones with only 512MB
2) Were you able to restart bigcoind?

Just bouncing some other ideas, since you had an issue running with an external drive, have you run something to really stress the RPi to see if it's a hardware issue?
A lot of people run external drives, so between that problem and this one it's not unreasonable to consider that it might be hardware. (Possible wall power adapter issue?)

-Dave

yes 1GB RAM. its the raspberry pi2
checked OS logs, nothing special happening.  restarted bitcoind, currently running, we'll see what happens.

i ran that memtest and ram passed.  i havent run anything to stress the CPU.  any suggestions?  

i'll test the wall adapter too.  

i have a raspberry pi3 also. maybe i will load up bitcoin core on that and see what happens.

also, doesnt look to me like bitcoin core is too resource hungry outside of IBD or reindex options.

[DaveF]

No tools that I can think of. Everything I have is for x86.
Just seemed that with the crashes & external drive issue it might be hardware.
If it's working now I would not worry about it.

-Dave

[callmejoe]

crashed again overnight.  got a segfault error this time.  i think that indicates a memory issue??

i moved my current sdcard over to a newer raspberry pi3 i have.  let's see how it does. 

[callmejoe]

well it's been 3 days on the raspberry pi3 and no crashes.  i guess it was the hardware on the raspi2. like a few of you mentioned here the bitcoin core software i guess will expose any issues not seen before.  

i am using the raspberry pi2 now as a Kodi box with no issues.

thanks for all the help.


*EDIT* spoke one day too soon. just crashed again.  but maybe this is a different issue

Code:

2019-05-07T21:43:34Z ERROR: ReadBlockFromDisk: Deserialize or I/O error - non-canonical ReadCompactSize(): iostream error at CBlockDiskPos(nFile=1625, nPos=8)

[ABCbits]

*EDIT* spoke one day too soon. just crashed again.  but maybe this is a different issue

Code:

2019-05-07T21:43:34Z ERROR: ReadBlockFromDisk: Deserialize or I/O error - non-canonical ReadCompactSize(): iostream error at CBlockDiskPos(nFile=1625, nPos=8)

According to https://github.com/bitcoin/bitcoin/issues/8081 and https://bitcoin.stackexchange.com/a/70193, start Bitcoin Core once with reindex should fix/avoid the problem.

Corrupt storage drive also possible, did you same SDCard/HDD on your Raspberry Pi 3?

[DaveF]

Agree with ETFbitcoin, that looks to be a SD card slowly going bad.

Why it worked better in the RPi3 vs the 2 I can only guess but I have seen something similar here even among the same model pi.
Keeps crashing in one, put it in another and it works fine for a bit then starts crashing again. I have never seen it make it 3+ days however. A day or two then the problem came back.

I would get a small powered USB drive and call it a day.
I know WD has discontinued the pidrive but you should be able to get one on the secondary market, or something similar.

-Dave