Sync stuck Bitcoin core

[hoptohodl]

When I rebooted a couple weeks ago I saw that the sync got stuck at block 860061.  I shut down and restarted, but not help.

Bitcoin Client Software and Version Number: Bitcoin Core version v26.0.0
Operating System:  MacOS Ventura Version 13.6.9
System Hardware Specs: 3 GHz 6 core Intel Core i5, 16 GB, 1.5 TB free space
Description of Problem: Bitcoin Core seems to be stuck while synching.
Any Related Addresses: None
Any Related Transaction IDs: None
Screenshot of the problem: https://imgur.com/a/ereO5SX
Log Files from the Bitcoin Client:  https://pastebin.com/DdzKWHqe

[BitMaxz]

I can't find any other errors except the one below with 1 failed.

Code:

2024-09-19T23:22:57Z New outbound-full-relay v1 peer connected: version: 70016, blocks=862024, peer=12
2024-09-19T23:22:57Z Imported mempool transactions from disk: 146 succeeded, 1 failed, 102669 expired, 0 already there, 0 waiting for initial broadcast
2024-09-19T23:22:57Z initload thread exit

Have you tried upgrading it to the latest version? Since you are using Bitcoin Core version v26.0.0 upgrading might fix this issue, the recent version is 27.1.

Or if you want to stay using the same version try to stop bitcoind first by using this command "bitcoind stop" and then reboot and try to run it again.

[nc50lc]

The logs don't tell much aside from a timeout when downloading the headers from a certain peer which coincides with your screenshot of stuck during "syncing headers".
Search your full debug.log file for blockchain-related errors

For now, start with (Open 'Window->Console'):

Code:

reconsiderblock 000000000000000000011b4932fb1ff50b1a96c3ced699f0b75318f3fb8b8d0c

result should be 'null' otherwise, an error.

If it didn't help,
Use this command and reply with the result, or your active chain's best "hash" and "height":

Code:

getchaintips

Bitcoin Client Software and Version Number: Bitcoin Core version v26.0.0

If you do not want to use v27.x, you must at least update to v26.2 / v26.1 that contains backported patches to some important updates.
Specially a fix against attacks related to mutated blocks.

[hoptohodl]

Upgraded to V27.0 (did not see version 27.1)

As suggested Ran:
reconsiderblock 000000000000000000011b4932fb1ff50b1a96c3ced699f0b75318f3fb8b8d0c

It started to move forward and then the Bitcoin Core froze.  I had to force a shutdown, but then after that, it progressed.  Had another error and froze several times. 
https://imgur.com/a/CzjgFVg

Finally caught up after several restarts.  Shut down Bitcoin Core and the restarted it.

I see that it still restarts back on Sept 4.  It syncs much faster, but this isn't right.

I cleared the log file, shut down and restarted, the whole log file is too big for past bin so here is much of the top..

https://pastebin.com/g8trLq8U

and then the end...
https://pastebin.com/5tGNZVUr

[nc50lc]

It started to move forward and then the Bitcoin Core froze.  I had to force a shutdown, but then after that, it progressed.

Bitcoin Core (Bitcoin-qt) normally freeze during the IBD in the latest blocks.
Its severity depends on your settings and hardware, but it should be running in the background and will catch-up the GUI once it become responsive again.

Had another error and froze several times.  
https://imgur.com/a/CzjgFVg

Code:

2024-09-21T20:10:24Z Fatal LevelDB error: Corruption: block checksum mismatch: /Users/michael/Library/Application Support/Bitcoin/chainstate/2953706.ldb
2024-09-21T20:10:24Z Error: A fatal internal error occurred, see debug.log for details
2024-09-21T20:10:24Z ForceFlushStateToDisk: failed to flush state (System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /Users/michael/Library/Application Support/Bitcoin/chainstate/2953706.ldb)

Probably caused by your force shutdowns since as described, Bitcoin Core is still running while the GUI is frozen.
Forcing shutdown while it's writing to disk can cause this kind of corruption issue.

I see that it still restarts back on Sept 4.  It syncs much faster, but this isn't right.
-snip-
I cleared the log file, shut down and restarted, the whole log file is too big for past bin -snip-

Usually, --reindex-chainstate is applied to fix corruption issue on the chainstate but the last parts of the log show that you've successfully synced your node:

Code:

2024-09-21T20:25:16Z Leaving InitialBlockDownload (latching to false)
-snip-
2024-09-21T20:25:23Z UpdateTip: new best=00000000000000000002baf6bece2d18b21e4255c0cbbbe7b7ebd48079824f05 height=862289 version=0x200ae000 log2_work=95.167307 tx=1083719859 date='2024-09-21T20:23:24Z' progress=0.999999 cache=1039.0MiB(8067229txo)

[hoptohodl]

I don't understand why it when shutdown properly it restarts the sync again on Sept 5.  This doesn't seem right.

[nc50lc]

I don't understand why it when shutdown properly it restarts the sync again on Sept 5.  This doesn't seem right.

That's weird indeed, in a "fortunate" way.
What should have happened is IBD would've restarted after the corruption incident since you have a pruned blockchain which cannot use --reindex-chainstate without redownloading the already pruned blocks.
But in your next restart, it's loaded successfully without errors:

Code:

2024-09-21T20:10:58Z Opening LevelDB in /Users/michael/Library/Application Support/Bitcoin/chainstate
2024-09-21T20:10:58Z Opened LevelDB successfully

For that specific shutdown, Bitcoin Core failed to flush to disk as the first part of the log indicates and perhaps the reason why it loaded successfully after restart.

[hoptohodl]

Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4


https://pastebin.com/2p2wEqJh
https://imgur.com/a/8zYOLWB

[LoyceV]

Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4

Can you add this option (from your debug.log) next time you start Bitcoin Core?

Code:

2024-09-25T23:09:36Z You can use -debug=leveldb to get more complete diagnostic messages

[nc50lc]

Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4

Hmm, same chainstate corruption error and same symptoms, same "2953706.ldb" file.
But for some weird reason, it fixed itself in your earlier reply after a restart.

usually, --reindex-chainstate fixes that kind of chainstate corruption error
But since your node's blockchain is pruned, your node has to restart IBD to do it. (re-sync from start)

BTW, as I've mentioned, that issue may have been caused by your previous force-closure and not related to the initial issue in the OP.

Once you decided to fix your chainstate and start from scratch, start Bitcoin-qt with --reindex-chainstate arg only once.
e.g.:

Code:

$ bitcoin-qt --reindex-chainstate

But I'd recommend you to remove your "prune block storage" setting first since you have enough disk space anyways.

Side question: is your datadir stored on an external HDD/SSD?
If so, check it's USB cable or USB port, most of the time, a bad cable/port can cause inconsistent data transfer.

[hoptohodl]

Can you add this option (from your debug.log) next time you start Bitcoin Core?

Code:

2024-09-25T23:09:36Z You can use -debug=leveldb to get more complete diagnostic messages

Ran with the -debug=leveldb

https://pastebin.com/U2JZatRi

Side question: is your datadir stored on an external HDD/SSD?
If so, check it's USB cable or USB port, most of the time, a bad cable/port can cause inconsistent data transfer.

Data is on local disk.

I have removed pruning and running --reindex-chainstate

[Cricktor]

This is odd in your debug.log:

Code:

...
2024-09-27T05:13:38Z Saw new header hash=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 height=863036
2024-09-27T05:13:38Z Saw new cmpctblock header hash=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 peer=59
2024-09-27T05:13:39Z [leveldb] WriteBatch memory usage: db=index, before=0.0MiB, after=0.0MiB
2024-09-27T05:13:39Z Fatal LevelDB error: Corruption: block checksum mismatch: /Users/michael/Library/Application Support/Bitcoin/chainstate/2953706.ldb
2024-09-27T05:13:39Z You can use -debug=leveldb to get more complete diagnostic messages
2024-09-27T05:13:39Z *** System error while flushing: Fatal LevelDB error: Corruption: block checksum mismatch: /Users/michael/Library/Application Support/Bitcoin/chainstate/2953706.ldb
2024-09-27T05:13:39Z Error: A fatal internal error occurred, see debug.log for details
2024-09-27T14:52:46Z ERROR: ConnectBlock: Consensus::CheckTxInputs: 8b6db3b4e43185484a33a6006fdc1549e6b164c9ccafe5f48335738403d3a94b, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
2024-09-27T14:52:46Z InvalidChainFound: invalid block=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5  height=863036  log2_work=95.176806  date=2024-09-27T05:13:12Z
2024-09-27T14:52:46Z InvalidChainFound:  current best=00000000000000000000c7b1a582ef0c5f1b5174df6d30d37d88cb07fce166bf  height=863035  log2_work=95.176793  date=2024-09-27T05:12:15Z
2024-09-27T14:52:46Z ERROR: ConnectTip: ConnectBlock 00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 failed, bad-txns-inputs-missingorspent, CheckTxInputs: inputs missing/spent
2024-09-27T14:52:46Z InvalidChainFound: invalid block=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5  height=863036  log2_work=95.176806  date=2024-09-27T05:13:12Z
2024-09-27T14:52:46Z InvalidChainFound:  current best=00000000000000000000c7b1a582ef0c5f1b5174df6d30d37d88cb07fce166bf  height=863035  log2_work=95.176793  date=2024-09-27T05:12:15Z
2024-09-27T14:52:46Z tor: Thread interrupt
2024-09-27T14:52:46Z torcontrol thread exit
2024-09-27T14:52:46Z Shutdown: In progress...
...

For block 863036 there's the "Saw new header..." and "Saw new cmpctblock..." message but then no "UpdateTip: new best=..." but rather some severe hiccups manifest. And a strange huge time gap between the logged entries

Code:

2024-09-27T05:13:39Z Error: A fatal internal error occurred, see debug.log for details
2024-09-27T14:52:46Z ERROR: ConnectBlock: Consensus::CheckTxInputs: 

Did your device sleep between 2024-09-27T05:13:39 UTC and 2024-09-27T14:52:46 UTC?

This is what my Raspibolt node had to say at the same time as yours when both have seen block 863036 being mined:

Code:

2024-09-27T05:13:38Z Saw new header hash=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 height=863036
2024-09-27T05:13:38Z Saw new cmpctblock header hash=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 peer=4755
2024-09-27T05:13:39Z UpdateTip: new best=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 height=863036 version=0x26f6c000 log2_work=95.176806 tx=1086622803 date='2024-09-27T05:13:12Z' progress=1.000000 cache=78.3MiB(688296txo)

[hoptohodl]

Did your device sleep between 2024-09-27T05:13:39 UTC and 2024-09-27T14:52:46 UTC?

I am pretty sure that gap is from when it had the error and when I came back to the computer and clicked on the error message.

[nc50lc]

Did your device sleep between 2024-09-27T05:13:39 UTC and 2024-09-27T14:52:46 UTC?

I am pretty sure that gap is from when it had the error and when I came back to the computer and clicked on the error message.

This checks out since your screenshot shows the same error in the log and pressing "ok" wont automatically close Bitcoin Core.

Side question: is your datadir stored on an external HDD/SSD?

Data is on local disk.

I have removed pruning and running --reindex-chainstate

I'd check for drive error first if weird issues that fixes itself happen to my node but since you already started IBD, let's see if you'll encounter the same issue.
Just make sure not to force close Bitcoin Core even if it seemingly non-responsive, you'll likely encounter that once it's nearing the tip.

[Cricktor]

Ah, OK, that makes sense, though I'm a bit surprised that your Bitcoin-Qt halts anything, likely to allow the user to see the error before it continues to shutdown. Running as daemon would've shutdown it without waiting when a data corruption error is detected.

So, the interesting part is why the corruption occured. It's usually a hardware problem. Could be a RAM issue, CPU overheating, storage device failure. Probably not easy to detect.

I'm no expert in what corruption your Bitcoin Core reported. Likely the chainstate is corrupted, block data I would assume, too. Maybe some other more experienced users here can chime in.

If I were you, I would stress test the RAM of your device, stress test the CPU and inspect S.M.A.R.T. data of the storage device. Additionally I would perform a thorough S.M.A.R.T. device extensive self-test to see if any errors are detected. I would also execute an extensive stress test of the storage device with other tools. (Make a full backup before those tests, just in case you trigger a failure. This will suck when your storage device is soldered as common on modern Apple gear and you're apparantly on MacOS.)

[hoptohodl]

Ran with the -debug=leveldb

locked up... log as follows

https://pastebin.com/U2JZatRi

Ran again and debug log gave the following...

https://pastebin.com/L83wghq4

Ran again and failed again at block 823222

https://pastebin.com/X7zWxPEb

[LoyceV]

Ran again and debug log gave the following...

https://pastebin.com/L83wghq4

This is the point where I'd replace the disk:

Code:

ERROR: ReadBlockFromDisk: Deserialize or I/O error

[nc50lc]

Ran again and failed again at block 823222

Check your disk's S.M.A.R.T report if it's failing.
How-to: https://support.apple.com/en/guide/mac-help/mchlp2548/13.0/mac/13.0
If there's no hardware problem, replace the sATA cable and use a different port if there's one available, then check if it can be fixed by "Disk Utility".

Or you can use third-party software to thoroughly check the report for bad values specially for "CRC Error Count" and "Bad Sectors".
The latter may be a sign of failing drive; the former may be caused by a bad cable or both.