Title: Sync stuck Bitcoin core Post by: hoptohodl on September 19, 2024, 11:45:02 PM When I rebooted a couple weeks ago I saw that the sync got stuck at block 860061. I shut down and restarted, but not help.
Bitcoin Client Software and Version Number: Bitcoin Core version v26.0.0 Operating System: MacOS Ventura Version 13.6.9 System Hardware Specs: 3 GHz 6 core Intel Core i5, 16 GB, 1.5 TB free space Description of Problem: Bitcoin Core seems to be stuck while synching. Any Related Addresses: None Any Related Transaction IDs: None Screenshot of the problem: https://imgur.com/a/ereO5SX Log Files from the Bitcoin Client: https://pastebin.com/DdzKWHqe Title: Re: Sync stuck Bitcoin core Post by: BitMaxz on September 19, 2024, 11:55:08 PM I can't find any other errors except the one below with 1 failed.
Code: 2024-09-19T23:22:57Z New outbound-full-relay v1 peer connected: version: 70016, blocks=862024, peer=12 Have you tried upgrading it to the latest version? Since you are using Bitcoin Core version v26.0.0 upgrading might fix this issue, the recent version is 27.1. Or if you want to stay using the same version try to stop bitcoind first by using this command "bitcoind stop" and then reboot and try to run it again. Title: Re: Sync stuck Bitcoin core Post by: nc50lc on September 20, 2024, 05:43:44 AM The logs don't tell much aside from a timeout when downloading the headers from a certain peer which coincides with your screenshot of stuck during "syncing headers".
Search your full debug.log file for blockchain-related errors For now, start with (Open 'Window->Console'): Code: reconsiderblock 000000000000000000011b4932fb1ff50b1a96c3ced699f0b75318f3fb8b8d0c If it didn't help, Use this command and reply with the result, or your active chain's best "hash" and "height": Code: getchaintips Bitcoin Client Software and Version Number: Bitcoin Core version v26.0.0 If you do not want to use v27.x, you must at least update to v26.2 / v26.1 that contains backported patches to some important updates.Specially a fix against attacks related to mutated blocks. Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 21, 2024, 08:33:00 PM Upgraded to V27.0 (did not see version 27.1)
As suggested Ran: reconsiderblock 000000000000000000011b4932fb1ff50b1a96c3ced699f0b75318f3fb8b8d0c It started to move forward and then the Bitcoin Core froze. I had to force a shutdown, but then after that, it progressed. Had another error and froze several times. https://imgur.com/a/CzjgFVg Finally caught up after several restarts. Shut down Bitcoin Core and the restarted it. I see that it still restarts back on Sept 4. It syncs much faster, but this isn't right. I cleared the log file, shut down and restarted, the whole log file is too big for past bin so here is much of the top.. https://pastebin.com/g8trLq8U and then the end... https://pastebin.com/5tGNZVUr Title: Re: Sync stuck Bitcoin core Post by: nc50lc on September 22, 2024, 03:58:21 AM It started to move forward and then the Bitcoin Core froze. I had to force a shutdown, but then after that, it progressed. Bitcoin Core (Bitcoin-qt) normally freeze during the IBD in the latest blocks.Its severity depends on your settings and hardware, but it should be running in the background and will catch-up the GUI once it become responsive again. Quote from: hoptohodl Had another error and froze several times. https://imgur.com/a/CzjgFVg Code: 2024-09-21T20:10:24Z Fatal LevelDB error: Corruption: block checksum mismatch: /Users/michael/Library/Application Support/Bitcoin/chainstate/2953706.ldb Forcing shutdown while it's writing to disk can cause this kind of corruption issue. I see that it still restarts back on Sept 4. It syncs much faster, but this isn't right. Usually, --reindex-chainstate is applied to fix corruption issue on the chainstate but the last parts of the log show that you've successfully synced your node:-snip- I cleared the log file, shut down and restarted, the whole log file is too big for past bin -snip- Code: 2024-09-21T20:25:16Z Leaving InitialBlockDownload (latching to false) Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 22, 2024, 08:11:33 PM I don't understand why it when shutdown properly it restarts the sync again on Sept 5. This doesn't seem right.
Title: Re: Sync stuck Bitcoin core Post by: nc50lc on September 23, 2024, 05:03:46 AM I don't understand why it when shutdown properly it restarts the sync again on Sept 5. This doesn't seem right. That's weird indeed, in a "fortunate" way.What should have happened is IBD would've restarted after the corruption incident since you have a pruned blockchain which cannot use --reindex-chainstate without redownloading the already pruned blocks. But in your next restart, it's loaded successfully without errors: Code: 2024-09-21T20:10:58Z Opening LevelDB in /Users/michael/Library/Application Support/Bitcoin/chainstate For that specific shutdown, Bitcoin Core failed to flush to disk as the first part of the log indicates and perhaps the reason why it loaded successfully after restart. Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 26, 2024, 01:49:46 PM Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4
https://pastebin.com/2p2wEqJh https://imgur.com/a/8zYOLWB Title: Re: Sync stuck Bitcoin core Post by: LoyceV on September 26, 2024, 02:09:03 PM Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4 Can you add this option (from your debug.log) next time you start Bitcoin Core?Code: 2024-09-25T23:09:36Z You can use -debug=leveldb to get more complete diagnostic messages Title: Re: Sync stuck Bitcoin core Post by: nc50lc on September 27, 2024, 04:37:24 AM Still crashing on me every couple hours and then always starting syncing blocks on disk on Sept 4 Hmm, same chainstate corruption error and same symptoms, same "2953706.ldb" file.But for some weird reason, it fixed itself in your earlier reply after a restart. usually, --reindex-chainstate fixes that kind of chainstate corruption error But since your node's blockchain is pruned, your node has to restart IBD to do it. (re-sync from start) BTW, as I've mentioned, that issue may have been caused by your previous force-closure and not related to the initial issue in the OP. Once you decided to fix your chainstate and start from scratch, start Bitcoin-qt with --reindex-chainstate arg only once. e.g.: Code: $ bitcoin-qt --reindex-chainstate Side question: is your datadir stored on an external HDD/SSD? If so, check it's USB cable or USB port, most of the time, a bad cable/port can cause inconsistent data transfer. Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 27, 2024, 03:09:10 PM Can you add this option (from your debug.log) next time you start Bitcoin Core? Code: 2024-09-25T23:09:36Z You can use -debug=leveldb to get more complete diagnostic messages Ran with the -debug=leveldb https://pastebin.com/U2JZatRi Side question: is your datadir stored on an external HDD/SSD? If so, check it's USB cable or USB port, most of the time, a bad cable/port can cause inconsistent data transfer. Data is on local disk. I have removed pruning and running --reindex-chainstate Title: Re: Sync stuck Bitcoin core Post by: Cricktor on September 27, 2024, 08:04:45 PM This is odd in your debug.log:
Code: ... For block 863036 there's the "Saw new header..." and "Saw new cmpctblock..." message but then no "UpdateTip: new best=..." but rather some severe hiccups manifest. And a strange huge time gap between the logged entries Code: 2024-09-27T05:13:39Z Error: A fatal internal error occurred, see debug.log for details This is what my Raspibolt node had to say at the same time as yours when both have seen block 863036 being mined: Code: 2024-09-27T05:13:38Z Saw new header hash=00000000000000000002b40b4a91ac617116d64a0f0eaa43fa96722572d1dec5 height=863036 Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 27, 2024, 09:35:18 PM Did your device sleep between 2024-09-27T05:13:39 UTC and 2024-09-27T14:52:46 UTC? I am pretty sure that gap is from when it had the error and when I came back to the computer and clicked on the error message. Title: Re: Sync stuck Bitcoin core Post by: nc50lc on September 28, 2024, 01:07:27 PM Did your device sleep between 2024-09-27T05:13:39 UTC and 2024-09-27T14:52:46 UTC? I am pretty sure that gap is from when it had the error and when I came back to the computer and clicked on the error message.Side question: is your datadir stored on an external HDD/SSD? Data is on local disk.I have removed pruning and running --reindex-chainstate Just make sure not to force close Bitcoin Core even if it seemingly non-responsive, you'll likely encounter that once it's nearing the tip. Title: Re: Sync stuck Bitcoin core Post by: Cricktor on September 28, 2024, 01:13:25 PM Ah, OK, that makes sense, though I'm a bit surprised that your Bitcoin-Qt halts anything, likely to allow the user to see the error before it continues to shutdown. Running as daemon would've shutdown it without waiting when a data corruption error is detected.
So, the interesting part is why the corruption occured. It's usually a hardware problem. Could be a RAM issue, CPU overheating, storage device failure. Probably not easy to detect. I'm no expert in what corruption your Bitcoin Core reported. Likely the chainstate is corrupted, block data I would assume, too. Maybe some other more experienced users here can chime in. If I were you, I would stress test the RAM of your device, stress test the CPU and inspect S.M.A.R.T. data of the storage device. Additionally I would perform a thorough S.M.A.R.T. device extensive self-test to see if any errors are detected. I would also execute an extensive stress test of the storage device with other tools. (Make a full backup before those tests, just in case you trigger a failure. This will suck when your storage device is soldered as common on modern Apple gear and you're apparantly on MacOS.) Title: Re: Sync stuck Bitcoin core Post by: hoptohodl on September 30, 2024, 08:36:45 PM Ran with the -debug=leveldb
locked up... log as follows https://pastebin.com/U2JZatRi Ran again and debug log gave the following... https://pastebin.com/L83wghq4 Ran again and failed again at block 823222 https://pastebin.com/X7zWxPEb Title: Re: Sync stuck Bitcoin core Post by: LoyceV on September 30, 2024, 08:56:27 PM Ran again and debug log gave the following... This is the point where I'd replace the disk:https://pastebin.com/L83wghq4 Code: ERROR: ReadBlockFromDisk: Deserialize or I/O error Title: Re: Sync stuck Bitcoin core Post by: nc50lc on October 01, 2024, 04:04:17 AM Ran again and failed again at block 823222 Check your disk's S.M.A.R.T report if it's failing.How-to: https://support.apple.com/en/guide/mac-help/mchlp2548/13.0/mac/13.0 (https://support.apple.com/en/guide/mac-help/mchlp2548/13.0/mac/13.0) If there's no hardware problem, replace the sATA cable and use a different port if there's one available, then check if it can be fixed by "Disk Utility". Or you can use third-party software to thoroughly check the report for bad values specially for "CRC Error Count" and "Bad Sectors". The latter may be a sign of failing drive; the former may be caused by a bad cable or both. |