Title: bitcoin daemon database error in a block Post by: darkv0rt3x on June 02, 2020, 11:06:36 PM Hello.
I'm having some issues keeping my node running. I'm using Bitcoin Core version v0.20.99.0-e478b11db under Debain Buster. I have been being surprised multiple times with my node shut down. When I check the debug.log file I can't see much. Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this: Code: 2020-06-02T23:06:05Z Writing 0 unbroadcast transactions to disk. What can I do to try to fix this? Title: Re: bitcoin daemon database error in a block Post by: jackg on June 02, 2020, 11:24:19 PM It tells you, try starting it up with - debug=leveldb
But it's likely you'll have to redownload the chain. Title: Re: bitcoin daemon database error in a block Post by: nc50lc on June 03, 2020, 03:32:13 AM Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this: Seems like there an issue with file: /media/Storage/Blockchain/chainstate/060607.ldbCode: 2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb Is it still crashing? If it is: Shutdown Core normally then try to delete the "chainstate" folder in "/media/Storage/Blockchain/". Start Core and it will rebuild the database (not redownload the blockchain); if it still crash, check the latest log. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 03, 2020, 06:16:34 PM It tells you, try starting it up with - debug=leveldb But it's likely you'll have to redownload the chain. I hope I don't have to. I have been having this problem since last week. And last week I stopped bitcoin daemon normally, tried the '-reindex' option but then I was told to cancel the '-reindex' and just try to restart the daemon normally. So I did and the laptop was almost 24h checking all transactions from the very beginning. Not downloading the entire chain, just re-checking txo's. Then it finished and was working since then. So I didn't pay much more attention to it until 2 days ago when I noticed the node was not running. Now I started the daemon again and after it was running normally, I shut it down with 'bitcoin-cli stop' and I can see this: Seems like there an issue with file: /media/Storage/Blockchain/chainstate/060607.ldbCode: 2020-06-02T23:06:06Z Fatal LevelDB error: Corruption: block checksum mismatch: /media/Storage/Blockchain/chainstate/060607.ldb Is it still crashing? If it is: Shutdown Core normally then try to delete the "chainstate" folder in "/media/Storage/Blockchain/". Start Core and it will rebuild the database (not redownload the blockchain); if it still crash, check the latest log. It just crashed again. It was working well since yesterday night (when I noticed it was stopped) until 5 minutes ago. It has been working but I can see this block error every now and then, probably when the daemon dumps the cache onto the drive. Do you think I need to delete the entire 'chainstate' folder? Wouldn't that single file be enough to delete and the daemon re-download it again? OP, you might want to check and test your HDD/SSD health if the problem persists after following advice by other members. Use tools called smartctl to do so. Just in case you never use it, here's short command that i usually use * See general information Code: sudo smartctl -a /dev/<your_storage_path> * Perform short or long test Code: sudo smartctl --test=short /dev/<your_storage_path> Code: sudo smartctl --test=long /dev/<your_storage_path> Then use sudo smartctl -a /dev/<your_storage_path> again to see the test result It tells you, try starting it up with - debug=leveldb But it's likely you'll have to redownload the chain. Only if the blockchain files corrupted as well, from the log, looks like only single chainstate file is corrupted. Thanks for the cheat sheet of commands. Edited: Well, I just noticed the error is now on a different file... A file that doesn't even exist in my chainstate folder. It's '061114.ldb' file. I also noticed something I'm not sure is normal. I have a symlink inside 'chainstate' to... 'chainstate'... Code: -rw------- 1 narayan narayan 88K ago 15 2019 104069.ldb Title: Re: bitcoin daemon database error in a block Post by: HCP on June 03, 2020, 09:10:45 PM It would appear increasingly likely that there is a hardware issue of some description. :-\
If you're getting corrupted files occurring randomly, then the most likely culprits are either your HDD/SSD or possibly bad RAM... there is also a possibility that it's the drive/memory controller and/or chipset on the mobo that is failing. Diagnosing the exact cause of these faults can be tricky... you might also want to try using Memtest86+ (https://www.memtest.org/) and see if there are any issues with your RAM. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 03, 2020, 11:12:08 PM It would appear increasingly likely that there is a hardware issue of some description. :-\ If you're getting corrupted files occurring randomly, then the most likely culprits are either your HDD/SSD or possibly bad RAM... there is also a possibility that it's the drive/memory controller and/or chipset on the mobo that is failing. Diagnosing the exact cause of these faults can be tricky... you might also want to try using Memtest86+ (https://www.memtest.org/) and see if there are any issues with your RAM. Well, at this point I just facked up this shit. I accidentally deleted part of my .bitcoin folder. Now I had to re-write my bitcoin.conf file and I don't remember the exact settings I had before. Anyways, I wrote a new bitcoin.conf file and I started bitcoin with the switch '-reindex' and pray that this fix the missing data in my .bitcoin folder. I see millions of messages like this in my terminal right now: Code: 2020-06-03T22:50:20Z Reindexing block file blk00085.dat... and Code: 2020-06-03T23:09:22Z Reindexing block file blk00719.dat... I hope this can download the missing data and just fix this once and for all. Title: Re: bitcoin daemon database error in a block Post by: nc50lc on June 04, 2020, 02:46:20 AM It just crashed again. It was working well since yesterday night (when I noticed it was stopped) until 5 minutes ago. It has been working but I can see this block error every now and then, probably when the daemon dumps the cache onto the drive. It will ask you to rebuild the whole database even if you delete that single file; it might even produce errors.Do you think I need to delete the entire 'chainstate' folder? Wouldn't that single file be enough to delete and the daemon re-download it again? Rebuilding the database wont take long compared to syncing the whole blockchain. Anyways, based from the new replies, it looks like you have more corrupted files than the one in the chainstate folder. Your .blk files seems to have inconsistencies like missing transactions probably because of corrupted data. You might need to check for hardware issues. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 04, 2020, 06:14:06 AM Ok, the proccess is now complete.
Later today, after work, I'll run a few hardware tests, just for the sake of it. Bitcoind seems to be running ok now. I'll post feedback later. Thanks Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 04, 2020, 10:15:48 PM Hello again...
Well, my node looks running ok, but when I run: Code: bitcoin-cli getblockcount I get only about 402000 blocks. What do I need to do to get the missing blocks from the Blockchain? Title: Re: bitcoin daemon database error in a block Post by: HCP on June 04, 2020, 11:34:27 PM I get only about 402000 blocks. For whatever reason, it seems like your node is not syncing... is the number of blocks increasing at all?What output do you get if you run bitcoin-cli getnetworkinfo? Do you see "networkactive": true and "connections": XX where XX is > 0? ??? Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 04, 2020, 11:52:38 PM I get only about 402000 blocks. For whatever reason, it seems like your node is not syncing... is the number of blocks increasing at all?What output do you get if you run bitcoin-cli getnetworkinfo? Do you see "networkactive": true and "connections": XX where XX is > 0? ??? Well, I decided to restart bitcoind with -reindex-chainstate... About getnetworkinfo, i get: Code: { So, yes, network is active and with 11 connections. However I can also see that "NETWORK_LIMITED" and I'm not sure what it means in this specific context. I don't know what is usually there when the node is fully running. Title: Re: bitcoin daemon database error in a block Post by: HCP on June 04, 2020, 11:58:32 PM Well, it's connected and achieving network connections to peers, so it should be receiving new blocks...
Using the reindex option, you might need to give it some time to finish that process before it starts receiving/validating new blocks... is there anything obvious in the debug.log that might explain why it's not? ??? Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 05, 2020, 06:12:44 AM Well, it's connected and achieving network connections to peers, so it should be receiving new blocks... Using the reindex option, you might need to give it some time to finish that process before it starts receiving/validating new blocks... is there anything obvious in the debug.log that might explain why it's not? ??? Well, my debug log is recent because I forgot to set it up after I accidentally deleted part of my folder. But I set it up yesterday and again I made a mistake. I set it up like: Code: debug=net,httprpc,reindex,proxy,leveldb I missed a coma. Now, I'm not sure what is it logging. And probably, if I stop bitcoin daemon, fix the debug setup and start -reindex-chainstate,it will restart the process from the very beginning, no? Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 05, 2020, 06:10:27 PM This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks.
Do I have any other way, other than '-reindx-chainstate' to try to fix my database? My 'blocks' folder size says 298GB and my 'index' folder (inside 'blocks' folder), says only 94MB. So, I think I have all the blocks files but looks like I might be missing some data from folder 'index', no? Will this be the only way to fix the problem or maybe some other approach can do the trick once I have all the data in 'blocks' folder? Title: Re: bitcoin daemon database error in a block Post by: nc50lc on June 06, 2020, 01:41:53 AM This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks. I think it's quite normal.Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes. Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks. Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size. The index folder's size seems fine to me. A new debug.log entries will help a lot. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 06, 2020, 10:02:27 AM This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks. I think it's quite normal.Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes. Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks. Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size. The index folder's size seems fine to me. A new debug.log entries will help a lot. Yeah, I know about the first blocks being faster and last blocks being significantly slower. In the meantime I tried to stop bitcoind -reindex-chainstate and restarted bitcoind normally, but using dbcache=12000 (I have 16GB of RAM) and this last night it progressed quite well, considering the last 24h. It's processing around file blk01800.dat. So, I expect this finishes during today an I hope I have my full data again. I'll then run the smart tools on this drive to check it's sanity. Title: Re: bitcoin daemon database error in a block Post by: Jet Cash on June 06, 2020, 11:55:39 AM Have you backed up your wallet and associated files? You should do this before you do any experimenting or attempt recovery.
Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 06, 2020, 01:46:46 PM This is taking ages... It only added about 2000 blocks in like 10 hours. I'm only at ~425000. And I started this almost 24h ago. In the first 12h it did about 22000 block, and in the remaining hours, only about 2000 blocks. I think it's quite normal.Sync is usually faster at the first few blocks when there're few transactions in them: smaller blocks sizes. Then considerably slower at the last few blocks because of the verification that was skipped in the old blocks. Consider adding tweaks to your config file like dbcache="half_of_your_RAM+" if you're using the default size. The index folder's size seems fine to me. A new debug.log entries will help a lot. I fail how it's normal, unless OP's internet is quite slow (200 kb/s or slower) or there's major bottleneck (indicated by 100% CPU or HDD usage almost all the time). I'll then run the smart tools on this drive to check it's sanity. If the result is quite bad, you just shorten age of your storage device. No, my internet is ok. It's Optics Fiber and it's rated at 200/100 Mbits. An I don't even know if the internet is playing any role right now, because I just add an internet disconnection and the process just continued! As I have my folder with the total amount it is supposed to have, I think that whatever it is doing, it is only using data that I already have on my drive. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 06, 2020, 05:03:15 PM Well, this is in the end but I started getting this messages:
Code: 2020-06-06T16:58:03Z Pre-allocating up to position 0x500000 in rev02102.dat Edited: This is doing the same whole shit again. My chainstate folder is only about 1000 items. I stopped the daemon again an I got the following output: Code: 2020-06-06T17:13:04Z FlushStateToDisk: write coins cache to disk (72150244 coins, 10273727kB) started Title: Re: bitcoin daemon database error in a block Post by: HCP on June 06, 2020, 11:29:13 PM Those first messages are just showing that it was syncing new blocks... you can ignore the "unknown version" messages... while it can be alarming to see, they're 'normal'.
It's a bit tricky to diagnose when you're giving snippets of debug.log... so you got that "Error: Out of memory" AFTER you executed bitcoin-cli stop and the node was shutting down? Did you get the normal shutting down debug like this: Code: 2020-06-05T06:31:18Z tor: Thread interrupt Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 07, 2020, 01:20:12 AM Those first messages are just showing that it was syncing new blocks... you can ignore the "unknown version" messages... while it can be alarming to see, they're 'normal'. It's a bit tricky to diagnose when you're giving snippets of debug.log... so you got that "Error: Out of memory" AFTER you executed bitcoin-cli stop and the node was shutting down? Did you get the normal shutting down debug like this: Code: 2020-06-05T06:31:18Z tor: Thread interrupt Sorry to miss critical information. My debug.log file is set to be split every 300Mb, so it would be silly of me to paste here the entire file. That's why I was just pasting here the error messages plus a few lines before. The error was happening when I issued 'bitcoin-cli stop'. I was not getting the same output as you because the error happened. Otherwise I would get similar output as yours. So, I think it was due to the fact that I tried to use a bigger 'dbcache' value of 12000 MB. And when it tried to dump the cache to the disk, it was running out of memory. I 'only' have 16Gb of RAM. Not sure this was being an issue. Anyway, the process has finished, and I think I already have my full node data back. And now that I have my full data back (I hope), I tried to import an address where I have a small amount of BTC. I imported the address using the PK. Then I tried to rescan the blockchain to update the wallet with this address transactions. I used 'bitcoin-cli rescanblockchain 597000 633472' but I got this error: Code: $ bitcoin-cli rescanblockchain 590000 633465 So I tried smaller ranges. With smaller ranges it is working and I get no errors. I tried between 597000 and 600000 and no errors. I tried other ranges and no errors either. But I still can't make that 'listunspent' command shows the utxos in this address I just imported. What else can I do to make listunspent show my utxos? Title: Re: bitcoin daemon database error in a block Post by: HCP on June 07, 2020, 04:35:59 AM If you stop Bitcoin Core... and then restart bitcoind using:
Code: bitcoind -rescan It should do a full rescan of all transactions relating to addresses in your wallet... WITHOUT needing to redownload the blocks etc. Also, what output do you get if you use: Code: bitcoin-cli getaddressinfo "YOUR_BTC_ADDRESS" Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 07, 2020, 10:49:17 AM If you stop Bitcoin Core... and then restart bitcoind using: Code: bitcoind -rescan It should do a full rescan of all transactions relating to addresses in your wallet... WITHOUT needing to redownload the blocks etc. Also, what output do you get if you use: Code: bitcoin-cli getaddressinfo "YOUR_BTC_ADDRESS" Would that -rescan take how much time? And isn't the '-rescanblockchain' option the same as that but with the possibility of limiting the range of that scan? 'getaddressinfo' shows me: (I hide some of the chars. Not sure if they contain any sensitive info) Code: { Title: Re: bitcoin daemon database error in a block Post by: HCP on June 07, 2020, 11:19:50 AM -rescan usually takes a few minutes to complete... possibly as long as 30mins (depending on your computer etc) but it shouldn't take days or hours etc.
Theoretically, -rescanblockchain should do the job, but it's not working for you... whether that is caused by issues you've had with your block data and reindexing, I don't know. I suggested -rescan as it will simply start at the beginning and run through everything making sure that everything is checked. Combined with the fact that the "big" rescanblockchain timed out on you, I'm not 100% convinced that your node is running without issues :-\ Title: Re: bitcoin daemon database error in a block Post by: LoyceV on June 07, 2020, 11:35:22 AM -rescan usually takes a few minutes to complete... possibly as long as 30mins (depending on your computer etc) but it shouldn't take days or hours etc. I did a rescan recently, and it took my (old) laptop several hours (reading from HDD).Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 07, 2020, 01:39:05 PM -rescan usually takes a few minutes to complete... possibly as long as 30mins (depending on your computer etc) but it shouldn't take days or hours etc. Theoretically, -rescanblockchain should do the job, but it's not working for you... whether that is caused by issues you've had with your block data and reindexing, I don't know. I suggested -rescan as it will simply start at the beginning and run through everything making sure that everything is checked. Combined with the fact that the "big" rescanblockchain timed out on you, I'm not 100% convinced that your node is running without issues :-\ Rescan is now complete but still no output from listunspent command. Rescan took probably 2h. I have an i7-3740QM with 16Gb DDR3 of RAM. I've just been told that: Quote a timeout means it's still going. rescans take potentially hours. it holds the main thread so your peers will disconnect, the RPC call will fail. I don't know what else can I do. When I run Code: bitcoin-cli getwalletinfo Code: { Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 07, 2020, 05:47:34 PM Nevermind.
I think it's fixed. I'll use this thread if I need to keep asking for help. Title: Re: bitcoin daemon database error in a block Post by: HCP on June 07, 2020, 09:03:06 PM Nevermind. Did the balance just show up? or did you do something else after the -rescan? ??? I think it's fixed. It would be helpful to know for anyone else that stumbles across this thread when trying to diagnose a similar problem to yours ;) Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 08, 2020, 08:05:44 PM Nevermind. Did the balance just show up? or did you do something else after the -rescan? ??? I think it's fixed. It would be helpful to know for anyone else that stumbles across this thread when trying to diagnose a similar problem to yours ;) Sure I'm sorry. I was the problem (as usual). I imported an address that had 2 inputs and 2 outputs and I didn't thought about that. So, there were no unspent utxo's to show. Address was empty and I was expecting to see the transactions there. The 2 inputs and the 2 outputs. My bad... What can I say. :( Title: Re: bitcoin daemon database error in a block Post by: LoyceV on June 09, 2020, 04:45:27 PM I imported an address that had 2 inputs and 2 outputs and I didn't thought about that. So, there were no unspent utxo's to show. Address was empty and I was expecting to see the transactions there. The 2 inputs and the 2 outputs. My bad... What can I say. :( I recently found out that importing addresses into Bitcoin Core has some bugs: the balance doesn't always show up. I tested this with 145 old addresses, and only one very recent transaction showed up, while it didn't show the old balances.Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 09, 2020, 06:57:38 PM I imported an address that had 2 inputs and 2 outputs and I didn't thought about that. So, there were no unspent utxo's to show. Address was empty and I was expecting to see the transactions there. The 2 inputs and the 2 outputs. My bad... What can I say. :( I recently found out that importing addresses into Bitcoin Core has some bugs: the balance doesn't always show up. I tested this with 145 old addresses, and only one very recent transaction showed up, while it didn't show the old balances.Really? I don't want to insult you because I don't know your level of knowledge, but you did the rescan, right? Title: Re: bitcoin daemon database error in a block Post by: LoyceV on June 09, 2020, 07:48:25 PM Really? Really!Quote I don't want to insult you because I don't know your level of knowledge, but you did the rescan, right? Yes. Don't worry about insulting me, if only the problem would be that simple :P It's worse:From Bitcointalk Party - Discord server (https://discord.gg/ahgHyku) (05/25/2020): Quote from: LoyceV Quick question... I have Bitcoin Core, import 145 addresses read-only, type "rescanblockchain" in Console, wait 2 hours, and none of the old transactions show up. Only a transaction received today shows up. Am I missing something? Quote from: Csmiami how did you import them in first place? Quote from: achow101 there are soem ... issues with imports and rescans issues that are difficult to reproduce and nail down Quote from: LoyceV I imported them through Console: copy/paste all at once. The "issues" I guess I ran into :stuck_out_tongue: I've never had this problem before, but I'm not using the latest version of Bitcoin Core, so the "issues" might be fixed already in later versions. Quote from: achow101 might not be fixed. I've ran into import and rescan issues before in various versions of the master branch and haven't been able to figure out what was wrong Quote from: Csmiami it is not; I tried imported one when Loyce posted (to try some new stuff out) and after the whole rescan (debug log shows it's completed), the balance shows 0 I trust achow101 (https://bitcointalk.org/index.php?action=profile;u=290195)'s knowledge on this. So Bitcoin Core has "issues with imports and rescans" :( Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 10, 2020, 09:45:52 AM That seems to be a bit of an issue indeed.
Is this already know by devs? Anyone opened an issue on Github? (I'm sure you did, obviously) Title: Re: bitcoin daemon database error in a block Post by: LoyceV on June 10, 2020, 10:00:25 AM Is this already know by devs? Anyone opened an issue on Github? (I'm sure you did, obviously) If achow101 (https://bitcointalk.org/index.php?action=profile;u=290195) knows, I'm sure it's known at the right places. I didn't open an issue myself.I want to test what happens if I re-download the entire blockchain with the same watch-only wallet instead of just doing a rescan, but as always I don't have enough time for everything I want to do. Title: Re: bitcoin daemon database error in a block Post by: darkv0rt3x on June 12, 2020, 12:08:14 PM Is this already know by devs? Anyone opened an issue on Github? (I'm sure you did, obviously) If achow101 (https://bitcointalk.org/index.php?action=profile;u=290195) knows, I'm sure it's known at the right places. I didn't open an issue myself.I want to test what happens if I re-download the entire blockchain with the same watch-only wallet instead of just doing a rescan, but as always I don't have enough time for everything I want to do. I know the feeling, for sure. Title: Re: bitcoin daemon database error in a block Post by: Amirofski on August 12, 2020, 01:31:56 PM i srart new node and processing somthing like this
Code: 20-08-12T13:29:56Z UpdateTip: new best=000000000000000002c8bb221485c0f******5c8bb904fde06fbeebd0153 height=423773 version=0x20000000 log2_work=85.087637 tx=147146353 date='2016-08-05T09:49:25Z' progress=0.265073 cache=110.2MiB(430827txo) it's normal? Title: Re: bitcoin daemon database error in a block Post by: HCP on August 12, 2020, 02:04:55 PM it's normal? I don't see any obvious error or problem with that output...It is simply showing that it is updating with each new block that is received and parsed. Note how the "height" number is incrementing. Looks like it's a little over 26.5% synced... the "progress" value is the percentage (divided by 100)... so 0.265078 * 100 = 26.5078%. Title: Re: bitcoin daemon database error in a block Post by: Amirofski on August 12, 2020, 05:06:58 PM it's normal? I don't see any obvious error or problem with that output...It is simply showing that it is updating with each new block that is received and parsed. Note how the "height" number is incrementing. Looks like it's a little over 26.5% synced... the "progress" value is the percentage (divided by 100)... so 0.265078 * 100 = 26.5078%. thanks for your useful reply :-* |