Database corrupt on re-start [Was: Bitcoin-qt cannot read the database, closing]

[tfeagle]

@benjamin07  

Sorry to re-activate an ancient thread.  Can you tell me more info about the Xeon processor?  I'm running a Xeon Nehalem x5570 2.93 GHz.  Quad core with hyperthreading.  48 GBytes of mainboard memory.  Windows 10, 64-bit, version 1511.  I'm using the 32-bit implementation of QT, version bitcoin-0.15.1.  

Clean, first-time installation.  Was downloading the blockchain from scratch.  Had to temporarily shut down QT after three days.  Clean shut down.  This morning, tried to restart QT and got the corrupt database message.  

Currently running "chkdsk /f/r" on the drive.  Cannot access the debug.log file until this is done.  

For what it's worth, an obscure or unknown Xeon errata, not checked-for by the QT software, could explain a lot of my QT issues.  

[tfeagle]

OK.  This is a follow-up.

"chkdsk /f/r" on the QT database drive concluded with no errors found.  This is not proof-positive that the hard drive is OK, but it's a reasonable indication that the problem lies elsewhere.  The drive itself is a high-reliability HP spinning SATA drive for use in servers.  It's almost new.  Odds are good that it's OK.  

Below is the shutdown sequence, from the debug.log.  I had thought it was a clean shutdown.  Note the time stamp at 16:59:34.  This is the time when I tried to restart bitcoin-QT.  The QT app wrote a blank line with a time stamp, followed by about 20 linefeeds. 

Debug log excerpt follows:

------------- ------------- -------------
2018-01-04 09:10:10 UpdateTip: new best=000000000000000003ddd3acef3ffb6e0317bd5787867662cf5be78010451526 height=437675 version=0x20000000 log2_work=85.507452 tx=168627592 date='2016-11-06 23:37:32' progress=0.591994 cache=1247.9MiB(11040514txo)
2018-01-04 09:10:10 UpdateTip: new best=0000000000000000006a8c4ad505c5f706275d2d57c905bbca07a854b9d7b7bc height=437676 version=0x00000004 log2_work=85.507481 tx=168630170 date='2016-11-07 00:00:05' progress=0.592003 cache=1248.0MiB(11041537txo)
2018-01-04 09:10:10 UpdateTip: new best=0000000000000000036dea4c8b425afd7ff0be1b9cb8dbd4adc9cf57b38bcaa1 height=437677 version=0x20000000 log2_work=85.50751 tx=168630220 date='2016-11-07 00:00:50' progress=0.592003 cache=1248.0MiB(11041589txo)
2018-01-04 09:10:10 UpdateTip: new best=00000000000000000262817365e28f5c5e6631f3078d7c5be68f7133464f5dd6 height=437678 version=0x20000000 log2_work=85.507538 tx=168632547 date='2016-11-07 00:03:04' progress=0.592011 cache=1248.2MiB(11042761txo)
2018-01-04 09:10:10 UpdateTip: new best=0000000000000000031e4b68f1591a920f069b2cdc06f4b34a0210fe870105bb height=437679 version=0x20000000 log2_work=85.507567 tx=168633976 date='2016-11-07 00:03:20' progress=0.592016 cache=1248.8MiB(11048687txo)
2018-01-04 09:10:11 UpdateTip: new best=00000000000000000336fb38dc25f841246536540b3a007d67819caccb210d9b height=437680 version=0x20000000 log2_work=85.507596 tx=168634706 date='2016-11-07 00:03:58' progress=0.592019 cache=1249.1MiB(11052010txo)
2018-01-04 09:10:11 UpdateTip: new best=00000000000000000074136a39836a35fdb36a872339d3b036cb10b28b6876ed height=437681 version=0x20000000 log2_work=85.507625 tx=168636123 date='2016-11-07 00:11:23' progress=0.592024 cache=1249.5MiB(11055343txo)
2018-01-04 09:10:35 tor: Thread interrupt
2018-01-04 09:10:35 opencon thread exit
2018-01-04 09:10:35 torcontrol thread exit
2018-01-04 09:10:35 addcon thread exit
2018-01-04 09:10:35 net thread exit
2018-01-04 09:10:35 scheduler thread interrupt
2018-01-04 09:10:35 Shutdown: In progress...
2018-01-04 09:10:35 msghand thread exit
2018-01-04 09:10:35 Dumped mempool: 0s to copy, 0.046998s to dump
2018-01-04 09:14:19 Shutdown: done
2018-01-04 16:59:34

[achow101]

his is not proof-positive that the hard drive is OK, but it's a reasonable indication that the problem lies elsewhere.  The drive itself is a high-reliability HP spinning SATA drive for use in servers.  It's almost new.  Odds are good that it's OK.  

If you think it is hardware, check your RAM.

This is the time when I tried to restart bitcoin-QT.  The QT app wrote a blank line with a time stamp, followed by about 20 linefeeds. 

That's normal.

Debug log excerpt follows:

This is a clean shutdown.

Can you post what the log of when you started it up?

[tfeagle]

Hello.

Short RAM check reports no errors.  I usually run a long RAM check about once every six weeks or so.  Never have found any errors.  I am clocking the RAMs slower than vendor max spec.  Last post (before this one) had full section of debug log, from Bitcoin-QT restart until panic shutdown due to corrupt database. 

Feel free to PM if you need additional info.  Total debug log is about 10 MBytes.   

[tfeagle]

Hello.

I'm not seeing my post that has the debug log excerpt, showing a normal restart followed by BitCoin-QT panic shutdown and corrupt database error message.  Were you able to view that portion of the log? 

[achow101]

I'm not seeing my post that has the debug log excerpt, showing a normal restart followed by BitCoin-QT panic shutdown and corrupt database error message.  Were you able to view that portion of the log? 

That post was deleted because individual posts cannot be migrated to a different thread. Please repost it.

[tfeagle]

Re-Post:  Full section of debug log showing "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..." message from GUI.  5-6 lines from the top, following application re-start after clean shutdown.  (Some of my personal comments have been modified.  But log is intact.)  Shutdown due to corrupt database is at the very end of the post.

-------------
Below is the portion of the debug.log that was recorded immediately after the 16.59.34 restart.  First line (after the 20 linefeeds, mentioned above) captures the version of the BitCoin-QT app.  

5 lines into the text, the GUI reports that Bitcoin Core "...didn't yet exit safely...".  My initial knee-jerk reaction is:  "What?  Bitcoin-QT requires more than 7 hours to shutdown safely?"  My second reaction is:  Hmm.  Looks like the app left something open, or else some of the shutdown status flags were not saved correctly.  

Unfortunately, I know nothing about the internals of BitCoin-QT.  Anybody out there know enough to comment on the unsafe shutdown message in the log?  

New comment:  I will try to reproduce the unwanted behavior.  It may take a few days.  

-------------
2018-01-04 16:59:34 Bitcoin version v0.15.1
2018-01-04 16:59:34 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2018-01-04 16:59:34 Assuming ancestors of block 0000000000000000003b9ce759c2a087d52abc4266f8f4ebd6d768b89defa50a have valid signatures.
2018-01-04 16:59:34 Setting nMinimumChainWork=000000000000000000000000000000000000000000723d3581fe1bd55373540a
2018-01-04 16:59:34 Using the 'standard' SHA256 implementation
2018-01-04 16:59:35 GUI: "registerShutdownBlockReason: Successfully registered: Bitcoin Core didn't yet exit safely..."
2018-01-04 16:59:38 Default data directory C:\Users\michaelt\AppData\Roaming\Bitcoin
2018-01-04 16:59:38 Using data directory Q:\qt_bitcoin
2018-01-04 16:59:38 Using config file Q:\qt_bitcoin\bitcoin.conf
2018-01-04 16:59:38 Using at most 125 automatic connections (2048 file descriptors available)
2018-01-04 16:59:38 Using 16 MiB out of 32/2 requested for signature cache, able to store 524288 elements
2018-01-04 16:59:38 Using 16 MiB out of 32/2 requested for script execution cache, able to store 524288 elements
2018-01-04 16:59:38 Using 0 threads for script verification
2018-01-04 16:59:38 init message: Verifying wallet(s)...
2018-01-04 16:59:38 scheduler thread start
2018-01-04 16:59:38 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2018-01-04 16:59:38 Using wallet wallet.dat
2018-01-04 16:59:38 CDBEnv::Open: LogDir=Q:\qt_bitcoin\database ErrorFile=Q:\qt_bitcoin\db.log
2018-01-04 16:59:38 Cache configuration:
2018-01-04 16:59:38 * Using 2.0MiB for block index database
2018-01-04 16:59:38 * Using 8.0MiB for chain state database
2018-01-04 16:59:38 * Using 1014.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)
2018-01-04 16:59:38 init message: Loading block index...
2018-01-04 16:59:38 Opening LevelDB in Q:\qt_bitcoin\blocks\index
2018-01-04 16:59:39 Opened LevelDB successfully
2018-01-04 16:59:39 Using obfuscation key for Q:\qt_bitcoin\blocks\index: 0000000000000000
2018-01-04 16:59:48 LoadBlockIndexDB: last block file = 671
2018-01-04 16:59:48 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=127, size=95867971, heights=437521...437745, time=2016-11-05...2016-11-07)
2018-01-04 16:59:48 Checking all blk files are present...
2018-01-04 16:59:49 LoadBlockIndexDB: transaction index disabled
2018-01-04 16:59:49 Opening LevelDB in Q:\qt_bitcoin\chainstate
2018-01-04 16:59:50 Opened LevelDB successfully
2018-01-04 16:59:50 Using obfuscation key for Q:\qt_bitcoin\chainstate: 96d278477b272041
2018-01-04 16:59:50 Loaded best chain: hashBestChain=00000000000000000074136a39836a35fdb36a872339d3b036cb10b28b6876ed height=437681 date=2016-11-07 00:11:23 progress=0.591842
2018-01-04 16:59:50 init message: Rewinding blocks...
2018-01-04 16:59:53 init message: Verifying blocks...
2018-01-04 16:59:53 Verifying last 6 blocks at level 3
2018-01-04 16:59:53 [0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
2018-01-04 17:00:10 No coin database inconsistencies in last 7 blocks (10843 transactions)
2018-01-04 17:00:10  block index           31304ms
2018-01-04 17:00:10 init message: Loading wallet...
2018-01-04 17:00:10 nFileVersion = 150100
2018-01-04 17:00:10 Keys: 2002 plaintext, 0 encrypted, 2002 w/ metadata, 2002 total
2018-01-04 17:00:10  wallet                  215ms
2018-01-04 17:00:10 setKeyPool.size() = 2000
2018-01-04 17:00:10 mapWallet.size() = 0
2018-01-04 17:00:10 mapAddressBook.size() = 1
2018-01-04 17:00:10 mapBlockIndex.size() = 502513
2018-01-04 17:00:10 nBestHeight = 437681
2018-01-04 17:00:10 Imported mempool transactions from disk: 0 successes, 0 failed, 0 expired
2018-01-04 17:00:10 torcontrol thread start
2018-01-04 17:00:10 Bound to [::]:8333
2018-01-04 17:00:10 Bound to 0.0.0.0:8333
2018-01-04 17:00:10 init message: Loading P2P addresses...
2018-01-04 17:00:10 Loaded 65418 addresses from peers.dat  550ms
2018-01-04 17:00:10 init message: Loading banlist...
2018-01-04 17:00:10 init message: Starting network threads...
2018-01-04 17:00:10 net thread start
2018-01-04 17:00:10 init message: Done loading
2018-01-04 17:00:10 dnsseed thread start
2018-01-04 17:00:10 addcon thread start
2018-01-04 17:00:10 opencon thread start
2018-01-04 17:00:10 msghand thread start
2018-01-04 17:00:10 GUI: Platform customization: "windows"
2018-01-04 17:00:10 GUI: PaymentServer::LoadRootCAs: Loaded  51  root certificates
2018-01-04 17:00:21 Loading addresses from DNS seeds (could take a while)
2018-01-04 17:00:22 139 addresses found from DNS seeds
2018-01-04 17:00:22 dnsseed thread exit
2018-01-04 17:00:33 receive version message: /Satoshi:0.15.1/: version 70015, blocks=502567, us=50.38.113.130:54521, peer=0
2018-01-04 17:00:40 receive version message: /Satoshi:0.15.1/: version 70015, blocks=502567, us=50.38.113.130:54527, peer=1
2018-01-04 17:00:42 Pre-allocating up to position 0x900000 in rev00671.dat
2018-01-04 17:00:42 UpdateTip: new best=000000000000000003736fb238a27d9465e0e5b9b7293bc43781ee121896b94b height=437682 version=0x20000000 log2_work=85.507653 tx=168638786 date='2016-11-07 00:48:46' progress=0.591851 cache=1.0MiB(9004txo)
2018-01-04 17:00:43 UpdateTip: new best=0000000000000000018aed817cad0210ae279e16846e3cadb05e37924848077c height=437683 version=0x20000000 log2_work=85.507682 tx=168641192 date='2016-11-07 00:53:47' progress=0.591860 cache=1.9MiB(16519txo)
2018-01-04 17:00:44 UpdateTip: new best=0000000000000000038ac7709c75b5a2416a88338e75e6026328cfc7cca64c37 height=437684 version=0x20000000 log2_work=85.507711 tx=168643681 date='2016-11-07 01:29:59' progress=0.591869 cache=2.5MiB(22339txo)
2018-01-04 17:00:45 UpdateTip: new best=0000000000000000018d29a7d972326f050aa159756cfc6789d781aeeb273e16 height=437685 version=0x20000000 log2_work=85.507739 tx=168645399 date='2016-11-07 01:29:49' progress=0.591875 cache=3.1MiB(28010txo)
2018-01-04 17:00:46 UpdateTip: new best=0000000000000000023a5dc70740498bbd31c0baf49384e174682290522908ab height=437686 version=0x20000000 log2_work=85.507768 tx=168647400 date='2016-11-07 01:31:55' progress=0.591882 cache=3.9MiB(34533txo)
2018-01-04 17:00:47 UpdateTip: new best=00000000000000000217d4d6b6e7aff5f3b0688b9c1e9417429d32dd056fc5e1 height=437687 version=0x20000000 log2_work=85.507797 tx=168649259 date='2016-11-07 01:42:12' progress=0.591888 cache=4.4MiB(39411txo)
2018-01-04 17:00:48 UpdateTip: new best=00000000000000000237de7fb25dff1a5017ab7ad6c0c99e835ad831aeee57a6 height=437688 version=0x20000000 log2_work=85.507825 tx=168651391 date='2016-11-07 01:54:40' progress=0.591896 cache=5.1MiB(45346txo)
2018-01-04 17:00:49 UpdateTip: new best=000000000000000000da4d7991f5ffae9926a00085fc79e79408ea8b1db61a49 height=437689 version=0x20000000 log2_work=85.507854 tx=168652562 date='2016-11-07 01:55:48' progress=0.591900 cache=6.2MiB(55908txo)
2018-01-04 17:00:50 UpdateTip: new best=00000000000000000354c67ed12386b00e39dfbb62df55110e4ccdbedbb751df height=437690 version=0x20000000 log2_work=85.507883 tx=168655027 date='2016-11-07 02:28:32' progress=0.591908 cache=6.7MiB(60380txo)
2018-01-04 17:00:51 Pre-allocating up to position 0xa00000 in rev00671.dat
2018-01-04 17:00:51 UpdateTip: new best=000000000000000000def58b719d8da750ed4754a5f71cc372684407c83d2584 height=437691 version=0x20000000 log2_work=85.507911 tx=168657161 date='2016-11-07 02:29:25' progress=0.591916 cache=7.6MiB(66316txo)
2018-01-04 17:00:52 UpdateTip: new best=00000000000000000009561c636c3d5d6a0474da3a5226b20c918bdbafa32e3e height=437692 version=0x20000000 log2_work=85.50794 tx=168658654 date='2016-11-07 02:37:21' progress=0.591921 cache=8.0MiB(70176txo)
2018-01-04 17:00:52 UpdateTip: new best=000000000000000000fafc2ec0ccf2df5ce6ba6aa7c68a1c1fe035ed778b3ee2 height=437693 version=0x20000000 log2_work=85.507969 tx=168660864 date='2016-11-07 02:53:10' progress=0.591929 cache=8.4MiB(74192txo)
2018-01-04 17:00:53 UpdateTip: new best=0000000000000000043a207bd780207bffe830265f9c31db1e6afc71b39c5213 height=437694 version=0x20000000 log2_work=85.507997 tx=168662633 date='2016-11-07 03:05:30' progress=0.591935 cache=9.4MiB(83099txo)
2018-01-04 17:00:53 UpdateTip: new best=00000000000000000172afd5249d2502b8fd422abd99003ccada74df488fd2e5 height=437695 version=0x00000004 log2_work=85.508026 tx=168663222 date='2016-11-07 03:09:17' progress=0.591937 cache=9.7MiB(86342txo)
2018-01-04 17:00:54 UpdateTip: new best=0000000000000000020005ce401726949ae5302c472a8a835d348989d0c999f0 height=437696 version=0x20000000 log2_work=85.508055 tx=168664304 date='2016-11-07 03:15:27' progress=0.591941 cache=10.2MiB(90970txo)
2018-01-04 17:00:54 UpdateTip: new best=000000000000000003e0af361d432ab783a599449f80600bc5be8c6870e64d69 height=437697 version=0x20000000 log2_work=85.508083 tx=168664774 date='2016-11-07 03:18:44' progress=0.591943 cache=10.4MiB(93029txo)
2018-01-04 17:00:54 UpdateTip: new best=0000000000000000043601a83cc2e01140490b7685d6d063231db3fe91604601 height=437698 version=0x20000000 log2_work=85.508112 tx=168665774 date='2016-11-07 03:24:23' progress=0.591946 cache=10.6MiB(94316txo)
2018-01-04 17:00:55 UpdateTip: new best=0000000000000000008159b25b63be642e3e9e0691248ef1541d65884598e4b7 height=437699 version=0x20000000 log2_work=85.508141 tx=168667942 date='2016-11-07 03:40:30' progress=0.591954 cache=11.0MiB(98411txo)
2018-01-04 17:00:56 UpdateTip: new best=00000000000000000051718250ac61e9cd97af1e3309e301574b365b1c62e8c1 height=437700 version=0x20000000 log2_work=85.508169 tx=168668659 date='2016-11-07 03:42:43' progress=0.591956 cache=11.6MiB(103900txo)
2018-01-04 17:00:57 Pre-allocating up to position 0xb00000 in rev00671.dat
2018-01-04 17:00:57 UpdateTip: new best=0000000000000000034dfc747e0914b3639096538632ac05b85e9088eae95a70 height=437701 version=0x20000000 log2_work=85.508198 tx=168670820 date='2016-11-07 03:58:09' progress=0.591964 cache=12.0MiB(107646txo)
2018-01-04 17:00:58 UpdateTip: new best=000000000000000003401c4a21a7f12a3977b6e74f489009d85cecd5340cf94a height=437702 version=0x20000000 log2_work=85.508227 tx=168672596 date='2016-11-07 04:11:48' progress=0.591970 cache=12.4MiB(111329txo)
2018-01-04 17:00:58 receive version message: /Satoshi:0.15.99/: version 70015, blocks=502567, us=50.38.113.130:54537, peer=2
2018-01-04 17:00:59 UpdateTip: new best=000000000000000001d2f8a19e200c6271a2a6f90105aab16a930202fa1b8e5a height=437703 version=0x20000000 log2_work=85.508255 tx=168673719 date='2016-11-07 04:17:08' progress=0.591974 cache=12.8MiB(115557txo)
2018-01-04 17:00:59 UpdateTip: new best=00000000000000000378c3d9e009669ebb45977747670d6e4bff2087505ed7eb height=437704 version=0x20000000 log2_work=85.508284 tx=168674980 date='2016-11-07 04:25:59' progress=0.591978 cache=13.1MiB(117994txo)
2018-01-04 17:01:00 UpdateTip: new best=0000000000000000043a742c320c9a93ed8ace62d5a7b50fba6259b497454918 height=437705 version=0x20000000 log2_work=85.508313 tx=168675841 date='2016-11-07 04:30:58' progress=0.591981 cache=13.2MiB(119504txo)
2018-01-04 17:01:00 UpdateTip: new best=000000000000000000dcf1e86060cb7e635ac136fa7d77ca31db4882a2ccbe8e height=437706 version=0x20000000 log2_work=85.508342 tx=168677171 date='2016-11-07 04:39:23' progress=0.591986 cache=13.5MiB(122120txo)
2018-01-04 17:01:00 UpdateTip: new best=000000000000000001ef0ba8a0d1a4f0309cfcc04a4d4f0e534df5594d5bba1b height=437707 version=0x20000000 log2_work=85.50837 tx=168677476 date='2016-11-07 04:40:58' progress=0.591987 cache=13.6MiB(122504txo)
2018-01-04 17:01:01 UpdateTip: new best=000000000000000000ea1b8ed8f5d066e27f473392dfad5b6143b5f6f1915ae4 height=437708 version=0x20000000 log2_work=85.508399 tx=168678754 date='2016-11-07 04:50:14' progress=0.591992 cache=13.8MiB(124801txo)
2018-01-04 17:01:01 UpdateTip: new best=0000000000000000026e830834b8a1f6439342af76e75f22d63a96ae71efa6a7 height=437709 version=0x20000000 log2_work=85.508428 tx=168680305 date='2016-11-07 04:58:56' progress=0.591997 cache=14.7MiB(128671txo)
2018-01-04 17:01:02 UpdateTip: new best=00000000000000000195221ac3a4682e02c0ae82c7780220dab7a6679151de54 height=437710 version=0x20000000 log2_work=85.508456 tx=168682825 date='2016-11-07 05:20:11' progress=0.592006 cache=15.3MiB(133749txo)
2018-01-04 17:01:03 UpdateTip: new best=0000000000000000034dde56267aa664e511ea82f5f11dd5d9d92f05430d3f21 height=437711 version=0x20000000 log2_work=85.508485 tx=168684612 date='2016-11-07 05:28:20' progress=0.592012 cache=15.7MiB(137861txo)
2018-01-04 17:01:03 Pre-allocating up to position 0xc00000 in rev00671.dat
2018-01-04 17:01:03 UpdateTip: new best=00000000000000000157e244f59c6d96ddae09d4042f6a2f0d934a94c78e519b height=437712 version=0x20000000 log2_work=85.508514 tx=168685225 date='2016-11-07 05:31:26' progress=0.592014 cache=15.8MiB(138670txo)
2018-01-04 17:01:04 receive version message: /Satoshi:0.15.1/: version 70015, blocks=502567, us=50.38.113.130:54542, peer=3
2018-01-04 17:01:05 receive version message: /Satoshi:0.13.1/: version 70014, blocks=502567, us=50.38.113.130:54545, peer=4
2018-01-04 17:01:06 receive version message: /Satoshi:0.15.1/: version 70015, blocks=502567, us=50.38.113.130:54546, peer=5
2018-01-04 17:01:12 UpdateTip: new best=0000000000000000006d2794b477ec8f31ca65312702c8d7204f998aa3f9c0ee height=437713 version=0x20000000 log2_work=85.508542 tx=168686072 date='2016-11-07 05:35:24' progress=0.592017 cache=16.2MiB(142786txo)
2018-01-04 17:01:12 UpdateTip: new best=00000000000000000400446fcdb4adf325c6c601cf68224df06307058c832270 height=437714 version=0x20000000 log2_work=85.508571 tx=168686891 date='2016-11-07 05:41:32' progress=0.592020 cache=16.4MiB(144244txo)
2018-01-04 17:01:12 receive version message: /Satoshi:0.13.1/: version 70014, blocks=502567, us=50.38.113.130:54549, peer=6
2018-01-04 17:01:13 receive version message: /Satoshi:0.15.1/: version 70015, blocks=502567, us=50.38.113.130:54550, peer=7
2018-01-04 17:01:19 Pre-allocating up to position 0x7000000 in blk00671.dat
2018-01-04 17:01:26 UpdateTip: new best=000000000000000002968cbbe6cd7172533e48a65e7d6a8ce39a6bfe9a7a50e0 height=437715 version=0x20000000 log2_work=85.5086 tx=168687796 date='2016-11-07 05:46:37' progress=0.592023 cache=16.6MiB(145944txo)
2018-01-04 17:01:26 UpdateTip: new best=00000000000000000423452eedf1458866674161ad8be52178b18bde0c0b7fd0 height=437716 version=0x20000000 log2_work=85.508628 tx=168688548 date='2016-11-07 05:51:30' progress=0.592026 cache=16.7MiB(147212txo)
2018-01-04 17:01:34 UpdateTip: new best=00000000000000000364a19d4e936a365606ebc8a4ed84a727ae13ddd8db4a7c height=437717 version=0x20000000 log2_work=85.508657 tx=168689608 date='2016-11-07 05:58:59' progress=0.592029 cache=16.9MiB(149033txo)
2018-01-04 17:01:34 UpdateTip: new best=000000000000000002dc25552c59000d8f374bec25ae76d0636e1abe55b2d1bf height=437718 version=0x30000000 log2_work=85.508686 tx=168689700 date='2016-11-07 05:59:31' progress=0.592030 cache=16.9MiB(149102txo)
2018-01-04 17:01:36 UpdateTip: new best=00000000000000000444290761e7512c86a1b009b4e42b62df54f9e2d9549ac9 height=437719 version=0x20000000 log2_work=85.508714 tx=168690365 date='2016-11-07 06:02:56' progress=0.592032 cache=17.5MiB(154684txo)
2018-01-04 17:01:36 UpdateTip: new best=0000000000000000032ffac28d07f3b10e090843aa78cc9d273d4c26350a6bec height=437720 version=0x20000000 log2_work=85.508743 tx=168690873 date='2016-11-07 06:05:58' progress=0.592034 cache=17.6MiB(155645txo)
2018-01-04 17:01:36 UpdateTip: new best=0000000000000000037c58a5ed94fe7fed52c81b165ed49fddfea4ebc6859004 height=437721 version=0x20000000 log2_work=85.508772 tx=168691507 date='2016-11-07 06:09:41' progress=0.592036 cache=17.7MiB(156819txo)
2018-01-04 17:01:38 UpdateTip: new best=0000000000000000007a04e9b741396c43cec1249dec9d216a4e54707b3ee0f0 height=437722 version=0x20000000 log2_work=85.5088 tx=168693285 date='2016-11-07 06:21:30' progress=0.592042 cache=18.3MiB(161789txo)
2018-01-04 17:01:54 LevelDB read failure: Corruption: block checksum mismatch
2018-01-04 17:01:54 Corruption: block checksum mismatch
2018-01-04 17:02:08 Error reading from database: Database corrupted
2018-01-04 17:03:00

[achow101]

Have you tried deleting the highest numbered blk*.dat and rev*.dat files and then reindexing (deleting those files will force a reindex anyways)? This will remove the corrupted block (and a few others) which will then be redownloaded after the reindex completes.

[tfeagle]

Hello.  I am running reindex command right now.  "Q:\qt_bitcoin\bitcoin-0.15.1\bin\bitcoin-qt.exe -dbcache=4096 -reindex" 

I did not delete anything.  I'm not familiar enough with BitCoin-QT files (yet) to know what is appropriate to delete.  Reindex has been running for a few hours.  Still more than four years behind.  :-(  At present rate, I should be able to report success/failure in 2-3 days. 

Also, at the same time, I am downloading a new original BitCoin blockchain from scratch, using a netbook.  Atom processor (Intel) with two cores.  Hyperthreaded.  This is using WiFi connection.  Netbook and WiFi combo is slooow.  With luck, new blockchain may be downloaded in 3-4 days.  Maybe longer if next door neighbor saturates WiFi. 

At the moment, this data corruption event is an interesting problem.  But if I did not already have coins on deposit in exchanges, then I would be dead in the water.  I think maybe for small users like me, with no understanding of application internals, BitCoin-QT application may not be sustainable. 

[achow101]

I did not delete anything.  I'm not familiar enough with BitCoin-QT files (yet) to know what is appropriate to delete.

If the reindex fails, you should delete the highest numbered blk*.dat and rev*.dat files in the blocks/ folder in the datadir.

I think maybe for small users like me, with no understanding of application internals, BitCoin-QT application may not be sustainable. 

This rarely happens and is usually not Bitcoin Core's fault. In many cases, it is just spurious/random corruption (cosmic rays, random bit flipping, etc.). Other times it is a hardware problem.

[tfeagle]

I did not delete anything.  I'm not familiar enough with BitCoin-QT files (yet) to know what is appropriate to delete.

If the reindex fails, you should delete the highest numbered blk*.dat and rev*.dat files in the blocks/ folder in the datadir.

Ah.  Thank you.  If simple "-reindex" fails, then I will delete the content that you describe.  Good to know. 

I think maybe for small users like me, with no understanding of application internals, BitCoin-QT application may not be sustainable. 

This rarely happens and is usually not Bitcoin Core's fault. In many cases, it is just spurious/random corruption (cosmic rays, random bit flipping, etc.). Other times it is a hardware problem.

I have no blame for the software or for the developers.  Crypto-coin applications are non-trivial.  That being said, non-recoverable errors appear to be increasing in frequency, out in the world.  (This is my gut feel.  I have no hard data.)  Blockchain is 150 GBytes now, approx.  Grows daily.  5 Megabit FIOS connection will download new blockchain in 2-3 days, if I am lucky.  Reindex also appears to take a long time.  I don't know exact duration, yet.  If reindex takes 1-2 days to run, but has a high chance of failure, then the most reliable recovery process is to always download a new blockchain.  Maybe dedicate a netbook or laptop...keep hot spare blockchain ready at all times.  Alternately, I can quit use of personal wallet, and rely upon exchanges.  This solution (no personal wallet) is not happy for me.  If my crypto-coins are always on deposit in an exchange or coin-bank, and I must use traditional processes to buy/sell, then I might as well simply stay with fiat.  Wallets exist that do not require a complete blockchain...but at some level these depend upon existence of a trustworthy blockchain at some third-party entity.  Seems as though exchanges are starting to fulfill this need.  Not much different from central bank for fiat, in the long run. 

[tfeagle]

I have no blame for the software or for the developers.  Crypto-coin applications are non-trivial.  That being said, non-recoverable errors appear to be increasing in frequency, out in the world.  (This is my gut feel.  I have no hard data.)  Blockchain is 150 GBytes now, approx.  Grows daily.  5 Megabit FIOS connection will download new blockchain in 2-3 days, if I am lucky.  Reindex also appears to take a long time.  I don't know exact duration, yet.  If reindex takes 1-2 days to run, but has a high chance of failure, then the most reliable recovery process is to always download a new blockchain.  Maybe dedicate a netbook or laptop...keep hot spare blockchain ready at all times.  Alternately, I can quit use of personal wallet, and rely upon exchanges.  This solution (no personal wallet) is not happy for me.  If my crypto-coins are always on deposit in an exchange or coin-bank, and I must use traditional processes to buy/sell, then I might as well simply stay with fiat.  Wallets exist that do not require a complete blockchain...but at some level these depend upon existence of a trustworthy blockchain at some third-party entity.  Seems as though exchanges are starting to fulfill this need.  Not much different from central bank for fiat, in the long run. 

OK.  I have reproduced the problem to my own satisfaction.  I believe it to be a "perfect storm" issue that stems from the simultaneous manifestation of several infrequent individual events.  Given that I've spent more than 40 hours of serious focused work upon this matter, I am faced with a dilemma.  Should I continue to pursue this matter alone and at my own expense?  Or should I ask for help?  Or possibly a reward?  At any rate...here/below is the end of the log file from the reproduced database corruption. 

2018-01-13 02:24:53 version handshake timeout from 116
2018-01-13 02:24:58 UpdateTip: new best=000000000000000000f5ef9ca0220abeac7b474f6470aa9397660531ef86c6a2 height=478537 version=0x20000002 log2_work=86.86067 tx=243257756 date='2017-08-01 08:59:33' progress=0.847051 cache=195.2MiB(1750287txo)
2018-01-13 02:25:01 UpdateTip: new best=0000000000000000000e62e0df477f899f485a6da366adef6467ceeb72214dd5 height=478538 version=0x20000002 log2_work=86.860708 tx=243258446 date='2017-08-01 09:07:25' progress=0.847053 cache=195.3MiB(1751171txo)
2018-01-13 02:25:02 version handshake timeout from 118
2018-01-13 02:25:08 UpdateTip: new best=0000000000000000010897d24cf560af85f46b8f82d1ebc5fc3c4d79f9375cef height=478539 version=0x20000002 log2_work=86.860746 tx=243260041 date='2017-08-01 09:25:55' progress=0.847059 cache=195.9MiB(1756839txo)
2018-01-13 02:25:11 Pre-allocating up to position 0xf00000 in rev00952.dat
2018-01-13 02:25:11 UpdateTip: new best=00000000000000000110048746f38cdfac533e1e2c3b8285c229cbb43eabdd02 height=478540 version=0x20000002 log2_work=86.860784 tx=243260307 date='2017-08-01 09:28:17' progress=0.847060 cache=196.4MiB(1761983txo)
2018-01-13 02:25:11 UpdateTip: new best=00000000000000000094abdb2dca7ed8c828f5c5f9c5840e9accfe1e6ef69fe0 height=478541 version=0x20000002 log2_work=86.860822 tx=243260370 date='2017-08-01 09:29:10' progress=0.847060 cache=196.4MiB(1762252txo)
2018-01-13 02:25:16 version handshake timeout from 119
2018-01-13 02:25:18 UpdateTip: new best=000000000000000000bdf7c7ba3aeef751e26182e391f9cb6e54bad6694e1b85 height=478542 version=0x20000012 log2_work=86.86086 tx=243262230 date='2017-08-01 10:08:40' progress=0.847066 cache=196.8MiB(1765749txo)
2018-01-13 02:25:26 UpdateTip: new best=0000000000000000004d7a5d2b057e4b44b4ff760f1216d3a4529ecb7fad0cf4 height=478543 version=0x20000002 log2_work=86.860898 tx=243263611 date='2017-08-01 10:16:16' progress=0.847071 cache=197.3MiB(1770574txo)
2018-01-13 02:25:31 UpdateTip: new best=000000000000000000007e8dd0ee030a2ecffab1580faab7bbbeccc70a38a608 height=478544 version=0x20000002 log2_work=86.860936 tx=243264418 date='2017-08-01 10:25:55' progress=0.847074 cache=198.6MiB(1782674txo)
2018-01-13 02:25:33 version handshake timeout from 120
2018-01-13 02:25:33 tor: Thread interrupt
2018-01-13 02:25:33 addcon thread exit
2018-01-13 02:25:33 torcontrol thread exit
2018-01-13 02:25:33 scheduler thread interrupt
2018-01-13 02:25:33 GUI: QItemSelectionModel: Selecting when no model has been set will result in a no-op.
2018-01-13 02:25:33 Shutdown: In progress...
2018-01-13 02:25:33 GUI: QItemSelectionModel: Selecting when no model has been set will result in a no-op.
2018-01-13 02:25:33 GUI: QItemSelectionModel: Selecting when no model has been set will result in a no-op.
2018-01-13 02:25:37 opencon thread exit
2018-01-13 02:25:39 UpdateTip: new best=000000000000000000c2d9fce2f37b5d68baa7550d6496346c2ed156f28e7196 height=478545 version=0x20000002 log2_work=86.860974 tx=243265249 date='2017-08-01 10:35:22' progress=0.847077 cache=198.9MiB(1785550txo)
2018-01-13 02:25:39 net thread exit
2018-01-13 02:25:40 msghand thread exit
2018-01-13 02:25:41 Dumped mempool: 0s to copy, 0.045002s to dump
2018-01-13 02:25:41 Corruption: block checksum mismatch
2018-01-13 02:25:41 *** System error while flushing: Database corrupted
2018-01-13 02:36:30 Corruption: block checksum mismatch
2018-01-13 02:36:30 *** System error while flushing: Database corrupted
2018-01-13 02:36:32 Shutdown: done

[CoinXpert]

Try to contact these guys: https://www.walletsos.org/

[tfeagle]

Try to contact these guys: https://www.walletsos.org/

Ah.  Many thanks for your suggestion.  But, I have no serious worries.  I have multiple copies of my wallet.  None of my coins are at risk.  Also, the bulk of my crypto coins are on deposit at exchanges. 

This thread has two themes.  (1) I worry that some instances of database corruption...happens ever more frequently with all databases that fork from original bitcoin...is/are related to processor architecture.  Specifically, Intel Xeon architecture.  I asked about this at the beginning of the thread...but nobody ever responded.  Many of my systems contain Xeon processors.  I worry that they may cause high risk of local database corruption.  (2) I believe that the current generation of full-node wallets (such as qt_bitcoin-0.15.1) become more fragile as number of blocks in the blockchain grows.  If true, then eventually full-node databases (containing complete blockchains) will only be supported by large corporations with ample money, backups, and software engineers.  Days of folks supporting a full node on the family desktop will be done.  Exchanges and financial industry will then be the only entities in possession of a complete blockchain.  Just like the current banking system...individuals will lose visibility and control of crypto coins.  Might as well stay with fiat. 

I was hoping that some developer would want to work on failure modes.  Look for root causes of bugs.  But nobody seems interested. 

[AbsentSixth]

I seem to be having a similar issue with mine as well, and also a lack of resources in order to find a fix. achow101 has been kind enough to try and help me in another thread as well so thanks to him there has been some feedback. Before I posted here I also tried the IRC #bitcoin channel but at the time there wasn't anyone who was able to offer any assistance.
Like you mentioned I too am concerned for the folk who run their own nodes and may not have a working knowledge of how to fix these types of problems on their own.
I hope you find a solution and will keep an eye on the thread in case you do.

[achow101]

I was hoping that some developer would want to work on failure modes.  Look for root causes of bugs.  But nobody seems interested. 

The root causes are usually out of our hands. In many cases it is hardware failure. Bitcoin Core does a lot of disk I/O and uses quite a bit of memory and processing power. So often times hardware issues (typically in memory issues or disk issues) are the root cause of such problems. I suggest that you run some hardware diagnostics on your machine and see if anything turns up.

[tfeagle]

The root causes are usually out of our hands. In many cases it is hardware failure. Bitcoin Core does a lot of disk I/O and uses quite a bit of memory and processing power. So often times hardware issues (typically in memory issues or disk issues) are the root cause of such problems. I suggest that you run some hardware diagnostics on your machine and see if anything turns up.

I'm relatively sure (let's call it 80% certainty) that some database corruption events stem from the way qt handles an interrupted incoming bitstream (ie. downloading blocks) when the interruption is not flagged by hardware or other lower level layer.  Boxes which connect different telecomm technologies to local ethernet network (DSL to ethernet, coax to ethernet, FIOS to ethernet, etc...sometimes called "broadband modems") are an ongoing headache.  For ethernet (OSI model) there are seven layers to ethernet...more or less.  This is "ivory tower" perspective.  In reality, vendors who make little boxes/modems will merge and/or divide existing layers whenever they wish.  In an ivory tower implementation, there exist diagnostic hardware/code hooks to detect issues with data stream and somehow mitigate possibility of bad data.  In a low-cost box with fewer or merged layers, some specific types of bad data are knowingly passed northwards.  (Passed upwards from phy layers to processing and application layers.)  Some types of bad data are too expensive to identify and flag at lower levels.  Brute force experiment (to detect absence of flags) is easy to design, annoying to execute.  Simply inject noise into DSL signal on outside of modem while downloading blockchain.  (Be careful...some ISPs may be angered by this.  Also...use protective circuit to prevent damage to modem or to ISP hardware.)  Last log I posted (above) from corrupt database was generated in this manner.  Took less than three hours, with only a small amount of noise injection.

If I am correct, then yes.  This would be clearly a network/hardware failure, stemming from design omissions.  Or maybe a management failure, stemming from corporate practice of giving as little value as possible to buyers of equipment.  If so, then application software is completely innocent.  Nevertheless, it is unlikely that all modem vendors will upgrade their products in order to prevent blockchain corruption.  Even if this were practical, any dysfunctional equipment which is already in service will continue to pass bad data, and annoy us, for decades to come.  Choices are: fix the problem(s) at the application level, or post process and repair the database, or require that users endure corruption events.  Also...as blockchains grow, it is likely that users' overhead expenses (download time, offline time, network loading during download, extra systems with hot spare blockchain, etc) will also grow.  Will corruption events increase in frequency as blockchains grow?  Difficult to predict.  

To developers: I would recommend creation of an offline stand-alone point tool (or possibly several tools) to examine idle database on user's hard drive, and provide block-by-block diagnosis.  My opinion: embedding diagnostic tools within an application package is generally a bad idea.  Also, some sort of block editor would be useful.  I should not need to download 150 GBytes of data, whenever I wish to delete and replace one specific block.  

Sorry for the rant.  Can you guess that I once designed tecomm hardware?  

[tfeagle]

I seem to be having a similar issue with mine as well, and also a lack of resources in order to find a fix. achow101 has been kind enough to try and help me in another thread as well so thanks to him there has been some feedback. Before I posted here I also tried the IRC #bitcoin channel but at the time there wasn't anyone who was able to offer any assistance.
Like you mentioned I too am concerned for the folk who run their own nodes and may not have a working knowledge of how to fix these types of problems on their own.
I hope you find a solution and will keep an eye on the thread in case you do.

Thanks for speaking out.  And welcome to the discussion. 

[LoyceV]

After reading the entire thread, I wonder if there is any specific reason to use 32-bit software on a 64-bit OS. I doubt it's related, but it seems to be an unnecessary limitation.

[tfeagle]

After reading the entire thread, I wonder if there is any specific reason to use 32-bit software on a 64-bit OS. I doubt it's related, but it seems to be an unnecessary limitation.

At this point, I run all qt based wallets from external USB drive(s).  Each crypto-coin gets a unique hard drive.  (Litecoin, Original Bitcoin, Bitcoin Cash, etc)  For backup, I mirror each drive into a single compressed bundle.  Use of 32-bit versions allows me to attach external drive to any system in the building.  CPU hardware doesn't really matter, version of Windows OS doesn't really matter.  Typical data corruption event only disables one type of coin.  Recovery of an entire wallet and database from bundle takes a couple of hours.  Process to create stand-alone qt wallet on external USB drive is a little tricky, but is well worth the effort (IMHO). 

That being said...

I'm doubting that open source wallets will be fixed any time soon.  Remaining bugs are subtle...typically stem from conceptual/architectural errors.  Original developers/coders cannot "see" the problems, because they are reviewing their own work.  Must reliably reproduce the failure in order to identify and isolate the malfunctioning code.  With involvement of asynchronous components...sometimes from sources external to code-running systems...fault reproduction may require very expensive test & measurement equipment, and/or hundreds of person-hours of tedious and frustrating bug hunting.  Most open source developers cannot expend this quantity of resources.  Thus last 2% of bugs inherently require "big corporation" resources if a timely solution is to be found. 

Do serious, show-stopping bugs, still exist in mature software?  Undoubtedly.  Really bad bugs only emerge during "perfect storm" situations.  Also known as "black swan" events.

Here is a classic conceptual problem (below).  Unrelated to this thread, but instructional/educational nonetheless.  Portions of "Project Web" on sourceforge is/are down.  Database migration is in progress.  Applications that check with SF for version update/release BEFORE opening control panel are all broken.  This is worldwide, most likely.  Apps cannot fetch version numbers, because SF is down.  End users cannot easily disable version/update check, because control panel will not open.  Tsk.  From my perspective, this bug should be fixed within application code...even though the application itself is not to blame. 

>
> Exception in thread "AWT-EventQueue-0" java.lang.NumberFormatException: For input string: "Project web is currently offline pending the final migration of its data to our new datacenter"
>         at java.lang.NumberFormatException.forInputString(Unknown Source)
>         at java.lang.Integer.parseInt(Unknown Source)
>         at java.lang.Integer.valueOf(Unknown Source)
>         at me.mabra.hellonzb.HelloNzbToolkit.getVersionParts(Unknown Source)
>         at me.mabra.hellonzb.HelloNzbToolkit.isLatestNewer(Unknown Source)
>         at me.mabra.hellonzb.HelloNzbToolkit.isUpdateAvailable(Unknown Source)
>         at me.mabra.hellonzb.HelloNzb.checkProgramUpdate(Unknown Source)
>         at me.mabra.hellonzb.HelloNzb.<init>(Unknown Source)
>         at me.mabra.hellonzb.HelloNzb$1.run(Unknown Source)
>         at java.awt.event.InvocationEvent.dispatch(Unknown Source)
>         at java.awt.EventQueue.dispatchEventImpl(Unknown Source)
>         at java.awt.EventQueue.access$500(Unknown Source)
>         at java.awt.EventQueue$3.run(Unknown Source)
>         at java.awt.EventQueue$3.run(Unknown Source)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(Unknown Source)
>         at java.awt.EventQueue.dispatchEvent(Unknown Source)
>         at java.awt.EventDispatchThread.pumpOneEventForFilters(Unknown Source)
>         at java.awt.EventDispatchThread.pumpEventsForFilter(Unknown Source)
>         at java.awt.EventDispatchThread.pumpEventsForHierarchy(Unknown Source)
>         at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
>         at java.awt.EventDispatchThread.pumpEvents(Unknown Source)
>         at java.awt.EventDispatchThread.run(Unknown Source)
>