Reindexing a Full Node (ubuntu linux) "ERROR: ConnectTip(): ConnectBlock"

[miguelmorales85]

I am trying to reindex my node (linux).

checkin debug.log -f I can see this error

Code:

ERROR: ConnectTip(): ConnectBlock 00000000000000001c9ae7f614afce81a9418f2bc38fcd47035de048e1d69270 failed

over and over again, I have checked all over the internet to see somebody with same issue and it appears that the only solution (AS ALWAYS) is to REINDEX AGAIN.. i am trying to deploy this full node and ALWAYS appears a new issue that forces me to REINDEX the ... blockchain...

Can somebody help me?

 

[unholycactus]

I'm not sure I'm reading your post right.

Are you saying you're trying to reindex your blockchain, that it doesn't work and the solution suggested by most people is to reindex?

[miguelmorales85]

I'm not sure I'm reading your post right.

Are you saying you're trying to reindex your blockchain, that it doesn't work and the solution suggested by most people is to reindex?

most people just reindex their nodes when there is problem.
I had to reindex mine, every time I get past 80% height there is always a problema and I have to reindex again.

The last report I saw in the debug.log file was

Code:

ERROR: ConnectTip(): ConnectBlock .. 

Could you help me explaining why does this happen?

[unholycactus]

I'm not sure I'm reading your post right.

Are you saying you're trying to reindex your blockchain, that it doesn't work and the solution suggested by most people is to reindex?

most people just reindex their nodes when there is problem.
I had to reindex mine, every time I get past 80% height there is always a problema and I have to reindex again.

The last report I saw in the debug.log file was

Code:

ERROR: ConnectTip(): ConnectBlock .. 

Could you help me explaining why does this happen?

Maybe your database is corrupted? You can try deleting the top half of your blocks and let them re-download.
To be honest, every time I have a problem with my blockchain I usually wipe out everything and do a clean install.

[miguelmorales85]

I'm not sure I'm reading your post right.

Are you saying you're trying to reindex your blockchain, that it doesn't work and the solution suggested by most people is to reindex?

most people just reindex their nodes when there is problem.
I had to reindex mine, every time I get past 80% height there is always a problema and I have to reindex again.

The last report I saw in the debug.log file was

Code:

ERROR: ConnectTip(): ConnectBlock .. 

Could you help me explaining why does this happen?

Maybe your database is corrupted? You can try deleting the top half of your blocks and let them re-download.
To be honest, every time I have a problem with my blockchain I usually wipe out everything and do a clean install.

it's official.. cant reindex.. it seems it always get to 80ish % before this

Code:

2014-06-10 19:07:55 progress=0.289942  cache=61.1MiB(28202tx)
2015-09-11 22:42:30 Corruption: block checksum mismatch
2015-09-11 22:42:30 *** System error while flushing: Database corrupted
2015-09-11 22:42:30 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-11 22:42:30 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-09-11 22:42:30 Loaded 750 blocks from external file in 1726311ms
2015-09-11 22:42:30 Reindexing block file blk00148.dat...
2015-09-11 22:42:30 scheduler thread interrupt
2015-09-11 22:42:30 addcon thread interrupt
2015-09-11 22:42:30 opencon thread interrupt
2015-09-11 22:42:30 msghand thread interrupt
2015-09-11 22:42:30 net thread interrupt
2015-09-11 22:42:30 Shutdown: In progress...
2015-09-11 22:42:30 StopNode()
2015-09-11 22:42:30 Corruption: block checksum mismatch
2015-09-11 22:42:30 *** System error while flushing: Database corrupted
2015-09-11 22:42:30 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-11 22:42:30 Shutdown: done

[unholycactus]

Try deleting blk00148.dat, it clearly says it's corrupted in the log. There might be more corrupted blocks like this but you should be able to do the same.

[miguelmorales85]

Try deleting blk00148.dat, it clearly says it's corrupted in the log. There might be more corrupted blocks like this but you should be able to do the same.

I think I might come with a solution, I see theres files in the blocks folder

rev00151.dat
rev00151.dat
and then a jump to rev00320.dat

the same with

blk00151.dat
blk00151.dat
and then a jump to blk00320.dat

I just erase rev and blk files from rev00147 and blk00147 onwards...

reindexing again... 

[miguelmorales85]

Ok, it failed AGAIN..

this is last night problem with reindexing the node.. another 5 days lost..

2015-09-16 05:15:05 LoadExternalBlockFile: Processing out of order child 0000000000000000087d41de97e50f2c63151b0caee4a63f1387dd0eb8ae2153 of 000000000000000016629ca1dab82f1c78c2a401d51021a890310b867190ac7d
2015-09-16 05:15:09 UpdateTip: new best=0000000000000000087d41de97e50f2c63151b0caee4a63f1387dd0eb8ae2153  height=345409  log2_work=82.326044  tx=60898518  date=2015-02-27 14:50:06 progress=0.724568  cache=64.9MiB(14575tx)
2015-09-16 05:15:09 Corruption: block checksum mismatch
2015-09-16 05:15:09 *** System error while flushing: Database corrupted
2015-09-16 05:15:10 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 05:15:10 ERROR: ProcessNewBlock: ActivateBestChain failed
2015-09-16 05:15:10 addcon thread interrupt
2015-09-16 05:15:10 opencon thread interrupt
2015-09-16 05:15:10 scheduler thread interrupt
2015-09-16 05:15:10 net thread interrupt
2015-09-16 05:15:10 msghand thread interrupt
2015-09-16 05:15:10 Shutdown: In progress...
2015-09-16 05:15:10 RPCAcceptHandler: Error: Operation canceled
2015-09-16 05:15:10 StopNode()
2015-09-16 05:15:11 Corruption: block checksum mismatch
2015-09-16 05:15:11 *** System error while flushing: Database corrupted
2015-09-16 05:15:11 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 05:15:11 Shutdown: done
2015-09-16 10:14:11

when I try bitcoind again..

2015-09-16 10:14:11 Bitcoin version v0.11.0.0-gd26f951 (Fri, 10 Jul 2015 19:23:55 +0200)
2015-09-16 10:14:11 Using OpenSSL version OpenSSL 1.0.1f 6 Jan 2014
2015-09-16 10:14:11 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2015-09-16 10:14:11 Default data directory /home/mike/.bitcoin
2015-09-16 10:14:11 Using data directory /home/mike/.bitcoin
2015-09-16 10:14:11 Using config file /home/mike/.bitcoin/bitcoin.conf
2015-09-16 10:14:11 Using at most 125 connections (1024 file descriptors available)
2015-09-16 10:14:11 Using 0 threads for script verification
2015-09-16 10:14:11 scheduler thread start
2015-09-16 10:14:11 Binding RPC on address ::1 port 8332 (IPv4+IPv6 bind any: 0)
2015-09-16 10:14:11 Binding RPC on address 127.0.0.1 port 8332 (IPv4+IPv6 bind any: 0)
2015-09-16 10:14:11 Bound to [::]:8333
2015-09-16 10:14:11 Bound to 0.0.0.0:8333
2015-09-16 10:14:11 Cache configuration:
2015-09-16 10:14:11 * Using 2.0MiB for block index database
2015-09-16 10:14:11 * Using 32.5MiB for chain state database
2015-09-16 10:14:11 * Using 65.5MiB for in-memory UTXO set
2015-09-16 10:14:11 init message: Loading block index...
2015-09-16 10:14:11 Opening LevelDB in /home/mike/.bitcoin/blocks/index
2015-09-16 10:14:11 Opened LevelDB successfully
2015-09-16 10:14:11 Opening LevelDB in /home/mike/.bitcoin/chainstate
2015-09-16 10:14:16 Opened LevelDB successfully
2015-09-16 10:14:21 LoadBlockIndexDB: last block file = 236
2015-09-16 10:14:21 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=253, size=117172286, heights=344944...345409, time=2015-02-24...2015-02-27)
2015-09-16 10:14:21 Checking all blk files are present...
2015-09-16 10:14:22 LoadBlockIndexDB: transaction index disabled
2015-09-16 10:14:22 LoadBlockIndexDB: hashBestChain=000000000000000013b16f33611d5f3c4bf85974ead0b81f1320df72a2cfe467 height=345398 date=2015-02-27 13:42:51 progress=0.724279
2015-09-16 10:14:22 init message: Verifying blocks...
2015-09-16 10:14:22 Verifying last 288 blocks at level 3
2015-09-16 10:14:27 No coin database inconsistencies in last 7 blocks (3640 transactions)
2015-09-16 10:14:27  block index           16603ms
2015-09-16 10:14:27 Wallet disabled!
2015-09-16 10:14:27 init message: Activating best chain...
2015-09-16 10:14:29 UpdateTip: new best=00000000000000000dd9e1229fba800321d7577a620dc301928bd3eef6ac3b3f  height=345399  log2_work=82.325567  tx=60893089  date=2015-02-27 13:46:34 progress=0.724283  cache=3.4MiB(1138tx)
2015-09-16 10:14:32 UpdateTip: new best=0000000000000000066e33cbe32b2ee2b52500d66fcbe609f813b659c978c790  height=345400  log2_work=82.325614  tx=60893573  date=2015-02-27 13:54:16 progress=0.724292  cache=7.2MiB(2634tx)
2015-09-16 10:14:34 UpdateTip: new best=00000000000000000924d9eefcd031876d0a721b62f783e636767cba4cc2fca6  height=345401  log2_work=82.325662  tx=60893780  date=2015-02-27 13:55:11 progress=0.724294  cache=15.0MiB(3469tx)
2015-09-16 10:14:35 UpdateTip: new best=0000000000000000001d7d40758a315577a60c335ef1ada2a514ca36c46e456d  height=345402  log2_work=82.32571  tx=60893947  date=2015-02-27 13:57:09 progress=0.724296  cache=16.5MiB(4064tx)
2015-09-16 10:14:35 UpdateTip: new best=00000000000000000090811b2f21d44df503e67997f5afe017f1d23a283b63c3  height=345403  log2_work=82.325758  tx=60894032  date=2015-02-27 13:57:51 progress=0.724297  cache=16.6MiB(4347tx)
2015-09-16 10:14:38 UpdateTip: new best=00000000000000000ab0ed41c5ff279532b73fab78a3f80927eb651018392f5e  height=345404  log2_work=82.325805  tx=60894645  date=2015-02-27 14:05:23 progress=0.724306  cache=23.3MiB(5951tx)
2015-09-16 10:14:39 UpdateTip: new best=000000000000000016afece55f1ec5b2090d695ac8aa65ba8271457a45372470  height=345405  log2_work=82.325853  tx=60894816  date=2015-02-27 14:07:18 progress=0.724309  cache=23.8MiB(6377tx)
2015-09-16 10:14:46 UpdateTip: new best=00000000000000000329b9f3a9d2916b280fe69fe0eab51eca5817ddce10ca17  height=345406  log2_work=82.325901  tx=60896344  date=2015-02-27 14:29:31 progress=0.724333  cache=44.4MiB(9939tx)
2015-09-16 10:14:53 UpdateTip: new best=0000000000000000022dad674ea6b69de08b8dd194a81431eb5b3417922e054a  height=345407  log2_work=82.325948  tx=60897556  date=2015-02-27 14:41:40 progress=0.724349  cache=55.9MiB(12556tx)
2015-09-16 10:14:54 UpdateTip: new best=000000000000000016629ca1dab82f1c78c2a401d51021a890310b867190ac7d  height=345408  log2_work=82.325996  tx=60897812  date=2015-02-27 14:46:26 progress=0.724354  cache=56.1MiB(13001tx)
2015-09-16 10:14:58 UpdateTip: new best=0000000000000000087d41de97e50f2c63151b0caee4a63f1387dd0eb8ae2153  height=345409  log2_work=82.326044  tx=60898518  date=2015-02-27 14:50:06 progress=0.724361  cache=63.0MiB(14575tx)
2015-09-16 10:14:58 Corruption: block checksum mismatch
2015-09-16 10:14:58 *** System error while flushing: Database corrupted
2015-09-16 10:14:58 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 10:14:58 Error: Failed to connect best block
2015-09-16 10:14:58 Reindexing block file blk00000.dat...
2015-09-16 10:14:58 Shutdown: In progress...
2015-09-16 10:14:58 scheduler thread interrupt
2015-09-16 10:14:58 StopNode()
2015-09-16 10:14:59 Corruption: block checksum mismatch
2015-09-16 10:14:59 *** System error while flushing: Database corrupted
2015-09-16 10:14:59 Error: Error: A fatal internal error occured, see debug.log for details
2015-09-16 10:14:59 Shutdown: done


PLease can somebody help me? Dont tell me I have to reindex again... 

[achow101]

Go to the data directory and delete everything except the wallet folks. This will cause a full resync of everything

[Newar]

Hardware specs?

[miguelmorales85]

Go to the data directory and delete everything except the wallet folks. This will cause a full resync of everything

Does the best solution always be DELETE EVERYTHING and start from scratch?
I cant believe its so disappointing..

Hardware specs?

250 GB HD SATA 3.
2 Gh Intel Pentium 4.
4 GB RAM.

[unholycactus]

Go to the data directory and delete everything except the wallet folks. This will cause a full resync of everything

Does the best solution always be DELETE EVERYTHING and start from scratch?
I cant believe its so disappointing..

Hardware specs?

250 GB HD SATA 3.
2 Gh Intel Pentium 4.
4 GB RAM.

It's nearly impossible to pinpoint what data is corrupted in a stack of 50GB, so yes, it's the best solution. Unless you have limited bandwidth, it shouldn't be much of a problem to download the chain again.

If you have space and bandwidth limitations, it's not a good idea to run a node anyways.

[miguelmorales85]

Go to the data directory and delete everything except the wallet folks. This will cause a full resync of everything

Does the best solution always be DELETE EVERYTHING and start from scratch?
I cant believe its so disappointing..

Hardware specs?

250 GB HD SATA 3.
2 Gh Intel Pentium 4.
4 GB RAM.

It's nearly impossible to pinpoint what data is corrupted in a stack of 50GB, so yes, it's the best solution. Unless you have limited bandwidth, it shouldn't be much of a problem to download the chain again.

If you have space and bandwidth limitations, it's not a good idea to run a node anyways.

not a bandwidth limitation my friend... it's all about time.. download 55 GB will take like a week...

I will keep this thread documented. See you in a week...

thanks for all

[miguelmorales85]

over again... after all these days reindexing..

I got closer...

Code:

2015-09-18 06:28:25 UpdateTip: new best=000000000000000005e3cef9e0cf1b278d6c6e0d2520acc2c3e9a4d841f9202f  height=372039  log2_work=83.274521  tx=81585840  date=2015-08-29 08:49:22 progress=0.977698  cache=46.7MiB(10884tx)
2015-09-18 06:28:25 LevelDB read failure: Corruption: block checksum mismatch
2015-09-18 06:28:25 Corruption: block checksum mismatch
2015-09-18 06:28:26 Error: Error reading from database, shutting down.
2015-09-18 06:28:26 Error reading from database: Database corrupted

[achow101]

Do you have a hardware error? Run some hardware diagnostics. Bitcoin core usually is a good indicator of failing hardware.

[miguelmorales85]

Do you have a hardware error? Run some hardware diagnostics. Bitcoin core usually is a good indicator of failing hardware.

I ran fschk.ext4 and it shown no errors.

[Newar]

Do you have a hardware error? Run some hardware diagnostics. Bitcoin core usually is a good indicator of failing hardware.

I ran fschk.ext4 and it shown no errors.

Just for giggles, can you try using an external HDD for the datadir?

[miguelmorales85]

Do you have a hardware error? Run some hardware diagnostics. Bitcoin core usually is a good indicator of failing hardware.

I ran fschk.ext4 and it shown no errors.

Just for giggles, can you try using an external HDD for the datadir?

Im reindexing AGAIN... see you in 3 or 4 days...