Bitcore node.js db error?

[s2]

So I've been using bitcore to sync the blockchain on Ubuntu 16.  After running out of space on a 250Gb HD I've moved it to a 1Tb external HD.  It came up with syncing errors so decided to just restart from scratch with this external HD.

The other week it finally reached 100% however a couple days ago I accidentally pulled out the external HD USB lead
I next put the USB lead back in but on a different USB port to make sure I didn't accidentally pull it out again in future.
This seemed to change the mapping from

Code:

/media/myhd/mydir/[b]blockchain[/b]

to

Code:

/media/myhd/mydir/[b]blockchain1[/b]

Argh!  So I changed the bitcore.json file to reflect the new data directory but it gives the following result

Code:

warn: Verifying blocks...
warn: Verifying blocks...
warn: Verifying blocks...
...
warn: Verifying blocks...
error: Failed to start servicees
error: RPCError: Verifying blocks.
<callstack in bitcoind.js>
error: Failed to stop services: Error: bitcoind spawned process exited with status code 1.

I tried adding the reindex=1 into the bitcoin.conf file (Is this the right place??) to try to reindex it but this doesn't appear to be picked up.  Looking in bitcoin.js it appeared as though some help text should appear saying reindexing flag has been noticed.

Anyone know how I can prevent yet another weeks delay in needing to redownload the entire blockchain again??? 

There are no obvious corrupt files on the disk (unlike the disk full scenario which did exhibit this)

I love bitcoin but this setup/usability of bitcore is a nightmare if the db  (not even sure if this is bitcoin or bitcore's db) is really this fragile.  My current concern is when in production if we have a HD crash or power cut we need to somehow fix the db and can't afford to do a full resync each time.

 
I'm pretty sure this is just because I'm new to bitcore and ubuntu but I have no idea how to resolve this Verifying blocks error now so any advice is really appreciated!

[1714874017]

1714874017

[1714874017]

1714874017

[s2]

Ok, bit of an update.  Sure enough I made a mistake on the reindex bit and had put it in the wrong bitcoin.conf file (I had a duplicate due to the path changing).
Trying to reindex now to see if this fixes it....

[s2]

Anyone looked into why it's so easy for the DB to get corrupted and/or anything been tried to make it more robust?


(From reading other posts it appears I'm no the first to encounter lots of corrupt database issues.)

[cr1776]

If you are using bitcore, why not just dump bitcore and use Bitcoin Core?   Then see if you have the issues using only bitcoin core.

For what it is worth, I had bitcoin running for several years fall 2010-2014 on one machine and never had an issue.  And then for 2 years on a different host with similar results.  So I think it is one of those ymmv scenarios.  If it is in a production environment, UPSs, SSDs, and similar robust measures are pretty much required.

[s2]

Thanks, yes I think it's more getting it setup and stable is proving difficult.

I'd like to use bitcore because of the node and HD wallet features... just seems it could be perfect for this project apart from getting the thing running on the new machine.

The reindex feature looks like it's going to take days to rescan.

Any idea why it would be considered insecure for bitcoin to just sync a snapshot, test the hash of the DBs in the snapshot and just test the block headers and merkel tree to ensure it's legitimate (up to the snapshot date)?

[cr1776]

If you have a fast drive and fast CPU, it shouldn't take days, but if you do have a backup of the entire directory, you should be able to just swap it out so that you don't need the full rescan and might only need to catch up between the date of the last backup and the current date.

Thanks, yes I think it's more getting it setup and stable is proving difficult.

I'd like to use bitcore because of the node and HD wallet features... just seems it could be perfect for this project apart from getting the thing running on the new machine.

The reindex feature looks like it's going to take days to rescan.

Any idea why it would be considered insecure for bitcoin to just sync a snapshot, test the hash of the DBs in the snapshot and just test the block headers and merkel tree to ensure it's legitimate (up to the snapshot date)?

[s2]

Ok, thanks. That's reassuring.

Has anyone described what the various files are in the bitcoin directory?

Guessing all the blk files are the container files for the blockchain blocks ( but many blocks are in each file given how few files there are)

Is it right to assume the -reindex is merely building the blocks/index files from the blocks/blk0?.dat files?

What happens to orphaned blocks?   Are these also stored in these dat files?

Sorry for all the questions, last one... would it be sensible to assume -reindex could actually be done from a number much higher than 1 it's simply the last LDB file that was corrupted because the FS was disconnected?

[cr1776]

Hi,
This should help:  https://en.bitcoin.it/wiki/Data_directory

Orphan blocks are stored too if you are connected to the network when it occurs.

I am not sure about the -reindex from a higher number.  I suppose it could in theory, but to have complete consistency, it starts from 1.

Ok, thanks. That's reassuring.

Has anyone described what the various files are in the bitcoin directory?

Guessing all the blk files are the container files for the blockchain blocks ( but many blocks are in each file given how few files there are)

Is it right to assume the -reindex is merely building the blocks/index files from the blocks/blk0?.dat files?

What happens to orphaned blocks?   Are these also stored in these dat files?

Sorry for all the questions, last one... would it be sensible to assume -reindex could actually be done from a number much higher than 1 it's simply the last LDB file that was corrupted because the FS was disconnected?

[s2]

Ok, thanks again.
Yes, I bit the bullet and deleted it all and started the resync and will just take a copy when 100% is reached.