Old blocks changed

[d_eddie]

My bitcoin node runs on a linux box. I've been away a few weeks, and I kept it switched off. When I got back, I found the main drive had some problem, so I replaced it and reinstalled the system and some of the software, including bitcoind. So far so good. After this, I proceeded to restore the latest daily backup of the .bitcoin directory. I then left the node running to verify and sync.

The surprise came when the catching up was done, and I launched the new daily backup. I use rsync, so only files that have actually changed do get transferred. Well, apparently about half of the blocks need uploading to backup, which was never the case during my previous daily backup routine. Why have older block files changed?

I'm talking about blk0XXXX.dat files. The latest is blk01790.dat, so I expected the backup to begin well after blk01000.dat, but rsync's output begins like this:

Code:

.bitcoin/blocks/blk00028.dat
.bitcoin/blocks/blk00031.dat
.bitcoin/blocks/blk00033.dat
.bitcoin/blocks/blk00036.dat
.bitcoin/blocks/blk00037.dat
.bitcoin/blocks/blk00038.dat
...

Why the change in all these old blocks?

[1713562928]

1713562928

[1713562928]

1713562928

[1713562928]

1713562928

[1713562928]

1713562928

[fillippone]

Just me asking.
Which version of Bitcoin Core are you running? Which were you running before reinstalling the software?
If you updated the version .dat format might have changed (maybe you didn’t upgrade core since a long time), hence the backup.

[d_eddie]

I'm still on 0.18.0 as I was before.

Besides, the format of raw blk*.dat files (basically, a copy of the blockchain) hasn't ever changed AFAIK.

[DireWolfM14]

Did the name of your hard drive change?  Any difference between the old hard drive's directory path and the new hard drive's directory path will trigger a discrepancy between the files for backup purposes.  

[fillippone]

I'm still on 0.18.0 as I was before.

Besides, the format of raw blk*.dat files (basically, a copy of the blockchain) hasn't ever changed AFAIK.

Ok,
Sorry for the poor reply.
I went back and checked what I was referring to.
It is something different it changed in version 0.17:

If your node has a txindex, the txindex db will be migrated the first time you run 0.17.0 or newer, which may take up to a few hours. Your node will not be functional until this migration completes.

So, yes, different thing.

At least I hope knowing which version you are running will help others with your issue.

[d_eddie]

Did the name of your hard drive change?  Any difference between the old hard drive's directory path and the new hard drive's directory path will trigger a discrepancy between the files for backup purposes.  

I don't think the name changed. It's linux.
But even if it changed, why should this affect only part of the files? The backup didn't begin with blk0000.dat, and several files have been skipped - half, more or less.

[Abdussamad]

manually compare the hashes of the old and new files. sha256sum them for instance. the reason being that timestamp changes can also trigger rsync. from the man page:

Rsync  finds  files  that  need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

if the hashes are the same then that means the file contents haven't changed and only the timestamp has been changed. then see if there's an rsync option that lets you ignore file timestamps and only compare file hashes.

[d_eddie]

manually compare the hashes of the old and new files. sha256sum them for instance. the reason being that timestamp changes can also trigger rsync. from the man page:

Rsync  finds  files  that  need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

if the hashes are the same then that means the file contents haven't changed and only the timestamp has been changed. then see if there's an rsync option that lets you ignore file timestamps and only compare file hashes.

Good call, but... unfortunately the old files have all been replaced by the new backup, so I can't check again. However, I did check the timestamps before launching the backup (I knew the timestamp could trigger a "changed" state for rsync) and all the blk*.dat files had the same date on the backup server and on the bitcoin node. This is consistent with my intuitive hypothesis that after being downloaded/assembled, they are only read: probably never written to again.

EDIT  The rsync dry runs were 100% consistent with what happened in the real run and among themselves. File corruption is the only possible explanation for this behavior I can come up with. Possibly the corruption occured while copying back the backup to the target machine, which had a busy disk and a busy network connection from downloading/installing several software packages. However, I just made another daily backup (and several dry runs just to check), and only the expected files were transferred. If anyone has a better hypothesis than selective file corruption, I'd like to hear it.

[LFC_Bitcoin]

I’d be interested to see achow101’s take on this.

[achow101]

Did you have to reindex after restoring the backup?

[d_eddie]

Did you have to reindex after restoring the backup?

I think so, but I'm not sure. Any easy way to check after the fact?

The bitcoind daemon has kept itself quite busy (20-25% CPU according to the top utility) as soon as I launched it. It seems a little too much for simple verification of the new blocks.

[achow101]

I think so, but I'm not sure. Any easy way to check after the fact?

You can check in the debug.log file. It'll say something about reindexing there.

IIRC, reindexing can result in the block files having different modification times, even if they weren't actually modified.

[d_eddie]

The log doesn't mention any more indexing than I expected - that is, apparently only from the first "new block" onwards. I've filtered the content for relevance.

Code:

2019-09-11T08:59:44Z * Using 2.0 MiB for block index database
2019-09-11T08:59:44Z * Using 255.8 MiB for transaction index database
2019-09-11T08:59:44Z init message: Loading block index...
2019-09-11T08:59:44Z Opening LevelDB in /home/btc/.bitcoin/blocks/index
2019-09-11T08:59:44Z Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
2019-09-11T09:01:45Z  block index          121209ms
2019-09-11T09:01:45Z Opening LevelDB in /home/btc/.bitcoin/indexes/txindex
2019-09-11T09:01:46Z Using obfuscation key for /home/btc/.bitcoin/indexes/txindex: 0000000000000000
2019-09-11T09:01:46Z txindex thread start
2019-09-11T09:01:46Z txindex is enabled at height 585264
2019-09-11T09:01:46Z txindex thread exit
2019-09-11T09:11:52Z * Using 2.0 MiB for block index database
2019-09-11T09:11:52Z * Using 255.8 MiB for transaction index database
2019-09-11T09:11:52Z init message: Loading block index...
2019-09-11T09:11:52Z Opening LevelDB in /home/btc/.bitcoin/blocks/index
2019-09-11T09:11:52Z Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
2019-09-11T09:12:07Z  block index           15025ms
2019-09-11T09:12:07Z Opening LevelDB in /home/btc/.bitcoin/indexes/txindex
2019-09-11T09:12:08Z Using obfuscation key for /home/btc/.bitcoin/indexes/txindex: 0000000000000000
2019-09-11T09:12:08Z txindex thread start
2019-09-11T09:12:08Z Syncing txindex with block chain from height 585265
2019-09-11T09:13:21Z Syncing txindex with block chain from height 585266
2019-09-11T09:13:46Z txindex is enabled at height 585366

[achow101]

The log doesn't mention any more indexing than I expected - that is, apparently only from the first "new block" onwards. I've filtered the content for relevance.

Do you see any of the files that were replaced mentioned in the debug.log?

[d_eddie]

^^^

In the debug log, I only see blocks that "make sense": height 585264 onwards. No explicit mapping between block number and blk file, but I'd say these are the newest - blocks that never were on the backup server to begin with. They are, now.