Bitcoin Forum
September 15, 2019, 05:42:49 PM *
News: Latest Bitcoin Core release: 0.18.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Old blocks changed  (Read 182 times)
d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 11, 2019, 08:27:31 PM
Merited by LoyceV (1)
 #1

My bitcoin node runs on a linux box. I've been away a few weeks, and I kept it switched off. When I got back, I found the main drive had some problem, so I replaced it and reinstalled the system and some of the software, including bitcoind. So far so good. After this, I proceeded to restore the latest daily backup of the .bitcoin directory. I then left the node running to verify and sync.

The surprise came when the catching up was done, and I launched the new daily backup. I use rsync, so only files that have actually changed do get transferred. Well, apparently about half of the blocks need uploading to backup, which was never the case during my previous daily backup routine. Why have older block files changed?

I'm talking about blk0XXXX.dat files. The latest is blk01790.dat, so I expected the backup to begin well after blk01000.dat, but rsync's output begins like this:

Code:
.bitcoin/blocks/blk00028.dat
.bitcoin/blocks/blk00031.dat
.bitcoin/blocks/blk00033.dat
.bitcoin/blocks/blk00036.dat
.bitcoin/blocks/blk00037.dat
.bitcoin/blocks/blk00038.dat
...

Why the change in all these old blocks?
1568569369
Hero Member
*
Offline Offline

Posts: 1568569369

View Profile Personal Message (Offline)

Ignore
1568569369
Reply with quote  #2

1568569369
Report to moderator
1568569369
Hero Member
*
Offline Offline

Posts: 1568569369

View Profile Personal Message (Offline)

Ignore
1568569369
Reply with quote  #2

1568569369
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1568569369
Hero Member
*
Offline Offline

Posts: 1568569369

View Profile Personal Message (Offline)

Ignore
1568569369
Reply with quote  #2

1568569369
Report to moderator
fillippone
Sr. Member
****
Offline Offline

Activity: 462
Merit: 1110


Legendary Member Wannabe


View Profile
September 11, 2019, 09:04:23 PM
 #2

Just me asking.
Which version of Bitcoin Core are you running? Which were you running before reinstalling the software?
If you updated the version .dat format might have changed (maybe you didn’t upgrade core since a long time), hence the backup.

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 11, 2019, 09:58:08 PM
Merited by fillippone (1)
 #3

I'm still on 0.18.0 as I was before.

Besides, the format of raw blk*.dat files (basically, a copy of the blockchain) hasn't ever changed AFAIK.
DireWolfM14
Hero Member
*****
Offline Offline

Activity: 518
Merit: 750



View Profile WWW
September 11, 2019, 10:20:17 PM
 #4

Did the name of your hard drive change?  Any difference between the old hard drive's directory path and the new hard drive's directory path will trigger a discrepancy between the files for backup purposes.  

fillippone
Sr. Member
****
Offline Offline

Activity: 462
Merit: 1110


Legendary Member Wannabe


View Profile
September 11, 2019, 10:24:10 PM
 #5

I'm still on 0.18.0 as I was before.

Besides, the format of raw blk*.dat files (basically, a copy of the blockchain) hasn't ever changed AFAIK.


Ok,
Sorry for the poor reply.
I went back and checked what I was referring to.
It is something different it changed in version 0.17:

Quote
If your node has a txindex, the txindex db will be migrated the first time you run 0.17.0 or newer, which may take up to a few hours. Your node will not be functional until this migration completes.

So, yes, different thing.

At least I hope knowing which version you are running will help others with your issue.

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 11, 2019, 10:35:02 PM
 #6

Did the name of your hard drive change?  Any difference between the old hard drive's directory path and the new hard drive's directory path will trigger a discrepancy between the files for backup purposes.  
I don't think the name changed. It's linux.
But even if it changed, why should this affect only part of the files? The backup didn't begin with blk0000.dat, and several files have been skipped - half, more or less.
Abdussamad
Legendary
*
Offline Offline

Activity: 2226
Merit: 1183



View Profile WWW
September 11, 2019, 10:39:01 PM
 #7

manually compare the hashes of the old and new files. sha256sum them for instance. the reason being that timestamp changes can also trigger rsync. from the man page:

Quote
Rsync  finds  files  that  need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

if the hashes are the same then that means the file contents haven't changed and only the timestamp has been changed. then see if there's an rsync option that lets you ignore file timestamps and only compare file hashes.

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 11, 2019, 10:46:56 PM
Last edit: September 12, 2019, 10:02:22 AM by d_eddie
 #8

manually compare the hashes of the old and new files. sha256sum them for instance. the reason being that timestamp changes can also trigger rsync. from the man page:

Quote
Rsync  finds  files  that  need to be transferred using a "quick check" algorithm (by default) that looks for files that have changed in size or in last-modified time.

if the hashes are the same then that means the file contents haven't changed and only the timestamp has been changed. then see if there's an rsync option that lets you ignore file timestamps and only compare file hashes.

Good call, but... unfortunately the old files have all been replaced by the new backup, so I can't check again. However, I did check the timestamps before launching the backup (I knew the timestamp could trigger a "changed" state for rsync) and all the blk*.dat files had the same date on the backup server and on the bitcoin node. This is consistent with my intuitive hypothesis that after being downloaded/assembled, they are only read: probably never written to again.

EDIT  The rsync dry runs were 100% consistent with what happened in the real run and among themselves. File corruption is the only possible explanation for this behavior I can come up with. Possibly the corruption occured while copying back the backup to the target machine, which had a busy disk and a busy network connection from downloading/installing several software packages. However, I just made another daily backup (and several dry runs just to check), and only the expected files were transferred. If anyone has a better hypothesis than selective file corruption, I'd like to hear it.
LFC_Bitcoin
Copper Member
Legendary
*
Online Online

Activity: 1834
Merit: 2072


One of the world's leading Bitcoin-powered casinos


View Profile
September 12, 2019, 05:07:11 PM
 #9

I’d be interested to see achow101’s take on this.

achow101
Moderator
Legendary
*
Offline Offline

Activity: 1890
Merit: 2729


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
September 12, 2019, 05:39:51 PM
 #10

Did you have to reindex after restoring the backup?

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 12, 2019, 06:11:22 PM
 #11

Did you have to reindex after restoring the backup?
I think so, but I'm not sure. Any easy way to check after the fact?

The bitcoind daemon has kept itself quite busy (20-25% CPU according to the top utility) as soon as I launched it. It seems a little too much for simple verification of the new blocks.
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1890
Merit: 2729


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
September 13, 2019, 03:48:13 AM
 #12

I think so, but I'm not sure. Any easy way to check after the fact?
You can check in the debug.log file. It'll say something about reindexing there.

IIRC, reindexing can result in the block files having different modification times, even if they weren't actually modified.

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 13, 2019, 01:48:05 PM
 #13

The log doesn't mention any more indexing than I expected - that is, apparently only from the first "new block" onwards. I've filtered the content for relevance.

Code:
2019-09-11T08:59:44Z * Using 2.0 MiB for block index database
2019-09-11T08:59:44Z * Using 255.8 MiB for transaction index database
2019-09-11T08:59:44Z init message: Loading block index...
2019-09-11T08:59:44Z Opening LevelDB in /home/btc/.bitcoin/blocks/index
2019-09-11T08:59:44Z Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
2019-09-11T09:01:45Z  block index          121209ms
2019-09-11T09:01:45Z Opening LevelDB in /home/btc/.bitcoin/indexes/txindex
2019-09-11T09:01:46Z Using obfuscation key for /home/btc/.bitcoin/indexes/txindex: 0000000000000000
2019-09-11T09:01:46Z txindex thread start
2019-09-11T09:01:46Z txindex is enabled at height 585264
2019-09-11T09:01:46Z txindex thread exit
2019-09-11T09:11:52Z * Using 2.0 MiB for block index database
2019-09-11T09:11:52Z * Using 255.8 MiB for transaction index database
2019-09-11T09:11:52Z init message: Loading block index...
2019-09-11T09:11:52Z Opening LevelDB in /home/btc/.bitcoin/blocks/index
2019-09-11T09:11:52Z Using obfuscation key for /home/btc/.bitcoin/blocks/index: 0000000000000000
2019-09-11T09:12:07Z  block index           15025ms
2019-09-11T09:12:07Z Opening LevelDB in /home/btc/.bitcoin/indexes/txindex
2019-09-11T09:12:08Z Using obfuscation key for /home/btc/.bitcoin/indexes/txindex: 0000000000000000
2019-09-11T09:12:08Z txindex thread start
2019-09-11T09:12:08Z Syncing txindex with block chain from height 585265
2019-09-11T09:13:21Z Syncing txindex with block chain from height 585266
2019-09-11T09:13:46Z txindex is enabled at height 585366
achow101
Moderator
Legendary
*
Offline Offline

Activity: 1890
Merit: 2729


bc1qshxkrpe4arppq89fpzm6c0tpdvx5cfkve2c8kl


View Profile WWW
September 13, 2019, 06:51:40 PM
 #14

The log doesn't mention any more indexing than I expected - that is, apparently only from the first "new block" onwards. I've filtered the content for relevance.
Do you see any of the files that were replaced mentioned in the debug.log?

d_eddie
Hero Member
*****
Offline Offline

Activity: 812
Merit: 671



View Profile
September 14, 2019, 04:12:41 AM
 #15

^^^

In the debug log, I only see blocks that "make sense": height 585264 onwards. No explicit mapping between block number and blk file, but I'd say these are the newest - blocks that never were on the backup server to begin with. They are, now.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!