Title: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 20, 2019, 05:21:02 PM I apologise if this had been asked and answered before, but I failed to find that on this forum.
Every time I found a new VPS provider offering cheaper and bigger resources, I move my full node to the new VPS. I make sure the integrity of blk*.dat and rev*.dat files by comparing their md5sum on the source and target VPS. I recently decided not to renew the contract of one of my VPS'. Instead of letting it waiting for its contract expiry date and doing nothing, I configured another full node with pruning as it has only about 100 GB storage space. When I compared the pruning full node with the main one, the most recent blk*.dat and rev*.dat files have different md5sum as below. Code: . That makes me wonder on the integrity of my blockchain files. As far as I know, the only command to verify the blockchain database is "bitcoin-cli verifychain" as below bitcoin-cli shell Code: anto@deeppurple:~$ bitcoin-cli verifychain 4 100 debug.log Code: . But I don't think that will tell me which blk*.dat and rev*.dat files that are corrupted (if any). And it will take quite a long time to verify the whole blockchain files. Is there any better and faster way to manually verify those blk*.dat and rev*.dat? What will happen if my full node sends corrupted data to its peers who asked very old blocks, due to the corrupted blk*.dat and rev*.dat files? Will my full node get notified so that it can automatically repair the relevant corrupted blk*.dat and rev*.dat files? Thanks a lot in advance for your help. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: amaclin1 on February 20, 2019, 06:31:46 PM Your files are not corrupted.
a) BitcoinCore writes blocks into blk*.dat files not in their order. b) BitcoinCore keeps orphan blocks in blk*.dat files This causes differences. The databases are equal - but the files can be different Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 20, 2019, 07:00:41 PM a) BitcoinCore writes blocks into blk*.dat files not in their order. Thanks a lot. That explains why my pruning full node and my main full node have different md5sum from blk01513.dat and rev01513.dat onward, as starting from those files they are running independently.b) BitcoinCore keeps orphan blocks in blk*.dat files How about my other question? I am sorry for a basic question as I didn't realise that until now. What will happen if my main full node (which has all blocks from blk00000.dat) fails to send valid data to its peers due to corrupted (for whatever reasons) blk*.dat and/or rev*.dat files? What kind of mechanism applies in bitcoind? I am running bitcoin core 0.17.1 by the way. And does it make sense to run for instance "bitcoin-cli verifychain 4 563927" once in a while to make sure the integrity of my full node? On my VPS that might take about Edited: Sorry... wrong calculation on "4 seconds to verify 100 blocks". It took about 1 minutes 14 seconds to verify 100 blocks. So to verify 563927 blocks will take about 5 days on my VPS. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: amaclin1 on February 20, 2019, 07:30:47 PM What will happen if my main full node (which has all blocks from blk00000.dat) fails to send valid Nothing will happen.data to its peers due to corrupted (for whatever reasons) blk*.dat and/or rev*.dat files? Your peers will disconnect/ban your misbehaviour node and will try to get valid data from another sources. (Nodes do not send rev*.dat information to each other. These files are local database files for updating the current utxo set) Quote What kind of mechanism applies in bitcoind? Trust to nobody. Check everything youself. Follow the Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: BitCoinDream on February 20, 2019, 07:59:24 PM Your files are not corrupted. Why the old orphans are not removed? Is it a flaw in architecture?a) BitcoinCore writes blocks into blk*.dat files not in their order. b) BitcoinCore keeps orphan blocks in blk*.dat files This causes differences. The databases are equal - but the files can be different Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 20, 2019, 08:21:54 PM Your peers will disconnect/ban your misbehaviour node and will try to get valid data from another sources. This is exactly what I want to avoid, hence my questions.If something went wrong outside my control causing the corruption on blk*dat and/or rev*.dat files, e.g. power outage or glitch, my full node should not be categorised as "misbehave node". There must be better mechanism to avoid such full nodes from being banned. In my case, I want to be able to make sure the integrity of the blockchain files on my full node. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 20, 2019, 11:30:07 PM Out of curiosity, I just executed
Code: anto@deeppurple:~$ bitcoin-cli verifychain 1 10000 And the result Code: 2019-02-20T22:24:01Z Verifying last 10000 blocks at level 1 It took 8 minutes and 37 seconds to verify just the validity of the last 10000 blocks. It will take about 8 hours to verify 565947 blocks. I think that is do-able. So the next time I restart my bitcoind, I will use the following command to check the validity of all block files. Code: bitcoind -checkblocks=0 -checklevel=1 I think that would be enough. But that means I assume that everything must be valid as all block files are valid. I really don't like that kind of assumption though. I am wondering how many people (who run full nodes obviously) check all block files (checkblocks=0) with checklevel=4. Is there anybody who does that regularly? There must be better way than that, as otherwise it will take a lot longer time next year when we reach above 1 million blocks. For instance, applying some kind of a process to notify the peer that the block which it sent out is invalid so that the peer can update its block file based on the data on its outbound peers. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: amaclin1 on February 21, 2019, 06:05:55 AM I think that would be enough. But that means I assume that everything must be valid as all block files are valid. I really don't like that kind of assumption though. Take the binary file editor, open one of the oldest files ( for example blk00005.dat ) and change some data in it. I am quite sure that the bitcoind on your node will not discover any problems. Next time I will ask you to write a small program which intercepts network traffic from your node to 8333 ports and puts some garbage in it. ;D Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 21, 2019, 08:29:00 AM Next time I will ask you to write a small program which intercepts network traffic Why do you think I would do that? Even if I were a top Bitcoin programmer, that would defy my own personal principle. I have already spent a lot of my own personal efforts and (some) money to support Bitcoin network since version 0.9.x as I like the idea and the objective. I hate the change of its name to "Bitcoin Core" though, just because of some stupid people tried to disrupt it.from your node to 8333 ports and puts some garbage in it. ;D Take the binary file editor, open one of the oldest files ( for example blk00005.dat ) I believe this would only happen on a crappy in-house developed software and only maintained by 1 or 2 programmers. There are a lot of companies involve as this becomes an industry with huge market valuation and supported by thousands of developers around the globe. So I really doubt that we (including me) are so stupid to trust software that has no mechanism to maintain the integrity of the database.and change some data in it. I am quite sure that the bitcoind on your node will not discover any problems. What I asked in this topic is how to do manual check which is better and faster than using "bitcoin-cli verifychain" command. Maybe there is no other way to do manual check. But I believe there must be a mechanism within bitcoind (and bitcoin-qt) software to make sure the integrity of blockchain data is properly maintained. I asked about that mechanism is for me to understand the impact on my full node. A few years back when the blockchain data on my full node was corrupted, I had to do reindex resulting it to re-download the entire blocks from blk00000.dat. But you pointed out something that I have to double check myself. I think I will intentionally corrupt the 2nd last blk*.dat file and run my full node on Bitcoin testnet to see what will happen. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: almightyruler on February 23, 2019, 03:05:54 PM How about my other question? I am sorry for a basic question as I didn't realise that until now. What will happen if my main full node (which has all blocks from blk00000.dat) fails to send valid data to its peers due to corrupted (for whatever reasons) blk*.dat and/or rev*.dat files? What kind of mechanism applies in bitcoind? It's my understanding that the blk*.dat files are append only, so everything but the current file is 100% immutable. (Until I started pruning on my oldest Bitcoin node, my low numbered blk*.dat files had a last-modified timestamp from 2011.) If this is correct, there's no need to repeatedly verify the entire blockchain with exhaustive checks, since the data in the large majority of the dat files will never be updated by the client. Something like this should suffice: 1. A byte level checksum of all blk*.dat and rev*.dat files except the current file to confirm that previously verified data has not changed; 2. A complete verification of individual blocks and transactions in the current blk*.dat file (-checkblocks=x at startup, or the verifychain RPC call) The value of x would need to be generous in order to ensure it covers the final block file. (Would be handy if there was a way to specify the height, blockhash, or blk*.dat file to start at.) Back in 2013 it seems the client checked the last 2500 blocks by default! https://bitcointalk.org/index.php?topic=141200.0 Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 25, 2019, 11:19:01 AM It's my understanding that the blk*.dat files are append only, so everything but the current file is 100% immutable. (Until I started pruning on my oldest Bitcoin node, my low numbered blk*.dat files had a last-modified timestamp from 2011.) If this is correct, there's no need to repeatedly verify the entire blockchain with exhaustive checks, since the data in the large majority of the dat files will never be updated by the client. Something like this should suffice: Indeed. It is only the latest blk*.dat file that is being updated. The rest of the blk*.dat files are left untouched and they are only being read when peers request the blocks located in those older blk*.dat.My question was, what happen when a peer requests old blocks that are located in a corrupted blk*.dat file (e.g. due to bitcoind crashes or not properly shutdown), so bitcoind sends corrupted blocks as well. From Bitcoin network perspective, I think those peers will just reject the corrupted blocks (perhaps banned the peer which sends it) and request the same blocks to other peers. So there is no issue at bitcoin network level. But the peer which has corrupted blk*.dat files will eventually be isolated as a lot of peers ban it. So I was wondering if there would be a mechanism to notify the peer which sends corrupted blocks, so that it can update its blk*.dat based on the data from its outbound peers. But the main problem is that, nobody can know for sure whether the corrupted block is due to the peer has corrupted blk*.dat file or the peer intentionally sends garbage block. So I think it might be difficult to implement such mechanism. But perhaps the top Bitcoin programmers have some ideas to deal with this kind of issues. Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: amaclin1 on February 25, 2019, 11:34:00 AM But the peer which has corrupted blk*.dat files will eventually be isolated as a lot of peers ban it. Existed network nodes do not ask old blocks. So, your node will be fine.Transaction validation does not require data from old blocks either. So, if your UTXO database is fine - your node is also fine and robust. The only thing you can not do is cloning. Quote mechanism to notify the peer which sends corrupted blocks No reason to notify a daemon.Title: Re: How to manually verify blk*.dat and rev*.dat files? Post by: .anto. on February 25, 2019, 11:57:16 AM Existed network nodes do not ask old blocks. So, your node will be fine. Thanks for your confirmation.Transaction validation does not require data from old blocks either. So, if your UTXO database is fine - your node is also fine and robust. The only thing you can not do is cloning. Perhaps I am just being too paranoid about maintaining the integrity of my full nodes. That is perhaps because I recently keep updating my criteria to ban misbehaving peers, on top of the ones automatically banned by bitcoind. I think I will post another topic about this as I am not sure about something. |