Bitcoin Forum
June 20, 2019, 08:31:54 AM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Very hard to sync Core Client from scratch  (Read 74 times)
2J87qy5arXEXVhc9hfT2
Newbie
*
Offline Offline

Activity: 7
Merit: 7


View Profile
April 09, 2018, 09:09:40 AM
 #1

Recently I started running a Core 15.1 node with the blockchain on an external hard drive, and found it very hard to fully sync the blockchain.
The problem I keep running into is that (in about 90% of my attempts) the sync does not complete. A DB corruption occurs at some point (usually pretty late in the game, maybe 36 hours into it).

I did some digging in forums and saw that I am not the first user to notice this problem. Peter Wuille mentioned that he does a full sync on like a weekly basis and it always works, and he thus suspects that these type of issues are due to sub optimal hardware.

That seemed very reasonable at first because my initial attempt was done using a pretty old 400gig pocket hard drive. So I switched hard drives.
By now, about 2 months later, I have tried to sync the blockchain from scratch about 20 or 30 times and only twice managed to complete the process.
I did that on 4 different types of hard drives, including a brand new 1T Western Digital.

One may say that all 4 hard drives I used are "suboptimal", but I would object to that because
1. After the sync does complete, the node runs just fine.
2. I think it is desirable to be able to run a full node on a raspberry pi with a pocket size hard drive (note that I run the node on a raspberry pi but use my linux desktop to do the initial sync)

Now the issue I have here is not so much the crash during sync, but the fact that when I restart the client after the crash the only option is to basically start from scratch (If I understand correctly, the client is not necessarily downloading everything from scratch but that is not my point) and sync again from Genesis.
Ideally I would like the client to keep synching from the point it successfully got to and not re sync from scratch on such errors.

I am wondering if this resync from scratch on error is necessary. I assume that corruption occurs in one (or more) of the files in the blocks or chainstate directories. If that is so then why sync from Genesis? Is it not possible to, say, delete all files post the corrupted file and sync from there?

If there is no fundamental reason for this behaviour, I would also appreciate a pointer to the relevant code in the client.

Thanks!!
1561019514
Hero Member
*
Offline Offline

Posts: 1561019514

View Profile Personal Message (Offline)

Ignore
1561019514
Reply with quote  #2

1561019514
Report to moderator
Try The Brand New Ethereum Game
50 Last Players Also Win The Bank
Works On Any iOS/Android Device With Standard Browser
Join Us On Telegram To Get Notified When You Can Win
COLOR PIXELS
AND WIN
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1561019514
Hero Member
*
Offline Offline

Posts: 1561019514

View Profile Personal Message (Offline)

Ignore
1561019514
Reply with quote  #2

1561019514
Report to moderator
1561019514
Hero Member
*
Offline Offline

Posts: 1561019514

View Profile Personal Message (Offline)

Ignore
1561019514
Reply with quote  #2

1561019514
Report to moderator
ScripterRon
Full Member
***
Offline Offline

Activity: 136
Merit: 106


View Profile
April 09, 2018, 01:22:25 PM
 #2

You don't need to delete the blocks directory.  But you do need to delete the chainstate directory if it is corrupted.  LevelDB (which is used for chainstate) causes most of the disk activity and is the most likely source of any corruption, especially if you kill the node without going through the normal shutdown.
2J87qy5arXEXVhc9hfT2
Newbie
*
Offline Offline

Activity: 7
Merit: 7


View Profile
April 09, 2018, 03:04:53 PM
 #3

Thank you for your answer.
So say I sync the node and get a db corruption when the node is, say, 90% synced. Now I delete the chainstate directory and restart the node. Will the node have to re-sync from Genesis (and take the usual 36~48 hours) or will it resume from 90%?
ranochigo
Legendary
*
Offline Offline

Activity: 1694
Merit: 1141

Somewhat inactive.


View Profile WWW
April 09, 2018, 03:55:19 PM
 #4

Now I delete the chainstate directory and restart the node. Will the node have to re-sync from Genesis (and take the usual 36~48 hours) or will it resume from 90%?
Your node would have to rebuild the chainstate by going through all the blocks that you've already downloaded and synchronized. I'm not sure if deleting it would actually require the blocks to be validated again (achow101 said Core does). Anyhow, you can execute -reindex-chainstate to rebuild the chainstate only and that wouldn't need to validate the blocks again since your client is getting the information from your already compiled block index.

If you have a spare hard drive, you can think about synchronized it first and copying the entire data directory to it. If it gets corrupted again, you can just replace the data directory.

2J87qy5arXEXVhc9hfT2
Newbie
*
Offline Offline

Activity: 7
Merit: 7


View Profile
April 09, 2018, 06:19:50 PM
 #5

Now I delete the chainstate directory and restart the node. Will the node have to re-sync from Genesis (and take the usual 36~48 hours) or will it resume from 90%?
Your node would have to rebuild the chainstate by going through all the blocks that you've already downloaded and synchronized. I'm not sure if deleting it would actually require the blocks to be validated again (achow101 said Core does). Anyhow, you can execute -reindex-chainstate to rebuild the chainstate only and that wouldn't need to validate the blocks again since your client is getting the information from your already compiled block index.

If you have a spare hard drive, you can think about synchronized it first and copying the entire data directory to it. If it gets corrupted again, you can just replace the data directory.

Thanks.
Rsyncing from another hard drive is exactly what I did once I realised achieving a full sync from scratch is not likely to happen. Anyway, your remark about the -reindex-chainstate flag is interesting. I'll try that. However, I would still like to understand why the default behaviour is the way it is.
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!