Bitcoin Forum
October 05, 2022, 03:24:53 AM *
News: Latest Bitcoin Core release: 23.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Bitcoin core reindexing  (Read 210 times)
Kuffy (OP)
Member
**
Offline Offline

Activity: 127
Merit: 31

The real Jet Cash.


View Profile WWW
February 24, 2020, 08:16:38 AM
Merited by suchmoon (4), bones261 (4), LoyceV (2), DdmrDdmr (2), LFC_Bitcoin (1), BitMaxz (1), ETFbitcoin (1), Heisenberg_Hunter (1), Devawnm367 (1)
 #1

I've got 3 computers with Bitcoin core installed, and I decided to do this as a form of backup, as well as an experiment as part of my digital nomad life. I have Ubuntu installed on a Netbook with a smallish hard drive, so my wallet and the blockchain are kept on an external 250Gb Samsung SSD. It is a while since I synchronised this, and I decided to bring it up to date. It appeared to start correctly, but then told me it was reindexing the blockchain from about 7 years ago. This probably means there is a corrupted record on the disk. I could have copied the blockchain, and associated files such as chainstate from a backup drive, but I decided to run the reindexing to see how long it took. The wallet data seems to be correct, and I am able to see some early transactions. I stated this thread to remind new Bitcoiners to backup their files regularly if they are running their own node.

Whilst reindexing does not download the complete blockchain, it does try to reuse the data you have stored, and it seems to discard the  blockchain and chainstate indices, so it will still consume a large quantity of ISP bandwidth. I'm running this over public WiFi, so I expect it will take a couple of weeks to rebuild the files.

Please feel free to comment if I have misunderstood this process, or if you have any suggestions or comments about recovery from minor disk errors. I would also remind users of external SSD drives that they should "eject" the drive before disconnecting it - this ensures that write buffers are cleared to maintain the integrity of the data structure.
1664940293
Hero Member
*
Offline Offline

Posts: 1664940293

View Profile Personal Message (Offline)

Ignore
1664940293
Reply with quote  #2

1664940293
Report to moderator
Each block is stacked on top of the previous one. Adding another block to the top makes all lower blocks more difficult to remove: there is more "weight" above each block. A transaction in a block 6 blocks deep (6 confirmations) will be very difficult to remove.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1664940293
Hero Member
*
Offline Offline

Posts: 1664940293

View Profile Personal Message (Offline)

Ignore
1664940293
Reply with quote  #2

1664940293
Report to moderator
joniboini
Legendary
*
Offline Offline

Activity: 1596
Merit: 1564


View Profile
February 24, 2020, 10:04:50 AM
 #2

The bandwidth you required depends on your blockchain data. If the corrupted files aren't that big then it won't take long. Your computer specs will also affects it. CMIIW. Further reading:

https://bitcointalk.org/index.php?topic=582554.0
https://coinguides.org/rescan-reindex-wallet/
https://bitcoin.stackexchange.com/questions/60709/when-should-i-use-reindex-chainstate-and-when-reindex
Kuffy (OP)
Member
**
Offline Offline

Activity: 127
Merit: 31

The real Jet Cash.


View Profile WWW
February 24, 2020, 10:09:05 AM
Last edit: February 24, 2020, 12:14:35 PM by Kuffy
 #3

It running faster than I expected. It has already caught up about a year of data, so I'm now in the high 6 years recovery band. I think that is good for McDonalds WiFi using a Celeron netbook.

It's a shame I need to leave and sort out the van. The resync' is rattling along, and I'm under 6 years backlog now.
LFC_Bitcoin
Legendary
*
Offline Offline

Activity: 2940
Merit: 7291


1st licensed crypto casino; the future of gaming.


View Profile
February 24, 2020, 12:20:13 PM
 #4

The first couple of years are always quick, trust me that the last 2 or 3 years will seem to take forever Smiley

What version of Bitcoin Core are you using?


       ▀
   ▄  ███ ███
      ▄▄▄ ▀▀▀
▀  ▀  ███     ███
      ▄▄▄ ▄▄▄ ▀▀▀
▀  ▀  ███ ███
      ▄▄▄     ███
▀  ▀  ███     ▀▀▀
   ▄  ▄▄▄ ███
      ███ ▀▀▀
       ▄
.
Bitcasino.io
.
Loyalty Club
.
Casino boost
.
2000+ Games
.
Weekly tourneys
.
24/7 Support
.
Fast Payouts
███████████████████████████
███████████████████████████
███████████████████████████
██████████████████▀▀███████
█████████████▀▀▀    ███████
████████▀▀▀   ▄▀   ████████
█████▄     ▄█▀     ████████
████████▄ █▀      █████████
█████████▌▐       █████████
██████████ ▄██▄  ██████████
████████████████▄██████████
███████████████████████████
███████████████████████████
███████████████████████████
███████████████████████████
███████████████████████████
██████▀███████▀   ▀▀▀▄█████
█████▌  ▀▀███▌       ▄█████
█████▀               ██████
█████▄              ███████
██████▄            ████████
███████▄▄        ▄█████████
██████▄       ▄████████████
███████████████████████████
███████████████████████████
███████████████████████████
███████████████████████████
██████▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀██████
████░░░▄▄▄▄▄▄▄▄▄▄▄▄▄░░░████
███░░▄█████████████▀▀▄░░███
███░░████▀▀░░░░░▀▀█▄▄█░░███
███░░███▀░▄█████▄░▀███░░███
███░░███░░███████░░███░░███
███░░███▄░▀█████▀░▄███░░███
███░░████▄▄░░░░░▄▄████░░███
███░░▀███████████████▀░░███
████░░░▀▀▀▀▀▀▀▀▀▀▀▀▀░░░████
██████▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄██████
███████████████████████████
.
...SIGN UP...
jackg
Copper Member
Legendary
*
Offline Offline

Activity: 2604
Merit: 2755


https://bit.ly/387FXHi ← lightning theory


View Profile
February 24, 2020, 06:04:25 PM
Merited by Jet Cash (5), ETFbitcoin (1)
 #5

If the disk cam with a drive utility to check it, I'd suggest running that - it'll mark the bad cells.

Ssds with a io driver (most have them nowadays) should have a nice amount of ram to check the block they write to the drive. It's especially important to eject an ssd safely rather than pulling it out. And if it says drive in use, I'd shutdown the computer to avoid further issue. As the others have suggested an initial sync takes a long time when you get to the later years... And for a few months in 2017 it looks as if it gets stuck.

If it has to reindex again after this one then the corrupted file may need renaming (the last blk file it uses) afaik bitcoin core checks the last 7 on startup so you can already work out where the error may arise but it may not cause a problem anyway as core can usually overwrite it.

HCP
Legendary
*
Offline Offline

Activity: 1946
Merit: 4246

<insert witty quote here>


View Profile
February 25, 2020, 05:54:39 AM
Merited by Jet Cash (5)
 #6

...so my wallet and the blockchain are kept on an external 250Gb Samsung SSD. It is a while since I synchronised this, and I decided to bring it up to date.
Are you pruning? If not, you're on a bit of a fool's errand trying to sync everything to a 250Gb drive!!?! Shocked

Given that the current size of Blockchain on it's own is currently larger than that (nevermind the chainstate and other assorted data files), you're going to run out of room before it finishes syncing.

Kuffy (OP)
Member
**
Offline Offline

Activity: 127
Merit: 31

The real Jet Cash.


View Profile WWW
February 25, 2020, 08:38:17 AM
 #7


Are you pruning? If not, you're on a bit of a fool's errand trying to sync everything to a 250Gb drive!!?! Shocked


That will teach me to avoid doing my homework. The last time I used this SSD the blockchain was about 128Gb. Given that I believe you should leave at least a third of an SSD free for processing files. I probably need at least 1Tb. I've got a 2Tb HDD, but it is slower, and it needs more power, and that defeats the purpose of using the Netbook. I've also got a 1Tb SSD that I was going to use to replace the HDD in the Notebook, so that I could get rid of Windows at last. Probably the sensible thing to do is to copy the 256Gb data onto the 1Tb SSD, and carry on with that. I can then buy a 2Tb SSD for the Notebook.

Note on the current synchronisation status. I went through the correct sequence when shutting down yesterday - close core and wait for completion, eject the SSD, and then shut down Ubuntu. I've just restarted the reindexing, and it started again at 6 years and 42 weeks, that's about a year earlier than the previous day's closing point,  but it has skipped quite quickly to 5 years and 23 weeks. Thinking about the post by HCP, it looks as if the disk is short of space, and I need to switch to the 1Tb drive before going much further. I'll leave it to run for another hour or so, as I've got some other stuff to sort out. I'll probably swop drives tomorrow, and let you guys know how it goes. I've got Mint installed on the !Tb SSD, but I might reformat that before copying the Bitcoin files.
Kuffy (OP)
Member
**
Offline Offline

Activity: 127
Merit: 31

The real Jet Cash.


View Profile WWW
February 25, 2020, 11:31:38 AM
 #8

It looks as if the SSD is overloaded. The Netbook went to standby, as I wasn't using it for anything other than sync'ing Bitcoin. When I restarted it, the sync report indicated that it had dropped back over a year, so obviously the disk is unhappy. It's a decent Samsung one, and it hasn't been used all that much, but it is reporting only 47Gb of free space, and it is going to use a lot more than that to get much further. I think I'll buy a new drive for the blockchain.

I like Samsung, and there is quite a lot of choice in 2Tb SSDs. I need to weigh performance, cost and reliability, and my initial decision is to get an 860 QVO internal drive, and insert it into a case for USB connection. As the drive will just be used for the Bitcoin node, price was a consideration. The QVO seems to be slightly inferior to the EVO, but this relates to disk activity. As the blockchain is incremental rather than constantly being moved and replaced, I thought that the cost saving was worth the marginal loss in performance and endurance. Models other than the QVO and EVO are more expensive, and I don't think the improvements are worth the extra costs. I decided to case the drive myself for two reasons. I may want to move it into a notebook at some stage in the future, and there is a cost saving. Transfer speeds may be faster with a factory built external drive, but I don't believe that this will be significant on low to mid spec note/net books. At the moment the netbook is running under Ubuntu, but I may replace this if the machine is going to be used mainly for crypto projects. I'll need to research that.

Any comments or advice will be received gratefully. Smiley
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!