Bitcoin Forum
March 03, 2024, 10:42:59 PM *
News: Pie Baking Contest
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Extremely slow reindexing  (Read 130 times)
BlackHatCoiner (OP)
Legendary
*
Online Online

Activity: 1442
Merit: 6920


Farewell, Leo


View Profile
February 09, 2024, 12:14:18 PM
Merited by LoyceV (4), ABCbits (2)
 #1

Bitcoin Client Software and Version Number: Bitcoin Core 26.0
Operating System: Raspbian Lite OS (64-bit)
System Hardware Specs: 4 GB memory, 2 TB external HDD.
Description of Problem:

TL;DR: I switched disks, and somehow either the blocks or the chainstate got corrupted, and I had to reindex. I'm one week now, at progress=0.313028. It seems like it goes much slower now. Yesterday it was in 0.30. At this rate, it will take more than a month to reach the tip. I was thinking if it's possible to eject the HDD, plug it a computer with better specs, so it could finish... this month. It should be possible to do, right? The disk is loaded with Raspbian, so I could theoretically load it during boot in any other machine.

(Asking just in case I don't mess things up)

  BTC
.
BTC
.
 BTC
.
BTC
..JAMBLER.io..
██
██
██
██
██
██
██

██

██

██

██
YOUR OPPORTUNITY TO
HAVE BITCOIN BUSINESS

██
██
██
██
██
██
██

██

██

██

██
.
  BTC
. BTC
.
.
 
BTC
  BTC
You get merit points when someone likes your post enough to give you some. And for every 2 merit points you receive, you can send 1 merit point to someone else!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1709505779
Hero Member
*
Offline Offline

Posts: 1709505779

View Profile Personal Message (Offline)

Ignore
1709505779
Reply with quote  #2

1709505779
Report to moderator
1709505779
Hero Member
*
Offline Offline

Posts: 1709505779

View Profile Personal Message (Offline)

Ignore
1709505779
Reply with quote  #2

1709505779
Report to moderator
apogio
Sr. Member
****
Offline Offline

Activity: 364
Merit: 765


Intern at Foxpup's merit cycling club


View Profile WWW
February 09, 2024, 01:44:24 PM
 #2

I don't see a problem with doing what you suggest. I have never tried it though. To be honest, I don't understand how the directories were corrupted, but reindexing can be painful, indeed.

What I have tried in the past, and it worked, was to download the blockchain on my SSD using Windows 11 OS. Then I deleted Bitcoin Core, because I performed a format. Then I downloaded Core again and set it to look in the SSD again. It worked flawlessly. I guess it's a relevant situation, isn't it ?

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
BlackHatCoiner (OP)
Legendary
*
Online Online

Activity: 1442
Merit: 6920


Farewell, Leo


View Profile
February 09, 2024, 02:29:04 PM
 #3

What I have tried in the past, and it worked, was to download the blockchain on my SSD using Windows 11 OS.
I've considered doing that, but I don't have a terabyte of SSD, and I don't want to buy one just for that purpose. HDD satisfies my needs. Sure, Bitcoin Core would be less troublesome, but if it works, even slowly, then I don't care.

Then I downloaded Core again and set it to look in the SSD again. It worked flawlessly. I guess it's a relevant situation, isn't it ?
Hmm. Maybe if I used Bitcoin Core from my main computer without booting into Raspbian, and pointed to the Bitcoin data directory in the plugged HDD, then maybe it would reindex more rapidly. There are quite a lot of permission issues this way, though. (The datadir is read only, and I'll have to run Bitcoin Core as root)

  BTC
.
BTC
.
 BTC
.
BTC
..JAMBLER.io..
██
██
██
██
██
██
██

██

██

██

██
YOUR OPPORTUNITY TO
HAVE BITCOIN BUSINESS

██
██
██
██
██
██
██

██

██

██

██
.
  BTC
. BTC
.
.
 
BTC
  BTC
ranochigo
Legendary
*
Offline Offline

Activity: 2926
Merit: 4041



View Profile
February 09, 2024, 02:57:33 PM
 #4

Yes you can. Format of data directories between versions are the exact same and there is no compatibility issues whatsoever. Of course, remember to shut it down gracefully.

In fact, this was how I encouraged people to do IBD on a proper Desktop instead of their RPi. Resync on RPi was really long for me, and using the Desktop was much less frustrating in comparison.

  BTC
.
BTC
.
 BTC
.
BTC
..JAMBLER.io..
██
██
██
██
██
██
██

██

██

██

██
YOUR OPPORTUNITY TO
HAVE BITCOIN BUSINESS

██
██
██
██
██
██
██

██

██

██

██
.
  BTC
. BTC
.
.
 
BTC
  BTC
LoyceV
Legendary
*
Offline Offline

Activity: 3234
Merit: 16185


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
February 09, 2024, 03:44:56 PM
Last edit: February 09, 2024, 03:55:41 PM by LoyceV
Merited by Findingnemo (1)
 #5

I was thinking if it's possible to eject the HDD, plug it a computer with better specs, so it could finish...
That should work, although I'd prefer to use it as a secondary disk in another system, and not boot from it.
Your problem is the 4 GB RAM in combination with 11 GB chainstate. Add HDD, and it's a very bad combination.
As far as I know, reindexing chainstate requires almost as much writing to disk as a full fresh sync. It only saves writing 580 GB blocks. I wrote 4.58 GB to disk with 8 GB RAM during IBD. Remove the blocks, and you're left with 4 TB. And that was with twice the ram, so you'll need to write more than 4 TB. That's 4+ TB reading and writing more or less random access on a HDD. If it takes a month, that's 1.5+ MB/s on average, which could very well be the best the HDD can do with random reads and writes. This is why I prefer to put chainstate on SSD.
Progress will get worse once you're further ahead, see my graph.

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
satscraper
Hero Member
*****
Offline Offline

Activity: 658
Merit: 1151


Non-custodial BTC Wallet


View Profile
February 12, 2024, 01:51:17 PM
 #6

Bitcoin Client Software and Version Number: Bitcoin Core 26.0
Operating System: Raspbian Lite OS (64-bit)
System Hardware Specs: 4 GB memory, 2 TB external HDD.
Description of Problem:

TL;DR: I switched disks, and somehow either the blocks or the chainstate got corrupted, and I had to reindex. I'm one week now, at progress=0.313028. It seems like it goes much slower now. Yesterday it was in 0.30. At this rate, it will take more than a month to reach the tip. I was thinking if it's possible to eject the HDD, plug it a computer with better specs, so it could finish... this month. It should be possible to do, right? The disk is loaded with Raspbian, so I could theoretically load it during boot in any other machine.

(Asking just in case I don't mess things up)

I had the same experience some time ago. After  deterioration of chainstate data on my external SSD I have fallen for reindex. This  process  was as slow as it was  less than effective. Thus I had to start the sync from scratch.

Hence, to not waste time,  I would advocate  IBD rather than reindex in the case of any chainstate corruption.

█▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

█          ▄         ▄      ▄▄▄▄▄
█       ▄███      ▄███      █████
█        ████      ████     ▀▀▀▀▀
█         ████      ████
█          ████▄▄▄▄▄▄████▄▄▄▄▄▄▄▄
█           █████████████████████
█            ▀█████▄   ▀█████▄
█              ▀█████▀   ▀█████▀
█                 ▀▀        ▀▀

█▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
.....Your private Bitcoin wallet for desktop.....█▀▀▀▀▀▀











█▄▄▄▄▄▄
▀▀▀▀▀▀█











▄▄▄▄▄▄█
DaveF
Legendary
*
Offline Offline

Activity: 3402
Merit: 6098


Crypto Swap Exchange


View Profile WWW
February 12, 2024, 03:14:44 PM
 #7

If you do it on a full desktop system, mind the file permissions. 99% of the time there is no issue that 1% will drive you nuts.

And I know it's been said but I'll say it again, get a SSD

And, with all the disk I/O and everything else make sure the RPi is not throttling for either temperature because the CPU being hammered or power because of the constant pull from the drive.

-Dave

█▀▀▀











█▄▄▄
▀▀▀▀▀▀▀▀▀▀▀
e
▄▄▄▄▄▄▄▄▄▄▄
█████████████
████████████▄███
██▐███████▄█████▀
█████████▄████▀
███▐████▄███▀
████▐██████▀
█████▀█████
███████████▄
████████████▄
██▄█████▀█████▄
▄█████████▀█████▀
███████████▀██▀
████▀█████████
▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀
c.h.
▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄
▀▀▀█











▄▄▄█
▄██████▄▄▄
█████████████▄▄
███████████████
███████████████
███████████████
███████████████
███░░█████████
███▌▐█████████
█████████████
███████████▀
██████████▀
████████▀
▀██▀▀
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!