Indexing new node has slowed to a crawl

[doctrbob]

I'm trying to download a bitcoin node. Bitcoin Core Ver 26.0  Indexing started out fast but has now slowed to a crawl.  Bitcoin core did close down twice on its own (once because the hard drive went to sleep before I changed the setting and not sure why the second time).  Did that possibly corrupt the indexing? It started up fine when restarted but now is barely moving.   I closed the program with control q and let it gracefully shut down and rebooted the computer, but that didn't help. I'm down to 361,355 blocks left to index, but it says 10 days until completion. It's been running 30 hours.  I was at 47 hours last night and decreasing quickly.  Thanks in advance.

[1714565120]

1714565120

[1714565120]

1714565120

[1714565120]

1714565120

[Mia Chloe]

What you are experiencing is normal.
Firstly you don't download the node rather the computer you are using runs as the node. You are actually downloading all of Bitcoin's block chain data and it's actually over 500Gb except you are trying to run a prune node by download some data and omitting some from a particular time frame.
Depending on the computation power of your computer and network speed, downloading the Blockchain data from the Genesis block till date and also syncing, could take a couple of days. The faster you computational power and network strength, the faster the syncing process.

Make sure you have a storage of up to 1tb preferably SSD to prevent issues due to space as it syncs and also when you attempt running it.

[doctrbob]

Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

When you say start resyncing, you mean delete everything and start over?

[Cricktor]

Increasing dbcache=<about half of RAM in MB> might help a bit (add in your bitcoin.conf file).

blocksonly=1 during initial blockchain download (IBD) lets your node only deal with block data over the network, don't bother at that time with unconfirmed transations.

External storage via USB isn't ideal, USB protocol eats some performance and with an UTXO set of around 16GB size your storage has to shuffle quite some data around when it doesn't fit all in your RAM during blockchain sync.

With optimized settings for a Raspi 4B with 8GB RAM and a 1TB SSD attached via USB3 a full IBD took about 4 days some months ago (my internet speed is 100MBit/s downstream).
I was kind of surprised, I expected more time as a Raspi 4B isn't really fast compared to normal PC CPUs. I did this with an Umbrel node as an experiment and I hope Umbrel doesn't use some sort of shortcuts for IBD. I might still have the debug.log of this experimental IBD.

[Amphenomenon]

Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

I don't think you have to restart syncing, try Increasing the database cache size to

Code:

dbcache=2048

since you are using an external drive this would increase the syncing process

When you say start resyncing, you mean delete everything and start over?

yes but you don't have to use this

[promise444c5]

Firstly, check the usage of your hardisk if it's  getting  to 100% it will definitely  be slow no matter what .
I've been unable to download mine due to low bandwidth  in my country  but I found a great trick that makes is fast .
Get an external SSD drive
Create a copy of your bitcoin data in the new SSD drive and rename the initial to another name to serve as backup incase of any  error
Now follow this step as directed  from @NotATether if you're using the GUI mode

you need to move the datadir to be on the partition that has your external drive.

You will need about 650-700 GB free space on the external drive.

The datadir is located in C:\Users\<you>\AppData\Local\Bitcoin (or maybe it's in AppData\Roaming\Bitcoin, I don't remember which). You need to shut down Core and move all the contents of the folder into a new folder like D:\Bitcoin

Then in the C drive, go to the original folder like AppData\Local\Bitcoin or like that and create a file called bitcoin.conf and use a text editor to add the following:

Code:

datadir=D:\Bitcoin (or whatever your new folder is called)

Additionally, you can also add this setting:

Code:

dbcache=(write down 50% of ram, in MB: for example just write 1024 to use 1 GB or 2048 for 2GB and so on)

If you have a lot of RAM like at least 8 or 16 gigs then using this option will make syncing faster because it is not constantly writing the chainstate to the disk.

Then you can open Bitcoin Core again and it will store everything in the datadir.

In addition , the file is the bitcoin and you should save it as a conf file not as a text (note this)

[LoyceV]

Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

[promise444c5]

Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

What If I use a  no sdd drive?? Just  asking  because I got an external   ssd drive because my disk functionality  was been  used up entirely to 100% while synchronising.
Does the RAM provides a way of not getting  the disk  used up ??

[Cricktor]

Does the RAM provides a way of not getting  the disk  used up ??

What is a "no SSD drive", a spinning magnetic platter aka HDD drive?

What kind of an interface does your external drive use? Most common would be USB3 or up and USB protocol, especially when UASP is not used (your USB adapter needs to support it and the driver must have it enabled, that's not always on by default, depending what OS you use), slows down the performance of a SSD quite a bit. That's what LoyceV summerized with "internal drives are preferable", especially for SSDs.

More RAM means, more data can be kept in RAM where access speed is still orders of magnitude faster than for internal and to greater extend for external drives. Less often data needs to be flushed out to storage. The OS has more space for file I/O buffers, can better optimize how data is written out to storage (moving R/W heads for HDDs is snail slow compared to flash storage with no moving parts).

[LoyceV]

What If I use a  no sdd drive??

Without enough RAM, that sucks If the 11 GB from chainstate doesn't fit in your file cache, you're going to be scraping that HDD.

I have to ask: if your system still runs on an old HDD, why don't you replace it for an internal SSD instead of external? I made that switch years ago, and I'll never go back! Buy a decent one: some (budget) SSDs have low performance on sustained writes.

Just  asking  because I got an external   ssd drive because my disk functionality  was been  used up entirely to 100% while synchronising.

I did another IBD test recently, with 8 GB RAM, Linux, and SSD, and the SSD was the bottleneck. It's was reading/writing at maximum capacity for hours in a row.

Does the RAM provides a way of not getting  the disk  used up ??

Yes.

[promise444c5]

Without enough RAM, that sucks If the 11 GB from chainstate doesn't fit in your file cache, you're going to be scraping that HDD.

Explains why my OS  crashed without  giving  any sign
Almost lost my important  files though

Or does Windows sucks entirely??

I have to ask: if your system still runs on an old HDD, why don't you replace it for an internal SSD instead of external?

Yes
Never cared about HDD until I was running  a bitcoin Core on the PC

[LoyceV]

Explains why my OS  crashed without  giving  any sign

Not really: the OS should kill the program that runs out of memory instead of crashing.

[promise444c5]

Explains why my OS  crashed without  giving  any sign

Not really: the OS should kill the program that runs out of memory instead of crashing.

But I got a crashed os luckily  I know how I could use the cmd-pmt for the retrieval of my file before resetting because i do keep my files hidden in the program data directory, what could have triggered it  because  It's  was my first  time experiencing it and that was when I started running  the Bitcoin core

[ABCbits]

I'm down to 361,355 blocks left to index, but it says 10 days until completion. It's been running 30 hours.  I was at 47 hours last night and decreasing quickly.  Thanks in advance.

This is extremely slow. Last time i had to perform reindex, it took about 1-2 days where all files stored on HDD.

Thank you for the replies.  I am running a PC with 8gb ram and and an i7 processor and 400 Mbps internet connection.  I have an attached Samsung T7 2tb external drive to run a full node, not pruned.  It just seemed like it was indexing faster and faster than all of a sudden barely moving.  If it's still downloading the blockchain regardless of speed, does that mean there was no corruption in the blockchain that is downloaded so far?

There are 2 things I would improve: get more RAM, and don't use an external drive. It's very easy to accidentally disconnect the USB cable, and that will lead to file corruption. Chainstate when fully synced is 11 GB, that doesn't fit your RAM which will lead to terabytes of data written to your disk.

Adding RAM will improve your overall computing experience, and is not very expensive anyway.

And if you don't use your PC for other task, you can close most application and background application so you can let Bitcoin Core use a bit more of RAM.

But I got a crashed os luckily  I know how I could use the cmd-pmt for the retrieval of my file before resetting because i do keep my files hidden in the program data directory, what could have triggered it  because  It's  was my first  time experiencing it and that was when I started running  the Bitcoin core

I recommend you to create new thread about your problem, rather than continue discussing on this thread.