Bitcoin core compiling/syncing but only uses fraction of my cpus

[mxnamvar]

Hi,

My IBD/Sync is going terribly slowly.  Have a gigabit connection and a 20 core Xenon CPU running @2.8 GHz

I see that less than 3% of my cpu processes are being used (verified on Vsphere and via system monitor on the linux VM).  Anyone have any thoughts on how I can get bitcoin core to use my full CPU resources?

I have all 20 cores allocated to the linux (ubuntu) machine as well as 4GB of memory.

Thanks for any help total nube here (BTC core / Linux / ESXI)

[jackg]

The large resources used by core are normally disk io/rw tasks so it's normal for it to use a small amount of your cpu power.

[mxnamvar]

Thank you for your response - I did run btc core on a windows machine last year and the IBD/Sync process was tons faster on a worse (Core I5 6th gen) processor.  That PC did have an SSD however where the current VM is assigned a non SSD HD. 

[jackg]

Thank you for your response - I did run btc core on a windows machine last year and the IBD/Sync process was tons faster on a worse (Core I5 6th gen) processor.  That PC did have an SSD however where the current VM is assigned a non SSD HD. 

How close are you on your current synchronisation? The chain grows over the years so if you're towards the end then it will take longer.

Generally, from my experience, at the start of the syn the intensity on everything is faily similar (but more intense on the network and processor) and then towards the end it's more network andhard drive.

There could be other problems with the machine, have you tried running a test on the hard drive to see how fast it can read and write data (a preinstalled analytics untiity by the OS might be able to determine this) - It also depends on the type of SSD you had I don't know if it's fully proven yet but I was told that SSDs without memory/ram in the controller drive were a similar speed to hard drives.

[mxnamvar]

Hey so I figured it out.  For some reason the memory was the bottleneck.  I stopped the vm and upped the mem to 30 from 4 (GBs) and restarted and now its running like a champ.  Wanted you to know so you let others know.  Thanks so much for your help.

[DaveF]

In addition to just adding memory you should change the dbcache size:
https://bitcointalk.org/index.php?topic=5246078.msg54478975#msg54478975

As you can see from the testing I did it will speed up the initial sync.
There is an upper limit of how much increasing it will help till you hit another bottleneck, but that is more system dependent.

-Dave

[ABCbits]

Hey so I figured it out.  For some reason the memory was the bottleneck.  I stopped the vm and upped the mem to 30 from 4 (GBs) and restarted and now its running like a champ.  Wanted you to know so you let others know.  Thanks so much for your help.

30 GB seems overkill, unless you configure dbcache to use most of your RAM. If you're not sure how to do it, you could either :
1. Add -dbcache=N (N is total of RAM used in MB) if you run Bitcoin Core from Terminal.

Code:

./bitcoin-qt -dbcache=20000

2. Open bitcoin.conf (located on folder where you store blockchain) and add dbcache=N on new line. If bitcoin.conf isn't exist, all you need to do is create new .txt file and rename it as bitcoin.conf.

Code:

dbcache=20000

[DaveF]

30 GB seems overkill, unless you configure dbcache to use most of your RAM. If you're not sure how to do it, you could either :
1. Add -dbcache=N (N is total of RAM used in MB) if you run Bitcoin Core from Terminal.

Code:

./bitcoin-qt -dbcache=20000

2. Open bitcoin.conf (located on folder where you store blockchain) and add dbcache=N on new line. If bitcoin.conf isn't exist, all you need to do is create new .txt file and rename it as bitcoin.conf.

Code:

dbcache=20000

You can also do it via the GUI if you are running it. It's under settings --> options
Once it's fully synced you can change it back if you need to free the RAM for something else if you keep it running.
If you shut it down and have to catch back up regularly then you might as well leave it.

-Dave