Block data download speed

[pawanjain]

So recently, I had setup a bitcoin full node on my Raspberry Pi 400 and started to download the block data.
I noticed that although my internet speed is good the block data is getting downloaded at a very slow speed.
I am not sure but I think it might be because of one of the following reasons

Internet traffic (I use the wifi on 3 devices including the raspberry)
Low system config (obviously the raspberry has a low end config. Does it have something to do with slow downloading of block data ?)

My internet speed is 100 Mbps (megabits) constant yet it takes about 45 minutes to download 150 MB of block data.
Is that normal ?



If anyone wants to look at my Raspberry Pi 400 setup
https://bitcointalk.org/index.php?topic=5333360

[ranochigo]

Bitcoin Core has a parallel download during the IBD with the headers-first synchronization. The speed shouldn't be solely limited to a single peer as the blocks can be downloaded in no specific order, as long as the block hashes are in the headers on the best tip.

Now, with Raspberry Pi, there are several bottlenecks, the USB port (for the external HDD) and the CPU speed. The CPU speed is probably your main problem; Raspberry Pis are very underpowered and it would take days for a completely synchronization without assumevalid (in my case) but some script validations are skipped with assumevalid. Bitcoin Core still has to validate the blocks, transactions, etc and also build the chainstate and index the block. These are heavily reliant on your CPU which is unfortunately not an area Raspberry Pis excel in.

What I've done when I ran a node on my Pi is to synchronize on my actual computer before transferring the entire directory to an external HDD and point my Raspberry Pi's Bitcoin Core datadir at it. This skips most of the heavily validation on the Pi and reduces the time taken for it to be up and running.

[pawanjain]

Bitcoin Core has a parallel download during the IBD with the headers-first synchronization. The speed shouldn't be solely limited to a single peer as the blocks can be downloaded in no specific order, as long as the block hashes are in the headers on the best tip.

Now, with Raspberry Pi, there are several bottlenecks, the USB port (for the external HDD) and the CPU speed. The CPU speed is probably your main problem; Raspberry Pis are very underpowered and it would take days for a completely synchronization without assumevalid (in my case) but some script validations are skipped with assumevalid. Bitcoin Core still has to validate the blocks, transactions, etc and also build the chainstate and index the block. These are heavily reliant on your CPU which is unfortunately not an area Raspberry Pis excel in.

So there's pretty much nothing I can do about it. I will try overclocking the Pi to 2GHz to boost it a little.

What I've done when I ran a node on my Pi is to synchronize on my actual computer before transferring the entire directory to an external HDD and point my Raspberry Pi's Bitcoin Core datadir at it. This skips most of the heavily validation on the Pi and reduces the time taken for it to be up and running.

Even I had downloaded the first 35% of block data through my Ryzen 5 laptop initially but since I couldn't keep it on for a long time I though to transfer the data to my Pi and run it 24x7. Well, yeah it's been a few days now and I am at 42% still.

Let's see how much boost does the overclocking gives to the Pi.

[ranochigo]

So there's pretty much nothing I can do about it. I will try overclocking the Pi to 2GHz to boost it a little.

You can increase your dbcache for the time being and it'll help with the synchronization a lot more significantly as they're cached in the RAM.

I've got a RPi 4 B and it runs at 75 deg C at stock, not too sure about RPi 400 but do be careful with the temperatures. Keep in mind that the synchronization will get slower as you're approaching the current blocks.

[BitMaxz]

What are the ram/memory specs of this unit? Is that 4GB ram?
Follow the ranochigo suggestion and put dbcache in bitcoin.conf file with this setting "dbcache=2048" only use this if that raspi 400 has 4GB ram.

Another suggestion if you want to speed it up more there is a big increase in performance if you switch to SSD as your storage. I know HDD is cheap but if you want to run a full node and want it to speed up the syncing process you need to switch to SSD.

Or borrow SSD from your friends temporarily until you downloaded the whole blockchain and transfer them to your HDD.

[ranochigo]

Another suggestion if you want to speed it up more there is a big increase in performance if you switch to SSD as your storage. I know HDD is cheap but if you want to run a full node and want it to speed up the syncing process you need to switch to SSD.

Or borrow SSD from your friends temporarily until you downloaded the whole blockchain and transfer them to your HDD.

No harm trying, really but from my experience the bottleneck wasn't with my disk. My Pi decided to throttle severely before I slapped on a heat sink on it. The only time I saw any significant changes to the speed was only with the SSD and the HDD on my desktop, really had no other bottlenecks other than the disk in that case.

Again, no harm trying but the bottleneck is *definitely* not the disk.

[pawanjain]

What are the ram/memory specs of this unit? Is that 4GB ram?
Follow the ranochigo suggestion and put dbcache in bitcoin.conf file with this setting "dbcache=2048" only use this if that raspi 400 has 4GB ram.

Another suggestion if you want to speed it up more there is a big increase in performance if you switch to SSD as your storage. I know HDD is cheap but if you want to run a full node and want it to speed up the syncing process you need to switch to SSD.

Or borrow SSD from your friends temporarily until you downloaded the whole blockchain and transfer them to your HDD.

Thanks for that info. I do have a spare 480 GB SSD. I will use it for the time being to download the block data and then transfer the block data back to HDD when it's fully synchronized.

Regarding dbcache, I just looked up and there seems to be no folder called .bitcoin in my home directory.
I am using an external HDD and whole installation has been done on the HDD itself.
Should I create a new folder ./bitcoin and then place the new conf while there ?

[ranochigo]

Regarding dbcache, I just looked up and there seems to be no folder called .bitcoin in my home directory.
I am using an external HDD and whole installation has been done on the HDD itself.
Should I create a new folder ./bitcoin and then place the new conf while there ?

No. The bitcoin.conf is stored in your data directory. It should be in the same folder with your mempool.dat, peers.dat,anchors.dat etc. This requires a restart of your Bitcoind.

Alternatively, start bitcoind with -dbcache=2048.

[HCP]

Definitely a CPU bottleneck... the download of the data is only a small part of the syncing process... it's the actual number crunching of the block data to verify it (and all the IO read/writes) that will be making the Pi slow.

From a review of the Pi 400 that I saw... they had put in a large metal plate to act as a kind of heat spreader off the CPU to try and dissipate heat...

And from a write-up testing this cooling solution, it seems that this heat spreader actually does a better job for the Pi400 than a standard Pi4B with a cooling fan!

So, it'll likely be a combination of "low end" CPU + Disk I/O that is slowing things down, tbh.

[LoyceV]

I do have a spare 480 GB SSD. I will use it for the time being to download the block data and then transfer the block data back to HDD when it's fully synchronized.

I have my .bitcoin directory on SSD, and inside it is a symlink that puts the blocks directory on HDD. That's enough to remove the HDD as bottleneck, without taking too my space on my (much smaller) SSD.

[pawanjain]

Alternatively, start bitcoind with -dbcache=2048.

Yup that worked. Used SSD, overclocked and increased the dbcache.
The download speed has increased drastically now.
I can say that the speed has increased at least by 30% - 40% from what it was earlier.

Definitely a CPU bottleneck... the download of the data is only a small part of the syncing process... it's the actual number crunching of the block data to verify it (and all the IO read/writes) that will be making the Pi slow.

From a review of the Pi 400 that I saw... they had put in a large metal plate to act as a kind of heat spreader off the CPU to try and dissipate heat...

And from a write-up testing this cooling solution, it seems that this heat spreader actually does a better job for the Pi400 than a standard Pi4B with a cooling fan!

So, it'll likely be a combination of "low end" CPU + Disk I/O that is slowing things down, tbh.

Yes the CPU temperature is much better in Pi 400. I had been monitoring the CPU temperature and even after overclocking the temperature never crossed 55 degrees.

I do have a spare 480 GB SSD. I will use it for the time being to download the block data and then transfer the block data back to HDD when it's fully synchronized.

I have my .bitcoin directory on SSD, and inside it is a symlink that puts the blocks directory on HDD. That's enough to remove the HDD as bottleneck, without taking too my space on my (much smaller) SSD.

That's a nice way to get around the CPU bottleneck. Will keep in this mind in case I need to implement it.
So far the performance is increased since I am now using a SSD itself to store the block data.