Initial block download drastically slowed down

[28e7166c23e70c2014feb5d64]

I am attempting to run a bitcoin node for the first time. I downloaded bitcoin core from the official website. I am running `bitcoind` in a Raspberry Pi 4 which has a Samsung T7 with approximately 220 GB still free (checked 2 minutes ago). I started downloading the blockchain on August 17, exactly 2 weeks ago. The first 10 or so days it was downloading about half a year of blockchain per day, and roughly constant speed. However, since the last 3-4 days this drastically slowed down, and now it downloads about one or two days of blockchain per day. At this rate, I will need like one more year to complete the initial download, instead of a few more days as with the initial download rate. There seems to be no error, it just became very slow. Is this normal?

[LoyceV]

Which model Pi do you have exactly? Google tells me they range from 1 to 8 GB RAM, which is the first thing I suspect to be the cause.

[28e7166c23e70c2014feb5d64]

It has 8 GB of RAM.

```
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       452Mi        79Mi       456Ki       7.2Gi       7.2Gi
Swap:          511Mi       324Mi       187Mi
```

[BitMaxz]

Did you edit the bitcoin.conf file?
If not, try editing it and add dbcache to 8GB of RAM. Since you have 8GB, dbcache=4000 might help speed things up.


Or try this config below for Raspberry Pi.It was generated from the jlopp config generator setup for Raspberry Pi. Paste it to your Bitcoin.conf file and save, restart the node, and check for improvements.

Code:

# This config should be placed in following path:
# ~/.bitcoin/bitcoin.conf

# [core]
# Set database cache size in MB; machines sync faster with a larger cache. Recommend setting to 4000 if RAM is available.
dbcache=100
# Keep at most <n> unconnectable transactions in memory.
maxorphantx=10
# Keep the transaction memory pool below <n> megabytes.
maxmempool=50

# [network]
# Maintain at most N connections to peers.
maxconnections=40
# Tries to keep outbound traffic under the given target (in MiB per 24h), 0 = no limit.
maxuploadtarget=5000


# [Sections]
# Most options automatically apply to mainnet, testnet, and regtest networks.
# If you want to confine an option to just one network, you should add it in the relevant section.
# EXCEPTIONS: The options addnode, connect, port, bind, rpcport, rpcbind and wallet
# only apply to mainnet unless they appear in the appropriate section below.

# Options only for mainnet
[main]

# Options only for testnet
[test]

# Options only for regtest
[regtest]

[ABCbits]

Did you edit the bitcoin.conf file?
If not, try editing it and add dbcache to 8GB of RAM. Since you have 8GB, dbcache=4000 might help speed things up.

If OP doesn't use run other app that use lots of RAM, i would recommend higher value (dbcache=6500) and enable ZRAM.

Or try this config below for Raspberry Pi.It was generated from the jlopp config generator setup for Raspberry Pi. Paste it to your Bitcoin.conf file and save, restart the node, and check for improvements.

Code:

# This config should be placed in following path:
# ~/.bitcoin/bitcoin.conf

# [core]
# Set database cache size in MB; machines sync faster with a larger cache. Recommend setting to 4000 if RAM is available.
dbcache=100
# Keep at most <n> unconnectable transactions in memory.
maxorphantx=10
# Keep the transaction memory pool below <n> megabytes.
maxmempool=50

# [network]
# Maintain at most N connections to peers.
maxconnections=40
# Tries to keep outbound traffic under the given target (in MiB per 24h), 0 = no limit.
maxuploadtarget=5000


# [Sections]
# Most options automatically apply to mainnet, testnet, and regtest networks.
# If you want to confine an option to just one network, you should add it in the relevant section.
# EXCEPTIONS: The options addnode, connect, port, bind, rpcport, rpcbind and wallet
# only apply to mainnet unless they appear in the appropriate section below.

# Options only for mainnet
[main]

# Options only for testnet
[test]

# Options only for regtest
[regtest]

Setting by that generator website would slow down IBD process, since it reduce default dbcache value from 500 to 100.

[28e7166c23e70c2014feb5d64]

Code:

Thank you for your comments. I have `dbcache=4096`. I still have 
[code]
$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.6Gi       280Mi        85Mi       4.0Ki       7.4Gi       7.4Gi
Swap:          511Mi       337Mi       174Mi

and `top` is:

Code:

top - 21:26:42 up  1:02,  1 user,  load average: 1.11, 1.14, 1.38
Tasks: 172 total,   1 running, 171 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  4.0 sy,  0.0 ni, 67.8 id, 27.8 wa,  0.0 hi,  0.1 si,  0.0 st 
MiB Mem :   7809.4 total,     78.1 free,    278.7 used,   7574.2 buff/cache     
MiB Swap:    512.0 total,    174.5 free,    337.5 used.   7530.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                 
    742 btcusr    20   0 9912.5m 881320 833248 S   6.9  11.0  11:42.52 bitcoind

Also, using `iotop` I get:

Code:

Total DISK READ:        36.06 M/s | Total DISK WRITE:         0.00 B/s
Current DISK READ:      36.06 M/s | Current DISK WRITE:       3.75 K/s
    TID  PRIO  USER     DISK READ DISK WRITE>    COMMAND                                                                                                                                       
    773 be/4 btcusr     36.06 M/s    0.00 B/s bitcoind [b-initload]

all the time. This could be the bottleneck.[/code]

[philipma1957]

why oh why did you use a raps pi and not an inexpensive laptop

and what size is that external ssd 1tb?

You are about the 20th person that has done this and asked the same questions.

to all of them I say do not use a rasp pi.

and use an empty 2tb ssd

not a 1tb ssd

lastly. 16gb ram is helpful.



as an aside what interest connection do you have/

some services will punish you for doing a 600gb plus download.

[28e7166c23e70c2014feb5d64]

I left `bitcoind` running for some hours in a very powerful machine and can confirm that now it is going very fast again, so the bottleneck is something in the Raspberry Pi.

In the powerful machine I am running it with the default settings, i.e. no bitcoin.conf, except for `-datadir` which I point to the USB storage SSD. In this powerful machine, `pmap` reports a total memory usage of `bitcoind` of about 14e9. It seems that 8 GB of RAM is at least in part responsible for the bottleneck, no matter what you put in `dbcache`.

[LoyceV]

In this powerful machine, `pmap` reports a total memory usage of `bitcoind` of about 14e9. It seems that 8 GB of RAM is at least in part responsible for the bottleneck, no matter what you put in `dbcache`.

Your chainstate directory will grow to about 12 GB, which means anything less than that largely increases disk activity.

[nc50lc]

It seems that 8 GB of RAM is at least in part responsible for the bottleneck, no matter what you put in `dbcache`.

Your first reply has a clue on this.

It may say 7.2Gi "available" memory, but that's after the 7.2Gi "buff/cache" are written to your disk.
Given that Core is already using the drive extensively, increasing the database cache doesn't help as much
since it will otherwise require your OS to swap more memory, further increasing the disk usage.

I have `dbcache=4096`. I still have -snip-

For the new setting like dbcache to apply, you'll have to restart Core (just in case you missed it instead of the above)

[28e7166c23e70c2014feb5d64]

I am curious, if I may ask, why is it so memory hungry? From my limited understanding, while downloading the blockchain it is also validating it. This has to be done one block at a time, because of the nature of the blockchain itself (I guess I can validate a random block, but at some point I will need a hash of the previous, which I must check is valid before I can say this one is valid). If each block weights 1 MB, then how are we needing > 8 GB?

[LoyceV]

I am curious, if I may ask, why is it so memory hungry? From my limited understanding, while downloading the blockchain it is also validating it. This has to be done one block at a time, because of the nature of the blockchain itself (I guess I can validate a random block, but at some point I will need a hash of the previous, which I must check is valid before I can say this one is valid). If each block weights 1 MB, then how are we needing > 8 GB?

The blocks directory contains all confirmed transactions. The chainstate directory contains all knowledge on all existing UTXOs ("funds"). With each block Bitcoin Core downloads, it verifies all transactions in that block, and needs to check if no UTXO that doesn't exist gets spent (as that would mean Bitcoin is created out of thin air), and it updates chainstate by removing the spent UTXOs and adding the newly created ones. So for each block, it has to check (up to) thousands of UTXOs, and chainstate tends to grow over time. If it fits in your RAM, it's quite fast. If it doesn't come from RAM, it has to read and write thousands of UTXOs from disk for each block it verifies. Even on SSDs that just takes time, and USB only makes things worse.

I'm not a blockchain expert, so this is my layman interpretation

[ABCbits]

So for each block, it has to check (up to) thousands of UTXOs, and chainstate tends to grow over time. If it fits in your RAM, it's quite fast. If it doesn't come from RAM, it has to read and write thousands of UTXOs from disk for each block it verifies. Even on SSDs that just takes time, and USB only makes things worse.

For additional information,
1. https://statoshi.info/d/000000009/unspent-transaction-output-set?orgId=1&from=now-5y&to=now&timezone=browser&refresh=10m shows total and size UTXO growth in last 5 years. Peak UTXO size was about 12.8GB.
2. AFAIK the read/write for those UTXO considered as random read/write operation. So you'll notice the slowdown if you use HDD, very cheap SSD or other slow storage.

[zabzob]

why oh why did you use a raps pi and not an inexpensive laptop

and what size is that external ssd 1tb?

You are about the 20th person that has done this and asked the same questions.

to all of them I say do not use a rasp pi.

and use an empty 2tb ssd

not a 1tb ssd

lastly. 16gb ram is helpful.



as an aside what interest connection do you have/

some services will punish you for doing a 600gb plus download.

I'm wondering if the bottleneck issues OP experienced with the Pi 4 could be avoided by using a Pi 5? I'm asking because I'm planning on doing exactly this to set up a node and Fulcrum server. I prefer this option to a laptop because I want something portable that I can travel with. The Pi 5 has a 16GB RAM option, plus it has a faster CPU and PCIe. It seems to me these specs should make the Pi 5 a capable cheap option for running a node with a fast 2TB ssd. Any reason not to do this? I don't need the fastest sync time, don't mind waiting a few days. But wouldn't want to be the 21st person to have this issue

[ABCbits]

--snip--

I'm wondering if the bottleneck issues OP experienced with the Pi 4 could be avoided by using a Pi 5? I'm asking because I'm planning on doing exactly this to set up a node and Fulcrum server. I prefer this option to a laptop because I want something portable that I can travel with. The Pi 5 has a 16GB RAM option, plus it has a faster CPU and PCIe. It seems to me these specs should make the Pi 5 a capable cheap option for running a node with a fast 2TB ssd. Any reason not to do this? I don't need the fastest sync time, don't mind waiting a few days. But wouldn't want to be the 21st person to have this issue

The RAM/USB storage bottleneck can be avoided, just make sure configure Bitcoin Core to use most of RAM capacity. But if you don't need customizability offered by Pi 5, you may want to consider Mini PC. At similar cost, you can find mini PC that use somewhat faster CPU (i refer to Intel N100 for comparison[1-2]).

[1] https://www.cpu-monkey.com/en/compare_cpu-raspberry_pi_5_b_broadcom_bcm2712-vs-intel_processor_n100
[2] https://www.cpubenchmark.net/compare/6054vs5157/BCM2712-vs-Intel-N100

[zabzob]

--snip--

I'm wondering if the bottleneck issues OP experienced with the Pi 4 could be avoided by using a Pi 5? I'm asking because I'm planning on doing exactly this to set up a node and Fulcrum server. I prefer this option to a laptop because I want something portable that I can travel with. The Pi 5 has a 16GB RAM option, plus it has a faster CPU and PCIe. It seems to me these specs should make the Pi 5 a capable cheap option for running a node with a fast 2TB ssd. Any reason not to do this? I don't need the fastest sync time, don't mind waiting a few days. But wouldn't want to be the 21st person to have this issue

The RAM/USB storage bottleneck can be avoided, just make sure configure Bitcoin Core to use most of RAM capacity. But if you don't need customizability offered by Pi 5, you may want to consider Mini PC. At similar cost, you can find mini PC that use somewhat faster CPU (i refer to Intel N100 for comparison[1-2]).

[1] https://www.cpu-monkey.com/en/compare_cpu-raspberry_pi_5_b_broadcom_bcm2712-vs-intel_processor_n100
[2] https://www.cpubenchmark.net/compare/6054vs5157/BCM2712-vs-Intel-N100

I'll consider the mini PC option as well, thanks, although I like the aesthetics of a Pi with Argon case. I might just stick with my clunky 15-inch Thinkpad running Debian for now--save myself the expense of any new hardware purchases, except for the 2tb ssd I'll have to buy in any case. My current 1tb ssd is running out of space with the growing blockchain, I recently had to delete a bunch of files when my node stopped working. Would be nice to replace the thinkpad with a much smaller and lighter server device, but I'm used to lugging it around on trips as a second laptop. so I can live with it for another year or two. Shouldn't be long before the Pi 6 comes out, might as well wait for that...

[pooya87]

The first 10 or so days it was downloading about half a year of blockchain per day, and roughly constant speed. However, since the last 3-4 days this drastically slowed down, and now it downloads about one or two days of blockchain per day.
There seems to be no error, it just became very slow. Is this normal?

I didn't see anybody clarify this so here it goes.
This is normal and there doesn't need to be any errors. When you download the blocks from first year or two, you are downloading blocks that were mostly empty. So your verification is basically computing a bunch of SHA256 hashes which are very fast and little to no signature verification which is the slowest part of the computation.

As you move forward in the history, the bitcoin adoption grows so does the number of transactions in each block. Therefore as the blocks become fuller, you'll have perform more computation (more expensive signature verification compared to hashes) which slows your verification process down.

[takuma sato]

In this powerful machine, `pmap` reports a total memory usage of `bitcoind` of about 14e9. It seems that 8 GB of RAM is at least in part responsible for the bottleneck, no matter what you put in `dbcache`.

Your chainstate directory will grow to about 12 GB, which means anything less than that largely increases disk activity.

Thanks to mempool spammers, and it will only get worse if we don't stop it. Soon it will be impossible to run a Bitcoin node on a hardware that is not compromised (all modern hardware is). It is no longer viable to fully sync with a nice piece of hardware such as the Thinkpad x60 anymore because the CPU and RAM limitation gets a beating thanks to the anti-Bitcoin movement by Jameson Lopp and other spammers powered by Core 30.

[LoyceV]

It is no longer viable to fully sync with a nice piece of hardware such as the Thinkpad x60 anymore because the CPU and RAM limitation gets a beating

You can up to some point replace RAM for disk, it just gets much slower. Even with an SSD, 4 GB RAM is just low. But if you insist on using dinosaur hardware, you will eventually run into it's limitations.
If you're truely paranoid when it comes to running modern hardware, and also don't want an SPV wallet, you'll just have to be very patient.

[takuma sato]

It is no longer viable to fully sync with a nice piece of hardware such as the Thinkpad x60 anymore because the CPU and RAM limitation gets a beating

You can up to some point replace RAM for disk, it just gets much slower. Even with an SSD, 4 GB RAM is just low. But if you insist on using dinosaur hardware, you will eventually run into it's limitations.
If you're truely paranoid when it comes to running modern hardware, and also don't want an SPV wallet, you'll just have to be very patient.

You can get safe laptops with allow for 16GB ram, but it would be cool if we can get rid of spammers at some point so Bitcoin goes back to handling monetary transactions only because eventually they will only make things worse.