Linux SysAdmin Help Debugging Slow Blockchain DL

[bagpuss]

I'm setting up a full bitcoin node on a raspberry pi and an external HDD - it's taken two weeks so far and still going. I'd like to learn how to debug where the bottleneck is. Any linux sysadmin folk can spare some time to guide me would be much appreciated.


RPi 3B+, 4GB RAM, Raspbian 4.19.66
USB3, to 2TB HDD
50Mbps up/down link

TOP
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.4 us,  2.0 sy,  0.0 ni, 74.0 id, 19.5 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3906.0 total,     66.9 free,   1175.5 used,   2663.6 buff/cache
MiB Swap:    100.0 total,      0.4 free,     99.6 used.   2572.3 avail Mem

IOTOP
Total DISK READ:      1087.31 K/s | Total DISK WRITE:         0.00 B/s
Current DISK READ:    1093.92 K/s | Current DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                        
 1686 be/4 pi       1087.31 K/s    0.00 B/s  0.00 % 88.99 % bitcoin-qt [bitcoin-msghand]
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init splash

IOSTAT -MX
Linux 4.19.66-v7l+ (xxx)      22/09/19        _armv7l_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           6.65    0.00    2.20   22.27    0.00   68.88

Device            r/s     w/s     rMB/s     wMB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
mmcblk0          0.43    0.25      0.01      0.01     0.12     0.10  22.18  27.57    2.35   63.74   0.00    26.21    46.69   1.65   0.11
sda             83.01    4.43      0.94      1.06     0.01     5.06   0.01  53.35   17.57  137.77   1.92    11.56   244.82   9.16  80.12

[LoyceV]

From what I've seen, the HDD is most likely the bottleneck.
If you can use an SSD, that should give a large improvement. If you have one that's not big enough, it already helps a lot if you put only the "chainstate" directory on it.
You could also complete the download on a normal PC, then transfer the data directory to the Raspberry Pi when it's done.

[bob123]

 TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                        
 1686 be/4 pi       1087.31 K/s    0.00 B/s  0.00 % 88.99 % bitcoin-qt [bitcoin-msghand]

The bottleneck definitely is the hard drive.


If you don't want to get a SSD or don't want to use another computer to sync, you could still increase the syncing speed by increasing the database cache (dbcache parameter).

With 4GB of ram, 2GB dbcache would be a good value.
Either put it into the config file or start it with the parameter:

Code:

-dbcache=2048

[bagpuss]

Thanks. How can the iops be so far off the stated 120 Mbps capability - 1Mbps is surely not the top of this drives performance?

https://www.amazon.com/gp/product/B00TKFEE5S/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

[ABCbits]

Thanks. How can the iops be so far off the stated 120 Mbps capability - 1Mbps is surely not the top of this drives performance?

https://www.amazon.com/gp/product/B00TKFEE5S/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

120 MB/s is maximum speed (sequential speed), not random speed (which is far lower than sequential speed)

See https://stackoverflow.com/a/2471684 if you need detailed technical explanation

[Carlton Banks]

120 MB/s is maximum speed (sequential speed), not random speed (which is far lower than sequential speed)

indeed. very few workloads can actually make use of the sequential maximum speed, most are either random reads, or a mixture of random reads and writes. Not least Bitcoin, the blockchain is written sequentially, but you cannot possibly hope to verify the data at 120MB/s, lol (the Bitcoin devs have said that they've tested Bitcoin in "ideal" conditions with high end hardware, and cannot get blocks to verify faster than 50MB/s, although that was maybe 1 year ago).

The major disk performance bottleneck is computing the chainstate, which is a much more random workload than the blockchain, and what slows down anything that's RAM or disk speed restrained (such as a rasPi). Bitcoin chainstate (i.e. UTXO set) is constantly doing short burst of reads from the blocks and writes to the chainstate database. SSDs are best for that kind of usage pattern.



Make the chainstate (and the whole blockchain syncing) go faster thusly:

• set dbcache= to at least 1GB (default is 450MB if you don't set it)
• put your chainstate folder on an SSD, make a symlink to it in your .bitcoin directory (use the ln -s command)
• put your swapfile on an SSD, make it equal or double your RAM, then disable the swapfile on the SD card (only if you're running from an SD card, of course)

you can even put the whole /dev/mmcblk0p2 partition (i.e. root partition) on the SSD, and leave /boot (i.e. /dev/mmcblk0p1 partition) on the SD card. But the biggest performance gains will come from the 3 steps above.

[seoincorporation]

TOP
Tasks: 150 total,   1 running, 149 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.4 us,  2.0 sy,  0.0 ni, 74.0 id, 19.5 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :   3906.0 total,     66.9 free,   1175.5 used,   2663.6 buff/cache

You could drop the cache ram with the next command as Root:

Code:

sync && echo 3 | sudo tee /proc/sys/vm/drop_caches

This way you clean the buff/cache ram and get memory fo somer other new tasks.

source: https://www.commandlinefu.com/commands/view/9509/clear-filesystem-memory-cache

And LoyceV is right, an SSD would help a lot  

[adroitful_one]

Which version of the Raspberry Pi are you using? If it's not a newer one with usb 3.0, that will really slow you down. I was mining burst with an 8 gb external on my solar in my camper just because I had everything to do it and wanted to play with it. It took several minutes to read the thing. So, if at all possible, upgrade that to a usb 3.0 and it will help tremendously.

Edit:Nevermind.. Seen the Pi 3 thing. I would use a desktop to sync the wallet first if at all possible. Should be fine after that.

[Carlton Banks]

Which version of the Raspberry Pi are you using? If it's not a newer one with usb 3.0, that will really slow you down. I was mining burst with an 8 gb external on my solar in my camper just because I had everything to do it and wanted to play with it. It took several minutes to read the thing. So, if at all possible, upgrade that to a usb 3.0 and it will help tremendously.

it's not so much USB 3 as the new disk protocol that makes the Pi4 better.

USB 3 or 2 on their own only lets disks queue up 1 instruction simultaneously, whereas this new thing allows multiple disk commands to be sent in parallel over USB. You need a SATA hub (or an external drive) that talks the new protocol though, otherwise it won't work. any old sata disk will work over the protocol though, it's backwards compatible (at least back as far as SATA-II, you'd think, does anyone still have a SATA-I disk that works ? )

trouble is, I forgot the name of this new USB disk protocol. UFA? totally forgot, sorry everyone