How to initially sync Bitcoin as fast as possible

[cryptoworld99]

If you have >4GB ram, consider increasing your dbcache setting.

If you store the blockchain on HDD. Stop the bitcoin client - defragment - start the bitcoin client again. You will be surprised by the difference!

[DaveF]

If you have >4GB ram, consider increasing your dbcache setting.

If you store the blockchain on HDD. Stop the bitcoin client - defragment - start the bitcoin client again. You will be surprised by the difference!

If you read above, that is part of the test. :-)
Start stock then increase the dbcache but want to get the I downloaded and installed this app and ran it experience 1st.
But it does kind of makes you wonder why they don't do some probing on the install and either create the conf file or at lest suggest one with some settings.

Stay safe.

-Dave

And so the 1st test begins:

2020-05-13T22:44:18Z UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 version=0x00000001 log2_work=33.000022 tx=2 date='2009-01-09T02:54:25Z' progress=0.000000 cache=0.0MiB(1txo)

-Dave

And it's done

2020-05-18T23:20:09Z UpdateTip: new best=00000000000000000007d614c0b374fb92737b358386cfe10318d8d99702c02d height=630881 version=0x20000000 log2_work=91.961602 tx=531173082 date='2020-05-18T23:20:04Z' progress=1.000000 cache=307.0MiB(2275078txo) warning='68 of last 100 blocks have unexpected version'

So 5 days start to finish.

Going to re-do this test while allowing the machine access to the internet instead of just the local machines. Want to make sure that that it it about the same amount of time. Will only do the time to block 630,881 to keep everything consistent.

See you the end of the week :-)
-Dave

[aliashraf]

So 5 days start to finish.

Congratulations! You are the champion of dealing with stupid nightmares 

[DaveF]

So 5 days start to finish.

Congratulations! You are the champion of dealing with stupid nightmares 

Feel free to go to github and open some issues.
Or clone it and make your own version of core

What I decided to do here after yogg started the thread was take a few of what I felt were some of the more common scenarios and see how long they take to sync.

AND then make 1 change to the conf file (dbcache) and see how long it takes.

Feel free to do your own testing, just remember much like the one run I just did and am re-doing for verification when you make the change you NEED a baseline.

-Dave

[aliashraf]

@DaveF

I Didn't mean to be disrespectful bro, just wanted to remind you that it is not how things work in the software world. Once you encounter  a stupid situation you need to ask "Why should I be here, facing such a nightmare, after all?"

Sorry, if I said it in the worst way ever.

[DaveF]

@DaveF

I Didn't mean to be disrespectful bro, just wanted to remind you that it is not how things work in the software world. Once you encounter  a stupid situation you need to ask "Why should I be here, facing such a nightmare, after all?"

Sorry, if I said it in the worst way ever.

I did not take offense, it's just that yeah it's broken and as of now we have a limited amount of tools to fix it.

So I figured this is a good test to do with what we have. And, it's somewhat not time intensive. I can start it and walk away. Even if I had not checked in before I went to sleep I could have had the answer in the morning.

-Dave



 

[HCP]

5 Days? That's not so terrible. Not exactly great, but not super terrible.

Just to confirm... that test was using Windows 10 as the OS wasn't it? you're not using ubuntu or some other flavour of Linux are you?

[DaveF]

5 Days? That's not so terrible. Not exactly great, but not super terrible.

Just to confirm... that test was using Windows 10 as the OS wasn't it? you're not using ubuntu or some other flavour of Linux are you?

Yeah, Win 10. I am going for the "average" user experience. Figure most people are running home PCs and are going to install core on that.
*nix people are probably going to be more customized installs so it's a bit tougher to get a basline / tweak settings.

Just my view.

-Dave

[DaveF]

Since you mentioned Windows 10, i wonder if you've disabled it's automatic update since it could intervene with the benchmark. Sometimes it's quite annoying since it takes all bandwidth.

No automatic updates were running.

But it was fully updated / patched to the day I started the sync. Which was why I started a bit later in the day on the 13th. Patch Tuesday was the day before and I wanted to make sure it was fully done with updates before I began.

One of the next tests is definitely going to hit over June 9th, so running it again after the updates will give an interesting bit of data of how much they really kill performance.
 
achow101 came in and cleaned up a bit, I will just do the updates in this post.

-Dave


So the 2nd sync just finished to the same block as the last one. Same as before except for instead of just the 4 local nodes I let it connect to the the net.
Came up about 2 hours less. Over 5 days that seems to be a small enough difference that it does not matter.

The SSD has not arrived yet, and I am not motivated to drive to the office to add more ram the the machine I am going to leave it at 8GB installed but I will set the db cache for 4GB. Going to do it with the GUI not even the conf file to more imitate how a user might do it.

Lets see how it plays out.



And the results for the 4GB db cache are in.
A bit under 3 days. So a major improvement. And since most PCs these days are coming with 8GB of RAM having that at a high setting for a few days for an initial sync is probably not going to be that bad for most people.

Start:
2020-05-23T21:42:20Z UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 version=0x00000001 log2_work=33.000022 tx=2 date='2009-01-09T02:54:25Z' progress=0.000000 cache=0.0MiB(1txo)

End:
2020-05-26T18:17:06Z UpdateTip: new best=00000000000000000007d614c0b374fb92737b358386cfe10318d8d99702c02d height=630881 version=0x20000000 log2_work=91.961602 tx=531173082 date='2020-05-18T23:20:04Z' progress=0.995235 cache=3079.8MiB(21238640txo)

Shut it down, added 8GB of RAM, set the dbcache to 8192, deleted the AppData\Roaming\Bitcoin folder and started again.
Lets see how it goes.


So the 8GB dbcache with 16GB of ram really made it faster. A bit under 24 hours start to finish.

2020-05-26T19:53:45Z UpdateTip: new best=00000000839a8e6886ab5951d76f411475428afc90947ee320161bbf18eb6048 height=1 version=0x00000001 log2_work=33.000022 tx=2 date='2009-01-09T02:54:25Z' progress=0.000000 cache=0.0MiB(1txo)

020-05-27T19:26:51Z UpdateTip: new best=00000000000000000007d614c0b374fb92737b358386cfe10318d8d99702c02d height=630881 version=0x20000000 log2_work=91.961602 tx=531173082 date='2020-05-18T23:20:04Z' progress=0.994597 cache=4882.4MiB(31913169txo)


As of now the SSD STILL has not come it. So testing paused for a bit.....

-Dave

[lcharles123]

Answering the OP I just found a way to sync as fast as possible using RAMdisk, it will work even if you has a slow HDD or NAS. RAMdisk is faster than SSD but requires some complexity, like linking files

I observed which files are written frequently on sync, and I saw that are only the blk and rev dat ones on blocks folder.
bitcoind has the -blocksdir option, to specify the blocks folder, you just need to point it to a folder on RAMdisk.
I write the following script that check for completed blk and rev files on ramdisk/blocks folder and move them to persistent storage, and create a link to them, because these .dat files are used to build chainstate after completing initial sync (or successfully shutting down before it).

With this strategy the initial sync will have only network speed as bottleneck. At least for me, I have at most 10 Mb/s.

I am using a single core notebook from 2011 with debian 11, 16GB of RAM and two 500GB HDDs merged using mergerfs.
mergerfs was always using CPU before, and with this strategy, it use 0 almost all time, only when a dat file are completed.
I will post the results here after finishing the initial sync

Code:

#!/bin/bash


# first of all create and mount a ramdisk, i.e.
# sudo mkdir -p /mount/point/folder -m 755
# sudo mount -o size=4G -t tmpfs tmpdir /mount/point/folder
# ajust the variables on this script

# point -blocksdir to ramdisk
# link all files from persistent storage 
# targeting blocks folder on ramdisk
# run this script on background to periodically flush 
# blk and rev files to persistent storage and link them
# make sure your pc doesnt run out of power
# I have a nobreak for that
# You can do a checkpoint by stoping bitcoind, and making a
# copy of chainstate and index folder and some blk and rev files
# After finishing up the sync,
# just copy all the real files from ramdisk to persistent storage

# get the blocks starting from BNBER
# 6 will let at least 6 blk and 6 rev files on ramdisk
BNBER=6
ACTUALDIR=/home/alice/bcdata/BTC/blocks
RAMDISK=/home/alice/temp/BTC/blocks

mkdir -p $RAMDISK
while true
do
    # iterate only over regular files, not folders nor links
    for blkfile in $(find $RAMDISK/blk*.dat  -type f -printf "%f\n" | sort -r | tail -n +$BNBER)
    do
        mv $RAMDISK/$blkfile $ACTUALDIR/$blkfile
        ln -s $ACTUALDIR/$blkfile $RAMDISK/$blkfile
    done
    for blkfile in $(find $RAMDISK/rev*.dat  -type f -printf "%f\n" | sort -r | tail -n +$BNBER)
    do
        mv $RAMDISK/$blkfile $ACTUALDIR/$blkfile
        ln -s $ACTUALDIR/$blkfile $RAMDISK/$blkfile
    done
    sleep 10m
done

[WhyFhy]

Answering the OP I just found a way to sync as fast as possible using RAMdisk, it will work even if you has a slow HDD or NAS. RAMdisk is faster than SSD but requires some complexity, like linking files

I observed which files are written frequently on sync, and I saw that are only the blk and rev dat ones on blocks folder.
bitcoind has the -blocksdir option, to specify the blocks folder, you just need to point it to a folder on RAMdisk.
I write the following script that check for completed blk and rev files on ramdisk/blocks folder and move them to persistent storage, and create a link to them, because these .dat files are used to build chainstate after completing initial sync (or successfully shutting down before it).

With this strategy the initial sync will have only network speed as bottleneck. At least for me, I have at most 10 Mb/s.

I am using a single core notebook from 2011 with debian 11, 16GB of RAM and two 500GB HDDs merged using mergerfs.
mergerfs was always using CPU before, and with this strategy, it use 0 almost all time, only when a dat file are completed.
I will post the results here after finishing the initial sync

Code:

#!/bin/bash


# first of all create and mount a ramdisk, i.e.
# sudo mkdir -p /mount/point/folder -m 755 alice
# sudo mount -o size=12G -t tmpfs tmpdir /mount/point/folder
# ajust the variables below

# point only blocks folder to ramdisk, i.e. -blocksdir=/mount/point/folder
# use it on first run
# run this script BEFORE starting bitcoind 
# otherwise move some last blk and rev files to ramdisk
# symlink all blk and rev files from ramfs -> hdd/ssd
# make sure your pc doesnt run out of power
# I have a nobreak for that
# with this setup, the sync will be limited by others factors than media speed
# shutdown bitcoind with ctrl + c, and move all non link elements to hdd/ssd

# get the blocks starting from BNBER
# this will let at least 7 blk and 7 rev files on ramdisk
BNBER=7
ACTUALDIR=/home/alice/bcdata/BTC/blocks
RAMDISK=/home/alice/temp/BTC/blocks
set -e

while true
do
    # iterate only over regular files excluding links and the BNBER most recent ones
    for blkfile in $(find $RAMDISK/blk*.dat  -type f -printf "%f\n" | sort -r | tail -n +$BNBER)
    do
        mv $RAMDISK/$blkfile $ACTUALDIR/$blkfile
        ln -s $ACTUALDIR/$blkfile $RAMDISK/$blkfile
    done
    for blkfile in $(find $RAMDISK/rev*.dat  -type f -printf "%f\n" | sort -r | tail -n +$BNBER)
    do
        mv $RAMDISK/$blkfile $ACTUALDIR/$blkfile
        ln -s $ACTUALDIR/$blkfile $RAMDISK/$blkfile
    done

    sleep 10m
done

I wouldnt help yogg with anything if I where you.
(Yogg hasn't been around since he swept all the keys and coin from coldkey wallets from the collectibles section)
And why are you necro posting? This topic is years old now.

[jojo69]

This is still a valid avenue of inquiry.

[satscraper]

~

Interesting solution for IBD, never encountered it before.

Having in mind 16 Gb RAM, what  size should be assigned to RAMdisk ? And  how to set up  dbcache in this case, dbcache=1/4 RAMdisk or 1/4(RAM - RAMdisk).

 And another question. What is gain in comparison with  M.2 NVme SSD?

[lcharles123]

~

Interesting solution for IBD, never encountered it before.

Having in mind 16 Gb RAM, what  size should be assigned to RAMdisk ? And  how to set up  dbcache in this case, dbcache=1/4 RAMdisk or 1/4(RAM - RAMdisk).

 And another question. What is gain in comparison with  M.2 NVme SSD?

I assigned 10GB for dbcache, and 4GB for RAMDISK. Not needed big RAMdisk because it will store only a few blk and rev files, the rest will be links of that files to persistent storage.
There are many access to latest blk and rev files during syncronization.
The writes on chainstate are made up periodically or if you interrupt the sync, so this work will be amortized by download activity.
I said gain because generally RAM are faster than any SSD. But it depends on the machine you are using too. I do not have machines with M.2 connector to test.

I just synced the blockchain with this setup, have to reindex from 49% because the system crashed, since I am using 2x 500GB HDDs on a docking station connected though USB to notebook. The sync from scratch was going well with about 15% a day, but the reindex process was really slow, because i have put only index folder and chain state on RAMdisk, but not helped much.
The script needs some adjusts, removing set -e, because it not needed since if the script exits, the RAMdisk will be filled up and the bitcoind will stop after that.

So instead re-indexing, downloading from scratch is much more faster, and you can make checkpoints by interrupting the program at 30% for example, and saving chainstate and index folders and some of the recent blk and rev files. To put it back in the case of any critical event occurs further on the process.

Beginning of 2023 I was trying to run a full-node, but it doesn't worked because some problems with mergerfs. The initial sync long lasted about 4 months. But now with this strategy, i can conclude it in the same machine in less than one week.

...
I wouldnt help yogg with anything if I where you.
(Yogg hasn't been around since he swept all the keys and coin from coldkey wallets from the collectibles section)
And why are you necro posting? This topic is years old now.

I searched in this forum using many terms about "speeding up initial sync" and all the time it returned this thread as the only relevant one. But it lacks a satisfactory answer, so I just give one.

[satscraper]

The sync from scratch was going well with about 15% a day,

Thanks,"15% a day" speaks volumes.

My gained experience evidences that IBD involving 16 Gb RAM machine and external SSD (870 EVO Samsung) goes much faster. 4 weeks ago it took less than 36 hours to download full blockchain from the scratch. The properly tuned bincoin.conf (dbcache=1/4 RAM and blocksonly=1) facilitates IBD.