Running a node dumb question

[Eternad]

PS; I don't want to spam the challenge thread so I decided to create this thread for my personal question as my curiosity started to go deeper on the technical aspect of running a node.

• I notice that blocks are being downloaded while my Bitcoin core wallet is syncing in the form of .dat file which is the same format as my wallet on Electrum. Can anyone explain to me in ELI5 what is this format and how block files differ from my wallet files while they have the same format?
• If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.
• Is it possible to skip certain blocks like blocks around 2009 to 2012 and directly download 2013 onwards when I want to start running a Node since I find it inefficient to download everything from the start while the pruning process removes previously downloaded blocks and continues the sync process using the onwards downloaded blocks without the need of the old block.
• Lastly, What I'm achieving in running a node? Am I helping the blockchain to become more decentralized? 

I'm completely a noob but I want to explore this so please be gentle in answering my question. Please always use ELI5 format so that I can grasp your idea.

[1714223808]

1714223808

[1714223808]

1714223808

[1714223808]

1714223808

[ABCbits]

• I notice that blocks are being downloaded while my Bitcoin core wallet is syncing in the form of .dat file which is the same format as my wallet on Electrum. Can anyone explain to me in ELI5 what is this format and how block files differ from my wallet files while they have the same format?

They just happen to have same extension, the format of those files are different. They have different format and usually Electrum create wallet without extension. If you actually refer to Bitcoin Core's wallet.dat, they just happen to have same extensions.

• If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.

I'm not fully understand your question, but you need to copy whole folder (which also contain other stuff such as "chainstate" folder) rather than only block files.

• Is it possible to skip certain blocks like blocks around 2009 to 2012 and directly download 2013 onwards when I want to start running a Node since I find it inefficient to download everything from the start while the pruning process removes previously downloaded blocks and continues the sync process using the onwards downloaded blocks without the need of the old block.

Theoretically it's possible, assuming you have programming skill to modify Bitcoin Core (or other full node) software. But that means you're skipping some verification process which is point of full node.

[un_rank]

• Lastly, What I'm achieving in running a node? Am I helping the blockchain to become more decentralized?  

You get to improve your own privacy and the decentralization of the network. Just as with mining, the more divers nodes we have keeping a record of their own mempool, the more decentralized blockchain information is, making the information immutable.

- Jay -

[Eternad]

• If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.

I'm not fully understand your question, but you need to copy whole folder (which also contain other stuff such as "chainstate" folder) rather than only block files.

I back up that too including all the files on the file directory for Bitcoin core. I manually pasted all the files that I previously downloaded to the new file directory of Bitcoin Core after I reinstalled the apps. Still, the synchronization of "headers(not sure of the exact word)" is so slow that it almost didn't progress anymore so I just forgot my previous progress and started fresh again.

[ABCbits]

I'm not fully understand your question, but you need to copy whole folder (which also contain other stuff such as "chainstate" folder) rather than only block files.

I back up that too including all the files on the file directory for Bitcoin core. I manually pasted all the files that I previously downloaded to the new file directory of Bitcoin Core after I reinstalled the apps. Still, the synchronization of "headers(not sure of the exact word)" is so slow that it almost didn't progress anymore so I just forgot my previous progress and started fresh again.

If it stuck during syncing block header (NOT block), it's probably due to internet connection. There's no need to start fresh again (as in download blockchain from beginning).

[NotATether]

If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.

Is it possible to skip certain blocks like blocks around 2009 to 2012 and directly download 2013 onwards when I want to start running a Node since I find it inefficient to download everything from the start while the pruning process removes previously downloaded blocks and continues the sync process using the onwards downloaded blocks without the need of the old block.

Bitcoin nodes need to assemble the global UTXO set while they are syncing. This is basically the outputs of every bitcoin address that have not been spent yet. In order to make such a list, it needs to read the blocks in order, without skipping anything so that it can verify that there are no invalid transactions inside a block (which should never happen in theory but you can always edit the .dat file to make such a corruption). That is also why it is reading the blocks even after you save them.

[BlackHatCoiner]

• If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.

In case you still find it strange, it is verification which takes most of the time. Download speed is pretty neat in almost every part of the world. With an average 10 MB/s, it would take you about just a day to download the blockchain. However, verification might even take a week, depending on your computer resources.

Lastly, What I'm achieving in running a node? Am I helping the blockchain to become more decentralized?

You know the truth.  You need no third parties to figure out how much money you have. It also a privacy upgrade, since you let no third party scrutinize your Bitcoin activity.

[khaled0111]

I notice that blocks are being downloaded while my Bitcoin core wallet is syncing in the form of .dat file which is the same format as my wallet on Electrum. Can anyone explain to me in ELI5 what is this format and how block files differ from my wallet files while they have the same format?

The file extension doesn't matter much, it could be .txt or anything else. What matters is the data stored on those files. blk.dat files contain data related to the blocks and transactions that have been mined while wallet.dat contains data related to your wallet.

If I have the record of all the block files, why is the syncing process still manually reading all my current blocks which is the same speed as when I downloaded them manually? I did an experiment about this in which I saved all my downloaded blocks and uninstalled my Bitcoin core then I manually put the block file in the directory folder when I reinstalled the new app.

The blockchain, as the name suggests, is a chain of block where each block is connected to the previous one. To verify the validity of the blocks and transactions you have downloaded, your node needs to verify the validity of each of them going back to the genesis block (first block ever mined).

Is it possible to skip certain blocks like blocks around 2009 to 2012

I believe not for the reason explained above.

Lastly, What I'm achieving in running a node? Am I helping the blockchain to become more decentralized?

Yes, by running a full node you contribute to the decentralization of the network and by using bitcoin core as your wallet or connecting your wallet to your own full node you improve your anonymity.

[Eternad]

I appreciate all the replies and learnings to this uncharted territory for me.

@NotATether I will use this thread to inquire about my problem with your running a node challenge.

Bitcoin core syncs very slowly when it reaches 15% progress while Bitcoind performs at a usual speed. I'm not sure what is the problem because my RAM and CPU usage were still the same when I started this challenge including the storage on the hard disk since I used the prune command that you suggested on the challenge thread.

the screenshot below is my current stats


it doesn't have any good progress even though I left it overnight running.

[khaled0111]

Bitcoin core syncs very slowly when it reaches 15% progress while Bitcoind performs at a usual speed.

There is no problem and syncing becoming slower is completely normal.
As you know, your node verifies the validity of each block it downloads and the transactions in it. Currently, your node is synced up to 2016 (according to the screenshot you posted) and prior to that date bitcoin wasn't so popular so blocks back then we're either empty or had few transactions on them therefore verifying their validity (syncing) doesn't take too long.

[Eternad]

Bitcoin core syncs very slowly when it reaches 15% progress while Bitcoind performs at a usual speed.

There is no problem and syncing becoming slower is completely normal.
As you know, your node verifies the validity of each block it downloads and the transactions in it. Currently, your node is synced up to 2016 (according to the screenshot you posted) and prior to that date bitcoin wasn't so popular so blocks back then we're either empty or had few transactions on them therefore verifying their validity (syncing) doesn't take too long.

Gotcha. This makes sense a lot. Thank you, I'm just a little bit carried away by how fast it was synced on the first day it almost completed 15% so I thought that it would be inconsistent speed. So this means that the 2016 onwards block content is much heavier in file size due to the volume of transactions.

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

Follow-up question, What is the data shown by the Bitcoind window? Is it related to the syncing process on Bitcoin core because my Bitcoin core is progressing even though Bitcoind is closed?

[pooya87]

.dat file which is the same format as my wallet on Electrum.

Last I checked Electrum wallet file didn't have any extension and it was not a dat file either. It's plain text using JSON format.

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

You should be able to increase your cache size and speed up the verification process. Set the dbcache in your bitcoin.conf file to 8 GB (since you have 16 GB memory, I believe half is good).

Code:

dbcache=8000

[satscraper]

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

Set blocksonly=1 in bitcoin.config file. As it was mentioned by gmaxwell  it may result in  "reduced the node's bandwidth usage by 88%." After completing IBD you may set default , i.e.   blocksonly=0

[ABCbits]

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

Set blocksonly=1 in bitcoin.config file. As it was mentioned by gmaxwell  it may result in  "reduced the node's bandwidth usage by 88%." After completing IBD you may set default , i.e.   blocksonly=0

88% was obtained from Bitcoin Core 0.12 though. Since then, Bitcoin Core implement compact block which allow faster block propagation while reducing node's bandwidth usage (since node doesn't have to send transaction data (standalone and in block). And i recall node which in sync progress doesn't obtain TX mempool, although i can't find source of that statement.

[satscraper]

88 % was obtained from Bitcoin Core 0.12 though. Since then, Bitcoin Core implement compact block which allow faster block propagation while reducing node's bandwidth usage

You are correct,  even though it is  not 88% nowadays but  "absolute minimum"  matters. In  fact , I had a chance to appreciate blocksonly=1 setting   a  few months ago at my node's IBD.

[Eternad]

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

You should be able to increase your cache size and speed up the verification process. Set the dbcache in your bitcoin.conf file to 8 GB (since you have 16 GB memory, I believe half is good).

Code:

dbcache=8000

How can I know if the "dbcache=8000" setup is already working? The Bitcoind windows process too much data quickly which makes it hard to read the initial summary when I open it.

why this code doesn't have "#" in the beginning compared to other commands listed on my bitcoin.conf. Lastly, what's the difference between Bitcoin.conf file located on the file path in which I installed my Bitcoin core and Bitcoin.conf that I created on C:\Users\YourUserName\AppData\Roaming\Bitcoin path when I input the pruned command? Thanks for your guidance sir.

[satscraper]

Is there a way to speed this up a little bit using the remaining unused RAM and CPU?

You should be able to increase your cache size and speed up the verification process. Set the dbcache in your bitcoin.conf file to 8 GB (since you have 16 GB memory, I believe half is good).

Code:

dbcache=8000

How can I know if the "dbcache=8000" setup is already working? The Bitcoind windows process too much data quickly which makes it hard to read the initial summary when I open it.

why this code doesn't have "#" in the beginning compared to other commands listed on my bitcoin.conf. Lastly, what's the difference between Bitcoin.conf file located on the file path in which I installed my Bitcoin core and Bitcoin.conf that I created on C:\Users\YourUserName\AppData\Roaming\Bitcoin path when I input the pruned command? Thanks for your guidance sir.

# is a commenting character. Putting it before the command disables this command.

Regarding the value of dbcache. As a rule of thumb dbcache=1/4 RAM. Thus in 16 GB case the optimal would be dbcache=4096

Read more on dbcache which   essentially acts as write buffer rather than a cache.

[promise444c5]

why this code doesn't have "#" in the beginning compared to other commands listed on my bitcoin.conf.

You should create your bitcoin.conf not .txt  and once it's setted you should see changes
The # makes it a comment jut like writing comments for codes
Comments  don't get to be read in a code just for readability and reference mssg
On

[Eternad]

Bitcoin core syncs very slowly when it reaches 15% progress while Bitcoind performs at a usual speed.

There is no problem and syncing becoming slower is completely normal.
As you know, your node verifies the validity of each block it downloads and the transactions in it. Currently, your node is synced up to 2016 (according to the screenshot you posted) and prior to that date bitcoin wasn't so popular so blocks back then we're either empty or had few transactions on them therefore verifying their validity (syncing) doesn't take too long.

I have a follow-up question to this, why did blocks during 2016 give me a slower sync while Bitcoin transactions during 2017 onwards were much higher due to the demand? I sync faster now after I finished syncing blocks in 2016. I'm already at 85% progress compared to when I posted this which is just 15% sync stuck for a long period.

I'm curious what's with 2016 blocks that cause slow sync?

[Cricktor]

...

You have been given a few meaningful optimization tweaks. Did you apply those and now wonder why sync speed improved?

We can't know how your computer is utilized and thus what all affects the sync process of your Core node.

I once made a test with my Raspi 4B 8GB RAM with a 1TB SATA-USB3 SSD (UASP active) running a Umbrel node: I had to uninstall any app in that Umbrel and thus wiped all Bitcoin Core blockchain data on purpose. I then installed only the Bitcoin Core app and tweaked dbcache to use about half of RAM (4883MB were accepted as dbcache value). The initial blockchain download and sync took (Tor-only for outgoing connections) to my surprise only about 95h on my Raspi 4B (I have stable ~100MBit downstream for my internet connection, you hardly get this speed sustained over Tor all the time; but it's possible this Umbrel node talked to any of my other three local Core nodes that are running in my local network, I haven't excluded this possibility to keep things simple).

All those who complain a Raspi 4B takes weeks to sync, I can reply that with optimized parameters a Raspi 4B is capable of executing an IBD in less than one week with a normal 1TB SATA-USB3 SSD enclosure for data storage.


P.S.
I still have the debug.log of this IBD experiment. I started it on approx. 2023-06-22T18:00:00Z and finished sync to chaintip at block 796033 on 2023-06-26T16:46:31Z. Later bloating of the UTXO set won't make the sync process any easier for low RAM devices.