Installing a Full Node under Linux

[citb0in]

Hi. In order to better understand various contexts, but also to make a small contribution to the bitcoin community, I would like to install a full node. I have an older SBC with ARMv7 CPU @ 1GHz and 2GB RAM with 1TB SATA hard disk (no SSD) which should be sufficient for the first steps.  As far as I read bitcoin core should run on armhf/32bit architecture. The OS used is GNU/Linux Debian 10 (Buster). I want to understand and learn the basics before I reinstall later on a more performant hardware.

However, I run into the following important key question right at the start. On https://bitcoin.org/en/download the version "Bitcoin Core 22.0" is offered for download. By chance I just came across the page https://bitcoincore.org/en/download/, where 23.0 is offered. Why the heck are there two different websites, what's the deal and which one is recommended and why? I would have instinctively used the 23.0 version of bitcoincore.org, but I would still like to know what it is all about.

In the course of this I would also like to get rid of the following question: what is the minimum hardware requirement you would recommend if I want to add Lightning functionality later on? What is the most important hardware criterion, is it RAM, fast disk access times or the speed or bandwidth of the internet connection?

Thanks so far.
citb0in

[1715426071]

1715426071

[jackg]

I think Internet bandwidth and a fast(ish) ping is the most important for hosting lightning network channels but a lot route through tor anyway (as well as just generally ensuring your device keeps functioning fast - so you're not waiting for seconds of lag for something to install).

Your initial sync will take a long time and I think it should be able to install on 32 bit systems but that might slow it down further - with the smaller number of registers 32 bit offers afaik.

[BlackHatCoiner]

However, I run into the following important key question right at the start. On https://bitcoin.org/en/download the version "Bitcoin Core 22.0" is offered for download. By chance I just came across the page https://bitcoincore.org/en/download/, where 23.0 is offered. Why the heck are there two different websites, what's the deal and which one is recommended and why?

Bitcoin.org is not the official site for Bitcoin Core releases. Bitcoincore.org is. You can see the differences between these two versions and decide which to install here:

• 23.0
• 22.0

In the course of this I would also like to get rid of the following question: what is the minimum hardware requirement you would recommend if I want to add Lightning functionality later on?

C-Lightning is lightweight enough. In my RPi 4 I'm using about 1.5 GB memory, and it works fine.

[citb0in]

So I'll stick with bitcoincore.org and 23.0 version. Thanks!

[LoyceV]

I have an older SBC with ARMv7 CPU @ 1GHz and 2GB RAM with 1TB SATA hard disk (no SSD) which should be sufficient for the first steps
~
In the course of this I would also like to get rid of the following question: what is the minimum hardware requirement you would recommend if I want to add Lightning functionality later on? What is the most important hardware criterion, is it RAM, fast disk access times or the speed or bandwidth of the internet connection?

I can't answer LN-questions from experience, but for Bitcoin Core itself, 2 GB and HDD is going to be slow. I did that years ago on an Atom netbook, and eventually gave up. Any chance you can upgrade the memory, or at least use a (very small) SSD for the chainstate directory?

[citb0in]

Hi LoyceV and thanks for your feedback and suggestion. Can you explain more detailled please, what exactly is the issue when running bitcoincd on a usual SATA (magnetic) disk vs. SSD ? Is access time crucial for bitcoind ? I could install and run it on another host I have, it's an Intel NUC with 16GB of RAM and 512GB NVMe disk but then I'd need to use prune=262144 for limiting the disk space to about 256GB for bitcoin-core.

[LoyceV]

Can you explain more detailled please, what exactly is the issue when running bitcoincd on a usual SATA (magnetic) disk vs. SSD ?

It's slow
When syncing, Bitcoin Core verifies all blocks. I've tested it in the past, and putting chainstate on SSD gives a significant performance improvement. Using 4 GB as dbcache helps a lot too, but you'll need more RAM for that.

Is access time crucial for bitcoind ?

It's not crucial, but makes a big difference.

I could install and run it on another host I have, it's an Intel NUC with 16GB of RAM and 512GB NVMe disk but then I'd need to use prune=262144 for limiting the disk space to about 256GB for bitcoin-core.

If at all possible, plug in the SATA disk and run chainstate from the NVMe. Add 4096 dbcache and syncing should be done in half a day. Then move everything to your other system.

[citb0in]

Do I interpret correctly from this that the disk performance and thus your suggestion is based on the fact that it is only decisive for the IBD? As soon as the IBD is completed, the hard disk performance plays a rather subordinate role?

[LoyceV]

As soon as the IBD is completed, the hard disk performance plays a rather subordinate role?

Correct. One block per 10 minutes isn't very demanding, and (as far as I know) uploading blocks requires mostly linear reading from disk.

[citb0in]

Alright, I don't have any problem waiting for weeks until the sync is finished so I'll go that way nevertheless I'd like to know: in case of running a pruned node (let's say with pruned=10000) do I have any restrictions when using this node for mining ?

[Sarah Azhari]

What is the most important hardware criterion, is it RAM, fast disk access times or the speed or bandwidth of the internet connection?

1GHz and 2GB RAM with 1TB SATA hard disk is too slow in middle of step, 50% downloading block will have trouble slow sync and stuck. I ever used core 2.5 GHz and 4GB RAM with 1TB SATA hard disk, the trouble begin on middle block. I suggest to you upgrade all spec than wasted

[citb0in]

Let's say I have computer1 which is low-spec and computer2 which has better hardware and thus is much faster. I have installed bitcoin-core 23.0 on both of them and started bitcoind, the setting on both computers is to run a pruned node with 4 GiB. Computer2 was very fast as I utilized a lot of db cache because of the high available RAM on that computer2. It finished already and is up-to-date. Side note: I wonder why I have two big directories under ./bitcoin/

5.0 G   chainstate
4.0 G   blocks

I'd have expect to see only 4GB of data because I configured prune=4096 in my bitcoin.conf file.

Now I stopped bitcoind on both computers and I would like to copy the necessary files from computer2 to computer1 to save time and bandwidth for computer1. My goal is to bring up both nodes and run them simultaneously.

How should I do that, which folders are necessary to copy over to computer1 ? I assume it's not good to copy the whole .bitcoin directory over to the slower computer1 cause I think computer2 might use some sort of fixed data that are relevant only to computer2 (maybe some hardware or cookie IDs, or whatever I am not aware of...) Will it be enough to copy the folders chainstate and blocks, or maybe only blocks folder is enough ?

please enligthen me   thank you

EDIT: I found some helpful information here and if understood correctly I was on the right path. I guess I will try to copy only those two mentioned folder chainstate and blocks and keep the rest of the folders/files content on each installation to their own.

However, anyone please give me a reply to the above question about the high folder sizes ?

[DireWolfM14]

~

Yes, the chainstate and the blocks directories will transfer from one node to another.  However, you'll still need to synchronize the new install on computer2, and if you're using a HDD, that will be slow anyway.  Once upon a time I tried to install core on an RPi with a HDD plugged into one of the USB ports.  IIRC it took over three weeks to synchronize, and that was with another local node specified by addnode, so download speeds weren't an issue.  Also, if I recall the blockchain was around 360GB at the time, quite a bit less than it is right now.

Another action that can lead impatience with a slow HDD is importing addresses.  Read speeds on HDDs are faster than wright speeds, so it's less of an issue, but it's still noticeably slower than a node running on a SSD.

[LoyceV]

I wonder why I have two big directories under ./bitcoin/

5.0 G   chainstate
4.0 G   blocks

Chainstate is the one I mentioned earlier: it gets a lot of write access during syncing:

the blocks directory contains the actual blocks. The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

Will it be enough to copy the folders chainstate and blocks, or maybe only blocks folder is enough ?

I think that's enough, but I also think files like mempool.dat and peers.dat won't hurt to add. So I'd just copy everything except for your wallet.

Back to your original question:

to make a small contribution to the bitcoin community, I would like to install a full node.

Why did you switch to a pruned node? I've tested it, and a pruned node can indeed upload more than it downloads, but it's not much in total. To compare: my full node uploads about 40 GB per day.

[n0nce]

However, I run into the following important key question right at the start. On https://bitcoin.org/en/download the version "Bitcoin Core 22.0" is offered for download. By chance I just came across the page https://bitcoincore.org/en/download/, where 23.0 is offered. Why the heck are there two different websites, what's the deal and which one is recommended and why?

Bitcoin.org is not the official site for Bitcoin Core releases. Bitcoincore.org is. You can see the differences between these two versions and decide which to install here:

• 23.0
• 22.0

In general, I would always refer to the (very prominently placed) link on top of this forum, or go directly to GitHub with its easy to memorize URL: https://github.com/bitcoin/bitcoin

In the course of this I would also like to get rid of the following question: what is the minimum hardware requirement you would recommend if I want to add Lightning functionality later on?

C-Lightning is lightweight enough. In my RPi 4 I'm using about 1.5 GB memory, and it works fine.

Exactly; don't bother with lnd on a low-power system, get Core Lightning right away.

You may also be able to use large parts of my FutureBit Apollo BTC full node install guide. It is just an Orange Pi 4.
https://bitcointalk.org/index.php?topic=5401747

One key difference is that you're running 32-bit ARM instead of 64-bit ARM, and the path to your external drive will be different. You may also be running a different OS with different package manager / package names.
But in general, you can follow the steps as outlined there.

I have an older SBC with ARMv7 CPU @ 1GHz and 2GB RAM with 1TB SATA hard disk (no SSD) which should be sufficient for the first steps
~
In the course of this I would also like to get rid of the following question: what is the minimum hardware requirement you would recommend if I want to add Lightning functionality later on? What is the most important hardware criterion, is it RAM, fast disk access times or the speed or bandwidth of the internet connection?

I can't answer LN-questions from experience, but for Bitcoin Core itself, 2 GB and HDD is going to be slow. I did that years ago on an Atom netbook, and eventually gave up. Any chance you can upgrade the memory, or at least use a (very small) SSD for the chainstate directory?

Very good point! The initial block download will be an issue. It is possible to do the IBD on a powerful desktop PC, gracefully shut down Bitcoin Core and then remove the HDD and put it into the SBC node.

See my experience below. With 4GB of RAM and HDD, it took almost a week for the first 50% (and it's not linear growth), as soon as I added 4GB of RAM, it finished in around a day. It would have taken multiple more weeks to finish with just the 4GB, unfortunately.

Alright, I don't have any problem waiting for weeks until the sync is finished so I'll go that way nevertheless I'd like to know: in case of running a pruned node (let's say with pruned=10000) do I have any restrictions when using this node for mining ?

You don't normally need to run a full node for mining. You just choose a solo (or regular) pool like https://kano.is/ and use Stratum protocol.

[citb0in]

I thank you for the helpful responses so far. I have already read your HowTo, I have set up bitcoin-core very similar. At the moment I still have the following questions:

I had installed bitcoin-core on computer1 and computer2 in the same way. Computer1 is the slow armbian and computer2 is the fast computer with NVMe disk. Both nodes shut down before I got to work. I thought I had enough space on the fast computer2 so that I could download the whole blockchain, but unfortunately I was wrong and the space is not enough for the full blockchain. So I decided to run computer2 as a pruned node with the same settings as computer1. Both use the same configuration except that I could increase the dbcache on computer2. Computer2 was very quickly done with the IDB. Then I stopped bitcoin-core on computer2 again, so it was both nodes offline. Then I copied the folders "blocks" and "chainstate" from the fast computer2 to the slow armbian computer1. When I then started computer1 I got the error message that it could not find a block file and bitcoin-core was shut down again. This surprised me so I checked it out.

To my surprise I found out that computer2 used following folder structure for its block files:

Code:

bitcoin@computer2:~/.bitcoin/blocks$ ls

blk03189.dat  blk03198.dat  blk03207.dat  blk03216.dat  rev03196.dat  rev03205.dat  rev03214.dat
blk03190.dat  blk03199.dat  blk03208.dat  index         rev03197.dat  rev03206.dat  rev03215.dat
blk03191.dat  blk03200.dat  blk03209.dat  rev03189.dat  rev03198.dat  rev03207.dat  rev03216.dat
blk03192.dat  blk03201.dat  blk03210.dat  rev03190.dat  rev03199.dat  rev03208.dat
blk03193.dat  blk03202.dat  blk03211.dat  rev03191.dat  rev03200.dat  rev03209.dat
blk03194.dat  blk03203.dat  blk03212.dat  rev03192.dat  rev03201.dat  rev03210.dat
blk03195.dat  blk03204.dat  blk03213.dat  rev03193.dat  rev03202.dat  rev03211.dat
blk03196.dat  blk03205.dat  blk03214.dat  rev03194.dat  rev03203.dat  rev03212.dat
blk03197.dat  blk03206.dat  blk03215.dat  rev03195.dat  rev03204.dat  rev03213.dat

bitcoin@computer2:~/.bitcoin/blocks$ ls index

001144.ldb  001239.ldb  001332.ldb  001395.ldb  001489.ldb  001497.ldb  001536.ldb  001573.ldb
001145.ldb  001240.ldb  001333.ldb  001396.ldb  001490.ldb  001498.ldb  001566.log  CURRENT
001146.ldb  001241.ldb  001334.ldb  001397.ldb  001491.ldb  001530.ldb  001567.ldb  LOCK
001147.ldb  001242.ldb  001335.ldb  001398.ldb  001492.ldb  001531.ldb  001568.ldb  MANIFEST-001564
001235.ldb  001243.ldb  001336.ldb  001399.ldb  001493.ldb  001532.ldb  001569.ldb
001236.ldb  001244.ldb  001337.ldb  001400.ldb  001494.ldb  001533.ldb  001570.ldb
001237.ldb  001330.ldb  001393.ldb  001401.ldb  001495.ldb  001534.ldb  001571.ldb
001238.ldb  001331.ldb  001394.ldb  001402.ldb  001496.ldb  001535.ldb  001572.ldb

While computer1 shows following structure:

Code:

[bitcoin@computer1:~/.bitcoin/blocks]## ls

blocks  index

[bitcoin@computer1:~/.bitcoin/blocks]# ls *

blocks:
blk03189.dat  blk03197.dat  blk03205.dat  blk03213.dat  rev03193.dat  rev03201.dat  rev03209.dat
blk03190.dat  blk03198.dat  blk03206.dat  blk03214.dat  rev03194.dat  rev03202.dat  rev03210.dat
blk03191.dat  blk03199.dat  blk03207.dat  blk03215.dat  rev03195.dat  rev03203.dat  rev03211.dat
blk03192.dat  blk03200.dat  blk03208.dat  blk03216.dat  rev03196.dat  rev03204.dat  rev03212.dat
blk03193.dat  blk03201.dat  blk03209.dat  rev03189.dat  rev03197.dat  rev03205.dat  rev03213.dat
blk03194.dat  blk03202.dat  blk03210.dat  rev03190.dat  rev03198.dat  rev03206.dat  rev03214.dat
blk03195.dat  blk03203.dat  blk03211.dat  rev03191.dat  rev03199.dat  rev03207.dat  rev03215.dat
blk03196.dat  blk03204.dat  blk03212.dat  rev03192.dat  rev03200.dat  rev03208.dat  rev03216.dat

index:
001144.ldb  001239.ldb  001332.ldb  001395.ldb  001489.ldb  001497.ldb  001536.ldb  001567.ldb
001145.ldb  001240.ldb  001333.ldb  001396.ldb  001490.ldb  001498.ldb  001560.log  CURRENT
001146.ldb  001241.ldb  001334.ldb  001397.ldb  001491.ldb  001530.ldb  001561.ldb  LOCK
001147.ldb  001242.ldb  001335.ldb  001398.ldb  001492.ldb  001531.ldb  001562.ldb  MANIFEST-001559
001235.ldb  001243.ldb  001336.ldb  001399.ldb  001493.ldb  001532.ldb  001563.ldb
001236.ldb  001244.ldb  001337.ldb  001400.ldb  001494.ldb  001533.ldb  001564.ldb
001237.ldb  001330.ldb  001393.ldb  001401.ldb  001495.ldb  001534.ldb  001565.ldb
001238.ldb  001331.ldb  001394.ldb  001402.ldb  001496.ldb  001535.ldb  001566.ldb

(Question 1)
Simply said --> computer1 had no block files in its root folder ./blocks stored ! they are stored in another subfolder called blocks. Why did this happen, how is that possible, what caused this discrepancy ?

So after I copied the blocks/chainstate folder from computer2 to computer1, I had to manually create another sub-folder "blocks" on computer1 and move all *.dat files into this new subfolder. After that bitcoin-core started and found what it looked for and continued its work. However I am puzzled about this because as stated before both computers use the same bitcoin.conf, the configuration directives are exactly the same on both machines. Any clues ?

(Question 2)
My other question which unfortunately is still unanswered:

5.0 G   chainstate
4.0 G   blocks

I'd have expect to see only 4GB of data because I configured prune=4096 in my bitcoin.conf file.

Let's say I have 6 GB of free space which I like to run my pruned bitcoin-core node on, so I configure prune=4096 on my node. I would expect that the node would not occupy more than this. But as you see it occupied more than twice the configured size. In that case the node would not be able to run because the file space would have been filled up already. So why do I have two folders with 4GB, total 9GB in my real life example ? I would have expected to see only 4GB file space claimed here.

(Question 3)
As far as I understood it correctly according to this wiki here ...

-prune=<n>

Reduce storage requirements by enabling pruning (deleting) of old blocks. This allows the pruneblockchain RPC to be called to delete specific blocks, and enables automatic pruning of old blocks if a target size in MiB is provided. This mode is incompatible with -txindex, -coinstatsindex and -rescan. Warning: Reverting this setting requires re-downloading the entire blockchain. (default: 0 = disable pruning blocks, 1 = allow manual pruning via RPC, >=550 = automatically prune block files to stay under the specified target size in MiB)

... my pruned node should have no further restrictions except that I cannot use -txindex, -coinstatsindex and -rescan, correct ? as long as I don't run/use the wallet function on that full-node I shouldn't miss anything important by those three switch functions, right?

Thus, I should not have any noticeable disadvantages with my pruned node, right?

(Question 4)
What about if I can't accept incoming connections on port 8333? Then my node only connects to a maximum of 10 outgoing nodes. Is that bad except that the whole network can't work correctly as peer-to-peer if everyone would operate it that way. But do I have any serious disadvantages otherwise? EDIT: I guess I've found an answer to this question. I conclude that it's not a real disadvantage when the node is running only as outbound. Of course it is not as much supportful for the network but I understand this. However, by the explained method using a TOR hidden service I could make it a supportful node again

[LoyceV]

(Question 1)
Simply said --> computer1 had no block files in its root folder ./blocks stored ! they are stored in another subfolder called blocks. Why did this happen, how is that possible, what caused this discrepancy ?

I guess you changed the data directory when you first started Bitcoin Core.

I am puzzled about this because as stated before both computers use the same bitcoin.conf, the configuration directives are exactly the same on both machines. Any clues ?

Can you post the contents of bitcoin.conf?

Additional question: why are you running Bitcoin Core as root on computer1? Generally, that's bad practice.

(Question 2)
My other question which unfortunately is still unanswered

Did you overlook this post?

the blocks directory contains the actual blocks. The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

Let's say I have 6 GB of free space which I like to run my pruned bitcoin-core node on, so I configure prune=4096 on my node. I would expect that the node would not occupy more than this. But as you see it occupied more than twice the configured size. In that case the node would not be able to run because the file space would have been filled up already. So why do I have two folders with 4GB, total 9GB in my real life example ? I would have expected to see only 4GB file space claimed here.

If you want it to fit in 6 GB, you'll have to prune to the minimum (550). But that doesn't leave much space for future growth.

(Question 3)
Thus, I should not have any noticeable disadvantages with my pruned node, right?

You also can't upload older blocks to other nodes.

[citb0in]

I guess you changed the data directory when you first started Bitcoin Core.

hum..not quite sure any more, maybe yes maybe no ...

Can you post the contents of bitcoin.conf?

Code:

daemon=1
prune=4096
dbcache=10284 # this value is from the fast host, the other one has a much lower value
avoidpartialspends=1
[main]
[test]
[regtest]

Additional question: why are you running Bitcoin Core as root on computer1? Generally, that's bad practice.

I don't. If I accidently posted some bash outputs with "root" showing than it was just because I was logged in with root at that moment I took the screen dump. But the daemon runs as user "bitcoin" which I created extra for this purpose. It's a restricted user account.

Did you overlook this post?

the blocks directory contains the actual blocks. The chainstate directory contains the state as of the latest block (in simplified terms, it stores every spendable coin, who owns it, and how much it's worth).

indeed. Thanks for pointing out. Related to the data structure l also found some interesting information here and here. Nevertheless, I would suggest to explicitly state that in the bitcoin-core documentation or wiki. A user would mistakenly assume that the prune value is what would be expected for the node setup related to the disk space required. However, we have just clarified that this is not the case. Unfortunately, I have not found any reference to this in any documentation so far. Only my recommendation to emphasize this.

You also can't upload older blocks to other nodes.

Can you give examples why an average user should need to upload older blocks to other nodes, please? Is this something the user manually need to initiate and for what reasons ?

[LoyceV]

I guess you changed the data directory when you first started Bitcoin Core.

hum..not quite sure any more, maybe yes maybe no ...

Can you post the contents of bitcoin.conf?

I just checked: it turns out the data directory isn't stored in that (which is inside the data directory). It's stored here:

Code:

~/.config/Bitcoin/Bitcoin-Qt.conf

I would suggest to explicitly state that in the bitcoin-core documentation or wiki. A user would mistakenly assume that the prune value is what would be expected for the node setup related to the disk space required. However, we have just clarified that this is not the case. Unfortunately, I have not found any reference to this in any documentation so far. Only my recommendation to emphasize this.

You have a point, it would make sense to share this information in the popup during installation.

Can you give examples why an average user should need to upload older blocks to other nodes, please? Is this something the user manually need to initiate and for what reasons ?

It's absolutely not necessary. I just thought you wanted it:

to make a small contribution to the bitcoin community, I would like to install a full node.

[citb0in]

I am running headless, so no QT on any of my nodes.

Yes, I understand the point of a full (unpruned) node. However I was curious if "uploading older blocks" has a certain reason and when this needs to happen, just for my understanding. BTW: Meanwhile I have installed and running five different nodes on various sites across the planet, I am trying to contribute as much as possible and affordable to me