Cheap Node Self Hosting: Just because you CAN does not mean you SHOULD

[DaveF]

Since the same things keep coming up:

1) Yes you can force a RPi2 or 3 to run a node but it is going to be a slow miserable time and you will have lots of issues.

2) Yes you can download the entire blockchain and run a node on a 500GB drive. But you WILL run out of space and then probably have corruption and then have to buy a larger drive and start again. And you might not even be able to copy the data from the old drive if the corruption was bad enough.

3) Yes you can use older equipment to do it. You can even use 10 year old laptops with a 2nd gen core i5. But much older then that the pure performance issues are going to be a killer. And as time goes on it's only going to get worse.

4) You can even use an old 5400 RPM drive and store your OS and blockchain on it. But if for whatever reason you have to do a full re-scan of the blockchain I hope you have a couple of days to kill.

5) Yes you can do it with 2GB of RAM, but your IBD is going to take 2 days short of forever and general performance is going to be miserable.

n0nce made a nice post about doing a node for under $60. https://bitcointalk.org/index.php?topic=5364742
You can do it for a bit more, you can do it for a bit less.

Don't try to cut that price in 1/2 unless you are technically inclined and can work around the issues. Otherwise you are going to have a miserable time.

This is not to stop people from trying, far from it. This is a post letting you know that it WILL WORK. But the time and effort you are going to put into it, although probably a learning experience is going to be a lot more then having SLIGHTLY better equipment and doing it more efficiently.

-Dave

[NotATether]

I believe the bitcoin.org website lists the minimum required PC specs for running Bitcoin core (with the exception of CPU), but I don't think it lists recommended specs. I'll fill these in from my own experience:

CPU: at least 2.5GHz (block verification is single-threaded. Additional cores won't speed this up.)
Memory: 4GB RAM (healthy buffer for dbcache)
Disk: 1TB HDD (7200RPM separate from OS - but only because 1TB SSDs are still fairly expensive.)
Network: 50Mbit/s (anything less than that and block & tx download/upload will affect your connection quality)

[dkbit98]

Yes you can force a RPi2 or 3 to run a node but it is going to be a slow miserable time and you will have lots of issues.

Raspberry Pi4 is much better option because they have much bigger option to choose more RAM, that is if yo can find them available in stock 

Yes you can download the entire blockchain and run a node on a 500GB drive. But you WILL run out of space and then probably have corruption and then have to buy a larger drive and start again. And you might not even be able to copy the data from the old drive if the corruption was bad enough.

Unless you are running pruned node, that can run on even smaller hard disk without any issues.
I don't think larger hard drives are problem for anyone nowadays, old style HDD are very cheap now and you can find them anywhere.

Yes you can use older equipment to do it. You can even use 10 year old laptops with a 2nd gen core i5. But much older then that the pure performance issues are going to be a killer. And as time goes on it's only going to get worse.

If you look at ebay or craiglist you can find very cheap good laptops with more than enough RAM, processor and hard disk.
I would say that for around $100 you can get great used laptop with 3rd or 4th generation i5 or i7 intel with SSD and 8gb RAM, but better look models that can be upgraded in future, with socketed parts, not soldered.

but only because 1TB SSDs are still fairly expensive.

I don't think they are so much expensive anymore, you can quickly find very good new 1TB for around $80 to $90, and even cheaper on some deals.
There are a bunch of used 100% health 2.5 inch SSD drives being sold online, so you can find good Samsung 860 evo for around $60 to $70.
However, you should always look for TBW and higher number means that disk will last much longer, for 860 evo that would be 600TB.

[BitMaxz]

but only because 1TB SSDs are still fairly expensive.

I don't think they are so much expensive anymore, you can quickly find very good new 1TB for around $80 to $90, and even cheaper on some deals.
There are a bunch of used 100% health 2.5 inch SSD drives being sold online, so you can find good Samsung 860 evo for around $60 to $70.
However, you should always look for TBW and higher number means that disk will last much longer, for 860 evo that would be 600TB.

You can buy low space SSD for OS which is cheaper than buying 1TB SSD and you can split them with HDD for blockchains just to improve the syncing process.
We already have a guide here on how to split them if you want to speed up the syncing process you can check this guide "[Guide] Split Bitcoin blockchain between SSD & HDD and speed up the initial sync"

[NeuroticFish]

2) Yes you can download the entire blockchain and run a node on a 500GB drive. But you WILL run out of space and then probably have corruption and then have to buy a larger drive and start again. And you might not even be able to copy the data from the old drive if the corruption was bad enough.

It's a very informative post, thank you very much for it. I did intend to "cut corners" and see whether it would work (and also how badly!) with the intention to just use the (few) hardware I have at hand.
Luckily I didn't have the time for the setup, so, again, this helps me a lot.

I will also vouch for the 500GB HDD "approach" problem. Right now my setup works with an external 500GB HDD for the blockchain and I already got rid of txindex in order to make sure it doesn't suddenly stop because of the lack of space (and that hurts badly my local block explorer, obviously).

Of course, one of my problems was that I've thought late to make myself a local setup and RasPi are overly expensive for now, so I'll keep the current setup (that unfortunately doesn't run 24/7), but I still have to look for a HDD soon.

You can buy low space SSD for OS which is cheaper than buying 1TB SSD and you can split them with HDD for blockchains just to improve the syncing process.

That's correct. Even more, if you have another high spec computer (with SSD), you can use that one for the initial sync and just use HDD (some temporary symlinks to SSD and copying of files will be needed though).
While a normal spec is great, I think that SSD can be seen a bit "overzealous" by those very careful with the costs.
But: SSD is silent, HDD is not. SSD is probably much more energy efficient. So one has to make his choice properly for the long run.


PS: is there any info available about how much heating a RasPi do if used as a node? I fear that anything under RasPi 4 may need active cooling too (again, noise...)

[ranochigo]

But: SSD is silent, HDD is not. SSD is probably much more energy efficient. So one has to make his choice properly for the long run.


PS: is there any info available about how much heating a RasPi do if used as a node? I fear that anything under RasPi 4 may need active cooling too (again, noise...)

I think given the price to performance as well as the longevity aspect, the energy efficiency of HDD vs SSD would be negligible or simply a dollar or two extra per year, maximum. The price of a HDD is ridiculously low compared to SSD of similar size. The noise produced by HDD is actually quite overstated most of the time and it is barely audible.

Regarding the cooling required for RPi4, I ran into some throttling with RPi4 during synchronization and slapped a heatsink, which more or less solved the issue. Active cooling wasn't really necessary in my case. Never really an issue with the previous revisions though (7w vs 5w). It is still painfully slow regardless.

[NeuroticFish]

The price of a HDD is ridiculously low compared to SSD of similar size.

I thought that this is obvious enough to not state it

The noise produced by HDD is actually quite overstated most of the time and it is barely audible.

Then maybe I'm wrong. The HDD I use for blockchain is more than 12 years old, it's a WD Black and it's pretty noisy.

Regarding the cooling required for RPi4, I ran into some throttling with RPi4 during synchronization and slapped a heatsink, which more or less solved the issue. Active cooling wasn't really necessary in my case. Never really an issue with the previous revisions though (7w vs 5w). It is still painfully slow regardless.

The "painfully slow" part is indeed useful (I'd guess though that it happens in the initial sync, or when it's a couple days behind), but I was more curious if anybody had the (expectedly bad) experience with lower spec Pi (3,2,1, or even Zero).

[DaveF]

The "painfully slow" part is indeed useful (I'd guess though that it happens in the initial sync, or when it's a couple days behind), but I was more curious if anybody had the (expectedly bad) experience with lower spec Pi (3,2,1, or even Zero).

Think weeks vs. days.

Not quite the same but, a RPi4 with a SSD can sync Umbrel in 3 or 4 days. That is over TOR and indexing the blockchain and the other stuff to run a lightning node.

A RPi3, not a prepacked node setup, just the bare RPi software and bitcoind with a 7200 RPM is still churning through just the IBD  after 2 1/2 weeks.
Both are behind the same router on the same cable modem so it's not bandwidth just purely a performance issue.

-Dave

[ABCbits]

Regarding the cooling required for RPi4, I ran into some throttling with RPi4 during synchronization and slapped a heatsink, which more or less solved the issue. Active cooling wasn't really necessary in my case. Never really an issue with the previous revisions though (7w vs 5w). It is still painfully slow regardless.

Also update the firmware of the Pi 4, there are few update which reduce the heat without sacrifice the performance.

The noise produced by HDD is actually quite overstated most of the time and it is barely audible.

Then maybe I'm wrong. The HDD I use for blockchain is more than 12 years old, it's a WD Black and it's pretty noisy.

WD Black isn't exactly average customer HDD, it's designed for best performance at high price. Besides, i find newer HDD has less noise than older ones.

[dkbit98]

I think given the price to performance as well as the longevity aspect, the energy efficiency of HDD vs SSD would be negligible or simply a dollar or two extra per year, maximum. The price of a HDD is ridiculously low compared to SSD of similar size. The noise produced by HDD is actually quite overstated most of the time and it is barely audible.

Some disks are creating more noise than others, but if the HDD noise is the problem, I found easy solution for that by purchasing silencing case enclosure that almost totally removed noise from my HDD.
It is made from solid metal from inside (I think copper) with combination of foam and other materials that makes drive cool and quiet.
I found this very cheap, but I guess you can easily make something similar if you like DIY projects.

Regarding the cooling required for RPi4, I ran into some throttling with RPi4 during synchronization and slapped a heatsink, which more or less solved the issue. Active cooling wasn't really necessary in my case. Never really an issue with the previous revisions though (7w vs 5w). It is still painfully slow regardless.

I can confirm that Rpi400 has a huge heatsink that made massive cooling improvements, so you don't need to install any fans and you will have 100% silent computer.
Version with 4GB and 8GB is not slow at all, but there is option of using Rpi computing module, or some Rpi clones that are even stronger.

 

[hZti]

Version with 4GB and 8GB is not slow at all, but there is option of using Rpi computing module, or some Rpi clones that are even stronger.

 

Watt Raspi clones do you mean?

[NeuroticFish]

Watt Raspi clones do you mean?

I'd guess that one in the list could be Orange Pi which I heard about not long ago and I don't know it it's indeed good, still, you can read some comparison here: https://versus.com/en/orange-pi-rk3399-vs-raspberry-pi-4-model-b
There may be others too, let's see what @dkbit98 meant.

Besides, i find newer HDD has less noise than older ones.

Yeah, probably. I guess that I've just proven the point that old hardware is probably not worth it

[dkbit98]

Watt Raspi clones do you mean?

There are bunch of Rpi clones for years, some of them are RockPi, Odroid, Pine64, Banana Pi, Orange Pi and some others could be considered as alternative, for example ASUS Tinker Board.
I am sure there are other single board computers that are inspired by raspberry pi.

[LoyceV]

The HDD I use for blockchain is more than 12 years old, it's a WD Black and it's pretty noisy.

I've had old server disks that make a loud KRRR KRRR sound, but I can barely hear my modern 2.5" disks.

In my experience, anything with 8 GB RAM or more will work just fine to run a full Bitcoin Core on Linux. If the hardware supports 8 GB, I assume it's modern enough to have sufficient processing power. I've used Bitcoin Core with 2 GB and HDD on an Atom netbook years ago (back then I liked to keep a backup blockchain ready), but I abandoned it because it took painfully long to update.
I'm a sucker for laptops nowadays, but unfortunately most of them can handle only one HDD. My preferred setup is adding a large HDD for data that doesn't need to be on the (smaller) SSD.

[NeuroticFish]

I'm a sucker for laptops nowadays, but unfortunately most of them can handle only one HDD. My preferred setup is adding a large HDD for data that doesn't need to be on the (smaller) SSD.

I usually try to avoid investing into hardware, since its price keeps dropping sharply just after buying. I've bought my laptop in 2017 and I've already been using your strategy: I've bought the laptop with SSD and then added a HDD to it.
My data stays on a NAS that seems to be aging pretty well. I've upgraded there not long ago an 500 GB HDD to 4TB (I feared that bigger HDD won't work there) and now that 500 GB HDD I use for the blockchain, plugged into a thingy that I can connect to on USB3. Clearly suboptimal solution, but I just didn't have the hearth yet to throw away that HDD

But you're right. My laptop's HDD is indeed silent. All I can hear (when I take my headphones off, that is) is the cooler the laptop stands on.

---
I usually don't have the hearth to throw away old stuff, but this topic may be just the thing to convince me do it! I have yet to find a device I can leave running 24/7 without fearing it'll set the room on fire, but I find today's 150-200 EUR too much for a decent RasPi (board only).

[hZti]

As far as I understand the Raspi 4 is the first Raspi that can be easily used to set up a node. So in 2-3 years we can expect to have a very strong increase in the number of nodes when all the current Raspi 4 are sold as second hand for 20 Dollars. So for me I will just wait for that and also don't put 200-300 USD in a decent raspi node setup.

[nullama]

This post reminded me of the message in the README from the mempool.space production setup:

HDD vs SSD vs NVMe

If you don't have a fast SSD or NVMe-backed disk, that's fine—go online and buy some fast new NVMe drives. When they arrive, install them, throw away your old HDDs, and then proceed with the rest of this guide.

 

[ABCbits]

This post reminded me of the message in the README from the mempool.space production setup:

HDD vs SSD vs NVMe

If you don't have a fast SSD or NVMe-backed disk, that's fine—go online and buy some fast new NVMe drives. When they arrive, install them, throw away your old HDDs, and then proceed with the rest of this guide.

 

To be fair, that instruction meant to run blockexplorer for 6 (six) different cryptocurrency network. Mempool also mention this on their installation methods.

Mempool can be self-hosted on a wide variety of your own hardware, ranging from a simple one-click installation on a Raspberry Pi full-node distro all the way to a robust production instance on a powerful FreeBSD server.

[DaveF]

The point, in general does remain the same.
You can cram a lot of stuff on under-powered slower devices with I/O and other things that can't keep up.
However, in the end you are not doing yourself any favors.

And as the blockchain gets bigger and the other apps that use it need more and more storage and everything else, you will hit sooner or later hit a wall.
Sometimes the wall is soft and padded and all you have to do is dd your older slower smaller drive to another one. Other times the entire thing gets corrupted and you have to start from scratch.

-Dave

[Dabs]

I would put a budget of around maybe $500 USD equivalent to run a full node full time on enterprise level rack server machines. You can get refurbished to save money, it does not have to be brand new. That amount of money buys you a Xeon dual processor running faster than 2 Ghz, with at least 16 GB of RAM and another $100 for whatever hard drive space or pair of SSDs you need for the near future. 1 TB should last awhile.