Set up Bitcoin node for data analysis

[Schabernack]

Hi,

I want to become an active part of the Bitcoin network and host my own node. Besides contributing to the network, I would like to gain access to the data stored on the blockchain to conduct different types of analyses. I want to set up my wallet on the node as well.

I am currently searching for the proper hardware for this purpose. Is a Raspberry Pi (4 or 5) a valid option or should I go for something with a little bit more performance if I also want to extract the data from the blockchain and run some analysis?

-Should I aim for 16GB RAM or already go for 32GB RAM?
-Is a 4TB SSD sufficient or should I also look for a harddrive with more storage space, given that Ordinals increase the pace of data growth on the Bitcoin blockchain?
-What other hardware is crucial for my purpose? I am happy about any any additional feedback on that.

I am willing to spend some money (I do not require the cheapest possible setup) on the hardware if it is necessary for my use case.

Kind regards,
Schabernack

[nc50lc]

I am willing to spend some money (I do not require the cheapest possible setup) on the hardware if it is necessary for my use case.

In that case, you might want to consider to "max-out" the motherboard of your choice. (for futureproofing purposes)

-Should I aim for 16GB RAM or already go for 32GB RAM?
-Is a 4TB SSD sufficient or should I also look for a harddrive with more storage space, given that Ordinals increase the pace of data growth on the Bitcoin blockchain?
-What other hardware is crucial for my purpose? I am happy about any any additional feedback on that.

Ordinals haven't increased the pace since the maximum block weight is still at 4WU (1vMB) even after TapRoot upgrade.
Maybe a bit because of the more consistent full blocks that's caused by the large transaction volume that we're experiencing since then.
Refer to this chart: https://ycharts.com/indicators/bitcoin_blockchain_size
At that pace, your 4TB SSD should be sufficient enough.

[apogio]

Besides contributing to the network, I would like to gain access to the data stored on the blockchain to conduct different types of analyses. I want to set up my wallet on the node as well.

May I ask what exactly do you want to do? For personal reasons?

-Should I aim for 16GB RAM or already go for 32GB RAM?
-Is a 4TB SSD sufficient or should I also look for a harddrive with more storage space, given that Ordinals increase the pace of data growth on the Bitcoin blockchain?
-What other hardware is crucial for my purpose? I am happy about any any additional feedback on that.

Prefferably aim for:
8GB RAM
2TB SSD

Anything higher than that is good but not necessary / mandatory.

[ABCbits]

-Should I aim for 16GB RAM or already go for 32GB RAM?

Honestly it comes down to which data analysis software you use and additional software you run (for example, self-hosted block explorer).

-Is a 4TB SSD sufficient or should I also look for a harddrive with more storage space, given that Ordinals increase the pace of data growth on the Bitcoin blockchain?

Current blockchain is size is about 533GB[1] and maximum possible growth per month is 17280MB (4MB * 144 blocks/day * 30 day). So 4TB should be sufficient.

-What other hardware is crucial for my purpose? I am happy about any any additional feedback on that.

I'd say CPU assuming data analysis software you use support using multiple core/thread.

[1] https://blockchair.com/bitcoin

[Schabernack]

Hi guys,

first of all thank you all for your feedback, including details and links with further information!

To answer your questions, I will break down my answer into several points:

1.) Data Analysis

-I want to run analyses to "explore" and "visualize" the Bitcoin ecosystem (some data providers do not provide the raw data that I intend to use).

-There are obviously data providers that I could use but I want to make sure that
1. I always have access to the data
2. I familiarize myself with interacting with Bitcoin on a "data-level"
3. I have access to data I need for my analysis and on a high level of granularity

-Initially, I only intend to download data, store it in tables, and use Python to run further analysis. This might change as time goes by.

2.) I will run the node and use the data for personal reasons (being part of the Bitcoin network, research interest)

3.) I was actually considering to use a Raspberry Pi as it is very compact in size and easy to store somewhere in a room. However, I am unsure about how well it performs when I want to do some data analysis. That is why I want to use a device with more performance so that I can host the node and conduct further activities on the device simultaneously. I will try to get a decent CPU and sufficient RAM.

As for now, I will use a 4TB SSD for storage. Regarding the computer, I have one more question: which OS is the "best" to use? I am a Microsoft user and I am aware of its vulnerability compared to other OS. Is it still a valid option to use, even if I also want to use my node to host my own wallet, or are the risks to high if I use Microsoft?

[Cricktor]

I have multiple headless nodes on a few Raspberry Pi 4B with 8GB RAM (RaspiBlitz, RaspiBolt, Umbrel). All work fine and are nicely power efficient, especially when you run them 24/7/365. While such nodes like RaspiBlitz or especially Umbrel abstract the underlying Linux from the user, it's still beneficial to have or learn some Linux knowledge. You can do it all on Windows, too, but frankly I would rather choose a Linux base.

Another suggestion is to get a used business laptop with 16GB RAM (or option to upgrade to it) which are usually not very expensive. On my ThinkPad T540 I run Ubuntu and another Bitcoin node (no, I'm no Bitcoin node addict, I think). While idle (Bitcoin Core runs still in the background) and no active display this laptop consumes on average somewhere in the ballpark of 11-12W (about double of a Raspi Pi 4B).

[ABCbits]

As for now, I will use a 4TB SSD for storage. Regarding the computer, I have one more question: which OS is the "best" to use? I am a Microsoft user and I am aware of its vulnerability compared to other OS. Is it still a valid option to use, even if I also want to use my node to host my own wallet, or are the risks to high if I use Microsoft?

Since you only mention Python, Linux probably is better option since Python comes with many Linux distro and you could allocate more RAM and CPU for your Python script since Linux use less RAM/CPU.

[Schabernack]

Thank you again very much for the information!

Right now, I am thinking about running two nodes on two devices:

1. Windows laptop (as of now, easier for me)
Is a setup including a 16GB RAM and Intel Core i5 (12th or 13th generation) sufficient to run a node and do some data analysis (Python, SQL) simultaneously? I know it is not easy to answer such a question, but I would appreciate your opinion.

Can someone please tell me if the following idea makes sense:
I will set up a node that I will only use for actions related to interacting with Bitcoin. Ocassionally, I will download some data from the blockchain and store the data on another external harddrive. Then I will use this harddrive to transfer the data to my current laptop (which has a good performance) and do the data analysis on my current laptop.
Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop? Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again?

2. Rasperry Pi
I am eager to run one node on the Rasperry Pi for the previously mentioned reasons and to get more familiar with Linux. Accumulating knowledge and skills will take time but I am willing to invest the time.

[apogio]

Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop? Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again?

Bitcoin nodes can be shut down and then they will continue exactly from where they stopped. But of course they need to be terminated gracefully!

Why do you need to transfer the SSD from one computer to the other? You want one computer for the node and one computer for the "analysis"?

[LoyceV]

Before doing all this for data analysis, have a look at these topics:

Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
[~500 886 GB] Bitcoin block data: inputs, outputs and transactions

[satscraper]

Can someone please tell me if the following idea makes sense:
I will set up a node that I will only use for actions related to interacting with Bitcoin. Ocassionally, I will download some data from the blockchain and store the data on another external harddrive. Then I will use this harddrive to transfer the data to my current laptop (which has a good performance) and do the data analysis on my current laptop.
Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop? Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again?

In my view you should try both options and select the most optimal one which depends on that how resource-avid  is your your data analysis.

If you current laptop   has a juice to do both jobs simultaneously, i.e. to support the copy of  blockchain on your drive in sync state and at same time perform data analysis then there is  no need in the second  laptop.

In any case don't forget   to have txindex=1 in your conf file so that you could analyse all transactions rather than your own ones.

[ABCbits]

1. Windows laptop (as of now, easier for me)
Is a setup including a 16GB RAM and Intel Core i5 (12th or 13th generation) sufficient to run a node and do some data analysis (Python, SQL) simultaneously? I know it is not easy to answer such a question, but I would appreciate your opinion.

Probably yes. You can configure how much RAM used by your SQL database. And after you download all blockchain, you can setup Bitcoin Core to use less RAM as well without much problem. Just don't try to load all data to RAM at once.

Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop?

Yes, assuming you use Bitcoin Core to download blockchain data.

Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again?

It's the latter.

2. Rasperry Pi
I am eager to run one node on the Rasperry Pi for the previously mentioned reasons and to get more familiar with Linux. Accumulating knowledge and skills will take time but I am willing to invest the time.

I expect Raspberry Pi will struggle for data analysis.

[ranochigo]

Thank you again very much for the information!

Right now, I am thinking about running two nodes on two devices:

1. Windows laptop (as of now, easier for me)
Is a setup including a 16GB RAM and Intel Core i5 (12th or 13th generation) sufficient to run a node and do some data analysis (Python, SQL) simultaneously? I know it is not easy to answer such a question, but I would appreciate your opinion.

Can someone please tell me if the following idea makes sense:
I will set up a node that I will only use for actions related to interacting with Bitcoin. Ocassionally, I will download some data from the blockchain and store the data on another external harddrive. Then I will use this harddrive to transfer the data to my current laptop (which has a good performance) and do the data analysis on my current laptop.
Or can I simply disconnect my SDD (where the blockchain data is stored) from my node for a few hours and download the blockchain data directly to my other laptop? Will it cause problems if my node is offline for several hours or does it just need some time to synch once it is online again?

2. Rasperry Pi
I am eager to run one node on the Rasperry Pi for the previously mentioned reasons and to get more familiar with Linux. Accumulating knowledge and skills will take time but I am willing to invest the time.

I second what is said above: Don't use Raspberry Pi. My Pi 4 was overheating constantly when running a node and I had to get active cooling on it.

It is okay to remove your data directory, provided that your Core has shutdown properly and is not writing any data on it, or it'll get corrupted and you'll have to do a reindex again. You might just want to run everything on a single computer instead; indexing the transaction data and parsing them to another format can take quite a while and external harddrive are slow, and are prone to corruption.

[apogio]

I second what is said above: Don't use Raspberry Pi. My Pi 4 was overheating constantly when running a node and I had to get active cooling on it.

Out of curiousity, what where you running on the Raspberry and how much RAM did it have?

I run 2 Raspberry Pi 4b 8GB RAM and I never had overheating issues.

I use this case on both of them and the temperature never exceeds 58-60 celcius. It has passive cooling and there is no noise at all.

[ranochigo]

I second what is said above: Don't use Raspberry Pi. My Pi 4 was overheating constantly when running a node and I had to get active cooling on it.

Out of curiousity, what where you running on the Raspberry and how much RAM did it have?

I run 2 Raspberry Pi 4b 8GB RAM and I never had overheating issues.

I use this case on both of them and the temperature never exceeds 58-60 celcius. It has passive cooling and there is no noise at all.

I think I ran it with my RPi 4, 4GB. How were your temps looking during the synchronization? Mine was fairly hot throughout, granted that it was just a heatsink and not a fancy heat dispersing case.

Not to mention, it took weeks.

[apogio]

I think I ran it with my RPi 4, 4GB. How were your temps looking during the synchronization? Mine was fairly hot throughout, granted that it was just a heatsink and not a fancy heat dispersing case.

Not to mention, it took weeks.

What OS did you run? Mine is downloading the monero blockchain now and the temp is currently 43.8 celcius. I don't remember exactly with Bitcoin but I can tell you for sure that it never went over 58-60 celcius.

But RAM is very important for this. So 8GB work much better than 4GB.

[ranochigo]

What OS did you run? Mine is downloading the monero blockchain now and the temp is currently 43.8 celcius. I don't remember exactly with Bitcoin but I can tell you for sure that it never went over 58-60 celcius.

But RAM is very important for this. So 8GB work much better than 4GB.

Raspbian. Mine was above 70 for most of the time, but again your cooling method makes a huge difference. Having the entire casing as a heatsink is undoubtedly better with what I've had on hand. RPi4s were notorious for overheating actually, might've been the issue with the earlier versions.

RAM is important, but RPis are pretty expensive so I had to skimp on it back then. RAM shouldn't affect temperature anyways.

[apogio]

What OS did you run? Mine is downloading the monero blockchain now and the temp is currently 43.8 celcius. I don't remember exactly with Bitcoin but I can tell you for sure that it never went over 58-60 celcius.

But RAM is very important for this. So 8GB work much better than 4GB.

Raspbian. Mine was above 70 for most of the time, but again your cooling method makes a huge difference. Having the entire casing as a heatsink is undoubtedly better with what I've had on hand. RPi4s were notorious for overheating actually, might've been the issue with the earlier versions.

RAM is important, but RPis are pretty expensive so I had to skimp on it back then. RAM shouldn't affect temperature anyways.

To be honest I have created a node as a series of tutorials for Bitcointalk here.

This node runs Raspbian Lite and to be honest the temperature went close to 65-67 degrees during the IBD.

The other node that I have runs Start9 and the IBD went very smoothly and much faster than the other one's.

I used to have Umbrel too and I don't remember any issues too. I just switched to Start9 because I loved its simplicity and the fact that it is much easier  to customize.

[LoyceV]

My Pi 4 was overheating constantly when running a node and I had to get active cooling on it.

The thing burns 6.4W max. That shouldn't be too difficult to dissipate with passive cooling.

[ranochigo]

The thing burns 6.4W max. That shouldn't be too difficult to dissipate with passive cooling.

On a fairly compact surface area and my aluminium heat sink was apparently not enough, the earlier versions were at 7-8Ws and were known to overheat. Might be better to get a bigger heatsink and a better thermal pad for it.