Hardware requirements for analyzing ouput of rusty-blockparser

[eugx0]

Hello,

I am trying to do some data analytics on the bitcoin blockchain. I am running a node and then parsing the .dat files using the rusty-blockparser.
This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size,  I can't load it on a python jupyter notebook. I am wondering what common programs and strategies that people use to process this data.

Thanks for the help,

Best,

[1713579814]

1713579814

[1713579814]

1713579814

[1713579814]

1713579814

[LoyceV]

This produces four .csv files which the biggest one is nearly 100 GB of data. Due to the size,  I can't load it on a python jupyter notebook.

You can read the data into some database that does what you need, or use things like grep and split to get only the lines you need.

I am wondering what common programs and strategies that people use to process this data.

Can you start by sharing what you're trying to accomplish? I usually pipe the data through some Linux command line tools to get what I need, you'll find some examples in those topics:

Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used
[~500 GB] Bitcoin block data: inputs, outputs and transactions

[DaveF]

Jupyter Notebook is also bloaty software.

Along with what ETF and Loyce said. Use the proper tool for the job, there are better ways to get at the data. I'm assuming (yes I know don't assume) you are a student and have to use Jupyter for class so in the end it has to wind up in there. But, you can't just do A->B you need to get it to a size and format that it can handle.

-Dave