Congratulation, Bitcoin has reached 500 GB size hard disk data

[Jason Brendon]

Is 500GB a lot? 5TB drives cost less than $100, so $20 for 1 TB. $20 gives you enough space to run a system, some basic software and have enough space to run full node at least until 2025.

man, you're talking about HHD right? it is slow as fuck.

[Jason Brendon]

Student got 1 terabyte gdrive when login in using college email. and maybe more than 5 terabytes when you have enough shrewd to modify your friend email using Cbackups software.

yeah but i don't want my blockchain data has anything to do with google drive....especially when there is a risk of KYC relation..

[Hydrogen]

Compressing data into a format like .zip or .tar can roughly decrease file sizes by a factor of 1/3rd.

500 / 3 = 166 gigabytes.

While I do not support most updates or amendments to bitcoin core, I do wonder if compression of file sizes may become necessary at some point to maintain node support.

[LoyceV]

Compressing data into a format like .zip or .tar can roughly decrease file sizes by a factor of 1/3rd.

500 / 3 = 166 gigabytes.

You're wrong on several points: First, .tar doesn't compress on it's own. Second, dividing by 3 means a 2/3rd decrease in size. But most important: compression of (more or less) random data isn't that good. I just tried gzip, and block the file got only 14% smaller.

[NeuroticFish]

Is 500GB a lot? 5TB drives cost less than $100, so $20 for 1 TB. $20 gives you enough space to run a system, some basic software and have enough space to run full node at least until 2025.

man, you're talking about HHD right? it is slow as fuck.

You can keep the block data on the "slow as fuck" HDDs and only the chainstate (and indexes if you have them) on a small SSD. And it's already very fast.
Even more, you no longer need ultra-speed after the IBD, so you can move then everything onto HDD. That's what I did not long ago I had to do an IBD (external HDD is not so great when one doesn't have UPS).

Compressing data into a format like .zip or .tar can roughly decrease file sizes by a factor of 1/3rd.

500 / 3 = 166 gigabytes.

While I do not support most updates or amendments to bitcoin core, I do wonder if compression of file sizes may become necessary at some point to maintain node support.

Did you actually try this out? I would be overly surprised if you get for real these numbers. Plus, as LoyceV said, your math is wrong.

[AverageGlabella]

If anyone really cares the new price everywhere I can see here in the US for a 2TB Samsung 870EVO SSD is $149.99
Amazon, newegg, Microcenter, wherever it's $150.
These are all brand new sealed in a box drives.

Just about EVERY 4TB spinning drive is under $100 until you get into the higher end ones which you DO NOT NEED for this. Yes, your IBD might take a bit longer if the rest of your system is good enough. But if there is ANY other bottleneck be it RAM, CPU, download speed, then the drive speed does not matter.

Back to what I said back in November higher hour used 1TB and 2TB drives are being given away by a lot of people since they have NO real value anymore.

-Dave

You might not need a 4tb hard drive now but would it be better to get one if you plan on running a full copy of the blockchain to future proof? It would be cheaper to buy a bigger hard drive now then having to upgrade your 1TB in the next years. SSD would be a waste of money they are only needed for running operating systems to increase the speed but for storing the blockchain a HDD is the best option.

[BlackHatCoiner]

You might not need a 4tb hard drive now but would it be better to get one if you plan on running a full copy of the blockchain to future proof?

The blockchain, in gigabytes, is less than 600. I know that it's growing more abruptly than a few years ago, but 2 TB is enough for another 30 years according to NotATether (who's proved a little bit inaccurate, but even if you take 10 years, it's still sufficient). 

This philosophy is similarly applied when setting transaction fees manually.

What does transaction fee have anything to do with running a full, non-pruned node in an HDD? What philosophy?

[franky1]

lets assume that (unlike a certain groups desires) that we do not grow the blockchain by just 60gb a year
nor wait 30 years before chaing parameters of the dev politic code cludge preventing scaling

lets assume we finally convince devs to remove some code cludge and have full utility of the 4mb space per block, (they have said for YEARS is network safe way way way before certain peoples desires of a "wait patiently" for 30 years(facepalm))
(full utility of 4m safe space is not a big ask)

1mb=52.5gb so 4mb is 210gb

lets say we round of the year being patient and get to about 500gb by year end 2023
and start afresh uncludgy in 2024 with better transaction count capability in 2024

december 2024 =710gb
december 2025 =920gb
december 2026 =1.13tb
december 2027 =1.34tb
december 2028 =1.55tb
december 2029 =1.76tb
december 2030 =1.97tb

by 2030 people will be wanting to upgrade their computer again anyway.. obviously owning a 2020 computer that has 2tb(average expectation) and having it already for a couple years and still good until 2030 it would be about time to upgrade anyway

[larry_vw_1955]

Back to what I said back in November higher hour used 1TB and 2TB drives are being given away by a lot of people since they have NO real value anymore.

-Dave

people still sell sub terabyte hdds on ebay and no they're not giving them away for free. depending on the hdd model, people might really cough up some big bucks for 1 or 2TB USED hdd. i've seen it.

[franky1]

Back to what I said back in November higher hour used 1TB and 2TB drives are being given away by a lot of people since they have NO real value anymore.

-Dave

people still sell sub terabyte hdds on ebay and no they're not giving them away for free. depending on the hdd model, people might really cough up some big bucks for 1 or 2TB USED hdd. i've seen it.

if your paying premium for a used hard drive(sub 1tb).. at a higher price than a new hard drive with more capacity(retail 2tb) and where if on the same site you admire... then find a 1-2tb USED which is also bigbucks..  your searching the wrong listings

maybe its time you stop using ebay,  searching for particular models, as the models you see on ebay are premium

its much the same as a games consoles
people can buy new games consoles at retail stores cheaper then some ebay listings..
the fault lays in people search ebay for games consoles

[AverageGlabella]

Student got 1 terabyte gdrive when login in using college email. and maybe more than 5 terabytes when you have enough shrewd to modify your friend email using Cbackups software.

Hosting anything that is not student related on that cloud account is against their terms of service and could mean that the cloud storage gets terminated. It is only limited to the time that the person is a student to so you would have to think about long term solutions even if you could host the blockchain in the student cloud that a couple of years. It is not a permanent solution but a cheap 1tb, 2tb, 4tb hard drive is a permanent solution.

[Jason Brendon]

You can keep the block data on the "slow as fuck" HDDs and only the chainstate (and indexes if you have them) on a small SSD. And it's already very fast.

Interesting. didn't know that before. Thanks! For the 'chainstate' (and indexes if you have them), where can i find them and what are these folders or files exactly? 

So, basically are you meaning that after the IBD, it is okay to move the blockchain data to a HDD (keping the indexes in SSD)?
Thanks!

[NeuroticFish]

For the 'chainstate' (and indexes if you have them), where can i find them and what are these folders or files exactly?  

I don't know what they contain. Files  
chainstate and indexes are folders in the data folder, usually at the same level with blocks.
But it depends if you use standard settings/config or you've already set up something, since there are config settings for them.
You may want to read:
https://en.bitcoin.it/wiki/Data_directory
I don't know the settings for those folders, I've always used symlinks.

So, basically are you meaning that after the IBD, it is okay to move the blockchain data to a HDD (keping the indexes in SSD)?

What I meant was that the blocks can always stay on HDD (including during IBD!). And after the IBD even having everything on HDD no longer hurts that much.

[Jason Brendon]

For the 'chainstate' (and indexes if you have them), where can i find them and what are these folders or files exactly?  

I don't know what they contain. Files  
chainstate and indexes are folders in the data folder, usually at the same level with blocks.
But it depends if you use standard settings/config or you've already set up something, since there are config settings for them.
You may want to read:
https://en.bitcoin.it/wiki/Data_directory
I don't know the settings for those folders, I've always used symlinks.

So, basically are you meaning that after the IBD, it is okay to move the blockchain data to a HDD (keping the indexes in SSD)?

What I meant was that the blocks can always stay on HDD (including during IBD!). And after the IBD even having everything on HDD no longer hurts that much.

ok. a bit of confusion here. The purpose is to prove that using HDD is not losing too much comparing using SSD.

So, you are saying that "blocks" directory can be stored on a hdd, but "chainstate" directory should be stored on ssd to not to lose any speed.

[ABCbits]

Compressing data into a format like .zip or .tar can roughly decrease file sizes by a factor of 1/3rd.

500 / 3 = 166 gigabytes.

While I do not support most updates or amendments to bitcoin core, I do wonder if compression of file sizes may become necessary at some point to maintain node support.

Did you actually try this out? I would be overly surprised if you get for real these numbers. Plus, as LoyceV said, your math is wrong.

Some time ago i did that, although using Bitcoin Signet (similar with testnet, except SegWit is always enabled).

Depending on capacity of your external storage, you could try compress all files/folders into .rar or .7z file with maximum compression. I did quick experiment with blockchain data for Bitcoin Signet (similar with Bitcoin testnet) where ~798MB folder compressed into ~505MB .7z file.

The result isn't bad, but we know strong compression/decompression process require more CPU/RAM.

Student got 1 terabyte gdrive when login in using college email. and maybe more than 5 terabytes when you have enough shrewd to modify your friend email using Cbackups software.

Hosting anything that is not student related on that cloud account is against their terms of service and could mean that the cloud storage gets terminated. It is only limited to the time that the person is a student to so you would have to think about long term solutions even if you could host the blockchain in the student cloud that a couple of years. It is not a permanent solution but a cheap 1tb, 2tb, 4tb hard drive is a permanent solution.

Unless you claim your study/research is related with Bitcoin . Although AFAIK storing file which violate copyright is more risky.

[NeuroticFish]

Depending on capacity of your external storage, you could try compress all files/folders into .rar or .7z file with maximum compression. I did quick experiment with blockchain data for Bitcoin Signet (similar with Bitcoin testnet) where ~798MB folder compressed into ~505MB .7z file.

The result isn't bad, but we know strong compression/decompression process require more CPU/RAM.

It's much more impressive than I expected. 7z is probably also more powerful than what was previously proposed, depending on the compression level of choice, still, I'm impressed.
Does the node need the entire blockchain "at hand" in order to validate transactions, or it has the UTXOs somewhere else? Because if it needs to continuously unpack old blocks then indeed the computing power spent probably not worth it. And if one wants to save space he can just use pruned node.

[ABCbits]

Depending on capacity of your external storage, you could try compress all files/folders into .rar or .7z file with maximum compression. I did quick experiment with blockchain data for Bitcoin Signet (similar with Bitcoin testnet) where ~798MB folder compressed into ~505MB .7z file.

The result isn't bad, but we know strong compression/decompression process require more CPU/RAM.

It's much more impressive than I expected. 7z is probably also more powerful than what was previously proposed, depending on the compression level of choice, still, I'm impressed.

To be specific, that 7z use LZMA2 algorithm which is known to be slow.

Does the node need the entire blockchain "at hand" in order to validate transactions, or it has the UTXOs somewhere else?

No, Bitcoin Core has directory called chainstate which contain all UTXO. That's why people say pruned node still perform full verification.

Because if it needs to continuously unpack old blocks then indeed the computing power spent probably not worth it.

It's also not worth if you need to look up for specific transaction or block frequently, which is needed when you run self-hosted block explorer or Electrum server.

And if one wants to save space he can just use pruned node.

People also could try built-in compression feature on the disk format. IIRC NTFS and BTRFS have such feature.

[AverageGlabella]

Unless you claim your study/research is related with Bitcoin . Although AFAIK storing file which violate copyright is more risky.

Maybe but I think you have to be registered with a college to qualify for the student cloud? You have to show proof of enrollment too. AFAIK that is how it works.

[franky1]

Unless you claim your study/research is related with Bitcoin . Although AFAIK storing file which violate copyright is more risky.

Maybe but I think you have to be registered with a college to qualify for the student cloud? You have to show proof of enrollment too. AFAIK that is how it works.

Honestly i don't know. I simply assume collage would just create bunch of google student account at once and send authentication detail to all students through email.

a lil bit of research reveals
students get a college-uni email address
then
when signing up to google they have to give them their college-uni address and an email confirmation is sent to students uni-college email

..

anyways this topic had started with the OP saying he had a 10 terrabyte hard drive and wondering why it was taking 3 days to sync

its been noted several times its not due to the hard drive. with multiple examples why..
. and that includes that a ~774,000 blockheight of ~450gb would be even at 10 blocks per second only be 24 hours and thats on a slow day. but where even generic hard drives can handle way way more then that.

thus debunking the hard drive being bottleneck theory
thus not a storage amount problem or a write-read speed problem

even 6-13mb a second is way less than the limits of even old hard drives
(plus all the other blk rev file stuff ontop still doesnt exceed hard drive tolerances)

.. all that said i do find it funny how people went into a whole tangent to try to get someone to buy some majorly expensive new server-grade hard drive and pretend bitcoin wont be good unless large server cost investment went into it (facepalm)

however its more likely a bandwidth (intermittent internet) issue

ram, also deals in GB amounts not megabyte amounts
also even if ram is over utilised the pc can use hard drive as virtual memory (page file swap files) but again they are dealing with GB amount of data store/processing but only need a few mb per second utility.

the silly delays in syncing i see today happen more often by people using slow network protocols like Tor and VPN that have the worse ping rates and bandwidth speeds even if your device is linking to an ISP at highspeed
its the propagation/relay between nodes that slow the momentum down

i say this because
if hard drives can tolerate terrabytes of data
hundreds of mb of read-write speeds

ram deals in gbs of data and even using virtual memory hundreds of mb of read-write speeds per second

the only low baring fruit that is probably going to be only a few mb/s would be the internet. especially if using tor-vpn bad ping rate utilities

and searching OP post history, they like to talk about utilising tor alot. thus i feel that is their actual issue

but i still find it funny how people are trying to make it look like bitcoin cant function on hard drives bought circa 2018-2023 and pretending everyone needs to by some high spec hard drive that only servers use

[LoyceV]

People also could try built-in compression feature on the disk format. IIRC NTFS and BTRFS have such feature.

I would strongly recommend not to do that. Disk compression is a thing from the past, when disk space was scarce and expensive.
Even if you manage to reduce Bitcoin Core's data size by 20%, you'll go from 500 to 400 GB. That's 100 GB less, which is worth like $2. If you're that low on disk space, just prune your node and save 490 GB. Just imagine you need to use recovery software on a compressed disk.