Congratulation, Bitcoin has reached 500 GB size hard disk data

[franky1]

alot of people are still scratching at their devices wishing to say how that the blockchain is "huge" and cant scale and how blockchains are not fit for function

you know the types of people. that still want to pretend technology is in the 1990's

heres a prospective for how huge 500gb is
500gb.. double it. and then you have



its not a case of 500gb is "huge"
its that something that can secure hundreds of millions of peoples value, and 14 years history of transactions can all fit in the palm of your hand

[1714690563]

1714690563

[Sarah Azhari]

This is a really well done job by the Bitcoin devs. Really good job.

I'm not sure what the developer did until increased my HD data?.

So can we small that blockchain data?, 
like WinRAR, we can compress the data from 1 GB to 200 MB, maybe we can compress it from 475 Gb to 95GB, and node still we can use for running and transactions on the bitcoin core.

[nullama]

~snip~
I'm not sure what the developer did until increased my HD data?.

So can we small that blockchain data?, 
like WinRAR, we can compress the data from 1 GB to 200 MB, maybe we can compress it from 475 Gb to 95GB, and node still we can use for running and transactions on the bitcoin core.

Bitcoin devs make transactions and the data associated with it as compact as they can.

Also they make it so that processing the data is fast, so that the end result is that with updated software you can get faster and faster processing speeds of the whole blockchain.

Every new version of Core you download usually is faster than the previous one.

There is a limit of how much you can compress data in a lossless way. That means that if there's redundant data, sure, that can be compressed a lot, but if all the data or most of it, is vital, then you can't really compress it a lot.

For example, if you have one million zeros in a file, you can compress it by a rule that encodes "write a million zeros here". You'll have a huge compression rate.

On the other hand, if you have random numbers that follow no pattern, you have to just write them down, without any possible compression.

Bitcoin devs are making a good job of keeping the data structures as small and efficient as possible.

[LoyceV]

i heard that western digital is thinking of coming out with a hard drive that really is a tape drive inside.

That's not targeting the consumer market. Tapes are for backups, not for data you use.

Consumer response: That's €159 for something Electrum does for free.
Micro SD cards aren't really meant for blockchain storage, and even though it can work, I don't think many people do it.
Years ago, budget laptops came with 512 GB HDD. Nowadays they come with less\: 128 or 256 GB SSD/NVMe storage. The storage is much faster, but the small size doesn't work for downloading the blockchain. Many are still sold with 4 or 8 GB RAM. The more expensive models come with 16 GB RAM (which often is the maximum) and 512 GB or 1 TB. But the average consumer won't buy them. They also won't upgrade their storage capacity.
It's actually one of the reasons I'm still using my old laptop: it fits 2 disks.

There is a limit of how much you can compress data in a lossless way. That means that if there's redundant data, sure, that can be compressed a lot, but if all the data or most of it, is vital, then you can't really compress it a lot.

Blockchain data is more or less random data, which is hard to compress. I just tested it:

Code:

bzip2 --best blk02506.dat

The compressed file got 15.58% smaller. That's really not worth making the data inaccessible.

[LoyceV]

I know microSD is just example of storage with big capacity, but average microSD have poor I/O speed which hardly suitable to store/manage Blockchain data and other file (e.g. UTXO index).

I haven't tested it, but I expect that exchanging my HDD for a high-end micro SD card wouldn't make it any slower. It will depend on the setup though: I currenly only have my blocks directory on the HDD, not chainstate.

[DaveF]

I know microSD is just example of storage with big capacity, but average microSD have poor I/O speed which hardly suitable to store/manage Blockchain data and other file (e.g. UTXO index).

I haven't tested it, but I expect that exchanging my HDD for a high-end micro SD card wouldn't make it any slower.

If we're talking about high-end microSD, it's random read/write speed could surpass HDD sometimes. But i was talking about average microSD which has quite lower price per GB.

It will depend on the setup though: I currenly only have my blocks directory on the HDD, not chainstate.

Setup which consist SSD and HDD is fairly common, but combination of SSD and microSD is very unusual (outside Raspberry Pi).

I can see it happening more and more on budget setups. There are a lot of laptops and PCs out there on the used market with smaller SSDs. Remember 6 years ago the 6th gen Intel processors were new and a lot of machines were coming with 256gb SSDs. Small by today's standards but faster then spinning drives and more then enough for an office PC. Around here they are all over the used market for under $200 with a warranty from a store and under $100 on the Craigslist / Facebook marketplace from people.

The SSD is not going to be big enough for the blockchain. BUT, just about all of them have SD slots. So any 1TB microSD with an adapter to regular size and you are done.
Yes it's close to $100 to get a 1TB card vs. $50 to replace the drive but not everyone is comfortable taking apart a PC. Sliding an SD card into a slot it easy....

Don't know if there are going to be a lot of people doing it. But once again for ~$250 a full node with room to grow. And it's a useful PC when you want to upgrade to something else.

-Dave

[franky1]

my point is that those people who shout "blockchain huge" and "bitcoin not fit for purpose"
are the ones that then reference some windows vista era hardware,
you know sata 1 or even IDE cable hard drives

where we all know solid state memory is faster, even on "average" priced SSD/ micro media cards

and no. its not a opportunity to show high spec hard drive(sata 3) vs low spec microsd

[Sarah Azhari]

I know microSD is just example of storage with big capacity, but average microSD have poor I/O speed which hardly suitable to store/manage Blockchain data and other file (e.g. UTXO index).

I haven't tested it, but I expect that exchanging my HDD for a high-end micro SD card wouldn't make it any slower.

If we're talking about high-end microSD, it's random read/write speed could surpass HDD sometimes. But i was talking about average microSD which has quite lower price per GB.

is it right if use micro SD express with PCIe technology inside? because when using that new technology, we can write data about 1 GB/s.
So if that micro SD is good to copy all data bitcoin, it will be very convenient and efficient to bring it on everywhere and anywhere if use it on different PC.

https://hexus.net/tech/news/storage/127829-microsd-express-integrates-pcie-nvme-985mbs/

[larry_vw_1955]

That's not targeting the consumer market.

An embedded tape would still be more expensive than a normal one (LTO-9 tapes retail for about $130 a pop) because of the extra electronics but you don’t need a tape drive to get started. As long as it sits somewhere between tapes ($4 per TB) and enterprise hard drives ($20 per TB), there will be a significant market for it.

i think price (ultimately) determines who ends up adopting it. but consumers pay $20 per TB for HDDs all day long don't they?

Tapes are for backups, not for data you use.

but if it was faster then that becomes less of a rule right?

What this alludes to is the intriguing possibility of getting the basic components of a tape drive merged with the actual tape media in a bid to reduce the inherent environmental and technological complexity of tape libraries as well as improving the access time by at least one order of magnitude.


how fast do you need to access your data?

Most likely no, but it depends on the performance. Current LTO tape currently is designed and only have great performance for linear/batch data. But otherwise it perform poorly since it need wind/unwind the tape.

but the blockchain is kind of like a linear structure in that blocks are ordered in a time manner one after the other. older blocks come first. seems like tape would be perfect for that. you don't need to access older blocks as often as you do newer ones probably. for a casual user of bitcoin core that only needs to do a transaction every so often, i don't see how access time would be a huge deterrent to using a tape drive.

[LoyceV]

So if that micro SD is good to copy all data bitcoin, it will be very convenient and efficient to bring it on everywhere and anywhere if use it on different PC.

How often do you want to use your own Bitcoin Core installation on someone else's computer? I'd never do that, just like I don't check my email on systems that aren't mine.

i think price (ultimately) determines who ends up adopting it. but consumers pay $20 per TB for HDDs all day long don't they?

It's not about the price, it's about the use case:

Tapes are for backups, not for data you use.

but if it was faster then that becomes less of a rule right?

Tape will never be fast when it comes to random reading. One order of magnitude faster is still very slow.
I see another problem with this: tape should be stored in a safe location, which means physically removing it from the system after the backup is done.

but the blockchain is kind of like a linear structure in that blocks are ordered in a time manner one after the other. older blocks come first. seems like tape would be perfect for that. you don't need to access older blocks as often as you do newer ones probably. for a casual user of bitcoin core that only needs to do a transaction every so often, i don't see how access time would be a huge deterrent to using a tape drive.

Use prune, it solves all those problems.

[ABCbits]

--snip--

is it right if use micro SD express with PCIe technology inside? because when using that new technology, we can write data about 1 GB/s.
So if that micro SD is good to copy all data bitcoin, it will be very convenient and efficient to bring it on everywhere and anywhere if use it on different PC.

https://hexus.net/tech/news/storage/127829-microsd-express-integrates-pcie-nvme-985mbs/

PCIe is just the port, the actual speed depends on microSD itself and most microSD isn't fast enough to utilize maximum speed on PCIe port.

Most likely no, but it depends on the performance. Current LTO tape currently is designed and only have great performance for linear/batch data. But otherwise it perform poorly since it need wind/unwind the tape.

but the blockchain is kind of like a linear structure in that blocks are ordered in a time manner one after the other. older blocks come first. seems like tape would be perfect for that. you don't need to access older blocks as often as you do newer ones probably. for a casual user of bitcoin core that only needs to do a transaction every so often, i don't see how access time would be a huge deterrent to using a tape drive.

On theoretical level, it's true blockchain has linear structure. Bun on implementation level, there are many random read/write such as read/update index file and read certain UTXO/block.

[franky1]

ok some people are saying about data rate of read/writes

so lets say we have 773377 blocks.. and over the 14 years average block is about 1mb (remember we had hundreds of thousands of blocks under 1mb in early days

now lets say we put a sync time of 24 hours
24 hours is
1440 minutes which is
86400 seconds

which is 8.95 blocks per second ..so about 9mb a second (rounded)

so when saying 'but ssd is more then platter', 'but both are less than tape'..
..it doesnt really matter much because ultimately if you take away the download time and the validation time..
 it takes the amount of data per second to read/write, less time... meaning there is alot of idle time between writes in that 9mb/sec
so because there are plenty of gaps between writes per block every ~0.1sec.. (due to the other tasks happening unrelated to writes happening between each block) there is no harm or over use of a data store even if said data store is cheap

[NotATether]

Consumer response: That's €159 for something Electrum does for free.

But you know that Electrum cannot function without these data stores slotted into full nodes in the first place.

We're not asking people to run full nodes - we have enough of those for now. But more SPV and hardware wallets is absolutely a must.

Besides, I wouldn't gloss too much over storage capacity as ultimately it's the read/write speeds that matter the most.

[franky1]

i was actually being favourable with my rounding of numbers to the hard drive doomsayers.. whilst still debunking them, i

however if you really wanna go full on best case if you compute it more accurately
the read-write data per second even with rev, blk and all other files. the data is still not even close to the basic read-write rate of cheap data storage

for instance the 14 years of blockchain is actually closer to 0.65mb per block average
meaning even handing 10 blocks a second. in octuple the file saving(im going extreme now)
is still only 48mb a second.. far below standard storage media capability and still able to sync within 24 hours

yes lopp was showing how core can sync in 7 hours ..

my point is the other type of people saying bitcoin cant function, hard drives are bad, bitcoin bad, bitcoin takes weeks.. needed to be debunked. but fairly so they cant come back crying with new absent theories

and so, even on the most cheap boring hard drive a sync can be done with no storage device burnout in way under 24 hours, no special equipment or expense needed. (as lopp also shows)

but i didnt want to be stating the overly efficient numbers i was trying to be fair to all parties by not being the optimistic "7 hours" guy

[LoyceV]

i was actually being favourable with my rounding of numbers to the hard drive doomsayers..

I don't think many people here are saying it's not possible to sync Bitcoin Core.

yes lopp was showing how core can sync in 7 hours ..

I'm pretty sure it can be much faster too, given powerful enough hardware. Unfortunately, I no longer have access to the 32 core 256 GB RAM dedicated server I was given to play with a few months ago, otherwise I would test it right now.

my point is the other type of people saying bitcoin cant function, hard drives are bad, bitcoin bad, bitcoin takes weeks.. needed to be debunked. but fairly so they cant come back crying with new absent theories

Bitcoin functions, that's a fact. Some people indeed have hardware or configuration problems that makes Bitcoin Core take weeks or longer to sync.

and so, even on the most cheap boring hard drive a sync can be done with no storage device burnout in way under 24 hours, no special equipment or expense needed. (as lopp also shows)

A cheap boring hard drive isn't enough, you'll need RAM and CPU power. Take an old netbook with 1 GB RAM and 1 TB HDD, and you won't be able to sync Bitcoin Core in less than a month.

[nullama]

~snip~
I'm pretty sure it can be much faster too, given powerful enough hardware. Unfortunately, I no longer have access to the 32 core 256 GB RAM dedicated server I was given to play with a few months ago, otherwise I would test it right now.

With a 2020 Mac Mini you have the storage needed and also it syncs the whole blockchain roughly overnight or so.

That's a pretty normal device, more than two years old already, and has zero issues to sync the blockchain.

I don't think there's an issue with the size of the blockchain and the speed of devices.

[Sarah Azhari]

So if that micro SD is good to copy all data bitcoin, it will be very convenient and efficient to bring it on everywhere and anywhere if use it on different PC.

How often do you want to use your own Bitcoin Core installation on someone else's computer? I'd never do that, just like I don't check my email on systems that aren't mine.

not too often, I ever try it on my external HD. and I just used it for downloaded blockchain data, I don't put the main wallet folder inside. Because I have an office with good Internet, and that can spend my money also if I used it on my home with not good internet.

so lets say we have 773377 blocks.. and over the 14 years average block is about 1mb (remember we had hundreds of thousands of blocks under 1mb in early days

now lets say we put a sync time of 24 hours
24 hours is
1440 minutes which is
86400 seconds

which is 8.95 blocks per second ..so about 9mb a second (rounded)

You look good to calculate complicated mathematics, so after 3 days my bitcoin folder size increased from 480 to 481 GB, I am just curious what time the blockchain data reached max 850 GB?

[franky1]

@loyce
the MANY topics talking about blockchain vs hard drive size.. mention the same doomsday about how hard drives and blockchains cant cope.. its the norm expected evolution of the topic of such. so best bite the bullet and state the facts before the fud
 funny part is you start by saying not many say it, then.. YOU go and say it and re-join the club

@sarah
850-481 = 369
imagining a 1.3mb average data load now per block
~ another 283k blocks
so about the 1.05m blockheight mark.. about 5 years from now

[larry_vw_1955]

It's not about the price, it's about the use case:

Well lets say you could get a 10TB embedded tape drive for $50. people would find uses for that. even if the access time was on the order of seconds. i would expect tape to last longer than hard drives too.

Tape will never be fast when it comes to random reading. One order of magnitude faster is still very slow.

Lets say they got it down to 10 seconds access time. I bet some old cdrom drives that people use for running Live CDs has seek times of 1 second no doubt. Yeah it's a bit slow but you learn to deal with it. Same idea here. For all the TB of storage, you would be willing to put up with it, maybe not you in particular but some group of people would no doubt.

I see another problem with this: tape should be stored in a safe location, which means physically removing it from the system after the backup is done.

You're thinking of tape only being used to backup data. Maybe think again if embedded tape drives ever come about and have sub 10-second access times. People would definitely use them to store videos and movies and stuff. maybe even the blockchain. Tape doesn't have to be just for "backing up". Why would it need to be?

I have a feeling too that tape would outlast hard drives. In terms of durability. As evidence of that, look at all the vhs tapes that are from 30 years ago still work. how many hard drives you know that old?  

[Kakmakr]

I don't know why people want to specifically buy SSD drives for this?...... you can go much cheaper and just buy bigger SATA drives for the same price or less. SATA drives are also slower to boot up and slower in retrieving data than SSDs, but will a few milliseconds really make that much of a difference for average users?

You get some pretty good high performance SATA drives within the 1TB to 2TB range, with a lot of cache and good RPM that will be more than enough for the average user.... with an affordable price tag.