Congratulation, Bitcoin has reached 500 GB size hard disk data

[buwaytress]

my point was.
when people shout moores law and say we are at the end

i recall when i was last at the edges of the end of moores law in computing over 25 years ago
.. until it wasnt the end and life moved on

I recall that same period too. The excitement of zip drives (I invested pretty heavily in that dealing with clients and showing them graphics and stuff on the move), the deals I thought were crazy in those days to buy new drives. The memory disks for Playstation that we thought were amazing... this was back when we were doing our best to keep file sizes low, compressing images on websites. Even had all kinds of golden rules for uploading things so people don't die on dialup waiting to load our sites.

Now we're uploading things to the best resolution possible, and my kids are eating up hundreds of gbs of data on their phones daily.

So yeah, I don't think we're at the edge, or that edge is getting farther out than is possible for most.

Big deal 500gb =D

[1714064481]

1714064481

[1714064481]

1714064481

[1714064481]

1714064481

[LoyceV]

Did you actually create bootstrap.dat file (see https://bitcoin.stackexchange.com/q/10381) or just compress Bitcoin Core directory?

I didn't know there's a procedure for it, I just compressed blocks and chainstate.

7z with LZMA2 algorithm could do better job, but i expect it'll be extremely slow.

Even if it saves a few GB, that's only a few percent anyway.

[LoyceV]

That's true. Last time i tried that with Bitcoin Signet, i got compression ratio 0.63 (505MB / 798MB). For comparison, compression you choose have ratio about 0.81 (397GB / ~486GB).

This took over 2 hours on all CPU cores already.

[Coin Gorilla]

That's true. Last time i tried that with Bitcoin Signet, i got compression ratio 0.63 (505MB / 798MB). For comparison, compression you choose have ratio about 0.81 (397GB / ~486GB).

This took over 2 hours on all CPU cores already.

Does faster drive (something like PCI-e 4.0 NVMe or regular even regular SATA SSD) mean anything in this case or is it only CPU bound? I assume that in this case the CPU and core clock (IPC performance too) is the most important aspect?

[LoyceV]

Does faster drive (something like PCI-e 4.0 NVMe or regular even regular SATA SSD) mean anything in this case

No. The CPU is already the limiting factor, stronger compression means there's even less to write (per second).

Back to the growth of the blockchain: the daily amount of data increased a lot in the past week:

[ ]   blockchair_bitcoin_inputs_20230128.tsv.gz   2023-01-29 00:59    205M   
[ ]   blockchair_bitcoin_inputs_20230129.tsv.gz   2023-01-30 00:59    200M   
[ ]   blockchair_bitcoin_inputs_20230130.tsv.gz   2023-01-31 00:59    235M   
[ ]   blockchair_bitcoin_inputs_20230131.tsv.gz   2023-02-01 00:59    221M   
[ ]   blockchair_bitcoin_inputs_20230201.tsv.gz   2023-02-02 00:59    274M   
[ ]   blockchair_bitcoin_inputs_20230202.tsv.gz   2023-02-03 00:59    257M   
[ ]   blockchair_bitcoin_inputs_20230203.tsv.gz   2023-02-04 00:59    238M   
[ ]   blockchair_bitcoin_inputs_20230204.tsv.gz   2023-02-05 00:59    289M   
[ ]   blockchair_bitcoin_inputs_20230205.tsv.gz   2023-02-06 00:59    282M   
[ ]   blockchair_bitcoin_inputs_20230206.tsv.gz   2023-02-07 00:59    363M   
[ ]   blockchair_bitcoin_inputs_20230207.tsv.gz   2023-02-08 00:59    330M   
[ ]   blockchair_bitcoin_inputs_20230208.tsv.gz   2023-02-09 00:59    319M   
[ ]   blockchair_bitcoin_inputs_20230209.tsv.gz   2023-02-10 00:59    366M   
[ ]   blockchair_bitcoin_inputs_20230210.tsv.gz   2023-02-11 00:59    328M   
[ ]   blockchair_bitcoin_inputs_20230211.tsv.gz   2023-02-12 00:59    377M   
[ ]   blockchair_bitcoin_inputs_20230212.tsv.gz   2023-02-13 00:59    447M   
[ ]   blockchair_bitcoin_inputs_20230213.tsv.gz   2023-02-14 00:59    331M   

[Wronsk]

I think that when the technology of data storage has evolved by changing scale, i.e. for example when the price of 10TB is at the current price of 1TB, it may be possible to increase the size of the blocks as this will not harm decentralisation, however we may be close to the theoretical limit in data storage

[franky1]

I think that when the technology of data storage has evolved by changing scale, i.e. for example when the price of 10TB is at the current price of 1TB, it may be possible to increase the size of the blocks as this will not harm decentralisation, however we may be close to the theoretical limit in data storage

(facepalm)
1990's hit the theoretical limit of portable data storage of 1.4mb(floppies)
oh wait CD-rw's
1990's hit the theoretical limit of portable data storage of 700mb(CD)
oh wait DVD-rw's
DVD hit the limit.. oh wait blue ray
an along side all of that flash drives
portable hard drives

so theoretical limit.. is what, and when exactly?
and with a current 210gb bitcoin growth limit.
lets call it 4 years 6 months per terrabyte
how long is this deadline before
if you wish to call 10tb a hard technical limit of storage
thats 46years for geeks to invent new storage options

[AverageGlabella]

I think that when the technology of data storage has evolved by changing scale, i.e. for example when the price of 10TB is at the current price of 1TB, it may be possible to increase the size of the blocks as this will not harm decentralisation, however we may be close to the theoretical limit in data storage

We have not shown any signs of being close to the theoretical limit but we are slowing down. Franky is right that we have evolved very quickly from things that we thought would be with us for a long time. CDs were very popular and people thought that it would be a method of storage for many years but they are now redundant so redundant that people do not have dvd players and do not have a cd tray in their computers.

I think the motivation for consumer storage might be the theoretical limit instead of the actual limit. hard drives wil continue to expand by a lot but they will be aimed at companies that need a lot of storage and cloud storage provides. The average consumer does not use more then 4TB atm and with other technologies slowing down and hitting their limit like games where the hardware is stronger then the things it relies on I think that could be what happens in the next 5-10 years. Consumer level hard drives might continue to be cheap but probably will not be made in smaller sizes corporate ones will probably continue to expand at a fast rate but will be more expensive to purchase.

[nullama]

~snip~
We have not shown any signs of being close to the theoretical limit but we are slowing down. Franky is right that we have evolved very quickly from things that we thought would be with us for a long time. CDs were very popular and people thought that it would be a method of storage for many years but they are now redundant so redundant that people do not have dvd players and do not have a cd tray in their computers.

I think the motivation for consumer storage might be the theoretical limit instead of the actual limit. hard drives wil continue to expand by a lot but they will be aimed at companies that need a lot of storage and cloud storage provides. The average consumer does not use more then 4TB atm and with other technologies slowing down and hitting their limit like games where the hardware is stronger then the things it relies on I think that could be what happens in the next 5-10 years. Consumer level hard drives might continue to be cheap but probably will not be made in smaller sizes corporate ones will probably continue to expand at a fast rate but will be more expensive to purchase.

Yeah, I would agree that whatever is being sold mainstream gives you a good indication of what the average user is going to be expecting to be the maximum storage.

For example, the new Mac mini with the M2 Pro comes with configuration options ranging from 512GB to 8TB. So that gives you an idea of what's mainstream these days. 512GB being the bare minimum to run a system, and if you need some extra space, you can get up to 8TB, without doing any installation yourself.

So, clearly 500GB of Bitcoin data is not too much these days.

[Sarah Azhari]

And another question that came to my mind while reading the OP was , why do i need to download the entire bitcoin blockchain data when I am not a developer? I mean what is the essence actually,

for me is just a sense of pride, I don't know next. maybe when having the time I want to learn how to make a Lighting network node, and earning satoshi from that, or create a wallet and be a developer like you said.

and there is no harm if we have unused internet at home and pc because is not disturb also when running my pc with another program.

actually, I step to download a bit by bit, certainly not so felt long if in the future I make the blockchain program or etc.

[AmoreJaz]

And another question that came to my mind while reading the OP was , why do i need to download the entire bitcoin blockchain data when I am not a developer? I mean what is the essence actually,

for me is just a sense of pride, I don't know next. maybe when having the time I want to learn how to make a Lighting network node, and earning satoshi from that, or create a wallet and be a developer like you said.

and there is no harm if we have unused internet at home and pc because is not disturb also when running my pc with another program.

actually, I step to download a bit by bit, certainly not so felt long if in the future I make the blockchain program or etc.

well, set aside the pride and all. if you are not a developer, there's no reason for you to download such heavy data. am using electrum right now for my btc holdings. light and txs are fast as well. i was using the btc core before but yeah, it came to a point i ask myself why i was using it. if you have no valid or worthwhile reasons, just go for wallets that are not heavy especially for beginners and just regular users.

[LoyceV]

so theoretical limit.. is what, and when exactly?

When talking about theoretical limits for data storage, I'm thinking about storage on a molecular level. You can't store much more than 1 bit per molecule. The closest thing I could find was 215 PB in 1 gram of DNA, which is still several orders of magnitude away from current storage systems.

We have not shown any signs of being close to the theoretical limit but we are slowing down.

Is that because of technical limits, or because newer storage types focus more on access time than size?

[franky1]

theoretical limit in data storage (in general) or just theoretical limit of current technology used on HDD? Even if you're talking specifically about HDD, theoretically we can see at least 80TB capacity[1].

--snip--

well, set aside the pride and all. if you are not a developer, there's no reason for you to download such heavy data. am using electrum right now for my btc holdings. light and txs are fast as well.

I'll have to remind privacy concern is also valid reason to run full node. For example, you don't want random electrum server know list of address belong to same person.

[1] https://www.pcmag.com/news/next-gen-hamr-platters-promise-80tb-hard-drives

its also not relying on one source supply of balance accounting
imagine if a litewallet didnt tell you a payment arrived at your address. reliance on a litewallet means you only know what they choose to tell you.
they can send you a tx, say its confirmed. but the utxo does not taint/track back to any coin mined nor any real coin existing on a blockchain. should they go down a deceitful route

running a full node means you get to see it all and know all coins have a track/taint back to their origins of creation(coinbase reward) that validates that all coins are real coins and not ghost coins made up

yes not everyone needs to know everything. but if your receiving regular payment from alot of random people.. just seeing a tx designated for you is not enough, you will want to ensure it is confirmed and has valid funding before releasing your part of the deal, 
because the more payments you receive the more risk of abuse if you did operate without a non independent checking method

[Sarah Azhari]

--snip--

well, set aside the pride and all. if you are not a developer, there's no reason for you to download such heavy data. am using electrum right now for my btc holdings. light and txs are fast as well.

I'll have to remind privacy concern is also valid reason to run full node. For example, you don't want random electrum server know list of address belong to same person.

Maybe AmoreJaz means if don't run a full node, and just use the core wallet just store the fund.
Sure, I was thinking before making the decision to download bitcoin core. because the core wallet is complicated, can not be used for transactions before the blockchain is completed, But cmon, core wallet is from satoshi nakamoto legacy, same as this forum we live. there is a sense of pride if we talk it on seminar or other forum if we used what satoshi create, bitcoin, core and bitcointalk forum.

[LoyceV]

the core wallet is complicated, can not be used for transactions before the blockchain is completed

For the record: you can send a transaction before Bitcoin Core is fully synced. The moment an input shows up in Bitcoin Core, you can send it. The tricky part is of course that it might have been spent already, and in that case your transaction will indeed be rejected by other nodes.

[franky1]

the core wallet is complicated, can not be used for transactions before the blockchain is completed

For the record: you can send a transaction before Bitcoin Core is fully synced. The moment an input shows up in Bitcoin Core, you can send it. The tricky part is of course that it might have been spent already, and in that case your transaction will indeed be rejected by other nodes.

if your copying over a whole wallet file that also contains UTXO details of your spends. then yes. it can pop up instantly when initiating core, and will appear as pending confirmation until synced

but if just importing a private key you will have to wait for the section of a sync where by it recognises the association between the key pair to then display it mid sync while its building the blockchain

those that only got involved in say 2020 and just imported a private key/seed key, will still have to wait until 350gb+ is download to get to blockheight sync of 2020+ for their input to show up..

luckily my 2012 stash pops up on GUI display in the first ~100gb*
luckily my 2014 stash pops up on GUI display in the first ~150gb*
luckily my 201x stash's pop up on GUI display in the first ~300gb*
unluckily my 2022 play amount pops up after 400gb

* (im hoarding thus dont want to spend these)

[LoyceV]

luckily my 2012 stash pops up on GUI display in the first ~100gb*
~
* (im hoarding thus dont want to spend these)

Are you using a hot wallet for long-term holding?

[franky1]

luckily my 2012 stash pops up on GUI display in the first ~100gb*
~
* (im hoarding thus dont want to spend these)

Are you using a hot wallet for long-term holding?

my old stash is on legacy
.. am i doing a LukeJR.. nope

i have a few other features on my node i added myself so i just do "watch only" for the old stash thus not importing private, but importing public address just as a balance display

but if it was time to spend them. then yes i treat it as a hot wallet meaning get a (offline) new set of keypair. to use only public address as change address. and then spend old private key to destination address+ to the new public address(change) i import. and then wipe drive. and redo it all again.

its a hassle to spend old stash but that hassle is avoided by making it another reason to not wanna spend old stash.. making it complicated avoids easy temptation to spend

i have not put my private key of old stash in my node..ever

my 1frank funds are not my main stash. the 1frank funds was/are just a side 'dont give a crap' play money small fund. and i havnt bothered with that private key for 5 ish years which was the last time i had to do the hassle method of wiping hard drive and starting again due to treating it as a hot wallet in relation to those funds.
newer play money amounts i have(different keys) as display of balance using public keys on the node. but spend on a different device thats easier to wipe and start again.(lite wallet)

[larry_vw_1955]

luckily my 2012 stash pops up on GUI display in the first ~100gb*
luckily my 2014 stash pops up on GUI display in the first ~150gb*
luckily my 201x stash's pop up on GUI display in the first ~300gb*
unluckily my 2022 play amount pops up after 400gb

wow franky that's alot of stashes! you were smart to get involve in bitcoin so early 

[franky1]

luckily my 2012 stash pops up on GUI display in the first ~100gb*
luckily my 2014 stash pops up on GUI display in the first ~150gb*
luckily my 201x stash's pop up on GUI display in the first ~300gb*
unluckily my 2022 play amount pops up after 400gb

wow franky that's alot of stashes! you were smart to get involve in bitcoin so early  

luck.. not smarts
its a learning process. dont stop learning

i dont need to keep buying im self sustainable and live a good life thanks to bitcoin
i choose to buy more when there are undeniably cheap dips worthy of grabbing more

im no where near the desire to just sell it all. so i dont
its not "smart". its just not needing to.
yea i use small play money amounts as a general living/short term income accumulator but its like 0.0X% 0.0xxx of stashes so risk is low

but my main goal.. i want to ensure bitcoin functions as a financial payment network to protect my value..  and is not turned into a meme-stock storage network

from a technical side of data storage growth of 210gb a year is not disastrous growth. but how that data is used, is a disaster if majority of it is just memes. memes is very inefficient use of the available data growth and it limits/decreases of the financial utility of bitcoin