SSD vs HDD: what it means for Bitcoin scaling?

[hatshepsut93]

If you know at least a little bit about how Bitcoin works and its scaling problem, you know that it all comes down to the storage capacities of average users who run full node - we can't allow blocks that are big enough to knock those users out of the network.

Now, technology keeps improving, and sometimes people argue that if we went from 700 MB CD's to terabytes of modern storage, it won't be a big problem in the future. Enter SSD - they are quickly overtaking HDD's because they offer much higher speed, but they come with a drawback - they come with much lower capacity than HDD's. This can mean that people of the nearest future might have less storage then they have now, or it won't improve much, and meanwhile Bitcoin's blockchain will keep growing.

I think that any ideas of potential hard forks for increasing blocksize (they resurface every time fees get high) should be forgotten for the next 10 years, until we figure the amount of storage an average user can dedicate to running a full node.

[seoincorporation]

If you know at least a little bit about how Bitcoin works and its scaling problem, you know that it all comes down to the storage capacities of average users who run full node - we can't allow blocks that are big enough to knock those users out of the network.

Now, technology keeps improving, and sometimes people argue that if we went from 700 MB CD's to terabytes of modern storage, it won't be a big problem in the future. Enter SSD - they are quickly overtaking HDD's because they offer much higher speed, but they come with a drawback - they come with much lower capacity than HDD's. This can mean that people of the nearest future might have less storage then they have now, or it won't improve much, and meanwhile Bitcoin's blockchain will keep growing.

I think that any ideas of potential hard forks for increasing blocksize (they resurface every time fees get high) should be forgotten for the next 10 years, until we figure the amount of storage an average user can dedicate to running a full node.

The storage for blockchain isn't a problem nowadays, but it will be a problem in the future, i mean, if now the bitcoin blockchain is bigger than 200Gb, then it will become huge in the next years, and only servers will be allowed to handle them.

And about SSD vs HDD, some coins like monero recommend to their users to run the node with an SSD because this way the chain could work better, or at least you will sync it fastest, but we know ssd isn't cheap at all and use them to run a node feels like a waste.

[squatter]

SSD storage is getting cheaper and cheaper. Between that and the use of pruning, I don't think there's much concern here. Efficiently accessing the UTXO set is a much bigger concern. So, transitioning to SSDs should actually be a good thing for scalability. From the Bitcoin wiki:

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

[jseverson]

This can mean that people of the nearest future might have less storage then they have now, or it won't improve much, and meanwhile Bitcoin's blockchain will keep growing.

I think that any ideas of potential hard forks for increasing blocksize (they resurface every time fees get high) should be forgotten for the next 10 years, until we figure the amount of storage an average user can dedicate to running a full node.

I don't think HDDs will be phased out for as long as they have significant advantages over SSDs. They're going to have a solid niche until SSDs are priced competitively, so any person who wants to run a node should be able to easily get their hands on one. They're only going to get cheaper from here on out, so I don't think it's going to be a big issue.

We should definitely figure out the sweet spot, but I feel like compromise is necessary at some point. People should at least talk about it once we start getting full blocks from real adoption. I don't think it's a concern now because fees only really skyrocket when Bitcoin surges.

[pooya87]

size of the blockchain is a problem but it is not the biggest problem we have specially since we have cheap storage nowadays with high capacity and we also have option to prune the blockchain. the bigger problem is the time it takes to verify these blocks, as they get bigger they will contain a lot more sigops which need to be run and verified. we have had improvements in speed with SegWit and will have a huge improvement with Schnorr that can help so i don't think we need to wait 10 years to be able to handle bigger sizes. we already can handle 4 MB blocks! and if we want bitcoin adoption to grow and not become obsolete we need to scale.

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

by the way the problem with UTXO set has nothing to do with block size that OP is talking about, UTXO set can grow even if block size was 0.5 MB since it is the "Unspent Transaction Outputs".

[uray]

I think that any ideas of potential hard forks for increasing blocksize (they resurface every time fees get high) should be forgotten for the next 10 years, until we figure the amount of storage an average user can dedicate to running a full node.

This is the most debatable topic for a very long time, just imagine the amount of money we have to spent for the mining hardware if you are running a mining farm and you think it will be a big hurdle to spend $70 for a 2 TB HDD and $130 for a 1 TB SSD  , what ever gives the best solution must be implemented for the general welfare of  the network. The hard disk price is not the big issue here which is stopping us from scaling, there are other issues which have to be looked at.

[squatter]

SSD storage is getting cheaper and cheaper. Between that and the use of pruning, I don't think there's much concern here. Efficiently accessing the UTXO set is a much bigger concern. So, transitioning to SSDs should actually be a good thing for scalability. From the Bitcoin wiki:

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

by the way the problem with UTXO set has nothing to do with block size that OP is talking about, UTXO set can grow even if block size was 0.5 MB since it is the "Unspent Transaction Outputs".

Who said it had anything to do with block size?

Since the OP was asking about SSD adoption's effect on scalability, I suggested that SSD adoption is actually good for scalability because of their efficiency in accessing the UTXO set. It seemed like a relevant observation...

[pooya87]

SSD storage is getting cheaper and cheaper. Between that and the use of pruning, I don't think there's much concern here. Efficiently accessing the UTXO set is a much bigger concern. So, transitioning to SSDs should actually be a good thing for scalability. From the Bitcoin wiki:

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

by the way the problem with UTXO set has nothing to do with block size that OP is talking about, UTXO set can grow even if block size was 0.5 MB since it is the "Unspent Transaction Outputs".

Who said it had anything to do with block size?

Since the OP was asking about SSD adoption's effect on scalability, I suggested that SSD adoption is actually good for scalability because of their efficiency in accessing the UTXO set. It seemed like a relevant observation...

sorry, i made the connection because this discussion is also about blockchain size which is then affected by the block sizes.

[Wind_FURY]

If you know at least a little bit about how Bitcoin works and its scaling problem, you know that it all comes down to the storage capacities of average users who run full node - we can't allow blocks that are big enough to knock those users out of the network.

Now, technology keeps improving, and sometimes people argue that if we went from 700 MB CD's to terabytes of modern storage, it won't be a big problem in the future. Enter SSD - they are quickly overtaking HDD's because they offer much higher speed, but they come with a drawback - they come with much lower capacity than HDD's. This can mean that people of the nearest future might have less storage then they have now, or it won't improve much, and meanwhile Bitcoin's blockchain will keep growing.

I think that any ideas of potential hard forks for increasing blocksize (they resurface every time fees get high) should be forgotten for the next 10 years, until we figure the amount of storage an average user can dedicate to running a full node.

Storage is just part of the problem. Some would say that it's "not a problem anymore", but the truth is, it is for some users.

The other, bigger part of the problem is bandwidth, not all users have access or can afford the fastest internet connections.

You want real scaling? This is real scaling, https://bitcointalk.org/index.php?topic=5148054.0

[aad140386]

It is likely that in the near future, SSD drives will become more accessible to people and then the problem will go away by itself. Well, or rather, the problem will remain as for desktop wallet storage volume will constantly increase and it is really a problem. It is likely that many will eventually switch to cold wallets like Ledger and this will partially solve the problem. Well, or the problem will be solved in a completely different way and on computers it will not be necessary to store a wallet that will take up a lot of space.

[hatshepsut93]

SSD storage is getting cheaper and cheaper. Between that and the use of pruning, I don't think there's much concern here. Efficiently accessing the UTXO set is a much bigger concern. So, transitioning to SSDs should actually be a good thing for scalability. From the Bitcoin wiki:

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

That's interesting, when I was running a full node, it was putting a ton of pressure on my old HDD's read/write resource, so I can see how SSD can improve it with their superb speed.

Sure, SSD's are getting cheaper, but for now it seems like we've faced a setback - I was recently choosing between SSD and HDD with the same price, and I picked 500 GB SSD over 2 TB HDD, because speed is important for my applications, and I no longer would be able to run full node, because I just can't put the blockchain on SSD, I need that space for other things.

[Wind_FURY]

SSD storage is getting cheaper and cheaper. Between that and the use of pruning, I don't think there's much concern here. Efficiently accessing the UTXO set is a much bigger concern. So, transitioning to SSDs should actually be a good thing for scalability. From the Bitcoin wiki:

The primary limiting factor in Bitcoin's performance is disk seeks once the unspent transaction output set stops fitting in memory. Once hard disks are phased out in favour of SSDs, it is quite possible that access to the UTXO set never becomes a serious bottleneck.

That's interesting, when I was running a full node, it was putting a ton of pressure on my old HDD's read/write resource, so I can see how SSD can improve it with their superb speed.

Sure, SSD's are getting cheaper, but for now it seems like we've faced a setback - I was recently choosing between SSD and HDD with the same price, and I picked 500 GB SSD over 2 TB HDD, because speed is important for my applications, and I no longer would be able to run full node, because I just can't put the blockchain on SSD, I need that space for other things.

You can prune it if you need the space. Open the bitcoin.conf file and type "prune =" without the quotes, and set it to 5,000 for 5 GB. It will only store 5 GB of the blockchain, prune the rest. But you will still need to sync, and validate the whole 200 GB of data.

[bob123]

Storage ? No. Not a problem at all.

There are a lot reasons why increasing the blocksize will harm the state of decentralization of BTC. But storage is not one of them.

You can buy a 1 TB SSD for ~150$ currently. That's enough storage for the next 15 years at least.
Even with a blocksize increase, do you think 1TB SSD's are the maximum for an average buyer ? That the technology will not improve in the next 10-20 years ?

Or one could simply buy a 4TB HDD for ~120$. You don't need the blockchain to be stored on a SSD. A HDD is fast enough for the blocks itself.
What you should store on a SDD is the chainstate folder, which has quite a lot read/write operations happening.

[CryptoReggae]

But is there really anyone who believes that the size of the blockchain is a problem?
Are spending thousands of dollars/euros for the purchase of any type of hardware and there is no money for an HD with the necessary capabilities? say 70/80 €?

The real problem is maintaining decentralization, encouraging, instructing users to maintain a knot, making them understand the importance.
And in an era where smartphones are supplanting PCs, it's important to really understand how essential and essential it is to keep the network neutral.

The size of the BTC blockchain is in my opinion a false problem.

[hatshepsut93]

You can prune it if you need the space. Open the bitcoin.conf file and type "prune =" without the quotes, and set it to 5,000 for 5 GB. It will only store 5 GB of the blockchain, prune the rest. But you will still need to sync, and validate the whole 200 GB of data.

Thanks for the suggestion, I totally forgot about it, and I might actually do it in the future. But the problem is that everyone started pruning, it would increase centralization, because few nodes will have the full blockchain and all new nodes will have to rely on them for the initial sync.

Storage ? No. Not a problem at all.

There are a lot reasons why increasing the blocksize will harm the state of decentralization of BTC. But storage is not one of them.

You can buy a 1 TB SSD for ~150$ currently. That's enough storage for the next 15 years at least.
Even with a blocksize increase, do you think 1TB SSD's are the maximum for an average buyer ? That the technology will not improve in the next 10-20 years ?

Or one could simply buy a 4TB HDD for ~120$. You don't need the blockchain to be stored on a SSD. A HDD is fast enough for the blocks itself.
What you should store on a SDD is the chainstate folder, which has quite a lot read/write operations happening.

I don't think a lot of people are going to buy hardware specifically for running Bitcoin, why spend $150 if you can just run a light client even on your phone instead? And I know there are always Bitcoin enthusiasts who own multiple full node machines, but they are a minority, most people run nodes if they can spare resources on the machines that were bought for other purposes. It's true that the storage is increasing, but the demand for it also increases - programs weight more and more, Bitcoin blockchain is growing.

[bob123]

I don't think a lot of people are going to buy hardware specifically for running Bitcoin, why spend $150 if you can just run a light client even on your phone instead?

Either they do, or they don't.
But the same applies to the current state too.

Why are people running a full node nowadays if they can run a light client on their phone ?

This has nothing to do with storage.

And I know there are always Bitcoin enthusiasts who own multiple full node machines, but they are a minority, most people run nodes if they can spare resources on the machines that were bought for other purposes.

A lot of people already own 1 TB HDDs.
With the increasing popularity of SSDs and their decreasing price.. what will the 1TB HDD's be used of in 5 years ? They won't be worth anything. Might as well use them to store blockchain data if you don't want to throw it away or store them on one of your big future-SSDs.

It's true that the storage is increasing, but the demand for it also increases - programs weight more and more, Bitcoin blockchain is growing.

The 'demand' does not increase at that pace. By far not.

20 years ago you could barely store the operating system on an optical drive.
Now you get 8TB storage for a fraction(!) of the price. And those 8TB are way more than needed for 'programs' nowadays.


Storage really isn't any issue at all.
People who are running a full node today, won't have a problem running one in 20 years. At least not storage-wise.

[Haunebu]

Storage really isn't any issue at all.
People who are running a full node today, won't have a problem running one in 20 years. At least not storage-wise.

You are grossly underestimating how serious of an issue storage really is in this case. In my opinion, this is easily one of the biggest issues with the Bitcoin blockchain. The number of people who are running a full node today are a small minority who are primarily trying to support the Bitcoin network in this case.

Better solutions need to be available in order to help their numbers grow. For example, the Neutrino protocol is helping people support the LN without having to download the entire blockchain.

In my opinion, 1TB SSDs will help in the future once their prices continue to drop with time.

[bob123]

Storage really isn't any issue at all.
People who are running a full node today, won't have a problem running one in 20 years. At least not storage-wise.

You are grossly underestimating how serious of an issue storage really is in this case. In my opinion, this is easily one of the biggest issues with the Bitcoin blockchain.

How ?
If a 1 TB HDD costs ~40$ and has enough space to store the blockchain for the coming ~15 years.. how can you claim that storage is a problem at all 

The number of people who are running a full node today are a small minority who are primarily trying to support the Bitcoin network in this case.

And they can continue to do so.
The extremely low price for storage nowadays (which is drastically decreasing btw) doesn't prevent them from doing so in the future.

[dothebeats]

SSDs are becoming cheaper as years go by, and the storage capacities are also increasing too, so I don't think it'll be much of a problem in the future. But yeah, if the blockchain continues to expand at a rapid pace in terms of size, the miners have to increase their storage capacities as well, and so the cost of mining will effectively increase too. Then again, technological developments on that area isn't stagnating, so I think we're still okay. Just 40 or so years ago, we managed to get by with very little storage spaces compared to what we have today for very important events for Mankind, so we're good, I think.

[NeuroticFish]

how can you claim that storage is a problem at all 

Storage size is, indeed, a non-problem, at least not directly.
Bandwidth is more a bottleneck in many cases (great link, @Wind_FURY) and also the speed of accessing the data is important (and we are back to the data size and HDDs too, somehow). SSD can help, but it doesn't have the lifespan of the HDD and SSD is well, let's say not cheap.

I was certain that the devs already know that scaling has to be improved, still got a nice surprise from reading ^^ that link.
And there's SegWit, the work on LN and I expect that more will be done in the near future, since unlike "the competition", Bitcoin really processes a huge number of transactions already and the numbers are expected to rise exponentially.