Bitcoin Core blockchain impossible to download now, too slow and too big

[v1.0]

As a newcomer to Bitcoin Core. I find the Bitcoin Core blockchain impossible to download now, it is growing exponentially and is just too slow and too big for the average person. What's the point? I wanted to have the entire blockchain on my computer, but it ain't happening. Here's why:

1) Bitcoin has been in existance for almost 10 years. But 50% of the entire blockchain is just from the last year or so! I mean, I'm at 30% downloaded right now, and that's right up until November 2015!

2) If your hard drive screws up, you're screwed! I was three weeks into the Bitcoin Core blockchain download, made it to 90%. Then, BAM! My hard drive died. Got a new drive, decided to try again. Seems slower than ever, my computer runs to a crawl at times.

3) Even if someday I do manage to catch up to the most current block/day, I believe soon if not yet already there, we will never be able to get past a certain percentage, say 98%, due to the ever-growing nature of the blockchain. It will get to the point where it is growing faster than than possible to download.

4) Bitcoin is not new-user friendly! Basically, if you missed this boat, the only other one to hop on are the life rafts (exchanges) circling around the boat like a body of sharks. Anyone that the blockchain download after 2016 I bet put in a lot of wasted time and resources trying.

5) Windows automatic updates really, really screw things up. With Bitcoin Core download, there is no time that your computer is idle for the updates, yet Windows thinks it is. So it kills the process with the reboot.

6) Actually, any sort of blip whatsoever in your Bitcoin Core client or storage device causes havoc. Its not like a few years ago, where you're looking at 20 gigs. We're dealing with over 100 gigs now. What happens when this reaches a terabyte? Not too far off, watch.

So, although it is enticing to attempt having the entire blockchain for failsafe transactions, it simply isn't feasible anymore. What happens now? Well, one could say Bitcoin should just adapt permament light wallets ala Electrum using multiple redundent cloud-based storage for the blockchain. Question is, who pays for it? Easy. Have silicon-based chip storage, with a percentage coming off transaction fees for the maintenance. Except then you have individual governance, which does not work with the Bitcoin system.

Thoughts?

[1714171477]

1714171477

[bitperson]

How does the blockchain grow exponentially with one 1 MB block added, on average, every ten minutes? Isn’t that linear growth? Never mind. You can speed up synchronisation by putting the data directory, or at least the chainstate, on an SSD. Also, connection speeds, storage capacity and other such metrics keep improving as hardware evolves. And you can put Windows Update on manual, or ditch Windows altogether.

[v1.0]

How does the blockchain grow exponentially with one 1 MB block added, on average, every ten minutes? Isn’t that linear growth? Never mind. You can speed up synchronisation by putting the data directory, or at least the chainstate, on an SSD. Also, connection speeds, storage capacity and other such metrics keep improving as hardware evolves. And you can put Windows Update on manual, or ditch Windows altogether.

Thanks for reply.

The 1MB block limit is linear, sure, but thats related to mining. The transactions are what make it exponential. No limit on ledger size. How else to explain 3 times growth in just last 2 years with a 9 year old platform?

The hardware must evolve to keep up. Thanks for hardware tips- and I agree Windows is terrible for bitcoining. Also running a Linux server on SSD, but using that one for BiblePay

--very new to this mind you. Just so discouraged at how difficult its proving to get into Bitcoin Core. I didn't want to just be a holder of the coins- anyone can do that. I much prefer to have the source and contribute to the community.

[emuLOAD]

Unless you're on dialup, you'll never have problems staying up to date. Keep in mind the first few years we were nowehere near filling blocks, which means the chain grew less than it could have. nowadays we are closer to saturation, so growth is a lot more linear.

As far as bootstrapping goes, it's insane to download the Whole thing from scratch through the core client. Look for a bootstrap torrent, and sync from there. yes, first start will be slow, but it won't take nearly as long.

All this said, nobody's ever said you need to run a full node. Wallets like Electrum, especially if combined with Trezor or other hardware wallets, are safe enough--indeed more in some cases--than running a full node.
there is no need to run a full node, and not doing so won't make you anybody's slave.

If you want core and contribute, you have more studying to do, because you simply havn't learnt enough yet to be useful.

[v1.0]

Unless you're on dialup, you'll never have problems staying up to date. Keep in mind the first few years we were nowehere near filling blocks, which means the chain grew less than it could have. nowadays we are closer to saturation, so growth is a lot more linear.

As far as bootstrapping goes, it's insane to download the Whole thing from scratch through the core client. Look for a bootstrap torrent, and sync from there. yes, first start will be slow, but it won't take nearly as long.

All this said, nobody's ever said you need to run a full node. Wallets like Electrum, especially if combined with Trezor or other hardware wallets, are safe enough--indeed more in some cases--than running a full node.
there is no need to run a full node, and not doing so won't make you anybody's slave.

If you want core and contribute, you have more studying to do, because you simply havn't learnt enough yet to be useful.

Thanks for the reply.

Oh yes, lots of studying to be had, and learning on this constantly, so thanks!

So, regarding bootstrapping, actually it's hardly any faster than downloading through the client- this is through my personal experience. Which is really what annoyed me about it all. My Core client has been "processing blocks" for over two days now via bootstrap, it hit 30% and basically stuck at the same speed, even worse, than previously on my old drive through traditional method.

Of course I don't have to run a full node. But imagine if everyone thought this way- "why bother running a full node, I can just grab an Electrum wallet and feed through that." Well, I bet most people do go this route, but as stated in my original post, its better and safer to have the blockchain locally, given the option.

As far as "learning enough to be useful", not sure what you mean by this statement. Even newcomers can be usefull by providing nodes to the community via Core to assist with transaction verifications, even though we can't mine anymore.

I agree there is no need to run a full node, which is really the crux of this thread. Its unfortunate and discouraging, really, that those of us who do are pretty much SOL now. We should be encouraging nodes, but the current state has made it an impossibility to start from scratch, bootstrap or otherwise.

[DannyHamilton]

As a newcomer to Bitcoin Core.

Welcome.

I find the Bitcoin Core blockchain impossible to download now,

You may want to consider a lightweight wallet if your computer specs are not modern enough to handle it.

it is growing exponentially

It is not growing exponentially. It is growing linearly.

and is just too slow

You'll find that a modern computer with enough RAM and an SSD will help a lot.

and too big for the average person.

It is currently less than 157 gigabytes.

An "average person" can purchase a 256 gigabyte SD Card for less than $25 at Amazon.com:
http://a.co/7Wu4ib4

An "average person" can purchase a 1 terabyte external SSD drive  for less than $350 at Amazon.com:
http://a.co/6TtcGMx

An average person can also run Bitcoin Core in "pruning mode", since the "average person" doesn't have any personal need to store the blockchain.

What's the point?

The point to running Bitcoin Core?  There may not be one.  That's for each user to decide for themselves.  You could establish an account with a service such as Coinbase, you could run a lightweight wallet such as Electrum, you could run Core with pruning.  Each user can decide for themselves what best fits their needs.

I wanted to have the entire blockchain on my computer,

Why?

but it ain't happening.

That's ok.  If you can't store the blockchain, then others will.

1) Bitcoin has been in existance for almost 10 years. But 50% of the entire blockchain is just from the last year or so!

You may want to check your math on that.

Blockchain size a year ago was just a bit more than 102 gigabytes.

(157 - 102) / 157 = 35.03%

That's a LOT less than 50%

I mean, I'm at 30% downloaded right now, and that's right up until November 2015!

Correct.  The blocks were not full the first 6 or 7 years of Bitcoin's existence.  There just weren't many people using it yet.

2) If your hard drive screws up, you're screwed! I was three weeks into the Bitcoin Core blockchain download, made it to 90%. Then, BAM! My hard drive died. Got a new drive, decided to try again.

I create regular backups of data that is important to me. That way I don't lose it when my hard drive "screws up".  You may want to try doing that.  You can recover a backup of your hard drive, and then just pick up where you left off.

Seems slower than ever, my computer runs to a crawl at times.

If it is slower now than before purchasing the new drive, then I'm guessing you purchased a drive with lesser specifications.  Perhaps purchase storage with the same OR BETTER specs next time?

3) Even if someday I do manage to catch up to the most current block/day, I believe soon if not yet already there, we will never be able to get past a certain percentage, say 98%, due to the ever-growing nature of the blockchain. It will get to the point where it is growing faster than than possible to download.

The block chain grows at a maximum average rate of 4 megabytes every 10 minutes.  If your internet is incapable up supplying 4 megabytes every 10 minutes (that's less than a 56k dial-up modem) or your computer is incapable of processing 4 megabytes every 10 minutes, then you should probably not be trying to run Bitcoin Core.

4) Bitcoin is not new-user friendly! Basically, if you missed this boat, the only other one to hop on are the life rafts (exchanges) circling around the boat like a body of sharks. Anyone that the blockchain download after 2016 I bet put in a lot of wasted time and resources trying.

I started up a new node recently.  It took less than 5 days for me to get completely caught up.

5) Windows automatic updates really, really screw things up. With Bitcoin Core download, there is no time that your computer is idle for the updates, yet Windows thinks it is. So it kills the process with the reboot.

Windows is crap.  That's not Bitcoin's fault.  Windows will screw up a lot of things.  Sounds like a good reason not to use Windows.  However, if you feel you need to use Windows, then I recommend frequent backups so you can recover from any mess it makes.

6) Actually, any sort of blip whatsoever in your Bitcoin Core client or storage device causes havoc. Its not like a few years ago, where you're looking at 20 gigs. We're dealing with over 100 gigs now.

As I said, modern computer with good internet, or lots of patience and good backups, or choose something other than Core.

What happens when this reaches a terabyte? Not too far off, watch.

At a MAXIMUM possible growth of 4 megabytes per block, a terabyte is nearly 5 years away.  At a TYPICAL growth of 2.5 megabytes per block, a terabyte is more than 7.5 years away.

So, although it is enticing to attempt having the entire blockchain for failsafe transactions, it simply isn't feasible anymore.

You don't need to store the entire blockchain. You can run Core in pruning mode and your transactions will be just as secure.

What happens now?

People that are able to run Core will.  Those that are not able to, won't.

Well, one could say Bitcoin should just adapt permament light wallets ala Electrum using multiple redundent cloud-based storage for the blockchain.

Electrum is a very good option.

I've mentioned other options earlier in this post.

Question is, who pays for it?

That depends on the software you use.

Easy. Have silicon-based chip storage, with a percentage coming off transaction fees for the maintenance. Except then you have individual governance, which does not work with the Bitcoin system.

Bitcoin is working just fine the way it is. I'm not even sure what you're trying to suggest here.

[atliens99]

Just sue light wallets instead of downloading the entire bitcoin blockchain from scratch.

[emuLOAD]

Basically, what I am saying (though DH said it far more eloquently) is that there really isn't a need for "full nodes" that are having trouble setting themselves up. There are plenty of full replicas of the blockchain, and frankly there is no real reson for an end user to keep one more. Keep in mind it's only even slightly useful if you're Always on, 24/7, with good reachability. and even if you are, your addition to the network isn't essential.

This isn't to discourage people from running full nodes. I personally run two, 24/7, but you have to learn to deal with the software and its management. It will take you anywhere from 3 days to a week to fully sync. If bootstrapping doesn't make the process any quicker then you're not bandwidth bound but CPU bound.

[v1.0]

Welcome.

Thankyou, and for breaking it apart so nicely. You gave a lot to consider, some great info there. Going to dig in to it.

[v1.0]

If bootstrapping doesn't make the process any quicker then you're not bandwidth bound but CPU bound.

I never considered this- good to know. My connection speed certainly wouldn't be a factor @ 25/10 megabits..

[v1.0]

** Some awesome responses on here by experienced coiners, I want to address some of them, particularly by DannyHamilton **

** This is all from my point of view and experience to date, which is <2 months **

1) I keep seeing "linear growth" mentioned. This is incorrect IMO. The data is growing exponentially, you can see this by checking the blockchain growth charts.

2) Lightweight wallets are fine and dandy yes, but they require 100% dependency on an outside source :shudder:

3) Modern computer w/ RAM, SSD, some good cores etc---> I really appreciate this info, I had no idea the Bitcoin Core client was so dependant on this with the block processing, I had assumed the software itself was limited in how fast it accomplished this.. Silly oversight on my part, software is ALWAYS dependent on the quality of the hardware it resides in. It's like asking "why can't my Balckberry Playbook run Autocad?"

4) The download is still impossible. Because of the exponential growth, it gets slower and slower as it inches closer to the last year and a half, to the point of ridiculousness.

5) Pray, do tell, why the average person doesn't need to store the blockchain? Again, imagine if this type of thinking was the modus operative of the Bitcoin Core initiative.

6) I learned something new---> blocks "filling up". So, please correct me if I'm wrong. The size of the blockchain is the sum of two parts: the mined blocks & the ledger

7) Pruning mode--> neat! But you still need the entire blockchain first before entering this mode.

Backups are very important. If you aren't backing up your data, don't cry when its lost. But I'm not going to back up something I don't even have yet. Unless, I suppose, you do a backup of the partial data.

9) If its processing slower now than my old drive, than yes I suppose the specs are worse...I didn't realize how important this was, and should have wend SSD.

10) Silicone-chip based storage w/ fee percentages, never mind. A discussion for a different time. Typical newbie blurt.

[codewench]

6) I learned something new---> blocks "filling up". So, please correct me if I'm wrong. The size of the blockchain is the sum of two parts: the mined blocks & the ledger

The blocks are the ledger! The pre-Segwit 1MB block limit meant that the ledger can only grow by 1MB every ~10 minutes. Now with Segwit, the ledger can theoretically grow 4MB in ~10 minutes, though the practical rate is about 2MB/~10min and the current actual rate is only slightly over 1MB/~10 min.

[DannyHamilton]

** Some awesome responses on here by experienced coiners, I want to address some of them, particularly by DannyHamilton **

** This is all from my point of view and experience to date, which is <2 months **

1) I keep seeing "linear growth" mentioned. This is incorrect IMO. The data is growing exponentially, you can see this by checking the blockchain growth charts.

Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth.

2) Lightweight wallets are fine and dandy yes, but they require 100% dependency on an outside source :shudder:

They are not the best choice for everyone, but if you can't handle synchronizing and maintaining the blockchain they are a workable option.  They don't require an outside source at all for securing your bitcoins, nor do they require an outside source for sending/spending bitcoins.  They ONLY require an outside source for determining if (and how many) bitcoins you've received.  That risk can be significantly reduced by using multiple methods to check.

4) The download is still impossible. Because of the exponential growth, it gets slower and slower as it inches closer to the last year and a half, to the point of ridiculousness.

There was a lot of growth in block sizes for the first 6 or 7 years of Bitcoin's existence, because there was not enough transaction volume to fill the blocks.  For about 2 years now the blocks have been full at 1 megabyte per block.  So, you'll see it slow down until it is processing the transactions from the year 2016.  Then the speed will level off for the remaining 2+ years of data. The introduction of SegWit 6 months ago has allowed for a bit more growth in block size.  The maximum was bumped up to 4 megabytes, but the average size isn't likely to grow to be much more than 2.6 megabytes.


It is only ridiculous if you are using insufficient hardware.  As I said, I recently started up a new node. It took less than 5 days for me to get completely caught up.

5) Pray, do tell, why the average person doesn't need to store the blockchain?

The blockchain is only needed during initial synchronization.  After that, storing it is only useful for assisting others in starting up a new node.  

If you WANT to help others start up new nodes, that's great.  But, if you are unable to, that's okay. There are many of us that can and will.

Again, imagine if this type of thinking was the modus operative of the Bitcoin Core initiative.

Bitcoin Core implemented pruning specifically because they were aware that it isn't necessary for everybody to store a complete copy of the blockchain forever.

Additionally, Satoshi himself said:

It's not the download so much as verifying all the signatures in all the blocks as it downloads that takes a long time.

- snip -

Simplified Payment Verification is for lightweight client-only users who only do transactions and don't generate and don't participate in the node network.  They wouldn't need to download blocks, just the hash chain, which is currently about 2MB and very quick to verify (less than a second to verify the whole chain).  If the network becomes very large, like over 100,000 nodes, this is what we'll use to allow common users to do transactions without being full blown nodes.  At that stage, most users should start running client-only software and only the specialist server farms keep running full network nodes, kind of like how the usenet network has consolidated.
- snip -

AND

The design outlines a lightweight client that does not need the full block chain.  In the design PDF it's called Simplified Payment Verification.  The lightweight client can send and receive transactions, it just can't generate blocks.  It does not need to trust a node to verify payments, it can still verify them itself.

- snip -

I anticipate there will never be more than 100K nodes, probably less.  It will reach an equilibrium where it's not worth it for more nodes to join in.  The rest will be lightweight clients, which could be millions.

At equilibrium size, many nodes will be server farms with one or two network nodes that feed the rest of the farm over a LAN.

AND

The current system where every user is a network node is not the intended configuration for large scale.  That would be like every Usenet user runs their own NNTP server.  The design supports letting users just be users.  The more burden it is to run a node, the fewer nodes there will be.  Those few nodes will be big server farms.  The rest will be client nodes that only do transactions and don't generate.
- snip -

6) I learned something new---> blocks "filling up". So, please correct me if I'm wrong. The size of the blockchain is the sum of two parts: the mined blocks & the ledger

Incorrect.

The ledger IS the mined blocks.  The mined blocks ARE the ledger.  The blockchain is a chain of blocks of transactions which have a valid proof-of-work.

7) Pruning mode--> neat! But you still need the entire blockchain first before entering this mode.

No.  You can set pruning mode before you start synchronization.  It will prune old blocks as it downloads the newer blocks.  You still need to process the entire blockchain, but you don't need to store it all. That way you don't need as big of a hard drive, and it is easier to stop your node and create a backup once a day during synchronization.  Then, if your node crashes in a disasterous way (due to Windoze?) you can pick up from where you left off less than a day ago.

Backups are very important. If you aren't backing up your data, don't cry when its lost. But I'm not going to back up something I don't even have yet. Unless, I suppose, you do a backup of the partial data.

Correct.  Backup what you have so far. That way, if you have a disaster, you don't have to start all over from the beginning.

9) If its processing slower now than my old drive, than yes I suppose the specs are worse...I didn't realize how important this was, and should have wend SSD.

Yes.  The synchronization process accesses the storage a LOT.  A faster drive (or better yet SSD) makes a HUGE difference.

[v1.0]

Wow- really appreciate all the detailed clarifications plus responses! Going to grind through them some more on the weekend.

At about 80.5% with the blocks..so is it impossible? We shall see. I want to back it up before 90%.

Currently kept on seperate drive than my Windows OS. So just a matter of copying entire Bitcoin folder (subdirectories+files) to my backup drive, correct? Exit the client first..

[alyssa85]

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

[v1.0]

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

Ok- but I won't hold my breathe 

80% to 84% in 10 hours boy o boy. I turned it off and unplugged the drive to give it rest, maybe getting too worked. I should have known better to get an SSD as poster DannyHamilton said. I honestly didn't think it would matter. Oh well, lesson learned.

Good to know about the 10 minute update/week thats not bad at all.

Thanks for response!

[alyssa85]

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

Ok- but I won't hold my breathe 

80% to 84% in 10 hours boy o boy. I turned it off and unplugged the drive to give it rest, maybe getting too worked.

Don't unplug! Keep going. Once you are up to date it really is easy and smooth. You won't remember the hassle of getting up to date!

[v1.0]

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

Ok- but I won't hold my breathe 

80% to 84% in 10 hours boy o boy. I turned it off and unplugged the drive to give it rest, maybe getting too worked.

Don't unplug! Keep going. Once you are up to date it really is easy and smooth. You won't remember the hassle of getting up to date!

Thanks for the encouragement! Gave it a couple hours of rest, and went back to it..there is hope its at 96.99% now.

Going to grind through everyones responses now, fell behind on the weekend.

So, in terms of linear vs exponential growth- I apologize in advance if I keep rehashing this- and someone responded with a graph. So yes looks like linear to me, and it appears my definitions of linear and exponential are somewhat skewed. Anyone wish to elaborate in detail about how Bitcoin growth in terms of the blockchain is not exponential? Would linear growth not indicate that there are never massice fluctuations in increase?

Ok, so if the blocks ARE the ledger, what diffirentiates mined blocks vs a day when suddenly there's 500,000 transactions vs a day of only 20,000? Does this not contribute toward determining a linear vs exponential growth of the blockchain?

@ DannyHamilton said:

"Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth."

Ok, so you're saying there is also a transaction limit per block?

Hmm, maybe you already answered my question, with your response of the Segwit introduction. Great, now I'm even more confused. Is Segwit implementation backwards-compatible with transactions pre-Segwit? Is there anything I have to do with the Bitcoin core client or will everything just run? I vaguely remember when I set up Electrum it asking me something about if I wanted a classic wallet or Segwit wallet, no idea what it was talking about so I went classic.

Thnx @ CodeWrench, atliens99 and emuLOAD too!

DannyHamilton, your Satoshi references are awesome! Thanks so much. I can't believe he had foresight to know that the average user would actually NOT need the entire blockchain! Especially the quote "The more burden it is to run a node, the fewer nodes there will be." which rings true. People would just get discouraged, and interest would wane off. Also, I appreciate your further detail about pruning without having to download the entire blockchain first- thanks.

Dammit, my response is all over the place and completely disorganized, ugh! Apologies. Usually not my MO.

[Aura]

Harddrives are not that expensive anymore, considering that a 1 Tb HDD costs less than $40. I don't agree that an average person can't afford 0.15*40=$6 to store the full blockchain. Also downloading the full blockchain is not impossible either, you just need to have some patience. It took my less than 2 days to download the full blockchain, while I don't have the best hardware. Besides that, you don't really need to have the full blockchain on your computer to use the bitcoin ecosystem. It's your own choice whether you contribute to decentralization by having a full node or use Bitcoin by trusting other's blockchain. Last thing, deframenting your HDD will speed up the syncing process, although Windows does this automatically each week.

[DannyHamilton]

- snip -
Ok, so if the blocks ARE the ledger, what diffirentiates mined blocks vs a day when suddenly there's 500,000 transactions vs a day of only 20,000? Does this not contribute toward determining a linear vs exponential growth of the blockchain?
- snip -
Ok, so you're saying there is also a transaction limit per block?
- snip -
Great, now I'm even more confused.
- snip -

It seems you are lacking some understanding of what bitcoin is and how it works.  As such, you're unable to put your frustrations with the time to synchronize a new wallet into a mental framework that allows you to understand why it takes as long as it does, and what the future is likely to look like.

To get a grasp on the general concepts, I suggest pretending that you don't know anything about bitcoin (or at least assuming that everything you know is wrong) and then watching this video as if it is about something completely different than your current understandings:

https://www.youtube.com/watch?v=bBC-nXj3Ng4

Some of the specific details are generalized or simplified in that video, so it won't give you an exact understanding of every technical detail, but it is a good introduction to the more basic concepts.

Then once you've watched it, let me know if you still have some questions.

Note: That video was created before SegWit was implemented, so it won't cover anything about SegWit at all.

what differentiates mined blocks vs a day when suddenly there's 500,000 transactions vs a day of only 20,000? Does this not contribute toward determining a linear vs exponential growth of the blockchain?

Transactions are that are not yet in a block are not a reliable indication of transfer of value. A transaction without "confirmations" simply means that the transaction is not yet in the blockchain. Those transactions may make it into a block eventually, but they also may be replaced with some other transaction.  This is why it is important to wait for "confirmations" when receiving a transaction from someone that you don't already have a trust relationship with.

So, when there is a HUGE volume of transactions (transactions are being created at a faster rate than they fit into a block) then the miners/pools choose to include the transactions with the highest fees in their blocks (since they get to keep the fees). Transactions with lower fees sit around waiting for later blocks and become part of a backlog of unconfirmed transactions.  If the rate of new high fee transactions eventually drops off, then the backlog of lower fee transactions fills in the block space that becomes available.

There has pretty much been a continuous backlog of transactions waiting for confirmation for a bit more than a year now. As older low fee transactions become confirmed, or abandoned, newer low fee transactions get broadcast and take their place waiting for confirmation. Prior to 2016 blocks were nearly never full.  Bitcoin was brand new and there just wasn't enough use to result in full blocks yet.  As such, the growth of the blockchain from 2009 through 2016 was exponential.  Then starting sometime in 2016, most blocks were nearly full and transaction fees started to increase.  As such, growth of the blockchain throughout 2016 and 2017 was linear.  Then in late 2017 SegWit was introduced.  With that change, the blocks are no longer limited by "size", but instead by "weight" where the "weight" of the SegWit portion of a transaction counts for less than the "weight" of the non-SegWit portion.  As such, the effective size of a block will depend on how much of it is SegWit transactions, however even if it were 100% SegWit, the block size would not exceed 4 megabytes, with the typical size closer to 2.6 megabytes.

SegWit transactions ARE backward compatible, however you won't be able to take advantage of the features of SegWit (such as increased probability of confirmation in any given block for the same transaction & fees) unless you are using SegWit receiving addresses in a SegWit enabled wallet.  Furthermore, a pre-SegWit full node (such as an old version of Bitcoin Core) will not fully validate transactions that spend bitcoins that were previously received at a SegWit address.  Instead it will rely on the hope that someone else on the network is running a SegWit enabled full node which will filter out invalid transactions before they are relayed.