Bitcoin Forum

As a newcomer to Bitcoin Core. I find the Bitcoin Core blockchain impossible to download now, it is growing exponentially and is just too slow and too big for the average person. What's the point? I wanted to have the entire blockchain on my computer, but it ain't happening. Here's why:

1) Bitcoin has been in existance for almost 10 years. But 50% of the entire blockchain is just from the last year or so! I mean, I'm at 30% downloaded right now, and that's right up until November 2015!

2) If your hard drive screws up, you're screwed! I was three weeks into the Bitcoin Core blockchain download, made it to 90%. Then, BAM! My hard drive died. Got a new drive, decided to try again. Seems slower than ever, my computer runs to a crawl at times.

3) Even if someday I do manage to catch up to the most current block/day, I believe soon if not yet already there, we will never be able to get past a certain percentage, say 98%, due to the ever-growing nature of the blockchain. It will get to the point where it is growing faster than than possible to download.

4) Bitcoin is not new-user friendly! Basically, if you missed this boat, the only other one to hop on are the life rafts (exchanges) circling around the boat like a body of sharks. Anyone that the blockchain download after 2016 I bet put in a lot of wasted time and resources trying.

5) Windows automatic updates really, really screw things up. With Bitcoin Core download, there is no time that your computer is idle for the updates, yet Windows thinks it is. So it kills the process with the reboot.

6) Actually, any sort of blip whatsoever in your Bitcoin Core client or storage device causes havoc. Its not like a few years ago, where you're looking at 20 gigs. We're dealing with over 100 gigs now. What happens when this reaches a terabyte? Not too far off, watch.

So, although it is enticing to attempt having the entire blockchain for failsafe transactions, it simply isn't feasible anymore. What happens now? Well, one could say Bitcoin should just adapt permament light wallets ala Electrum using multiple redundent cloud-based storage for the blockchain. Question is, who pays for it? Easy. Have silicon-based chip storage, with a percentage coming off transaction fees for the maintenance. Except then you have individual governance, which does not work with the Bitcoin system.

Thoughts?

How does the blockchain grow exponentially with one 1 MB block added, on average, every ten minutes? Isn’t that linear growth? Never mind. You can speed up synchronisation by putting the data directory, or at least the chainstate, on an SSD. Also, connection speeds, storage capacity and other such metrics keep improving as hardware evolves. And you can put Windows Update on manual, or ditch Windows altogether.

Thanks for reply.

The 1MB block limit is linear, sure, but thats related to mining. The transactions are what make it exponential. No limit on ledger size. How else to explain 3 times growth in just last 2 years with a 9 year old platform?

The hardware must evolve to keep up. Thanks for hardware tips- and I agree Windows is terrible for bitcoining. Also running a Linux server on SSD, but using that one for BiblePay :)

--very new to this mind you. Just so discouraged at how difficult its proving to get into Bitcoin Core. I didn't want to just be a holder of the coins- anyone can do that. I much prefer to have the source and contribute to the community.

Unless you're on dialup, you'll never have problems staying up to date. Keep in mind the first few years we were nowehere near filling blocks, which means the chain grew less than it could have. nowadays we are closer to saturation, so growth is a lot more linear.

As far as bootstrapping goes, it's insane to download the Whole thing from scratch through the core client. Look for a bootstrap torrent, and sync from there. yes, first start will be slow, but it won't take nearly as long.

All this said, nobody's ever said you need to run a full node. Wallets like Electrum, especially if combined with Trezor or other hardware wallets, are safe enough--indeed more in some cases--than running a full node.
there is no need to run a full node, and not doing so won't make you anybody's slave.

If you want core and contribute, you have more studying to do, because you simply havn't learnt enough yet to be useful.

Thanks for the reply.

Oh yes, lots of studying to be had, and learning on this constantly, so thanks!

So, regarding bootstrapping, actually it's hardly any faster than downloading through the client- this is through my personal experience. Which is really what annoyed me about it all. My Core client has been "processing blocks" for over two days now via bootstrap, it hit 30% and basically stuck at the same speed, even worse, than previously on my old drive through traditional method.

Of course I don't have to run a full node. But imagine if everyone thought this way- "why bother running a full node, I can just grab an Electrum wallet and feed through that." Well, I bet most people do go this route, but as stated in my original post, its better and safer to have the blockchain locally, given the option.

As far as "learning enough to be useful", not sure what you mean by this statement. Even newcomers can be usefull by providing nodes to the community via Core to assist with transaction verifications, even though we can't mine anymore.

I agree there is no need to run a full node, which is really the crux of this thread. Its unfortunate and discouraging, really, that those of us who do are pretty much SOL now. We should be encouraging nodes, but the current state has made it an impossibility to start from scratch, bootstrap or otherwise.

Welcome.


You may want to consider a lightweight wallet if your computer specs are not modern enough to handle it.


It is not growing exponentially. It is growing linearly.


You'll find that a modern computer with enough RAM and an SSD will help a lot.


It is currently less than 157 gigabytes.

An "average person" can purchase a 256 gigabyte SD Card for less than $25 at Amazon.com:
http://a.co/7Wu4ib4

An "average person" can purchase a 1 terabyte external SSD drive  for less than $350 at Amazon.com:
http://a.co/6TtcGMx

An average person can also run Bitcoin Core in "pruning mode", since the "average person" doesn't have any personal need to store the blockchain.


The point to running Bitcoin Core?  There may not be one.  That's for each user to decide for themselves.  You could establish an account with a service such as Coinbase, you could run a lightweight wallet such as Electrum, you could run Core with pruning.  Each user can decide for themselves what best fits their needs.


Why?


That's ok.  If you can't store the blockchain, then others will.


You may want to check your math on that.

Blockchain size a year ago was just a bit more than 102 gigabytes.

(157 - 102) / 157 = 35.03%

That's a LOT less than 50%


Correct.  The blocks were not full the first 6 or 7 years of Bitcoin's existence.  There just weren't many people using it yet.


I create regular backups of data that is important to me. That way I don't lose it when my hard drive "screws up".  You may want to try doing that.  You can recover a backup of your hard drive, and then just pick up where you left off.


If it is slower now than before purchasing the new drive, then I'm guessing you purchased a drive with lesser specifications.  Perhaps purchase storage with the same OR BETTER specs next time?


The block chain grows at a maximum average rate of 4 megabytes every 10 minutes.  If your internet is incapable up supplying 4 megabytes every 10 minutes (that's less than a 56k dial-up modem) or your computer is incapable of processing 4 megabytes every 10 minutes, then you should probably not be trying to run Bitcoin Core.


I started up a new node recently.  It took less than 5 days for me to get completely caught up.


Windows is crap.  That's not Bitcoin's fault.  Windows will screw up a lot of things.  Sounds like a good reason not to use Windows.  However, if you feel you need to use Windows, then I recommend frequent backups so you can recover from any mess it makes.


As I said, modern computer with good internet, or lots of patience and good backups, or choose something other than Core.


At a MAXIMUM possible growth of 4 megabytes per block, a terabyte is nearly 5 years away.  At a TYPICAL growth of 2.5 megabytes per block, a terabyte is more than 7.5 years away.


You don't need to store the entire blockchain. You can run Core in pruning mode and your transactions will be just as secure.


People that are able to run Core will.  Those that are not able to, won't.


Electrum is a very good option.

I've mentioned other options earlier in this post.


That depends on the software you use.


Bitcoin is working just fine the way it is. I'm not even sure what you're trying to suggest here.

Just sue light wallets instead of downloading the entire bitcoin blockchain from scratch.

Basically, what I am saying (though DH said it far more eloquently) is that there really isn't a need for "full nodes" that are having trouble setting themselves up. There are plenty of full replicas of the blockchain, and frankly there is no real reson for an end user to keep one more. Keep in mind it's only even slightly useful if you're Always on, 24/7, with good reachability. and even if you are, your addition to the network isn't essential.

This isn't to discourage people from running full nodes. I personally run two, 24/7, but you have to learn to deal with the software and its management. It will take you anywhere from 3 days to a week to fully sync. If bootstrapping doesn't make the process any quicker then you're not bandwidth bound but CPU bound.

Thankyou, and for breaking it apart so nicely. You gave a lot to consider, some great info there. Going to dig in to it.

I never considered this- good to know. My connection speed certainly wouldn't be a factor @ 25/10 megabits..

** Some awesome responses on here by experienced coiners, I want to address some of them, particularly by DannyHamilton **

** This is all from my point of view and experience to date, which is <2 months **

1) I keep seeing "linear growth" mentioned. This is incorrect IMO. The data is growing exponentially, you can see this by checking the blockchain growth charts.

2) Lightweight wallets are fine and dandy yes, but they require 100% dependency on an outside source :shudder:

3) Modern computer w/ RAM, SSD, some good cores etc---> I really appreciate this info, I had no idea the Bitcoin Core client was so dependant on this with the block processing, I had assumed the software itself was limited in how fast it accomplished this.. Silly oversight on my part, software is ALWAYS dependent on the quality of the hardware it resides in. It's like asking "why can't my Balckberry Playbook run Autocad?"

4) The download is still impossible. Because of the exponential growth, it gets slower and slower as it inches closer to the last year and a half, to the point of ridiculousness.

5) Pray, do tell, why the average person doesn't need to store the blockchain? Again, imagine if this type of thinking was the modus operative of the Bitcoin Core initiative.

6) I learned something new---> blocks "filling up". So, please correct me if I'm wrong. The size of the blockchain is the sum of two parts: the mined blocks & the ledger

7) Pruning mode--> neat! But you still need the entire blockchain first before entering this mode.

8) Backups are very important. If you aren't backing up your data, don't cry when its lost. But I'm not going to back up something I don't even have yet. Unless, I suppose, you do a backup of the partial data.

9) If its processing slower now than my old drive, than yes I suppose the specs are worse...I didn't realize how important this was, and should have wend SSD.

10) Silicone-chip based storage w/ fee percentages, never mind. A discussion for a different time. Typical newbie blurt.

The blocks are the ledger! The pre-Segwit 1MB block limit meant that the ledger can only grow by 1MB every ~10 minutes. Now with Segwit, the ledger can theoretically grow 4MB in ~10 minutes, though the practical rate is about 2MB/~10min and the current actual rate is only slightly over 1MB/~10 min.

https://i.imgur.com/SxEhQek.png

Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth.


They are not the best choice for everyone, but if you can't handle synchronizing and maintaining the blockchain they are a workable option.  They don't require an outside source at all for securing your bitcoins, nor do they require an outside source for sending/spending bitcoins.  They ONLY require an outside source for determining if (and how many) bitcoins you've received.  That risk can be significantly reduced by using multiple methods to check.


There was a lot of growth in block sizes for the first 6 or 7 years of Bitcoin's existence, because there was not enough transaction volume to fill the blocks.  For about 2 years now the blocks have been full at 1 megabyte per block.  So, you'll see it slow down until it is processing the transactions from the year 2016.  Then the speed will level off for the remaining 2+ years of data. The introduction of SegWit 6 months ago has allowed for a bit more growth in block size.  The maximum was bumped up to 4 megabytes, but the average size isn't likely to grow to be much more than 2.6 megabytes.


It is only ridiculous if you are using insufficient hardware.  As I said, I recently started up a new node. It took less than 5 days for me to get completely caught up.


The blockchain is only needed during initial synchronization.  After that, storing it is only useful for assisting others in starting up a new node.  

If you WANT to help others start up new nodes, that's great.  But, if you are unable to, that's okay. There are many of us that can and will.


Bitcoin Core implemented pruning specifically because they were aware that it isn't necessary for everybody to store a complete copy of the blockchain forever.

Additionally, Satoshi himself said:


AND


AND



Incorrect.

The ledger IS the mined blocks.  The mined blocks ARE the ledger.  The blockchain is a chain of blocks of transactions which have a valid proof-of-work.


No.  You can set pruning mode before you start synchronization.  It will prune old blocks as it downloads the newer blocks.  You still need to process the entire blockchain, but you don't need to store it all. That way you don't need as big of a hard drive, and it is easier to stop your node and create a backup once a day during synchronization.  Then, if your node crashes in a disasterous way (due to Windoze?) you can pick up from where you left off less than a day ago.


Correct.  Backup what you have so far. That way, if you have a disaster, you don't have to start all over from the beginning.


Yes.  The synchronization process accesses the storage a LOT.  A faster drive (or better yet SSD) makes a HUGE difference.

Wow- really appreciate all the detailed clarifications plus responses! Going to grind through them some more on the weekend.

At about 80.5% with the blocks..so is it impossible? We shall see. I want to back it up before 90%.

Currently kept on seperate drive than my Windows OS. So just a matter of copying entire Bitcoin folder (subdirectories+files) to my backup drive, correct? Exit the client first..

Be patient - if you are new, it will take at least a week to download the entire blockchain from scratch.

BUT - once downloaded, you just need to update your wallet for ten minutes a week to keep updated and it should function as smoothly as butter.

Ok- but I won't hold my breathe  :)

80% to 84% in 10 hours boy o boy. I turned it off and unplugged the drive to give it rest, maybe getting too worked. I should have known better to get an SSD as poster DannyHamilton said. I honestly didn't think it would matter. Oh well, lesson learned.

Good to know about the 10 minute update/week thats not bad at all.

Thanks for response!

Don't unplug! Keep going. Once you are up to date it really is easy and smooth. You won't remember the hassle of getting up to date!

Thanks for the encouragement! Gave it a couple hours of rest, and went back to it..there is hope its at 96.99% now.

Going to grind through everyones responses now, fell behind on the weekend.

So, in terms of linear vs exponential growth- I apologize in advance if I keep rehashing this- and someone responded with a graph. So yes looks like linear to me, and it appears my definitions of linear and exponential are somewhat skewed. Anyone wish to elaborate in detail about how Bitcoin growth in terms of the blockchain is not exponential? Would linear growth not indicate that there are never massice fluctuations in increase?

Ok, so if the blocks ARE the ledger, what diffirentiates mined blocks vs a day when suddenly there's 500,000 transactions vs a day of only 20,000? Does this not contribute toward determining a linear vs exponential growth of the blockchain?

@ DannyHamilton said:

"Looks pretty linear to me.

Regardless, the math for the future is pretty simple.  Maximum 4 MB per block.  Average 1 block per 10 minutes.  That's linear growth."

Ok, so you're saying there is also a transaction limit per block?

Hmm, maybe you already answered my question, with your response of the Segwit introduction. Great, now I'm even more confused. Is Segwit implementation backwards-compatible with transactions pre-Segwit? Is there anything I have to do with the Bitcoin core client or will everything just run? I vaguely remember when I set up Electrum it asking me something about if I wanted a classic wallet or Segwit wallet, no idea what it was talking about so I went classic.

Thnx @ CodeWrench, atliens99 and emuLOAD too!

DannyHamilton, your Satoshi references are awesome! Thanks so much. I can't believe he had foresight to know that the average user would actually NOT need the entire blockchain! Especially the quote "The more burden it is to run a node, the fewer nodes there will be." which rings true. People would just get discouraged, and interest would wane off. Also, I appreciate your further detail about pruning without having to download the entire blockchain first- thanks.

Dammit, my response is all over the place and completely disorganized, ugh! Apologies. Usually not my MO.

Harddrives are not that expensive anymore, considering that a 1 Tb HDD costs less than $40. I don't agree that an average person can't afford 0.15*40=$6 to store the full blockchain. Also downloading the full blockchain is not impossible either, you just need to have some patience. It took my less than 2 days to download the full blockchain, while I don't have the best hardware. Besides that, you don't really need to have the full blockchain on your computer to use the bitcoin ecosystem. It's your own choice whether you contribute to decentralization by having a full node or use Bitcoin by trusting other's blockchain. Last thing, deframenting your HDD will speed up the syncing process, although Windows does this automatically each week.

It seems you are lacking some understanding of what bitcoin is and how it works.  As such, you're unable to put your frustrations with the time to synchronize a new wallet into a mental framework that allows you to understand why it takes as long as it does, and what the future is likely to look like.

To get a grasp on the general concepts, I suggest pretending that you don't know anything about bitcoin (or at least assuming that everything you know is wrong) and then watching this video as if it is about something completely different than your current understandings:

https://www.youtube.com/watch?v=bBC-nXj3Ng4

Some of the specific details are generalized or simplified in that video, so it won't give you an exact understanding of every technical detail, but it is a good introduction to the more basic concepts.

Then once you've watched it, let me know if you still have some questions.

Note: That video was created before SegWit was implemented, so it won't cover anything about SegWit at all.


Transactions are that are not yet in a block are not a reliable indication of transfer of value. A transaction without "confirmations" simply means that the transaction is not yet in the blockchain. Those transactions may make it into a block eventually, but they also may be replaced with some other transaction.  This is why it is important to wait for "confirmations" when receiving a transaction from someone that you don't already have a trust relationship with.

So, when there is a HUGE volume of transactions (transactions are being created at a faster rate than they fit into a block) then the miners/pools choose to include the transactions with the highest fees in their blocks (since they get to keep the fees). Transactions with lower fees sit around waiting for later blocks and become part of a backlog of unconfirmed transactions.  If the rate of new high fee transactions eventually drops off, then the backlog of lower fee transactions fills in the block space that becomes available.

There has pretty much been a continuous backlog of transactions waiting for confirmation for a bit more than a year now. As older low fee transactions become confirmed, or abandoned, newer low fee transactions get broadcast and take their place waiting for confirmation. Prior to 2016 blocks were nearly never full.  Bitcoin was brand new and there just wasn't enough use to result in full blocks yet.  As such, the growth of the blockchain from 2009 through 2016 was exponential.  Then starting sometime in 2016, most blocks were nearly full and transaction fees started to increase.  As such, growth of the blockchain throughout 2016 and 2017 was linear.  Then in late 2017 SegWit was introduced.  With that change, the blocks are no longer limited by "size", but instead by "weight" where the "weight" of the SegWit portion of a transaction counts for less than the "weight" of the non-SegWit portion.  As such, the effective size of a block will depend on how much of it is SegWit transactions, however even if it were 100% SegWit, the block size would not exceed 4 megabytes, with the typical size closer to 2.6 megabytes.

SegWit transactions ARE backward compatible, however you won't be able to take advantage of the features of SegWit (such as increased probability of confirmation in any given block for the same transaction & fees) unless you are using SegWit receiving addresses in a SegWit enabled wallet.  Furthermore, a pre-SegWit full node (such as an old version of Bitcoin Core) will not fully validate transactions that spend bitcoins that were previously received at a SegWit address.  Instead it will rely on the hope that someone else on the network is running a SegWit enabled full node which will filter out invalid transactions before they are relayed.

the main gripe is that people cant really spend their funds until the chain is uptodate. which can be days-weeks to set it up

if only there was a smart dev that could think of some simple solution..
like

an uptodate UTXO set of (lets say current blockheight minus 1) gets its own hash. and that hash is set in the current block data.
making the UTXO set of 1confirms+ verifiable as part of block checks

thus people can then have their wallet grab a very recent UTXO set FIRST. so they can actually go ahead and spend their funds, the same day.
after all sending out a tx using the UTXO set wont hurt the network. if it contains a input that is actually spend or non existent it wont get relayed/accepted into a new block anyway..

...
this way it is also good for light wallet users that just want the UTXO set have some faith that the UTXO set has some checks and validations by the network.

and then it makes downloading the blockchain if they want to part of the networks full validation/data archiving, more of a background feature..
thus then alleviates the requirement for some people to not have to wait 2 weeks or so to download the blockchain before they are upto date enough just to spend funds.

It's done, it is not impossible. You all were right. And I've been won over. :) So now I shall view the title to the thread as merely now an exasperation of sorts, more of "because it took so long, it seemed impossible". Man, does that feel great though, after such a long time blocks are up to date and finally syncs.

So now what. Well, I exited it the program and disconnected the drive. Replugged, started up the core again, and it was up and synced fairly quick. So, that was about a week ago. I've since unplugged the drive again and its been sitting cold for a week. I want to see how long it takes to resync after a couple weeks. I think its going to annoy me enough to the point of upgrading some hardware..it always spirals into ridiculousness though. For example, if I want more ram, I'll have to upgrade my MB to support it. MY new MB might not be compatible with my current cpu. So then I'll have to change that. etc But what I'm really looking forward to is putting Core on an SSD as many on here have suggested.

DannyHamilton thank you for the video suggestion, I am going to look at it. This is going to be a slow and steady race for me, but eventually I hope to at least grasp enough of the basics to have a proper dialogue about it. The rest of your response I'm going to have to sort through at least 5 or 6 times, there is such great detail in there.

So a newer Bitcoin Core version can process SegWit transactions? I will have to check my version. That is interesting to know about how an older version handles (or doesn't, to be correct) a transaction from a SegWit wallet.

Also, invalid transactions seem important to understand. I want to know what kind of reprocussions it causes in the ecosystem, if any, or do they just eventually filter into non-existance?

Thanks again for all the responses/help.

They almost immediately filter into non-existence.

Every node on the network verifies every transaction it receives before it adds the transaction to it's own mempool, or relays the transaction to any other connected peers.

Therefore, if anyone tries to connect to any nodes and broadcast an invalid transaction, every one of those connected peers will reject the invalid transaction.  It won't be accepted into their own mempools, AND it won't be relayed to ANY other nodes.

Furthermore, if a node is found to be repeatedly sending invalid data, then the nodes that it is connected to will close that connection so that they don't need to repeatedly check invalid data.

What is holding back implementing this kind of idea? There must be trade-offs of some sort, otherwise it would have been done by now?

Excellent- thanks!

Taking this a bit further to rejected transactions...on another altcoin I was mining (not bitcoin, as individual users no longer can mine), apparently I mined my first block ever ( super excited) and got paid, it showed the number of coins I had in my wallet. But then apparently the crypto "forked". I had to upgrade my software to continue mining. However, after the upgrade, my payment for my newly mined block disappeared! I inquired to the sysop about it, he responded that likely my new mined block was mined during a "fork", thereby rejected by all the nodes I was attempting to connect to.

If this is inappropriate to ask about in a bitcoin thread, by all means just tell me I wont bring it up again. But I'm thinking perhaps the same thing could have occurred in the past with Bitcoin itself during a fork? Mined blocks would not be valid?

***

So, after a couple weeks of testing, ie. having my wallet and the entire chain offline on a removable drive, than plugging in and starting the resync all over again, happy to report ( I know I know I'm preaching to the choir) that everything works perfectly! Both times Bitcoin Core has synced back up to the network and brought the chain up to date even after being offline for a week. So---I can happily say that: It is NOT impossible to download now, but still is definitely too slow and too big! :)

Why is this a problem? I must be missing something.

Hi, I'm tired of waiting for a long sync :( and stumbled upon the blocks with a torrent and there are still coins torrented
https://krypto-wallet.org/en/coin/download-blockchain-bitcoin-torrent-file.html

 "New info escalates importance: upgrading to 0.16.3 is REQUIRED"

Does the entire blockchain have to be downloaded again now??

Learning as I go folks! So no, as I found out, an upgrade does not require the blockchain to be downloaded again. Once downloaded and unpacked, the upgrade is really just the bitcoin-qt file plus a couple more files. Basically copied them into my bitcoin folder and viola! everything runs just fine.

Thanks,

v1.0

p.s. new core version released today:  v0.17.0