broflof (OP)
Newbie
Offline
Activity: 3
Merit: 4
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 24, 2024, 03:54:41 PM |
|
I'm syncing my Bitcoin wallet blockchain using latest Bitcoin Core on Linux.
It progresses very very slowly, right now 2 to 3 weeks in 12 hours.
CPU and net usage stay low most of the time, problem is not there (most likely). HDD usage is constant, I think that's probably the bottleneck. It's a 7200rpm standard desktop HDD.
10 years ago I could sync the blockchain on a slow 5400rpm laptop HDD without major issues.
Time to buy big SSD perhaps.
But the interesting question is, Does syncing get harder over time as the blockchain gets longer?
Or has something happened to the protocol so it would be harder now?
Or has the code quality of Bitcoin Core gone down?
The GUI also stalls badly during sync. I don't remember when this problem began but it did not always exist.
Syncing monero is also extremely slow and it has been so as long as I can remember. But that's another story.
|
|
|
|
promise444c5
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 392
Merit: 254
Keep Promises !
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 24, 2024, 04:01:56 PM Last edit: June 24, 2024, 04:44:11 PM by promise444c5 |
|
In the last 10 years there're fewer transactions in a block... now there's more. Use the current version check the latest news section of the forum for the torrent file,Then if you are using bitcoin-qt GUI check the dbcache increase it, it could make it sync process more faster(note the value is in MB ) this should be done carefully in accordance to your ram capacity . Does syncing get harder over time as the blockchain gets longer?
Not really but yes and might depends on the hardware , people can still get it done in just 2~4 days. The GUI also stalls badly during sync. I don't remember when this problem began but it did not always exist.
Sometimes when when block index is writing to disk you might experience that and you would think it's just stall
|
R |
▀▀▀▀▀▀▀██████▄▄ ████████████████ ▀▀▀▀█████▀▀▀█████ ████████▌███▐████ ▄▄▄▄█████▄▄▄█████ ████████████████ ▄▄▄▄▄▄▄██████▀▀ | LLBIT | │ | CRYPTO FUTURES | | | | | | | | | [ | 1,000x LEVERAGE | ] | [ | . COMPETITIVE FEES | ] | [ | INSTANT EXECUTION | ] | | | ██████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██████ | ████████████████████████████████████████████████████████ . TRADE NOW . ████████████████████████████████████████████████████████ | ██████ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██ ██████ |
|
|
|
un_rank
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 24, 2024, 04:16:55 PM |
|
Does syncing get harder over time as the blockchain gets longer?
Or has something happened to the protocol so it would be harder now?
It should, but the chain is not increasing as such a rate that will cause an significant lag in download time. Could be from the internet connection. Nothing happened on the protocol level to make it harder either. - Jay -
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17277
Thick-Skinned Gang Leader and Golden Feather 2021
|
CPU and net usage stay low most of the time, problem is not there (most likely). HDD usage is constant, I think that's probably the bottleneck. It's a 7200rpm standard desktop HDD. You have 2 bottlenecks: RAM, and HDD speed. Both are more or less interchangeable: with 32 GB RAM and 12000 MB dbcache, your sync will go just fine. With 8 GB RAM and an SSD, the SSD is still the bottleneck. I tested both of these examples myself. 10 years ago I could sync the blockchain on a slow 5400rpm laptop HDD without major issues. Ten years ago, the entire blockchain was only 20 GB. Does syncing get harder over time as the blockchain gets longer? Yes. The chainstate directory keeps growing, and requires more and more RAM (or a LOT of disk activity) to verify the download. Or has the code quality of Bitcoin Core gone down? Nope. Older versions were much slower to sync. Syncing monero is also extremely slow and it has been so as long as I can remember. But that's another story. Monero is even more demanding than Bitcoin Core, despite a much smaller blockchain. TL;DR: get an SSD. If you have a small SSD, put the chainstate directory on it. That will make a huge difference for Bitcoin Core. If you don't have an SSD yet, buy one. Get a decent one for best performance. It's going to largely improve your entire computing experience.
|
|
|
|
philipma1957
Legendary
Offline
Activity: 4228
Merit: 8392
'The right to privacy matters'
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 24, 2024, 07:06:05 PM |
|
a decent i5 gen cpu say i5 8500 or newer a 1 tb samsung ssd or bigger 16gb ram or more a decent internet connection say 100Mbps or more
if you have all of the above you will be under 1 day
obvious that
an i7 14700 a 2tb nvme2 quality ssd 64gb ram and a 1Gbps internet
will be faster.
In fact the second setup may be faster than the offered download speed.
I have used 200Mbps and 128gb ram 1tb nvme2 ssd threadripper cpu
My setup was able to do 190Mbps constantly which makes me suspect my internet connection was my bottle neck
I will soon get 1gb internet and try to see how fast I can do this.
|
|
|
|
ABCbits
Legendary
Offline
Activity: 2982
Merit: 7821
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 25, 2024, 08:45:15 AM Last edit: June 25, 2024, 09:03:54 AM by ABCbits Merited by LoyceV (6), pooya87 (4), Husna QA (2) |
|
Does syncing get harder over time as the blockchain gets longer?
Short answer, yes. Not only each block become bigger, total UTXO keep growing. Edit: I assume "gets longer" refers to newer blocks. Or has something happened to the protocol so it would be harder now?
AFAIK no. New stuff (such as SegWit) solve few problem including quadratic verification time. Or has the code quality of Bitcoin Core gone down?
No. Jameson Lopp did benchmark which prove the opposite on https://blog.lopp.net/bitcoin-core-performance-evolution/. The GUI also stalls badly during sync. I don't remember when this problem began but it did not always exist.
That's true, it's known issue for few years. See this issue, https://github.com/bitcoin-core/gui/issues/299.
|
|
|
|
broflof (OP)
Newbie
Offline
Activity: 3
Merit: 4
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 06, 2024, 03:36:37 PM |
|
Thank you, for answers. Finally, actually, I got "pro level" answers instead of just "works fine for me", "can u show ur logfiles", "just get SSD bruh". ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif) I got a cheap RAM upgrade, set dbcache to 16GB (was 4GB or 6GB), started syncing. At some point the GUI had stalled very badly, I could do nothing so I shut down Bitcoin-qt and it shut down cleanly. When I restarted it the blockchain was in fully synced state. So my remaining 5 weeks were synced in 2 to 3 hours, much faster than before. I don't know if the GUI would have recovered if I had waited for some more time. Maybe I tried to use it just at the critical moment. It's unknown for now. Monero did not get faster than before and the last 50000 to 100000 blocks seem to be much slower than anything before. I didn't find any option similar to dbcache in Bitcoin. 600 blocks/h is the speed now. Used to be 2200 blocks/h for quite long. I benchmarked different file systems and certain types were occasionally faster than that but fell to the same slow level at some point. Maybe it's SSD time next.
|
|
|
|
larry_vw_1955
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2024, 02:36:01 AM |
|
In fact the second setup may be faster than the offered download speed.
I have used 200Mbps and 128gb ram 1tb nvme2 ssd threadripper cpu
My setup was able to do 190Mbps constantly which makes me suspect my internet connection was my bottle neck
I will soon get 1gb internet and try to see how fast I can do this.
how much did that setup cost you? i'm guessing around $2500. most people would be better off just buying bitcoin instead of trying to spend that money to download the blockchain. it's just going to be an ever increasing game of having to throw more and more hardware at the problem big waste of money unless you just have that stuff lying around from something else...
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17277
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 07, 2024, 07:39:59 AM |
|
Monero did not get faster than before and the last 50000 to 100000 blocks seem to be much slower than anything before. I didn't find any option similar to dbcache in Bitcoin. 600 blocks/h is the speed now. Used to be 2200 blocks/h for quite long. I benchmarked different file systems and certain types were occasionally faster than that but fell to the same slow level at some point.
Maybe it's SSD time next. Last time I synced Monero (on a server with 32 GB RAM, and HDD), the first thing it told me was to get a SSD. I couldn't do that, and eventually removed Monero because even after it was synced, the entire server felt "sluggish" while it was running.
|
|
|
|
ABCbits
Legendary
Offline
Activity: 2982
Merit: 7821
Crypto Swap Exchange
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 08, 2024, 10:28:34 AM |
|
At some point the GUI had stalled very badly, I could do nothing so I shut down Bitcoin-qt and it shut down cleanly. When I restarted it the blockchain was in fully synced state. So my remaining 5 weeks were synced in 2 to 3 hours, much faster than before.
I don't know if the GUI would have recovered if I had waited for some more time. Maybe I tried to use it just at the critical moment. It's unknown for now.
Based on my experience, the GUI would be responsive again and show latest information. Although if you need to know latest sync progress, you could open debug.log and look for newest line which contain text "UpdateTip". Monero did not get faster than before and the last 50000 to 100000 blocks seem to be much slower than anything before. I didn't find any option similar to dbcache in Bitcoin. 600 blocks/h is the speed now. Used to be 2200 blocks/h for quite long. I benchmarked different file systems and certain types were occasionally faster than that but fell to the same slow level at some point.
Maybe it's SSD time next.
There are other parameter you could try, such as --block-sync-size, --fast-block-sync and --db-sync-mode. If you have question specifically about Monero, you should create new thread/reply on different board.
|
|
|
|
kevinteohuiwei
Newbie
Offline
Activity: 1
Merit: 0
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 09, 2024, 02:11:18 PM |
|
I just got mine sync and its 660GB .... done in 5 days with a 3MBPS line on my phone data. Just 16gb amd ryzen 9 7950 and 1tb ssd ext drive.
Yea, i did set the cache to 1800MB compare to originally 450MB. It does help. Even on my raspberry pi 5, i did sync it within 5 days on the same ext drive.
same core version 27.0. I think it could be the cache 450MB default setting. Try changing it and trying different setting. Hope it helps.
|
|
|
|
BlackHatCoiner
Legendary
Offline
Activity: 1624
Merit: 7958
Bitcoin is a royal fork
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 13, 2024, 07:44:37 PM |
|
Does syncing get harder over time as the blockchain gets longer? It depends on the transactions. If the future transactions create more UTXO than spend, then it will be worse over time, yes. Or has the code quality of Bitcoin Core gone down? I believe it's just your HDD. It's extremely slow to download and index all that information on an HDD. Have you tried SSD? Short answer, yes. Not only each block become bigger, total UTXO keep growing. How do you know that the UTXO keeps growing? Are there stats that support this?
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17277
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 13, 2024, 08:46:16 PM |
|
How do you know that the UTXO keeps growing? Are there stats that support this? I don't keep track of UTXOs, but I have data on funded addresses. Apart from small temporary reductions (usually due to consolidation at low fees), they're generally growing in number. The way I see it: Bitcoin gets more expensive, and more people own smaller and smaller bits of it.
|
|
|
|
BlackBoss_
Sr. Member
![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif) ![*](https://bitcointalk.org/Themes/custom1/images/star.gif)
Offline
Activity: 742
Merit: 437
Rollbit is for you. Take $RLB token!
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 14, 2024, 03:35:42 AM |
|
The way I see it: Bitcoin gets more expensive, and more people own smaller and smaller bits of it.
It's natural that with a same cake, if there are more people know about the cake, want to have pieces of the cake, each will have smaller piece. People who have bitcoins need to take care of their bitcoins, hold their coins tightly and sit down to think of solid plans for investment, holding and personal, family financial plans. They need very good plans to avoid of any situation that forces them to sell their bitcoins and turn their pieces to be smaller and smaller ones with time. They had chances to join the market earlier than late comers but if they don't have proper plans, they will end with no differences to late comers. Bitcoin: Addresses Holding > X BTC by Year
|
|
|
|
R |
▀▀▀▀▀▀▀██████▄▄ ████████████████ ▀▀▀▀█████▀▀▀█████ ████████▌███▐████ ▄▄▄▄█████▄▄▄█████ ████████████████ ▄▄▄▄▄▄▄██████▀▀ | LLBIT | | | 4,000+ GAMES███████████████████ ██████████▀▄▀▀▀████ ████████▀▄▀██░░░███ ██████▀▄███▄▀█▄▄▄██ ███▀▀▀▀▀▀█▀▀▀▀▀▀███ ██░░░░░░░░█░░░░░░██ ██▄░░░░░░░█░░░░░▄██ ███▄░░░░▄█▄▄▄▄▄████ ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀ | █████████ ▀████████ ░░▀██████ ░░░░▀████ ░░░░░░███ ▄░░░░░███ ▀█▄▄▄████ ░░▀▀█████ ▀▀▀▀▀▀▀▀▀ | █████████ ░░░▀▀████ ██▄▄▀░███ █░░█▄░░██ ░████▀▀██ █░░█▀░░██ ██▀▀▄░███ ░░░▄▄████ ▀▀▀▀▀▀▀▀▀ |
| | | ██░░░░░░░░░░░░░░░░░░░░░░██ ▀█▄░▄▄░░░░░░░░░░░░▄▄░▄█▀ ▄▄███░░░░░░░░░░░░░░███▄▄ ▀░▀▄▀▄░░░░░▄▄░░░░░▄▀▄▀░▀ ▄▄▄▄▄▀▀▄▄▀▀▄▄▄▄▄ █░▄▄▄██████▄▄▄░█ █░▀▀████████▀▀░█ █░█▀▄▄▄▄▄▄▄▄██░█ █░█▀████████░█ █░█░██████░█ ▀▄▀▄███▀▄▀ ▄▀▄▀▄▄▄▄▀▄▀▄ ██▀░░░░░░░░▀██ | | | | | | | . ▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄ ░▀▄░▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄░▄▀ ███▀▄▀█████████████████▀▄▀ █████▀▄░▄▄▄▄▄███░▄▄▄▄▄▄▀ ███████▀▄▀██████░█▄▄▄▄▄▄▄▄ █████████▀▄▄░███▄▄▄▄▄▄░▄▀ ████████████░███████▀▄▀ ████████████░██▀▄▄▄▄▀ ████████████░▀▄▀ ████████████▄▀ ███████████▀ | ▄▄███████▄▄ ▄████▀▀▀▀▀▀▀████▄ ▄███▀▄▄███████▄▄▀███▄ ▄██▀▄█▀▀▀█████▀▀▀█▄▀██▄ ▄██▀▄██████▀████░███▄▀██▄ ███░█████████▀██░████░███ ███░████░█▄████▀░████░███ ███░████░███▄████████░███ ▀██▄▀███░█████▄█████▀▄██▀ ▀██▄▀█▄▄▄██████▄██▀▄██▀ ▀███▄▀▀███████▀▀▄███▀ ▀████▄▄▄▄▄▄▄████▀ ▀▀███████▀▀ | | OFFICIAL PARTNERSHIP FAZE CLAN SSC NAPOLI | | |
|
|
|
nc50lc
Legendary
Offline
Activity: 2520
Merit: 5967
Self-proclaimed Genius
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 14, 2024, 05:30:26 AM |
|
At some point the GUI had stalled very badly, I could do nothing so I shut down Bitcoin-qt and it shut down cleanly. -snip- I don't know if the GUI would have recovered if I had waited for some more time. Maybe I tried to use it just at the critical moment. It's unknown for now.
That point must be after the specific block hash set here: github.com/bitcoin/bitcoin/blob/c4d45b695ef27e21d35e886b08887e2ecf272ce7/src/kernel/chainparams.cpp#L108But it should've recovered after it's finished with all the heavy signature script verifications. Finally, actually, I got "pro level" answers instead of just "works fine for me", "can u show ur logfiles", "just get SSD bruh". Tongue Well, a " pro" will ask for the debug.log to look for the root of the issue instead of suggesting solutions that may not be necessary. The reason why you're not asked for logs here is because slow sync is pretty much hardware and/or database cache issue and the " dbcache" solution that you found is consistent in topics like this. If you followed this format ( sticky topic): [READ BEFORE POSTING] Tech Support Help Request FormatSomeone pro can easily point that your ram and dbcache settings can be improved just from looking at the logs and the provided info.
|
|
|
|
|
philipma1957
Legendary
Offline
Activity: 4228
Merit: 8392
'The right to privacy matters'
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 14, 2024, 11:35:02 PM |
|
In fact the second setup may be faster than the offered download speed.
I have used 200Mbps and 128gb ram 1tb nvme2 ssd threadripper cpu
My setup was able to do 190Mbps constantly which makes me suspect my internet connection was my bottle neck
I will soon get 1gb internet and try to see how fast I can do this.
how much did that setup cost you? i'm guessing around $2500. most people would be better off just buying bitcoin instead of trying to spend that money to download the blockchain. it's just going to be an ever increasing game of having to throw more and more hardware at the problem big waste of money unless you just have that stuff lying around from something else... that pc was the best i ever owned. it was serious money the threadripper alone was 2500 you don’t need all that. Any i5 intel 8500 or newer 16 gb better to have 32gb and any good 2tb ssd. nmve 2 oh. 200. mh internet .
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17277
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 15, 2024, 06:00:14 AM |
|
Any i5 intel 8500 or newer
16 gb better to have 32gb
and any good 2tb ssd. nmve 2
oh. 200. mh internet . This is still excessive for syncing a Bitcoin blockchain. All it does is shave a few hours off the total sync time. I'd base my computing requirements on my daily needs, not on a one-time event.
|
|
|
|
larry_vw_1955
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 16, 2024, 02:25:31 AM |
|
Any i5 intel 8500 or newer
16 gb better to have 32gb
and any good 2tb ssd. nmve 2
oh. 200. mh internet . This is still excessive for syncing a Bitcoin blockchain. All it does is shave a few hours off the total sync time. I'd base my computing requirements on my daily needs, not on a one-time event.well i don't know what world you are living in but a core i5 is not excessive and neither is a 2TB storage space especially if 25% of that is being hogged up by the blockchain...
|
|
|
|
broflof (OP)
Newbie
Offline
Activity: 3
Merit: 4
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
July 20, 2024, 09:08:12 AM |
|
Got SSDs and tested. My conclusion is here.
So,
With HDD and maybe 4GB for dbcache, 2 to 3 weeks in 12 hours. GUI stalled badly.
With HDD and 16GB for dbcache, 5 weeks in 2 to 3 hours. GUI still stalled badly I think.
With SSD and 16GB for dbcache, 2 weeks in less than one hour. Maybe less than 30 minutes? And GUI felt less unresponsive.
Roughly, 10 times speed increase with bigger dbcache, another 2x increase with SSD and same dbcache.
My benchmarks aren't equal because I didn't have much blockchain left when I got RAM and SSD upgrades. But roughly.
With Monero,
RAM upgrade didn't help.
With HDD speed was 500 to 600 blocks/h during the final 50000 to 100000 blocks.
With SSD, speed was around 5000 blocks/h, a bit more or a bit less at times. So 10x increase in speed.
A year ago perhaps, when I benchmarked file systems, I got over 5000 blocks/h on HDD. Up to 9000 blocks/h at times. But Monero has some kind of checkpointing that speeds up syncing I think? So these cannot be compared to today's results?
It must be said that this computer is not new, and my cryptocurrency stuff are running in a virtual machine. The disk is passed to the virtual machine and I don't know how much overhead this brings.
|
|
|
|
|