Again a topic about wallet sync

[LightningSphere]

I'm writing this because I counld't find a recent topic about this.
Wallet syncing is extremly slow, I'm syncing for 3 days now and it tells me it will take another 3 days. The developers should find a solution for this, because it is ridiculous, it will prevent people of using bitcoin at all.

From 2009 to januari 2018 takes a day orso, still long, but well I can live with that. But than after this date progress drops and takes another hole week! to get the last 2 years!
Developers FIX this, this not acceptable anymore. Syncing will soon be more than a month, nobody it going to wait a month the get their wallet synced.
I'm seriously considering getting my coins out of the wallet when I'm synced and change them into another crypto, to never touch a bitcoin wallet again...

[Rath_]

Developers FIX this, this not acceptable anymore. Syncing will soon be more than a month, nobody it going to wait a month the get their wallet synced.
I'm seriously considering getting my coins out of the wallet when I'm synced and change them into another crypto, to never touch a bitcoin wallet again...

The sync time depends not only on the client's code but also on your hardware. Upgrading to an SSD made a huge difference in my case. Your CPU is also important since you are also verifying all the blocks while downloading them. If you don't want to wait then you can use light clients, for example, Electrum as long as you follow proper security measures (verifing installation files, downloading them from the official websites and avoiding malware by even using a separate computer for cryptocurrencies).

[jackg]

There are a lot of alternatives to running a full node and waiting this long to download. I think the favourite for desktop at the moment is electrum (from electrum.org/#download).

You can wait for core to finish syncing though if you want to it's much faster than it used to be when I started. You might want to tell us your system specs too a standard laptop should handle it in a few days, you could try upgrading your hardware if you want to run a full node like putting an ssd in your hard drive slot. You could try looking at other ways to improve performance from the confignfile if you have spare ram.

[LightningSphere]

When I look at the system performance it seems the HD is the bottleneck. CPU is only at 15%, RAM 3.9G/8G, Ethernet below 5%, but the HD is 100% flat.
So maybe SDD will improve sync, I will try that.
But it raises a question for me, why is syncing so hard on the HD? Blockchain is about 250G, that's only a few minutes for a SATA6 drive

[Rath_]

But it raises a question for me, why is syncing so hard on the HD? Blockchain is about 250G, that's only a few minutes for a SATA6 drive

Sequential reads and writes are not important. You should be more concerned about input/output operations per second (IOPS). HDDs usually reach ~75 IOPS while SATA III SSDs can go up to 120000 IOPS or more. NVMe SSDs are even better in this aspect.

[LightningSphere]

But it raises a question for me, why is syncing so hard on the HD? Blockchain is about 250G, that's only a few minutes for a SATA6 drive

Sequential reads and writes are not important. You should be more concerned about input/output operations per second (IOPS). HDDs usually reach ~75 IOPS while SATA III SSDs can go up to 120000 IOPS or more. NVMe SSDs are even better in this aspect.

So sync performance could be improved if the wallet would cluster the read/write operations. Not sure if this would be possible, I never looked at the wallet code.

[Rath_]

So sync performance could be improved if the wallet would cluster the read/write operations. Not sure if this would be possible, I never looked at the wallet code.

No, sequential read and write speeds are relevant only to large files. While synchronizing, you are dealing with a large amount of small files which are being verified by the client one by one.

[Lauda]

But it raises a question for me, why is syncing so hard on the HD? Blockchain is about 250G, that's only a few minutes for a SATA6 drive

You're not transferring data, you're downloading and validating it. The later being very resource heavy.

When I look at the system performance it seems the HD is the bottleneck. CPU is only at 15%, RAM 3.9G/8G, Ethernet below 5%, but the HD is 100% flat.

You seem to have room for a large dbcache, that should help. Try setting it to 2 or 3 GB.

So maybe SDD will improve sync, I will try that.

In most cases the HDD is the most limiting factor in sync speed, yes.

[LoyceV]

So maybe SDD will improve sync, I will try that.

That's not a "maybe", it's a given

I keep my "blocks" directory on my old hdd, and the rest of Bitcoin Core on ssd. Especially the chainstate-directory benefits a lot from much faster storage.

I'm seriously considering getting my coins out of the wallet when I'm synced

That would be a waste: once synced, you'll never have to go through this again. Just keep Bitcoin Core running and it downloads 6 blocks per hour to remain fully synced.

[Jet Cash]

I can keep my node synchronised over public WiFi without much difficulty. That's on a Windows 10 machine with an HDD. As others have said, it is much faster if I use a Linux machine with an SSD. I can handle the current situation, and I appreciate the extra security that comes from it. I may well switch to keeping a pruned node in the future though.

[LFC_Bitcoin]

I can keep my node synchronised over public WiFi without much difficulty. That's on a Windows 10 machine with an HDD. As others have said, it is much faster if I use a Linux machine with an SSD. I can handle the current situation, and I appreciate the extra security that comes from it. I may well switch to keeping a pruned node in the future though.

Me too, in another 5-10 years it’s going to become ridiculous to keep up (storage wise).

[LightningSphere]

Tried the SDD (970 EVO 1Tb) and it did make a HUGE difference. I synced the complete blockchain in two days. My 20bps internet was the next limiting factor, but two days is ok for me.

[wwzsocki]

Tried the SDD (970 EVO 1Tb) and it did make a HUGE difference. I synced the complete blockchain in two days. My 20bps internet was the next limiting factor, but two days is ok for me.

Such a summary was needed at the end of this topic, exactly when I wanted to ask about the time difference between synchronization on HDD and SDD.

I think 2 days is acceptable, taking into consideration slow internet, sync can be possibly improved to 24 hours or even less on really good performing machines.

Still, I am surprised that there is no other way to sync faster. I was sure this is possible with the help of bootstrap files but maybe not for Bitcoin Core client?

[ABCbits]

I was sure this is possible with the help of bootstrap files but maybe not for Bitcoin Core client?

That won't help at all, you still need to verify all blocks from the bootstrap file. The speed when you download from other nodes (for Bitcoin Core or P2P client) and centralized server more/less is same.

Besides, you need to trust the uploader not to give malicious/altcoin bootstrap file.

[jackg]

Still, I am surprised that there is no other way to sync faster. I was sure this is possible with the help of bootstrap files but maybe not for Bitcoin Core client?

I think they optimised it too so it's faster to do it by core of its at minimum the same. Unless you buy hardware with the blockchain on it (and verify a few merkle roots, hashes and tx) then it'll still take the same amount of time. 

[wwzsocki]

...The speed when you download from other nodes (for Bitcoin Core or P2P client) and centralized server more/less is same.
Besides, you need to trust the uploader not to give malicious/altcoin bootstrap file.

...it's faster to do it by core of its at minimum the same...

Always when there were problems with synchronization of the wallets in many projects, the first solution provided was the bootstrap file with all blockchain data, mostly ending on the last checkpoint. But to be honest, I have never seen this solution for Bitcoin Core client. When I have used bootstrap files a couple of times, always the synchronization was smooth and took only a fraction of the time needed if I would try to sync without the file. It was like a couple of days, to a few hours difference.

Keep thinking about this and maybe bootstrap file works better (faster) in POS coins because if I remember correctly, I have used it only for alts like Pivx, Dash, Bitsend, Masternodecoin and never in POW mined coins like our old good Bitcoin?

[mocacinno]

Some nice advice has been given, but i'd like to put some emphasis on part of BitCryptex and jackg's initial posts... They are correct, but it seems their posts have been overlooked...

Running a full node isn't necessary if you don't want to... If you have problems with the time it takes to do a full sync, or the amount of data you need to store, start looking at an SPV wallet.
An SPV wallet only downloads the block headers, not the complete blocks. A "full sync" of all headers happens in a matter of minutes and only uses a couple Mb on disk, and i find SPV wallets sufficient for most day-to-day task.
Let's be honest: it's a good thing for the network if a lot of people run full nodes, it's good for an enduser to verify all blocks and all transactions, but it isn't absolutely necessary for everybody to run such a full node if they are only going to perform enduser tasks. If we have several thousand full nodes, endusers can use SPV wallets without significant impact on the network's security (imho).

Granted, if you use an SPV wallet, you do rely on other nodes and you give away a small part of your privacy... But security-wise SPV wallets seem to be fine.
If you worry so much about synctime and diskusage, download and verify the latest version of electrum, or buy a ledger or trezor (both spv hardware wallets). By using a HW wallet, you'll even add an extra layer of security....

[NeuroticFish]

Keep thinking about this and maybe bootstrap file works better (faster) in POS coins because if I remember correctly, I have used it only for alts like Pivx, Dash, Bitsend, Masternodecoin and never in POW mined coins like our old good Bitcoin?

Bitcoin has bootstrap file too in the past. But since about version 0.10 something has change and the verify blocks steps seems to be the bottleneck, being slower than the actual download. And yeah, bootstrap.dat is a security risk. But that doesn't stop people post and use bootstrap. Old habits don't die easy.


And whoever doesn't have the time, space, willingness to do this sync properly, there are plenty of good SPV wallets. That's what I also use.

[wwzsocki]

Some nice advice has been given, but i'd like to put some emphasis on part of BitCryptex and jackg's initial posts... They are correct, but it seems their posts have been overlooked...

I don't think these posts were overlooked.

OP take these suggestions seriously and even changed his drive to SSD which significantly accelerated the synchronization speed as @Bitcryptex advised.

Additionally, @Jackg suggested alternatives like SPV wallets (electrum) for people who don't really need a full node and this was also discussed above.

...bootstrap.dat is a security risk. But that doesn't stop people post and use bootstrap. Old habits don't die easy...

Of course, one should do all what possible to check such file before download but this is true that security risk is the main reason that bootstrap files should not be used if possible.

[wwzsocki]

...Keep thinking about this and maybe bootstrap file works better (faster) in POS coins because if I remember correctly, I have used it only for alts like Pivx, Dash, Bitsend, Masternodecoin and never in POW mined coins like our old good Bitcoin?

That's because Bitcoin have many full nodes with fast upload speed while AFAIK most altcoin have few full nodes with mediocre upload speed.

If there aren't many Bitcoin full node or upload speed of bitcoin node isn't fast, downloading bootstrap would be faster.

I was afraid that the number of full nodes may decrease drastically taking into consideration the size of the blockchain which is technically too high for a hard disk of a PC.

I was running a full node for almost 4 years because Bitcoin-QT was the only one available wallet when I started to use and later mine BTC. Additionally, I always heard that the best thing we can do to support the network is to run a full node but in today's reality, I rather doubt if average people decide to do it through downloading the whole blockchain because of the size. There is no comparison to the times when a few hours of synchronization and a medium-class computer was enough to be a happy owner of a full node.

Despite everything, the number of full nodes increases every year and today we have already 9137 according to https://bitnodes.earn.com/. It looks like a phenomenon to me.



It would be great if some solution could be implemented to speed up the synchronization because it would be only worst during the time.