ElectrumX - extremely long sync time

[Rath_]

Hey,

I am running a virtual machine under Proxmox with bitcoind and ElectrumX. It took about 24 hours for the ElectrumX to process about 430,000 blocks. It's been a few more hours and only 15,000 blocks were processed.

ElectrumX dbcache is set to 2000 MB. Bitcoind dbcache is 2048 MB. I assigned 4 cores and 5 GB of RAM to the VM. htop reports that electrumx-server process uses less than 10% of the CPU (11th gen Intel i5). Both bitcoind and ElectrumX use less than 2 GB of RAM in total. They both read/write data from the same external SSD.

Height       Machine A     Machine B
447,168       2d 13h 20m      9h 47m

Machine B: a late 2012 iMac running Sierra 10.12.2, 2.9GHz quad-core Intel i5 CPU with an HDD and 24GB RAM. Running bitcoind on the same machine. DB_CACHE set to 1,800. LevelDB

I wonder if there is a major bottleneck in my setup or if I misconfigured something.

[ABCbits]

I've used ElectrumX, but since i never use Proxmox (VirtualBox is easier for casual user/geek), i have 3 common suggestion
1. Check I/O usage of the partition, is it always near 100%?
2. Check whether proxmox have disk performance problem. I mention it because other virtualization (such as WSL) have this problem.
3. Check log file created by ElectrumX, does it give any hint?

[Rath_]

2. Check whether proxmox have disk performance problem. I mention it because other virtualization (such as WSL) have this problem.

I have just realised that I have plugged in my SSD into an USB 2.0 port It looks like my motherboard has only two USB 3.2 gen 2 ports and I couldn't get passthrough to work for them, so I plugged in my SSD into some random working port. I will report back once I have managed to get it to work. How much time did your sync take?

[DireWolfM14]

2. Check whether proxmox have disk performance problem. I mention it because other virtualization (such as WSL) have this problem.

I have just realised that I have plugged in my SSD into an USB 2.0 port It looks like my motherboard has only two USB 3.2 gen 2 ports and I couldn't get passthrough to work for them, so I plugged in my SSD into some random working port. I will report back once I have managed to get it to work. How much time did your sync take?

After reading the OP I was going to say that the external drive sounds like your most likely bottle neck.  If you have an open SATA port internally, that's probably the best way to reduce the synchronization time.

I have a dedicated tower with an I5 processor, 4 gigs of RAM, and a 1TB ssd running both bitcoind and ElectrumX.  IIRC it took at least a couple of days, maybe three for ElectrumX to fully sync.

[Rath_]

I managed to get USB 3 passthrough to work by using a front panel connector and replacing the data cable. The sync ETA has dropped from 13 days to just two and I can see that the VM is now using up to 30% of the assigned CPU power.

If you have an open SATA port internally, that's probably the best way to reduce the synchronization time.

I don't think there is a significant performance difference between a SATA and an USB 3.0 SSD. Even if I disassembled my SSD, it would probably have an mSATA connector which my motherboard does not have. I will upgrade to an NVMe SSD at some point, but I wanted to reuse an SSD from my previous Raspberry Pi setup for the time being.

[Rath_]

For future reference: the sync up to the 684,456 block height took 6 days and 15 hours. It would have been faster if I had noticed earlier that my external drive was connected to a USB 2.0 port. The database directory weights about 66 GB.

[HCP]

I would be interested to know how much of that 6 days and 15 hour sync was at USB2.0 and how much was at USB3.0?

I've been considering setting up an ElectrumX server... but if the sync is really that long, even using USB3.0, I'm not sure it is worth the time investment for what is essentially just a "test".

[DireWolfM14]

It took my system some time to sync, but I'm sure it wasn't a week.  Like I said before, I think it was a max of three days and IIRC the block height was around 660k at the time, so not that long ago.  I'm using an internal sata ssd, and I doubt my internet connection throttled the process in any way.

The first time I set up ElectrumX I had just built the system for it, sync'd bitcoind, then realized I hadn't turned on txindex.  
I had to sync the bitcoin blockchain again, then sync ElectrumX.  It seemed like I was waiting for a month.

[Rath_]

I would be interested to know how much of that 6 days and 15 hour sync was at USB2.0 and how much was at USB3.0?

My SSD was connected to the USB 2.0 port for the first 2 days. About 430,000 blocks were processed in the first 24 hours and about 25,000 blocks in another 24 hours.

I've been considering setting up an ElectrumX server... but if the sync is really that long, even using USB3.0, I'm not sure it is worth the time investment for what is essentially just a "test".

If it is just for testing, I can share my database with you. I have no idea where to upload it, though.

[NeuroticFish]

I have no idea where to upload it, though.

If there are no security concerns with the data, https://wetransfer.com/ does the job gracefully.
You can click the (...) and select "Get Transfer Link" and you won't also need to mess with e-mails.
Then, after upload, you'll just get a link you can post or PM.

[Rath_]

If there are no security concerns with the data, https://wetransfer.com/ does the job gracefully.

2 GB of upload for free accounts and 20 GB for paid users are nowhere near the 67 GB which I would have to send. Thanks for your suggestion, though!

[NeuroticFish]

If there are no security concerns with the data, https://wetransfer.com/ does the job gracefully.

2 GB of upload for free accounts and 20 GB for paid users are nowhere near the 67 GB which I would have to send. Thanks for your suggestion, though!

Ouch. Sorry.
You are kind too, it's clearly not as useful as I first thought.

You can make 2GB volumes, but it's not as nice and comfortable anymore...  

Edit: I've done a web search and fromsmash.com and terashare.net may not have such limits, although terashare is a bit odd, "files larger than 10 GB are not uploaded to the Terashare servers. This means that your receiver can only download these large files when your computer is switched on. If you switch your computer off, the download will stop, but it will automatically resume once you come back online."

[DireWolfM14]

I was going to suggest bit torrent, but you can probably sync bitcoind and electrumx five time over before transferring ~60GB that way.

If one of you has a MS Teams account or a Dropbox Pro account, they allow you transfers up to 100GB.   There's also transferxl.com, and transfernow.net which allow up to 5GB transfers for the basic accounts, but you'll have to pay to get more.  None of that is going to be fast, and if one of you has metered data usage that would make it impractical.  I think synchronizing the db yourself is probably the most practical solution.

Or, just play around with testnet.

[HCP]

If it is just for testing, I can share my database with you. I have no idea where to upload it, though.

Thanks for the offer... but that's just not really feasible.

Testnet might be an option for my particular use case... but, for #reasons, the preference would be for "live" data on mainnet... at this stage, I think I'll just leave it and maybe look at it again when I have some more time.

[DireWolfM14]

I just set up another system to run bitcoin core and ElectrumX.  My other system has a 1TB SSD that I want to repurpose, so I decided to set up another system before taking the first one down.

Machine: Dell Optiplex
Processor: Intel Core i5-4590 @ 3.30GHz
Ram: 16GB Total - 4x 4GB DDR3 @ 1600MHz
Hard Drive: Seagate Barracuda STX-720011
OS: Ubuntu Server 20.04.3

I didn't bother copying over either of the data bases, I wasn't in that big of a hurry.  I still need some components for my other project and I'm waiting until those go on sale in the next few months.  I used addnode to add my other server, so bitcoin core took about 24 hours to synchronize, probably not much longer than it would have taken to transfer 411GB over the LAN anyway.  I was asleep for a third of that time and at work for another third, so it didn't hurt much.

ElectrumX took a little longer to synchronize, about 36.5 hours.

Code:

Sep 03 09:06:02 dell-xxxxxxxx electrumx_server[791]: INFO:DB:sync time: 1d 12h 24m  ETA: 00s
Sep 03 09:06:02 dell-xxxxxxxx electrumx_server[791]: INFO:BlockProcessor:ElectrumX 1.16.0 synced to height 698,860

[HCP]

Seems the take away from all this is that while backups are not strictly necessary (as you can always just resync)... they can be a huge "time saver"!

36.5 hours to sync ElectrumX is fairly slow. Even electrs on my Raspberry Pi 4B didn't take anywhere near that long to build it's databases.

[Rath_]

ElectrumX took a little longer to synchronize, about 36.5 hours.

That's interesting. I wonder what is the bottleneck in my system then. At first, I thought that the USB SSD was the culprit but it should still be faster than your HDD. Does anyone have any idea what could be wrong with my setup?

[DireWolfM14]

36.5 hours to sync ElectrumX is fairly slow. Even electrs on my Raspberry Pi 4B didn't take anywhere near that long to build it's databases.

It seems rather slow by modern internet standards, but compared to a torrent it's not crazy.

That's interesting. I wonder what is the bottleneck in my system then. At first, I thought that the USB SSD was the culprit but it should still be faster than your HDD. Does anyone have any idea what could be wrong with my setup?

I agree there's something about your set up throttling your sync.  I have no experience with Proxmox, but I know VM environments can be finicky.  I imagine you've scrutinized the setting for anything that could hinder the connection.  Did you have to sync core on the system, and if so did that take long as well? Have you tried with a different VM environment?  

[Rath_]

I imagine you've scrutinized the setting for anything that could hinder the connection.

Proxmox should have a negligible impact on the I/O. I suspect that the USB protocol might be imposing some kind of limitation (like limited IOPS?).

Did you have to sync core on the system, and if so did that take long as well?

No, I was using that drive with my previous Raspberry Pi build. I reused it so I didn't have to sync from the beginning. My node catches up quickly when I lose my Internet connection so I guess it wouldn't take a long time.

Have you tried with a different VM environment?

I can't really do that. I have a bunch of other VMs.