[BUG] electrum unable to fully sync a large wallet

[balambala]

Alright so here is my situation :


I am running in local the latest version bitcoin full node index=1, server=1.
I am running in local the latest ElectrumX, docker with network exposure.
I am running Electrum wallet 4.0.2, also latest.


I have a wallet with 1ML addresses in it and I am attempting to sync it to the network, however it fails and restarts again without success for day, screenshot :  https://ibb.co/HdMH0dJ


Now last few months, I did not have this issue because I was running an older version of bitcoin, Electrum Personal Server (not electrumx) and Electrum 3.8.and all addresses and balances were synchronized.

Any idea on a solution ?

[HCP]

Electrum isn't really designed for wallets containing 1 million addresses... I would think that the issue is most likely that the Electrum Server isn't handling the sheer volume of requests very well.

Either break your wallet file down into multiple wallets containing less addresses (ie 10 wallet files with 100,000 or 100 files with 10,000 addresses) or move to a completely different option as I suspect you're likely to continue to face these problems if you need/want to track such a large number of addresses. (why? )

You could also try a different Electrum Server option: https://github.com/romanz/electrs

I use this one, but have less than 100 addresses, so I can't comment on how well it will handle trying to sync 1 million.

[BitMaxz]

According to ThomasV "many Electrum servers have a DoS-protection limit of 10000 addresses."
So there's a possibility that the server you use before can handle unlimited addresses but since you use your own ElectrumX server I think you can edit your own server and increase the limit?

I think you can find this limit under .conf file on ElectrumX folder under "/etc/" folder check that folder and maybe you can find something ElectrumX.conf

Here's the sample text inside the .conf file.

Code:

COIN = Komodo
DB_DIRECTORY = /electrumdb/KMD
DAEMON_URL = http://$rpcuser:$rpcpass@127.0.0.1:<port>/
SERVICES = tcp://:10001,rpc://:8001
EVENT_LOOP_POLICY = uvloop
PEER_DISCOVERY = self
INITIAL_CONCURRENT = 50
COST_SOFT_LIMIT = 10000
COST_HARD_LIMIT = 100000
BANDWIDTH_UNIT_COST = 10000

And I got the idea from https://docs.komodoplatform.com/komodo/setup-electrumX-server.html#coin-specific-part

So the cost_soft_limit; cost_hard_limit and bandwidth_unit_cost is the one responsible for this limit protection?

Check this documentation of ElectrumX about these below.
- https://electrumx.readthedocs.io/_/downloads/en/stable/pdf/

[HCP]

It would indeed appear to be the reason why ElectrumX is disconnecting the client... The "soft" limit is when throttling starts to occur and the "hard" limit is when you get disconnected:

As the total cost of a session goes over the soft limit, its requests start to be throttled in two ways. First, the
number of requests for that session that the server will process concurrently is reduced. Second, each request starts to sleep a little before being handled.
...
Once the hard limit is reached, the session is disconnected.

So, not only will it eventually disconnect you, but due to the throttling it'll slow down your syncing when you hit the soft limit.



additionally...

If a session disconnects, ElectrumX continues to associate its cost with its IP address, so if it immediately reconnects it will re-acquire its previous cost allocation

So, you will just continuously get disconnected once you hit that hard limit

I'd suggest setting the soft and hard limits up to a value of 100000000 and the bandwidth cost to 1000000... that should hopefully stop ElectrumX from reaching the hard limit and disconnecting your client

[balambala]

Thank you all for your replies.

You were all right, I passed the following parameters to docker and was able to fully sync the wallet.

Code:

-e COST_SOFT_LIMIT=1000000 -e COST_HARD_LIMIT=1000000000 -e INITIAL_CONCURRENT=100 -e BANDWIDTH_UNIT_COST=1000000