LoyceV's 0.1 sat/vbyte Electrum Server Adventure

[Cricktor]

I tested this watch-only wallet with LoyceV's chosen 1000 addresses on my Linux desktop running appimage of Electrum 4.6.2 under Ubuntu 22.04.x connected via LAN to my Fulcrum 2.1 server (Bitcoin Core, Fulcrum and a few other containers run in a Virtualbox VM hosted on Linux Mint running on a Dell Latitude E7470 with 32GB RAM).

The test wallet synced after approx. 18m52s +/- few seconds, where Electrum sat quite some minutes during sync at 13244/20852 kinda frozen and seemingly doing nothing and not counting up on the sync. But then larger count jumps in the sync happened. I have no idea of the logic behind this and what Electrum is doing in those periods of no visual progress. No errors where thrown out (wallet and server).

The wallet synced to 9426 transactions (4 are newer and added compared to LoyceV's post of his results) with a balance of 372.79298051BTC, last transaction was a0219bd9...d26d6045 confirmed 2026-02-14 21:39:35 UTC.

As I don't have any electrs (romanz) or other Electrum server running, I can't make a comparison to something else than Fulcrum on my hardware. But I will try to find a public electrs and see how and if it syncs this wallet. I'll amend results if I have any.

[LoyceV]

Did I manage to explain the idea?

A bit But I wouldn't know how to do this in Electrum, other than manually adding the addresses. But that doesn't give a good overview.

The test wallet synced after approx. 18m52s +/- few seconds

This makes my test even look good

I have no idea of the logic behind this and what Electrum is doing in those periods of no visual progress.

That's what made me think this:

Or maybe Electrum itself gets exponentially more inefficient when loading many addresses?

If anyone wants to test with a larger address sample (I don't dare do this to my Electrum as I'm pretty sure it will run out of memory):

Code:

wget http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz -O- | gunzip | shuf | head -n10000 > 10000_randomaddresses.txt

tmp.loyce.club/10000_randomaddresses.txt

[joker_josue]

Did I manage to explain the idea?

A bit But I wouldn't know how to do this in Electrum, other than manually adding the addresses. But that doesn't give a good overview.

This would be done more on the server side, via Fulcrum. Let's say it would be asking it to create a list of addresses near address X.

But never mind, it was just to get an idea for a project I'm planning. Thank you.

[Cricktor]

This makes my test even look good

Yes, sort of. I had no expectations how my Fulcrum will serve its data job. All I want is it to finish the sync in a reasonable time.


I tried the 1000 addresses collection with public Electrum servers. Oh boy, that didn't age well.

My first attempt was to choose one or two servers manually, e.g. electrum.emzy.de:50002, but this one kicked the connection after about 8min of sync. My wallet could reconnect but it didn't last long. So, I conclude to get banned for exhaustive resource demand.

I tried another public server but my wallet kept resetting "Synchronizing..." and spewed error messages. I aborted trying to stick to one particular server (electrs or ElectrumX, not another Fulcrum server, as long as the server banner shows what kind of a server I'm connected to).

Next attempt was to switch to "Auto-connect" mode and hope this works better? Well... my wallet somehow found a "patient" server 34.18.69.244:50002, but even after ~7h30m sync wasn't finishes, maybe 2/3 through until that Electrum server gave up on me.

Code:

27007.73 | W | interface.[34.18.69.244:50002] | disconnecting due to GracefulDisconnect(RPCError(-101, 'excessive resource usage'))

Not sure how patient I will still be today, other servers keep disconnecting...

Code:

29251.70 | E | synchronizer.[wallet_5_temp260215] | taskgroup died (0x7ae1782cda00). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['25bd8fabe42d2b200965ccf173e0de4c874bfed99c6df5cb48ba586a208bdd66']) (id: 3136)")

This is exactly what I hated with public servers, when I did some occasional blockchain digging and research. You stumble over address histories that simply aren't served gracefully.

I had such issues with my personal electrs server, too, which I couldn't configure away. That's why I switched to Fulcrum which simply works very gracefully so far. It finishes the serving job! That the receiving Electrum wallet sometimes chokes hard is another story to be examined.

Post edit: the wallet finished syncing after around 8h19m minus a few minutes because I missed the moment it finished. The last error message spewed to stdout had a timestamp of 8h13m. Balance and number of transaction is the same as with my Fulcrum server. At least all the warnings and errors didn't screw up the important data.

Code:

wget http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz -O- | gunzip | shuf | head -n10000 > 10000_randomaddresses.txt

tmp.loyce.club/10000_randomaddresses.txt

I'll likely give it a try. Am genuinely curious how this will turn out.

[LoyceV]

I'll likely give it a try. Am genuinely curious how this will turn out.

Me too It's a bit tricky though to use random addresses: one of them could have had millions of transactions. If that's the case, just that address alone would be enough to stop Electrum from syncing.

[Cricktor]

That's true. A random addresses sample may exhibit "surprises".

Anyway, after about 5h18m of choking synchronization or your 10k addresses sample my Electrum wallet got killed, likely due to exhausting memory resources. My Ubuntu desktop has 16GB of RAM, a bitcoind runs 24/7 in the background to maintain a few watch-only wallets. The Fulcrum server (running on another machine) didn't appear to flinch at any moment, at least not visible in its log file.

These were the only errors logged by my Electrum wallet to stdout during the sync:

Code:

13202.07 | E | synchronizer.[wallet_6_temp260215_10k] | taskgroup died (0x7935e3a1d670). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['291196541f27fcd0fc055e8503936db9102dc5057b3444c1fac670fb8ffe8e42']) (id: 3419)")
16766.75 | E | synchronizer.[wallet_6_temp260215_10k] | taskgroup died (0x793536f83890). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['d84e423d056bfe3edf7bf30085c475c2a02b3c45d80609a1e45d1b6a00671f8f']) (id: 4029)")

The process kill happened while I was sleeping, but I had a screencast recording to monitor runtime and what the syncing Electrum wallet does. While my Electrum 10k samples wallet got killed, the bitcoind and recording processes were not noticeably affected, both kept running. Before the Electrum processes got killed the GUI showed about 116k synced transactions in the wallet.

The synchronization lost connection a few times and restarted the sync counters a few times. From what I observed not with repeating patterns and counter targets.

My Fulcrum server is setup to allow large address histories: max_history = 2000000, max_subs_per_ip = 250000 and my Electrum wallet accepts large responses from an Electrum server: "network_max_incoming_msg_size": 80000000, (default value is too low for larger histories, this setting is not even present in default Electrum .electrum/config file).

I might give your 10k sample a try with Sparrow. Maybe Sparrow handles larger address histories without being such a memory hog like Electrum. Though not sure if Sparrow supports watch-only wallets with arbitrarily unrelated addresses (maybe via descriptors). I haven't tried such a thing with Sparrow yet (arbitrary addresses).