LoyceV's 0.1 sat/vbyte Electrum Server Adventure

[Cricktor]

I tested this watch-only wallet with LoyceV's chosen 1000 addresses on my Linux desktop running appimage of Electrum 4.6.2 under Ubuntu 22.04.x connected via LAN to my Fulcrum 2.1 server (Bitcoin Core, Fulcrum and a few other containers run in a Virtualbox VM hosted on Linux Mint running on a Dell Latitude E7470 with 32GB RAM).

The test wallet synced after approx. 18m52s +/- few seconds, where Electrum sat quite some minutes during sync at 13244/20852 kinda frozen and seemingly doing nothing and not counting up on the sync. But then larger count jumps in the sync happened. I have no idea of the logic behind this and what Electrum is doing in those periods of no visual progress. No errors where thrown out (wallet and server).

The wallet synced to 9426 transactions (4 are newer and added compared to LoyceV's post of his results) with a balance of 372.79298051BTC, last transaction was a0219bd9...d26d6045 confirmed 2026-02-14 21:39:35 UTC.

As I don't have any electrs (romanz) or other Electrum server running, I can't make a comparison to something else than Fulcrum on my hardware. But I will try to find a public electrs and see how and if it syncs this wallet. I'll amend results if I have any.

[LoyceV]

Did I manage to explain the idea?

A bit But I wouldn't know how to do this in Electrum, other than manually adding the addresses. But that doesn't give a good overview.

The test wallet synced after approx. 18m52s +/- few seconds

This makes my test even look good

I have no idea of the logic behind this and what Electrum is doing in those periods of no visual progress.

That's what made me think this:

Or maybe Electrum itself gets exponentially more inefficient when loading many addresses?

If anyone wants to test with a larger address sample (I don't dare do this to my Electrum as I'm pretty sure it will run out of memory):

Code:

wget http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz -O- | gunzip | shuf | head -n10000 > 10000_randomaddresses.txt

tmp.loyce.club/10000_randomaddresses.txt

[joker_josue]

Did I manage to explain the idea?

A bit But I wouldn't know how to do this in Electrum, other than manually adding the addresses. But that doesn't give a good overview.

This would be done more on the server side, via Fulcrum. Let's say it would be asking it to create a list of addresses near address X.

But never mind, it was just to get an idea for a project I'm planning. Thank you.

[Cricktor]

This makes my test even look good

Yes, sort of. I had no expectations how my Fulcrum will serve its data job. All I want is it to finish the sync in a reasonable time.


I tried the 1000 addresses collection with public Electrum servers. Oh boy, that didn't age well.

My first attempt was to choose one or two servers manually, e.g. electrum.emzy.de:50002, but this one kicked the connection after about 8min of sync. My wallet could reconnect but it didn't last long. So, I conclude to get banned for exhaustive resource demand.

I tried another public server but my wallet kept resetting "Synchronizing..." and spewed error messages. I aborted trying to stick to one particular server (electrs or ElectrumX, not another Fulcrum server, as long as the server banner shows what kind of a server I'm connected to).

Next attempt was to switch to "Auto-connect" mode and hope this works better? Well... my wallet somehow found a "patient" server 34.18.69.244:50002, but even after ~7h30m sync wasn't finishes, maybe 2/3 through until that Electrum server gave up on me.

Code:

27007.73 | W | interface.[34.18.69.244:50002] | disconnecting due to GracefulDisconnect(RPCError(-101, 'excessive resource usage'))

Not sure how patient I will still be today, other servers keep disconnecting...

Code:

29251.70 | E | synchronizer.[wallet_5_temp260215] | taskgroup died (0x7ae1782cda00). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['25bd8fabe42d2b200965ccf173e0de4c874bfed99c6df5cb48ba586a208bdd66']) (id: 3136)")

This is exactly what I hated with public servers, when I did some occasional blockchain digging and research. You stumble over address histories that simply aren't served gracefully.

I had such issues with my personal electrs server, too, which I couldn't configure away. That's why I switched to Fulcrum which simply works very gracefully so far. It finishes the serving job! That the receiving Electrum wallet sometimes chokes hard is another story to be examined.

Post edit: the wallet finished syncing after around 8h19m minus a few minutes because I missed the moment it finished. The last error message spewed to stdout had a timestamp of 8h13m. Balance and number of transaction is the same as with my Fulcrum server. At least all the warnings and errors didn't screw up the important data.

Code:

wget http://addresses.loyce.club/Bitcoin_addresses_LATEST.txt.gz -O- | gunzip | shuf | head -n10000 > 10000_randomaddresses.txt

tmp.loyce.club/10000_randomaddresses.txt

I'll likely give it a try. Am genuinely curious how this will turn out.

[LoyceV]

I'll likely give it a try. Am genuinely curious how this will turn out.

Me too It's a bit tricky though to use random addresses: one of them could have had millions of transactions. If that's the case, just that address alone would be enough to stop Electrum from syncing.

[Cricktor]

That's true. A random addresses sample may exhibit "surprises".

Anyway, after about 5h18m of choking synchronization or your 10k addresses sample my Electrum wallet got killed, likely due to exhausting memory resources. My Ubuntu desktop has 16GB of RAM, a bitcoind runs 24/7 in the background to maintain a few watch-only wallets. The Fulcrum server (running on another machine) didn't appear to flinch at any moment, at least not visible in its log file.

These were the only errors logged by my Electrum wallet to stdout during the sync:

Code:

13202.07 | E | synchronizer.[wallet_6_temp260215_10k] | taskgroup died (0x7935e3a1d670). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['291196541f27fcd0fc055e8503936db9102dc5057b3444c1fac670fb8ffe8e42']) (id: 3419)")
16766.75 | E | synchronizer.[wallet_6_temp260215_10k] | taskgroup died (0x793536f83890). exc=RequestTimedOut("request timed out: ('blockchain.transaction.get', ['d84e423d056bfe3edf7bf30085c475c2a02b3c45d80609a1e45d1b6a00671f8f']) (id: 4029)")

The process kill happened while I was sleeping, but I had a screencast recording to monitor runtime and what the syncing Electrum wallet does. While my Electrum 10k samples wallet got killed, the bitcoind and recording processes were not noticeably affected, both kept running. Before the Electrum processes got killed the GUI showed about 116k synced transactions in the wallet.

The synchronization lost connection a few times and restarted the sync counters a few times. From what I observed not with repeating patterns and counter targets.

My Fulcrum server is setup to allow large address histories: max_history = 2000000, max_subs_per_ip = 250000 and my Electrum wallet accepts large responses from an Electrum server: "network_max_incoming_msg_size": 80000000, (default value is too low for larger histories, this setting is not even present in default Electrum .electrum/config file).

I might give your 10k sample a try with Sparrow. Maybe Sparrow handles larger address histories without being such a memory hog like Electrum. Though not sure if Sparrow supports watch-only wallets with arbitrarily unrelated addresses (maybe via descriptors). I haven't tried such a thing with Sparrow yet (arbitrary addresses).

[LoyceV]

I pulled the trigger.  They give you 10% discount if you pay for the annual plan, so it ends up being $197.76 for a year, or $16.48 per month.  Not bad for all that you get when compared to the big boys.

In retrospect, you were just in time They've now raised the prices to $330 per year. I don't think they'll raise prices for renewing.
The reason: the sharp increase in hardware cost:

RAM is now costing us approximately 5× more than it did 6 months ago.
NVMe storage is now costing us about 3× more than it did 6 months ago.

AI is driving up memory prices.

[DireWolfM14]

In retrospect, you were just in time They've now raised the prices to $330 per year. I don't think they'll raise prices for renewing.

Wow, that's a significant difference, I'm glad I made up my mind when I did.  One can only cross their fingers that renewal won't cost so much.

The reason: the sharp increase in hardware cost:

I noticed this myself recently when I bought a couple of 2TB SSDs for my servers.  I was dismayed when I saw the prices, I expected they would have fallen from the time of my last purchase.  The opposite is true.

AI is driving up memory prices.

Yet another reason to dislike AI. 

[LoyceV]

I received an email from my webhost: my server is "scheduled for decommissioning". I'll try to keep the same IP-address, but I'm waiting for them to get back to me on that. This may cause some downtime, I hope not too much.

[TryNinja]

I received an email from my webhost: my server is "scheduled for decommissioning". I'll try to keep the same IP-address, but I'm waiting for them to get back to me on that. This may cause some downtime, I hope not too much.

Is this from Servarica? Why are they decommissioning your server?

In retrospect, you were just in time They've now raised the prices to $330 per year. I don't think they'll raise prices for renewing.
The reason: the sharp increase in hardware cost:

That's what I thought about netcup, but they just raised prices for old and new customers.

[LoyceV]

Is this from Servarica?

Yep.

Why are they decommissioning your server?

I don't know I'm kinda curious myself. My VPS was rebooted a couple of months ago, I don't know why, so I can only guess if hardware failure is the reason.

That's what I thought about netcup, but they just raised prices for old and new customers.

It's just one more reason to hate AI

[DireWolfM14]

I received an email from my webhost: my server is "scheduled for decommissioning".

Bummer, dude.  I haven't received any such email.  Is your contract due for renewal some time soon?  If so, I may get the same treatment early next year.

I'll try to keep the same IP-address, but I'm waiting for them to get back to me on that. This may cause some downtime, I hope not too much.

Not sure who's hosting your domain name, but redirecting it to a different IP is usually a pretty simple task.  It may take up to 48 hours to refresh, but it's often much faster than that.

[LoyceV]

Bummer, dude.  I haven't received any such email.  Is your contract due for renewal some time soon?

It's not that bad: the new server is a lot faster. The old one had "CPU steal", even though the cores were supposed to be dedicated. So I think it must have been overloaded. At least it has a faster CPU now. I've paid up to December already.

Not sure who's hosting your domain name, but redirecting it to a different IP is usually a pretty simple task.  It may take up to 48 hours to refresh, but it's often much faster than that.

It's more complicated than that: I have to bug 2 people for an IP-change (theymos for whitelisting and suchmoon because the DNS is hosted by AWS for DDoS protection).

Update: the new server has just as much CPU steal. So far for being dedicated cores, they're also oversold.

[LoyceV]

I still hope to keep my old IP-address, but until that's sorted out, here's my new temporary server location:
162.250.188.205:50002:s

[LoyceV]

My Electrum server on the new IP-address is temporarily down. I needed the disk space to move around other data.

[bitmover]

I am using your electrum server LoyceV. Thanks for making this. I have used 2 weeks ago to make a consolidation of signature campaign addresses .  

About 40 UTXO consolidated a 0.2 sat/vB for just $ 0.46.

I think it is worth making consolidations now, we don't know for how long fees will remain so low

[LoyceV]

About 40 UTXO consolidated a 0.2 sat/vB for just $ 0.46.
I think it is worth making consolidations now, we don't know for how long fees will remain so low

I thought the same years ago, when I consolidated at several sats/vbyte But as always, fees can only go down so much, while there's not really an upper limit. Now is a good time.

Optimistic timeline:
I'm hoping to get my Electrum server running again today. Moving many files from slow storage just takes a lot of time.
I'm also hoping to move my old IP to the new server within a few days, so everything will be back to normal.

Update: it's not going to be today For some reason rsync takes 1 full CPU core at 6 1.8 MB/s. When I Google the problem, I'm not alone, but I don't know how to fix it. So it just takes a bit longer.

[cygan]

✂️
Update: it's not going to be today For some reason rsync takes 1 full CPU core at 6 1.8 MB/s. When I Google the problem, I'm not alone, but I don't know how to fix it. So it just takes a bit longer.

thanks for always keeping us up to date here!
i had been using your Electrum server up until this downtime and now i've switched over to DireWolfM14's server for the time being while yours is down - i hope you'll forgive me

[LoyceV]

thanks for always keeping us up to date here!

It's back:

162.250.188.205:50002:s

I'm still waiting for the IP-move so all DNS records go back to normal.

i had been using your Electrum server up until this downtime and now i've switched over to DireWolfM14's server for the time being while yours is down - i hope you'll forgive me

I'll forgive you, if you forgive me for the downtime

[DireWolfM14]

I still hope to keep my old IP-address, but until that's sorted out, here's my new temporary server location:
162.250.188.205:50002:s

Still using Servarica, or did you find a more suitable hosting service?