ElectrumX server load?

[electrumx_throwaway3]

Hey,

I have been running a public ElectrumX server for over a year. However the load on it has been increasing quite a lot lately. Pics attached:

https://imgur.com/a/StOQcCu

It's running virtualized on a Ryzen CPU, on NVMe storage as well. I am not sure if there is really a huge increase on the network load or if I am just being DOS'd. (I would have to play with FW rules and decrease server soft/hard cost limits parameters).

Is anyone else seeing the same on their server?

Thanks

[DireWolfM14]

~

~

That seems like a lot of hits, but was your server running the whole year without a restart?  

Anyway, I don't think electrumx_rpc getinfo is the best way to see what load your server is under currently.  That list will just catalogue the pings and requests, but doesn't really tell you if your server is working hard at the moment.  To actually see what processes are currently running and how much of your resources they're using run the command htop.

Just to put things into perspective, I just restarted my server, then connected to it from one my wallets that received a transaction and got this:

Code:

    },
    "pid": 736,
    "request counts": {
        "blockchain.block.header": 1,
        "blockchain.block.headers": 1,
        "blockchain.estimatefee": 4,

        "blockchain.headers.subscribe": 40,

        "blockchain.relayfee": 1,
        "blockchain.scripthash.get_history": 1,
        "blockchain.scripthash.subscribe": 3,
        "blockchain.transaction.get": 1,
        "blockchain.transaction.get_merkle": 1,
        "getinfo": 2,
        "mempool.get_fee_histogram": 1,
        "server.banner": 1,
        "server.donation_address": 1,
        "server.features": 39,
        "server.peers.subscribe": 40,
        "server.version": 40
    },
    "request total": 177,
    "sessions": {
        "count": 2,
        "count with subs": 1,
        "errors": 0,
        "logged": 0,
        "pending requests": 1,
        "subs": 3
    },

[electrumx_throwaway3]

~
https://i.imgur.com/xMVm8c2.png
~

That seems like a lot of hits, but was your server running the whole year without a restart?  

Anyway, I don't think electrumx_rpc getinfo is the best way to see what load your server is under currently.  That list will just catalogue the pings and requests, but doesn't really tell you if your server is working hard at the moment.  To actually see what processes are currently running and how much of your resources they're using run the command htop.

Just to put things into perspective, I just restarted my server, then connected to it from one my wallets that received a transaction and got this:

Code:

    },
    "pid": 736,
    "request counts": {
        "blockchain.block.header": 1,
        "blockchain.block.headers": 1,
        "blockchain.estimatefee": 4,

        "blockchain.headers.subscribe": 40,

        "blockchain.relayfee": 1,
        "blockchain.scripthash.get_history": 1,
        "blockchain.scripthash.subscribe": 3,
        "blockchain.transaction.get": 1,
        "blockchain.transaction.get_merkle": 1,
        "getinfo": 2,
        "mempool.get_fee_histogram": 1,
        "server.banner": 1,
        "server.donation_address": 1,
        "server.features": 39,
        "server.peers.subscribe": 40,
        "server.version": 40
    },
    "request total": 177,
    "sessions": {
        "count": 2,
        "count with subs": 1,
        "errors": 0,
        "logged": 0,
        "pending requests": 1,
        "subs": 3
    },

I think I have finally kinda figured it out.

I am getting around 300 connections per second from random IPs around the world. They just connect and disconnect all over, it is a huge number of different source IPs as I have done some per-IP connection rate limiting with not much success. If I just general rate-limit connections to my server then it will become kinda useless to the public. It does appear to be a botnet.

Anything similar happening to you?

[DireWolfM14]

I am getting around 300 connections per second from random IPs around the world. They just connect and disconnect all over, it is a huge number of different source IPs as I have done some per-IP connection rate limiting with not much success. If I just general rate-limit connections to my server then it will become kinda useless to the public. It does appear to be a botnet.

Anything similar happening to you?

Damn, what a shitshow.  Sorry you're going through that.  I have not experienced anything like that.  My server is open to the general public, but I'm not advertising it in any way.  Just out curiosity, did you list it on the Bitcoin All Seeing Eye site?  I thought about listing mine there, but decided against it.

[mocacinno]

It isn't the first time a botnet gets used to DDos electrum nodes... Many years ago, i setup an electrs electrum node with nginx as a reverse proxy in front of it using a whitelist... That way i could whitelist bitcointalk members so they could use my node. It was the same situation: most (if not all) public electrum nodes got DDos'ed for several weeks without pause.

Maybe it's time to start that project again

[DaveF]

Just looked in and one of my nodes is getting hit.
Which is odd, because it would have to have been found at random and was on a non-standard port so I have no idea how they found it.
Never advertised it at all. It is on an IP that is also running one of my lightning nodes so someone might have just walked the ports till they fond something.

Just changed it to a new port, will see if it comes back.

-Dave

[mocacinno]

Just looked in and one of my nodes is getting hit.
Which is odd, because it would have to have been found at random and was on a non-standard port so I have no idea how they found it.
Never advertised it at all. It is on an IP that is also running one of my lightning nodes so someone might have just walked the ports till they fond something.

Just changed it to a new port, will see if it comes back.

-Dave

If it's just for personal use, and the DDos is really bothering you, you could always do the same setup i did with electrs: let electrumX listen on localhost only, then use nginx as a reverse proxy (let it listen on your external ip and reverse proxy to port 50002 on 127.0.0.1) and use a whitelist for nginx so you can manually whitelist your own ip(s).
Not very user-friendly, but it does get the job done...

I have the nginx config file laying around somewhere in my backups in case you're interested...

[DireWolfM14]

Just looked in and one of my nodes is getting hit.
Which is odd, because it would have to have been found at random and was on a non-standard port so I have no idea how they found it.
Never advertised it at all. It is on an IP that is also running one of my lightning nodes so someone might have just walked the ports till they fond something.

Just changed it to a new port, will see if it comes back.

-Dave

I was home all day yesterday and checked my node a few times and didn't see anything weird, but this is scaring me.  I'm at the office today but I'll take a look once I get home.  I don't think I'm getting hit (at least I hope not,) I just opened a wallet and it connected to my server without issue.  I only have one node open to the world, and it's listening on standard (default) ports.

If it's just for personal use, and the DDos is really bothering you, you could always do the same setup i did with electrs: let electrumX listen on localhost only, then use nginx as a reverse proxy (let it listen on your external ip and reverse proxy to port 50002 on 127.0.0.1) and use a whitelist for nginx so you can manually whitelist your own ip(s).
Not very user-friendly, but it does get the job done...

I have the nginx config file laying around somewhere in my backups in case you're interested...

Sounds like a pretty complex set up, or as I like to look at it: a learning opportunity.    
I'd love to see that config file, if you don't mind sharing.  I have Nginx installed for use with Mempool, but I don't have a lot of experience with it.  Will it work for both services?

[mocacinno]

It isn't that hard tbh...

Here's my config:

Code:

load_module /usr/lib/nginx/modules/ngx_stream_module.so;
user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/electrum_error.log;
events {
    worker_connections  1024;
}

stream {
       upstream electrs {
                server 127.0.0.1:50001;
        }
                log_format upstream_time '[$time_local] [$connection] $remote_addr:$remote_port => $server_addr:$server_port bytes in/out $bytes_received/$bytes_sent [$status]';
        server {

                access_log /var/log/nginx/electrum_access.log upstream_time;
                listen 50002 ssl;
                proxy_pass electrs;
                ssl_certificate /etc/letsencrypt/live/electrum.mocacinno.com/fullchain.pem;
                ssl_certificate_key /etc/letsencrypt/live/electrum.mocacinno.com/privkey.pem;
                ssl_session_cache shared:SSL:1m;
                ssl_session_timeout 4h;
                ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
                ssl_prefer_server_ciphers on;
                include blockips.conf;
        }
}

Thenin the 3rd line from the bottom (include blockips.conf), there's a reference to my whitelist... Whitelists are pretty easy aswell:

Code:

allow <i like this ip>;
allow <this one aswell>;
allow <ipv4 or ipv6 should work>;
deny all;

in my setup, i had electrs listening on 127.0.0.1:50001,
Nginx was listening on myexternalip:50002, using a letsencrypt x3 certificate, and nginx was reverse proxy'ing to 127.0.0.1:50001

Worked like a charm... The only downside was that i had to maintain the whitelist, manually adding each ip proceeded with "allow", "deny all" has to be at the complete last line of the whitelist.

[electrumx_throwaway3]

It isn't the first time a botnet gets used to DDos electrum nodes... Many years ago, i setup an electrs electrum node with nginx as a reverse proxy in front of it using a whitelist... That way i could whitelist bitcointalk members so they could use my node. It was the same situation: most (if not all) public electrum nodes got DDos'ed for several weeks without pause.

Maybe it's time to start that project again

This might be a good idea.

Do you know if these bad guys are also attacking via IPv6 (I do not have it enabled so far)?. I could deploy another ElectrumX to serve exclusively via IPv6.

FYI my server is listed in the all seeing eye. for me it is a way to be thankful and give back/support the network.

Thanks!

[DaveF]

...
Worked like a charm... The only downside was that i had to maintain the whitelist, manually adding each ip proceeded with "allow", "deny all" has to be at the complete last line of the whitelist.

The problem is I use it when I am on the phone so I never know what IP I would be connecting from.
The home PC is connecting on a 192.168.1.x address.
Since I changed the port to another one it has stopped. At a guess something walked the IP stack and then walked the ports once it found an IP that responded.

-Dave

[DireWolfM14]

~

Thanks for sharing that, it actually looks like a good solution.  It reminds me of an idea I had at one point, but I'm not sure if it'll work because I don't have any web dev experience; could one set up a reverse proxy on a VPS that points to a server that's hosted locally?

For example, I have a server running on my local home network, and I want to allow global access but don't want to expose my local IP.  Could I install Nginx on an AWS instance with an elastic IP that is advertised to the world, but have that server point all requests to my local IP?

[electrumx_throwaway3]

~

Thanks for sharing that, it actually looks like a good solution.  It reminds me of an idea I had at one point, but I'm not sure if it'll work because I don't have any web dev experience; could one set up a reverse proxy on a VPS that points to a server that's hosted locally?

For example, I have a server running on my local home network, and I want to allow global access but don't want to expose my local IP.  Could I install Nginx on an AWS instance with an elastic IP that is advertised to the world, but have that server point all requests to my local IP?

Hey,

If you are comfortable with networking you could do something similar with a VPS, you set up pfsense/opnsense/vyos or equivalent in your VPS and create a tunnel to your home, then you go from there.

Also I see several ppl discussing some kind of reverse proxy setup to restrict access to your server, when it is way easier to just set up some firewall rules provided you actually have a firewall in between server and internet.

[DireWolfM14]

~

Thanks for sharing that, it actually looks like a good solution.  It reminds me of an idea I had at one point, but I'm not sure if it'll work because I don't have any web dev experience; could one set up a reverse proxy on a VPS that points to a server that's hosted locally?

For example, I have a server running on my local home network, and I want to allow global access but don't want to expose my local IP.  Could I install Nginx on an AWS instance with an elastic IP that is advertised to the world, but have that server point all requests to my local IP?

Hey,

If you are comfortable with networking you could do something similar with a VPS, you set up pfsense/opnsense/vyos or equivalent in your VPS and create a tunnel to your home, then you go from there.

Also I see several ppl discussing some kind of reverse proxy setup to restrict access to your server, when it is way easier to just set up some firewall rules provided you actually have a firewall in between server and internet.

That's been my approach, so far: Firewall enabled on all my devices; port forwarding from my modem to my router; from my router to my server, which also has a firewall (ufw) enabled.  Currently I don't have any IPs restricted, the node ports are open to the world (except 22, of course.)  I suppose that if I wanted to whitelist specific IPs I could do so on my modem as well.

[spirit-receiver]

I've been having the same problem for about a week now, intermittently. ElectrumX has been running on my Raspberry Pi at home for several years, but now it is even bringing my internet connection to a halt.

[electrumx_throwaway3]

Is there some memory leak on ElectrumX?

My ElectrumX process is now eating up 6.5GB of ram and increasing. Always was below 3GB before.

Possibly it's the server tracking the cost for the huge bunch of IPs?

[mocacinno]

I guess it's possible that the number of connections has a cost in terms of memory consumption... Maybe that's something to ask on the dev's github repo, since he has the most intimate knowledge on how electrumX functions?

This being said, a tad bit offtopic, since a couple of weeks, i see increased hacking activity's on a lot of my bitcoin related websites... My main one is behind cloudflare (i'm not a cloudflare fan, but it's just a site with basic info and tips and tricks, nothing sensitive gets sent over, so the tradeoff is getting DDos'ed myself or using cloudflare), but my other one's do get hit pretty regular. No idear if there's a general anti-bitcoin sentiment and people using botnets and hacking skills against several services, or if these 2 things are isolated.

[spirit-receiver]

Just out curiosity, did you list it on the Bitcoin All Seeing Eye site?  I thought about listing mine there, but decided against it.

I found mine on that list even though I didn't sign up.

[mocacinno]

Just out curiosity, did you list it on the Bitcoin All Seeing Eye site?  I thought about listing mine there, but decided against it.

I found mine on that list even though I didn't sign up.

I had mine up there many, many years ago... IIRC, there was no real "verification", so anybody could have added your server to the list... As long as it was online, open and more or less synced to a relatively recent blockheight, it would be added to the list.

[spirit-receiver]

Yes, and I'm fine with that. It's supposed to be a public server, after all.