4. Run Electrum server and configure your Electrum only connect to your Electrum server
I don't know how would this help, sorry. ElectrumX needs Bitcoind and the whole blockchain.
You must to be running a non-pruning bitcoin daemon
OP, if you change frequently the wallet file, I think that you need either a non-pruning Bitcoin wallet, either an Electrum (client) with default settings (if that's OK for you on privacy).
Or maybe you rethink what you do there and find a way to not change the wallet files. OR, for example if you have a (very) small number of them, you keep switching maybe you trick it with symlinks to point to the correct data folder depending on the wallet you use (of course, you'll still download the whole blockchain once for each wallet).
But I think that Electrum could be just fine.