Electrum server on Windows (or VirtualBox, or, even better, WSL)

[NeuroticFish]

Later updates:
* I managed to make Electrum Personal Server work on Windows (see here and here)
* I compiled/installed electrs under WSL as HCP advised and it works on Debian!
* I compiled/installet BTC RPC Explorer under the same WSL as ETFbitcoin advised and it also works.

I am moving here a discussion which (mea culpa) went off topic somewhere else.

The relevant bits:

And since o_e_l_e_o came with very valid points for having own server and block explorer, I tried my luck and... it's not easy. I didn't manage yet to make any of those work on Windows (heh) and I am a bit undecided about trying out fulcrum, since I didn't find much of discussion about it here on bitcointalk.

I've also looked about using Bitcoin core with HW and.. hmm.. it looks unfinished and far from straightforward, so I pass that.

Any ideas?

The easiest way is to use virtualbox: https://www.virtualbox.org/
And then deploy the virtalbox version of mynode: https://www.mynodebtc.com/download

The upsides: it's easy, you don't have to install or learn anything, and there is a lot of support out there.
The downsides: You have to trust the people involved.
But, you do get a BTC node, a lightning node, an electrum server and a basic block explorer.

Keep in mind you are going to need enough free RAM & disk space on your PC to run it. And you are going to have to keep your machine running 24/7 to get access to it.

-Dave

I've tried that too yesterday, and didn't work out because the disk for the blockchain is connected though USB and VBox/myNode didn't like that.

In VBox my last attempt was with a proper Debian. I managed to get Bitcoin Core run and use that USB drive for the data, but I completely failed in making it start with the system. I suck at Linux and I know it 

And on Windows, bitcoind is fine, but the Electrum Personal Server just doesn't do anything. Whatever I do it just tells "press any key to continue" and Electrum doesn't see it.

Getting a bit OT, but you moved the

Code:

mynode_vm-disk002.vdi

file to your USB dive and then told virtual box where it was and it still gave you problems?



Remember that vdi has more then just the blockchain on it. So, it will get much larger then just the blockchain.
If you really want to keep the data external, you might want to just get another external drive and put all of the mynode files on there. This way you can pick it up and move it to another PC with no real effort.

-Dave

I'll continue from here.

[1713551330]

1713551330

[1713551330]

1713551330

[NeuroticFish]

No, I didn't move the vdi disk file, don't want to mix with that. I only want bitcoin core use that specific data dir.
The point is that since bitcoind worked well on Windows, I have already the data pretty much up to date on that HDD and I would rather not spend ages for a resync (I also don't have space for moving the data too much around).
Even more, that USB disk is only 500GB, which is fine for the blockchain, but not enough for the rest, hence that vdi should stay where VBox created it, where I do have other ~250GB free.


I tried to mount the USB drive directly into Virtual Box and it didn't work out.
If I remember correctly, the guest additions made me troubles. But, you know what? I'll give it a try a bit later and I'll come back with this.

Update:
I tried to insert Guest Additions CD, I cannot find it anywhere to run the additions.
I tried to use VBox Shared Folders feature (which works for me on the clean Debian!) and I don't find anywhere the corresponding folder.

So the directions I have:
1. [preferred] Make an Electrum server and a block explorer work on Windows
2. Anything I can install on Virtual Box as long as Bitcoin DataDir remains on the USB attached HDD. This is a special problem, since I am badly inexperienced with Linux.
2.1. I do have a Debian configured on a Virtual Box, with DataDir just fine, but I don't know how to make bitcoind start automatically with the system (and with that datadir); I didn't go with other steps yet there.

[DaveF]

Take a look here: https://www.virtualbox.org/wiki/Downloads

What filesystem is the external drive formatted with? If it's NTFS mynode may not have installed that driver by default.

Also keep in mind, in general I don't know if it's going to work at all with mynode.
Getting all the other stuff to run with the data directory someplace else should be no issue.

BUT, since it was setup as an all in one, plug and play appliance I don't know if they made any assumptions about what files are where for other integrated applications.

They do have the standard telegram / discord / other support that you can ask them directly about.
Takes some time to get a reply now and then, but they always have replied.

-Dave

[NeuroticFish]

Take a look here: https://www.virtualbox.org/wiki/Downloads

What filesystem is the external drive formatted with? If it's NTFS mynode may not have installed that driver by default.

Also keep in mind, in general I don't know if it's going to work at all with mynode.
Getting all the other stuff to run with the data directory someplace else should be no issue.

BUT, since it was setup as an all in one, plug and play appliance I don't know if they made any assumptions about what files are where for other integrated applications.

They do have the standard telegram / discord / other support that you can ask them directly about.
Takes some time to get a reply now and then, but they always have replied.

-Dave

You made me hope for a minute  
I indeed installed "VirtualBox Extension Pack", but it didn't help.

The drive is indeed NTFS, but, I think that the problem is not that because:

1. Also the guest additions CD is not seen.
2. The same NTFS drive is seen (as Shared Folder) in the Debian VM, and myNode is also Debian.

I think that for the shared folder I need the guest additions installed; since that's not seen, I'm stuck.


When I get some more time again, I'll give another shot to EPS on Windows. I think that I'm missing something very simple there. It would also make more sense than an extra VM.
myNode looks overly restrictive for my needs and I feel like even if that problem gets answered for me, I'll run very soon into another problem. But I'll keep it as a backup option if I have to get back to a VM.
Thank you.

[Dabs]

I'm going to try fulcrum one day. It says on the github that "it should just work." and I discovered it because I keep looking at my electrum wallet console all the time and watch what server it connects to.

[NeuroticFish]

I'm going to try fulcrum one day. It says on the github that "it should just work." and I discovered it because I keep looking at my electrum wallet console all the time and watch what server it connects to.

The thing is that form my search, here on Bitcointalk English areas, it was only you mentioning it. No offense, but I'd wait for a bit more, since "it should just work" doesn't imply that it would also do exactly what we'd expect. Maybe I'm paranoid, still, it would mean I'm running new unknown software with not much of trusted reviews.
I've seen a bit of info on Reddit, but mostly in BCash areas, which I skipped super fast.

[Dabs]

I had asked about it before on these forums, but even the regular Electrum server there isn't much other than the docs on it's website. But ... source code is out there if you know how to read and audit it, can also compile it yourself. The whole idea of running anything in it's own Virtual Machine should probably be the default for most users. At least it's a full node server that doesn't do anything else it's not supposed to do (or you can monitor it for that.)

[DireWolfM14]

So the directions I have:
1. [preferred] Make an Electrum server and a block explorer work on Windows
2. Anything I can install on Virtual Box as long as Bitcoin DataDir remains on the USB attached HDD. This is a special problem, since I am badly inexperienced with Linux.
2.1. I do have a Debian configured on a Virtual Box, with DataDir just fine, but I don't know how to make bitcoind start automatically with the system (and with that datadir); I didn't go with other steps yet there.

In the ElectumX documentation, on the "HOWTO" page there's a claim that ElectrumX should be able to run on Windows.

ElectrumX should run on any flavour of unix. I have run it successfully on MacOS and DragonFlyBSD. It won’t run out-of-the-box on Windows, but the changes required to make it do so should be small - pull requests are welcome.

I've never tried to run it on Windows, but it should just be a matter of getting Python3 installed, then using PIP to install the remaining dependencies.  You would then need to set up the proper service command to create a service that starts automatically when the machine boots up or the user logs in.  I'll try it one of these days, and report back.

As for bitcoind running in the background as a service in Linux, the instructions to use systemd are available in the bitcoind.service sample file found in the master repository.

Note:  If you're not running around as root, you'll need to use sudo before each of the following commands.

This code will create the systemd configuration file:

Code:

systemctl edit bitcoind.service

Copy the commands you want into the file and then save it.

Enabling the service will cause it to start at boot, to do so use this command:

Code:

systemctl enable bitcoind.service

To start the service manually use this command:

Code:

systemctl start bitcoind.service

[NeuroticFish]

but it should just be a matter of getting Python3 installed, then using PIP to install the remaining dependencies.

First of all, thanks for the help.
Now... I've just tried this ElectrumX for Windows path. Installed Python (for the 1st time), used pip for aiohttp, then failed at pylru:

Code:

py -m pip install pylru
Defaulting to user installation because normal site-packages is not writeable
Collecting pylru
  Downloading pylru-1.2.0.tar.gz (18 kB)
    ERROR: Error [WinError 87] The parameter is incorrect while executing command python setup.py egg_info
ERROR: Could not install packages due to an OSError: [WinError 87] The parameter is incorrect

[NeuroticFish]

V i c t o r y ! (sort of)
Later note: I managed to install Electrs too. I will provide steps at a later point. Electrs allows as many wallets as you want and block explorer too. I find that a better approach.

With some knowledge earned from last night's tests with Sparrow wallet and from reading a tutorial for RasPi which I found better than the one everybody reads it, I managed to make Electrum Personal Server work on my Windows.

Highlights

bitcoin.conf must have:

Code:

txindex=1
server=1

I have some more (probably unnecessary) things from the previous tests with various servers, but if there are problems, these can be added:

Code:

dbcache=2400
rpcbind=127.0.0.1
zmqpubhashblock=tcp://127.0.0.1:8433
rpcallow=127.0.0.1

In EPS config.ini I added only 2 lines, nothing else had to be changed (during my tests I had other settings there and just made the things worse).

Code:

myHWallet = zpub12345678901234567890 <- put your own zPub
datadir = x:\BitcoinDataDir

One thing (or two) that was always missing in my tests and may be just the part I've missed in the official tutorial (thank you Sparrow wallet) :

Code:

bitcoin-cli.exe -datadir=x:\BitcoinDataDir createwallet myHWallet

in subsequent runs it will be

Code:

bitcoin-cli.exe -datadir=x:\BitcoinDataDir loadwallet myHWallet

From here, according to the tutorial:

Code:

electrum-personal-server.exe config.ini
electrum-personal-server-rescan.bat config.ini

Now Electrum connects:

Code:

electrum --oneserver --server 127.0.0.1:50002:s

I have a new problem now.
For now I am testing with my current HW having quite a number of addresses and transactions.
But most of my addresses are displayed with red background, my transactions are shown as "This transaction is only available on your local machine", and the final sum of coins shown is 0.
Something is incorrect and I don't even know where (any ideas?).

Of course that I'd also prefer a server that works with more wallets. But it's a start.

If you're looking for simple setup, you should use ElectrumX which packaged on Docker (https://github.com/lukechilds/docker-electrumx). You just need to run one command to run ElectrumX server. But AFAIK installing Docker on Windows is a bit annoying.

Thank you. I will look into this too; hopefully tomorrow.

[DaveF]

If you're looking for simple setup, you should use ElectrumX which packaged on Docker (https://github.com/lukechilds/docker-electrumx). You just need to run one command to run ElectrumX server. But AFAIK installing Docker on Windows is a bit annoying.

Thank you. I will look into this too; hopefully tomorrow.

And for extra fun if you still have VirtualBox installed and running and since Docker uses Hyper-V....there are more steps that you have to go though at times.

You can google for more but here are a few links:

https://us.informatiweb.net/tutorials/it/virtualization/virtualbox-use-virtualbox-and-hyper-v-at-the-same-time.html
https://derekgusoff.wordpress.com/2012/09/05/run-hyper-v-and-virtualbox-on-the-same-machine/

Or just remove virtual box if you don't need it.

-Dave

[NeuroticFish]

I have a new problem now.
For now I am testing with my current HW having quite a number of addresses and transactions.
But most of my addresses are displayed with red background, my transactions are shown as "This transaction is only available on your local machine", and the final sum of coins shown is 0.

I've changed EPS config from

Code:

initial_import_count = 1000
gap_limit = 25

to

Code:

initial_import_count = 10000
gap_limit = 125

It may very well be overkill (the rescan was sooo slooow....), but my wallet looks good now.
Of course, next step is a server that works with more than one wallet. And I've read that mempool may need electrs...

Hyper-V

Thanks, I will have a read, although my VBox stays 95% of the time stopped (and I intend to keep it so). But I expect to be interesting.

[NeuroticFish]

And for extra fun if you still have VirtualBox installed and running and since Docker uses Hyper-V....there are more steps that you have to go though at times.

You can google for more but here are a few links:

https://us.informatiweb.net/tutorials/it/virtualization/virtualbox-use-virtualbox-and-hyper-v-at-the-same-time.html
https://derekgusoff.wordpress.com/2012/09/05/run-hyper-v-and-virtualbox-on-the-same-machine/

Or just remove virtual box if you don't need it.

-Dave

I've done my reading, it was indeed interesting and useful, thank you for that. It cleared up some of old things 

I was using VirtualBox - very rarely, but still - for many years. Since before Win10 that's for sure. And I remember I had errors related to Hyper-V in VirtualBox, including that black screen.
I don't know if it's because of those problems, I really don't remember, but my host Win10 doesn't have Hyper-V installed. For a while I was even installing only x86 systems (Windows?) on VBox because the x64 failed. And I blame Hyper-V 

So.. I don't know why I would keep enabled Hyper-V; I've been using my computer very well with it off. Maybe the fact my Intel CPU supports VT-x is of help; maybe this is the trick that kept me away of problems lately.

[HCP]

Any particular reason you didn't want to use "Windows Subsystem for Linux" (WSL)?

I was able to get both "electrs" running under WSL. electrs was reading Windows-based Bitcoin Core block data and I was able to connect to electrs from a Windows-based Electrum client. In my opinion it was certainly a lot easier than messing about with VirtualBox.

I vaguely recall I also managed to get ElectrumX working, but preferred electrs.


Was using VirtualBox, simply a way to keep everything segregated for "security" reasons?

[NeuroticFish]

Any particular reason you didn't want to use "Windows Subsystem for Linux" (WSL)?

Mostly because I'm a stupid windows-head and although I heard about it, I've never used WSL and never actually needed it, since VBox (and before that VMWare) always did all I needed (usually clean Windows for tests).
And it didn't occur to me at all that this could be the way. Thank you!

I was able to get both "electrs" running under WSL. electrs was reading Windows-based Bitcoin Core block data and I was able to connect to electrs from a Windows-based Electrum client. In my opinion it was certainly a lot easier than messing about with VirtualBox.

Since you have that, I'll try to abuse your kindness and try to replicate your setup  , basically if I run into odd newbish issues at least I have the chance somebody can answer 
electrs actually sounds very good to me, at least it solves the requirements for a block explorer which I also wanted (as a step 2).

So.. what Linux flavor you suggest I should install? Ubuntu?

[NotATether]

@NeuroticFish, why are you running bitcoind and ElectrumX inside a virtual machine (if I'm understanding the discussion correctly), when disk ops are notoriously slow on an emulated disk-as-file and disk controller?

Yes I know your datadir is on an external drive, but Virtualbox emulating the disk controller means that bitcoind will take a serious performance hit.

[also for some reason I have discovered bugs where VM network traffic is dropped, so beware of that as well].

[NeuroticFish]

@NeuroticFish, why are you running bitcoind and ElectrumX inside a virtual machine (if I'm understanding the discussion correctly), when disk ops are notoriously slow on an emulated disk-as-file and disk controller?

Yes I know your datadir is on an external drive, but Virtualbox emulating the disk controller means that bitcoind will take a serious performance hit.

[also for some reason I have discovered bugs where VM network traffic is dropped, so beware of that as well].

I was planning to not use emulated disk, at least (not) for bitcoin datadir. I was planning to use the shared disk functionality (or connect directly to the USB) and use the external USB disk I already have with a pretty much up to date datadir.

The VBox direction was chosen because I have to have Windows as main OS and I failed to install - at least at start - any reasonable Electrum server onto Windows. And on VBox I managed to have a Linux working with my USB disk. I've noticed that's not the best direction (and thanks for the heads up about the traffic!), but it was the best I could think of. After reading about Docker I am afraid to use it on this machine (I don't want to have to reinstall windows if I do something wrong).


Later on I managed to make Electrum Personal Server work on Windows, but that's still not enough, since as step 2 I intend to also get a block explorer.
I intend to try now @HCP direction with WSL; from what he said here, his setup looks very promising.

[NotATether]

Later on I managed to make Electrum Personal Server work on Windows, but that's still not enough, since as step 2 I intend to also get a block explorer.
I intend to try now @HCP direction with WSL; from what he said here, his setup looks very promising.

Alright, however you should know you can always compile ElectrumX and whatever block explorer you need using MSYS2 (https://msys2.org) windows terminal - which includes the MinGW build tools that let you compile Linux programs on Windows.

But forget about this if any of your tools are written in NodeJS because it starts to become very messy to compile (they don't provide a NodeJS interpreter in their package manager).

[HCP]

So.. what Linux flavor you suggest I should install? Ubuntu?

Ubuntu is as good as any other distro... plus it's basically available on the Windows App Store.

WSL is actually insanely easy to use... and it does nifty things like offer you /mnt/<windows drive letter>/ access, so you can easily access external drives and the like...

You might want to have a quick read of some of the basic stuff here: https://docs.microsoft.com/en-us/windows/wsl/

[NeuroticFish]

Many thanks HCP, this was the correct solution.

It wasn't straightforward at all, as a matter of fact I almost gave up at some point and I even unistalled Ubuntu after a couple of tries. But I've tried again, with Debian under WSL. I may have been making useless steps too, but I've finally got to the sweet:

Code:

[2021-10-16T13:31:39.392Z INFO  electrs::server] serving Electrum RPC on 127.0.0.1:50001

Some notes:

• For now already the Debian folder takes 4 GB on my C: drive although Bitcoin runs under Windows and electrs actual data dir (30 GB) is set to somewhere else.
• The errors under Ubuntu were from not finding librocksdb-dev 6.11.4-3 to not "knowing" how to create files and folders on a Windows NTFS drive/folder. Not nice.
• For now it works well, but it uses bitcoind .cookie instead of user/pass, and I want to also make it start automatically, maybe also not fail if bitcoind didn't finish syncing, and obviously also install a block explorer.

When all is done I may even make another post with the steps, since I expect to not be the only Windows user here  


Questions (need help) :
1. Is there any way to make electrs not fail (and instead retry later) if bitcoind is not available yet?
2. I could use some suggestions for the block explorer to install. Is it mempool.space or is there any better/easier one?