Bitcoin Forum

I've been working on an Electrum server implementation that uses ZeroMQ and libbitcoin as its backend so that I can use Electrum with my libbitcoin node/server.
It is written with using libbitcoin v4 in mind, so with public servers, only testnet is available because v4 is not yet finished, or if you have your own mainnet v4 server, then you can use mainnet.

The code can be found here: https://github.com/parazyd/obelisk

In total, it's about 1000 lines of Python code, which is relatively straightforward. This means it's 10 times smaller than ElectrumX for example.

If you're interested, I'd appreciate some feedback, help, and code review  :D

Happy hacking!

Since i never use libbitcoin, i'll just give basic feedback,
1. Have you compare the performance with ElectrumX or other Electrum server implementation?
2. AFAIK Electrum have it's own protocol and updated few times, so which Electrum (client wallet) version supported by Obelisk?
3. Is it right to assume Obelisk only support Linux?

1.
I haven't yet. This is something I would appreciate help with, ideally from someone who is experienced in this kind of profiling.

2.
Electrum uses the ElectrumX server protocol as found in https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html
I followed and implemented the spec.

3.
I develop and use it on Linux, but there should be no blockers to running it on any system that can use Python 3.

Well it's still a work in progress since the methods for peers and drawing a histogram are stuns.

Histograms can be drawn with matplotlib but I don't think libbitcoin can help with making a list of peers or inserting yourself into other people's peer lists. I wonder how many more lines of Python will that electrum-specific functionality take plus the existing 1000.

The peer stuff is trivial, since it's just about telling other servers about itself, and other servers can be retrieved through DNS seeds.
I didn't implement this yet because I don't really need it, and because I want the code reviewed first.

The mempool fee histogram stuff is deprecated since protocol 1.4.2, so I'll just drop that. The 1.5 protocol is going to have some new additions which I will have to follow up after the spec is written.

https://github.com/cculianu/Fulcrum/releases

This one is Fulcrum. Have you tried that? There was an update just today I think, version 1.5.2. How does your server compare with that one? It also runs on Windows.

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another ;)

Oh, so how does your Electrum server work then? Where does it get it's blockchain data for the addresses and coins and all those other stuff?

*edit* oh, libbitcoin = is another full node, like Bitcoin Core but different?

Yes, libbitcoin-server is a full node and a query server.

The existing implementations do their separate indexing on top of Core which is extremely inefficient.
libbitcoin is the fastest implementation of Bitcoin and it already indexes everything that is necessary.

You can use your own libbitcoin server or public instances, it's up to the user to choose.

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Do you think it's doable? Sure it would be nice if we could just fire up an Electrum server with a --server switch, but the libbitcoin dependency could be an obstacle to merging. You have to go to the package manager to install deps like Qt5 and libsecp256k1 so the logical thing to do is make sure that it's in distribution's official repositories, and not some PPA or community maintained repo.

libbitcoin is not a direct dependency. Obelisk simply uses zeromq to talk to a libbitcoin server and queries for information.

It is very much doable to have it merged into Electrum, but it will mostly depend on the maintainers and the community to choose if they want to move forward this way or not.

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

Basically Electrum 3.3.0 and newer are supported, at least according to these
https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4 (https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4)
https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4 (https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4)


That's great, i'll try it if i can setup libbitcoin-server without any major problem.

I'm still reading how to setup libbitcoin-server and found out how to make it connect only to my Bitcoin Core client with peer = localhost:8333 on bs.cfg. However, i can't find out how to configure RAM usage and whether HDD will cause bottleneck or not.

Thanks :)

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.
I'm not sure if RAM usage is configurable, but you shouldn't go OOM or something.

I'll try to get some setup.py ready soon. Done.

Yeah, the README.md isn't complete and didn't mention that libzmq is required for compiling. libzmq repository shows there's .deb file, but there are dependency problem when i attempt to install it with apt and dpkg. I would rather not risking broke my system right now.

In apt-based distros there is python3-zmq which should do the trick.

EDIT: I added some more notes in the Readme. Thanks for the feedback.

Ah, for libbitcoin-server, it's best if you run the install.sh script.

Something like:

These install scripts are made for easy deployment and a self-contained build process.

By the way, if you're interested, also check out libbitcoin-explorer, which is complementary to libbitcoin-server and can be used for querying.

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

If it's not a hassle for you, I can help you out if there's a way for you to share the install.sh build output somehow.

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

I'm sure i didn't use different user or sudo to run the script. However, i solved the problem simply by replacing $HOME/.local with absolute path (on new directory solely for libbitcoin). I can run ./bs -v without any problem.


Now i just need to configure libbitcoin-server and perform initial sync. However, i just notice i'll have 2 copy of Bitcoin blockchain with different format (owned by Bitcoin Core and libbitcoin), so i'll free my HDD for libbitcoin's blockchain. There shouldn't be any more problem and i assume libbitcoin will take few days to sync since i use HDD before i can test Obelisk.


Yeah, i noticed that.

Ok, glad you got it set up.
One thing I recommend is to first sync testnet, it's about 30 gigabytes only and play with that before going mainnet.

Good thing you mentioned it, i was thinking which file should i move to external drive. Coincidentally, i also run Bitcoin Core on testnet, so i can save some time and internet bandwidth.

Finished most of the code now, and almost all is covered by test units :)

libbitcoin-server v4 still needs some code written, regarding tx broadcasting, but hopefully this will be finished soon.

If you have an unstable power source or something, you can enable the "flush_writes" option in the config file. This will slow down the sync a bit, but it should avoid corrupting the databases. By default libbitcoin decides to keep most of the stuff in memory and just do periodic writes to disk.

But yeah, v4 is still in development so there are bugs for sure. However, Eric tells me that he's back on working on libbitcoin full time again so I expect it to be more stable sooner than later.

On my local testnet libbitcoin-server, I modified the configuration to the testnet values and didn't change anything else except "use_libconsensus=true" from the defaults. The comments in the config file tell you what needs to be changed to work on mainnet/testnet.