Obelisk: An Electrum server using libbitcoin

[Parazyd]

I've been working on an Electrum server implementation that uses ZeroMQ and libbitcoin as its backend so that I can use Electrum with my libbitcoin node/server.
It is written with using libbitcoin v4 in mind, so with public servers, only testnet is available because v4 is not yet finished, or if you have your own mainnet v4 server, then you can use mainnet.

The code can be found here: https://github.com/parazyd/obelisk

In total, it's about 1000 lines of Python code, which is relatively straightforward. This means it's 10 times smaller than ElectrumX for example.

If you're interested, I'd appreciate some feedback, help, and code review  

Happy hacking!

[1714468259]

1714468259

[1714468259]

1714468259

[1714468259]

1714468259

[ABCbits]

Since i never use libbitcoin, i'll just give basic feedback,
1. Have you compare the performance with ElectrumX or other Electrum server implementation?
2. AFAIK Electrum have it's own protocol and updated few times, so which Electrum (client wallet) version supported by Obelisk?
3. Is it right to assume Obelisk only support Linux?

[Parazyd]

Since i never use libbitcoin, i'll just give basic feedback,
1. Have you compare the performance with ElectrumX or other Electrum server implementation?
2. AFAIK Electrum have it's own protocol and updated few times, so which Electrum (client wallet) version supported by Obelisk?
3. Is it right to assume Obelisk only support Linux?

1.
I haven't yet. This is something I would appreciate help with, ideally from someone who is experienced in this kind of profiling.

2.
Electrum uses the ElectrumX server protocol as found in https://electrumx-spesmilo.readthedocs.io/en/latest/protocol-methods.html
I followed and implemented the spec.

3.
I develop and use it on Linux, but there should be no blockers to running it on any system that can use Python 3.

[NotATether]

Well it's still a work in progress since the methods for peers and drawing a histogram are stuns.

Histograms can be drawn with matplotlib but I don't think libbitcoin can help with making a list of peers or inserting yourself into other people's peer lists. I wonder how many more lines of Python will that electrum-specific functionality take plus the existing 1000.

[Parazyd]

Well it's still a work in progress since the methods for peers and drawing a histogram are stuns.

Histograms can be drawn with matplotlib but I don't think libbitcoin can help with making a list of peers or inserting yourself into other people's peer lists. I wonder how many more lines of Python will that electrum-specific functionality take plus the existing 1000.

The peer stuff is trivial, since it's just about telling other servers about itself, and other servers can be retrieved through DNS seeds.
I didn't implement this yet because I don't really need it, and because I want the code reviewed first.

The mempool fee histogram stuff is deprecated since protocol 1.4.2, so I'll just drop that. The 1.5 protocol is going to have some new additions which I will have to follow up after the spec is written.

[Dabs]

https://github.com/cculianu/Fulcrum/releases

This one is Fulcrum. Have you tried that? There was an update just today I think, version 1.5.2. How does your server compare with that one? It also runs on Windows.

[Parazyd]

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another

[Dabs]

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another

Oh, so how does your Electrum server work then? Where does it get it's blockchain data for the addresses and coins and all those other stuff?

*edit* oh, libbitcoin = is another full node, like Bitcoin Core but different?

[Parazyd]

That one requires Bitcoin Core. All servers like this that use Bitcoin core are inefficient in one way or another

Oh, so how does your Electrum server work then? Where does it get it's blockchain data for the addresses and coins and all those other stuff?

*edit* oh, libbitcoin = is another full node, like Bitcoin Core but different?

Yes, libbitcoin-server is a full node and a query server.

The existing implementations do their separate indexing on top of Core which is extremely inefficient.
libbitcoin is the fastest implementation of Bitcoin and it already indexes everything that is necessary.

You can use your own libbitcoin server or public instances, it's up to the user to choose.

[Parazyd]

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

[NotATether]

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Do you think it's doable? Sure it would be nice if we could just fire up an Electrum server with a --server switch, but the libbitcoin dependency could be an obstacle to merging. You have to go to the package manager to install deps like Qt5 and libsecp256k1 so the logical thing to do is make sure that it's in distribution's official repositories, and not some PPA or community maintained repo.

[Parazyd]

The eventual next step would be to merge Obelisk directly into Electrum and avoid the server boilerplate.

Do you think it's doable? Sure it would be nice if we could just fire up an Electrum server with a --server switch, but the libbitcoin dependency could be an obstacle to merging. You have to go to the package manager to install deps like Qt5 and libsecp256k1 so the logical thing to do is make sure that it's in distribution's official repositories, and not some PPA or community maintained repo.

libbitcoin is not a direct dependency. Obelisk simply uses zeromq to talk to a libbitcoin server and queries for information.

It is very much doable to have it merged into Electrum, but it will mostly depend on the maintainers and the community to choose if they want to move forward this way or not.

[Parazyd]

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

[ABCbits]

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Basically Electrum 3.3.0 and newer are supported, at least according to these
https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4
https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

That's great, i'll try it if i can setup libbitcoin-server without any major problem.

I'm still reading how to setup libbitcoin-server and found out how to make it connect only to my Bitcoin Core client with peer = localhost:8333 on bs.cfg. However, i can't find out how to configure RAM usage and whether HDD will cause bottleneck or not.

[Parazyd]

Yeah, 1.4 up to 1.4.2., but still a few things need to be finished which I need help with, or just more spare time.

Basically Electrum 3.3.0 and newer are supported, at least according to these
https://github.com/spesmilo/electrum/blob/3.3.0/electrum/version.py#L4
https://github.com/spesmilo/electrum/blob/3.2.4/electrum/version.py#L4

Will do @ setup.py
I've been planning to make such a thing so Obelisk can even be installed with pip.

That's great, i'll try it if i can setup libbitcoin-server without any major problem.

I'm still reading how to setup libbitcoin-server and found out how to make it connect only to my Bitcoin Core client with peer = localhost:8333 on bs.cfg. However, i can't find out how to configure RAM usage and whether HDD will cause bottleneck or not.

Thanks

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.
I'm not sure if RAM usage is configurable, but you shouldn't go OOM or something.

I'll try to get some setup.py ready soon. Done.

[ABCbits]

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.

Yeah, the README.md isn't complete and didn't mention that libzmq is required for compiling. libzmq repository shows there's .deb file, but there are dependency problem when i attempt to install it with apt and dpkg. I would rather not risking broke my system right now.

[Parazyd]

Keep in mind that libbitcoin v4 is still not fully finished, but Eric and the community are working on it.

Yeah, the README.md isn't complete and didn't mention that libzmq is required for compiling. libzmq repository shows there's .deb file, but there are dependency problem when i attempt to install it with apt and dpkg. I would rather not risking broke my system right now.

In apt-based distros there is python3-zmq which should do the trick.

EDIT: I added some more notes in the Readme. Thanks for the feedback.

[Parazyd]

Ah, for libbitcoin-server, it's best if you run the install.sh script.

Something like:

Code:

git clone https://github.com/libbitcoin/libbitcoin-server
cd libbitcoin-server
./install.sh --build-boost --build-zmq --disable-shared --prefix=$HOME/.local

These install scripts are made for easy deployment and a self-contained build process.

By the way, if you're interested, also check out libbitcoin-explorer, which is complementary to libbitcoin-server and can be used for querying.

[Parazyd]

Ah, for libbitcoin-server, it's best if you run the install.sh script.

Something like:

Code:

git clone https://github.com/libbitcoin/libbitcoin-server
cd libbitcoin-server
./install.sh --build-boost --build-zmq --disable-shared --prefix=$HOME/.local

These install scripts are made for easy deployment and a self-contained build process.

Actually i've used the install.sh and follow the guide step by step. I also tried Autotools, but there's error when i run ./configure which says libbitcoin-node is missing. But since you mention different parameter, i'll try it now.

Edit: Although there are many warning (such as "extra ;"), there's no single error. However, i can't find the executable bs even though i check $HOME/.local is on $PATH variable. Using find command, i also can't find the executable on $HOME/.local.

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

If it's not a hassle for you, I can help you out if there's a way for you to share the install.sh build output somehow.

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

[ABCbits]

Strange. if this prefix is correct, then the binary should be in $HOME/.local/bin/bs, and the config file should be in $HOME/.local/etc/libbitcoin/bs.cfg
You can run it with the full path, after you configure bs.cfg. If they're not there, then something did error (or you perhaps ran install.sh with another user's privileges so it ended up in a different $HOME, like /root/.local).

I'm sure i didn't use different user or sudo to run the script. However, i solved the problem simply by replacing $HOME/.local with absolute path (on new directory solely for libbitcoin). I can run ./bs -v without any problem.

Code:

Version Information:

libbitcoin-server:     4.0.0
libbitcoin-protocol:   4.0.0
libbitcoin-node:       4.0.0
libbitcoin-blockchain: 4.0.0
libbitcoin:            4.0.0

Now i just need to configure libbitcoin-server and perform initial sync. However, i just notice i'll have 2 copy of Bitcoin blockchain with different format (owned by Bitcoin Core and libbitcoin), so i'll free my HDD for libbitcoin's blockchain. There shouldn't be any more problem and i assume libbitcoin will take few days to sync since i use HDD before i can test Obelisk.

For what it's worth, the autotools is generally the tool that is ran in a standalone context. The install.sh script also downloads all other necessary dependencies, like libbitcoin-system, and libbitcoin-node and does all that for you transparently.

Yeah, i noticed that.