Electrum server discussion thread

[ThomasV]

It happens a lot that my server (electrum.0x0000.de) gets kicked from the list.

I'm thinking it might be because it takes quite a while for it to digest new blocks and it's marked as "lagging".

Is that possible?

No, a server does not get kicked because it is lagging.
It happens because your database is corrupted, and your server fails to produce the correct root hash.

database corruption may occur if you do not shutdown your server cleanly.

[1714508417]

1714508417

[1714508417]

1714508417

[1714508417]

1714508417

[molecular]

It happens a lot that my server (electrum.0x0000.de) gets kicked from the list.

I'm thinking it might be because it takes quite a while for it to digest new blocks and it's marked as "lagging".

Is that possible?

No, a server does not get kicked because it is lagging.
It happens because your database is corrupted, and your server fails to produce the correct root hash.

database corruption may occur if you do not shutdown your server cleanly.

Well, it's on and off the list without me doing anything.

If the root hash is wrong, is the db automatically repaired or something?

[JWU42]

If the root hash is wrong, is the db automatically repaired or something?

Nope, need to download a new version from the foundry (http://foundry.electrum.org/leveldb-dump/).  I wouldn't suggest building a new one as it will take over a month (I know, doing it now)

[molecular]

just pulled newest master from https://github.com/spesmilo/electrum-server.git

Not sure if It's new, but I'm seeing this in the log:

Code:

INFO:electrum:joining IRC
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/home/nick/electrum-server/src/ircthread.py", line 118, in run
    client = irc.client.Reactor()
AttributeError: 'module' object has no attribute 'Reactor'

my wild guess: my python-irclib is outdated? I'm on debian:

Code:

user@null:~/electrum-server$ apt-cache show python-irclib
Package: python-irclib
Version: 0.4.8-1

EDIT: problem now obsolete. I had to migrate to a different server (now ssd ;-) for harddrive failure reasons and the problem doesn't occur there. I also again have more than 10 session. I guess my irc stuff was borked all along.

[ozboy2014]

Hi, can you please take a look and assist if you can. Thanks.

https://bitcointalk.org/index.php?topic=1053864.msg11494471#msg11494471

[JWU42]

Hi, can you please take a look and assist if you can. Thanks.

https://bitcointalk.org/index.php?topic=1053864.msg11494471#msg11494471

Electrum for DOGE?!?

[Dogedarkdev]

Hi, can you please take a look and assist if you can. Thanks.

https://bitcointalk.org/index.php?topic=1053864.msg11494471#msg11494471

Electrum for DOGE?!?

DOGED*

the client is complete, we are just having some issues with the back end (electrum server)

https://bitcointalk.org/index.php?topic=1053864.msg11504413#msg11504413

[jadedjack]

Now if I try "sudo electrum-server getinfo" the electrum-server process blocks and nothing happens. The log does not show anything. Process cpu usage is at ~100%.

Any suggestions on how to debug this?

@fex,

The electrum-server DB has to catch-up.  It isn't a verbose process but seeing the CPU usage will assure you it is working.  The log will only update after passing every 1000 blocks.  Note it can take an hour or two to catch-up just 10-12 hours (on a modern CPU with an SSD).

Yes, I found this to be the case, but there was another thing.

I finally had some time to look into the problem: there were too many files open. I did change the limit for this in /etc/security/limits.conf as described on github, however, two things to note:

1. If you put an entry like "<bitcoinuser> hard nofile 65536" in limits.conf, it will only work if <bitcoinuser> is really logged in. It does not work (at least on Ubuntu 14.10) when another user account loggs in and then uses "su" to get into the <bitcoinuser> account. This might be a problem when using "electrum-server" binary as it seems to use "su". To solve this, I added limits.conf-entries for the "really logged in" user account that starts the "electrum-server" binary.

Maybe this information could be added to the readme on Github somehow.

2. The error indicating that too many files were open was burried under a stack trace that I got when I was running run_electrum_server directly (instead of "electrum-server start"). It did not show up in the log file.



Regards,

Felix

This page describes how to increase the open files limit for a user
https://github.com/spesmilo/electrum-server/blob/master/HOWTO.md#step-10-tweak-your-system-for-running-electrum

Code:

 sudo echo "bitcoin hard nofile 65536" >> /etc/security/limits.conf
 sudo echo "bitcoin soft nofile 65536" >> /etc/security/limits.conf

In Ubuntu 14.04 you must also edit /etc/pam.d/common-session

Code:

sudo nano  /etc/pam.d/common-session

Add this to the end of the file

Code:

session required pam_limits.so

[kamk]

It happened yesterday, from /var/log/electrum.log:

[04/06/2015-18:35:52] addr not in tree 1c0b8f470c0b7a32792c6cbacf3a9b00e72d93b1e516b1d2de130ab4a4e2be1dde3570be097e005 f0ebacca65b21aa859bfa4f1a01000000 13ZHmZ15qKNLbttUFZoTYDdwhqfpfjZGmP !7l<NP><NUL><NUL><NUL><NUL>s{<NUL>

The server then stopped catching up with bitcoind and I had to download snapshot from foundry. I checked syslog but found no errors reported regarding the disk.

[btcash]

Thomas could you please post your opinion on this issue:

http://www.reddit.com/r/Electrum/comments/38qi2m/20_mb_blocks_the_end_of_electrum/

[tl121]

Thomas could you please post your opinion on this issue:

http://www.reddit.com/r/Electrum/comments/38qi2m/20_mb_blocks_the_end_of_electrum/

It seems likely that only a small change will be needed to the server code to handle larger blocks.  However, there is a more serious problem related to the traffic load of the bitcoin network.  The Electrum server code is inefficient and with a slow processor can often fall behind the Bitcoin network.  Even with the present size blocks if a large block hits the network immediately followed by a second block then the server will be two blocks back and the client will report that the server is lagging.

I run a private Electrum server on an Atom (Haswell) based Intel NUC, with 8 GB RAM and an SSD. Bitcoind has no problem keeping up with the network, using only a few percent of the CPU load and this slow machine would have a chance of keeping up with 20MB blocks, given that it is running at about 3 percent CPU loading today.  However, the Electrum server uses about seven times as much CPU time per block as bitcoind, so there is zero chance it will handle a large expansion in network transaction rate. Actually, I consider this server code to be marginal with present network traffic, because I get occasional "server lagging" messages with my clients. Incidentally, one of the things that put me off from Electrum a few years ago were these "server lagging" messages, which I took to be an indication that something was wrong with the Electrum system.  People want to start their client and immediately do a transaction without any network delays or other BS. (In this regard even with the typical "lagging" delays, quick start up with Electrum is still better than Multibit if the client has been off-line for several days, at least with my network access speed.  If bitcoind is not kept online 24/7 then its start up performance is worse than hopeless.)

I assume that the Electrum server code could be tweaked or rewritten to make it run substantially faster.  I can think of no good reason why the Electrum server should use any more processor time than Bitcoin while keeping up with the network. Indeed, on my system the unavoidable use of processor time is probably doing signature checking by bitcoind. It don't believe there is anything that the Electrum Server needs to do that is remotely as CPU intensive.  If I had limited RAM or a spinning rust disk then perhaps there would be a hard reason why the Electrum server was slow.

I'm not complaining at present, but I think some work is needed and is possible to cover future increases in network traffic.

[ThomasV]

if you run an electrum server, it is time to upgrade!

Two fixes were added recently:
 - support for arbitrary p2sh scripts (will be required by electrum 2.4 client)
 - memory pool efficiency fix, added after the recent 'stress test' (servers without this patch are too slow)

note: 2.4 clients will only connect to servers that have upgraded

[tl121]

if you run an electrum server, it is time to upgrade!

Two fixes were added recently:
 - support for arbitrary p2sh scripts (will be required by electrum 2.4 client)
 - memory pool efficiency fix, added after the recent 'stress test' (servers without this patch are too slow)

note: 2.4 clients will only connect to servers that have upgraded

Thanks for the update.  It was quick and easy to install.

The server seems to run smoother between blocks.  However, when a new full block shows up the server still takes a long time to process the block, over two minutes for a full 1 MB block. In the past 12 hours electrum-server has consumed more than four times as much CPU time as bitcoin core. The server machine is completely CPU bound while processing a new block. The machine hardware is a Haswell Atom machine, running with 8 GB of RAM and SSD.

[BadAss.Sx]

Hi, how do i upgrade...is there a simple method?

Sorry for asking if it is too much asked

[JWU42]

Assumes you are starting in your home directory and the server is in ~/electrum-server

Code:

cd electrum-server
git pull
sudo python setup.py install

[BadAss.Sx]

I did that and all went fine without errors, but now i read this in my log and can't connect to my server:

[04/07/2015-21:38:48] Signal handler called with signal 15
[04/07/2015-21:38:48] Stopping Stratum
[04/07/2015-21:38:49] Closing database...
[04/07/2015-21:38:49] Database is closed
[04/07/2015-21:38:49] Electrum Server stopped
[04/07/2015-21:40:10] Starting Electrum server on electrum.badass.sx
[04/07/2015-21:40:17] Database version 3
[04/07/2015-21:40:17] Blockchain height 363830
[04/07/2015-21:40:17] UTXO tree root hash: d229909f0021c67165fd317a9bb519125996b68335b2aff3ef4651339d5174d7
[04/07/2015-21:40:17] Coins in database: 1434314010655316
[04/07/2015-21:40:17] TCP server started on 94.102.50.70:50001
[04/07/2015-21:40:17] SSL server started on 94.102.50.70:50002
[04/07/2015-21:40:17] bitcoind still warming up...
[04/07/2015-21:40:27] bitcoind still warming up...
[04/07/2015-21:40:37] bitcoind still warming up...
[04/07/2015-21:40:47] bitcoind still warming up...
[04/07/2015-21:40:57] bitcoind still warming up...
[04/07/2015-21:41:07] bitcoind still warming up...
[04/07/2015-21:41:17] bitcoind still warming up...
[04/07/2015-21:41:27] bitcoind still warming up...
[04/07/2015-21:41:37] bitcoind still warming up...
[04/07/2015-21:41:47] bitcoind still warming up...

Any idea what is going on? In top i don't see anything strange.

When i do bitcoin-cli getblockchaininfo

{
    "chain" : "main",
    "blocks" : 363834,
    "headers" : 363834,
    "bestblockhash" : "0000000000000000011cef43d1c689b097bd15410151f6b2666a6f683a7ff659",
    "difficulty" : 49402014931.22746277,
    "verificationprogress" : 0.99999607,
    "chainwork" : "00000000000000000000000000000000000000000008295a3c16c8d1fb7190dc"
}

But when i read the electrum-server log

[04/07/2015-21:57:55] Starting Electrum server on electrum.badass.sx
[04/07/2015-21:57:56] Database version 3
[04/07/2015-21:57:56] Blockchain height 363830
[04/07/2015-21:57:56] UTXO tree root hash: d229909f0021c67165fd317a9bb519125996b68335b2aff3ef4651339d5174d7
[04/07/2015-21:57:56] Coins in database: 1434314010655316
[04/07/2015-21:57:56] TCP server started on 94.102.50.70:50001
[04/07/2015-21:57:56] SSL server started on 94.102.50.70:50002

It looks like the blockchain height isn't synchronising properly because it says 363830 instead of 363834.

EDIT: changed the file limits to 64k to see what that does, but i had it already static in limits.conf, but when i did ulimit -n it told me 1024. Because electrum-server needs to be run under root i have added the same lines for root also in the limits.conf. Now it is syncing i guess, because cpu load it around 20% all the time, but no specific new lines in the log...so i wait until it is done with its job and see what happens then.

[JWU42]

Now it is syncing i guess, because cpu load it around 20% all the time, but no specific new lines in the log...so i wait until it is done with its job and see what happens then.

The log only outputs every 1000 blocks.

You can do...

Code:

electrum-server getinfo

To get an output of what block it is working on...

[BadAss.Sx]

Thanks, it is working now....apparently is was the size limits issue, because server came online and i can connect now with my client

[BadAss.Sx]

Have a new problem:

[05/07/2015-17:43:36] Starting Electrum server on electrum.badass.sx
[05/07/2015-17:43:37] Database version 3
[05/07/2015-17:43:37] Blockchain height 363959
[05/07/2015-17:43:37] UTXO tree root hash: 83d729bc609764ee6bfd7f8e33ece94e33d68aada186554fe1c9579b622f2bc9
[05/07/2015-17:43:37] Coins in database: 1434845972693919
[05/07/2015-17:43:37] TCP server started on 94.102.50.70:50001
[05/07/2015-17:43:37] SSL server started on 94.102.50.70:50002
[05/07/2015-17:43:39] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef

How can i pass this? Doesn't matter if i restart the server, the problem remains and electrum server isn't public.

[JWU42]

Have a new problem:

[05/07/2015-17:43:36] Starting Electrum server on electrum.badass.sx
[05/07/2015-17:43:37] Database version 3
[05/07/2015-17:43:37] Blockchain height 363959
[05/07/2015-17:43:37] UTXO tree root hash: 83d729bc609764ee6bfd7f8e33ece94e33d68aada186554fe1c9579b622f2bc9
[05/07/2015-17:43:37] Coins in database: 1434845972693919
[05/07/2015-17:43:37] TCP server started on 94.102.50.70:50001
[05/07/2015-17:43:37] SSL server started on 94.102.50.70:50002
[05/07/2015-17:43:39] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef

How can i pass this? Doesn't matter if i restart the server, the problem remains and electrum server isn't public.

Not a problem - it hit my log as well.  Reorg's happen...

Code:

[05/07/2015-15:25:23] blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15ef