Electrum server discussion thread

[TheButterZone]

Submitted
https://github.com/spesmilo/electrum-server/issues/98
for
https://bitcointalk.org/index.php?topic=1109497.0

[memvola]

Hi!

Unfortunately my server couldn't get past the reorg; seems like it can't revert. bitcoind running fine, updated to the latest version, regenerated/imported blockchain_headers to no avail. Any ideas?

Code:

electrum-server[2001]: INFO:electrum:blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15e
electrum-server[2001]: Exception in thread Thread-4:
electrum-server[2001]: Traceback (most recent call last):
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
electrum-server[2001]: self.run()
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 763, in run
electrum-server[2001]: self.__target(*self.__args, **self.__kwargs)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 76, in do_catch_up
electrum-server[2001]: self.catch_up(sync=False)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 658, in catch_up
electrum-server[2001]: self.import_block(block, self.storage.last_hash, self.storage.height, sync, revert=True)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 408, in import_block
electrum-server[2001]: self.storage.revert_transaction(txid, tx, block_height, touched_addr, undo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 602, in revert_transaction
electrum-server[2001]: self.revert_add_to_history(addr, txid, x.get('index'), x.get('value'), block_height)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 511, in revert_add_to_history
electrum-server[2001]: self.delete_address(key + txo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 416, in delete_address
electrum-server[2001]: items.pop(letter)
electrum-server[2001]: KeyError: '7'

[JWU42]

Hi!

Unfortunately my server couldn't get past the reorg; seems like it can't revert. bitcoind running fine, updated to the latest version, regenerated/imported blockchain_headers to no avail. Any ideas?

Code:

electrum-server[2001]: INFO:electrum:blockchain reorg 363959 000000000000000010f14e0687867ae37dafce2a918cdf3f9eccdfa992445eb5 00000000000000001262efe584ec7ac039dba15b56d6ed239249c483636b15e
electrum-server[2001]: Exception in thread Thread-4:
electrum-server[2001]: Traceback (most recent call last):
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
electrum-server[2001]: self.run()
electrum-server[2001]: File "/usr/lib/python2.7/threading.py", line 763, in run
electrum-server[2001]: self.__target(*self.__args, **self.__kwargs)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 76, in do_catch_up
electrum-server[2001]: self.catch_up(sync=False)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 658, in catch_up
electrum-server[2001]: self.import_block(block, self.storage.last_hash, self.storage.height, sync, revert=True)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/blockchain_processor.py", line 408, in import_block
electrum-server[2001]: self.storage.revert_transaction(txid, tx, block_height, touched_addr, undo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 602, in revert_transaction
electrum-server[2001]: self.revert_add_to_history(addr, txid, x.get('index'), x.get('value'), block_height)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 511, in revert_add_to_history
electrum-server[2001]: self.delete_address(key + txo)
electrum-server[2001]: File "/usr/lib/python2.7/site-packages/electrumserver/storage.py", line 416, in delete_address
electrum-server[2001]: items.pop(letter)
electrum-server[2001]: KeyError: '7'

It seems the best option for now is grabbing a new DB from foundry...

https://foundry.electrum.org/

[ThomasV]

Hi!

Unfortunately my server couldn't get past the reorg; seems like it can't revert. bitcoind running fine, updated to the latest version, regenerated/imported blockchain_headers to no avail. Any ideas?

...

This traceback means that your database was corrupted.
There is a robot, 'electra', that detects servers with a corrupted DB, and kicks them from the irc channel.
Unfortunately this robot is temporarily broken (I will try to fix it this week), and it seems that a few servers have been running with db corruption without noticing it.
This reorg made the job that electra usually does

[memvola]

There is a robot, 'electra', that detects servers with a corrupted DB, and kicks them from the irc channel.
Unfortunately this robot is temporarily broken (I will try to fix it this week), and it seems that a few servers have been running with db corruption without noticing it.

Is there a way to check for corruption locally?

[BadAss.Sx]

There was an orphan block at height 363959.  Your bitcoind should have rejected the orphan block, if you are using Bitcoin-core v0.9.5 and above - which version are you using?

I used 0.10.0, but have upgraded now to 0.10.2. Also redownloaded the db, because mine was corrupt and couldn't get past the orphaned block.

[tl121]

My private Electrum server is having trouble keeping up with the present network activity,  being up to 11 blocks behind.  The bitcoin node on the same machine has no trouble keeping up and the CPU time used by electrum-server is now 7 times greater than that used by bitcoind.

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual).  In addition, it does not pull out a new unconfirmed received transactions that have been in the mempool for over an hour.  (Switching the client to another server does show this unconfirmed transaction.)  This is a private server and only one client is accessing it.  Perhaps a better balance in thread priority can be taken between resources spent tracking the network and updating the database vs. providing timely interaction to clients.

Looking at the logs, it looks like one block gave the server serious cramps:

[08/07/2015-12:24:49] utxo not in database; postponing mempool update
[08/07/2015-12:24:49] blockchain: 364428 (5284.813s)
[08/07/2015-12:25:15] mempool_update 16.0112478733

The machine has a Haswell Atom processor, but it has 8 GB of RAM and an SSD.  Running Ubuntu 14.04, Electrum Server updated from github yesterday.

[molecular]

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual).

Does a client restart fix this particular issue?

If so, maybe you're looking at this issue instead.

[tl121]

Worse, when I use an Electrum client to access the node I observe that the client reports an erroneous number of blocks lagging (more blocks than actual).

Does a client restart fix this particular issue?

If so, maybe you're looking at this issue instead.

Thanks for the suggestion. 

Because I had previously experienced the problem that you linked, I had already tried a number of client restarts. They didn't help.  At present my server is only 3 blocks behind (according to getinfo via SSH) and this is consistent with what my Electrum client  reports.  However, the unconfirmed transaction has been sitting for three hours (presumably due to the "test").  Blockchain.info has had the transaction for three hours, but my Electrum client still doesn't show it.  (It will show it if I connect it to a different Electrum server, so I know that this is server related. After changing servers, I restarted the client and it reverted back to showing the transaction unconfirmed.  I will have to wait until my server catches up and then I will see if it picks up the unconfirmed transaction.)

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

[molecular]

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

I agree to an extent. However I think the problem is mitigated for "newbie users" because they will likely use "auto connect" and I'm assuming lagging servers will not be autoconnected to.

Of course the electrum server software could probably be made to perform better, but I'm guessing it'd be quite a bit of work.

Until then and although it's a bit sad and maybe even mildly concerning (centralization), maybe your little atom (not to discredit it, but I assume it's the bottleneck in your system when electrum is digesting transactions) will have to leave the work to faster machines regarding servicing the masses?

[tl121]

I believe it should be high priority for the Electrum system to show unconfirmed payments as rapidly as possible, so as to make for a Newbie friendly user experience.

I agree to an extent. However I think the problem is mitigated for "newbie users" because they will likely use "auto connect" and I'm assuming lagging servers will not be autoconnected to.

Of course the electrum server software could probably be made to perform better, but I'm guessing it'd be quite a bit of work.

Until then and although it's a bit sad and maybe even mildly concerning (centralization), maybe your little atom (not to discredit it, but I assume it's the bottleneck in your system when electrum is digesting transactions) will have to leave the work to faster machines regarding servicing the masses?

A couple of years ago, I played around with Electrum and rejected it because of the "server lagging" issues and the unreliability of connections to the servers.  It seemed dodgy.  These problems are not new and not the fault of my particular server hardware. At the time I was not running a server and was using the auto selection feature.

I chose the little Atom deliberately, so that it would be a canary in the coal mine.  My little machine is not servicing the masses.  It's just servicing me.  If tracking the block chain can not be done efficiently, then the future of Electrum is at risk. I would say the future of bitcoin as well as a decentralized system, except that the little Atom manages to keep up with the present load using only a few percent of CPU time to run bitcoind.

Electrum-server used 5832 seconds to process one block!  This is a bug.  Not a slow processor. 

[it-zone]

Electrum-server used 5832 seconds to process one block!  This is a bug.  Not a slow processor. 

Unfortunately I must agree, the architecture of electrum server is screwed. If we use power of the internet to make electrum possible, we should share resources between electrum servers.

But there is no clear solution. Which Key-Value database engine is able work on the WAN scale?

[tl121]

Electrum-server used 5832 seconds to process one block!  This is a bug.  Not a slow processor.  

I wouldn't say that it is a bug, but instead, not optimized for slow processors, or bloated blocks, full of useless UTXOs that the Bitcoin spammer is currently injecting in the network today...

... over 50k transactions are in the waiting room at the moment, waiting to be included into a block... https://chain.so/BTC

If senders didn't put at least 0.0005 BTC per kb, as transaction fees, the waiting time to be included into a block is... 16h

One of those transactions is mine.  I sent it from my Bitcoin Core 10.2 wallet with the default transaction cost.  I believe this was a bug, with less than 1000 Satoshi's as fee.  It has been sitting unconfirmed for 24 hours.   I am sure that lots of other people are in the same boat.  About the only way to put it is, "Bitcoin is experimental.  It is not ready for production use by normal people."  These problems will either be fixed or bitcoin will cease to exist.

[BadAss.Sx]

I believe there is a 48hours timeout...so if you haven't confirmed them yet, they should come back into your wallet.

[tl121]

I believe there is a 48hours timeout...so if you haven't confirmed them yet, they should come back into your wallet.

Thanks.  I'm not worried.  One way or another, I'll get the bitcoins back since I sent them from my bitcoin core wallet to my Electrum wallet.  And they were pocket change, compared to the amount I've previously lost thanks to scamming bitcoin miner companies...   

[kamk]

It looks like the leveldb dumps are not updated in the foundry:
http://foundry.electrum.org/leveldb-dump/?C=M;O=D

Last dump is from July 8.

[BadAss.Sx]

Mostly it is done once a week

[JWU42]

It looks like the leveldb dumps are not updated in the foundry:
http://foundry.electrum.org/leveldb-dump/?C=M;O=D

Last dump is from July 8.

It used to be daily...

I update the 1000 limit weekly (on Saturday).

[mustyoshi]

I believe I am correctly running a public Electrum node at btc.mustyoshi.com:50001

[tnkflx]

I believe I am correctly running a public Electrum node at btc.mustyoshi.com:50001

You server doesn't seem to be connected on IRC... So no, you're not correctly running a public Electrum node