Electrumx not updating with mempool transactions

[tiagocs]

I'm running a bitcoin full node (v0.20.0) with an electrumx server (version 1.13.0) on top and I noticed that electrumx sometimes is veryyy slow to update itself with new transactions. What I am doing is: create a new signed transaction, broadcast it, make a CPFP transaction and broadcast it too. Then, I try to see if electrumx can detect the child transaction by querying electrumx for the scriptPubKeys of the parent transaction and see if anything is spending it. Around 2/3 of the times, electrumx updates quickly and can detect the child transaction but around 1/3 of the times it takes many minutes, like 15 minutes for it to detect and answer this child transaction. I have done this experiment like 9 times. I don't see any issues with my bitcoin node, its mempool contains the child transaction and when I run

Code:

bitcoin-cli getrawmempool true | jq '."parent_txid"'

I see the child txid in the "spentby" field.


Here is my electrumx.conf configuration:

Code:

DB_DIRECTORY = /media/pedro/Portable_2TB/Bitcoin/electrumx/data
DAEMON_URL = http://user:***@127.0.0.1
COIN = "BitcoinSegwit"
NET = "mainnet"
ALLOW_ROOT=1

PEER_DISCOVERY= self
SERVICES = rpc://localhost:8000,tcp://localhost:50001

INITIAL_CONCURRENT=500
COST_SOFT_LIMIT=10000000
COST_HARD_LIMIT=10000000
BANDWIDTH_UNIT_COST=1

NOTE: my bitcoin node gives a weird answer to `bitcoin-cli getchaintips` but looks good otherwise. Here is the answer:

Code:

>bitcoin-cli getchaintips
[
  {
    "height": 644185,
    "hash": "00000000000000000002fc463486fa70cc80cecdb0ebdfd8f7e2f056cbeb929c",
    "branchlen": 0,
    "status": "active"
  },
  {
    "height": 638519,
    "hash": "00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26",
    "branchlen": 1,
    "status": "valid-fork"
  }
]

I tried restarting the node and running 'bitcoin-cli reconsiderblock 00000000000000000011398b899fb881c61bd6e8670270fbc8e225c11fd95d26' but it didn't solve, could this be related? It's working fine otherwise it seems.

[DaveF]

It looks like you are running it on an external USB drive, if that is it that is probably your issue. USB is not always good to do this.

Beyond that standard questions:

What kind of hardware do you have running the server?
RAM / CPU / Drive space & type.
Physical or virtual?

Is it doing / running anything else or is it just bticoind & electrumx?
When it pauses if you run top are any other processes stealing CPU time / cycles?
Same with the drive what is iostat showing when it pauses?

-Dave

[tiagocs]

This issue is occuring locally on my laptop (Lenovo Thinkpad W541)  which is connected to an external HDD with 2T that stores the bitcoin blockchain and eletrumx data.

My laptop specs:
Ubuntu 18.04.4
16 GB RAM/ Intel(R) Core(TM) i7-4810MQ CPU @ 2.80GHz/ ~100 GB partition SSD (using a HDD with 2T to store the blockchain/electrumx data)

I have some open apps running in the background like chrome, telegram, keybase, pycharm, IntelliJ ... I don't see any specific app consuming a lot of resources.

Here is the output of iostat when the problem occurred:

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.1%    0.0%    9.0%    7.2%    0.0%   51.7%

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
loop0
                  0.00         0.0k         0.0k       6.3M       0.0k
loop1
                  0.00         0.0k         0.0k       3.7M       0.0k
loop2
                  0.00         0.0k         0.0k     111.0k       0.0k
loop3
                  0.01         0.0k         0.0k      10.8M       0.0k
loop4
                  0.00         0.0k         0.0k     118.0k       0.0k
loop5
                  0.00         0.0k         0.0k       5.6M       0.0k
loop6
                  0.00         0.0k         0.0k     813.0k       0.0k
loop7
                  0.00         0.0k         0.0k      44.0k       0.0k
sda
                 17.17       151.9k       162.7k     163.9G     175.6G
loop8
                  0.00         0.0k         0.0k       5.6M       0.0k
loop9
                  0.01         0.0k         0.0k       6.5M       0.0k
loop10
                  0.53         0.5k         0.0k     588.2M       0.0k
loop11
                  0.00         0.0k         0.0k       6.4M       0.0k
loop12
                  0.00         0.0k         0.0k      78.0k       0.0k
loop13
                  0.01         0.0k         0.0k       6.6M       0.0k
loop14
                  0.00         0.0k         0.0k       3.3M       0.0k
loop15
                  0.00         0.0k         0.0k     192.0k       0.0k
loop16
                  0.00         0.0k         0.0k     758.0k       0.0k
loop17
                  0.00         0.0k         0.0k       2.5M       0.0k
loop18
                  0.00         0.0k         0.0k       3.5M       0.0k
loop19
                  0.01         0.0k         0.0k       8.0M       0.0k
loop20
                  0.17         0.2k         0.0k     183.5M       0.0k
loop21
                  0.15         0.2k         0.0k     169.5M       0.0k
loop22
                  0.21         0.2k         0.0k     231.2M       0.0k
loop23
                  0.07         0.1k         0.0k      80.3M       0.0k
loop24
                  0.00         0.0k         0.0k       6.0M       0.0k
loop25
                  0.00         0.0k         0.0k       5.7M       0.0k
loop26
                  0.00         0.0k         0.0k       5.6M       0.0k
loop27
                  0.01         0.0k         0.0k       6.7M       0.0k
loop28
                  0.00         0.0k         0.0k     493.0k       0.0k
loop29
                  0.00         0.0k         0.0k      46.0k       0.0k
sdb
                 34.03         1.3M       504.2k       1.5T     544.1G
loop30
                  0.05         0.1k         0.0k      61.0M       0.0k
loop31
                  0.00         0.0k         0.0k       8.0k       0.0k

[DaveF]

Nothing is jumping out at me as an issue.
Can you run a speed test on the USB drive. See what it maxes out at?

It should be able to do a lot more then the 1.3M shown. So if that is all it's pulling then the issue is elsewhere. If that is all it can do then that may be the issue.

-Dave

[ABCbits]

Here is the output of iostat when the problem occurred:
--snip--

Can you use another tools such as iotop which shows drive usage in percentage when the problem occurred? if electrumx or bitcoin core have very high disk usage, then your HDD is likely the reason.

It should be able to do a lot more then the 1.3M shown. So if that is all it's pulling then the issue is elsewhere. If that is all it can do then that may be the issue.

Without knowing whether ElectrumX perform sequential or random read & write, i wouldn't say 1.3M/s is slow.

[bob123]

I'm with ETFbitcoin here.
The bottleneck seems to be your external disk.

Is your disk attached via USB2 or USB3?
Could you temporarily copy the required data onto an internal HDD and try it again? This would confirm whether the HDD is the bottleneck causing the issues. An external SSD connected via USB3 should be fine too.

[DaveF]

Without knowing whether ElectrumX perform sequential or random read & write, i wouldn't say 1.3M/s is slow.

Yes but...most external drives are crap.
However, from my non scientific testing over the years, mostly 2.5" portables, they tend to be consistent crap in terms of the drive but not so much in the controller.

From the ones I use most USBs will never see a large difference between sequential or random reads / writes. The USB controller that talks to the drive itself is the weak point and you never get to the limit of what the drive and it's controller itself can do.

Just what I have seen. YMMV.

-Dave

[tiagocs]

I changed from a 5 year-old 2TB external HDD to a recent 4TB external HDD and I think the problem is still there. I haven't experimented much today... I have already spent ~ 3 mBTC in the past two days in bitcoin fees

I ran "iotop -a" and waited for the numbers to stabilize. The result is:
https://i.imgur.com/vJu9ZTP.png 

I realised that "iostat" gives the average I/O stats and not the instantaneous ones. I moved a large file from the external HDD into my laptop's SSD and saw that it averaged around 30 MB/s

Code:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.3%    0.0%    3.2%   24.5%    0.0%   68.1%

Device             tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
sde
                293.00        36.0M         0.0k      36.0M       0.0k

If the problem is I/O speed from the drive, I am not worried because I will launch my app on AWS with an SSD... I am more concerned about excluding other causes.

[HCP]

May not get any answers due to the ElectrumX dev committing to BSV, but have you tried logging an issue on the ElectrumX GitHub? (or maybe the fork of ElectrumX that is still supporting BTC)

[tiagocs]

Based on average read speed, do you use USB 2 (whether the port is USB 2 or the external HDD uses USB 2) ?

I think it's USB 3 (the external HDD is the Seagate device - bus 004)

Code:

>lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp. 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 007: ID 0bc2:231a Seagate RSS LLC 
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 003: ID 138a:0017 Validity Sensors, Inc. Fingerprint Reader
Bus 003 Device 010: ID 0458:0186 KYE Systems Corp. (Mouse Systems) 
Bus 003 Device 002: ID 1058:10b8 Western Digital Technologies, Inc. Elements Portable (WDBU6Y, WDBUZG)
Bus 003 Device 016: ID 1b3f:2008 Generalplus Technology Inc. 
Bus 003 Device 005: ID 04ca:7035 Lite-On Technology Corp. 
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

May not get any answers due to the ElectrumX dev committing to BSV, but have you tried logging an issue on the ElectrumX GitHub? (or maybe the fork of ElectrumX that is still supporting BTC)

I know... I guess electrs will take the lead But last time I checked electrs was very bad at handling mempool transactions... any other electrum server implementation besides electrumX or electrs?

[HCP]

I know... I guess electrs will take the lead But last time I checked electrs was very bad at handling mempool transactions... any other electrum server implementation besides electrumX or electrs?

The only other one I know of is "Electrum Personal Server" aka "EPS"... might be worth a look?


Also, what exactly do you mean by "very bad at handling mempool transactions"? I've not had too many issues with it, but I'm not trying to do anything really extraordinary with electrs... basically just letting it run off of a standard Bitcoin Core node and connecting to it locally from my personal copy of Electrum...

[tiagocs]

EPS just keeps track of a set of scriptPubKeys associated with the user's seed, correct? I am not necessarily looking for something lightweight, I need something that can answer a lot of queries about any scriptPubKey fast.

Also, what exactly do you mean by "very bad at handling mempool transactions"?

I mean stuff like this https://github.com/romanz/electrs/issues/79

[gmaxwell]

I personally wouldn't use or run electrumx: It's author is a big outspoke advocate of a scammer.  It's not much of a leap to worry that in the future the software will be changed to exploit users more directly.

If that happened everyone else will be going "well, duh, what did you expect?"

[tiagocs]

Yes, using newer versions of Electrumx is out of the question and I am looking for an alternative. Currently, electrs seems like a good candidate but I need stuff issues like this https://github.com/romanz/electrs/issues/79 solved!

I need to query a lot of scriptPubKeys are not necessarily mine (like a block explorer). Any suggestions are greatly appreciated! Maybe I will try to get in touch with some block explorers and know owners of electrum servers and ask them what they are using (and if they are using electrumx, what they will switch to).