Ledger Nano with Electrum Wallet - getting history

[bitschoel]

Trying to connect my Ledger Nano S with an Electrum wallet, using Electrum Personal Server on my own Bitcoin Core Node.
My Ledger has an existing account/wallet that I've used with ledgerlive but I'm moving to a more private model.

Electrum Wallet: 3.3.6
EPS - 0.1.7

Guides I have followed:

https[Suspicious link removed]d-ledgerlive-electrum-eps-wasabi-6d638ec0f71f

EPS installed, configured and rescanned (per the block height of my first Bitcoin transaction) successfully, as far as I can tell.
EPS wallet installed on Mac OS and the setup of the hardware wallet via Electrum wallet went without a hitch.

The desktop Electrum wallet starts only syncing with my EPS/Bitcoin node.
Note - the freshAddressPath in LedgerLive for this showed "freshAddressPath": "49'/0'/0'/0/15".  I deduced that this would mean a "p2sh-segwit" Script type

My issue revealed itself when my Electrum wallet showed as completed the sync with my EPS/node, but none of the transactions associated with that wallet appeared in the History tab.
Upon further investigation, the xpub value for my wallet/account in LedgerLive didn't match the Master Public Key value set up in Electrum wallet, connected to my Ledger device
The Electrum Wallet value ends up as a "ypub...." value that doesn't match.


Why doesn't Electrum wallet use the same xpub value that LedgerLive reveals via the Advanced Log?
It was my assumption that to set up an Electrum Wallet with the Ledger device, the Master Public Key should match.
I tried both the Ledger xpub MPK value and the Electrum Wallet ypub MPK value in the EPS config file to see if either would sync, but neither produced history.

Any help is appreciated as I'm stuck.

[1715280000]

1715280000

[1715280000]

1715280000

[Rath_]

If you are trying to create a watch-only wallet then you can copy your Master Public Key from Ledger Live and paste it after creating new standard wallet and choosing 'Use a master key'. The link to the guide has been automatically removed so I am not sure if that's what you are looking for. Are you trying to import your seed?

[HCP]

Why doesn't Electrum wallet use the same xpub value that LedgerLive reveals via the Advanced Log?

Because Electrum is doing it "correctly" and using xpub for Legacy, ypub for P2SH-P2WPKH and zpub for P2WPKH:

These are the registered HD version bytes for extended serialization of public and private keys.

Coin   Public Key   Private Key   Address Encoding   BIP 32 Path
Bitcoin   0x0488b21e - xpub   0x0488ade4 - xprv   P2PKH or P2SH   m/44'/0'
Bitcoin   0x049d7cb2 - ypub   0x049d7878 - yprv   P2WPKH in P2SH   m/49'/0'
Bitcoin   0x04b24746 - zpub   0x04b2430c - zprv   P2WPKH   m/84'/0'

It's been a known "issue" for quite a while that Ledger Live displays the "wrong" MPK by showing an xpub for the P2SH-P2WPKH wallets.


Having said that, you can use this converter to confirm you have the "correct" values: https://jlopp.github.io/xpub-converter/

Either put in the ypub from Electrum... select "xpub (mainnet P2WPKH or P2SH)" option. You'll see the xpub from Ledger Live
or
Put in the xpub from Ledger Live... select "ypub (mainnet P2WPKH in P2SH)" option. You'll see the ypub from Electrum

It was my assumption that to set up an Electrum Wallet with the Ledger device, the Master Public Key should match.
I tried both the Ledger xpub MPK value and the Electrum Wallet ypub MPK value in the EPS config file to see if either would sync, but neither produced history.

I haven't tried using EPS, but I'm successfully using Bitcoin Core + electrs (Electrum Server in Rust) + Electrum in "oneserver" mode.

[bitschoel]

Thanks to HCP for help with LedgerLive's MPK confusion.

I used the URL to translate the link and verified the matching values between them.
I've defaulted to using the better "ypub" MPK, and while  my Electrum wallet shows a green connection to my EPS server, it still isn't showing any history.
I've confirmed that the EPS server has included the necessary transactions.

Anyone know if there are logs to check in Electrum Wallet to verify connectivity or to see why no history is appearing in the UI?

[HCP]

Perhaps a silly question... but did you follow the instructions on the EPS Github for rescanning AFTER first run and new addresses added as watch-only to the Bitcoin Core node?

The first time the server is run it will import all configured addresses as watch-only into the Bitcoin node, and then exit. If the wallets contain historical transactions you can use the rescan script (electrum-personal-server --rescan /path/to/config.ini) to make them appear. If using the windows packaged binary release build then drag the file config.ini onto the file electrum-personal-server-rescan.bat.

[bitschoel]

Perhaps a silly question... but did you follow the instructions on the EPS Github for rescanning AFTER first run and new addresses added as watch-only to the Bitcoin Core node?

Confirmed.

The EPS log, on startup, lists the first 3 transaction values per MPK from the config.  These have been matched to the existing first 3 transactions from my wallet, for confirmation that the EPS instance is
(1) connecting and receiving from my bitcoin node,
(2) Has rescanned the transactions desired

I'm now at the point where the Electrum wallet, using the MPK that I've listed in the config, still is not showing any transactions.

Any thoughts/ideas on how to debug the Electrum wallet?

[HCP]

Does the EPS have any logs that show that your Electrum wallet is actually connecting to it and requesting the correct info? Seems odd that EPS "has" your info... but isn't returning it to Electrum??!?

For Electrum logs: Tools -> Preferences -> General... There is a "write logs to file" option


You'll need to restart... and a "logs" directory will be created in the Electrum data directory


Also, I believe Electrum has a debug mode if you run it from the commandline using the -v argument... I believe this will only work for Linux and possibly Mac OSX. Fairly certain it doesn't work on Windows.

[bitschoel]

Below are
(1)the log from the Electrum Wallet as it was activated and run from my Mac, connecting to the EPS.
(2) the log from the EPS

Command line used to fire up Electrum wallet
open /Applications/Electrum.app/ --args --oneserver --server 192.XXX.XXX.XXX:50002:s -v

It seems to connect successfully and I don't see any error around history.  Please point me to anything I'm missing in the output.
Also, is there anything in particular to watch for in the EPS log that would indicate a problem?

20190624T220503.862382Z |     INFO | logging | Electrum version: 3.3.6 - https://electrum.org - https://github.com/spesmilo/electrum
20190624T220503.878355Z |     INFO | logging | Python version: 3.6.4 (default, Feb 28 2018, 03:24:19)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]. On platform: Darwin-18.6.0-x86_64-i386-64bit
20190624T220503.878891Z |     INFO | logging | Logging to file: /Users/XXXXXXXX/.electrum/logs/electrum_log_20190624T220503Z_27943.log
20190624T220503.879017Z |     INFO | logging | Log filters: verbosity '*', verbosity_shortcuts ''
20190624T220503.881530Z |     INFO | daemon | failed to connect to JSON-RPC server: [Errno 61] Connection refused
20190624T220503.884212Z |     INFO | plugin.Plugins | registering hardware coldcard: ('hardware', 'coldcard', 'Coldcard Wallet')
20190624T220503.885222Z |     INFO | plugin.Plugins | registering hardware digitalbitbox: ('hardware', 'digitalbitbox', 'Digital Bitbox wallet')
20190624T220503.887601Z |     INFO | plugin.Plugins | registering hardware keepkey: ('hardware', 'keepkey', 'KeepKey wallet')
20190624T220503.888795Z |     INFO | plugin.Plugins | registering hardware ledger: ('hardware', 'ledger', 'Ledger wallet')
20190624T220503.889868Z |     INFO | plugin.Plugins | registering hardware safe_t: ('hardware', 'safe_t', 'Safe-T mini wallet')
20190624T220503.890443Z |     INFO | plugin.Plugins | registering hardware trezor: ('hardware', 'trezor', 'Trezor wallet')
20190624T220503.891016Z |     INFO | plugin.Plugins | registering wallet type ('2fa', 'trustedcoin')
20190624T220503.891785Z |    DEBUG | util.profiler | Plugins.__init__ 0.0092
20190624T220503.893254Z |     INFO | network | blockchains

20190624T220503.893670Z |     INFO | exchange_rate.FxThread | using exchange CoinGecko
20190624T220503.894627Z |     INFO | network | starting network
20190624T220503.895015Z |     INFO | network | connecting to 192.XXX.XXX.XXX:50002:s as new interface
20190624T220503.896613Z |    DEBUG | util.profiler | Daemon.__init__ 0.0047
20190624T220504.055032Z |     INFO | gui.qt.history_list | could not import electrum.plot. This feature needs matplotlib to be installed.
20190624T220504.118047Z |     INFO | interface.[192.XXX.XXX.XXX:50002] | connection established. version: ['ElectrumPersonalServer 0.1.7', 1.4]
20190624T220504.137297Z |     INFO | interface.[192.XXX.XXX.XXX:50002] | set blockchain with height 582284
20190624T220504.137941Z |     INFO | interface.[192.XXX.XXX.XXX:50002] | could connect 582285
20190624T220504.139249Z |     INFO | network | switching to 192.XXX.XXX.XXX:50002:s
20190624T220504.265758Z |     INFO | network | fee_histogram [[0, 0]]
20190624T220504.266113Z |     INFO | network | fee_estimates {25: 1019, 10: 39980, 5: 62246, 2: 91427}
20190624T220504.502280Z |     INFO | network | fee_histogram [[0, 0]]
20190624T220504.502548Z |     INFO | network | fee_estimates {25: 1019, 10: 39980, 5: 62246, 2: 91427}
20190624T220504.520631Z |    DEBUG | util.profiler | ElectrumGui.__init__ 0.4627
20190624T220504.521154Z |     INFO | storage.WalletStorage | wallet path /Users/XXXXXXXX/.electrum/wallets/btc_wallet
20190624T220504.521537Z |    DEBUG | util.profiler | JsonDB.load_transactions 0.0000
20190624T220504.523006Z |    DEBUG | util.profiler | AddressSynchronizer.load_local_history 0.0000
20190624T220504.523311Z |    DEBUG | util.profiler | AddressSynchronizer.check_history 0.0001
20190624T220504.631734Z |    DEBUG | util.profiler | WalletStorage.write 0.0000
20190624T220504.781916Z |    DEBUG | util.profiler | Deterministic_Wallet.try_detecting_internal_addresses_corruption 0.0472
20190624T220504.782163Z |    DEBUG | util.profiler | ElectrumWindow.load_wallet 0.0775
20190624T220505.295330Z |     INFO | gui.qt.history_list.HistoryModel | refreshing... reason: update_tabs
20190624T220505.295769Z |    DEBUG | util.profiler | Abstract_Wallet.get_full_history 0.0001
20190624T220505.295940Z |    DEBUG | util.profiler | HistoryModel.refresh 0.0006
20190624T220505.541058Z |     INFO | gui.qt.update_checker.UpdateCheckThread | valid sig for version announcement '3.3.6' from address '13xjmXXXXXXXXXXXXXXXXXXXXXXXXp1P'
20190624T220509.649055Z |     INFO | interface.[192.XXX.XXX.XXX:50002] | skipping header 582285
20190624T220603.900479Z |     INFO | network | network: retrying connections
20190624T220606.996286Z |     INFO | network | fee_histogram [[0, 0]]
20190624T220606.996732Z |     INFO | network | fee_estimates {25: 1019, 10: 39980, 5: 62246, 2: 91427}
20190624T220709.006869Z |     INFO | network | network: retrying connections
20190624T220709.105801Z |     INFO | network | fee_histogram [[0, 0]]
20190624T220709.106267Z |     INFO | network | fee_estimates {25: 1019, 10: 39980, 5: 62246, 2: 91427}
20190624T220725.534575Z |     INFO | gui.qt.main_window.[btc_wallet] | close_wallet /Users/XXXXXXXX/.electrum/wallets/btc_wallet
20190624T220725.537937Z |     INFO | storage.WalletStorage | saved /Users/XXXXXXXX/.electrum/wallets/btc_wallet
20190624T220725.538259Z |    DEBUG | util.profiler | WalletStorage.write 0.0018
20190624T220725.584756Z |     INFO | daemon.Daemon | shutting down network
20190624T220725.585408Z |     INFO | network | stopping network
20190624T220725.590043Z |     INFO | plugin.Plugins | stopped
20190624T220725.605508Z |     INFO | interface.[192.XXX.XXX.XXX:50002] | disconnecting due to: CancelledError()
20190624T220725.606736Z |     INFO | daemon.Daemon | stopped

INFO:2019-06-24 16:00:48,547: Starting Electrum Personal Server
INFO:2019-06-24 16:00:48,548: Logging to /tmp/electrumpersonalserver.log
INFO:2019-06-24 16:00:50,582: Displaying first 3 addresses of each master public key:
INFO:2019-06-24 16:00:50,862:
btc_wallet =>
   [intentionally obfuscated]
INFO:2019-06-24 16:00:50,863: Obtaining bitcoin addresses to monitor . . .
INFO:2019-06-24 16:03:39,632: Obtained list of addresses to monitor in 170.69092750549316sec
INFO:2019-06-24 16:03:39,635: Building history with 2000 addresses . . .
INFO:2019-06-24 16:03:40,725: Found 0 txes. History built in 0.15528178215026855sec
INFO:2019-06-24 16:03:40,726: Starting electrum server
INFO:2019-06-24 16:03:41,517: Listening for Electrum Wallet on ('0.0.0.0', 50002)
INFO:2019-06-24 16:05:04,123: Electrum connected from ('192.XXX.XXX.XXX', 49851)
INFO:2019-06-24 16:07:25,599: Electrum wallet disconnected

[HCP]

INFO:2019-06-24 16:03:39,635: Building history with 2000 addresses . . .
INFO:2019-06-24 16:03:40,725: Found 0 txes. History built in 0.15528178215026855sec

That looks like the culprit right there... While it might have the correct addresses (based on what you said previously that the first 3 displayed addresses are correct), it would appear that EPS isn't actually finding any history.

Do these addresses/transactions actually show up within Bitcoin Core?

[bitschoel]

This is so curious.

The addresses listed in the log under
INFO:2019-06-24 16:00:50,582: Displaying first 3 addresses of each master public key:

1) can be found on blockstream.info
2) are in the list of already-imported addresses, as produced into the log

Since this installation of EPS has only ever connected to my bitcoin full node, it would have had to retrieve these addresses from the full node, I presume.
Which confirms my assumption that these are addresses/transactions on the block chain.

I'm trying to figure out why EPS still seems unable to find transactions related to the list of addresses it has in memory.

[HCP]

No, the addresses will be generated from the Master Public Key... which as I understand you have to put in the EPS config.

As it isn't able to find any transactions relating to those generated addresses, it would appear that your Bitcoin Core node has not indexed them for whatever reason. Perhaps try asking the EPS dev for assistance here: https://bitcointalk.org/index.php?topic=2664747.msg27179198