Remote ArmoryDB: "failed to connect socket"

[h.hanzo]

Hello everyone,

My Armory setup is playing up, and I was not able to figure it out in the last days.
I'll try to post as much hopefully relevant info as possible, please bear with me! :-)

I am running a bitcoin core node (non pruned) and ArmoryDB on a remote server, and connect to it via local Armory GUI (0.96.5-beta) (over local, wired ethernet).
This setup basically stayed the same for a long time. The server is running debian 12 bookworm, with 32 GB memory, a current i3-13100 (4 core 8 threads 4.5 Ghz boost), the bitcoin blockchain and armory DB are on a SATA HDD. The client is a debian 10 buster, which I only use for Armory.

Sometimes, maybe every twentieth try, I start Armory and within a second it is online.

Half of the time only the little splashscreen opens, and crashes after a few minutes. The other half of the times the Armory gui opens, but stays offline.

The server is completely idle, especially there's basically no CPU load and only little memory usage of ArmoryDB and bitcoin-core (checked with top). The disks are basically idle too, with only little usage from unrelated tasks (checked with iotop).
On the server, I kept an eye on the nginx and ArmoryDB logs:

/var/log/nginx/access.log

Code:

192.168.0.100 - - [07/Jun/2024:12:40:38 +0200] "POST / HTTP/1.1" 200 44 "-" "-"
192.168.0.100 - - [07/Jun/2024:12:40:38 +0200] "POST / HTTP/1.1" 200 48 "-" "-"

/home/bitcoind/.armory/dbLog.txt

Code:

-INFO  - 12:40:38: (BDM_Server.cpp:1121) registered bdv: add89c28f658da1b1d55
-INFO  - 12:45:38: (BDM_Server.cpp:1150) unregistered bdv: add89c28f658da1b1d55

To me, this looks like Armory client always manages to connect to nginx and to ArmoryDB.

The unstable behavior might have started when I played around with importing several wallets and several individual privatekeys.
After importing the first wallet, I remembered rescanning takes some time. I restarted Armory Client, it came back up offline, and I imported some more
wallets in one go before leaving it catching up over night.
The next day I tried to send a tx from one of the imported private keys with many small outputs to one of the imported wallets.
Sending that tx froze Armory, and I think after that my problems began.
Before that, I didn't use Armory much lately, bitcoin-core and ArmoryDB were running 24/7 as services though.

To remove possible factors, I then moved the armorydb folder and let it recreate it over night. It was 4.7GB before, and is 4.1GB now.
Also, I removed all wallets there were, stopping and restarting bitcoin-core, nginx and ArmoryDB several times on the way.

I run the Armory client with
python /opt/BitcoinArmory/ArmoryQt.py --debug --armorydb-ip=192.168.0.1 --armorydb-port=9002

Full (terminal) log when Armory came up online instantly:

Code:

(INFO) ArmoryUtils.py:1164 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:689 - Executing popen: free -m
(INFO) ArmoryUtils.py:689 - Executing popen: ['cat', '/proc/cpuinfo']
(ERROR) ArmoryUtils.py:1262 - Error getting system details:
Traceback (most recent call last):
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1260, in <module>
    SystemSpecs = GetSystemDetails()
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1255, in GetSystemDetails
    out.HddAvailB = getHddSize(BTC_HOME_DIR)    / (1024**3)
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1252, in getHddSize
    s = os.statvfs(adir)
OSError: [Errno 2] No such file or directory: '/home/user/.bitcoin'
(ERROR) ArmoryUtils.py:1263 - Skipping.
(INFO) ArmoryUtils.py:1281 - 
(INFO) ArmoryUtils.py:1282 - 
(INFO) ArmoryUtils.py:1283 - 
(INFO) ArmoryUtils.py:1284 - ************************************************************
(INFO) ArmoryUtils.py:1285 - Invoked: /opt/BitcoinArmory/ArmoryQt.py --debug --armorydb-ip=192.168.0.1 --armorydb-port=9002
(INFO) ArmoryUtils.py:1286 - ************************************************************
(INFO) ArmoryUtils.py:1287 - Loading Armory Engine:
(INFO) ArmoryUtils.py:1288 -    Armory Version        : 0.96.5
(INFO) ArmoryUtils.py:1289 -    Armory Build:         : None
(INFO) ArmoryUtils.py:1290 -    PyBtcWallet  Version  : 1.35
(INFO) ArmoryUtils.py:1291 - Detected Operating system: Linux
(INFO) ArmoryUtils.py:1292 -    OS Variant            : debian-10.13-
(INFO) ArmoryUtils.py:1293 -    User home-directory   : /home/user
(INFO) ArmoryUtils.py:1294 -    Satoshi BTC directory : /home/user/.bitcoin
(INFO) ArmoryUtils.py:1295 -    Armory home dir       : /home/user/.armory/
(INFO) ArmoryUtils.py:1296 - Detected System Specs    : 
(INFO) ArmoryUtils.py:1297 -    Total Available RAM   : -1.00 GB
(INFO) ArmoryUtils.py:1298 -    CPU ID string         : Unknown
(INFO) ArmoryUtils.py:1299 -    Number of CPU cores   : -1 cores
(INFO) ArmoryUtils.py:1300 -    System is 64-bit      : Unknown
(INFO) ArmoryUtils.py:1301 -    Preferred Encoding    : UTF-8
(INFO) ArmoryUtils.py:1302 -    Machine Arch          : x86_64
(INFO) ArmoryUtils.py:1303 -    Available HDD (ARM)   : -1 GB
(INFO) ArmoryUtils.py:1304 -    Available HDD (BTC)   : -1 GB
(INFO) ArmoryUtils.py:1305 - 
(INFO) ArmoryUtils.py:1306 - Network Name: Main Network
(INFO) ArmoryUtils.py:1307 - Satoshi Port: 8333
(INFO) ArmoryUtils.py:1308 - Do wlt check: True
(INFO) ArmoryUtils.py:1309 - Named options/arguments to armoryengine.py:
(INFO) ArmoryUtils.py:1311 -     force_segwit    : False
(INFO) ArmoryUtils.py:1311 -     nettimeout      : 2
(INFO) ArmoryUtils.py:1311 -     rescan          : False
(INFO) ArmoryUtils.py:1311 -     force_fcgi      : False
(INFO) ArmoryUtils.py:1311 -     ignoreAllZC     : False
(INFO) ArmoryUtils.py:1311 -     rescanBalance   : False
(INFO) ArmoryUtils.py:1311 -     disableModules  : False
(INFO) ArmoryUtils.py:1311 -     port            : None
(INFO) ArmoryUtils.py:1311 -     interport       : 8223
(INFO) ArmoryUtils.py:1311 -     coverageOutputDir: None
(INFO) ArmoryUtils.py:1311 -     forceWalletCheck: False
(INFO) ArmoryUtils.py:1311 -     regtest         : False
(INFO) ArmoryUtils.py:1311 -     rebuild         : False
(INFO) ArmoryUtils.py:1311 -     thread_count    : -1
(INFO) ArmoryUtils.py:1311 -     datadir         : DEFAULT
(INFO) ArmoryUtils.py:1311 -     clearMempool    : False
(INFO) ArmoryUtils.py:1311 -     offline         : False
(INFO) ArmoryUtils.py:1311 -     armoryDBDir     : DEFAULT
(INFO) ArmoryUtils.py:1311 -     armorydb_port   : 9002
(INFO) ArmoryUtils.py:1311 -     satoshiPort     : DEFAULT
(INFO) ArmoryUtils.py:1311 -     disableConfPermis: False
(INFO) ArmoryUtils.py:1311 -     useTorSettings  : False
(INFO) ArmoryUtils.py:1311 -     netlog          : False
(INFO) ArmoryUtils.py:1311 -     keypool         : 100
(INFO) ArmoryUtils.py:1311 -     coverageInclude : None
(INFO) ArmoryUtils.py:1311 -     forceOnline     : False
(INFO) ArmoryUtils.py:1311 -     disableWalletCheck: False
(INFO) ArmoryUtils.py:1311 -     redownload      : False
(INFO) ArmoryUtils.py:1311 -     rpcBindAddr     : 127.0.0.1
(INFO) ArmoryUtils.py:1311 -     armorydb_ip     : 192.168.0.1
(INFO) ArmoryUtils.py:1311 -     multisigFile    : DEFAULT
(INFO) ArmoryUtils.py:1311 -     ram_usage       : -1
(INFO) ArmoryUtils.py:1311 -     ignoreZC        : False
(INFO) ArmoryUtils.py:1311 -     logDisable      : False
(INFO) ArmoryUtils.py:1311 -     settingsPath    : /home/user/.armory/ArmorySettings.txt
(INFO) ArmoryUtils.py:1311 -     language        : en
(INFO) ArmoryUtils.py:1311 -     db_type         : DB_FULL
(INFO) ArmoryUtils.py:1311 -     doDebug         : True
(INFO) ArmoryUtils.py:1311 -     enableDetSign   : True
(INFO) ArmoryUtils.py:1311 -     mtdebug         : False
(INFO) ArmoryUtils.py:1311 -     testnet         : False
(INFO) ArmoryUtils.py:1311 -     rpcport         : DEFAULT
(INFO) ArmoryUtils.py:1311 -     satoshiHome     : DEFAULT
(INFO) ArmoryUtils.py:1311 -     satoshiRpcport  : DEFAULT
(INFO) ArmoryUtils.py:1311 -     logFile         : /home/user/.armory/ArmoryQt.py.log.txt
(INFO) ArmoryUtils.py:1311 -     verbosity       : None
(INFO) ArmoryUtils.py:1312 - Other arguments:
(INFO) ArmoryUtils.py:1315 - ************************************************************
(INFO) ArmoryUtils.py:1720 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(ERROR) ArmoryUtils.py:3735 - Unsupported language  specified. Defaulting to English (en)
(INFO) ArmoryUtils.py:3738 - Using Language: en
(INFO) BDM.py:353 - Using the asynchronous/multi-threaded BlockDataManager.
(INFO) BDM.py:354 - Blockchain operations will happen in the background.  
(INFO) BDM.py:355 - Devs: check TheBDM.getState() before asking for data.
(INFO) BDM.py:356 - Registering addresses during rescans will queue them for 
(INFO) BDM.py:357 - inclusion after the current scan is completed.
/opt/BitcoinArmory/armoryengine/Transaction.py:3126: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(DEBUG) ArmoryQt.py:5966 - Checking for already open socket...
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(INFO) ArmoryQt.py:2060 - loadWalletsAndSettings
(INFO) ArmoryQt.py:2120 - Loading wallets...
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/databases is a directory.
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/wallets is a directory.
(INFO) ArmoryQt.py:2185 - Number of wallets read in: 0
(INFO) ArmoryQt.py:2195 - Loading Multisig Lockboxes
(INFO) ArmoryQt.py:1763 - acquiring process mutex...
(DEBUG) ArmoryQt.py:1286 - setupSystemTray
(INFO) ArmoryQt.py:1395 - setupUriRegistration
(INFO) ArmoryUtils.py:689 - Executing popen: gconftool-2 --get /desktop/gnome/url-handlers/bitcoin/command
(INFO) ArmoryUtils.py:689 - Executing popen: xdg-mime query default x-scheme-handler/bitcoin
(DEBUG) ArmoryQt.py:3742 - setupDashboard
(INFO) ArmoryQt.py:566 - Usermode: Expert
(INFO) ArmoryQt.py:1698 - Changing usermode:
(INFO) ArmoryQt.py:1699 -    From: Expert
(INFO) ArmoryQt.py:1707 -      To: Expert
(INFO) ArmoryQt.py:1836 - startBitcoindIfNecessary
(WARNING) ArmoryQt.py:1826 - DB is already running
(INFO) ArmoryQt.py:1923 - Setting netmode: 1
(INFO) ArmoryQt.py:1905 - loadBlockchainIfNecessary
(INFO) ArmoryQt.py:1923 - Setting netmode: 1
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
BDM is ready!
BDM is ready!
(DEBUG) ArmoryQt.py:4923 - Running finishLoadBlockchain

Here it hang a while at the splashscreen, then closed itself:

Code:

(INFO) ArmoryUtils.py:1164 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:689 - Executing popen: free -m
(INFO) ArmoryUtils.py:689 - Executing popen: ['cat', '/proc/cpuinfo']
(ERROR) ArmoryUtils.py:1262 - Error getting system details:
Traceback (most recent call last):
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1260, in <module>
    SystemSpecs = GetSystemDetails()
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1255, in GetSystemDetails
    out.HddAvailB = getHddSize(BTC_HOME_DIR)    / (1024**3)
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1252, in getHddSize
    s = os.statvfs(adir)
OSError: [Errno 2] No such file or directory: '/home/user/.bitcoin'
(ERROR) ArmoryUtils.py:1263 - Skipping.
(INFO) ArmoryUtils.py:1281 - 
(INFO) ArmoryUtils.py:1282 - 
(INFO) ArmoryUtils.py:1283 - 
(INFO) ArmoryUtils.py:1284 - ************************************************************
<info deleted, see first full log>
(INFO) ArmoryUtils.py:1315 - ************************************************************
(INFO) ArmoryUtils.py:1720 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(ERROR) ArmoryUtils.py:3735 - Unsupported language  specified. Defaulting to English (en)
(INFO) ArmoryUtils.py:3738 - Using Language: en
(INFO) BDM.py:353 - Using the asynchronous/multi-threaded BlockDataManager.
(INFO) BDM.py:354 - Blockchain operations will happen in the background.  
(INFO) BDM.py:355 - Devs: check TheBDM.getState() before asking for data.
(INFO) BDM.py:356 - Registering addresses during rescans will queue them for 
(INFO) BDM.py:357 - inclusion after the current scan is completed.
/opt/BitcoinArmory/armoryengine/Transaction.py:3126: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(DEBUG) ArmoryQt.py:5966 - Checking for already open socket...
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(INFO) ArmoryQt.py:2060 - loadWalletsAndSettings
(INFO) ArmoryQt.py:2120 - Loading wallets...
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/databases is a directory.
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/wallets is a directory.
(INFO) ArmoryQt.py:2185 - Number of wallets read in: 0
(INFO) ArmoryQt.py:2195 - Loading Multisig Lockboxes
(INFO) ArmoryQt.py:1763 - acquiring process mutex...
(DEBUG) ArmoryQt.py:1286 - setupSystemTray
(INFO) ArmoryQt.py:1395 - setupUriRegistration
(INFO) ArmoryUtils.py:689 - Executing popen: gconftool-2 --get /desktop/gnome/url-handlers/bitcoin/command
(INFO) ArmoryUtils.py:689 - Executing popen: xdg-mime query default x-scheme-handler/bitcoin
(DEBUG) ArmoryQt.py:3742 - setupDashboard
(INFO) ArmoryQt.py:566 - Usermode: Expert
(INFO) ArmoryQt.py:1698 - Changing usermode:
(INFO) ArmoryQt.py:1699 -    From: Expert
(INFO) ArmoryQt.py:1707 -      To: Expert
(INFO) ArmoryQt.py:1836 - startBitcoindIfNecessary
(WARNING) ArmoryQt.py:1826 - DB is already running
(INFO) ArmoryQt.py:1923 - Setting netmode: 1

<here it hang a while at the splashscreen>

(ERROR) Traceback (most recent call last):
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5999, in <module>
    form = ArmoryMainWindow(splashScreen=SPLASH)
  File "/opt/BitcoinArmory/ArmoryQt.py", line 685, in __init__
    self.startBlockchainProcessingInitialization()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5761, in startBlockchainProcessingInitialization
    self.completeBlockchainProcessingInitialization()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5785, in completeBlockchainProcessingInitialization
    self.setupLedgerViews()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5834, in setupLedgerViews
    self.lockboxLedgModel.setLedgerDelegate(TheBDM.bdv().getLedgerDelegateForLockboxes())
  File "/opt/BitcoinArmory/CppBlockUtils.py", line 2797, in getLedgerDelegateForLockboxes
    return _CppBlockUtils.BlockDataViewer_getLedgerDelegateForLockboxes(self)
RuntimeError: failed to connect socket

Traceback (most recent call last):
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5999, in <module>
    form = ArmoryMainWindow(splashScreen=SPLASH)
  File "/opt/BitcoinArmory/ArmoryQt.py", line 685, in __init__
    self.startBlockchainProcessingInitialization()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5761, in startBlockchainProcessingInitialization
    self.completeBlockchainProcessingInitialization()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5785, in completeBlockchainProcessingInitialization
    self.setupLedgerViews()
  File "/opt/BitcoinArmory/ArmoryQt.py", line 5834, in setupLedgerViews
    self.lockboxLedgModel.setLedgerDelegate(TheBDM.bdv().getLedgerDelegateForLockboxes())
  File "/opt/BitcoinArmory/CppBlockUtils.py", line 2797, in getLedgerDelegateForLockboxes
    return _CppBlockUtils.BlockDataViewer_getLedgerDelegateForLockboxes(self)
RuntimeError: failed to connect socket

and finally, here the Armory GUI opened, but stayed offline. please notice the last line, where the GUI got info about a new block too:

Code:

(INFO) ArmoryUtils.py:1164 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:689 - Executing popen: free -m
(INFO) ArmoryUtils.py:689 - Executing popen: ['cat', '/proc/cpuinfo']
(ERROR) ArmoryUtils.py:1262 - Error getting system details:
Traceback (most recent call last):
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1260, in <module>
    SystemSpecs = GetSystemDetails()
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1255, in GetSystemDetails
    out.HddAvailB = getHddSize(BTC_HOME_DIR)    / (1024**3)
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 1252, in getHddSize
    s = os.statvfs(adir)
OSError: [Errno 2] No such file or directory: '/home/user/.bitcoin'
(ERROR) ArmoryUtils.py:1263 - Skipping.
(INFO) ArmoryUtils.py:1281 - 
(INFO) ArmoryUtils.py:1282 - 
(INFO) ArmoryUtils.py:1283 - 
(INFO) ArmoryUtils.py:1284 - ************************************************************
<info deleted, see first full log>
(INFO) ArmoryUtils.py:1315 - ************************************************************
(INFO) ArmoryUtils.py:1720 - C++ block utilities loaded successfully
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(ERROR) ArmoryUtils.py:3735 - Unsupported language  specified. Defaulting to English (en)
(INFO) ArmoryUtils.py:3738 - Using Language: en
(INFO) BDM.py:353 - Using the asynchronous/multi-threaded BlockDataManager.
(INFO) BDM.py:354 - Blockchain operations will happen in the background.  
(INFO) BDM.py:355 - Devs: check TheBDM.getState() before asking for data.
(INFO) BDM.py:356 - Registering addresses during rescans will queue them for 
(INFO) BDM.py:357 - inclusion after the current scan is completed.
/opt/BitcoinArmory/armoryengine/Transaction.py:3126: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(DEBUG) ArmoryQt.py:5966 - Checking for already open socket...
(INFO) ArmoryUtils.py:3578 - Using settings file: /home/user/.armory/ArmorySettings.txt
(INFO) ArmoryQt.py:2060 - loadWalletsAndSettings
(INFO) ArmoryQt.py:2120 - Loading wallets...
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/databases is a directory.
(DEBUG) ArmoryUtils.py:1134 - /home/user/.armory/wallets is a directory.
(INFO) ArmoryQt.py:2185 - Number of wallets read in: 0
(INFO) ArmoryQt.py:2195 - Loading Multisig Lockboxes
(INFO) ArmoryQt.py:1763 - acquiring process mutex...
(DEBUG) ArmoryQt.py:1286 - setupSystemTray
(INFO) ArmoryQt.py:1395 - setupUriRegistration
(INFO) ArmoryUtils.py:689 - Executing popen: gconftool-2 --get /desktop/gnome/url-handlers/bitcoin/command
(INFO) ArmoryUtils.py:689 - Executing popen: xdg-mime query default x-scheme-handler/bitcoin
(DEBUG) ArmoryQt.py:3742 - setupDashboard
(INFO) ArmoryQt.py:566 - Usermode: Expert
(INFO) ArmoryQt.py:1698 - Changing usermode:
(INFO) ArmoryQt.py:1699 -    From: Expert
(INFO) ArmoryQt.py:1707 -      To: Expert
(INFO) ArmoryQt.py:1836 - startBitcoindIfNecessary
(WARNING) ArmoryQt.py:1826 - DB is already running
(INFO) ArmoryQt.py:1923 - Setting netmode: 1
(INFO) ArmoryQt.py:1905 - loadBlockchainIfNecessary
(INFO) ArmoryQt.py:1923 - Setting netmode: 1
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
BDM is ready!
BDM is ready!
(ERROR) Traceback (most recent call last):
  File "/opt/BitcoinArmory/ui/QtExecuteSignal.py", line 31, in methodSlot
    _callable(*args)
  File "/opt/BitcoinArmory/ArmoryQt.py", line 4909, in handleCppNotification
    self.nodeStatus = TheBDM.bdv().getNodeStatus()
  File "/opt/BitcoinArmory/CppBlockUtils.py", line 2836, in getNodeStatus
    return _CppBlockUtils.BlockDataViewer_getNodeStatus(self)
RuntimeError: failed to connect socket

Traceback (most recent call last):
  File "/opt/BitcoinArmory/ui/QtExecuteSignal.py", line 31, in methodSlot
    _callable(*args)
  File "/opt/BitcoinArmory/ArmoryQt.py", line 4909, in handleCppNotification
    self.nodeStatus = TheBDM.bdv().getNodeStatus()
  File "/opt/BitcoinArmory/CppBlockUtils.py", line 2836, in getNodeStatus
    return _CppBlockUtils.BlockDataViewer_getNodeStatus(self)
RuntimeError: failed to connect socket
^CError in sys.excepthook:
Traceback (most recent call last):
  File "/opt/BitcoinArmory/armoryengine/ArmoryUtils.py", line 992, in logexcept_override
    def logexcept_override(_type, value, tback):
KeyboardInterrupt

Original exception was:
Traceback (most recent call last):
  File "/opt/BitcoinArmory/ui/QtExecuteSignal.py", line 31, in methodSlot
    _callable(*args)
  File "/opt/BitcoinArmory/ArmoryQt.py", line 4909, in handleCppNotification
    self.nodeStatus = TheBDM.bdv().getNodeStatus()
  File "/opt/BitcoinArmory/CppBlockUtils.py", line 2836, in getNodeStatus
    return _CppBlockUtils.BlockDataViewer_getNodeStatus(self)
RuntimeError: failed to connect socket

New Block:  846880

My big question is: if "failed to connect socket" is the fundamental problem, what "socket" is that?
If not the client connection to nginx nor the nginx connection to ArmoryDB?

Thank you for any hints!

Hattori

[goatpig]

My big question is: if "failed to connect socket" is the fundamental problem, what "socket" is that?
If not the client connection to nginx nor the nginx connection to ArmoryDB?

This means ArmoryQt is failing to connect to nginx. It's hard to know what's the cause right now. I would try connecting an ArmoryQt instance locally on that machine, or via a SSH tunnel/VPN (anything to bypass nginx, so as to confirm it as the culprit or rule it out).

Also check RAM usage and CPU load on the core/db machine, both idle and while connecting ArmoryQt.

[h.hanzo]

Thank you for looking into this!

This means ArmoryQt is failing to connect to nginx. It's hard to know what's the cause right now. I would try connecting an ArmoryQt instance locally on that machine, or via a SSH tunnel/VPN (anything to bypass nginx, so as to confirm it as the culprit or rule it out).

I'll investigate in this direction then.
What exactly would

Code:

-INFO  - 18:01:02: (BDM_Server.cpp:1121) registered bdv: f29e1bd6ecdf10d1b8c0

in dbLog.txt mean? I get this log entry every time I start ArmoryQt.

Also check RAM usage and CPU load on the core/db machine, both idle and while connecting ArmoryQt.

I checked RAM, CPU and IO on the core/ArmoryDB machine again.
When starting and connecting ArmoryQt, on the core/ArmoryDB machine bitcoind rises to around 2% CPU, nothing else with significant CPU. No observable HDD IO. at all.
ArmoryDB is at around 11GB VIRT memory and 6GB RES memory. System overall is 32GB total, 7GB used, 25GB buff/cache. No swapping.


On another start of ArmoryQt I got

Code:

[..]
(INFO) ArmoryQt.py:1923 - Setting netmode: 1
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
(INFO) ArmoryQt.py:4672 - Dashboard switched to "Scanning" mode
BDM is ready!
Segmentation fault

Never observed that before.
No other info in the shell, and nothing in ArmoryDBs dbLog.txt


Are there more ways to get debug info besides ArmoryDBs "dbLog.txt" and ArmoryQts "ArmoryQt.py --debug" ?

I'll try to check (or circumvent) the connection to nginx next.

edit:
syslog of ArmoryQt has

Code:

[Fri Jun  7 18:13:21 2024] python[26446]: segfault at 20 ip 00007625d551ebec sp 00007ffdb943cd60 error 4 likely on CPU 0 (core 0, socket 0)
[Fri Jun  7 18:13:21 2024] Code: 48 b8 ff ff ff ff ff ff ff 07 48 c7 c7 e0 ff ff ff 48 89 04 24 48 89 54 24 08 e8 ef e6 f1 ff 48 8b 54 24 08 49 89 c5 4d 01 ef <48> 8b 32 49 8d 47 10 4c 89 ff 48 89 44 24 08 49 89 07 48 8b 42 08

[h.hanzo]

This means ArmoryQt is failing to connect to nginx. It's hard to know what's the cause right now. I would try connecting an ArmoryQt instance locally on that machine, or via a SSH tunnel/VPN (anything to bypass nginx, so as to confirm it as the culprit or rule it out).

You were spot on!
I checked on the ArmoryDB machine, and even though I always get a

Code:

-INFO  - 18:01:02: (BDM_Server.cpp:1121) registered bdv: f29e1bd6ecdf10d1b8c0

entry in dbLog.txt, when I checked actual tcp connections with netstat, only some of the times there was any connection established between ArmoryQt and nginx.

I then opened a ssh tunnel between the two machines, tunneling port 9001. Then connecting ArmoryQt to 127.0.0.1:9001.
And sure enough, immediately everything worked just as it should, every time I start ArmoryQt!
Thank you so much, again, goatpig! 

So, what exactly does it mean when there's those entries in dbLog.txt?

I'll scrap nginx alltogether, and will use ssh tunneling from now on.

Cheers!

H.

[goatpig]

So, what exactly does it mean when there's those entries in dbLog.txt?

0.96.5 uses the FCGI protocol, think PHP but persistent. In PHP, each request to the HTTP deamon (nginx in your case) spawn a dedicated PHP process that handles the request. In the persistent approach, the process is spawned once and the daemon routes the traffic to it. This how ArmoryDB works in that version (stuff in dev branch uses WS instead, much better).

The CGI model is for REST like designs: client sends a request, server replies. There is no built in feature for push (server pushing unsolicited data to clients), so the way notifications are done is through what's called long polling. The client sends a long lived request to the server, which will reply when it has data to push, and that request will be cycled once in a while (once a minute with ArmoryDB).

The registration message you see is for the long poll request. It signals that the server has seen a client and started the long polling cycle.

What is failing in your setup seems to be the individual requests. My guess is that it's asking for packets that are too big for nginx to let them through? Either your wallets grew enough that the ledger dump at startup exceeds nginx's packet size limit, or maybe some default nginx setting got updated along the way? You said you tried with a fresh db and no wallets loaded so if the issue is indeed packet size, I'd check for those defaults.

I've come to this conclusion because traffic works via SSH and not nginx, but at the same time the long poll lives through nginx until the client crashes (longpoll traffic is very light, we talking 100~200 bytes per instance). Client crash could be explained by receiving a mangled packet as well.

You could try to verify my assumption by fidgeting with nginx's settings? This mostly to satisfy your curiosity, I think SSH is better for your use case.