DB errors in Armory prevent sending bitcoin

[zzzrob]

I have an armory instance running with bitcoin core 0.20.0 and armory 0.96.5.  It loads and I can see a balance in my wallet, but I cannot send any bitcoin from it.  Clicking the "Send Bitcoins" button results nothing happening in the GUI and the following errors on the console:

Code:

(ERROR) Traceback (most recent call last):
  File "/usr/local/bin/../lib/armory/ArmoryQt.py", line 3451, in clickSendBitcoins
    DlgSendBitcoins(self.getSelectedWallet(), self, self).exec_()
  File "/usr/local/lib/armory/qtdialogs.py", line 4594, in __init__
    spendFromLockboxID=spendFromLockboxID)
  File "/usr/local/lib/armory/ui/TxFrames.py", line 151, in __init__
    RBFcallback=RBFcallback)
  File "/usr/local/lib/armory/ui/WalletFrames.py", line 232, in __init__
    self.updateOnWalletChange()
  File "/usr/local/lib/armory/ui/WalletFrames.py", line 347, in updateOnWalletChange
    self.selectWltCallback(wlt)
  File "/usr/local/lib/armory/ui/TxFrames.py", line 347, in setWallet
    self.setupCoinSelectionInstance()
  File "/usr/local/lib/armory/ui/TxFrames.py", line 396, in setupCoinSelectionInstance
    self.coinSelection = self.wlt.cppWallet.getCoinSelectionInstance()
  File "/usr/local/lib/armory/CppBlockUtils.py", line 4163, in getCoinSelectionInstance
    return _CppBlockUtils.WalletContainer_getCoinSelectionInstance(self)
DbErrorMsg: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f286bca9510> >

Traceback (most recent call last):
  File "/usr/local/bin/../lib/armory/ArmoryQt.py", line 3451, in clickSendBitcoins
    DlgSendBitcoins(self.getSelectedWallet(), self, self).exec_()
  File "/usr/local/lib/armory/qtdialogs.py", line 4594, in __init__
    spendFromLockboxID=spendFromLockboxID)
  File "/usr/local/lib/armory/ui/TxFrames.py", line 151, in __init__
    RBFcallback=RBFcallback)
  File "/usr/local/lib/armory/ui/WalletFrames.py", line 232, in __init__
    self.updateOnWalletChange()
  File "/usr/local/lib/armory/ui/WalletFrames.py", line 347, in updateOnWalletChange
    self.selectWltCallback(wlt)
  File "/usr/local/lib/armory/ui/TxFrames.py", line 347, in setWallet
    self.setupCoinSelectionInstance()
  File "/usr/local/lib/armory/ui/TxFrames.py", line 396, in setupCoinSelectionInstance
    self.coinSelection = self.wlt.cppWallet.getCoinSelectionInstance()
  File "/usr/local/lib/armory/CppBlockUtils.py", line 4163, in getCoinSelectionInstance
    return _CppBlockUtils.WalletContainer_getCoinSelectionInstance(self)
<class 'CppBlockUtils.DbErrorMsg'>: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f286bca9510> >

The Armory dbLog has the following error (several times):

Code:

-WARN  - 16:05:46: (LedgerEntry.cpp:334) failed to get tx for ledger parsing
-ERROR - 16:05:46: (lmdb_wrapper.cpp:2086) BLKDATA DB does not have the requested ZC tx
-ERROR - 16:05:46: (lmdb_wrapper.cpp:2087) (0a22780009d6)

Do I need to upgrade bitcoin core to a newer version?
Rebuild my blockchain db?  Is this the pruning issue I've seen mentioned?
Anything else I should try?

[1714295507]

1714295507

[1714295507]

1714295507

[1714295507]

1714295507

[1714295507]

1714295507

[goatpig]

Do I need to upgrade bitcoin core to a newer version? Is this the pruning issue I've seen mentioned?

You can update Core but make sure to disable blockchain data pruning or it will erase historical block data from your disk, which Armory requires.

Rebuild my blockchain db? Anything else I should try?

Hard to say without seeing the full log file. From what you're showing, looks like some db data is mangled. I suggest a rescan, not a rebuild (yet).

[zzzrob]

Thanks!  I've been running a rescan since you posted, but I'm not sure if it's stuck.  The progress bar is at 100%, and the spinner next to it isn't moving.  CPU usage is at 100% for the armory python process though.  I'm not sure if I should be seeing any indication of progress.  Should I wait it out some more, or is it time to try something else?

The entirety of the dbLog is this (note - it's running as root in a docker container):

Code:

-INFO  - 14:47:09: (main.cpp:32) Running on 2 threads
-INFO  - 14:47:09: (main.cpp:33) Ram usage level: 50
-INFO  - 14:47:09: (BlockUtils.cpp:915) blkfile dir: /root/.bitcoin/blocks
-INFO  - 14:47:09: (BlockUtils.cpp:916) lmdb dir: /root/.armory/databases
-INFO  - 14:47:09: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:47:09: (BDM_Server.h:263) Listening on port 52933
-INFO  - 14:47:09: (BitcoinP2P.cpp:947) Connected to Bitcoin node
-INFO  - 14:47:09: (nodeRPC.cpp:57) RPC connection established
-INFO  - 14:47:09: (BlockDataManagerConfig.cpp:919) waiting on node sync: 99.9996%
-INFO  - 14:47:09: (nodeRPC.cpp:425) Node is ready
-INFO  - 14:47:09: (BlockUtils.cpp:1117) Executing: doInitialSyncOnLoad_Rescan
-INFO  - 14:47:09: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:47:09: (BDM_Server.cpp:1121) registered bdv: 9db1f452bb4f731cc2af
-INFO  - 14:47:10: (DatabaseBuilder.cpp:199) Reading headers from db
-INFO  - 14:48:24: (DatabaseBuilder.cpp:238) Found 826164 headers in db
-INFO  - 14:48:30: (DatabaseBuilder.cpp:64) Rewinding 100 blocks
-INFO  - 14:48:30: (DatabaseBuilder.cpp:71) updating HEADERS db
-INFO  - 14:48:30: (DatabaseBuilder.cpp:493) Found next block after skipping 1689752bytes
-INFO  - 14:48:30: (DatabaseBuilder.cpp:281) parsed block file #4056
-INFO  - 14:48:34: (DatabaseBuilder.cpp:281) parsed block file #4058
-INFO  - 14:48:42: (Blockchain.cpp:248) Organizing chain 
-INFO  - 14:48:42: (Blockchain.cpp:370) Organized chain in 0s
-INFO  - 14:48:42: (DatabaseBuilder.cpp:76) updated HEADERS db in 12s
-INFO  - 14:48:43: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:48:43: (DatabaseBuilder.cpp:1231) verifying txfilters integrity
-WARN  - 14:48:44: (DatabaseBuilder.cpp:1292) 14 mismatches in txfilter for file #11
-WARN  - 14:49:34: (DatabaseBuilder.cpp:1318) 1 damaged filters, repairing
-INFO  - 14:49:34: (DatabaseBuilder.cpp:1326) clearing damaged filters
-INFO  - 14:49:37: (DatabaseBuilder.cpp:1456) fixed txfilter for file #11
-INFO  - 14:49:37: (DatabaseBuilder.cpp:134) scanning new blocks from #-1 to #826135

[goatpig]

Are you trying to dockerize it because you're on a Windows host or is there another reason? Depending on what you're trying to achieve, I can help you with the docker file or the setup as a whole. If you're armed with patience, you could help me make the dev branch user friendly (at least the build & launch steps). That stuff works (I've been using it for my own stuff for over a year now), I just cant get around to make it user friendly. At any rate, a few notes:

1. ArmoryDB tracks the addresses in your wallets. If it sees them once, it will know then until you wipe/rebuild the "databases" folder (what the --dbdir argument points at). Since your db knows your addresses (it wouldn't try to scan otherwise), you can run "ArmoryDB --rescan" on its own (without either ArmoryQt or Core running) to get better verbose.

2. Dockerizing ArmoryDB isn't really useful imo. This is a headless C++ process, it doesn't have any linkage with Qt nor any other GUI library, it's the easiest part of the puzzle to build & run on any host OS. It needs read access to blockchain data and ~10GB for itself persisted across runs, so you will have to mount part of your host file system in there in all likelihood. It also mmap's the blockchain data, I don't know how docker/containerd/k8s deals with that. On the other hand, it talks to ArmoryQt via a socket, so you can run ArmoryDB and Core on your host, and containerize the client.

3. Containerizing the client is your best return on investment, this is where all the exotic requirements come from. I would run as a user, setup a python venv and install the dependencies via pip off of a lightweight image. This could probably work on alpine. I expect the main hurdle is to expose something that looks like a X server to the image.

[zzzrob]

I started running it in docker just because I had a bunch of issues getting it to build on Ubuntu 22.04 and it provided a good way to get an environment where could build everything and have it work correctly.  The docker container has all of Armory and Bitcoin in it, so I'm not trying to containerize the individual components, just manage the environment that way.

I tried to run ArmoryDB --rescan by itself and it looks like the actual ArmoryDB process is just crashing shortly after starting the rescan.  I'm going to try doing a rebuild instead.

Code:

-INFO  - 14:29:41: (main.cpp:32) Running on 2 threads
-INFO  - 14:29:41: (main.cpp:33) Ram usage level: 50
-INFO  - 14:29:41: (BlockUtils.cpp:915) blkfile dir: /root/.bitcoin/blocks
-INFO  - 14:29:41: (BlockUtils.cpp:916) lmdb dir: /root/.armory/databases
-INFO  - 14:29:41: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:29:41: (BDM_Server.h:263) Listening on port 9001
-INFO  - 14:29:41: (BlockUtils.cpp:1117) Executing: doInitialSyncOnLoad_Rescan
-INFO  - 14:29:41: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:29:41: (DatabaseBuilder.cpp:199) Reading headers from db
-INFO  - 14:32:05: (DatabaseBuilder.cpp:238) Found 826164 headers in db
-INFO  - 14:32:11: (DatabaseBuilder.cpp:64) Rewinding 100 blocks
-INFO  - 14:32:11: (DatabaseBuilder.cpp:71) updating HEADERS db
-INFO  - 14:32:12: (DatabaseBuilder.cpp:493) Found next block after skipping 1689752bytes
-INFO  - 14:32:12: (DatabaseBuilder.cpp:281) parsed block file #4056
-INFO  - 14:32:30: (DatabaseBuilder.cpp:281) parsed block file #4057
-INFO  - 14:32:34: (DatabaseBuilder.cpp:281) parsed block file #4058
-INFO  - 14:32:42: (DatabaseBuilder.cpp:281) parsed block file #4059
-INFO  - 14:32:42: (Blockchain.cpp:248) Organizing chain 
-INFO  - 14:32:43: (Blockchain.cpp:370) Organized chain in 0s
-INFO  - 14:32:43: (DatabaseBuilder.cpp:76) updated HEADERS db in 32s
-INFO  - 14:32:43: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:32:43: (DatabaseBuilder.cpp:1231) verifying txfilters integrity
-WARN  - 14:32:46: (DatabaseBuilder.cpp:1292) 14 mismatches in txfilter for file #11
-WARN  - 14:38:24: (DatabaseBuilder.cpp:1318) 1 damaged filters, repairing
-INFO  - 14:38:24: (DatabaseBuilder.cpp:1326) clearing damaged filters
-INFO  - 14:38:28: (DatabaseBuilder.cpp:1456) fixed txfilter for file #11
-INFO  - 14:38:28: (DatabaseBuilder.cpp:134) scanning new blocks from #-1 to #826267
terminate called recursively
Aborted (core dumped)

[goatpig]

I'm going to try doing a rebuild instead.

You're better off delete Armory's databases folder in this case (/root/.armory/databases)

I'd suggest you run as user instead of root too. You can add the following to your dockerfile:

Code:

#add the user while in root
ARG USERNAME=armory
RUN useradd -ms /bin/bash $USERNAME

#switch to user
USER $USERNAME

#run build steps here

[zzzrob]

OK, I've tried deleting the database folder and restarting everything.  It goes well until it get a "Missing file map for output scan, this is unexpected" error.  I also updated my Dockerfile to run everything as a regular user.

I did find this thread https://bitcointalk.org/index.php?topic=3055574.0 referring to that error back in 2018.  I'm not sure whether it's still applicable.  My bitcoin node is definitely online in this case and is keeping in sync.

Not sure if "ArmoryDB --db-type=DB_BARE --ram-usage=1" as mentioned in that thread would be helpful here either.

Abridged dbLog.txt:

Code:

INFO  - 02:02:56: (main.cpp:32) Running on 2 threads
-INFO  - 02:02:56: (main.cpp:33) Ram usage level: 50
-INFO  - 02:02:56: (BlockUtils.cpp:915) blkfile dir: /home/armory/.bitcoin/blocks
-INFO  - 02:02:56: (BlockUtils.cpp:916) lmdb dir: /home/armory/.armory/databases
-INFO  - 02:02:56: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 02:02:56: (BDM_Server.h:263) Listening on port 51776
-INFO  - 02:02:56: (BitcoinP2P.cpp:947) Connected to Bitcoin node
-INFO  - 02:02:56: (BDM_Server.cpp:1121) registered bdv: 05ff3a35925b8aea2457
-INFO  - 02:02:56: (nodeRPC.cpp:57) RPC connection established
-INFO  - 02:02:56: (BlockDataManagerConfig.cpp:919) waiting on node sync: 99.9998%
-INFO  - 02:02:56: (nodeRPC.cpp:425) Node is ready
-INFO  - 02:02:56: (BlockUtils.cpp:1108) Executing: doInitialSyncOnLoad
-INFO  - 02:02:57: (DatabaseBuilder.cpp:199) Reading headers from db
-WARN  - 02:02:57: (lmdb_wrapper.cpp:1241) No headers in DB yet!
-INFO  - 02:02:57: (DatabaseBuilder.cpp:238) Found 1 headers in db
-INFO  - 02:02:57: (DatabaseBuilder.cpp:71) updating HEADERS db
-INFO  - 02:03:03: (DatabaseBuilder.cpp:281) parsed block file #1
-INFO  - 02:03:08: (DatabaseBuilder.cpp:281) parsed block file #3
-INFO  - 02:03:13: (DatabaseBuilder.cpp:281) parsed block file #5
-INFO  - 02:03:21: (DatabaseBuilder.cpp:281) parsed block file #7
-INFO  - 02:03:27: (DatabaseBuilder.cpp:281) parsed block file #9
-INFO  - 02:03:31: (DatabaseBuilder.cpp:281) parsed block file #11
-INFO  - 02:03:37: (DatabaseBuilder.cpp:281) parsed block file #13
-INFO  - 02:03:41: (DatabaseBuilder.cpp:281) parsed block file #15
-INFO  - 02:03:47: (DatabaseBuilder.cpp:281) parsed block file #17
-INFO  - 02:03:53: (DatabaseBuilder.cpp:281) parsed block file #18
-INFO  - 02:03:54: (DatabaseBuilder.cpp:281) parsed block file #19
-SNIP-
-INFO  - 06:51:56: (DatabaseBuilder.cpp:281) parsed block file #4059
-INFO  - 06:52:04: (DatabaseBuilder.cpp:281) parsed block file #4061
-INFO  - 06:52:10: (DatabaseBuilder.cpp:281) parsed block file #4063
-INFO  - 06:52:16: (DatabaseBuilder.cpp:281) parsed block file #4065
-INFO  - 06:52:26: (DatabaseBuilder.cpp:281) parsed block file #4067
-INFO  - 06:52:37: (DatabaseBuilder.cpp:281) parsed block file #4069
-INFO  - 06:52:45: (DatabaseBuilder.cpp:281) parsed block file #4071
-INFO  - 06:52:56: (DatabaseBuilder.cpp:281) parsed block file #4073
-INFO  - 06:53:12: (DatabaseBuilder.cpp:281) parsed block file #4075
-INFO  - 06:53:18: (DatabaseBuilder.cpp:281) parsed block file #4077
-INFO  - 06:54:19: (DatabaseBuilder.cpp:281) parsed block file #4078
-INFO  - 06:54:19: (Blockchain.cpp:248) Organizing chain 
-INFO  - 06:54:24: (Blockchain.cpp:370) Organized chain in 5s
-INFO  - 06:54:35: (DatabaseBuilder.cpp:76) updated HEADERS db in 17498s
-INFO  - 06:54:35: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 06:54:35: (DatabaseBuilder.cpp:1231) verifying txfilters integrity
-INFO  - 06:57:55: (DatabaseBuilder.cpp:1314) done checking txfilters
-INFO  - 06:57:55: (BDM_supportClasses.cpp:1891) Enabling zero-conf tracking
-INFO  - 06:57:56: (BDM_supportClasses.cpp:401) Starting address registration process
-INFO  - 06:58:19: (BlockchainScanner.cpp:852) scanned from block #0 to #142691
-INFO  - 06:58:35: (BlockchainScanner.cpp:852) scanned from block #142692 to #170004
-INFO  - 06:58:54: (BlockchainScanner.cpp:852) scanned from block #170005 to #183110
-INFO  - 06:59:09: (BlockchainScanner.cpp:852) scanned from block #183111 to #189230
-INFO  - 06:59:32: (BlockchainScanner.cpp:852) scanned from block #189231 to #194895
-INFO  - 06:59:52: (BlockchainScanner.cpp:852) scanned from block #194896 to #200879
-INFO  - 07:00:18: (BlockchainScanner.cpp:852) scanned from block #200880 to #206654
-SNIP (there are some more "no history to scan" messages in here, but that's it)-
-INFO  - 10:32:35: (BlockchainScanner.cpp:852) scanned from block #677026 to #677428
-INFO  - 10:32:59: (BlockchainScanner.cpp:852) scanned from block #677429 to #677825
-INFO  - 10:33:20: (BlockchainScanner.cpp:852) scanned from block #677826 to #678227
-INFO  - 10:33:45: (BlockchainScanner.cpp:852) scanned from block #678228 to #678634
-INFO  - 10:34:11: (BlockchainScanner.cpp:852) scanned from block #678635 to #679027
-INFO  - 10:34:29: (BlockchainScanner.cpp:852) scanned from block #679028 to #679437
-INFO  - 10:34:53: (BlockchainScanner.cpp:852) scanned from block #679438 to #679845
-INFO  - 10:35:17: (BlockchainScanner.cpp:852) scanned from block #679846 to #680258
-INFO  - 10:35:37: (BlockchainScanner.cpp:852) scanned from block #680259 to #680661
-INFO  - 10:36:06: (BlockchainScanner.cpp:852) scanned from block #680662 to #681071
-INFO  - 10:36:30: (BlockchainScanner.cpp:852) scanned from block #681072 to #681464
-INFO  - 10:36:56: (BlockchainScanner.cpp:852) scanned from block #681465 to #681840
-INFO  - 10:37:24: (BlockchainScanner.cpp:852) scanned from block #681841 to #682236
-INFO  - 10:37:50: (BlockchainScanner.cpp:852) scanned from block #682237 to #682647
-INFO  - 10:38:18: (BlockchainScanner.cpp:852) scanned from block #682648 to #683039
-INFO  - 10:38:53: (BlockchainScanner.cpp:852) scanned from block #683040 to #683440
-INFO  - 10:39:10: (BlockchainScanner.cpp:52) no history to scan
-INFO  - 10:39:22: (BlockchainScanner.cpp:852) scanned from block #683441 to #683841
-INFO  - 10:39:52: (BlockchainScanner.cpp:852) scanned from block #683842 to #684245
-INFO  - 10:40:20: (BlockchainScanner.cpp:852) scanned from block #684246 to #684655
-INFO  - 10:40:50: (BlockchainScanner.cpp:852) scanned from block #684656 to #685055
-INFO  - 10:41:17: (BlockchainScanner.cpp:852) scanned from block #685056 to #685461
-INFO  - 10:41:36: (BlockchainScanner.cpp:852) scanned from block #685462 to #685837
-INFO  - 10:41:59: (BlockchainScanner.cpp:852) scanned from block #685838 to #686251
-INFO  - 10:42:24: (BlockchainScanner.cpp:852) scanned from block #686252 to #686733
-INFO  - 10:42:46: (BlockchainScanner.cpp:852) scanned from block #686734 to #687168
-INFO  - 10:43:10: (BlockchainScanner.cpp:852) scanned from block #687169 to #687576
-INFO  - 10:43:32: (BlockchainScanner.cpp:852) scanned from block #687577 to #687995
-INFO  - 10:43:52: (BlockchainScanner.cpp:852) scanned from block #687996 to #688429
-ERROR - 10:44:17: (BlockchainScanner.cpp:445) Missing file map for output scan, this is unexpected
-ERROR - 10:44:17: (BlockchainScanner.cpp:447) Has the following block files:
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2631
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2632
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2633
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2634
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2635
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2636
-ERROR - 10:44:17: (BlockchainScanner.cpp:449)  --- #2637
-ERROR - 10:44:17: (BlockchainScanner.cpp:451) Was looking for id #2641

[goatpig]

It complains that it cannot find older blkXXXXX.dat files, in this case blk02641.dat. Can you confirm that file is in your blockchain data folder? Im going to assume it is for the remainder of this post.

"ArmoryDB --db-type=DB_BARE --ram-usage=1"

This setting is too approximative, the main way to clamp memory usage in ArmoryDB is by reducing thread count (--thread-count=X).

I don't know how memory is managed between containers and host. I've ran into the oomkiller a few times when running beefy containers + using a bunch of memory on the host (im on arch) and while the system recovers, a lot of processes blow up unpredictably. I can't tell if you're running in this issue because of corrupted data or because of weirdness with resource load within containers.

My suspicion is that a parser thread dies mid scan and you're seeing the consequences of that (block files are grabbed as whole, this would explain one file failing to scan entirely). --ram-usage controls how many block files can be parsed concurrently. Bitcoin Core keeps block files within 128MB so that's a proxy for memory usage in ArmoryDB. --thread-count limits the amount of threads the scanner gets to use and is a better predictor of resource load.

This is what I suggest for now:

1. Try to rescan (not rebuild) your db. If it fails at a different block file, then you are running into a concurrency issue. Toy with --ram-usage and --thread-count, maybe it will pass.
2. If it fails at the same block file, delete your database folder and try again. Does it fail at the same block file again during the new scan? It's possible the concurrency issue hit during the build phase and you're seeing the symptoms at the scan phase
3. In general, this is old code I haven't looked at in a while. If this problem persist, I'd invite you to try out the new code in the dev branch, as that's stuff I can realistically debug and fix.

[zzzrob]

Problem solved, and thanks for all the help!  The issue seems to have been on the bitcoin core side.

I actually already tried rebuilding ArmoryDB (deleting the whole databases folder in between) and it seems to fail at the same block file.

I spun up a new VM running the same docker environment, but doing a fresh bitcoin download this time to see if maybe there is something actually wrong with my bitcoin database.  Everything worked correctly with the new bitcoin database.

[goatpig]

glad to hear