Questions about data directory/files

[shogoo9U]

Sorry in advance for being such a newb, alas, my Google-fu has failed me ...

datadir holds, among other things (correct me, if I'm wrong):
- one armory_XXXXXXXXX_.wallet file (main) and one armory_XXXXXXXXX_backup.wallet file (duplicate) per wallet, with XXXXXXXXX being the wallet ID; this is the legacy wallet format
- armory_YYYYYYYYY_wallet.lmdb file and armory_YYYYYYYYY_wallet.lmdb-lock files; this is the new LMDB-based walled format (incl. file-based lock)

1) encrypted digital backups result in one *.wallet file (on Linux) but one *.wallet.rootpubkey file (on Windows). Why the different suffixes? Is the content of these backups the same (rootpubkey sounds suspiciously like it might be missing the private key)?

2) Can the LMDB wallet be regenerated at will or should it be backed up as well? What kind of information does it (not) contain, compared to the legacy wallet, how sensitive is it privacy/security-wise?

3) How can I tell which legacy wallet corresponds to which LMDB one, seeing as the 9-char-IDs are all different? For whatever reason I have 3 sets of .lmdb* files for 2 active wallets (1 online, 1 offline, in case it matters) -- normal, or left-over stale data from a past wallet-creation experiment?

4) On Linux there's a file called "~/.wallet", even though datadir is default ~/.armory, that I did not, to my knowledge, create manually. Any idea how it could've gotten there? User error is most likely, but if there's a chance Armory writes wallets outside of its datadir, I'd like to know.

[1714779180]

1714779180

[1714779180]

1714779180

[1714779180]

1714779180

[1714779180]

1714779180

[1714779180]

1714779180

[goatpig]

1) encrypted digital backups result in one *.wallet file (on Linux) but one *.wallet.rootpubkey file (on Windows). Why the different suffixes? Is the content of these backups the same (rootpubkey sounds suspiciously like it might be missing the private key)?

"Encrypted digital backup" is just a copy of your wallet file. If the file has no pub keys, you're gonna end up with a copy of a WO, hence the .rootpubkey thing.

2) Can the LMDB wallet be regenerated at will or should it be backed up as well? What kind of information does it (not) contain, compared to the legacy wallet, how sensitive is it privacy/security-wise?

They're created on the fly, they only contain public keys and the chaincode.

3) How can I tell which legacy wallet corresponds to which LMDB one, seeing as the 9-char-IDs are all different? For whatever reason I have 3 sets of .lmdb* files for 2 active wallets (1 online, 1 offline, in case it matters) -- normal, or left-over stale data from a past wallet-creation experiment?

You can't, the ID computation is different. The extra you got is a left over.

4) On Linux there's a file called "~/.wallet", even though datadir is default ~/.armory, that I did not, to my knowledge, create manually. Any idea how it could've gotten there? User error is most likely, but if there's a chance Armory writes wallets outside of its datadir, I'd like to know.

Armory only operates inside its datadirs. Some other process created that folder.

[shogoo9U]

"Encrypted digital backup" is just a copy of your wallet file.

Hmm, the "encrypted digital backup" is not bit-identical to the live wallet file in ~/.armory, at least, only the same size (tested on both installations). If I make multiple EDBs in succession those are bit-identical to each other. (?)

If the file has no pub keys, you're gonna end up with a copy of a WO, hence the .rootpubkey thing.

Now I'm really confused ...   Aside from the fact that "copy of WO" and ".rootpubkey" sounds more like it has no priv keys ... -- why/when would that be the case and is it normal for a new wallet?

P.S.: The different extensions for Linux and Windows EDBs might just be cosmetic: The Linux file picker has .wallet as default extension for new backups but filters for "Root Pubkey Text Files" (so doesn't show old backups in the same folder), Windows has .rootpubkey for both. Could be a case of an extension not consistently changed.

So I guess the question becomes, are my wallets and EDBs fine, even though they're not identical and "have no pub keys"?

You can't, the ID computation is different. The extra you got is a left over.

Ok, so delete all three, two will regenerate. Voight-Kampff test for timelords. Gotcha.

[goatpig]

If the file has no pub keys, you're gonna end up with a copy of a WO, hence the .rootpubkey thing.

Now I'm really confused ...  

That's a typo, I meant to write "no priv keys".

why/when would that be the case and is it normal for a new wallet?

There's an option to delete the private keys in a wallet. Cryptic stuff, but the edge cases need covered in the GUI regardless.

P.S.: The different extensions for Linux and Windows EDBs might just be cosmetic: The Linux file picker has .wallet as default extension for new backups but filters for "Root Pubkey Text Files" (so doesn't show old backups in the same folder), Windows has .rootpubkey for both. Could be a case of an extension not consistently changed.

rootpubkey backups were added by droark a couple years after the wallet code was written by etotheipi. The 2 years apart and 2 different devs could be the cause for some inconsistencies in file extention naming. If youfind an inconsistency, let me know.

So I guess the question becomes, are my wallets and EDBs fine, even though they're not identical and "have no pub keys"?

Regardless of my reply, you should just test the backups to make sure.

Ok, so delete all three, two will regenerate. Voight-Kampff test for timelords. Gotcha.

For what it's worth, the ID computations are different because the legacy one is tied to the Armory specific chain derivation algo, whereas the new wallets can maintain several derivation schemes within the same file.

[shogoo9U]

There's an option to delete the private keys in a wallet.

I certainly didn't use that. To make doubly sure it's not PEBKAC, I just created a new testwallet and made an EDB when prompted -- still gives me a .rootpubkey file (though I've no idea if it's the file contents that are off or just the extension, of course). I couldn't have deleted anything by that point if I tried. (That's v0.96.1 official binaries on Windows, BTW).

Regardless of my reply, you should just test the backups to make sure.

At this point it could be one of three things:
1) The actual wallets don't have their priv keys.
2) The priv keys get stripped from the backup.
3) The backup gets assigned the wrong extension.

Is there any way to test which it is, without moving small amounts of BTC back and forth? Can you reproduce it at all?

Also, should the EDBs be bit-identical to their live wallets or is that just a red herring?

P.S.: Small buglet: after creating a new wallet, ArmoryDB logs "(BDM_supportClasses.cpp:497) Completed scan of wallet XXXXXXXX" pretty soonish, but GUI shows "Scanning: 0%" in the balance column until restart.

[goatpig]

I certainly didn't use that. To make doubly sure it's not PEBKAC, I just created a new testwallet and made an EDB when prompted -- still gives me a .rootpubkey file (though I've no idea if it's the file contents that are off or just the extension, of course). I couldn't have deleted anything by that point if I tried. (That's v0.96.1 official binaries on Windows, BTW).

Will look into it.

Is there any way to test which it is, without moving small amounts of BTC back and forth? Can you reproduce it at all?

The security column in the lobby's wallet list tells you if your wallet is WO or has private keys (Encrypted or Unencrypted).

Also, should the EDBs be bit-identical to their live wallets or is that just a red herring?

Backups of this nature do not carry comments nor labels IIRC.

P.S.: Small buglet: after creating a new wallet, ArmoryDB logs "(BDM_supportClasses.cpp:497) Completed scan of wallet XXXXXXXX" pretty soonish, but GUI shows "Scanning: 0%" in the balance column until restart.

Show me armorylog.txt

[shogoo9U]

The security column in the lobby's wallet list tells you if your wallet is WO or has private keys (Encrypted or Unencrypted).

That's "Offline" for the watch-only wallet and "Encrypted" for the hot one, looks fine.

[Encrypted digital backups] do not carry comments nor labels IIRC.

Ok, so it's actually better to just copy the files in ~/.armory while it isn't running? Anything I'm missing?

Show me armorylog.txt

Code:

2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1138 - C++ block utilities loaded successfully
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1255 - 
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1256 - 
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1257 - 
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1258 - ************************************************************
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1259 - Invoked: E:\Armory\ArmoryQt.exe
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1260 - ************************************************************
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1261 - Loading Armory Engine:
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1262 -    Armory Version        : 0.96.1
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1263 -    Armory Build:         : b77932c8e3
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1264 -    PyBtcWallet  Version  : 1.35
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1265 - Detected Operating system: Windows
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1266 -    OS Variant            : 7-6.1.7601-SP1-Multiprocessor Free
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1267 -    User home-directory   : D:\red\AppData\Roaming
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1268 -    Satoshi BTC directory : D:\red\AppData\Roaming\Bitcoin\
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1269 -    Armory home dir       : D:\red\AppData\Roaming\Armory\
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1270 - Detected System Specs    : 
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1271 -    Total Available RAM   : 15.91 GB
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1272 -    CPU ID string         : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1273 -    Number of CPU cores   : 4 cores
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1274 -    System is 64-bit      : True
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1275 -    Preferred Encoding    : cp1252
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1276 -    Machine Arch          : amd64
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1277 -    Available HDD (ARM)   : 140 GB
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1278 -    Available HDD (BTC)   : 0 GB
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1279 - 
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1280 - Network Name: Main Network
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1281 - Satoshi Port: 8333
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1282 - Do wlt check: True
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1283 - Named options/arguments to armoryengine.py:
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     thread_count    : -1
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     rescan          : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     ignoreAllZC     : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     rescanBalance   : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     disableModules  : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     port            : None
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     interport       : 8223
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     coverageOutputDir: None
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     forceWalletCheck: False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     regtest         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     rebuild         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     nettimeout      : 2
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     datadir         : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     clearMempool    : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     offline         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     armoryDBDir     : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     armorydb_port   : 9001
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     satoshiPort     : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     useTorSettings  : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     netlog          : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     keypool         : 100
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     coverageInclude : None
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     forceOnline     : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     redownload      : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     rpcBindAddr     : 127.0.0.1
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     armorydb_ip     : 127.0.0.1
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     multisigFile    : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     ram_usage       : -1
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     mtdebug         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     logDisable      : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     settingsPath    : D:\red\AppData\Roaming\Armory\ArmorySettings.txt
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     language        : en
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     db_type         : DB_FULL
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     doDebug         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     enableDetSign   : True
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     disableConfPermis: False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     testnet         : False
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     rpcport         : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     satoshiHome     : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     satoshiRpcport  : DEFAULT
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     logFile         : D:\red\AppData\Roaming\Armory\ArmoryQt.exe.log.txt
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1285 -     verbosity       : None
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1286 - Other arguments:
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1289 - ************************************************************
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:1692 - C++ block utilities loaded successfully
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:3590 - Using settings file: D:\red\AppData\Roaming\Armory\ArmorySettings.txt
2017-08-04 10:56:15 (ERROR) -- ArmoryUtils.pyc:3747 - Unsupported language  specified. Defaulting to English (en)
2017-08-04 10:56:15 (INFO) -- ArmoryUtils.pyc:3750 - Using Language: en
2017-08-04 10:56:15 (INFO) -- BDM.pyc:365 - Using the asynchronous/multi-threaded BlockDataManager.
2017-08-04 10:56:15 (INFO) -- BDM.pyc:366 - Blockchain operations will happen in the background.  
2017-08-04 10:56:15 (INFO) -- BDM.pyc:367 - Devs: check TheBDM.getState() before asking for data.
2017-08-04 10:56:15 (INFO) -- BDM.pyc:368 - Registering addresses during rescans will queue them for 
2017-08-04 10:56:15 (INFO) -- BDM.pyc:369 - inclusion after the current scan is completed.
2017-08-04 10:56:16 (INFO) -- ArmoryUtils.pyc:3590 - Using settings file: D:\red\AppData\Roaming\Armory\ArmorySettings.txt
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2050 - loadWalletsAndSettings
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2110 - Loading wallets...
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2175 - Number of wallets read in: 2
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2180 -    Wallet (XXXXXXXXXX):   "Online Wallet (*****)           "   (Encrypted)
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2180 -    Wallet (YYYYYYYYYY):   "Offline Wallet (*****) (W)   "   (No Encryption)
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:2185 - Loading Multisig Lockboxes
2017-08-04 10:56:16 (INFO) -- ArmoryQt.py:1752 - acquiring process mutex...
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1391 - setupUriRegistration
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1509 - URL-register action: AskUser
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:565 - Usermode: Expert
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1687 - Changing usermode:
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1688 -    From: Expert
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1696 -      To: Expert
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1825 - startBitcoindIfNecessary
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1913 - Setting netmode: 1
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1895 - loadBlockchainIfNecessary
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:1913 - Setting netmode: 1
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:4644 - Dashboard switched to "Scanning" mode
2017-08-04 10:56:19 (INFO) -- ArmoryQt.py:4644 - Dashboard switched to "Scanning" mode
2017-08-04 10:56:57 (INFO) -- ArmoryQt.py:1138 - Adding 27 keypress events to the entropy pool
2017-08-04 10:56:57 (INFO) -- ArmoryQt.py:1140 - Adding 5.5 kB bytes of filesystem data to the entropy pool
2017-08-04 10:56:57 (INFO) -- ArmoryQt.py:1142 - Adding 175.9 kB bytes from desktop screenshot to the entropy pool
2017-08-04 10:56:57 (INFO) -- PyBtcWallet.pyc:882 - ***Creating new deterministic wallet
2017-08-04 10:56:57 (INFO) -- PyBtcWallet.pyc:888 - (with encryption)
2017-08-04 10:56:57 (INFO) -- PyBtcWallet.pyc:890 - Target (time,RAM)=(0.250,33554432)
2017-08-04 10:56:58 (INFO) -- PyBtcWallet.pyc:957 -    New wallet will be written to: D:\red\AppData\Roaming\Armory\armory_3fwvpcKR_.wallet
2017-08-04 10:57:02 (INFO) -- ArmoryQt.py:2842 - addWalletToApplication
2017-08-04 11:24:42 (INFO) -- ArmoryQt.py:4918 - New Block! : 478982
2017-08-04 11:24:42 (INFO) -- ArmoryQt.py:4926 - Current block number: 478982

That's launching armory and creating a new test wallet. And from ArmoryDB's POV:

Code:

-INFO  - 10:56:39: (BDM_Server.cpp:1074) registered bdv: 540932974008305bb3ee
-INFO  - 10:57:23: (BDM_supportClasses.cpp:378) Starting address registration process
-WARN  - 10:57:23: (BDM_supportClasses.cpp:224) Updating ssh last scanned
-INFO  - 10:57:23: (BlockchainScanner.cpp:665) scanned from height #478981 to #478981
-INFO  - 10:57:23: (BDM_supportClasses.cpp:497) Completed scan of wallet 3fwvpcKR
-INFO  - 11:25:02: (BlockchainScanner.cpp:665) scanned from height #478982 to #478982

[goatpig]

Ok, so it's actually better to just copy the files in ~/.armory while it isn't running? Anything I'm missing?

Nothing. You're better off backing that stuff up than what the GUI spits out. The GUI feature is there for people who don't want to paw around in the file system.

[shogoo9U]

Ok, did some testing, fees be damned. Both wallets can send and receive BTC normally, so the priv keys are definitely there for both. I'm going to go with copies of ~/.armory for digital backups, so whether or not the GUI EDBs work and have the correct extension doesn't impact me, but I still think you should look at that, when you have the time, just to be safe.

[goatpig]

Ok, did some testing, fees be damned. Both wallets can send and receive BTC normally, so the priv keys are definitely there for both. I'm going to go with copies of ~/.armory for digital backups, so whether or not the GUI EDBs work and have the correct extension doesn't impact me, but I still think you should look at that, when you have the time, just to be safe.

Fixed.