0.94.1 Building Databases 99% 59 Years - Forever

[Ondart]

I am attempting an upgrade from Armory v0.93.3; Bitcoin Core v0.13.0 to Armory v0.94.1. I am on Ubuntu Xenial 16.04 LTS 64-bit. Bitcoind is fully synchronized and healthy per getinfo. Armory 0.94.1 was installed and Armory Synchronizes successfully and then hangs at Building Databases 99% 59 Years. The Armory log has all healthy [INFO] logs ending in '(INFO) -- Networking.py:341 - Handshake finished, connection open!'. I don't think this is related but to be thorough, I use symbolic links from home folders ~/.armory and ~/.bitcoin to a dedicated volume hosting /media/btc/.armory and /media/btc/.bitcoin folders respectively and this was a good setup for Armory v.0.93.3. The Armory database folder and files are created and logging stops when it hangs without any errors. Bitcoin Core logs continue without error.

I have also tried the following without success:
 - Started Armory --satoshi-datadir and --datadir to the Armory and Bitcoin folders directly, just to rule this out
 - Deleted the Armory databases folder and restarted
 - Selected Rebuild and Rescan Databases and restarted
 - Removed and reinstalled Armory

I could use some help on this one.

[1714780515]

1714780515

[1714780515]

1714780515

[1714780515]

1714780515

[1714780515]

1714780515

[achow101]

Log file please.

[Ondart]

Here you go...

Code:

2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1080 - C++ block utilities loaded successfully
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: free -m
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: ['cat', '/proc/cpuinfo']
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1190 - 
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1191 - 
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1192 - 
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1193 - ************************************************************
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1194 - Invoked: /usr/lib/armory/ArmoryQt.py --satoshi-datadir=/media/Virtual_Currency/.bitcoin/ --datadir=/media/Virtual_Currency/.armory/
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1195 - ************************************************************
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1196 - Loading Armory Engine:
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1197 -    Armory Version        : 0.94.1
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1198 -    Armory Build:         : 811798a928
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1199 -    PyBtcWallet  Version  : 1.35
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1200 - Detected Operating system: Linux
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1201 -    OS Variant            : Ubuntu-16.04-xenial
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1202 -    User home-directory   : /home/xxx
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1203 -    Satoshi BTC directory : /media/Virtual_Currency/.bitcoin/
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1204 -    Armory home dir       : /media/Virtual_Currency/.armory/
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1205 - Detected System Specs    : 
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1206 -    Total Available RAM   : 31.32 GB
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1207 -    CPU ID string         : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1208 -    Number of CPU cores   : 12 cores
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1209 -    System is 64-bit      : True
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1210 -    Preferred Encoding    : UTF-8
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1211 -    Machine Arch          : x86_64
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1212 -    Available HDD (ARM)   : 125 GB
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1213 -    Available HDD (BTC)   : 125 GB
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1214 - 
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1215 - Network Name: Main Network
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1216 - Satoshi Port: 8333
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1217 - Do wlt check: True
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1218 - Named options/arguments to armoryengine.py:
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     nettimeout      : 2
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     rescan          : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     ignoreAllZC     : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     rescanBalance   : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     disableModules  : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     port            : None
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     interport       : 8223
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     skipStatsReport : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     forceWalletCheck: False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     rebuild         : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     datadir         : /media/Virtual_Currency/.armory/
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     clearMempool    : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     offline         : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     coverageOutputDir: None
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     armoryDBDir     : DEFAULT
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     satoshiPort     : DEFAULT
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     useTorSettings  : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     netlog          : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     keypool         : 100
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     coverageInclude : None
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     forceOnline     : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     skipAnnounceCheck: False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     redownload      : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     multisigFile    : DEFAULT
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     disableTorrent  : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     testAnnounceCode: False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     mtdebug         : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     logDisable      : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     settingsPath    : /media/Virtual_Currency/.armory/ArmorySettings.txt
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     verbosity       : None
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     doDebug         : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     enableDetSign   : True
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     testnet         : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     rpcport         : DEFAULT
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     satoshiHome     : /media/Virtual_Currency/.bitcoin/
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     satoshiRpcport  : DEFAULT
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     logFile         : /media/Virtual_Currency/.armory/ArmoryQt.py.log.txt
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     disableConfPermis: False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1220 -     enableSupernode : False
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1221 - Other arguments:
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1224 - ************************************************************
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:1624 - C++ block utilities loaded successfully
2016-10-02 19:56 (INFO) -- BDM.py:420 - Using the asynchronous/multi-threaded BlockDataManager.
2016-10-02 19:56 (INFO) -- BDM.py:421 - Blockchain operations will happen in the background.  
2016-10-02 19:56 (INFO) -- BDM.py:422 - Devs: check TheBDM.getState() before asking for data.
2016-10-02 19:56 (INFO) -- BDM.py:423 - Registering addresses during rescans will queue them for 
2016-10-02 19:56 (INFO) -- BDM.py:424 - inclusion after the current scan is completed.
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:3535 - Using settings file: /media/Virtual_Currency/.armory/ArmorySettings.txt
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2709 - loadWalletsAndSettings
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2763 - Loading wallets...
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2822 - Number of wallets read in: 5
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2827 -    Wallet (xxxxxxxxx):    "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"   (No Encryption)
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2268 - Setting up networking...
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2297 - Internet status: 0
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2412 - startBitcoindIfNecessary
2016-10-02 19:56 (INFO) -- ArmoryQt.py:2461 - setSatoshiPaths
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: ['whereis', 'bitcoind']
2016-10-02 19:56 (INFO) -- SDM.py:453 - "whereis" returned: ['/usr/bin/bitcoind']
2016-10-02 19:56 (INFO) -- SDM.py:321 - Found bitcoind in the following places:
2016-10-02 19:56 (INFO) -- SDM.py:323 -    /usr/bin/bitcoind
2016-10-02 19:56 (INFO) -- SDM.py:323 -    /usr/bin/bitcoind
2016-10-02 19:56 (INFO) -- SDM.py:325 - Using: /usr/bin/bitcoind
2016-10-02 19:56 (INFO) -- SDM.py:496 - Reading bitcoin.conf file
2016-10-02 19:56 (INFO) -- SDM.py:536 - Setting permissions on bitcoin.conf
2016-10-02 19:56 (INFO) -- SDM.py:597 - Called startBitcoind
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: ['/usr/bin/bitcoind', '-datadir=/media/Virtual_Currency/.bitcoin/']
2016-10-02 19:56 (INFO) -- SDM.py:669 - PID of bitcoind: 28305
2016-10-02 19:56 (INFO) -- SDM.py:670 - PID of armory:   28286
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: ['python', '/usr/lib/armory/guardian.py', '28286', '28305']
2016-10-02 19:56 (INFO) -- SDM.py:848 - Creating proxy in SDM: host=127.0.0.1, port=8332
2016-10-02 19:56 (INFO) -- ArmoryQt.py:1527 - setupUriRegistration
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: gconftool-2 --get /desktop/gnome/url-handlers/bitcoin/command
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: xdg-mime query default x-scheme-handler/bitcoin
2016-10-02 19:56 (INFO) -- ArmoryUtils.py:593 - Executing popen: find /home/xxxi -type f -name "mimeTypes.rdf"
2016-10-02 19:56 (INFO) -- ArmoryQt.py:664 - Usermode: Advanced
2016-10-02 19:56 (INFO) -- ArmoryQt.py:1823 - Changing usermode:
2016-10-02 19:56 (INFO) -- ArmoryQt.py:1824 -    From: Advanced
2016-10-02 19:56 (INFO) -- ArmoryQt.py:1832 -      To: Advanced
2016-10-02 19:56 (INFO) -- ArmoryQt.py:5801 - Dashboard switched to auto-BadConnection
2016-10-02 19:57 (INFO) -- ArmoryQt.py:2498 - loadBlockchainIfNecessary
2016-10-02 19:57 (INFO) -- ArmoryQt.py:2519 - Setting netmode: 1
2016-10-02 19:57 (INFO) -- ArmoryQt.py:5935 - Dashboard switched to "Scanning" mode
2016-10-02 19:57 (INFO) -- Networking.py:64 - Connection initiated.  Start handshake
2016-10-02 19:57 (INFO) -- ArmoryQt.py:5935 - Dashboard switched to "Scanning" mode
2016-10-02 19:57 (INFO) -- Networking.py:155 - Received version message from peer:
2016-10-02 19:57 (INFO) -- Networking.py:156 -    Version:     70014
2016-10-02 19:57 (INFO) -- Networking.py:157 -    SubVersion:  /Satoshi:0.13.0/
2016-10-02 19:57 (INFO) -- Networking.py:158 -    TimeStamp:   1475459835
2016-10-02 19:57 (INFO) -- Networking.py:159 -    StartHeight: 432591
2016-10-02 19:57 (INFO) -- Networking.py:341 - Handshake finished, connection open!

Code:

2016-10-03 01:56:37 Bitcoin version v0.13.0.0-ga402396
2016-10-03 01:56:37 InitParameterInteraction: parameter interaction: -whitelistforcerelay=1 -> setting -whitelistrelay=1
2016-10-03 01:56:37 Default data directory /home/xxx/.bitcoin
2016-10-03 01:56:37 Using data directory /media/Virtual_Currency/.bitcoin/
2016-10-03 01:56:37 Using config file /media/Virtual_Currency/.bitcoin/bitcoin.conf
2016-10-03 01:56:37 Using at most 125 connections (1024 file descriptors available)
2016-10-03 01:56:37 Using 6 threads for script verification
2016-10-03 01:56:37 scheduler thread start
2016-10-03 01:56:37 HTTP: creating work queue of depth 16
2016-10-03 01:56:37 Config options rpcuser and rpcpassword will soon be deprecated. Locally-run instances may remove rpcuser to use cookie-based auth, or may be replaced with rpcauth. Please see share/rpcuser for rpcauth auth generation.
2016-10-03 01:56:37 HTTP: starting 4 worker threads
2016-10-03 01:56:37 Using BerkeleyDB version Berkeley DB 4.8.30: (April  9, 2010)
2016-10-03 01:56:37 Using wallet wallet.dat
2016-10-03 01:56:37 init message: Verifying wallet...
2016-10-03 01:56:37 CDBEnv::Open: LogDir=/media/Virtual_Currency/.bitcoin/database ErrorFile=/media/Virtual_Currency/.bitcoin/db.log
2016-10-03 01:56:37 Bound to [::]:8333
2016-10-03 01:56:37 Bound to 0.0.0.0:8333
2016-10-03 01:56:37 Cache configuration:
2016-10-03 01:56:37 * Using 2.0MiB for block index database
2016-10-03 01:56:37 * Using 8.0MiB for chain state database
2016-10-03 01:56:37 * Using 290.0MiB for in-memory UTXO set
2016-10-03 01:56:37 init message: Loading block index...
2016-10-03 01:56:37 Opening LevelDB in /media/Virtual_Currency/.bitcoin/blocks/index
2016-10-03 01:56:37 Opened LevelDB successfully
2016-10-03 01:56:37 Using obfuscation key for /media/Virtual_Currency/.bitcoin/blocks/index: 0000000000000000
2016-10-03 01:56:37 Opening LevelDB in /media/Virtual_Currency/.bitcoin/chainstate
2016-10-03 01:56:37 Opened LevelDB successfully
2016-10-03 01:56:37 Using obfuscation key for /media/Virtual_Currency/.bitcoin/chainstate: 0000000000000000
2016-10-03 01:56:40 LoadBlockIndexDB: last block file = 640
2016-10-03 01:56:40 LoadBlockIndexDB: last block file info: CBlockFileInfo(blocks=27, size=24770000, heights=432565...432591, time=2016-10-02...2016-10-03)
2016-10-03 01:56:40 Checking all blk files are present...
2016-10-03 01:56:40 LoadBlockIndexDB: transaction index disabled
2016-10-03 01:56:40 LoadBlockIndexDB: hashBestChain=000000000000000000af414a113e05fe53cbaae75dbac7724fad261b6222ed33 height=432591 date=2016-10-03 01:47:31 progress=0.999997
2016-10-03 01:56:40 init message: Rewinding blocks...
2016-10-03 01:56:41 init message: Verifying blocks...
2016-10-03 01:56:41 Verifying last 288 blocks at level 3
2016-10-03 01:56:41 [0%]...[10%]...[20%]...[30%]...[40%]...[50%]...[60%]...[70%]...[80%]...[90%]...[DONE].
2016-10-03 01:57:05 No coin database inconsistencies in last 94 blocks (153999 transactions)
2016-10-03 01:57:05  block index           28211ms
2016-10-03 01:57:05 init message: Loading wallet...
2016-10-03 01:57:05 nFileVersion = 130000
2016-10-03 01:57:05 Keys: 102 plaintext, 0 encrypted, 102 w/ metadata, 102 total
2016-10-03 01:57:05  wallet                    5ms
2016-10-03 01:57:05 mapBlockIndex.size() = 432670
2016-10-03 01:57:05 nBestHeight = 432591
2016-10-03 01:57:05 setKeyPool.size() = 101
2016-10-03 01:57:05 mapWallet.size() = 0
2016-10-03 01:57:05 mapAddressBook.size() = 1
2016-10-03 01:57:05 init message: Loading addresses...
2016-10-03 01:57:05 torcontrol thread start
2016-10-03 01:57:05 Loaded 57656 addresses from peers.dat  131ms
2016-10-03 01:57:05 init message: Loading banlist...
2016-10-03 01:57:05 init message: Done loading
2016-10-03 01:57:05 net thread start
2016-10-03 01:57:05 opencon thread start
2016-10-03 01:57:05 dnsseed thread start
2016-10-03 01:57:05 msghand thread start
2016-10-03 01:57:05 addcon thread start
2016-10-03 01:57:06 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:06 receive version message: /Classic:0.12.0/: version 70012, blocks=432591, us=xxx.xxx.xxx.xxx:53396, peer=1
2016-10-03 01:57:06 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:07 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:07 receive version message: /Satoshi:0.9.3/: version 70002, blocks=432591, us=xxx.xxx.xxx.xxx:44782, peer=2
2016-10-03 01:57:07 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:08 connect() to xxx.xxx.xxx.xxx:8333 failed after select(): No route to host (113)
2016-10-03 01:57:14 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:14 receive version message: /Satoshi:0.11.2/: version 70002, blocks=432591, us=xxx.xxx.xxx.xxx:56574, peer=4
2016-10-03 01:57:14 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:14 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:14 receive version message: /Satoshi:0.13.99/: version 70014, blocks=432591, us=xxx.xxx.xxx.xxx:58788, peer=3
2016-10-03 01:57:14 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:16 receive version message: Armory:0.94.1: version 40000, blocks=-1, us=[::127.0.0.1]:8333, peer=6
2016-10-03 01:57:16 P2P peers available. Skipped DNS seeding.
2016-10-03 01:57:16 dnsseed thread exit
2016-10-03 01:57:20 socket recv error Connection reset by peer (104)
2016-10-03 01:57:59 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:57:59 receive version message: /Satoshi:0.12.1/: version 70012, blocks=432591, us=xxx.xxx.xxx.xxx:53304, peer=8
2016-10-03 01:57:59 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:58:00 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:58:00 receive version message: /Satoshi:0.13.0/: version 70014, blocks=409384, us=xxx.xxx.xxx.xxx:58160, peer=9
2016-10-03 01:58:00 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:58:12 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:58:12 receive version message: /BitcoinUnlimited:0.12.1(EB16; AD4)/: version 80002, blocks=432591, us=xxx.xxx.xxx.xxx:40542, peer=10
2016-10-03 01:58:12 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:58:18 connect() to [2600:8805:2400:14e:12dd:b1ff:fef2:3013]:8333 failed: Network is unreachable (101)
2016-10-03 01:58:24 connect() to [2001:0:5ef5:79fd:3c43:636:e0ca:8fb5]:8333 failed: Network is unreachable (101)
2016-10-03 01:58:25 connect() to 41.75.96.80:8333 failed after select(): Connection refused (111)
2016-10-03 01:58:26 connect() to [2001:0:9d38:6abd:18b7:5a9:9d80:aca9]:8333 failed: Network is unreachable (101)
2016-10-03 01:58:32 socket recv error Connection reset by peer (104)
2016-10-03 01:58:32 connect() to [2601:406:300:c300:7d35:8507:1e30:1a3]:8333 failed: Network is unreachable (101)
2016-10-03 01:58:33 connect() to [2601:647:4d04:4a00:e4fc:6373:39:b64b]:8333 failed: Network is unreachable (101)
2016-10-03 01:58:34 connect() to 93.133.179.32:8333 failed after select(): No route to host (113)
2016-10-03 01:59:02 socket recv error Connection reset by peer (104)
2016-10-03 01:59:02 connect() to [2a02:810d:300:714:c32:a8c:42da:709e]:8333 failed: Network is unreachable (101)
2016-10-03 01:59:03 ProcessMessages: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 01:59:03 receive version message: /Satoshi:0.13.0/: version 70014, blocks=432591, us=xxx.xxx.xxx.xxx:48666, peer=13
2016-10-03 01:59:03 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 02:10:29 UpdateTip: new best=000000000000000002be5e613ae2df0e4703ba302993b086179c0c0bf2b896b3 height=432592 version=0x20000000 log2_work=85.354403 tx=160180645 date='2016-10-03 02:09:40' progress=1.000000 cache=5.0MiB(7608tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:10:38 UpdateTip: new best=000000000000000000fe2161f748268380bcbc6fb6f74fa3b4f37723f22aecc5 height=432593 version=0x20000000 log2_work=85.354434 tx=160182228 date='2016-10-03 02:11:22' progress=1.000000 cache=8.8MiB(11032tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:14:56 AdvertiseLocal: advertising address xxx.xxx.xxx.xxx:8333
2016-10-03 02:18:40 UpdateTip: new best=0000000000000000043b7f6c30cbfe25f15afdcd14da8dfb2e697ec392062f7c height=432594 version=0x20000000 log2_work=85.354464 tx=160184582 date='2016-10-03 02:20:25' progress=1.000001 cache=26.9MiB(17003tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:21:18 UpdateTip: new best=000000000000000004335f4c8947baced3d43689fe6098f0aa1a7eed60bd0390 height=432595 version=0x20000000 log2_work=85.354494 tx=160185919 date='2016-10-03 02:21:01' progress=1.000000 cache=28.9MiB(20756tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:23:44 UpdateTip: new best=000000000000000002acaf4d8989fec2d03a6a5fc9b7d8386d04de6c3055dc72 height=432596 version=0x20000000 log2_work=85.354524 tx=160187041 date='2016-10-03 02:22:59' progress=1.000000 cache=34.8MiB(24053tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:27:05 UpdateTip: new best=000000000000000002f58224200f7c6b1b445fa6a0bd1638b786188e44325021 height=432597 version=0x20000000 log2_work=85.354555 tx=160188750 date='2016-10-03 02:26:12' progress=1.000000 cache=43.3MiB(28618tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:32:37 UpdateTip: new best=0000000000000000005c2481cc3572268fca95565859a8c1cddd4c3b5d501519 height=432598 version=0x20000000 log2_work=85.354585 tx=160189823 date='2016-10-03 02:32:04' progress=1.000000 cache=50.3MiB(31160tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:47:25 UpdateTip: new best=0000000000000000002bc01e1736ede417174092e9e958a85733e0aa2b6dd991 height=432599 version=0x20000000 log2_work=85.354615 tx=160191771 date='2016-10-03 02:46:58' progress=1.000000 cache=55.7MiB(34746tx) warning='3 of last 100 blocks have unexpected version'
2016-10-03 02:50:02 Pre-allocating up to position 0x500000 in rev00640.dat
2016-10-03 02:50:02 UpdateTip: new best=0000000000000000023cc80de0ea07d4444897acf971697586a51687e4bf22db height=432600 version=0x20000000 log2_work=85.354645 tx=160192594 date='2016-10-03 02:49:18' progress=1.000000 cache=76.4MiB(37983tx) warning='3 of last 100 blocks have unexpected version'

[achow101]

Try disabling the auto-managed bitcoind and run Bitcoin Core manually.

[Ondart]

Try disabling the auto-managed bitcoind and run Bitcoin Core manually.

Auto-managed bitcoin in Armory disabled. bitcoind started manually in Terminal; getinfo shows healthy. Started Armory and it synced for 3 min and then, same behavior, hung at Building Databases 99% 30 years. The logs are the same as posted, without errors.

[Ondart]

Here is a similar thread with the exact same behavior. The solution there was to remove Bitcoin Core and Armory, install a newer Bitcoin Core 0.12.0 and Armory again and resync the entire blockchain. I am running 0.13.0 already but I think a complete resync was/is a bit extreme however.

https://bitcointalk.org/index.php?topic=1382534.0

[achow101]

Can you post the cpplog file too? That's somewhat important, it's where all the db stuff happens.

[Ondart]

Certainly...

Note: I modified this post with the log that matches the previous logs I posted previously.

Code:

Log file opened at 1475459791: /media/Virtual_Currency/.armory/armorycpplog.txt
-INFO  - 1475459836: (BlockUtils.cpp:924) blkfile dir: /media/Virtual_Currency/.bitcoin/blocks
-INFO  - 1475459836: (BlockUtils.cpp:925) lmdb dir: /media/Virtual_Currency/.armory/databases
-INFO  - 1475459836: (lmdb_wrapper.cpp:387) Opening databases...
-INFO  - 1475459836: (BlockUtils.cpp:1105) Executing: doInitialSyncOnLoad
-INFO  - 1475459836: (DatabaseBuilder.cpp:162) Reading headers from db
-WARN  - 1475459836: (lmdb_wrapper.cpp:1257) No headers in DB yet!
-INFO  - 1475459836: (DatabaseBuilder.cpp:195) Found 1 headers in db
-INFO  - 1475459836: (DatabaseBuilder.cpp:43) updating HEADERS db
-INFO  - 1475459844: (DatabaseBuilder.cpp:223) parsed block file #0
-INFO  - 1475459847: (DatabaseBuilder.cpp:223) parsed block file #12
-INFO  - 1475459852: (DatabaseBuilder.cpp:223) parsed block file #24
-INFO  - 1475459855: (DatabaseBuilder.cpp:378) Found next block after skipping 277494bytes
-INFO  - 1475459856: (DatabaseBuilder.cpp:223) parsed block file #36
-INFO  - 1475459860: (DatabaseBuilder.cpp:223) parsed block file #48
-INFO  - 1475459865: (DatabaseBuilder.cpp:223) parsed block file #60
-INFO  - 1475459869: (DatabaseBuilder.cpp:223) parsed block file #72
-INFO  - 1475459874: (DatabaseBuilder.cpp:223) parsed block file #84
-INFO  - 1475459878: (DatabaseBuilder.cpp:223) parsed block file #96
-INFO  - 1475459883: (DatabaseBuilder.cpp:223) parsed block file #108
-INFO  - 1475459887: (DatabaseBuilder.cpp:223) parsed block file #120
-INFO  - 1475459892: (DatabaseBuilder.cpp:223) parsed block file #132
-INFO  - 1475459896: (DatabaseBuilder.cpp:223) parsed block file #144
-INFO  - 1475459900: (DatabaseBuilder.cpp:223) parsed block file #156
-INFO  - 1475459905: (DatabaseBuilder.cpp:223) parsed block file #168
-INFO  - 1475459909: (DatabaseBuilder.cpp:223) parsed block file #180
-INFO  - 1475459914: (DatabaseBuilder.cpp:223) parsed block file #192
-INFO  - 1475459918: (DatabaseBuilder.cpp:223) parsed block file #204
-INFO  - 1475459923: (DatabaseBuilder.cpp:223) parsed block file #216
-INFO  - 1475459927: (DatabaseBuilder.cpp:223) parsed block file #228
-INFO  - 1475459932: (DatabaseBuilder.cpp:223) parsed block file #240
-INFO  - 1475459936: (DatabaseBuilder.cpp:223) parsed block file #252
-INFO  - 1475459937: (DatabaseBuilder.cpp:378) Found next block after skipping 749129bytes
-INFO  - 1475459941: (DatabaseBuilder.cpp:223) parsed block file #264
-INFO  - 1475459945: (DatabaseBuilder.cpp:223) parsed block file #276
-INFO  - 1475459950: (DatabaseBuilder.cpp:223) parsed block file #288
-INFO  - 1475459954: (DatabaseBuilder.cpp:223) parsed block file #300
-INFO  - 1475459959: (DatabaseBuilder.cpp:223) parsed block file #312
-INFO  - 1475459963: (DatabaseBuilder.cpp:223) parsed block file #324
-INFO  - 1475459963: (DatabaseBuilder.cpp:378) Found next block after skipping 281377bytes
-INFO  - 1475459968: (DatabaseBuilder.cpp:223) parsed block file #336
-INFO  - 1475459972: (DatabaseBuilder.cpp:223) parsed block file #348
-INFO  - 1475459977: (DatabaseBuilder.cpp:223) parsed block file #360
-INFO  - 1475459981: (DatabaseBuilder.cpp:223) parsed block file #372
-INFO  - 1475459986: (DatabaseBuilder.cpp:223) parsed block file #384
-INFO  - 1475459990: (DatabaseBuilder.cpp:223) parsed block file #396
-INFO  - 1475459994: (DatabaseBuilder.cpp:223) parsed block file #408
-INFO  - 1475459999: (DatabaseBuilder.cpp:223) parsed block file #420
-INFO  - 1475460003: (DatabaseBuilder.cpp:223) parsed block file #432
-INFO  - 1475460008: (DatabaseBuilder.cpp:223) parsed block file #444
-INFO  - 1475460012: (DatabaseBuilder.cpp:223) parsed block file #456
-INFO  - 1475460016: (DatabaseBuilder.cpp:223) parsed block file #468
-INFO  - 1475460021: (DatabaseBuilder.cpp:223) parsed block file #480
-INFO  - 1475460025: (DatabaseBuilder.cpp:223) parsed block file #492
-INFO  - 1475460030: (DatabaseBuilder.cpp:223) parsed block file #504
-INFO  - 1475460034: (DatabaseBuilder.cpp:223) parsed block file #516
-INFO  - 1475460038: (DatabaseBuilder.cpp:223) parsed block file #528
-INFO  - 1475460039: (DatabaseBuilder.cpp:378) Found next block after skipping 998043bytes
-INFO  - 1475460043: (DatabaseBuilder.cpp:223) parsed block file #540
-INFO  - 1475460047: (DatabaseBuilder.cpp:223) parsed block file #552
-INFO  - 1475460051: (DatabaseBuilder.cpp:223) parsed block file #564
-INFO  - 1475460056: (DatabaseBuilder.cpp:223) parsed block file #576
-INFO  - 1475460060: (DatabaseBuilder.cpp:223) parsed block file #588
-INFO  - 1475460065: (DatabaseBuilder.cpp:223) parsed block file #600
-INFO  - 1475460070: (DatabaseBuilder.cpp:223) parsed block file #612
-INFO  - 1475460074: (DatabaseBuilder.cpp:223) parsed block file #624
-INFO  - 1475460075: (DatabaseBuilder.cpp:223) parsed block file #636
-DEBUG - 1475460075: (Blockchain.cpp:213) Organizing chain  

[achow101]

I get the feeling that something is amiss with it skipping that many bytes..

I'm not sure what is wrong here. Redownloading the blockchain would most likely fix your issue. You could also try running the next testing version for 0.95 which should be out in a day or so (or run the one that is already released).

[Ondart]

Yeah, I was trying to avoid it but I am thinking I am going to have to do it. I have a bash script that saves a copy of the blockchain every 3 days off to another volume and I may do the reinstalls but use the copy to speed things up first. I have a 2nd copy on my bitcoin full node server I could try also.  If that doesn't do it, I'll resync the blockchain fresh. I'll go for 0.95.0 after that if the issue persists. I'll post my findings when i do. Thank you by the way for the immediate responses, it is very much appreciated.

One more thing...

I just restarted Armory and finally see two errors in the cpplog for the first time. Before I do a total nuke and resync everything, does this new error in the log provide any other clues?

Code:

/usr/lib/armory/armoryengine/Transaction.py:2653: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(ERROR) ArmoryQt.py:1336 - 3 attempts to load blockchain failed.  Remove mempool.bin.
(ERROR) ArmoryQt.py:1341 - File mempool.bin does not exist. Nothing deleted.
-INFO  - 1475468210: (BlockUtils.cpp:924) blkfile dir: /media/Virtual_Currency/.bitcoin/blocks
-INFO  - 1475468210: (BlockUtils.cpp:925) lmdb dir: /media/Virtual_Currency/.armory/databases
-INFO  - 1475468210: (lmdb_wrapper.cpp:387) Opening databases...
-INFO  - 1475468210: (BlockUtils.cpp:1105) Executing: doInitialSyncOnLoad
-INFO  - 1475468210: (DatabaseBuilder.cpp:162) Reading headers from db
-WARN  - 1475468210: (lmdb_wrapper.cpp:1257) No headers in DB yet!
-INFO  - 1475468210: (DatabaseBuilder.cpp:195) Found 1 headers in db
-INFO  - 1475468210: (DatabaseBuilder.cpp:43) updating HEADERS db
-INFO  - 1475468219: (DatabaseBuilder.cpp:223) parsed block file #0
-INFO  - 1475468222: (DatabaseBuilder.cpp:223) parsed block file #12
-INFO  - 1475468225: (DatabaseBuilder.cpp:223) parsed block file #24
-INFO  - 1475468227: (DatabaseBuilder.cpp:378) Found next block after skipping 277494bytes
-INFO  - 1475468229: (DatabaseBuilder.cpp:223) parsed block file #36
-INFO  - 1475468233: (DatabaseBuilder.cpp:223) parsed block file #48
-INFO  - 1475468236: (DatabaseBuilder.cpp:223) parsed block file #60
-INFO  - 1475468240: (DatabaseBuilder.cpp:223) parsed block file #72
-INFO  - 1475468243: (DatabaseBuilder.cpp:223) parsed block file #84
-INFO  - 1475468246: (DatabaseBuilder.cpp:223) parsed block file #96
-INFO  - 1475468249: (DatabaseBuilder.cpp:223) parsed block file #108
-INFO  - 1475468252: (DatabaseBuilder.cpp:223) parsed block file #120
-INFO  - 1475468256: (DatabaseBuilder.cpp:223) parsed block file #132
-INFO  - 1475468259: (DatabaseBuilder.cpp:223) parsed block file #144
-INFO  - 1475468262: (DatabaseBuilder.cpp:223) parsed block file #156
-INFO  - 1475468265: (DatabaseBuilder.cpp:223) parsed block file #168
-INFO  - 1475468269: (DatabaseBuilder.cpp:223) parsed block file #180
-INFO  - 1475468272: (DatabaseBuilder.cpp:223) parsed block file #192
-INFO  - 1475468276: (DatabaseBuilder.cpp:223) parsed block file #204
-INFO  - 1475468280: (DatabaseBuilder.cpp:223) parsed block file #216
-INFO  - 1475468284: (DatabaseBuilder.cpp:223) parsed block file #228
-INFO  - 1475468288: (DatabaseBuilder.cpp:223) parsed block file #240
-INFO  - 1475468290: (DatabaseBuilder.cpp:378) Found next block after skipping 749129bytes
-INFO  - 1475468292: (DatabaseBuilder.cpp:223) parsed block file #252
-INFO  - 1475468296: (DatabaseBuilder.cpp:223) parsed block file #264
-INFO  - 1475468300: (DatabaseBuilder.cpp:223) parsed block file #276
-INFO  - 1475468304: (DatabaseBuilder.cpp:223) parsed block file #288
-INFO  - 1475468308: (DatabaseBuilder.cpp:223) parsed block file #300
-INFO  - 1475468312: (DatabaseBuilder.cpp:223) parsed block file #312
-INFO  - 1475468317: (DatabaseBuilder.cpp:223) parsed block file #324
-INFO  - 1475468321: (DatabaseBuilder.cpp:223) parsed block file #336
-INFO  - 1475468325: (DatabaseBuilder.cpp:223) parsed block file #348
-INFO  - 1475468329: (DatabaseBuilder.cpp:223) parsed block file #360
-INFO  - 1475468333: (DatabaseBuilder.cpp:223) parsed block file #372
-INFO  - 1475468337: (DatabaseBuilder.cpp:223) parsed block file #384
-INFO  - 1475468342: (DatabaseBuilder.cpp:223) parsed block file #396
-INFO  - 1475468346: (DatabaseBuilder.cpp:223) parsed block file #408
-INFO  - 1475468350: (DatabaseBuilder.cpp:223) parsed block file #420
-INFO  - 1475468354: (DatabaseBuilder.cpp:223) parsed block file #432
-INFO  - 1475468359: (DatabaseBuilder.cpp:223) parsed block file #444
-INFO  - 1475468363: (DatabaseBuilder.cpp:223) parsed block file #456
-INFO  - 1475468367: (DatabaseBuilder.cpp:223) parsed block file #468
-INFO  - 1475468371: (DatabaseBuilder.cpp:223) parsed block file #480
-INFO  - 1475468376: (DatabaseBuilder.cpp:223) parsed block file #492
-INFO  - 1475468380: (DatabaseBuilder.cpp:223) parsed block file #504
-INFO  - 1475468384: (DatabaseBuilder.cpp:223) parsed block file #516
-INFO  - 1475468386: (DatabaseBuilder.cpp:378) Found next block after skipping 998043bytes
-INFO  - 1475468388: (DatabaseBuilder.cpp:223) parsed block file #528
-INFO  - 1475468392: (DatabaseBuilder.cpp:223) parsed block file #540
-INFO  - 1475468397: (DatabaseBuilder.cpp:223) parsed block file #552
-INFO  - 1475468401: (DatabaseBuilder.cpp:223) parsed block file #564
-INFO  - 1475468405: (DatabaseBuilder.cpp:223) parsed block file #576
-INFO  - 1475468410: (DatabaseBuilder.cpp:223) parsed block file #588
-INFO  - 1475468414: (DatabaseBuilder.cpp:223) parsed block file #600
-INFO  - 1475468418: (DatabaseBuilder.cpp:223) parsed block file #612
-INFO  - 1475468420: (DatabaseBuilder.cpp:223) parsed block file #624
-INFO  - 1475468422: (DatabaseBuilder.cpp:223) parsed block file #636
-DEBUG - 1475468422: (Blockchain.cpp:213) Organizing chain 

[achow101]

Well it means that has failed to load the blockchain before, which is usually indicative of some corruption in the blockchain.

[Ondart]

OK, I'll post more details when I get to this tomorrow evening after work. Thanks for the support achow101.

[Ondart]

I attempted to download the bitcoind blockchain from scratch and kept getting a database error after a few hours even with the -reindex switch to which I find odd since I have been running bitcoin 0.13.0 for over a year on this SSD without issue . I ended up restoring a backup copy of the blockchain that I have and bitcoind was stable and synchronizing. Then I removed and re-installed Armory 0.93.3 and in about an hour or so, Armory rebuilt its' database successfully and it is fully functional. The blockchain I have works fine with Arnory 0.93.3 but won't work with 0.94.1. So, I think I am going to try Armory 0.95 when its out and see if it works with my system next. I seriously doubt that I have corruption on this SSD.

[achow101]

I attempted to download the bitcoind blockchain from scratch and kept getting a database error after a few hours even with the -reindex switch to which I find odd since I have been running bitcoin 0.13.0 for over a year on this SSD without issue . I ended up restoring a backup copy of the blockchain that I have and bitcoind was stable and synchronizing. Then I removed and re-installed Armory 0.93.3 and in about an hour or so, Armory rebuilt its' database successfully and it is fully functional. The blockchain I have works fine with Arnory 0.93.3 but won't work with 0.94.1. So, I think I am going to try Armory 0.95 when its out and see if it works with my system next. I seriously doubt that I have corruption on this SSD.

That's quite interesting.

You can try out the latest testing build from https://github.com/goatpig/BitcoinArmory/releases/tag/v0.94.99.1-testing. This one is basically the final version as we have fixed all of the previous bugs and issues and haven't found anymore. We also need people to test it 

[Ondart]

I attempted to download the bitcoind blockchain from scratch and kept getting a database error after a few hours even with the -reindex switch to which I find odd since I have been running bitcoin 0.13.0 for over a year on this SSD without issue . I ended up restoring a backup copy of the blockchain that I have and bitcoind was stable and synchronizing. Then I removed and re-installed Armory 0.93.3 and in about an hour or so, Armory rebuilt its' database successfully and it is fully functional. The blockchain I have works fine with Arnory 0.93.3 but won't work with 0.94.1. So, I think I am going to try Armory 0.95 when its out and see if it works with my system next. I seriously doubt that I have corruption on this SSD.

That's quite interesting.

You can try out the latest testing build from https://github.com/goatpig/BitcoinArmory/releases/tag/v0.94.99.1-testing. This one is basically the final version as we have fixed all of the previous bugs and issues and haven't found anymore. We also need people to test it 

Thanks, I'll give this a try and report back my findings.

[Ondart]

achow101,

I resolved my issues and I wanted to provide some feedback on my experience. I decided to wait for Bitcoin Core v0.13.1 and Armory v0.95 before attempting an upgrade again from Bitcoin Core v0.13.0 and Armory v0.93.3. I installed the latest Bitcoin Core and Armory recently without issue and removed all /.armory files and folders except for my off-line Wallet files. Starting Armory hung permanently at Organizing the Blockchain. The Bitcoin debug.log was healthy. The Armory dblog.txt had these errors however:

Code:

-ERROR - 1477788588: (BlockDataMap.cpp:105) merkle root mismatch!
-ERROR - 1477788588: (BlockDataMap.cpp:106)    header has: b2b75721aae1017cf437651258cb306bd349805ca031f9283c778f3998eea9be
-ERROR - 1477788588: (BlockDataMap.cpp:107)    block yields: bbb666da44d603d199005469ab504ffd1226e3debc7826eff52729a2fc945dbd
-ERROR - 1477788588: (DatabaseBuilder.cpp:344) block deser except: invalid merkle root
-INFO  - 1477788588: (DatabaseBuilder.cpp:461) Found next block after skipping 949126bytes

I decided to rebuild the blockchain and try again. Last time it kept crashing with database errors that others always says one has a corrupted disk when this happens but I know it was not so I changed my network connection from wifi to a direct network connection to my router and this allowed the rebuild to complete without error. I have a Bitcoin Full Node so it only took about 8 hours surprisingly. Once complete, I cleaned out the Armory folder again and started it up and was successful in the upgrade. If segwit activates, I think I am good to go. Thanks for all the great support. Armory rocks!

Ondart

------------------------------------------------------------------------------
"It might make sense just to get some in case it catches on" - Satoshi Nakamoto 01/17/2009
1GbPxjXFttr53m4qcTHEHLZdSN9Tn6R969