Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help.

[jwputrl]

I'm trying to install Armory in Ubuntu 15.04, using the version at https://github.com/goatpig/BitcoinArmory/releases/tag/v0.93.3.  I have a few wallets already, from an installation on a laptop I had stolen.  So I'm trying to get these coins back.

Dashboard shows "Synchronizing with Network" complete.  It has finished "reading block headers", but now hangs on Organizing Blockchain.  It's been that way for maybe 18h overall (with a couple of attempted restarts in between).

Might be relevant:  I tried to give the new install a headstart by copying over some of the database files from a backup I had of the old installation (and also to the .bitcoin installation).

Also:  when I quit, it hangs on "Preparing to shut down", and I have to quit again.  When that happens I get the following repeated many times in  .armory/armorylog.txt:

2016-02-14 09:52 (WARNING) -- SDM.py:768 - Overriding not-available state. This should happen 0-5 times


Heres the cpplog.txt after re-starting, up to present moment where it hangs.

Log file opened at 1455443804: ~/.armory/armorycpplog.txt
-INFO  - 1455443847: (BlockUtils.cpp:918) blkfile dir: /home/gav/.bitcoin/blocks
-INFO  - 1455443847: (BlockUtils.cpp:919) lmdb dir: /home/gav/.armory/databases
-INFO  - 1455443847: (lmdb_wrapper.cpp:478) Opening databases...
-INFO  - 1455443847: (BlockUtils.cpp:1250) Executing: doInitialSyncOnLoad
-INFO  - 1455443847: (BlockUtils.cpp:1321) Total number of blk*.dat files: 444
-INFO  - 1455443847: (BlockUtils.cpp:1322) Total blockchain bytes: 59,390,537,918
-INFO  - 1455443847: (BlockUtils.cpp:1695) Reading headers from db
-WARN  - 1455443847: (lmdb_wrapper.cpp:1478) No headers in DB yet!
-INFO  - 1455443847: (BlockUtils.cpp:1721) Found 1 headers in db
-DEBUG - 1455443847: (Blockchain.cpp:211) Organizing chain w/ rebuild
-WARN  - 1455443847: (BlockUtils.cpp:1351) --- Fetching SSH summaries for 1342 registered addresses
-INFO  - 1455443847: (BlockUtils.cpp:1364) Left off at file 0, offset 0
-INFO  - 1455443847: (BlockUtils.cpp:1367) Reading headers and building chain...
-INFO  - 1455443847: (BlockUtils.cpp:1368) Starting at block file 0 offset 0
-INFO  - 1455443847: (BlockUtils.cpp:1370) Block height 0
-DEBUG - 1455444332: (Blockchain.cpp:211) Organizing chain w/ rebuild


And the last few lines of armorylog.txt

2016-02-14 09:56 (INFO) -- ArmoryQt.py:5887 - Dashboard switched to auto-InitSync
2016-02-14 09:57 (INFO) -- ArmoryQt.py:2525 - loadBlockchainIfNecessary
2016-02-14 09:57 (INFO) -- ArmoryQt.py:2546 - Setting netmode: 1
2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode
2016-02-14 09:57 (INFO) -- Networking.py:65 - Connection initiated.  Start handshake
2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode
2016-02-14 09:57 (INFO) -- Networking.py:156 - Received version message from peer:
2016-02-14 09:57 (INFO) -- Networking.py:157 -    Version:     70002
2016-02-14 09:57 (INFO) -- Networking.py:158 -    SubVersion:  /Satoshi:0.11.2/
2016-02-14 09:57 (INFO) -- Networking.py:159 -    TimeStamp:   1455443848
2016-02-14 09:57 (INFO) -- Networking.py:160 -    StartHeight: 398373
2016-02-14 09:57 (INFO) -- Networking.py:350 - Handshake finished, connection open!


Edit:  seems I get this error on starting from command line:

/usr/lib/armory/armoryengine/Transaction.py:2649: SyntaxWarning: import * only allowed at module level
  def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals):
(ERROR) announcefetch.py:287 - Unspecified error downloading URL
Traceback (most recent call last):
  File "/usr/lib/armory/announcefetch.py", line 277, in __fetchFile
    urlobj = urllib2.urlopen(url, timeout=CLI_OPTIONS.nettimeout)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 431, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 449, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1200, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 409, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib/python2.7/socket.py", line 480, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/usr/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
SSLError: ('The read operation timed out',)


Grateful for any help.

[goatpig]

None of these lines are relevant to your issue. You should start BitcoinQt manually and let it fully sync. In Armory, turn off auto bitcoind management then start it after BitcoinQt. Report what happens here.

[jwputrl]

Thanks.  Have done as you said:

• Started Bitcoin-qt directly
• Blockchain seemed synced, looking at the block height and activity.
• Started Armory first in offline mode, and unchecked box to stop Armory managing bitcoind
• Shutdown and restarted Armory in normal mode (leaving bitcoin-qt running). Now only 2 progress bars rather than 3
• Result appears the same - hanging on "Organizing Blockchain" - though so far it's only been 20 min
• Stuff in logs appears same as before, to my inexpert eye

(Noticed this in log, fwiw):
2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1323 - 8 attempts to load blockchain failed.  Remove mempool.bin.
2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1328 - File mempool.bin does not exist. Nothing deleted.

[goatpig]

Again these log lines are irrelevant. You should clean up your armorycpplog.txt or all personal info and pastebin it so I can look at it. In the mean time, delete the content of Armory's database folder and try again.

[jwputrl]

I have deleted contents of .armory/databases/, and restarted Armory.  Currently "organizing blockchain" as before.

The armorycpplog.txt is here:  http://pastebin.com/ruuVVeNK.

Thanks a lot for your help.

[goatpig]

-ERROR - 1455537157: (BlockWriteBatcher.cpp:980) No block in DB at height 195803
-INFO  - 1455537160: (BlockUtils.cpp:1518) Scanned Block range in 411.828s

That's your error. Your copy of the blockchain is corrupt, you need to redownload it.

[jwputrl]

OK.  Have deleted (actually moved) contents of .bitcoin folder - assuming will all be recreated when restart bitcoin-qt.  Rebuilding now and looks like it's going ok. 

Thanks a lot for your help.

[jwputrl]

-ERROR - 1455537157: (BlockWriteBatcher.cpp:980) No block in DB at height 195803
-INFO  - 1455537160: (BlockUtils.cpp:1518) Scanned Block range in 411.828s

That's your error. Your copy of the blockchain is corrupt, you need to redownload it.

Just to say this worked, and everything is good now. Thanks a lot for your help.

[Cobbler7]

Hi,

I am experiencing a similar problem: Armory hangs apparently endlessly in the "organizing blockchain"
stage, and never completes despite being run all day.

I am running BitcoinCore 0.21.1 and Armory 0.96.5.

This combination worked fine for me previously, but now something somewhere must have changed.
(I am thinking maybe from sudo apt-get update and upgrade etc.)

The only new clue (perhaps) as to what has changed is I get a new error message when I try
to quit the hung instance of Armory. This new error message says:

"ArmoryQt.py is not responding"
 
Any help or pointers would be most appreciated!

Thanks much,
Cobbler7

[nc50lc]

Any help or pointers would be most appreciated!

The pointers should be in the logs like goatpig requested in the previous replies.

"armorycpplog.txt" is in Armory's datadir; default paths: btcarmory.com/docs/pathing#armorydb

[Cobbler7]

Thank you NC501c,
I will check that out.
Cobbler7

[HCP]

armorycpplog.txt may not show anything of great interest... I'd check out armorylog.txt and dbLog.txt in the first instance.

If you want more assistance, you'd need to post the contents of those log files (in full) to pastebin.com and then link the generated pastebin URL here so we can view the logs.

If you don't see any obvious errors in the log files, then you might want to use the "Rebuild and Rescan Databases" option in Armory's "Help" menu and see if that helps.

[Cobbler7]

Hi,
I have a quite similar issue. I am by this point a fairly experienced BitcoinCore and Armory user (since 2014 or 2015). So I have tried my usual tricks
and tilts on the Armory pinball machine, but nothing seems to work now.

I have been running Ubuntu 20.4, Armory 0.96.5, and BitcoinCore 0.21.1, on a one year old System76 laptop with 64GB ram and an
Intel® Core™ i7-10870H CPU @ 2.20GHz × 16. Plus Verizon Fios for Internet. I don't think my system is underpowered to run a BitcoinCore/
Armory combination.

My specific problem is, BitcoinCore and Armory repeatedly hang/freezes after running for 2-6 hours. It hangs on either "Organizing Blockchain".
Or if that process completes, it hangs on "Scanning Tx".

I have run it (the boot up) maybe 20x plus over the past 3 weeks, to no avail.  One time, when I wasn't there/didn't see it, it showed me my Armory wallet balances,
but when I got there, the whole computer was frozen, so I couldn't use Armory or anything, and had to reboot. A couple of other times, it completed the "Scanning
Tx" part (at 100%), but then froze my whole computer, and did NOT show me my wallet balances. But that is as far as I have got, after 3 weeks.

I have tried the usual tilts, which have worked before. Like:
- rebuild and rescan databases
- letting, and not letting, Armory run bitcoind in the background
- running bitcoind and bitcoin-qt first, and then manually launching Armory

Also, when I run bitcoind or bitoin-qt manually, BitcoinCore 0.21.1 runs normally and updates the bitcoin blockchain apparently up to
current state.

So the problem appears to be the handshaking of Armory 0.96.5. with BitcoinCore.

I realized tonight that BitcoinCore has been upgraded to 0.22.0. I am currently running an update of blockchain with that. Maybe that will solve the problem,
But I dunno if that will work.

Any thoughts/ideas on what I can do to fix this? Any feedback most appreciated.
Thanks in advance, cobbler7.



Follow-up (by cobbler7): my Armory log files can be viewed at: https://github.com/truebar/pastebin

Thank you

[goatpig]

Hard to help you without logs.

[HCP]

Any thoughts/ideas on what I can do to fix this? Any feedback most appreciated.

As goatpig has mentioned, without being able to see the log files, it's going to be very difficult to understand what exactly is happening with your setup. Are there any obvious errors in the log files?


Also, no idea why your whole computer is "freezing"... That isn't a normal situation, so possibly indicative of something other than Armory or Bitcoin Core causing issues I've had Armory basically "freeze" (or seem like it has frozen) during the initial sync/scan... but the rest of the PC was functioning just fine.

[Cobbler7]

cobbler7:

My Armory logfiles can be viewed at:

https://github.com/truebar/pastebin

Any further advice most appreciated.
Thanks,
cobbler7

[goatpig]

1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned
2. Make sure there are no zombie ArmoryDB instances
3. Make sure nothing is listening on 9001

Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt.

[Cobbler7]

Thanks goatpig. I will try it.

Some further comments/questions below.

goatpig
Insert Quote
1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned

cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory.
No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before
I launched Armory. I don't know why Armory should be 100k blocks behind.

2. Make sure there are no zombie ArmoryDB instances
oat
cobbler7: pretty sure no zombies. I always hard boot before trying to re-run and reconstitute
bitcoincore/Armory.

3. Make sure nothing is listening on 9001

cobbler7: I tried netstat :9001, both with bitcoin/Armory, and then on clean boot without them. Both
netstat logs are now in my updated pastebin. I can't make heads or tails of these logs. Is something
there that shouldn't be there? And if so, what should I do about it?

Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt.

cobbler7: so I don't introduce anything new, can you tell me what the "clean" default terminal commands are to launch ArmoryDB, and then after that, ArmoryQt?

Thanks much.
cobbler7

[HCP]

cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory.
No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before
I launched Armory. I don't know why Armory should be 100k blocks behind.

Armory isn't 100k blocks behind... your bitcoind database is.

-INFO  - 21:24:04: (DatabaseBuilder.cpp:199) Reading headers from db
-INFO  - 21:24:07: (DatabaseBuilder.cpp:238) Found 626760 headers in db
...
-INFO  - 22:25:42: (DatabaseBuilder.cpp:134) scanning new blocks from #626760 to #626759
...

ArmoryDB reads your bitcoin block files directly... and these lines of debug from ArmoryDB show that ArmoryDB is only able to see 626760 blocks in your bitcoin block files.


If you're not running Bitcoin Core GUI, you might want to find the bitcoin "debug.log" file... and check in that to make sure that bitcoind is actually syncing properly... you should see "update tip" messages in debug.log if it is.... like this:

Code:

2021-12-06T04:20:24Z UpdateTip: new best=0000000000000000000652aa2bba5c40f43773386194967dc580446abbbd0be7 height=712815 version=0x20000004 log2_work=93.212644 tx=692645130 date='2021-12-06T04:20:16Z' progress=1.000000 cache=40.9MiB(219006txo)
2021-12-06T04:31:36Z UpdateTip: new best=000000000000000000011188e2434af142c05a91fdea0694e24631ea6a485fcc height=712816 version=0x3fffc004 log2_work=93.212656 tx=692646350 date='2021-12-06T04:31:25Z' progress=1.000000 cache=41.5MiB(223496txo)

Otherwise, it might also show errors as to why it isn't syncing properly.

[goatpig]

cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory.
No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before
I launched Armory. I don't know why Armory should be 100k blocks behind.

You have to start bitcoind and let it sync. 100k blocks short means you're missing some 25% of the chain? The later blocks are a lot heavier then the earlier ones. This would mean a few extra hours worth of bitcoind syncing to catch up.

cobbler7: I tried netstat :9001, both with bitcoin/Armory, and then on clean boot without them. Both
netstat logs are now in my updated pastebin. I can't make heads or tails of these logs. Is something
there that shouldn't be there? And if so, what should I do about it?

I checked the github page your linked in the previous thread, can't find any netstat output.

cobbler7: so I don't introduce anything new, can you tell me what the "clean" default terminal commands are to launch ArmoryDB, and then after that, ArmoryQt?

Type ArmoryDB or ArmoryQt in the terminal to start them. You are "introducing something new" in that approach in that you're not letting ArmoryQt control ArmoryDB, which reduces the degrees of freedom in the stack.