Title: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: jwputrl on February 14, 2016, 10:45:36 AM I'm trying to install Armory in Ubuntu 15.04, using the version at https://github.com/goatpig/BitcoinArmory/releases/tag/v0.93.3 (https://github.com/goatpig/BitcoinArmory/releases/tag/v0.93.3). I have a few wallets already, from an installation on a laptop I had stolen. So I'm trying to get these coins back.
Dashboard shows "Synchronizing with Network" complete. It has finished "reading block headers", but now hangs on Organizing Blockchain. It's been that way for maybe 18h overall (with a couple of attempted restarts in between). Might be relevant: I tried to give the new install a headstart by copying over some of the database files from a backup I had of the old installation (and also to the .bitcoin installation). Also: when I quit, it hangs on "Preparing to shut down", and I have to quit again. When that happens I get the following repeated many times in .armory/armorylog.txt: 2016-02-14 09:52 (WARNING) -- SDM.py:768 - Overriding not-available state. This should happen 0-5 times Heres the cpplog.txt after re-starting, up to present moment where it hangs. Log file opened at 1455443804: ~/.armory/armorycpplog.txt -INFO - 1455443847: (BlockUtils.cpp:918) blkfile dir: /home/gav/.bitcoin/blocks -INFO - 1455443847: (BlockUtils.cpp:919) lmdb dir: /home/gav/.armory/databases -INFO - 1455443847: (lmdb_wrapper.cpp:478) Opening databases... -INFO - 1455443847: (BlockUtils.cpp:1250) Executing: doInitialSyncOnLoad -INFO - 1455443847: (BlockUtils.cpp:1321) Total number of blk*.dat files: 444 -INFO - 1455443847: (BlockUtils.cpp:1322) Total blockchain bytes: 59,390,537,918 -INFO - 1455443847: (BlockUtils.cpp:1695) Reading headers from db -WARN - 1455443847: (lmdb_wrapper.cpp:1478) No headers in DB yet! -INFO - 1455443847: (BlockUtils.cpp:1721) Found 1 headers in db -DEBUG - 1455443847: (Blockchain.cpp:211) Organizing chain w/ rebuild -WARN - 1455443847: (BlockUtils.cpp:1351) --- Fetching SSH summaries for 1342 registered addresses -INFO - 1455443847: (BlockUtils.cpp:1364) Left off at file 0, offset 0 -INFO - 1455443847: (BlockUtils.cpp:1367) Reading headers and building chain... -INFO - 1455443847: (BlockUtils.cpp:1368) Starting at block file 0 offset 0 -INFO - 1455443847: (BlockUtils.cpp:1370) Block height 0 -DEBUG - 1455444332: (Blockchain.cpp:211) Organizing chain w/ rebuild And the last few lines of armorylog.txt 2016-02-14 09:56 (INFO) -- ArmoryQt.py:5887 - Dashboard switched to auto-InitSync 2016-02-14 09:57 (INFO) -- ArmoryQt.py:2525 - loadBlockchainIfNecessary 2016-02-14 09:57 (INFO) -- ArmoryQt.py:2546 - Setting netmode: 1 2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode 2016-02-14 09:57 (INFO) -- Networking.py:65 - Connection initiated. Start handshake 2016-02-14 09:57 (INFO) -- ArmoryQt.py:6010 - Dashboard switched to "Scanning" mode 2016-02-14 09:57 (INFO) -- Networking.py:156 - Received version message from peer: 2016-02-14 09:57 (INFO) -- Networking.py:157 - Version: 70002 2016-02-14 09:57 (INFO) -- Networking.py:158 - SubVersion: /Satoshi:0.11.2/ 2016-02-14 09:57 (INFO) -- Networking.py:159 - TimeStamp: 1455443848 2016-02-14 09:57 (INFO) -- Networking.py:160 - StartHeight: 398373 2016-02-14 09:57 (INFO) -- Networking.py:350 - Handshake finished, connection open! Edit: seems I get this error on starting from command line: /usr/lib/armory/armoryengine/Transaction.py:2649: SyntaxWarning: import * only allowed at module level def PyCreateAndSignTx_old(srcTxOuts, dstAddrsVals): (ERROR) announcefetch.py:287 - Unspecified error downloading URL Traceback (most recent call last): File "/usr/lib/armory/announcefetch.py", line 277, in __fetchFile urlobj = urllib2.urlopen(url, timeout=CLI_OPTIONS.nettimeout) File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 449, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open context=self._context) File "/usr/lib/python2.7/urllib2.py", line 1200, in do_open r = h.getresponse(buffering=True) File "/usr/lib/python2.7/httplib.py", line 1127, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 453, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 409, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 480, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 734, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 621, in read v = self._sslobj.read(len or 1024) SSLError: ('The read operation timed out',) Grateful for any help. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on February 14, 2016, 03:23:08 PM None of these lines are relevant to your issue. You should start BitcoinQt manually and let it fully sync. In Armory, turn off auto bitcoind management then start it after BitcoinQt. Report what happens here.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: jwputrl on February 14, 2016, 05:10:02 PM Thanks. Have done as you said:
(Noticed this in log, fwiw): 2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1323 - 8 attempts to load blockchain failed. Remove mempool.bin. 2016-02-14 16:38 (ERROR) -- ArmoryQt.py:1328 - File mempool.bin does not exist. Nothing deleted. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on February 14, 2016, 05:54:56 PM Again these log lines are irrelevant. You should clean up your armorycpplog.txt or all personal info and pastebin it so I can look at it. In the mean time, delete the content of Armory's database folder and try again.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: jwputrl on February 15, 2016, 09:17:56 PM I have deleted contents of .armory/databases/, and restarted Armory. Currently "organizing blockchain" as before.
The armorycpplog.txt is here: http://pastebin.com/ruuVVeNK (http://pastebin.com/ruuVVeNK). Thanks a lot for your help. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on February 15, 2016, 09:27:27 PM -ERROR - 1455537157: (BlockWriteBatcher.cpp:980) No block in DB at height 195803
-INFO - 1455537160: (BlockUtils.cpp:1518) Scanned Block range in 411.828s That's your error. Your copy of the blockchain is corrupt, you need to redownload it. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: jwputrl on February 15, 2016, 09:42:44 PM OK. Have deleted (actually moved) contents of .bitcoin folder - assuming will all be recreated when restart bitcoin-qt. Rebuilding now and looks like it's going ok.
Thanks a lot for your help. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: jwputrl on February 21, 2016, 09:28:02 AM -ERROR - 1455537157: (BlockWriteBatcher.cpp:980) No block in DB at height 195803 -INFO - 1455537160: (BlockUtils.cpp:1518) Scanned Block range in 411.828s That's your error. Your copy of the blockchain is corrupt, you need to redownload it. Just to say this worked, and everything is good now. Thanks a lot for your help. Title: Re: Armory hangs on "Organizing Blockchain" Post by: Cobbler7 on August 23, 2021, 03:29:10 AM Hi,
I am experiencing a similar problem: Armory hangs apparently endlessly in the "organizing blockchain" stage, and never completes despite being run all day. I am running BitcoinCore 0.21.1 and Armory 0.96.5. This combination worked fine for me previously, but now something somewhere must have changed. (I am thinking maybe from sudo apt-get update and upgrade etc.) The only new clue (perhaps) as to what has changed is I get a new error message when I try to quit the hung instance of Armory. This new error message says: "ArmoryQt.py is not responding" Any help or pointers would be most appreciated! Thanks much, Cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" Post by: nc50lc on August 23, 2021, 04:06:57 AM Any help or pointers would be most appreciated! The pointers should be in the logs like goatpig requested in the previous replies."armorycpplog.txt" is in Armory's datadir; default paths: btcarmory.com/docs/pathing#armorydb (https://btcarmory.com/docs/pathing#armorydb) Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on August 23, 2021, 05:12:03 AM Thank you NC501c,
I will check that out. Cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: HCP on August 23, 2021, 11:34:42 PM armorycpplog.txt may not show anything of great interest... I'd check out armorylog.txt and dbLog.txt in the first instance.
If you want more assistance, you'd need to post the contents of those log files (in full) to pastebin.com and then link the generated pastebin URL here so we can view the logs. If you don't see any obvious errors in the log files, then you might want to use the "Rebuild and Rescan Databases" option in Armory's "Help" menu and see if that helps. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 02, 2021, 07:31:45 AM Hi,
I have a quite similar issue. I am by this point a fairly experienced BitcoinCore and Armory user (since 2014 or 2015). So I have tried my usual tricks and tilts on the Armory pinball machine, but nothing seems to work now. I have been running Ubuntu 20.4, Armory 0.96.5, and BitcoinCore 0.21.1, on a one year old System76 laptop with 64GB ram and an Intel® Core™ i7-10870H CPU @ 2.20GHz × 16. Plus Verizon Fios for Internet. I don't think my system is underpowered to run a BitcoinCore/ Armory combination. My specific problem is, BitcoinCore and Armory repeatedly hang/freezes after running for 2-6 hours. It hangs on either "Organizing Blockchain". Or if that process completes, it hangs on "Scanning Tx". I have run it (the boot up) maybe 20x plus over the past 3 weeks, to no avail. One time, when I wasn't there/didn't see it, it showed me my Armory wallet balances, but when I got there, the whole computer was frozen, so I couldn't use Armory or anything, and had to reboot. A couple of other times, it completed the "Scanning Tx" part (at 100%), but then froze my whole computer, and did NOT show me my wallet balances. But that is as far as I have got, after 3 weeks. I have tried the usual tilts, which have worked before. Like: - rebuild and rescan databases - letting, and not letting, Armory run bitcoind in the background - running bitcoind and bitcoin-qt first, and then manually launching Armory Also, when I run bitcoind or bitoin-qt manually, BitcoinCore 0.21.1 runs normally and updates the bitcoin blockchain apparently up to current state. So the problem appears to be the handshaking of Armory 0.96.5. with BitcoinCore. I realized tonight that BitcoinCore has been upgraded to 0.22.0. I am currently running an update of blockchain with that. Maybe that will solve the problem, But I dunno if that will work. Any thoughts/ideas on what I can do to fix this? Any feedback most appreciated. Thanks in advance, cobbler7. Follow-up (by cobbler7): my Armory log files can be viewed at: https://github.com/truebar/pastebin Thank you Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 02, 2021, 07:46:25 AM Hard to help you without logs.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: HCP on December 02, 2021, 10:44:47 AM Any thoughts/ideas on what I can do to fix this? Any feedback most appreciated. As goatpig has mentioned, without being able to see the log files, it's going to be very difficult to understand what exactly is happening with your setup. Are there any obvious errors in the log files? ???Also, no idea why your whole computer is "freezing"... That isn't a normal situation, so possibly indicative of something other than Armory or Bitcoin Core causing issues ??? I've had Armory basically "freeze" (or seem like it has frozen) during the initial sync/scan... but the rest of the PC was functioning just fine. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 03, 2021, 07:13:40 AM cobbler7:
My Armory logfiles can be viewed at: https://github.com/truebar/pastebin Any further advice most appreciated. Thanks, cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 03, 2021, 08:02:48 AM 1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned
2. Make sure there are no zombie ArmoryDB instances 3. Make sure nothing is listening on 9001 Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 05, 2021, 09:15:49 PM Thanks goatpig. I will try it.
Some further comments/questions below. goatpig Insert Quote 1. Run bitcoind manually, you're behind some 100k blocks as far as Armory is concerned cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory. No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before I launched Armory. I don't know why Armory should be 100k blocks behind. 2. Make sure there are no zombie ArmoryDB instances oat cobbler7: pretty sure no zombies. I always hard boot before trying to re-run and reconstitute bitcoincore/Armory. 3. Make sure nothing is listening on 9001 cobbler7: I tried netstat :9001, both with bitcoin/Armory, and then on clean boot without them. Both netstat logs are now in my updated pastebin. I can't make heads or tails of these logs. Is something there that shouldn't be there? And if so, what should I do about it? Once bitcoind is caught up, start ArmoryDB from the terminal, let it catch up, post the log file back here if it doesn't get beyond block 700k. If ArmoryDB gets as far as "enabling zero conf parser", you can start ArmoryQt. cobbler7: so I don't introduce anything new, can you tell me what the "clean" default terminal commands are to launch ArmoryDB, and then after that, ArmoryQt? Thanks much. cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: HCP on December 06, 2021, 04:35:57 AM cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory. Armory isn't 100k blocks behind... your bitcoind database is. No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before I launched Armory. I don't know why Armory should be 100k blocks behind. Quote -INFO - 21:24:04: (DatabaseBuilder.cpp:199) Reading headers from db ArmoryDB reads your bitcoin block files directly... and these lines of debug from ArmoryDB show that ArmoryDB is only able to see 626760 blocks in your bitcoin block files.-INFO - 21:24:07: (DatabaseBuilder.cpp:238) Found 626760 headers in db ... -INFO - 22:25:42: (DatabaseBuilder.cpp:134) scanning new blocks from #626760 to #626759 ... If you're not running Bitcoin Core GUI, you might want to find the bitcoin "debug.log" file... and check in that to make sure that bitcoind is actually syncing properly... you should see "update tip" messages in debug.log if it is.... like this: Code: 2021-12-06T04:20:24Z UpdateTip: new best=0000000000000000000652aa2bba5c40f43773386194967dc580446abbbd0be7 height=712815 version=0x20000004 log2_work=93.212644 tx=692645130 date='2021-12-06T04:20:16Z' progress=1.000000 cache=40.9MiB(219006txo) Otherwise, it might also show errors as to why it isn't syncing properly. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 06, 2021, 07:30:11 AM cobbler7: I have run bitcoind and bitcoin-qt manually maybe 20+ times, then launched Armory. No luck, it always hangs. I am sure at least a half dozen times, blockchain was current state before I launched Armory. I don't know why Armory should be 100k blocks behind. You have to start bitcoind and let it sync. 100k blocks short means you're missing some 25% of the chain? The later blocks are a lot heavier then the earlier ones. This would mean a few extra hours worth of bitcoind syncing to catch up. Quote cobbler7: I tried netstat :9001, both with bitcoin/Armory, and then on clean boot without them. Both netstat logs are now in my updated pastebin. I can't make heads or tails of these logs. Is something there that shouldn't be there? And if so, what should I do about it? I checked the github page your linked in the previous thread, can't find any netstat output. Quote cobbler7: so I don't introduce anything new, can you tell me what the "clean" default terminal commands are to launch ArmoryDB, and then after that, ArmoryQt? Type ArmoryDB or ArmoryQt in the terminal to start them. You are "introducing something new" in that approach in that you're not letting ArmoryQt control ArmoryDB, which reduces the degrees of freedom in the stack. Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 07, 2021, 10:39:22 PM Thank s much goatpig and HCP for your continued help.
I have updated my pastebin https://github.com/truebar/pastebin with the following 4 files: -netstat run 1: scanning port 9001 with BOTH Armory and bitcoind running -netstat run 2: scanning port 9001 with NEITHER Armory NOR bitcoind running - armory debug.log - screenshot of latest bitcoind terminal run Re netstat, I still can't tell from these two files if anything is there that shouldn't be, or otherwise is interfering with either Armory or Bitcoin. Re armory debug.log, it is much like terminal output of bitcoind. Nothing jumps out to me as a problem. Re bitcoind run screenshot, this appears to me to indicate that the local bitcoin blockchain copy is complete, or near complete, and not 100k blocks short of being complete. Any further ideas as to what is going wrong in making Armory hang on "Organizing Blockchain" or "Scanning Tx"? One further background fact, in case it matters. In doing re-installs of Armory and BitcoinCore, I have been making use of a backup copy of the bitcoin blockchain that I keep on a external USB hard drive. (This is specifically the contents of the .bitcoin directory.) For example, I get a new computer, install BitcoinCore on it, but don't yet run it. Instead, I copy the contents of my back up .bitcoin directory from the USB external drive to the .bitcoin directory on my new PC. Only then do i run BitcoinCore Qt. Does/might this matter? Thanks much, cobblert7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 09, 2021, 06:58:19 PM Thanks goatpig and HCP -- this updated directory below should have two netstat files, debug.log, and bitcoind last run terminal log. Any ideas much appreciated. Thanks, cobbler7 https://github.com/truebar/pastebin Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 10, 2021, 07:39:45 AM Can you please make it all text files. ODT I can't read in browser =/. I don't really want to have to download users' log files locally.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 10, 2021, 08:13:53 AM I uploaded txt files of the odt ones to the pastebin, did you get/see them?
Thanks, cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 10, 2021, 10:28:49 AM From the the terminal, run ArmoryDB. Post the output from the terminal. You don't need bitcoind running for that.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 16, 2021, 06:02:41 AM Hi goatpig,
Please see terminal output after running ArmoryDB at terminal. File is ArmoryBD_log.txt https://github.com/truebar/pastebin The only thing I see in this I see of possible interest is: -INFO - 00:06:28: (BDM_Server.h:263) Listening on port 9001 Any ideas? Thanks much. cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 16, 2021, 05:05:10 PM It looks like it updated the chain but has had time to do much more. How long did you let it run?
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 19, 2021, 10:08:46 AM Hi goatpig,
I think I let it run about 18 hours. It appeared to have concluded a process, and I created another log file - ArmoryDB_log2.txt in my updated pastebin. https://github.com/truebar/pastebin The last line in the terminal was: -INFO - 01:10:12: (BDM_supportClasses.cpp:1891) Enabling zero-conf tracking Is this Ok, along with the early log line: -INFO - 00:06:28: (BDM_Server.h:263) Listening on port 9001 Also, I noticed that running ArmoryDB on the terminal did not hang in the usual 2-4 hours as it does with Armory GUI running and bitcoind running in the background. It just kept running on without stopping. Lastly, when I killed the terminal running ArmoryDB, it looked done, but there was still some kind of process running in it, though nothing new came on screen. Also updated in my pastebin is a new file armory_log3.txt. On December 17th, I ran first BitcoinCore Qt, and let up update the blockchain. Then I launched Armory. Armory still froze, but it got farther than usual. The Scanning Transactions progress bar went all the way to 100%, and the timer said only 1 second left. But at that point, Armory froze. Any ideas from this as to get Armory back up and running fully together with bitcoind? It has been a pretty frustrating process so far. Thanks much, cobbler7 Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: goatpig on December 23, 2021, 09:03:00 AM It looks like the DB is sane, you should start ArmoryQt now while ArmoryDB is running.
Title: Re: Armory hangs on "Organizing Blockchain" (>12h). Grateful for any help. Post by: Cobbler7 on December 25, 2021, 09:27:21 AM Thank - I will try it!
|