Upgraded to 0.96 building database takes long

[wachtwoord]

I'm using Bitcoin core version 0.13.0.

The blkxxxxx.dat files under Bitcoin data dir /blocks are 127-128 MB except for blk00000, blk00001, blk00002 and blk00003 (which are 1.95GB, 1.95GB, 1.95GB and 1.53 GB respectively). The last blk file is indeed blk00852.data

[1715073325]

1715073325

[1715073325]

1715073325

[goatpig]

You have an old copy of the chain, that explains it. Just start Armory the regular way now, see how that goes.

[wachtwoord]

You have an old copy of the chain, that explains it. Just start Armory the regular way now, see how that goes.

Okay, is having an old copy a problem? I always run a bit behind for security reasons (want to give it some time in the wild for bugs to be found). Version 0.13 is only 10 months old. Or is this because my blockchain is based on even older versions of Bitcoin and upgraded every time?

Meanwhile I started ArmoryQT by itself. I blazed passed the first steps (organizing chain, building database) and again is hanging my computer while scanning transaction history. Anything I can see or do?

Edit:

ArmoryDB memory usage is 3.6GB. Total memory usage is 93-99% ArmoryQT UI is hanging (like always, the little spinning banner is frozen)

Edit2:

Memory usage dropped to 1.4GB, 67% of total system memory usage.

[goatpig]

Okay, is having an old copy a problem? I always run a bit behind for security reasons (want to give it some time in the wild for bugs to be found). Version 0.13 is only 10 months old. Or is this because my blockchain is based on even older versions of Bitcoin and upgraded every time?

No that means you synced your chain with a super old version of Core and never had to nuke it since. Nothing wrong with that, but the size of the early block files is a tell.

Meanwhile I started ArmoryQT by itself. I blazed passed the first steps (organizing chain, building database) and again is hanging my computer while scanning transaction history. Anything I can see or do?

Edit:

ArmoryDB memory usage is 3.6GB. Total memory usage is 93-99% ArmoryQT UI is hanging (like always, the little spinning banner is frozen)

No this means your computer is so weak that you can 100% trigger this issue even on top of a completed build. I've got a VM gimped enough that I ran in the issue every other 5 scans now (I've never managed to reproduce the issue before), so I think I can get it under control. You'll have to wait for testing build with the fix or build it yourself once I push.

[wachtwoord]

Ok, so if I leave it running like this it will never finish?

Does this also have something to do with the weird big that started all this? (The, non-existing (?) outgoing transaction that Armory showed with the associated lower balance)

When are you planning to release the testing build? (I have to travel in a few days).

[goatpig]

Ok, so if I leave it running like this it will never finish?

Not when it triggers this issue.

Does this also have something to do with the weird big that started all this? (The, non-existing (?) outgoing transaction that Armory showed with the associated lower balance)

I don't know that they are related yet.

When are you planning to release the testing build? (I have to travel in a few days).

Once it's ready and tested. Can't tell you more.

[wachtwoord]

Ok thanks for all the help so far.

I found 2 more lines in my log that might help so I'm going to leave them here:

Code:

2017-06-18 19:53:30 (INFO) -- ArmoryQt.py:1890 - loadBlockchainIfNecessary
2017-06-18 19:53:30 (ERROR) -- ArmoryQt.py:1189 - 5 attempts to load blockchain failed.  Remove mempool.bin.

[wachtwoord]

Ok, I tried the latest #4 testing build last night until this morning. It ran for 12 hours and got to 4% (1.5 months left) and slowed my PC down to a very slow pace like before and was consuming most of the memory.

Does this mean the bug is still there or should I just let it run for >24 hours (or much more?)

BTW: I saw that the Armory installer is installing into "program files (x86)" folder by default while it's a 64 bit application. Is that correct?

[goatpig]

Ok, I tried the latest #4 testing build last night until this morning. It ran for 12 hours and got to 4% (1.5 months left) and slowed my PC down to a very slow pace like before and was consuming most of the memory.

Does this mean the bug is still there or should I just let it run for >24 hours (or much more?)

At this point it's probably still failing. Delete dbLog.txt and your databases folder, then start from scratch and post the log.

BTW: I saw that the Armory installer is installing into "program files (x86)" folder by default while it's a 64 bit application. Is that correct?

Legacy from the older versions. I'll fix that one of these days. All versions I've put out are x64 indeed.

[goatpig]

This is with 0.96.0.4?

[goatpig]

Are you building with --db-type==DB_BARE?

[wachtwoord]

Are you building with --db-type==DB_BARE?

I didn't build it but downloaded the 0.96.0.4 release from github.

If you mean whether I'm starting Armory with --db-type==DB_BARE as a command line argument, no I'm not. Should I try that? Do I need to delete the database folder again?

[goatpig]

Are you building with --db-type==DB_BARE?

Should I try that? Do I need to delete the database folder again?

Yes and yes.

[goatpig]

Try with --ram-usage=1 --thread-count=3. No need to wipe the DB this time.

[wachtwoord]

Try with --ram-usage=1 --thread-count=3. No need to wipe the DB this time.

It shows this pop-up:



But dblog.txt doesn't show anything. In Armorylog.txt I found:

2017-07-08 14:11:50 (ERROR) -- ArmoryQt.py:1819 - Failed to start Armory database: cannot concatenate 'str' and 'int' objects
Traceback (most recent call last):
  File "ArmoryQt.py", line 1797, in startArmoryDBIfNecessary
  File "SDM.pyc", line 392, in spawnDB
TypeError: cannot concatenate 'str' and 'int' objects

[goatpig]

Start the DB manually. Ignore the client for now.

[wachtwoord]

That works but it's done quite fast:

Log file opened at 14:44:32.000: XXX\Armory/dbLog.txt
-INFO  - 14:44:32.000: (..\main.cpp:32) Running on 3 threads
-INFO  - 14:44:32.000: (..\main.cpp:33) Ram usage level: 1
-INFO  - 14:44:32.015: (..\BlockUtils.cpp:908) blkfile dir: XXX\Bitcoin/blocks
-INFO  - 14:44:32.015: (..\BlockUtils.cpp:909) lmdb dir: XXX\Armory/databases
-INFO  - 14:44:32.031: (..\lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:44:32.124: (c:\users\goat\code\armory3\cppforswig\BDM_Server.h:249) Listening on port 9001
-INFO  - 14:44:34.215: (..\BlockUtils.cpp:1092) Executing: doInitialSyncOnLoad
-INFO  - 14:44:34.386: (..\DatabaseBuilder.cpp:190) Reading headers from db
-INFO  - 14:44:37.976: (..\DatabaseBuilder.cpp:229) Found 474938 headers in db
-INFO  - 14:44:40.127: (..\DatabaseBuilder.cpp:63) Rewinding 100 blocks
-INFO  - 14:44:40.127: (..\DatabaseBuilder.cpp:70) updating HEADERS db
-INFO  - 14:44:42.687: (..\DatabaseBuilder.cpp:272) parsed block file #873
-INFO  - 14:44:42.577: (..\DatabaseBuilder.cpp:482) Found next block after skipping 998141bytes
-INFO  - 14:44:43.201: (..\Blockchain.cpp:246) Organizing chain
-INFO  - 14:44:43.263: (..\Blockchain.cpp:360) Organized chain in 0.051s
-INFO  - 14:44:43.263: (..\DatabaseBuilder.cpp:75) updated HEADERS db in 3.139s
-INFO  - 14:44:43.279: (..\BDM_supportClasses.cpp:1873) Enabling zero-conf tracking

I ran it like: ArmoryDB.exe --datadir="XXX\Armory" --satoshi-datadir="XXX\Bitcoin" --db-type=DB_BARE --ram-usage=1 --thread-count=3

Next start the client?

[goatpig]

Next start the client?

Yes.

[wachtwoord]

It works! Thanks so much for the help!  

I tried to look up what DB_BARE means and I think it means I'm still a fill node (through bitcoin core) and everything is validated fully but some hash information is not stored making it some searches from Armory less rich. Is that correct? Can I just keep running like this?

Considering the difficulty I had with getting this to run does that mean relatively soon I won't be able to run Armory online on this computer or shouldn't I worry about that?


Edit:

When I close the client I see the following errors in the log (and it takes a while to close the client meanwhile it's not responding). Is that all ok?

Code:

-INFO  - 18:30:41.689: (..\BlockchainScanner.cpp:665) scanned from height #474800 to #474816
-ERROR - 18:36:38.775: (c:\users\goat\code\armory3\cppforswig\DataObject.h:286) exhausted entries in Arguments object
-INFO  - 18:38:49.956: (..\BDM_Server.cpp:1103) unregistered bdv: 23a775c696bc38acaafa
-ERROR - 18:38:49.159: (c:\users\goat\code\armory3\cppforswig\DataObject.h:286) exhausted entries in Arguments object
-ERROR - 18:38:52.905: (..\SocketObject.cpp:290) POLLERR error in readFromSocketThread
-ERROR - 18:38:52.920: (..\BitcoinP2P.cpp:1027) caught SocketError exception in processDataStackThread: POLLERR error in readFromSocketThread
-INFO  - 18:38:52.983: (..\BitcoinP2P.cpp:969) Disconnected from Bitcoin node

[goatpig]

I tried to look up what DB_BARE means and I think it means I'm still a fill node (through bitcoin core) and everything is validated fully but some hash information is not stored making it some searches from Armory less rich.

It means it does not maintain the dataset to resolve arbitrary tx hashes. On the GUI side that means you can't input tx nor fee of incoming transactions (people paying you). This is the state 0.94 was in.

Can I just keep running like this?

Well yes and no. You can keep on using this DB type but due to a mess up on my end, you have to specify the --db-type on each run. I fixed that issue but there's no build out with it yet. You're not supposed to be able to swap types once a DB is instantiated, but again, I messed up the check.

If you don't specify it, it will corrupt the DB on the next run. You can use armoryqt.conf to force that arg for now. Once the fix goes into the next build, you won't have to worry about the arg after bootstrapping.

Considering the difficulty I had with getting this to run does that mean relatively soon I won't be able to run Armory online on this computer or shouldn't I worry about that?

I can't tell. Clearly there is an issue that only exists with slow/old machines, since I never run into it, and all cases I can remember like yours are on old/weak machines. For now there seems to be a critical point at which a machine is too aged/weak and can't bootstrap the DB off of defaults. Until the point I figure out what's going on, you are stuck with using custom settings. What you ran is basically the fail safe setting.

I don't think the issue is related to block size but rather block load. Smaller blocks with lighter transactions (up around block #350k) are easy enough to parse. It's the recent stuff that's grinding some machines to a halt. Seeing you eventually got past it, I'm gonna guess your machine can keep on going.

When I close the client I see the following errors in the log (and it takes a while to close the client meanwhile it's not responding). Is that all ok?

False positives. A bunch of debug verbose I left there cause people on OSX are complaining about slow shutdowns. This helps me figure out what's going on. Ignore on anything but OSX.