Bitcoin Forum
May 18, 2021, 09:55:25 PM *
News: Latest Bitcoin Core release: 0.21.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 [2] 3 »  All
  Print  
Author Topic: Upgraded to 0.96 building database takes long  (Read 1582 times)
wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
June 18, 2017, 05:49:11 PM
 #21

I'm using Bitcoin core version 0.13.0.

The blkxxxxx.dat files under Bitcoin data dir /blocks are 127-128 MB except for blk00000, blk00001, blk00002 and blk00003 (which are 1.95GB, 1.95GB, 1.95GB and 1.53 GB respectively). The last blk file is indeed blk00852.data
1621374925
Hero Member
*
Offline Offline

Posts: 1621374925

View Profile Personal Message (Offline)

Ignore
1621374925
Reply with quote  #2

1621374925
Report to moderator
1621374925
Hero Member
*
Offline Offline

Posts: 1621374925

View Profile Personal Message (Offline)

Ignore
1621374925
Reply with quote  #2

1621374925
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1621374925
Hero Member
*
Offline Offline

Posts: 1621374925

View Profile Personal Message (Offline)

Ignore
1621374925
Reply with quote  #2

1621374925
Report to moderator
1621374925
Hero Member
*
Offline Offline

Posts: 1621374925

View Profile Personal Message (Offline)

Ignore
1621374925
Reply with quote  #2

1621374925
Report to moderator
1621374925
Hero Member
*
Offline Offline

Posts: 1621374925

View Profile Personal Message (Offline)

Ignore
1621374925
Reply with quote  #2

1621374925
Report to moderator
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
June 18, 2017, 05:52:03 PM
 #22

You have an old copy of the chain, that explains it. Just start Armory the regular way now, see how that goes.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
June 18, 2017, 05:57:01 PM
Last edit: June 18, 2017, 06:14:00 PM by wachtwoord
 #23

You have an old copy of the chain, that explains it. Just start Armory the regular way now, see how that goes.

Okay, is having an old copy a problem? I always run a bit behind for security reasons (want to give it some time in the wild for bugs to be found). Version 0.13 is only 10 months old. Or is this because my blockchain is based on even older versions of Bitcoin and upgraded every time?

Meanwhile I started ArmoryQT by itself. I blazed passed the first steps (organizing chain, building database) and again is hanging my computer while scanning transaction history. Anything I can see or do?

Edit:

ArmoryDB memory usage is 3.6GB. Total memory usage is 93-99% ArmoryQT UI is hanging (like always, the little spinning banner is frozen)

Edit2:

Memory usage dropped to 1.4GB, 67% of total system memory usage.
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
June 18, 2017, 06:13:12 PM
 #24

Okay, is having an old copy a problem? I always run a bit behind for security reasons (want to give it some time in the wild for bugs to be found). Version 0.13 is only 10 months old. Or is this because my blockchain is based on even older versions of Bitcoin and upgraded every time?

No that means you synced your chain with a super old version of Core and never had to nuke it since. Nothing wrong with that, but the size of the early block files is a tell.

Quote
Meanwhile I started ArmoryQT by itself. I blazed passed the first steps (organizing chain, building database) and again is hanging my computer while scanning transaction history. Anything I can see or do?

Edit:

ArmoryDB memory usage is 3.6GB. Total memory usage is 93-99% ArmoryQT UI is hanging (like always, the little spinning banner is frozen)

No this means your computer is so weak that you can 100% trigger this issue even on top of a completed build. I've got a VM gimped enough that I ran in the issue every other 5 scans now (I've never managed to reproduce the issue before), so I think I can get it under control. You'll have to wait for testing build with the fix or build it yourself once I push.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
June 18, 2017, 06:17:29 PM
 #25

Ok, so if I leave it running like this it will never finish?

Does this also have something to do with the weird big that started all this? (The, non-existing (?) outgoing transaction that Armory showed with the associated lower balance)

When are you planning to release the testing build? (I have to travel in a few days).
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
June 18, 2017, 06:26:19 PM
 #26

Ok, so if I leave it running like this it will never finish?

Not when it triggers this issue.

Quote
Does this also have something to do with the weird big that started all this? (The, non-existing (?) outgoing transaction that Armory showed with the associated lower balance)


I don't know that they are related yet.

Quote
When are you planning to release the testing build? (I have to travel in a few days).

Once it's ready and tested. Can't tell you more.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
June 18, 2017, 06:34:00 PM
 #27

Ok thanks for all the help so far.

I found 2 more lines in my log that might help so I'm going to leave them here:

Code:
2017-06-18 19:53:30 (INFO) -- ArmoryQt.py:1890 - loadBlockchainIfNecessary
2017-06-18 19:53:30 (ERROR) -- ArmoryQt.py:1189 - 5 attempts to load blockchain failed.  Remove mempool.bin.
wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
July 06, 2017, 12:55:21 PM
 #28

Ok, I tried the latest #4 testing build last night until this morning. It ran for 12 hours and got to 4% (1.5 months left) and slowed my PC down to a very slow pace like before and was consuming most of the memory.

Does this mean the bug is still there or should I just let it run for >24 hours (or much more?)

BTW: I saw that the Armory installer is installing into "program files (x86)" folder by default while it's a 64 bit application. Is that correct?
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 06, 2017, 02:12:42 PM
 #29

Ok, I tried the latest #4 testing build last night until this morning. It ran for 12 hours and got to 4% (1.5 months left) and slowed my PC down to a very slow pace like before and was consuming most of the memory.

Does this mean the bug is still there or should I just let it run for >24 hours (or much more?)

At this point it's probably still failing. Delete dbLog.txt and your databases folder, then start from scratch and post the log.

Quote
BTW: I saw that the Armory installer is installing into "program files (x86)" folder by default while it's a 64 bit application. Is that correct?

Legacy from the older versions. I'll fix that one of these days. All versions I've put out are x64 indeed.

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 06, 2017, 08:54:21 PM
 #30

This is with 0.96.0.4?


goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 07, 2017, 02:42:32 PM
 #31

Are you building with --db-type==DB_BARE?

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
July 07, 2017, 04:22:11 PM
 #32

Are you building with --db-type==DB_BARE?

I didn't build it but downloaded the 0.96.0.4 release from github.

If you mean whether I'm starting Armory with --db-type==DB_BARE as a command line argument, no I'm not. Should I try that? Do I need to delete the database folder again?
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 07, 2017, 05:29:03 PM
 #33

Are you building with --db-type==DB_BARE?
Should I try that? Do I need to delete the database folder again?

Yes and yes.

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 08, 2017, 10:16:43 AM
 #34

Try with --ram-usage=1 --thread-count=3. No need to wipe the DB this time.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
July 08, 2017, 12:18:31 PM
 #35

Try with --ram-usage=1 --thread-count=3. No need to wipe the DB this time.

It shows this pop-up:



But dblog.txt doesn't show anything. In Armorylog.txt I found:

2017-07-08 14:11:50 (ERROR) -- ArmoryQt.py:1819 - Failed to start Armory database: cannot concatenate 'str' and 'int' objects
Traceback (most recent call last):
  File "ArmoryQt.py", line 1797, in startArmoryDBIfNecessary
  File "SDM.pyc", line 392, in spawnDB
TypeError: cannot concatenate 'str' and 'int' objects

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 08, 2017, 12:25:19 PM
 #36

Start the DB manually. Ignore the client for now.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
July 08, 2017, 12:49:04 PM
 #37

That works but it's done quite fast:

Quote
Log file opened at 14:44:32.000: XXX\Armory/dbLog.txt
-INFO  - 14:44:32.000: (..\main.cpp:32) Running on 3 threads
-INFO  - 14:44:32.000: (..\main.cpp:33) Ram usage level: 1
-INFO  - 14:44:32.015: (..\BlockUtils.cpp:908) blkfile dir: XXX\Bitcoin/blocks
-INFO  - 14:44:32.015: (..\BlockUtils.cpp:909) lmdb dir: XXX\Armory/databases
-INFO  - 14:44:32.031: (..\lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 14:44:32.124: (c:\users\goat\code\armory3\cppforswig\BDM_Server.h:249) Listening on port 9001
-INFO  - 14:44:34.215: (..\BlockUtils.cpp:1092) Executing: doInitialSyncOnLoad
-INFO  - 14:44:34.386: (..\DatabaseBuilder.cpp:190) Reading headers from db
-INFO  - 14:44:37.976: (..\DatabaseBuilder.cpp:229) Found 474938 headers in db
-INFO  - 14:44:40.127: (..\DatabaseBuilder.cpp:63) Rewinding 100 blocks
-INFO  - 14:44:40.127: (..\DatabaseBuilder.cpp:70) updating HEADERS db
-INFO  - 14:44:42.687: (..\DatabaseBuilder.cpp:272) parsed block file #873
-INFO  - 14:44:42.577: (..\DatabaseBuilder.cpp:482) Found next block after skipping 998141bytes
-INFO  - 14:44:43.201: (..\Blockchain.cpp:246) Organizing chain
-INFO  - 14:44:43.263: (..\Blockchain.cpp:360) Organized chain in 0.051s
-INFO  - 14:44:43.263: (..\DatabaseBuilder.cpp:75) updated HEADERS db in 3.139s
-INFO  - 14:44:43.279: (..\BDM_supportClasses.cpp:1873) Enabling zero-conf tracking

I ran it like: ArmoryDB.exe --datadir="XXX\Armory" --satoshi-datadir="XXX\Bitcoin" --db-type=DB_BARE --ram-usage=1 --thread-count=3

Next start the client?
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 08, 2017, 02:19:26 PM
 #38

Next start the client?

Yes.

wachtwoord
Legendary
*
Offline Offline

Activity: 2100
Merit: 1044


View Profile
July 08, 2017, 04:34:05 PM
 #39

It works! Thanks so much for the help!  Smiley

I tried to look up what DB_BARE means and I think it means I'm still a fill node (through bitcoin core) and everything is validated fully but some hash information is not stored making it some searches from Armory less rich. Is that correct? Can I just keep running like this?

Considering the difficulty I had with getting this to run does that mean relatively soon I won't be able to run Armory online on this computer or shouldn't I worry about that?


Edit:

When I close the client I see the following errors in the log (and it takes a while to close the client meanwhile it's not responding). Is that all ok?

Code:
-INFO  - 18:30:41.689: (..\BlockchainScanner.cpp:665) scanned from height #474800 to #474816
-ERROR - 18:36:38.775: (c:\users\goat\code\armory3\cppforswig\DataObject.h:286) exhausted entries in Arguments object
-INFO  - 18:38:49.956: (..\BDM_Server.cpp:1103) unregistered bdv: 23a775c696bc38acaafa
-ERROR - 18:38:49.159: (c:\users\goat\code\armory3\cppforswig\DataObject.h:286) exhausted entries in Arguments object
-ERROR - 18:38:52.905: (..\SocketObject.cpp:290) POLLERR error in readFromSocketThread
-ERROR - 18:38:52.920: (..\BitcoinP2P.cpp:1027) caught SocketError exception in processDataStackThread: POLLERR error in readFromSocketThread
-INFO  - 18:38:52.983: (..\BitcoinP2P.cpp:969) Disconnected from Bitcoin node
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 2870
Merit: 1199

Armory Developer


View Profile
July 08, 2017, 05:03:46 PM
Merited by wachtwoord (35)
 #40

I tried to look up what DB_BARE means and I think it means I'm still a fill node (through bitcoin core) and everything is validated fully but some hash information is not stored making it some searches from Armory less rich.

It means it does not maintain the dataset to resolve arbitrary tx hashes. On the GUI side that means you can't input tx nor fee of incoming transactions (people paying you). This is the state 0.94 was in.

Quote
Can I just keep running like this?

Well yes and no. You can keep on using this DB type but due to a mess up on my end, you have to specify the --db-type on each run. I fixed that issue but there's no build out with it yet. You're not supposed to be able to swap types once a DB is instantiated, but again, I messed up the check.

If you don't specify it, it will corrupt the DB on the next run. You can use armoryqt.conf to force that arg for now. Once the fix goes into the next build, you won't have to worry about the arg after bootstrapping.

Quote
Considering the difficulty I had with getting this to run does that mean relatively soon I won't be able to run Armory online on this computer or shouldn't I worry about that?

I can't tell. Clearly there is an issue that only exists with slow/old machines, since I never run into it, and all cases I can remember like yours are on old/weak machines. For now there seems to be a critical point at which a machine is too aged/weak and can't bootstrap the DB off of defaults. Until the point I figure out what's going on, you are stuck with using custom settings. What you ran is basically the fail safe setting.

I don't think the issue is related to block size but rather block load. Smaller blocks with lighter transactions (up around block #350k) are easy enough to parse. It's the recent stuff that's grinding some machines to a halt. Seeing you eventually got past it, I'm gonna guess your machine can keep on going.

Quote
When I close the client I see the following errors in the log (and it takes a while to close the client meanwhile it's not responding). Is that all ok?

False positives. A bunch of debug verbose I left there cause people on OSX are complaining about slow shutdowns. This helps me figure out what's going on. Ignore on anything but OSX.

Pages: « 1 [2] 3 »  All
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!