Bitcoin Forum
May 06, 2024, 09:14:36 AM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Armory 0.95.1 out of memory rebuilding database  (Read 851 times)
carl.byington (OP)
Newbie
*
Offline Offline

Activity: 13
Merit: 0


View Profile
January 26, 2017, 11:32:26 PM
 #1

Running on Fedora 25, python 2.7, the box has 6GB memory, 6GB swap.

Armory 0.95.1 was working months ago, but failed to start today. The underlying bitcoind seems to be happy:

bitcoin-cli getinfo
{
  "version": 130100,
  "protocolversion": 70014,
  "walletversion": 60000,
  "balance": 0.00000000,
  "blocks": 450159,
  "timeoffset": 0,
  "connections": 124,
  "proxy": "",
  "difficulty": 392963262344.3704,
  "testnet": false,
  "keypoololdest": 1448647753,
  "keypoolsize": 105,
  "paytxfee": 0.00000000,
  "relayfee": 0.00001000,
  "errors": ""
}

Armory launched ArmoryDB, with DB_FULL. Based on other messages here, I tried to reduce the memory footprint with:

/usr/bin/ArmoryDB --db-type="DB_BARE" --spawnId="6RYQUcvejFYTxiPjDGhXABeP2pppQQyMXAv3bouBQNtE" --satoshi-datadir="/home/carl/.bitcoin/blocks" --datadir="/home/carl/.armory/" --dbdir="/home/carl/.armory/databases" --rebuild --ram-usage=1

That still dies with about 8GB resident.

dbLog.txt is the same as the console output:

Log file opened at 1485470827: /home/carl/.armory/dbLog.txt
-INFO  - 1485470827: (main.cpp:23) Running on 4 threads
-INFO  - 1485470827: (main.cpp:24) Ram usage level: 1
-INFO  - 1485470827: (BlockUtils.cpp:1338) blkfile dir: /home/carl/.bitcoin/blocks
-INFO  - 1485470827: (BlockUtils.cpp:1339) lmdb dir: /home/carl/.armory/databases
-INFO  - 1485470827: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 1485470827: (BlockUtils.cpp:1540) Executing: doInitialSyncOnLoad_Rebuild
-INFO  - 1485470827: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 1485470827: (BitcoinP2P.cpp:783) Connected to Bitcoin node
-INFO  - 1485470827: (DatabaseBuilder.cpp:169) Reading headers from db
-WARN  - 1485470827: (lmdb_wrapper.cpp:1175) No headers in DB yet!
-INFO  - 1485470827: (DatabaseBuilder.cpp:208) Found 1 headers in db
-INFO  - 1485470827: (DatabaseBuilder.cpp:51) updating HEADERS db
-INFO  - 1485470829: (DatabaseBuilder.cpp:268) parsed block file #1
-INFO  - 1485470829: (DatabaseBuilder.cpp:268) parsed block file #2
-INFO  - 1485470830: (DatabaseBuilder.cpp:268) parsed block file #3
....
-INFO  - 1485470944: (DatabaseBuilder.cpp:268) parsed block file #45
-INFO  - 1485470950: (DatabaseBuilder.cpp:268) parsed block file #47
-INFO  - 1485470974: (DatabaseBuilder.cpp:268) parsed block file #49

And it gets killed at that point. "journalctl -r" shows:

Killed process 14915 (ArmoryDB) total-vm:7897844kB, anon-rss:677480kB, file-rss:3882984kB
Out of memory: Kill process 14915 (ArmoryDB) score 386 or sacrifice child

Substantially the same results with DB_FULL or DB_BARE, ram-usage 1 or 4.


1714986876
Hero Member
*
Offline Offline

Posts: 1714986876

View Profile Personal Message (Offline)

Ignore
1714986876
Reply with quote  #2

1714986876
Report to moderator
"There should not be any signed int. If you've found a signed int somewhere, please tell me (within the next 25 years please) and I'll change it to unsigned int." -- Satoshi
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
goatpig
Moderator
Legendary
*
Offline Offline

Activity: 3668
Merit: 1345

Armory Developer


View Profile
January 27, 2017, 05:17:53 AM
 #2

Are you using a 32bit OS?

carl.byington (OP)
Newbie
*
Offline Offline

Activity: 13
Merit: 0


View Profile
January 27, 2017, 05:31:52 AM
 #3

Nope. uname -a shows:

4.8.10-300.fc25.x86_64 #1 SMP Mon Nov 21 18:59:16 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

file /usr/bin/python2.7
/usr/bin/python2.7: ELF 64-bit LSB shared object, x86-64,

ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 22761
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 22761
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 3668
Merit: 1345

Armory Developer


View Profile
January 27, 2017, 07:04:09 AM
 #4

Reduce the thread count then.

carl.byington (OP)
Newbie
*
Offline Offline

Activity: 13
Merit: 0


View Profile
January 27, 2017, 06:55:26 PM
 #5

Same results with --db-type="DB_BARE" --rebuild --ram-usage=1 --thread-count=1

Of course, "rm -rf ~/.armory/databases; mkdir ~/.armory/databases" before trying that rebuild.

This is packaged from the source:
Fetch URL: https://github.com/goatpig/BitcoinArmory.git

commit cc9eccf690c8534af1502f66176e2ca73a9f083c
Merge: 2db2622 0816758
Author: goatpig
Date:   Tue Nov 8 18:04:50 2016 +0100

    Merge pull request #128 from achow101/master
   
    Remove bitcoinarmory.com link from multisig dialog


Is that the current source location/version? changelog.txt says 0.95.1

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 3668
Merit: 1345

Armory Developer


View Profile
January 27, 2017, 10:41:26 PM
 #6

Are you running Core along ArmoryDB? Try without it.

carl.byington (OP)
Newbie
*
Offline Offline

Activity: 13
Merit: 0


View Profile
January 28, 2017, 01:23:51 AM
 #7

Yes, bitcoind was running. I stopped that, and the armory rebuild worked. Thanks!

carl.byington (OP)
Newbie
*
Offline Offline

Activity: 13
Merit: 0


View Profile
January 28, 2017, 02:53:30 AM
 #8

More info - it seems to have nothing to do with the box also running bitcoind. There were two stale logins (/usr/lib/systemd/systemd --user) with a bunch of child processes under them. Killing off both of those, the rebuild works with a simple --db-type="DB_FULL" --rebuild command. It eventually gets over 80GB virtual memory as shown by 'top', but only uses about 4GB of real memory and no swap.

I am not sure why those two stale logins caused this - none of the child processes seemed to be using up much memory.

goatpig
Moderator
Legendary
*
Offline Offline

Activity: 3668
Merit: 1345

Armory Developer


View Profile
January 28, 2017, 07:13:58 AM
 #9

Are the stale logins related to Armory at all? That's the first time I hear anything like that.

As for the virtual memory, Armory mmaps all blkXXXXX.dat files when building and scanning. No copy of that data is performed in memory though, it's basically just a caching hint for the OS.

The 4GB figure looks high. I usually don't break 1.5GB on my machine, but that's very dependent on your setup. Basically you are running into this:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/BlockchainScanner.cpp#L195

In full/bare node, there is very little data to write during scans (you can check verify by looking the size of your ssh file in the database folder, it should be tiny compared to the 100GB+ worth of blockchain data), so I just went with the assumptions the writer thread would always outpace the scanner threads (i.e there would never be any write queue depth).

I could fix the write queuing in 30 min but I don't want to release DB changes this time around, as very few people have run into your kind of issue (you are probably the worst case so far). I plan on speeding up the DB a bit in a later release (gotta keep up with the ever growing blockchain), I'll deal with it then.

Quote
More info - it seems to have nothing to do with the box also running bitcoind.

I only suggested that because Core can eat up a lot of RAM (2~3GB)

Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!