Starting preliminary 0.94 testing - "Headless fullnode"

[goatpig]

btchris: unless you consistantly get the same error, this probably the aftermath of a buffer overrun in another thread

Carlton Banks: That's usually symptomatic of a threading issue. Throw off the timing a little and the issue doesn't appear anymore.

[Carlton Banks]

Carlton Banks: That's usually symptomatic of a threading issue. Throw off the timing a little and the issue doesn't appear anymore.

I thought it couldn't possibly have been a threading problem, as yesterday the behaviour was totally consistent across multiple trials. And today it isn't, so I guess it must be threading related (someone might think you knew something about this code  ). That's pretty curious though, several consecutive trials that generate basically the same result, and then several more consecutive trials that produce completely different result, also all the same as each other, and all without any material change to the build. In addition, not one GTK+ CRITICAL assertion fails appeared on the console today.

To clarify the pattern of crashing yesterday: scanning from blocks 0 - 200,000, Armory crashed without even sending segfault messages to console, and it only got through 2-5 thousand blocks before the crashes. That's a lot of restarts to get it to scan the first 200,000 blocks.

So I've got no segfault tracebacks to post, as I've not had even one. Will carry on testing debug build using gdb in hope it will enter the same state again.

[goatpig]

Welcome to my world =P

Threading issues are tricky. Consider no one has ran into trouble in-house, and our first 2 testers get something right away.

You could try to force the thread count per task manually see how it reacts. In BlockWriteBatcher.cpp, around line 1540, add the lines in bold:

   else
   {
      //otherwise, give workers 90% of the processing power
      int32_t writers = (totalThreadCount_ * 10) / 100;
      if (writers < 1)
         writers = 1;

      int32_t workers = totalThreadCount_ - writers;
      if (workers < 1)
         workers = 1;
     
      workers = 1;
      writers = 1;

      nWorkers_.store(workers);
      nWriters_.store(writers);
   }

This controls the amount of workers (reading through transactions and extracting history) and writers (here you would force them both to 1 to test low performance stability).

Also, around line 385:

{
   //Grab the SSHheader static mutex. This makes sure only this scan is
   //creating new SSH keys. If several scanning threads were to take place,
   //it could possibly result in key collision, as scan threads are not aware
   //of each others' state
   unique_lock<mutex> addressingLock(SSHheaders::keyAddressingMutex_);

   uint32_t nThreads = 3;
   if (int(endBlock) - int(startBlock) < 100)
      nThreads = 1;

    nThreads = 1;
   
   prepareSshToModify(scf);

This controls the amount of "grab" threads, which pull raw data from the blockchain and serialize it into PulledBlock objects used by the workers. Be nice and don't comment on my naming sense, unlike some of my colleagues T__T.

This stuff needs consolidated but I'll get to that once the thread toggler is done. As you can see, it runs on static thread counts currently, but it's meant to adjust resource per task dynamically (to squeeze every last drop of processing power regardless of the machine, and eventually give the user live control over how many cpu cores it can use).

For max performance in Fullnode, you want a grab thread per worker and only 1 writer.

[Carlton Banks]

Code:

[New Thread 0x7fffd7fff700 (LWP 3900)]
[Thread 0x7fffd7fff700 (LWP 3900) exited]
-WARN  - 1431799612: (lmdb_wrapper.cpp:659) Clearing history
-WARN  - 1431799612: (BlockWriteBatcher.cpp:1556) Starting with: 
-WARN  - 1431799612: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1431799612: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1431799612: (BlockUtils.cpp:1071) Scanning from 0 to 356710
[New Thread 0x7fffd7fff700 (LWP 3901)]
[New Thread 0x7fffdc9ae700 (LWP 3902)]
[New Thread 0x7fffd77fe700 (LWP 3903)]
[New Thread 0x7fffd699e700 (LWP 3904)]
[New Thread 0x7fffd4d90700 (LWP 3905)]
[New Thread 0x7fffcffff700 (LWP 3906)]
[New Thread 0x7fffcf7fe700 (LWP 3907)]
[Thread 0x7fffd4d90700 (LWP 3905) exited]
[New Thread 0x7fffd4d90700 (LWP 3908)]
[Thread 0x7fffd4d90700 (LWP 3908) exited]
[New Thread 0x7fffd4d90700 (LWP 3909)]
[Thread 0x7fffd4d90700 (LWP 3909) exited]
-WARN  - 1431799616: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 2500
[New Thread 0x7fffd4d90700 (LWP 3911)]
[New Thread 0x7fffceffd700 (LWP 3912)]
[Thread 0x7fffceffd700 (LWP 3912) exited]
[New Thread 0x7fffceffd700 (LWP 3913)]
[Thread 0x7fffceffd700 (LWP 3913) exited]
[New Thread 0x7fffceffd700 (LWP 3914)]
[Thread 0x7fffceffd700 (LWP 3914) exited]
-WARN  - 1431799619: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 85000
[Thread 0x7fffd4d90700 (LWP 3911) exited]
[New Thread 0x7fffd4d90700 (LWP 3916)]
[New Thread 0x7fffceffd700 (LWP 3917)]
[Thread 0x7fffd4d90700 (LWP 3916) exited]
[New Thread 0x7fffce7fc700 (LWP 3918)]
[Thread 0x7fffce7fc700 (LWP 3918) exited]

{CARLTON EDIT}

[New Thread 0x7fffd4d90700 (LWP 5281)]
[New Thread 0x7fffcdffb700 (LWP 5282)]
[New Thread 0x7fffcd7fa700 (LWP 5283)]
[Thread 0x7fffd4d90700 (LWP 5281) exited]
[Thread 0x7fffcd7fa700 (LWP 5283) exited]
[New Thread 0x7fffcd7fa700 (LWP 5284)]
[Thread 0x7fffcdffb700 (LWP 5282) exited]
[Thread 0x7fffcd7fa700 (LWP 5284) exited]
[New Thread 0x7fffcdffb700 (LWP 5285)]
[New Thread 0x7fffcd7fa700 (LWP 5286)]
[New Thread 0x7fffceffd700 (LWP 5287)]
[Thread 0x7fffcd7fa700 (LWP 5286) exited]
[Thread 0x7fffceffd700 (LWP 5287) exited]
[New Thread 0x7fffceffd700 (LWP 5288)]
[Thread 0x7fffcdffb700 (LWP 5285) exited]
[New Thread 0x7fffd4d90700 (LWP 5290)]
[New Thread 0x7fffcdffb700 (LWP 5291)]
[New Thread 0x7fffcd7fa700 (LWP 5292)]
[Thread 0x7fffd4d90700 (LWP 5290) exited]
[Thread 0x7fffcd7fa700 (LWP 5292) exited]
[Thread 0x7fffcdffb700 (LWP 5291) exited]
[New Thread 0x7fffce7fc700 (LWP 5294)]
[New Thread 0x7fffcd7fa700 (LWP 5293)]
[Thread 0x7fffce7fc700 (LWP 5294) exited]
[New Thread 0x7fffce7fc700 (LWP 5295)]
[Thread 0x7fffceffd700 (LWP 5288) exited]
(WARNING) SDM.py:726 - bitcoind exited, bitcoind STDOUT:
(WARNING) SDM.py:728 - 
(WARNING) SDM.py:729 - bitcoind exited, bitcoind STDERR:
(WARNING) SDM.py:731 - 
[Thread 0x7fffce7fc700 (LWP 5295) exited]
(ERROR) Networking.py:366 - ***Connection to Satoshi client LOST!  Attempting to reconnect...
(ERROR) ArmoryQt.py:5842 - BitcoindNotAvailable: should not happen...
[Thread 0x7fffcd7fa700 (LWP 5293) exited]
[New Thread 0x7fffd4d90700 (LWP 5296)]
[New Thread 0x7fffcd7fa700 (LWP 5297)]
[New Thread 0x7fffce7fc700 (LWP 5298)]
[Thread 0x7fffd4d90700 (LWP 5296) exited]
[Thread 0x7fffce7fc700 (LWP 5298) exited]
[New Thread 0x7fffceffd700 (LWP 5300)]
[New Thread 0x7fffce7fc700 (LWP 5299)]
[Thread 0x7fffcd7fa700 (LWP 5297) exited]
[Thread 0x7fffceffd700 (LWP 5300) exited]
[New Thread 0x7fffceffd700 (LWP 5301)]
[Thread 0x7fffceffd700 (LWP 5301) exited]
[Thread 0x7fffce7fc700 (LWP 5299) exited]
[New Thread 0x7fffd4d90700 (LWP 5302)]
[New Thread 0x7fffce7fc700 (LWP 5303)]
[Thread 0x7fffd4d90700 (LWP 5302) exited]
[New Thread 0x7fffceffd700 (LWP 5304)]
[Thread 0x7fffceffd700 (LWP 5304) exited]
[Thread 0x7fffce7fc700 (LWP 5303) exited]
[New Thread 0x7fffcd7fa700 (LWP 5306)]
[New Thread 0x7fffceffd700 (LWP 5305)]
[Thread 0x7fffcd7fa700 (LWP 5306) exited]
[New Thread 0x7fffcd7fa700 (LWP 5307)]
[Thread 0x7fffcd7fa700 (LWP 5307) exited]
[Thread 0x7fffceffd700 (LWP 5305) exited]
[New Thread 0x7fffd4d90700 (LWP 5308)]
[New Thread 0x7fffceffd700 (LWP 5309)]
[New Thread 0x7fffcd7fa700 (LWP 5310)]
[Thread 0x7fffd4d90700 (LWP 5308) exited]
[Thread 0x7fffcd7fa700 (LWP 5310) exited]
[Thread 0x7fffceffd700 (LWP 5309) exited]
[New Thread 0x7fffcd7fa700 (LWP 5312)]
[New Thread 0x7fffce7fc700 (LWP 5313)]
[Thread 0x7fffce7fc700 (LWP 5313) exited]
[New Thread 0x7fffce7fc700 (LWP 5314)]
[Thread 0x7fffce7fc700 (LWP 5314) exited]
[Thread 0x7fffcd7fa700 (LWP 5312) exited]
[New Thread 0x7fffd4d90700 (LWP 5315)]
[Thread 0x7fffd4d90700 (LWP 5315) exited]
[New Thread 0x7fffcd7fa700 (LWP 5316)]
[New Thread 0x7fffce7fc700 (LWP 5317)]
[Thread 0x7fffce7fc700 (LWP 5317) exited]
[Thread 0x7fffcd7fa700 (LWP 5316) exited]
[New Thread 0x7fffceffd700 (LWP 5320)]
[New Thread 0x7fffce7fc700 (LWP 5318)]
-WARN  - 1431800044: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 217500

{CARLTON EDIT}

[New Thread 0x7fffce7fc700 (LWP 5617)]
[Thread 0x7fffceffd700 (LWP 5614) exited]
[Thread 0x7fffce7fc700 (LWP 5617) exited]
[New Thread 0x7fffd4d90700 (LWP 5619)]
[New Thread 0x7fffce7fc700 (LWP 5620)]
[New Thread 0x7fffceffd700 (LWP 5621)]
[Thread 0x7fffceffd700 (LWP 5621) exited]
[Thread 0x7fffd4d90700 (LWP 5619) exited]
[Thread 0x7fffce7fc700 (LWP 5620) exited]
[New Thread 0x7fffce7fc700 (LWP 5622)]
[New Thread 0x7fffceffd700 (LWP 5623)]
[Thread 0x7fffceffd700 (LWP 5623) exited]
[Thread 0x7fffce7fc700 (LWP 5622) exited]
[New Thread 0x7fffd4d90700 (LWP 5624)]
[New Thread 0x7fffce7fc700 (LWP 5625)]
[New Thread 0x7fffceffd700 (LWP 5626)]
[New Thread 0x7fffcd7fa700 (LWP 5627)]
[Thread 0x7fffceffd700 (LWP 5626) exited]
[Thread 0x7fffce7fc700 (LWP 5625) exited]
[New Thread 0x7fffce7fc700 (LWP 5628)]
[Thread 0x7fffce7fc700 (LWP 5628) exited]
[Thread 0x7fffd4d90700 (LWP 5624) exited]
[New Thread 0x7fffce7fc700 (LWP 5629)]
[Thread 0x7fffce7fc700 (LWP 5629) exited]
[Thread 0x7fffcd7fa700 (LWP 5627) exited]
[New Thread 0x7fffd4d90700 (LWP 5631)]
[New Thread 0x7fffcd7fa700 (LWP 5632)]
[New Thread 0x7fffce7fc700 (LWP 5633)]
[Thread 0x7fffd4d90700 (LWP 5631) exited]
[New Thread 0x7fffceffd700 (LWP 5634)]
[Thread 0x7fffce7fc700 (LWP 5633) exited]
[Thread 0x7fffcd7fa700 (LWP 5632) exited]
[New Thread 0x7fffcd7fa700 (LWP 5635)]
-INFO  - 1431800186: (BDM_mainthread.cpp:273) Stop requested detected
[Thread 0x7fffcd7fa700 (LWP 5635) exited]
[New Thread 0x7fffcd7fa700 (LWP 5636)]
terminate called without an active exception

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffdd1af700 (LWP 3645)]
0x00007ffff6fc8165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) run ArmoryQt.py
The program being debugged has been started already.
Start it from the beginning? (y or n) ^Cn
Program not restarted.
(gdb) quit
A debugging session is active.

	Inferior 1 [process 2698] will be killed.

Quit anyway? (y or n) y
Quitting: Couldn't get registers: No such process.
user@machine:~/BitcoinArmory$ gdb python
GNU gdb (GDB) 7.4.1-debian
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/python...(no debugging symbols found)...done.
(gdb) run ArmoryQt.py
Starting program: /usr/bin/python ArmoryQt.py
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
********************************************************************************
Loading Armory Engine:
   Armory Version:       0.93.99.1
   Armory Build:         578c0724b9
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('debian', '7.8', '')
   User home-directory   : /home/user
   Satoshi BTC directory : /home/user/.bitcoin
   Armory home dir       : /home/user/.armory
   ArmoryDB directory     : /home/user/.armory/databases
   Armory settings file  : /home/user/.armory/ArmorySettings.txt
   Armory log file       : /home/user/.armory/armorylog.txt
   Do wallet checking    : True
[New Thread 0x7fffddbb6700 (LWP 5671)]
[New Thread 0x7fffdd1af700 (LWP 5675)]
[New Thread 0x7fffdc9ae700 (LWP 5676)]
[Thread 0x7fffdc9ae700 (LWP 5676) exited]
[New Thread 0x7fffdc9ae700 (LWP 5712)]
[Thread 0x7fffdc9ae700 (LWP 5712) exited]
[New Thread 0x7fffdc9ae700 (LWP 5715)]
[New Thread 0x7fffdc1ad700 (LWP 5716)]
[New Thread 0x7fffdb9ac700 (LWP 5717)]
[Thread 0x7fffdb9ac700 (LWP 5717) exited]
[New Thread 0x7fffdb9ac700 (LWP 5718)]
[Thread 0x7fffdb9ac700 (LWP 5718) exited]
[New Thread 0x7fffdb9ac700 (LWP 5719)]
[Thread 0x7fffdb9ac700 (LWP 5719) exited]

{CARLTON EDIT}

[Thread 0x7fffdc9ae700 (LWP 6561) exited]
[New Thread 0x7fffdc9ae700 (LWP 6562)]
[Thread 0x7fffdc9ae700 (LWP 6562) exited]
[New Thread 0x7fffdc9ae700 (LWP 6563)]
[Thread 0x7fffdc9ae700 (LWP 6563) exited]
[New Thread 0x7fffdc9ae700 (LWP 6564)]
[Thread 0x7fffdc9ae700 (LWP 6564) exited]
[New Thread 0x7fffdc9ae700 (LWP 6565)]
[Thread 0x7fffdc9ae700 (LWP 6565) exited]
[New Thread 0x7fffdc9ae700 (LWP 6566)]
[Thread 0x7fffdc9ae700 (LWP 6566) exited]
[New Thread 0x7fffdc9ae700 (LWP 6567)]
[Thread 0x7fffdc9ae700 (LWP 6567) exited]
[New Thread 0x7fffdc9ae700 (LWP 6568)]
[Thread 0x7fffdc9ae700 (LWP 6568) exited]
[New Thread 0x7fffdc9ae700 (LWP 6569)]
[Thread 0x7fffdc9ae700 (LWP 6569) exited]
[New Thread 0x7fffdc9ae700 (LWP 6570)]
[Thread 0x7fffdc9ae700 (LWP 6570) exited]
[New Thread 0x7fffdc9ae700 (LWP 6571)]
[Thread 0x7fffdc9ae700 (LWP 6571) exited]
[New Thread 0x7fffdc9ae700 (LWP 6572)]
[Thread 0x7fffdc9ae700 (LWP 6572) exited]
[New Thread 0x7fffdc9ae700 (LWP 6573)]
[Thread 0x7fffdc9ae700 (LWP 6573) exited]
[New Thread 0x7fffdc9ae700 (LWP 6574)]
[Thread 0x7fffdd1af700 (LWP 5785) exited]
[New Thread 0x7fffdd1af700 (LWP 6575)]
-INFO  - 1431800624: (BlockUtils.cpp:850) blkfile dir: /.bitcoin/blocks
-INFO  - 1431800624: (BlockUtils.cpp:851) lmdb dir: /.armory/databases
-INFO  - 1431800624: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1431800624: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1431800624: (BlockUtils.cpp:1253) Total number of blk*.dat files: 270
-INFO  - 1431800624: (BlockUtils.cpp:1254) Total blockchain bytes: 36,138,815,515
-INFO  - 1431800624: (BlockUtils.cpp:1628) Reading headers from db
[Thread 0x7fffdc9ae700 (LWP 6574) exited]
[New Thread 0x7fffdc9ae700 (LWP 6576)]
[Thread 0x7fffdc9ae700 (LWP 6576) exited]
[New Thread 0x7fffdc9ae700 (LWP 6578)]
[Thread 0x7fffdc9ae700 (LWP 6578) exited]
[New Thread 0x7fffdc9ae700 (LWP 6579)]
[Thread 0x7fffdc9ae700 (LWP 6579) exited]
[New Thread 0x7fffdc9ae700 (LWP 6580)]
[Thread 0x7fffdc9ae700 (LWP 6580) exited]
[New Thread 0x7fffdc9ae700 (LWP 6581)]
[Thread 0x7fffdc9ae700 (LWP 6581) exited]
[New Thread 0x7fffdc9ae700 (LWP 6582)]
[Thread 0x7fffdc9ae700 (LWP 6582) exited]
[New Thread 0x7fffdc9ae700 (LWP 6583)]
[Thread 0x7fffdc9ae700 (LWP 6583) exited]
[New Thread 0x7fffdc9ae700 (LWP 6584)]
[Thread 0x7fffdc9ae700 (LWP 6584) exited]
[New Thread 0x7fffdc9ae700 (LWP 6585)]
[Thread 0x7fffdc9ae700 (LWP 6585) exited]
[New Thread 0x7fffdc9ae700 (LWP 6586)]
[Thread 0x7fffdc9ae700 (LWP 6586) exited]
-INFO  - 1431800642: (BlockUtils.cpp:1654) Found 356721 headers in db
-DEBUG - 1431800642: (Blockchain.cpp:214) Organizing chain w/ rebuild
[New Thread 0x7fffdc9ae700 (LWP 6587)]
[Thread 0x7fffdc9ae700 (LWP 6587) exited]
[New Thread 0x7fffdc9ae700 (LWP 6588)]
[Thread 0x7fffdc9ae700 (LWP 6588) exited]
[New Thread 0x7fffdc9ae700 (LWP 6589)]
[Thread 0x7fffdc9ae700 (LWP 6589) exited]
[New Thread 0x7fffdc9ae700 (LWP 6591)]
[Thread 0x7fffdc9ae700 (LWP 6591) exited]
-WARN  - 1431800645: (BlockUtils.cpp:1282) --- Fetching SSH summaries for 491 registered addresses
-INFO  - 1431800646: (BlockUtils.cpp:1295) Left off at file 269, offset 71957593
-INFO  - 1431800646: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1431800646: (BlockUtils.cpp:1299) Starting at block file 269 offset 71957593
-INFO  - 1431800646: (BlockUtils.cpp:1301) Block height 356710
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 00000000000000000456959f4335793c1ba488a78f6941c1b56baea08c098815
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000001d0f5ed714b78d466f97731beb4ca155367433bcdac028a
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000014386b1c3f544efa0b0fa0840a7993183027a6b0f1425dc4
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000006c5c96065b4f447d78cd761cec54aba452d7acb824a67c1
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 0000000000000000167945b0f576dcac6d20d64f13b20d5cf655308d848ff682
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000003390048ffcf6e78725d9863c032c690b9a479a6f130ac81
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000011c2bc499d60ea233f52e3c01a0e652d6c61f863395d980e
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 00000000000000000a4e923d2b9066d115ed317840a4bdbd4b86155b8d0d6381
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000008e12725ad276d16a4bd43434282f97d05d5cfb378ab1859
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000007c14b361d4fe8b0578d43051b70b078335d4ccdfabd24f6
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 0000000000000000160f830c3b4234adee0bbf88bec999658ddb251d231cc16d
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 000000000000000013d22ac7f29342ebaa7ea6dd6d0a0db8f511972b2666377a
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 00000000000000001114f7a74345ad847d27ae1ab6154aea01e459a5072c7d6d
-WARN  - 1431800646: (Blockchain.cpp:50) Somehow tried to add header that's already in map
-WARN  - 1431800646: (Blockchain.cpp:51)     Header Hash: 0000000000000000131705e6c6d3201c48fc237ff9ece5df8a99d88c30f878ec
-DEBUG - 1431800646: (Blockchain.cpp:214) Organizing chain w/ rebuild
[New Thread 0x7fffdc9ae700 (LWP 6592)]
[Thread 0x7fffdc9ae700 (LWP 6592) exited]
[New Thread 0x7fffdc9ae700 (LWP 6594)]
[Thread 0x7fffdc9ae700 (LWP 6594) exited]
-INFO  - 1431800646: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1431800646: (BlockUtils.cpp:1489) Loading block data... file 269 offset 69889767
-ERROR - 1431800646: (BlockUtils.cpp:516) Next block header found at offset 69889775
-INFO  - 1431800646: (BlockUtils.cpp:544) Reading raw blocks finished at file 269 offset 79963756
-INFO  - 1431800646: (BlockUtils.cpp:1354) Wrote blocks to DB in 0.06s
-INFO  - 1431800646: (BlockUtils.cpp:1371) Checking dupIDs from 226068 onward
-WARN  - 1431800646: (BlockWriteBatcher.cpp:1556) Starting with: 
-WARN  - 1431800646: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1431800646: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1431800646: (BlockUtils.cpp:1071) Scanning from 226068 to 356713
[New Thread 0x7fffdc9ae700 (LWP 6612)]
[New Thread 0x7fffd7fff700 (LWP 6613)]
[New Thread 0x7fffd77fe700 (LWP 6614)]
[New Thread 0x7fffd699e700 (LWP 6615)]
[New Thread 0x7fffc6e90700 (LWP 6616)]
[New Thread 0x7fffc668f700 (LWP 6617)]
[New Thread 0x7fffc5e8e700 (LWP 6618)]

(python:5764): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
[Thread 0x7fffc5e8e700 (LWP 6618) exited]
[New Thread 0x7fffc5e8e700 (LWP 6620)]
[New Thread 0x7fffc568d700 (LWP 6621)]
[Thread 0x7fffc568d700 (LWP 6621) exited]
[New Thread 0x7fffc568d700 (LWP 6622)]
[Thread 0x7fffc568d700 (LWP 6622) exited]
[Thread 0x7fffc5e8e700 (LWP 6620) exited]
[New Thread 0x7fffc5e8e700 (LWP 6623)]
[New Thread 0x7fffc568d700 (LWP 6624)]
[New Thread 0x7fffc4e8c700 (LWP 6625)]
[Thread 0x7fffc4e8c700 (LWP 6625) exited]
[Thread 0x7fffc5e8e700 (LWP 6623) exited]
[New Thread 0x7fffc4e8c700 (LWP 6626)]
[Thread 0x7fffc568d700 (LWP 6624) exited]
[Thread 0x7fffc4e8c700 (LWP 6626) exited]
[New Thread 0x7fffc4e8c700 (LWP 6627)]
[Thread 0x7fffc4e8c700 (LWP 6627) exited]
[New Thread 0x7fffc4e8c700 (LWP 6628)]
[New Thread 0x7fffc568d700 (LWP 6629)]
[Thread 0x7fffc568d700 (LWP 6629) exited]
[New Thread 0x7fffc568d700 (LWP 6630)]
[Thread 0x7fffc568d700 (LWP 6630) exited]
[New Thread 0x7fffc568d700 (LWP 6631)]
[Thread 0x7fffc4e8c700 (LWP 6628) exited]
[New Thread 0x7fffc5e8e700 (LWP 6632)]
[New Thread 0x7fffc4e8c700 (LWP 6633)]
[Thread 0x7fffc5e8e700 (LWP 6632) exited]
[New Thread 0x7fffb7fff700 (LWP 6634)]
[Thread 0x7fffb7fff700 (LWP 6634) exited]
[New Thread 0x7fffb77fe700 (LWP 6635)]
[Thread 0x7fffc4e8c700 (LWP 6633) exited]
[New Thread 0x7fffb7fff700 (LWP 6636)]
[Thread 0x7fffb7fff700 (LWP 6636) exited]
[New Thread 0x7fffb7fff700 (LWP 6637)]
[Thread 0x7fffc568d700 (LWP 6631) exited]
[Thread 0x7fffb7fff700 (LWP 6637) exited]
[New Thread 0x7fffb7fff700 (LWP 6638)]
[Thread 0x7fffb7fff700 (LWP 6638) exited]
[New Thread 0x7fffb7fff700 (LWP 6639)]
[Thread 0x7fffb7fff700 (LWP 6639) exited]
[Thread 0x7fffb77fe700 (LWP 6635) exited]
[New Thread 0x7fffc5e8e700 (LWP 6640)]
[New Thread 0x7fffb77fe700 (LWP 6641)]
[New Thread 0x7fffb7fff700 (LWP 6642)]
[Thread 0x7fffb7fff700 (LWP 6642) exited]
[Thread 0x7fffc5e8e700 (LWP 6640) exited]
[New Thread 0x7fffb7fff700 (LWP 6643)]
[Thread 0x7fffb77fe700 (LWP 6641) exited]
[New Thread 0x7fffc568d700 (LWP 6644)]
[Thread 0x7fffc568d700 (LWP 6644) exited]
[New Thread 0x7fffc568d700 (LWP 6645)]
[New Thread 0x7fffb77fe700 (LWP 6646)]
[Thread 0x7fffb77fe700 (LWP 6646) exited]
[Thread 0x7fffc568d700 (LWP 6645) exited]
-WARN  - 1431800658: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 227500
[New Thread 0x7fffc568d700 (LWP 6647)]
[Thread 0x7fffc568d700 (LWP 6647) exited]
[Thread 0x7fffb7fff700 (LWP 6643) exited]
[New Thread 0x7fffc5e8e700 (LWP 6648)]
[New Thread 0x7fffb7fff700 (LWP 6649)]
[New Thread 0x7fffc568d700 (LWP 6650)]
[Thread 0x7fffc5e8e700 (LWP 6648) exited]
[Thread 0x7fffc568d700 (LWP 6650) exited]
[New Thread 0x7fffb77fe700 (LWP 6651)]
[New Thread 0x7fffc568d700 (LWP 6652)]
[New Thread 0x7fffc4e8c700 (LWP 6653)]
[Thread 0x7fffb7fff700 (LWP 6649) exited]
[Thread 0x7fffc4e8c700 (LWP 6653) exited]
[New Thread 0x7fffc4e8c700 (LWP 6654)]
[Thread 0x7fffb77fe700 (LWP 6651) exited]
[Thread 0x7fffc4e8c700 (LWP 6654) exited]
[New Thread 0x7fffc4e8c700 (LWP 6655)]
[Thread 0x7fffc4e8c700 (LWP 6655) exited]
[Thread 0x7fffc568d700 (LWP 6652) exited]
[New Thread 0x7fffc5e8e700 (LWP 6656)]
[Thread 0x7fffc5e8e700 (LWP 6656) exited]
[New Thread 0x7fffc568d700 (LWP 6658)]
[New Thread 0x7fffc4e8c700 (LWP 6659)]
[Thread 0x7fffc4e8c700 (LWP 6659) exited]
[New Thread 0x7fffc4e8c700 (LWP 6660)]
[Thread 0x7fffc568d700 (LWP 6658) exited]
[New Thread 0x7fffb77fe700 (LWP 6661)]
[Thread 0x7fffb77fe700 (LWP 6661) exited]
[New Thread 0x7fffb77fe700 (LWP 6662)]
[Thread 0x7fffb77fe700 (LWP 6662) exited]
[New Thread 0x7fffb77fe700 (LWP 6663)]
[Thread 0x7fffb77fe700 (LWP 6663) exited]
[New Thread 0x7fffb77fe700 (LWP 6664)]
[Thread 0x7fffb77fe700 (LWP 6664) exited]
[Thread 0x7fffc4e8c700 (LWP 6660) exited]
[New Thread 0x7fffc5e8e700 (LWP 6665)]
[New Thread 0x7fffc4e8c700 (LWP 6666)]
[Thread 0x7fffc5e8e700 (LWP 6665) exited]
[New Thread 0x7fffb77fe700 (LWP 6667)]
[Thread 0x7fffb77fe700 (LWP 6667) exited]
[Thread 0x7fffc4e8c700 (LWP 6666) exited]
[New Thread 0x7fffc4e8c700 (LWP 6668)]
[New Thread 0x7fffb77fe700 (LWP 6669)]
[New Thread 0x7fffc568d700 (LWP 6670)]
[Thread 0x7fffc568d700 (LWP 6670) exited]
[Thread 0x7fffb77fe700 (LWP 6669) exited]
[New Thread 0x7fffb77fe700 (LWP 6671)]
[Thread 0x7fffb77fe700 (LWP 6671) exited]
[New Thread 0x7fffb77fe700 (LWP 6672)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd7fff700 (LWP 6613)]
0x00007ffff70b6d00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) backtrace
#0  0x00007ffff70b6d00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fffef6c44ac in BinaryData::copyFrom (this=0x7fffd7ffe800, inData=0x7fff6807d1e3 <Address 0x7fff6807d1e3 out of bounds>, sz=80) at BinaryData.h:196
#2  0x00007fffef7150b7 in BlockHeader::unserialize (this=0x7fffd7ffe800, ptr=0x7fff6807d1e3 <Address 0x7fff6807d1e3 out of bounds>, size=80) at BlockObj.cpp:30
#3  0x00007fffef71527b in BlockHeader::unserialize (this=0x7fffd7ffe800, str=...) at BlockObj.cpp:47
#4  0x00007fffef7152bd in BlockHeader::unserialize (this=0x7fffd7ffe800, brr=...) at BlockObj.cpp:53
#5  0x00007fffef6e0a8b in BlockHeader::BlockHeader (this=0x7fffd7ffe800, brr=...) at BlockObj.h:52
#6  0x00007fffef7a132a in PulledBlock::unserializeFullBlock (this=0x7fffa9016b40, brr=..., doFrag=true, withPrefix=false) at BlockWriteBatcher.h:230
#7  0x00007fffef78d4dd in GrabThreadData::pullBlockAtIter (pb=..., iter=..., db=0x165b6b0, bfa=...) at BlockWriteBatcher.cpp:489
#8  0x00007fffef78936d in GrabThreadData::grabBlocksFromDB (blockData=..., threadId=0) at BlockWriteBatcher.cpp:279
#9  0x00007fffef7ca79f in std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (this=0x7fffba7b9620) at /usr/include/c++/4.7/functional:1598
#10 0x00007fffef7ca461 in std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)>::operator()() (
    this=0x7fffba7b9620) at /usr/include/c++/4.7/functional:1586
#11 0x00007fffef7ca24c in std::thread::_Impl<std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)> >::_M_run() (this=0x7fffba7b9608) at /usr/include/c++/4.7/thread:115
#12 0x00007ffff4e65400 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff7bc7b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007ffff707195d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

Not typical of the first crashes, but there's probably something useful in here. There's two separate trials from the same tx scan; bitcoind crashed in the first trial, and the tx scan just carried on. I wasn't too sure about that, and thought that at least starting bitcoind again was a good idea. It wasn't/it didn't work out (SDM considered bitcoind reconnected only once Armory went into it's shutdown routine. Where it would normally throw up a balloon notifier to indicate bitcoind disconnection, instead a re-establishment notifier appeared, the shutdown routine then crashed). So there's definitely some information there to help with debugging the Db building when unexpected/disruptive things happen.

[btchris]

Not typical of the first crashes, but there's probably something useful in here.

That stack trace looks exactly the same as mine on Windows (some of the line #s are off, but I think that's just a symptom of using different debuggers). Even the most recent calls are showing a similar out-of-bounds pointer being passed. FYI I've had one additional crash, and the stack trace was the same (with pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ in GrabThreadData::pullBlockAtIter() corrupted).

[Carlton Banks]

Not typical of the first crashes, but there's probably something useful in here.

That stack trace looks exactly the same as mine on Windows (some of the line #s are off, but I think that's just a symptom of using different debuggers). Even the most recent calls are showing a similar out-of-bounds pointer being passed. FYI I've had one additional crash, and the stack trace was the same (with pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ in GrabThreadData::pullBlockAtIter() corrupted).

Ah, maybe this has nothing to do with those unexpected things happening in that trial (bitcoind crashing, SDM crashing etc). Debugging threaded applications really does seem as difficult as is made out. How people use other complex or subtle programming constructs in combination with threading, I do not know.

It's tempting to say I'd like a repeat of the failure patterns I got Thursday, those trials were very unstable. Armory mostly crashed after less than 30 seconds, then whatever caused that pattern shifted half through the scan, and then completed without complaint. But is that behaviour actually a whole other category of bug, that just so happens to be triggering the pointer scope bug as it's own natural consequence? It's difficult to tell. Could the debugging environment be invoked at the hypervisor level? I'm sure I've heard this is possible, whereby a VM has it's entire state recorded during software tests, which can then be "played back" should any given run yield bugs that are difficult to find after the fact.

Another little observation is: we're running to stand still, in performance terms. At the time 0.93 was finalised, just a few months ago, I think it took around the same length of time to build and scan as it does with this pre-0.94 build. Obviously we have more blocks/transactions to process/scan now, and goatpig says there's a little more performance to tweak out of this design. Could some of the work (reading the headers or Db writing?) be done concurrently with the blockchain sync? I suspect it would be easier once Armory is actually it's own node.

[btchris]

Another little observation is: we're running to stand still, in performance terms.

For me anyways, 0.94 allows me to place the Armory DB on an SSD whereas before both it and the Bitcoin DB were on an HDD. Practically speaking, this has hugely improved the initial "reading blocks" and "building database" steps, down to about 14 minutes. The tx scanning phase is only slightly faster IIRC (possibly because the Bitcoin DB block files are still on HDD).

In short, when 0.94 works, it's brilliant!

[Carlton Banks]

Another little observation is: we're running to stand still, in performance terms.

For me anyways, 0.94 allows me to place the Armory DB on an SSD whereas before both it and the Bitcoin DB were on an HDD. Practically speaking, this has hugely improved the initial "reading blocks" and "building database" steps, down to about 14 minutes. The tx scanning phase is only slightly faster IIRC (possibly because the Bitcoin DB block files are still on HDD).

In short, when 0.94 works, it's brilliant!

You're reminding me how spoilt I am, I can't imagine a computer without enough SSD space to hold a blockchain or two  . I get the same performance you do for the reading headers/blocks stage, and then about 40 minutes to scan; surprise me and tell me that the scan takes hours from a mechanical disk (are there not cryptography operations taking place in every iteration of that scan? The CPU is bound to be a significant limiting factor). I'm pretty sure the bottleneck with my machine when it comes to the scan is either RAM or CPU (as both, plus swap, get basically maxed out)

[btchris]

You're reminding me how spoilt I am

Well, your namesake certainly was

about 40 minutes to scan

I haven't timed it exactly, but it's somewhere around there even with my HDD. I'll pay closer attention next time I do a scan.

[zombieslayer9099]

My installation is running just fine

[Searinox]

Is there a downloadable Win x64 binary?

[doug_armory]

Is there a downloadable Win x64 binary?

Not for now. This is for people who are able to compile code we supply. We'll offer test binaries eventually.

[Carlton Banks]

Code:

(gdb) run ArmoryQt.py
Starting program: /usr/bin/python ArmoryQt.py
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
********************************************************************************
Loading Armory Engine:
   Armory Version:       0.93.99.1
   Armory Build:         578c0724b9
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('debian', '7.8', '')
   User home-directory   : /home/user
   Satoshi BTC directory : /home/user/.bitcoin
   Armory home dir       : /home/user/.armory
   ArmoryDB directory     : /home/user/.armory/databases
   Armory settings file  : /home/user/.armory/ArmorySettings.txt
   Armory log file       : /home/user/.armory/armorylog.txt
   Do wallet checking    : True
[New Thread 0x7fffddbb6700 (LWP 2525)]
[New Thread 0x7fffdd1af700 (LWP 2526)]
-INFO  - 1431981856: (BlockUtils.cpp:850) blkfile dir: /home/user/.bitcoin/blocks
-INFO  - 1431981856: (BlockUtils.cpp:851) lmdb dir: /home/user/.armory/databases
-INFO  - 1431981856: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1431981856: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1431981856: (BlockUtils.cpp:1253) Total number of blk*.dat files: 271
-INFO  - 1431981856: (BlockUtils.cpp:1254) Total blockchain bytes: 36,239,124,399
-INFO  - 1431981856: (BlockUtils.cpp:1628) Reading headers from db
[New Thread 0x7fffcd690700 (LWP 2552)]
[Thread 0x7fffcd690700 (LWP 2552) exited]
[New Thread 0x7fffcd690700 (LWP 2555)]
[New Thread 0x7fffccd54700 (LWP 2556)]
(ERROR) announcefetch.py:283 - Specified URL was inaccessible
(ERROR) announcefetch.py:284 - Tried: https://bitcoinarmory.com/announce.txt
[Thread 0x7fffccd54700 (LWP 2556) exited]
[Thread 0x7fffcd690700 (LWP 2555) exited]
(ERROR) announcefetch.py:283 - Specified URL was inaccessible
(ERROR) announcefetch.py:284 - Tried: https://s3.amazonaws.com/bitcoinarmory-media/announce.txt
(WARNING) announcefetch.py:297 - Error fetching announce digest
-INFO  - 1431981868: (BlockUtils.cpp:1654) Found 356730 headers in db
-DEBUG - 1431981868: (Blockchain.cpp:214) Organizing chain w/ rebuild
-WARN  - 1431981871: (BlockUtils.cpp:1282) --- Fetching SSH summaries for 491 registered addresses
-INFO  - 1431981871: (BlockUtils.cpp:1295) Left off at file 269, offset 83960315
-INFO  - 1431981871: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1431981871: (BlockUtils.cpp:1299) Starting at block file 269 offset 83960315
-INFO  - 1431981871: (BlockUtils.cpp:1301) Block height 356719
-DEBUG - 1431981872: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1431981875: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1431981875: (BlockUtils.cpp:1489) Loading block data... file 269 offset 83960307
-ERROR - 1431981875: (BlockUtils.cpp:516) Next block header found at offset 83960315
-INFO  - 1431981875: (BlockUtils.cpp:544) Reading raw blocks finished at file 269 offset 133863316
-INFO  - 1431981875: (BlockUtils.cpp:544) Reading raw blocks finished at file 270 offset 50901329
-INFO  - 1431981875: (BlockUtils.cpp:1354) Wrote blocks to DB in 0.3s
-INFO  - 1431981875: (BlockUtils.cpp:1371) Checking dupIDs from 356718 onward
-WARN  - 1431981875: (BlockWriteBatcher.cpp:1556) Starting with: 
-WARN  - 1431981875: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1431981875: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1431981875: (BlockUtils.cpp:1071) Scanning from 356719 to 357006

(python:2515): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
[New Thread 0x7fffcd690700 (LWP 2558)]
[New Thread 0x7fffccd54700 (LWP 2559)]
[New Thread 0x7fffd6a00700 (LWP 2560)]
[New Thread 0x7fffd61ff700 (LWP 2561)]
[New Thread 0x7fffd19fd700 (LWP 2562)]
[New Thread 0x7fffc79a0700 (LWP 2563)]
[New Thread 0x7fffc719f700 (LWP 2564)]
[Thread 0x7fffc719f700 (LWP 2564) exited]
[New Thread 0x7fffc719f700 (LWP 2565)]
[New Thread 0x7fffc699e700 (LWP 2566)]
[New Thread 0x7fffc619d700 (LWP 2567)]
[Thread 0x7fffc619d700 (LWP 2567) exited]
[Thread 0x7fffc719f700 (LWP 2565) exited]
[Thread 0x7fffc699e700 (LWP 2566) exited]
[New Thread 0x7fffc699e700 (LWP 2569)]
[New Thread 0x7fffc619d700 (LWP 2568)]
[Thread 0x7fffc699e700 (LWP 2569) exited]
[New Thread 0x7fffc699e700 (LWP 2570)]
[Thread 0x7fffc699e700 (LWP 2570) exited]
[Thread 0x7fffccd54700 (LWP 2559) exited]
[Thread 0x7fffd61ff700 (LWP 2561) exited]
[Thread 0x7fffc619d700 (LWP 2568) exited]
[New Thread 0x7fffc719f700 (LWP 2571)]
[New Thread 0x7fffc619d700 (LWP 2572)]
[New Thread 0x7fffd61ff700 (LWP 2573)]
[Thread 0x7fffc719f700 (LWP 2571) exited]
[Thread 0x7fffd61ff700 (LWP 2573) exited]
[New Thread 0x7fffd61ff700 (LWP 2574)]
[Thread 0x7fffc619d700 (LWP 2572) exited]
[New Thread 0x7fffc619d700 (LWP 2575)]
[Thread 0x7fffc619d700 (LWP 2575) exited]
[Thread 0x7fffd6a00700 (LWP 2560) exited]
[Thread 0x7fffd61ff700 (LWP 2574) exited]
[New Thread 0x7fffc719f700 (LWP 2576)]
[New Thread 0x7fffd61ff700 (LWP 2577)]
[New Thread 0x7fffd6a00700 (LWP 2578)]
[Thread 0x7fffc719f700 (LWP 2576) exited]
[Thread 0x7fffd6a00700 (LWP 2578) exited]
[Thread 0x7fffd61ff700 (LWP 2577) exited]
[New Thread 0x7fffd6a00700 (LWP 2579)]
[New Thread 0x7fffd61ff700 (LWP 2580)]
[Thread 0x7fffd61ff700 (LWP 2580) exited]
[Thread 0x7fffd6a00700 (LWP 2579) exited]
[New Thread 0x7fffc719f700 (LWP 2581)]
[New Thread 0x7fffd6a00700 (LWP 2582)]
[New Thread 0x7fffd61ff700 (LWP 2583)]
[Thread 0x7fffc719f700 (LWP 2581) exited]
[Thread 0x7fffd61ff700 (LWP 2583) exited]
[Thread 0x7fffd6a00700 (LWP 2582) exited]
[New Thread 0x7fffc719f700 (LWP 2584)]
[Thread 0x7fffc719f700 (LWP 2584) exited]
[New Thread 0x7fffc719f700 (LWP 2585)]
[Thread 0x7fffc79a0700 (LWP 2563) exited]
[Thread 0x7fffd19fd700 (LWP 2562) exited]
-WARN  - 1431981883: (BlockWriteBatcher.cpp:399) --- feedSleep: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:402) --- workers: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:405) --- writeStxo: 0.08 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:407) --- writeStxo_grabMutex: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:410) --- waitingOnSerThread: 0.13 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:412) --- waitForDataToWrite: 18.89 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:416) --- checkForCollisions: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:418) --- getExistingKeys: 0.01 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:420) --- getNewKeys: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:422) --- getSSHHeadersLock: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:424) --- prepareSSHheaders: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:426) --- computeDBKeys: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:428) --- getSshHeaders: 0.01 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:430) --- waitOnNewDBkeys: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:432) --- finishSerializeSSH: 0.05 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:436) --- serializeSSH: 0.12 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:439) --- serializeDataToCommit: 0.02 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:441) --- serializeSubSsh: 0.01 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:443) --- wait on serThread: 0 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:446) --- finishSer: 0.01 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:449) --- putSSH: 0.08 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:451) --- putSTX: 0.02 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:454) --- getnextfeed: 3.39 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:456) --- inControlThread: 19.01 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:459) --- parseTxOut: 4.56 s
-WARN  - 1431981883: (BlockWriteBatcher.cpp:461) --- parseTxIn: 7.5 s
[Thread 0x7fffcd690700 (LWP 2558) exited]
-INFO  - 1431981883: (BlockUtils.cpp:1451) --- bwbDtor: 0s
-INFO  - 1431981883: (BlockUtils.cpp:1452) Scanned Block range in 19.38s
-INFO  - 1431981883: (BlockUtils.cpp:1455) Finished loading at file 270, offset 50901329
-INFO  - 1431981883: (BlockDataViewer.cpp:155) Enabling zero-conf tracking 
[New Thread 0x7fffcd690700 (LWP 2586)]
[Thread 0x7fffcd690700 (LWP 2586) exited]
[Thread 0x7fffc719f700 (LWP 2585) exited]
-DEBUG - 1431981884: (Blockchain.cpp:214) Organizing chain 
[New Thread 0x7fffc719f700 (LWP 2587)]
[Thread 0x7fffc719f700 (LWP 2587) exited]
(ERROR) Networking.py:366 - ***Connection to Satoshi client LOST!  Attempting to reconnect...
[Thread 0x7fffdd1af700 (LWP 2526) exited]
[Thread 0x7ffff7fe5700 (LWP 2515) exited]
[Inferior 1 (process 2515) exited normally]

(gdb) run ArmoryQt.py
Starting program: /usr/bin/python ArmoryQt.py
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
********************************************************************************
Loading Armory Engine:
   Armory Version:       0.93.99.1
   Armory Build:         578c0724b9
   PyBtcWallet  Version: 1.35
Detected Operating system: Linux
   OS Variant            : ('debian', '7.8', '')
   User home-directory   : /home/user
   Satoshi BTC directory : /home/user/.bitcoin
   Armory home dir       : /home/user/.armory
   ArmoryDB directory     : /home/user/.armory/databases
   Armory settings file  : /home/user/.armory/ArmorySettings.txt
   Armory log file       : /home/user/.armory/armorylog.txt
   Do wallet checking    : True
Log file doesn't exist [yet]
[New Thread 0x7fffddbb6700 (LWP 2635)]
[New Thread 0x7fffdd1af700 (LWP 2636)]
-INFO  - 1431981950: (BlockUtils.cpp:850) blkfile dir: /home/user.bitcoin/blocks
-INFO  - 1431981950: (BlockUtils.cpp:851) lmdb dir: /home/user/.armory/databases
-INFO  - 1431981950: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1431981950: (BlockUtils.cpp:1181) Executing: doInitialSyncOnLoad
-INFO  - 1431981950: (BlockUtils.cpp:1253) Total number of blk*.dat files: 271
-INFO  - 1431981950: (BlockUtils.cpp:1254) Total blockchain bytes: 36,255,901,615
-INFO  - 1431981950: (BlockUtils.cpp:1628) Reading headers from db
-WARN  - 1431981950: (lmdb_wrapper.cpp:1396) No headers in DB yet!
-INFO  - 1431981950: (BlockUtils.cpp:1654) Found 1 headers in db
-DEBUG - 1431981950: (Blockchain.cpp:214) Organizing chain w/ rebuild
-WARN  - 1431981950: (BlockUtils.cpp:1282) --- Fetching SSH summaries for 491 registered addresses
-INFO  - 1431981950: (BlockUtils.cpp:1295) Left off at file 0, offset 0
-INFO  - 1431981950: (BlockUtils.cpp:1298) Reading headers and building chain...
-INFO  - 1431981950: (BlockUtils.cpp:1299) Starting at block file 0 offset 0
-INFO  - 1431981950: (BlockUtils.cpp:1301) Block height 0
[New Thread 0x7fffcc9db700 (LWP 2662)]
[Thread 0x7fffcc9db700 (LWP 2662) exited]
[New Thread 0x7fffcc9db700 (LWP 2665)]
[New Thread 0x7fffcc1da700 (LWP 2666)]
(ERROR) announcefetch.py:283 - Specified URL was inaccessible
(ERROR) announcefetch.py:284 - Tried: https://bitcoinarmory.com/announce.txt
[Thread 0x7fffcc1da700 (LWP 2666) exited]
[Thread 0x7fffcc9db700 (LWP 2665) exited]
-DEBUG - 1431982053: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1431982056: (BlockUtils.cpp:1337) Looking for first unrecognized block
-INFO  - 1431982087: (BlockUtils.cpp:1489) Loading block data... file 0 offset 0
-INFO  - 1431982091: (BlockUtils.cpp:544) Reading raw blocks finished at file 0 offset 134215774
-INFO  - 1431982092: (BlockUtils.cpp:544) Reading raw blocks finished at file 1 offset 134203642
-INFO  - 1431982092: (BlockUtils.cpp:544) Reading raw blocks finished at file 2 offset 134195561
-INFO  - 1431982094: (BlockUtils.cpp:544) Reading raw blocks finished at file 3 offset 134168011
-INFO  - 1431982095: (BlockUtils.cpp:544) Reading raw blocks finished at file 4 offset 134197561
-INFO  - 1431982096: (BlockUtils.cpp:544) Reading raw blocks finished at file 5 offset 134203484
-INFO  - 1431982096: (BlockUtils.cpp:544) Reading raw blocks finished at file 6 offset 134210618
-INFO  - 1431982097: (BlockUtils.cpp:544) Reading raw blocks finished at file 7 offset 134213443
-INFO  - 1431982098: (BlockUtils.cpp:544) Reading raw blocks finished at file 8 offset 134209487

{SNIP SNIP}

-INFO  - 1431982216: (BlockUtils.cpp:544) Reading raw blocks finished at file 267 offset 134096813
-INFO  - 1431982216: (BlockUtils.cpp:544) Reading raw blocks finished at file 268 offset 133744156
-INFO  - 1431982217: (BlockUtils.cpp:544) Reading raw blocks finished at file 269 offset 133863316
-INFO  - 1431982217: (BlockUtils.cpp:544) Reading raw blocks finished at file 270 offset 52407929
-INFO  - 1431982217: (BlockUtils.cpp:1354) Wrote blocks to DB in 66.61s
-INFO  - 1431982217: (BlockUtils.cpp:1371) Checking dupIDs from 0 onward
-WARN  - 1431982218: (lmdb_wrapper.cpp:659) Clearing history
-WARN  - 1431982218: (BlockWriteBatcher.cpp:1556) Starting with: 
-WARN  - 1431982218: (BlockWriteBatcher.cpp:1557) 1 workers
-WARN  - 1431982218: (BlockWriteBatcher.cpp:1558) 1 writers
-WARN  - 1431982218: (BlockUtils.cpp:1071) Scanning from 0 to 357009
[New Thread 0x7fffcc9db700 (LWP 2690)]
[New Thread 0x7fffcc1da700 (LWP 2691)]
[New Thread 0x7fffd74f8700 (LWP 2692)]
[New Thread 0x7fffd6cf7700 (LWP 2693)]
[New Thread 0x7fffd64f6700 (LWP 2694)]
[New Thread 0x7fffd564e700 (LWP 2695)]
-WARN  - 1431982221: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 2500
[New Thread 0x7fffceb7a700 (LWP 2697)]
-WARN  - 1431982223: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 85000
[Thread 0x7fffceb7a700 (LWP 2697) exited]
[New Thread 0x7fffceb7a700 (LWP 2699)]
[New Thread 0x7fffce379700 (LWP 2700)]
[Thread 0x7fffceb7a700 (LWP 2699) exited]
[New Thread 0x7fffcdb78700 (LWP 2701)]
[Thread 0x7fffcdb78700 (LWP 2701) exited]
[New Thread 0x7fffcdb78700 (LWP 2702)]
[Thread 0x7fffce379700 (LWP 2700) exited]
[New Thread 0x7fffce379700 (LWP 2703)]
-WARN  - 1431982225: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 105000
[Thread 0x7fffce379700 (LWP 2703) exited]
-WARN  - 1431982226: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 115000

{SNIP SNIP}

[New Thread 0x7fffcd377700 (LWP 2891)]
[Thread 0x7fffcd377700 (LWP 2891) exited]
-WARN  - 1431982303: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 170000
[New Thread 0x7fffcd377700 (LWP 2892)]
[Thread 0x7fffce379700 (LWP 2890) exited]
[New Thread 0x7fffce379700 (LWP 2893)]
[Thread 0x7fffce379700 (LWP 2893) exited]
[Thread 0x7fffcd377700 (LWP 2892) exited]
[New Thread 0x7fffceb7a700 (LWP 2894)]
[New Thread 0x7fffcd377700 (LWP 2895)]
[New Thread 0x7fffce379700 (LWP 2896)]
[Thread 0x7fffceb7a700 (LWP 2894) exited]
[Thread 0x7fffce379700 (LWP 2896) exited]
[Thread 0x7fffcd377700 (LWP 2895) exited]
[New Thread 0x7fffce379700 (LWP 2897)]
[New Thread 0x7fffcdb78700 (LWP 2898)]
[Thread 0x7fffce379700 (LWP 2897) exited]
[Thread 0x7fffcdb78700 (LWP 2898) exited]
[New Thread 0x7fffceb7a700 (LWP 2900)]
[New Thread 0x7fffcdb78700 (LWP 2901)]
[Thread 0x7fffceb7a700 (LWP 2900) exited]
[New Thread 0x7fffce379700 (LWP 2902)]
[Thread 0x7fffce379700 (LWP 2902) exited]
-WARN  - 1431982307: (BlockWriteBatcher.cpp:355) Finished applying blocks up to 172500
[New Thread 0x7fffce379700 (LWP 2903)]
[Thread 0x7fffcdb78700 (LWP 2901) exited]
[New Thread 0x7fffcd377700 (LWP 2904)]
[Thread 0x7fffcd377700 (LWP 2904) exited]
[Thread 0x7fffce379700 (LWP 2903) exited]
[New Thread 0x7fffceb7a700 (LWP 2905)]
[New Thread 0x7fffce379700 (LWP 2906)]
[Thread 0x7fffceb7a700 (LWP 2905) exited]
[New Thread 0x7fffcd377700 (LWP 2907)]
[Thread 0x7fffcd377700 (LWP 2907) exited]
[Thread 0x7fffce379700 (LWP 2906) exited]
[New Thread 0x7fffcd377700 (LWP 2908)]
[New Thread 0x7fffce379700 (LWP 2909)]
[Thread 0x7fffce379700 (LWP 2909) exited]
[Thread 0x7fffcd377700 (LWP 2908) exited]
[Thread 0x7fffd564e700 (LWP 2695) exited]
[Thread 0x7fffd64f6700 (LWP 2694) exited]
[Thread 0x7fffd6cf7700 (LWP 2693) exited]
[Thread 0x7fffd74f8700 (LWP 2692) exited]
[Thread 0x7fffcc1da700 (LWP 2691) exited]
[Thread 0x7fffcc9db700 (LWP 2690) exited]
[Thread 0x7fffdd1af700 (LWP 2636) exited]
[Thread 0x7fffddbb6700 (LWP 2635) exited]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) backtrace
No stack.

A pair of trials that crashed today. 

[Carlton Banks]

Code:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffcb19f700 (LWP 3421)]
0x00007ffff70b6d00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) backtrace
#0  0x00007ffff70b6d00 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#1  0x00007fffef6c44ac in BinaryData::copyFrom (this=0x7fffcb19e800, inData=0x7fff44278bc3 "\001", sz=80) at BinaryData.h:196
#2  0x00007fffef7150b7 in BlockHeader::unserialize (this=0x7fffcb19e800, ptr=0x7fff44278bc3 "\001", size=80) at BlockObj.cpp:30
#3  0x00007fffef71527b in BlockHeader::unserialize (this=0x7fffcb19e800, str=...) at BlockObj.cpp:47
#4  0x00007fffef7152bd in BlockHeader::unserialize (this=0x7fffcb19e800, brr=...) at BlockObj.cpp:53
#5  0x00007fffef6e0a8b in BlockHeader::BlockHeader (this=0x7fffcb19e800, brr=...) at BlockObj.h:52
#6  0x00007fffef7a132a in PulledBlock::unserializeFullBlock (this=0x7fff6415def0, brr=..., doFrag=true, withPrefix=false) at BlockWriteBatcher.h:230
#7  0x00007fffef78d4dd in GrabThreadData::pullBlockAtIter (pb=..., iter=..., db=0x165b6b0, bfa=...) at BlockWriteBatcher.cpp:489
#8  0x00007fffef78936d in GrabThreadData::grabBlocksFromDB (blockData=..., threadId=2) at BlockWriteBatcher.cpp:279
#9  0x00007fffef7ca79f in std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)>::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) (
    this=0x7fffc3db1620) at /usr/include/c++/4.7/functional:1598
#10 0x00007fffef7ca461 in std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)>::operator()() (this=0x7fffc3db1620)
    at /usr/include/c++/4.7/functional:1586
#11 0x00007fffef7ca24c in std::thread::_Impl<std::_Bind_simple<void (*(std::shared_ptr<LoadedBlockData>, unsigned int))(std::shared_ptr<LoadedBlockData>, unsigned int)> >::_M_run() (this=0x7fffc3db1608)
    at /usr/include/c++/4.7/thread:115
#12 0x00007ffff4e65400 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#13 0x00007ffff7bc7b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#14 0x00007ffff707195d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#15 0x0000000000000000 in ?? ()

backtrace from a subsequent run (no GTK+ CRITICAL entries in the log, or anything that appears unusual, like the threads all finishing one after another from the post above this one).

[njaard]

Thank you Carlton Banks,

SIGKILL is often caused by Linux's out-of-memory killer. Linux is saying "I'm out of memory, I'm gonna kill an application that I think is to blame", and in this case it guesses Armory, which is probably the correct one. Running dmesg should show that the "OOM" event. Maybe you can confirm?

The second crash is definitely our fault, but it's hard to tell what's causing it.

[goatpig]

Carlton Banks, btcchris: something new to try, disable the block data prefetch thread.

In BlockWriteBatcher.h#416, comment out these 2 lines:

  LoadedBlockData(const LoadedBlockData&) = delete;
   BFA_PREFETCH getPrefetchMode(void)
   {
      /*if (startBlock_ < endBlock_ && endBlock_ - startBlock_ > 100)
         return PREFETCH_FORWARD;*/
      
      return PREFETCH_NONE;
   }

Regarding speed, in fullnode the bottleneck is your CPU. On my Debian VM with 2 workers I scan fullnode in 40min. On my host with 9 workers I scan it in 5 (i7 5820). Once the thread toggler is ready it should run a lot better than going with some estimated value.

[Carlton Banks]

Carlton Banks, btcchris: something new to try, disable the block data prefetch thread.

In BlockWriteBatcher.h#416, comment out these 2 lines:

  LoadedBlockData(const LoadedBlockData&) = delete;
   BFA_PREFETCH getPrefetchMode(void)
   {
      /*if (startBlock_ < endBlock_ && endBlock_ - startBlock_ > 100)
         return PREFETCH_FORWARD;*/
      
      return PREFETCH_NONE;
   }

Regarding speed, in fullnode the bottleneck is your CPU. On my Debian VM with 2 workers I scan fullnode in 40min. On my host with 9 workers I scan it in 5 (i7 5820). Once the thread toggler is ready it should run a lot better than going with some estimated value.

I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom.

[goatpig]

I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom.

I don't know how the legal stuff goes. I can give you my personal guarantee but I don't know what that is worth either, legally speaking (I doubt I'm in a position to do that). I'm going to push some stuff today and I'll disable the prefetch thread along the way, how about you wait for that?

[Carlton Banks]

I don't know whether you're asking me to break the terms of the license by doing that. Can I get some assurance that I'm protected from legal action by your employers? I have zero interest in reading the license terms myself, it is, in my opinion, abhorrent to use litigation to enforce so-called freedom.

I don't know how the legal stuff goes. I can give you my personal guarantee but I don't know what that is worth either, legally speaking (I doubt I'm in a position to do that). I'm going to push some stuff today and I'll disable the prefetch thread along the way, how about you wait for that?

That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all).

[goatpig]

That's fine, as long as I perceive no risk to me or my affects, then I'm as happy doing that as I've ever been (you're publishing to github after all).

My understanding of the AGPL license is that if you were to modify Armory source and distribute binaries (or run a commercial activity) based on those changes without publishing the altered code publicly, you would be in infringement. I don't think this affects changes for personal and non commercial use. Regardless, if you were to fork Armory publicly (say on a public github repo), you could do as many changes as you like. That's just my interpretation however.

From my perspective there is no license infringement in doing those changes under these conditions, and you have my guarantee I won't be coming after you over this. Again, I'm no lawyer and these words only engage me. I don't have the credentials to speak for the business on this matter.

If anything the responsibility is mine since I asked you to do these changes.