CppnotificationHandler problem

[marekNowak]

I send 100 outgoing transactions from wallet (in c.a 30 seconds all):

and the daemon log an error like below:

(INFO) armoryd.py:3370 - New ZC tx: 2000000
(INFO) armoryd.py:3370 - New ZC tx: 33608
(INFO) armoryd.py:3370 - New ZC tx: -10527220
(INFO) armoryd.py:3370 - New ZC tx: -12913772
(INFO) armoryd.py:3370 - New ZC tx: -1990000
(INFO) armoryd.py:3370 - New ZC tx: -4193393
(INFO) armoryd.py:3370 - New ZC tx: -2020640
(INFO) armoryd.py:3370 - New ZC tx: -18500000
(INFO) armoryd.py:3370 - New ZC tx: -106172
(INFO) armoryd.py:3370 - New ZC tx: -735709
(INFO) armoryd.py:3370 - New ZC tx: -5020640
(INFO) armoryd.py:3370 - New ZC tx: -790640
(INFO) armoryd.py:3370 - New ZC tx: -320640
(INFO) armoryd.py:3370 - New ZC tx: -73220
(INFO) armoryd.py:3370 - New ZC tx: -517525
(INFO) armoryd.py:3370 - New ZC tx: -2257937
(INFO) armoryd.py:3370 - New ZC tx: -1013920
(INFO) armoryd.py:3370 - New ZC tx: -8194190
(INFO) armoryd.py:3370 - New ZC tx: -2324457
(INFO) armoryd.py:3370 - New ZC tx: -11291727
(INFO) armoryd.py:3370 - New ZC tx: -7860640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -4300000
(INFO) armoryd.py:3370 - New ZC tx: -1151953
(INFO) armoryd.py:3370 - New ZC tx: -11920640
(INFO) armoryd.py:3370 - New ZC tx: -318831
(INFO) armoryd.py:3370 - New ZC tx: -2143039
(INFO) armoryd.py:3370 - New ZC tx: -1020640
(INFO) armoryd.py:3370 - New ZC tx: -42221280
(INFO) armoryd.py:3370 - New ZC tx: -10532429
(INFO) armoryd.py:3370 - New ZC tx: -28539483
(INFO) armoryd.py:3370 - New ZC tx: -2257059
(INFO) armoryd.py:3370 - New ZC tx: -113000
(INFO) armoryd.py:3370 - New ZC tx: -2020640
(INFO) armoryd.py:3370 - New ZC tx: -4107234
(INFO) armoryd.py:3370 - New ZC tx: -9704437
(INFO) armoryd.py:3370 - New ZC tx: -15220640
(INFO) armoryd.py:3370 - New ZC tx: -4920640
(INFO) armoryd.py:3370 - New ZC tx: -79360
(INFO) armoryd.py:3370 - New ZC tx: -100000
(INFO) armoryd.py:3370 - New ZC tx: -52462500
(INFO) armoryd.py:3370 - New ZC tx: -138673605
(INFO) armoryd.py:3370 - New ZC tx: -10566460
(INFO) armoryd.py:3370 - New ZC tx: -10544487
(INFO) armoryd.py:3370 - New ZC tx: -1400000
(INFO) armoryd.py:3370 - New ZC tx: -1081640
(INFO) armoryd.py:3370 - New ZC tx: -3020640
(INFO) armoryd.py:3370 - New ZC tx: -33072396
(INFO) armoryd.py:3370 - New ZC tx: -945195
(INFO) armoryd.py:3370 - New ZC tx: -5030640
(INFO) armoryd.py:3370 - New ZC tx: -252600
(INFO) armoryd.py:3370 - New ZC tx: -223920640
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -11229000
(INFO) armoryd.py:3370 - New ZC tx: -9990000
(INFO) armoryd.py:3370 - New ZC tx: -14114489
(INFO) armoryd.py:3370 - New ZC tx: -7862640
(INFO) armoryd.py:3370 - New ZC tx: -14163165
(INFO) armoryd.py:3370 - New ZC tx: -11648277
(INFO) armoryd.py:3370 - New ZC tx: -2670298
(INFO) armoryd.py:3370 - New ZC tx: -2520640
(INFO) armoryd.py:3370 - New ZC tx: -2330640
(INFO) armoryd.py:3370 - New ZC tx: -1013640
(INFO) armoryd.py:3370 - New ZC tx: -1302327
(INFO) armoryd.py:3370 - New ZC tx: -481734
(INFO) armoryd.py:3370 - New ZC tx: -18670640
(INFO) armoryd.py:3370 - New ZC tx: -5650879
(INFO) armoryd.py:3370 - New ZC tx: -10490638
(INFO) armoryd.py:3370 - New ZC tx: -931080
(INFO) armoryd.py:3370 - New ZC tx: -11244500
(INFO) armoryd.py:3370 - New ZC tx: -220640
(INFO) armoryd.py:3370 - New ZC tx: -380640
(INFO) armoryd.py:3370 - New ZC tx: -1020640
(INFO) armoryd.py:3370 - New ZC tx: -1485205
(INFO) armoryd.py:3370 - New ZC tx: -2091110
(INFO) armoryd.py:3370 - New ZC tx: -1078640
-INFO  - 1483206291: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206293: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206304: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206309: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206310: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206314: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206316: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206319: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206327: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206328: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206344: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206365: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206367: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206382: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206396: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483206397: (SocketObject.cpp:517) POLLIN recv return 0
terminate called after throwing an instance of 'DbErrorMsg'

I have to start daemon again.

m.

[goatpig]

Try to patch this PR to your code base:

https://github.com/goatpig/BitcoinArmory/pull/158

If that doesn't fix it, I'll have to reproduce the issue myself and deliver the fix for 0.96.

[marekNowak]

I have patched the code. Now Armory deamon was terminated after c.a 9 hours of running, then I have started it again and it took c.a 8-9 hours and the error happen again:

In my opinion the patch helped but there must be something more which cuase error:

New Block:  446454
New Block:  446455
New Block:  446456
New Block:  446457
New Block:  446458
New Block:  446459
New Block:  446460
New Block:  446461
New Block:  446462
New Block:  446463
New Block:  446464
-INFO  - 1483465297: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465299: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465309: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465314: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465316: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465319: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465321: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465325: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465333: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465334: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465350: (SocketObject.cpp:517) POLLIN recv return 0
-INFO  - 1483465398: (SocketObject.cpp:517) POLLIN recv return 0


Can I help You with investigating problem by running my deamon with some kinde of debug etc? I don't know wheter it will be easy to find a reason of problem beause each time I had to wait couple of hours for it.


m.

[goatpig]

You can build with DEBUG=1 and run ArmoryDB in gdb. You can then post the backtrace after it hangs, although that will require some extra work to be really useful, as you would need to show me the bt for all relevant threads (seeing you run 100 ZC threads, it will be tedious to filter through).

[marekNowak]

should I decrease amount of threads to for example 20 ?

m.

[goatpig]

If the chance for the code to fail is not reduced, that's a way to go about it.

[marekNowak]

I will try.

m.

[marekNowak]

I have changed number of threads to 15, then

1) from command line "make DEBUG=1"
2) from command line "gdb ArmoryDB"
3) <gdb> run

now every couple of seconds log in gdb change and looks like:

[New Thread 0x7fff8f7fe700 (LWP 26326)]
[Thread 0x7fff8f7fe700 (LWP 26326) exited]
[New Thread 0x7fff8f7fe700 (LWP 26327)]
[Thread 0x7fff8f7fe700 (LWP 26327) exited]
[New Thread 0x7fff8f7fe700 (LWP 26328)]
[Thread 0x7fff8f7fe700 (LWP 26328) exited]
[New Thread 0x7fff8f7fe700 (LWP 26329)]
[Thread 0x7fff8f7fe700 (LWP 26329) exited]
[New Thread 0x7fff8f7fe700 (LWP 26330)]
[Thread 0x7fff8f7fe700 (LWP 26330) exited]
[New Thread 0x7fff8f7fe700 (LWP 26331)]
[Thread 0x7fff8f7fe700 (LWP 26331) exited]
[New Thread 0x7fff8f7fe700 (LWP 26332)]
[Thread 0x7fff8f7fe700 (LWP 26332) exited]
[New Thread 0x7fff8f7fe700 (LWP 26333)]
[Thread 0x7fff8f7fe700 (LWP 26333) exited]
[New Thread 0x7fff8f7fe700 (LWP 26334)]
[Thread 0x7fff8f7fe700 (LWP 26334) exited]
[New Thread 0x7fff8f7fe700 (LWP 26335)]
[Thread 0x7fff8f7fe700 (LWP 26335) exited]
[New Thread 0x7fff8f7fe700 (LWP 26336)]
[Thread 0x7fff8f7fe700 (LWP 26336) exited]
[New Thread 0x7fff8f7fe700 (LWP 26337)]
[Thread 0x7fff8f7fe700 (LWP 26337) exited]
[New Thread 0x7fff8f7fe700 (LWP 26338)]
[Thread 0x7fff8f7fe700 (LWP 26338) exited]
[New Thread 0x7fff8f7fe700 (LWP 26339)]
[Thread 0x7fff8f7fe700 (LWP 26339) exited]
[New Thread 0x7fff8f7fe700 (LWP 26340)]
[Thread 0x7fff8f7fe700 (LWP 26340) exited]
[New Thread 0x7fff8f7fe700 (LWP 26341)]
[Thread 0x7fff8f7fe700 (LWP 26341) exited]
[New Thread 0x7fff8f7fe700 (LWP 26342)]
[Thread 0x7fff8f7fe700 (LWP 26342) exited]


armoryD is running. Is it all right? Should I wait now for error and the gdb will stop? then I will may type "bt all"?

Thank You in advance for Your help

m.

[goatpig]

This is just gdb reporting threads starting and ending, nothing to see here. When the code crashes, you will see a SIGSEGV or SIGTERM at the bottom then prompt back at gdb> ready for your input.

Then post only the output for "bt" and wait on instructions.