Armory 0.95 testing phase

[goatpig]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

[achow101]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still getting tons of these errors:

Code:

Traceback (most recent call last):
  File "ArmoryQt.py", line 5771, in handleCppNotification
    self.updateWalletData()
  File "ArmoryQt.py", line 5735, in updateWalletData
    self.walletMap[wltid].getAddrDataFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 52, in inner
    return func(*args, **kwargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 3119, in getAddrDataFromDB
    countList = self.cppWallet.getAddrTxnCountsFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/CppBlockUtils.py", line 2354, in getAddrTxnCountsFromDB
    return _CppBlockUtils.BtcWallet_getAddrTxnCountsFromDB(self)
<class 'CppBlockUtils.DbErrorMsg'>: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f485abb8cf0> >

and it seems that it is stuck at "Scanning Transaction History 100% 1 second". Edit: It fixed itself.

Yay! It seems to work!

Edit2: Looking at the list of addresses in a wallet, it looks like any address that has a 0 balance is considered unused even though they have been used in the past. Double clicking on the address shows the transactions, but the adddresses in the list are marked as unused with 0 transactions.

Edit3: It doesn't detect new ZC transactions and nothing in the log indicates why.

[Carlton Banks]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still getting tons of these errors:

Code:

Traceback (most recent call last):
  File "ArmoryQt.py", line 5771, in handleCppNotification
    self.updateWalletData()
  File "ArmoryQt.py", line 5735, in updateWalletData
    self.walletMap[wltid].getAddrDataFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 52, in inner
    return func(*args, **kwargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 3119, in getAddrDataFromDB
    countList = self.cppWallet.getAddrTxnCountsFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/CppBlockUtils.py", line 2354, in getAddrTxnCountsFromDB
    return _CppBlockUtils.BtcWallet_getAddrTxnCountsFromDB(self)
<class 'CppBlockUtils.DbErrorMsg'>: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f485abb8cf0> >

and it seems that it is stuck at "Scanning Transaction History 100% 1 second".

Experiencing both issues, but I find the latter issue is just lengthy time without any progress feedback for the user completing the last step of the tx scan ("resolving tx hashes").

I appreciate that using a progress bar wouldn't work for an indicator... something to demonstrate the inability to estimate to the user might be an idea though (i.e. a striped progress bar that clearly doesn't ever complete, but does cascade at a speed related to how fast the tx-Resolve-Hash step is being called/finished). Otherwise that step could be mistaken for a program crash (potential source of frustration for new users, not to mention a source of common user complaints).

Otherwise, seems much better. Coinbase tx details bug fixed. ArmoryDb process still detaches itself from the parent process after Quit (i.e. survives when it should die).

[goatpig]

Edit3: It doesn't detect new ZC transactions and nothing in the log indicates why.

Is the ZC eventually mined?

[goatpig]

I appreciate that using a progress bar wouldn't work for an indicator... something to demonstrate the inability to estimate to the user might be an idea though (i.e. a striped progress bar that clearly doesn't ever complete, but does cascade at a speed related to how fast the tx-Resolve-Hash step is being called/finished). Otherwise that step could be mistaken for a program crash (potential source of frustration for new users, not to mention a source of common user complaints).

I'll add a progress bar for that stuff at some point. For now just assume this step has to happen before the scan completes.

[achow101]

Edit3: It doesn't detect new ZC transactions and nothing in the log indicates why.

Is the ZC eventually mined?

Yes. It only shows up after it is confirmed.

[goatpig]

Edit3: It doesn't detect new ZC transactions and nothing in the log indicates why.

Is the ZC eventually mined?

Yes. It only shows up after it is confirmed.

Do you experience the detached DB issue too?

[achow101]

Edit3: It doesn't detect new ZC transactions and nothing in the log indicates why.

Is the ZC eventually mined?

Yes. It only shows up after it is confirmed.

Do you experience the detached DB issue too?

Don't know, haven't and can't check (away from my machine).

[goatpig]

Still getting tons of these errors:

Code:

Traceback (most recent call last):
  File "ArmoryQt.py", line 5771, in handleCppNotification
    self.updateWalletData()
  File "ArmoryQt.py", line 5735, in updateWalletData
    self.walletMap[wltid].getAddrDataFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 52, in inner
    return func(*args, **kwargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 3119, in getAddrDataFromDB
    countList = self.cppWallet.getAddrTxnCountsFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/CppBlockUtils.py", line 2354, in getAddrTxnCountsFromDB
    return _CppBlockUtils.BtcWallet_getAddrTxnCountsFromDB(self)
<class 'CppBlockUtils.DbErrorMsg'>: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f485abb8cf0> >

Were you getting those while scanning?

[droark]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

[achow101]

Were you getting those while scanning?

I think so.

[goatpig]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

Can you:

1) upload the incriminated block file

2) try to sync against another copy of the testnet

[goatpig]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still getting tons of these errors:

Code:

Traceback (most recent call last):
  File "ArmoryQt.py", line 5771, in handleCppNotification
    self.updateWalletData()
  File "ArmoryQt.py", line 5735, in updateWalletData
    self.walletMap[wltid].getAddrDataFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 52, in inner
    return func(*args, **kwargs)
  File "/home/andy/bitcoin/BitcoinArmory/armoryengine/PyBtcWallet.py", line 3119, in getAddrDataFromDB
    countList = self.cppWallet.getAddrTxnCountsFromDB()
  File "/home/andy/bitcoin/BitcoinArmory/CppBlockUtils.py", line 2354, in getAddrTxnCountsFromDB
    return _CppBlockUtils.BtcWallet_getAddrTxnCountsFromDB(self)
<class 'CppBlockUtils.DbErrorMsg'>: <CppBlockUtils.DbErrorMsg; proxy of <Swig Object of type 'DbErrorMsg *' at 0x7f485abb8cf0> >

and it seems that it is stuck at "Scanning Transaction History 100% 1 second".

ArmoryDb process still detaches itself from the parent process after Quit (i.e. survives when it should die).

Both issues should be fixed now. Upgraded coin control to display utxos, give that a spin as well. Will work on ZC and progress bars next.

[goatpig]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

Can you:

1) upload the incriminated block file

2) try to sync against another copy of the testnet

Thanks for the file. Can you build a DB from scratch using --db-type=DB_BARE?

[droark]

I think most of the bugs and instability issues reported here should be fixed now. Testers, please pull dev and try again. DB format changed slightly, please start with a fresh folder.

Still seeing testnet crashes w/ ArmoryDB on OS X. Latest code. Same problem as before.

Can you:

1) upload the incriminated block file

2) try to sync against another copy of the testnet

Thanks for the file. Can you build a DB from scratch using --db-type=DB_BARE?

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

[goatpig]

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

Try the unit tests in the gtest folder. Both test binaries should pass valgrind --tool=massif

[droark]

You're welcome. Still crashes, unfortunately. Just to consolidate all the info, 0.94.1 on OSX doesn't crash, and these files don't crash on Linux at all. (I copied the testnet blockchain to Linux, and it was fine.)

Try the unit tests in the gtest folder. Both test binaries should pass valgrind --tool=massif

Hmmm. I'm seeing a couple of odd things after running Valgrind a few times.

Code:

[ RUN      ] BlockDir.HeadersFirst
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
[       OK ] BlockDir.HeadersFirst (21467 ms)

These messages pop up consistently at this point. These messages seem to be considered spurious by a lot of valgrind users. They apparently pop up quite a bit when running on OSX, which valgrind is slow to support. (In fact, AFAIK, they still haven't implemented full OSX 10.11 support.) I believe they sometimes come up when valgrind doesn't fully understand system calls. It's hard to say but it might be worth looking at this code anyway.

Code:

Failed to create map of file. Error Code: 22

Saw this once last night but can't reproduce it now for unknown reasons. I think it happened during the Load3Blocks_ZCchain test but I can't remember. Anyway, I believe 22 = EINVAL on OSX. I've written a small patch that uses strerror() in the future. The patch might be why this only came up once. (See below.)

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN      ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785== 
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785==  Non-existent physical address at address 0x1027D6000
==6785==    at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785==    by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785==    by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785==    by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function<bool (unsigned char const*, unsigned long, unsigned long)>) (DatabaseBuilder.cpp:381)
==6785==    by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>) (DatabaseBuilder.cpp:327)
==6785==    by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>, bool) const (DatabaseBuilder.cpp:217)
==6785==    by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785==    by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785==    by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785==    by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785==    by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785==    by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785== 
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Code:

[----------] 18 tests from BlockUtilsBare
[ RUN      ] BlockUtilsBare.Load5Blocks
[       OK ] BlockUtilsBare.Load5Blocks (30152 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_DamagedBlkFile
[       OK ] BlockUtilsBare.Load5Blocks_DamagedBlkFile (6538 ms)
[ RUN      ] BlockUtilsBare.Load4Blocks_Plus2
[       OK ] BlockUtilsBare.Load4Blocks_Plus2 (44924 ms)
[ RUN      ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2
[       OK ] BlockUtilsBare.Load4Blocks_ReloadBDM_ZC_Plus2 (82404 ms)
[ RUN      ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
[       OK ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers (89746 ms)
[ RUN      ] BlockUtilsBare.Load3Blocks_ZCchain
[       OK ] BlockUtilsBare.Load3Blocks_ZCchain (11110 ms)
[ RUN      ] BlockUtilsBare.Load3Blocks_RBF
[       OK ] BlockUtilsBare.Load3Blocks_RBF (497 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_FullReorg
[       OK ] BlockUtilsBare.Load5Blocks_FullReorg (3000 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_DoubleReorg
[       OK ] BlockUtilsBare.Load5Blocks_DoubleReorg (4670 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_ReloadBDM_Reorg
[       OK ] BlockUtilsBare.Load5Blocks_ReloadBDM_Reorg (19694 ms)
[ RUN      ] BlockUtilsBare.CorruptedBlock
[       OK ] BlockUtilsBare.CorruptedBlock (8482 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_RescanOps
[       OK ] BlockUtilsBare.Load5Blocks_RescanOps (136481 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_RescanEmptyDB
[       OK ] BlockUtilsBare.Load5Blocks_RescanEmptyDB (22092 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_RebuildEmptyDB
[       OK ] BlockUtilsBare.Load5Blocks_RebuildEmptyDB (68415 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_SideScan
[       OK ] BlockUtilsBare.Load5Blocks_SideScan (39226 ms)
[ RUN      ] BlockUtilsBare.Load5Blocks_GetUtxos
[       OK ] BlockUtilsBare.Load5Blocks_GetUtxos (55087 ms)
[ RUN      ] BlockUtilsBare.Load4Blocks_ZC_GetUtxos
[       OK ] BlockUtilsBare.Load4Blocks_ZC_GetUtxos (43231 ms)
[ RUN      ] BlockUtilsBare.LedgerSeDer
[       OK ] BlockUtilsBare.LedgerSeDer (20928 ms)
[----------] 18 tests from BlockUtilsBare (686681 ms total)

When the tests do succeed under valgrind, the BlockUtilsBare tests take forever. On their own, they don't take too long.

[goatpig]

Code:

[ RUN      ] BlockDir.HeadersFirst
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 2 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 4 times)
--6464-- UNKNOWN mach_msg unhandled MACH_SEND_TRAILER option (repeated 8 times)
[       OK ] BlockDir.HeadersFirst (21467 ms)

Never seen these. Sounds like OSX specific, not particularly concerned if they don't result in a segfault.

Code:

Failed to create map of file. Error Code: 22

Can happen sometimes when the OS doesn't create file handles fast enough. Each test completely wipes the db. On some rare occasions (like a system on full load), lmdb will create the db file and try to access it after the call returns but before the OS actually flushes the file to the system (which is a very rare and unexpected window to hit). Not a concern per se, file creation only takes place with empty db folders.

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN      ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785== 
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785==  Non-existent physical address at address 0x1027D6000
==6785==    at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785==    by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785==    by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785==    by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function<bool (unsigned char const*, unsigned long, unsigned long)>) (DatabaseBuilder.cpp:381)
==6785==    by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>) (DatabaseBuilder.cpp:327)
==6785==    by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>, bool) const (DatabaseBuilder.cpp:217)
==6785==    by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785==    by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785==    by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785==    by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785==    by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785==    by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785== 
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Now that's something I can't reproduce on my end. My valgrind tests all pass 100%. Can you try reproducing this on a linux setup too? Definitely worth looking into.

[droark]

Anyway, for some reason, valgrind tripped up here and caused some Armory error code to be triggered (or maybe valgrind just caught the error?). This is actually code that was put in place 6 months ago in order for Armory to compile properly after some other changes. Might be worth looking here too, even though I never see the message when running ArmoryDB.

Code:

[ RUN      ] BlockUtilsBare.Load3Blocks_ZC_Plus3_TestLedgers
==6785== 
==6785== Process terminating with default action of signal 10 (SIGBUS)
==6785==  Non-existent physical address at address 0x1027D6000
==6785==    at 0x101935DA0: _platform_memcmp (in /usr/lib/system/libsystem_platform.dylib)
==6785==    by 0x100107213: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==6785==    by 0x1000F97DC: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==6785==    by 0x1002E7C46: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::__1::function<bool (unsigned char const*, unsigned long, unsigned long)>) (DatabaseBuilder.cpp:381)
==6785==    by 0x1002E67BD: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>) (DatabaseBuilder.cpp:327)
==6785==    by 0x1002E62EA: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool)::$_2::operator()(unsigned short, unsigned long, std::__1::shared_ptr<BlockOffset>, bool) const (DatabaseBuilder.cpp:217)
==6785==    by 0x1002E4C06: DatabaseBuilder::updateBlocksInDB(std::__1::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool) (DatabaseBuilder.cpp:257)
==6785==    by 0x1002E8586: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==6785==    by 0x10013C241: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==6785==    by 0x1001F160D: BlockDataManagerThread::run()::$_4::operator()() const (BDM_mainthread.cpp:178)
==6785==    by 0x1001F0EB2: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==6785==    by 0x1001F04AC: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
--6785:0:schedule VG_(sema_down): read returned -4
==6785== 
Killed: 9

Seeing this consistently after one successful run. The invalid address varies slightly but it's always around this particular value. Definitely not good.

Now that's something I can't reproduce on my end. My valgrind tests all pass 100%. Can you try reproducing this on a linux setup too? Definitely worth looking into.

Hmmm. I'm seeing something different. After one successful run on Linux (Ubuntu 16.04), I kept getting the "Error Code: 22" error on Load3Blocks_ZC_Plus3_TestLedgers and had to disable it. After that, I did see this once.

Code:

[ RUN      ] BlockUtilsBare.Load3Blocks_ZCchain
==20207== 
==20207== Process terminating with default action of signal 7 (SIGBUS)
==20207==  Non-existent physical address at address 0x403B000
==20207==    at 0x5A5D851: __memcmp_sse4_1 (memcmp-sse4.S:812)
==20207==    by 0x88E929: BinaryDataRef::operator==(BinaryData const&) const (BinaryData.h:891)
==20207==    by 0x88E982: BinaryDataRef::operator!=(BinaryData const&) const (BinaryData.h:901)
==20207==    by 0x9BCB7D: DatabaseBuilder::parseBlockFile(unsigned char const*, unsigned long, unsigned long, std::function<bool (unsigned char const*, unsigned long, unsigned long)>) (DatabaseBuilder.cpp:381)
==20207==    by 0x9BC5F1: DatabaseBuilder::addBlocksToDB(BlockDataLoader&, unsigned short, unsigned long, std::shared_ptr<BlockOffset>) (DatabaseBuilder.cpp:328)
==20207==    by 0x9BB9FB: DatabaseBuilder::updateBlocksInDB(std::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool)::{lambda(unsigned short, unsigned long, std::shared_ptr<BlockOffset>, bool)#1}::operator()(unsigned short, unsigned long, std::shared_ptr<BlockOffset>, bool) const (DatabaseBuilder.cpp:217)
==20207==    by 0x9BBEA7: DatabaseBuilder::updateBlocksInDB(std::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool, bool) (DatabaseBuilder.cpp:258)
==20207==    by 0x9BD2CD: DatabaseBuilder::update() (DatabaseBuilder.cpp:473)
==20207==    by 0x8B97C8: BlockDataManager::readBlkFileUpdate(BlockDataManager::BlkFileUpdateCallbacks const&) (BlockUtils.cpp:1559)
==20207==    by 0x92A2F5: BlockDataManagerThread::run()::{lambda()#2}::operator()() const (BDM_mainthread.cpp:178)
==20207==    by 0x92AAEF: BlockDataManagerThread::run() (BDM_mainthread.cpp:236)
==20207==    by 0x92AFFB: BlockDataManagerThread::thrun(void*) (BDM_mainthread.cpp:261)
==20207== 
Killed

Unlike the OSX crash, which is consistent, this one only happens occasionally.

Also, I forgot to run ContainerTests. No crashes, at least not yet. First is the normal version, second is the Valgrind version, which is still running after almost an hour. (I'll update it if it finishes anytime soon.) This is on Linux. I'll update if OSX has significant differences.

Code:

[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 8 tests from ContainerTests
[ RUN      ] ContainerTests.TransactionalMap
[       OK ] ContainerTests.TransactionalMap (216 ms)
[ RUN      ] ContainerTests.PileTest_Sequential
[       OK ] ContainerTests.PileTest_Sequential (508 ms)
[ RUN      ] ContainerTests.PileTest_Concurrent
[       OK ] ContainerTests.PileTest_Concurrent (934 ms)
[ RUN      ] ContainerTests.StackTest_Sequential
[       OK ] ContainerTests.StackTest_Sequential (718 ms)
[ RUN      ] ContainerTests.StackTest_Concurrent
[       OK ] ContainerTests.StackTest_Concurrent (590 ms)
[ RUN      ] ContainerTests.BlockingStackTest_Sequential
[       OK ] ContainerTests.BlockingStackTest_Sequential (689 ms)
[ RUN      ] ContainerTests.BlockingStackTest_Concurrent
ContainerTests.cpp:576: Failure
Expected: (theStack.count()) != (0), actual: 0 vs 0
[  FAILED  ] ContainerTests.BlockingStackTest_Concurrent (1099 ms)
[ RUN      ] ContainerTests.TimedStackTest_Concurrent
[       OK ] ContainerTests.TimedStackTest_Concurrent (2850 ms)
[----------] 8 tests from ContainerTests (7604 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 1 test case ran. (7604 ms total)
[  PASSED  ] 7 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] ContainerTests.BlockingStackTest_Concurrent

 1 FAILED TEST

Code:

==23036== Massif, a heap profiler
==23036== Copyright (C) 2003-2015, and GNU GPL'd, by Nicholas Nethercote
==23036== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==23036== Command: ./ContainerTests
==23036== 
[==========] Running 8 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 8 tests from ContainerTests
[ RUN      ] ContainerTests.TransactionalMap
[       OK ] ContainerTests.TransactionalMap (9033 ms)
[ RUN      ] ContainerTests.PileTest_Sequential
[       OK ] ContainerTests.PileTest_Sequential (6640 ms)
[ RUN      ] ContainerTests.PileTest_Concurrent
ContainerTests.cpp:249: Failure
Value of: poptally
  Actual: 540229271904773
Expected: pushtally
Which is: 858672733906290
ContainerTests.cpp:250: Failure
Value of: 0
Expected: thePile.count()
Which is: 296875
[  FAILED  ] ContainerTests.PileTest_Concurrent (71804 ms)
[ RUN      ] ContainerTests.StackTest_Sequential
[       OK ] ContainerTests.StackTest_Sequential (3947 ms)
[ RUN      ] ContainerTests.StackTest_Concurrent
ContainerTests.cpp:400: Failure
Value of: poptally
  Actual: 809112423041638
Expected: pushtally
Which is: 859016864469035
ContainerTests.cpp:401: Failure
Value of: 0
Expected: theStack.count()
Which is: 46511
[  FAILED  ] ContainerTests.StackTest_Concurrent (106924 ms)
[ RUN      ] ContainerTests.BlockingStackTest_Sequential
[       OK ] ContainerTests.BlockingStackTest_Sequential (11527 ms)
[ RUN      ] ContainerTests.BlockingStackTest_Concurrent

EDIT: Interestingly, on OSX, ContainerTests passes every time, yet when running under valgrind, BlockingStackTest_Concurrent seems to get stuck, just like on Linux.

[goatpig]

Hmmm. I'm seeing something different. After one successful run on Linux (Ubuntu 16.04), I kept getting the "Error Code: 22" error on Load3Blocks_ZC_Plus3_TestLedgers and had to disable it. After that, I did see this once.

Clearly there's something off with the comparison function, or it is fed an empty pointer eventually. Is it happening at random in any test or always the same spot?

Also, I forgot to run ContainerTests. No crashes, at least not yet. First is the normal version, second is the Valgrind version, which is still running after almost an hour. (I'll update it if it finishes anytime soon.) This is on Linux. I'll update if OSX has significant differences.

All those are benign or false positives, nothing alarming here at least

EDIT: Interestingly, on OSX, ContainerTests passes every time, yet when running under valgrind, BlockingStackTest_Concurrent seems to get stuck, just like on Linux.

Yep, expected, forgot to rework the BlockingStack clean up method, which will end up hanging on heavy loads. It's only used when shutting down the DB, so I'm not in a hurry to fix it yet.