Starting preliminary 0.94 testing - "Headless fullnode"

[Carlton Banks]

My current machine is 6 core cpu (12 execution threads with HT), 32GB ddr4, SATA SSD RAID0 and fiber internet (~250Mb/s). And I'm thinking I should go with 64GB RAM soon. I had thought of buying a dual CPU board and go with a couple Haswell v3 Xeons but I backed off. Can't blow all my bitcoins on a PC =P.

For info, it takes 280~300 sec to scan the DB with ~5k addresses, about 80sec to build, and another ~100 sec for the header reading and what not (that could be reduced to 15sec with some effort).

Perhaps your FAQ should list minimum and recommended hardware specs.

I am running my full node on a $35 ARM single board computer with 1 GB ram.

The letstalkbitcoin guys say they used to use armory, and created a new tipping address for every episode, but stopped using it because it became too slow.

You can't compare the resource cost of initial synchronization with that of maintenance. I expect it would take your board over 2 weeks to sync the blockchain from scratch, yet it can process new blocks at an acceptable pace.

And the bitcoin core developers will likely improve that balance when they eventually get around to using libsecp256k1 for transaction verification as well as signing. That's probably the most significant bottleneck on these ARM systems, lack of clockspeed to handle compute-expensive crypto functions. Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor.

[goatpig]

Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor.

I expect it will be that much of an improvement, OpenSSL is painfully slow at ECDSA.

[Carlton Banks]

6e839bc fixed something for me unexpectedly, now getting 288 headers and 288 block files processed whereas before was getting either 285 or 286. Another difference is that the OS handles the interrupt overload when tx scanning alot more gracefully, background tasks in other domains are now given near full opportunity to take as contiguous processing cycles as they need, and the scanning appears to be hammering the CPU just as heavily. i.e. OS is usable simultaneously with scanning

[goatpig]

I changed an aggressive lock around key addressing that was meant to cover certain stuff for supernode, and is unnecessary for fullnode. Now it's limited to supernode. I guess that untangled some other part of the process.

[doug_armory]

0.94 is the first version to reduce hardware requirements (120MB disk space from 30GB+,  <2GB RAM from 8+) while increasing performance AND increasing scalability.

Seriously everyone, 0.94 is shaping up nicely. I've been flying by the seat of my pants and using 0.94 regularly since the first major chances were committed. After some initial crashes (totally understandable given the nature of software development), it worked really well for me. I don't doubt that there are bugs here and there - Carlton Banks inadvertently showed us quite a few serious bugs - and I encourage everybody to bang on the test builds when they come out. I'm just saying that, for my purposes, 0.94 is quite nice and does go a bit easier on the system.

Cutting the demand for CPU cycles by 6 or 8 as is claimed could cut the intial sync time by a similar factor.

I expect it will be that much of an improvement, OpenSSL is painfully slow at ECDSA.

I don't think it's that OpenSSL is painfully slow at ECDSA - I've found it to be okay, and probably a bit better than Crypto++ - as much as libsecp256k1 is specifically designed to optimize the hell out of working with the secp256k1 ECDSA curve (i.e., the curve Bitcoin uses). I believe Wladimir & Pieter have both reported 6-8x speed increases in verifications. They're just concerned about the possibility of a specially crafted Bitcoin transaction being validated in libsecp256k1 and not in OpenSSL or vice versa (i.e., a fork would occur). So, for now, libsecp256k1 is used only for signing, where it's apparently easier to test all the various use cases.

Anyway, yes, once the switch is thrown and libsecp256k1 is used for verification, BC Core should speed up quite a bit. The main contributors are doing some pretty cool things if you pay attention to the commits and the related conversations.

[goatpig]

I don't think it's that OpenSSL is painfully slow at ECDSA - I've found it to be okay, and probably a bit better than Crypto++ - as much as libsecp256k1 is specifically designed to optimize the hell out of working with the secp256k1 ECDSA curve (i.e., the curve Bitcoin uses). I believe Wladimir & Pieter have both reported 6-8x speed increases in verifications. They're just concerned about the possibility of a specially crafted Bitcoin transaction being validated in libsecp256k1 and not in OpenSSL or vice versa (i.e., a fork would occur). So, for now, libsecp256k1 is used only for signing, where it's apparently easier to test all the various use cases.

Anyway, yes, once the switch is thrown and libsecp256k1 is used for verification, BC Core should speed up quite a bit. The main contributors are doing some pretty cool things if you pay attention to the commits and the related conversations.

I should rephrase. What I meant is OpenSSL is painfully slow at secp256k1 because it has none of the specific optimizations for that curve. It is coded to sign any curve, so that's understandable.

[Carlton Banks]

What was added for 875ac898db seems to have killed off the GTK+ CRITICAL assertion fail warning, for me at least. It appeared before only in runs after the initial build and scan was shutdown and restarted, but now it doesn't appear even then. Not that it was causing any problems at that point, just thought it might help improve your understand of what's causing it.

[goatpig]

That's really weird because the change affects code that takes place after scans, and the Gtk error message shows up early on startup (most likely when a build or scan operation is underway).

[Carlton Banks]

That's really weird because the change affects code that takes place after scans, and the Gtk error message shows up early on startup (most likely when a build or scan operation is underway).

That pattern didn't apply to my later experience of the error (see above), but then again, it didn't cause any crashes or otherwise aberrant behaviour. And now it's not appearing at all. Make of it what you will (if the long term goal is still to ditch Qt 4, then it's possibly not worth pursuing).

[Carlton Banks]

Forget it completely lol, caught it again. I'm tempted to say that random instances of this warning can occur say 4-5 times consecutively for no apparent reason, likewise the inverse (no warning several times in a row).

[Carlton Banks]

Supernode

Just giving it a test. It didn't tolerate being interrupted and restarted; I gave it 2 hours to complete yesterday, and it only got ~40% finished. Starting again today produced a std:error exception relating to reading a block file. Otherwise seemingly ok, will mess around with the fully built Db later today.

[Carlton Banks]

Same issue again with supernode, some console output from a restart after a successful quit:

Code:

Do wallet checking    : True
(ERROR) ArmoryQt.py:1311 - 4 attempts to load blockchain failed.  Remove mempool.bin.
(ERROR) ArmoryQt.py:1316 - File mempool.bin does not exist. Nothing deleted.
-INFO  - 1435590364: (BlockUtils.cpp:873) blkfile dir: /home/user/.bitcoin/blocks
-INFO  - 1435590364: (BlockUtils.cpp:874) lmdb dir: /home/user/.armory/databases
-INFO  - 1435590364: (lmdb_wrapper.cpp:439) Opening databases...
-INFO  - 1435590364: (BlockUtils.cpp:1223) Executing: doInitialSyncOnLoad
-INFO  - 1435590364: (BlockUtils.cpp:1295) Total number of blk*.dat files: 291
-INFO  - 1435590364: (BlockUtils.cpp:1296) Total blockchain bytes: 38,901,027,826
-INFO  - 1435590364: (BlockUtils.cpp:1789) Reading headers from db
-INFO  - 1435590370: (BlockUtils.cpp:1815) Found 363050 headers in db
-DEBUG - 1435590370: (Blockchain.cpp:214) Organizing chain w/ rebuild
-INFO  - 1435590374: (BlockUtils.cpp:1337) Left off at file 290, offset 5090655
-INFO  - 1435590374: (BlockUtils.cpp:1340) Reading headers and building chain...
-INFO  - 1435590374: (BlockUtils.cpp:1341) Starting at block file 290 offset 5090655
-INFO  - 1435590374: (BlockUtils.cpp:1343) Block height 363035
-INFO  - 1435590374: (BlockUtils.cpp:345) parsing headers in file 290
-DEBUG - 1435590374: (Blockchain.cpp:214) Organizing chain w/ rebuild
-ERROR - 1435590454: (BlockUtils.cpp:559) std::exception (error encountered processing block at byte 99610185 file /home/user/.bitcoin/blocks/blk00143.dat, blocksize 707891)
(ERROR) Networking.py:366 - ***Connection to Satoshi client LOST!  Attempting to reconnect...
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times
-INFO  - 1435590523: (BlockUtils.cpp:1380) Looking for first unrecognized block
-INFO  - 1435590523: (BlockUtils.cpp:1384) Updating Headers in DB
-INFO  - 1435590523: (BDM_mainthread.cpp:273) Stop requested detected
-INFO  - 1435590523: (BDM_mainthread.cpp:317) UI asked build/scan thread to finish
(WARNING) SDM.py:769 - Overriding not-available state. This should happen 0-5 times

[goatpig]

Looks like some false positive being blown out of proportion. Sadly, I have some short term assignment till Tuesday so I have to put 0.94 on hold until then.

[Carlton Banks]

Looks like some false positive being blown out of proportion. Sadly, I have some short term assignment till Tuesday so I have to put 0.94 on hold until then.

Following more testing, I think I may have been misinterpreting. Armory appears to deadlock upon std:exception file not found, but this actually seems to be attributable to some interaction between SDM mode and the supernode DB build code; running bitcoind manually will generate the exception, but not the apparent deadlock. I suspect if I'd walked away for 10 minutes then it may have continued correctly, and it's already been said that SDM needs an overhaul so I didn't try testing it with supernode again (I figured testing multi-threaded scans with SDM would be worthwhile seeing as it's the default Armory setting that new users will encounter, I can confirm that it works as well with multi-threaded scans as it did with the previous scanning logic)

Supernode DB builds fine (given the many hours that takes), that code seems robust to hours of testing, including several deliberate quit/restarts. Segfaults ensue during tx scanning.

Also tested 5c0497f03b today (rebuild+scan), no problems so far.

[goatpig]

I have not had time to deal with your bug reports on supernode yet, working a few issues with parallel address scanning and zero conf balance. I expect I'll have time to go over your issues before the testing builds are released (that's the last item on my bug list btw).

Also, it is expected supernode users will manage their own Core instance, and I've personally never tested it with auto managed bitcoind.

[m.fridge]

I get a segmentation fault during the "Organizing Blockchain" step.

The last messages:

Code:

[New Thread 0xac9f1b40 (LWP 12492)]
[Thread 0xac9f1b40 (LWP 12492) exited]
[New Thread 0xac9f1b40 (LWP 12493)]
[Thread 0xac9f1b40 (LWP 12493) exited]
[New Thread 0xac9f1b40 (LWP 12494)]
[Thread 0xac9f1b40 (LWP 12494) exited]
-INFO  - 1436209883: (BlockUtils.cpp:1651) Loading block data... file 0 offset 0
[New Thread 0xac9f1b40 (LWP 12495)]
-INFO  - 1436209883: (BlockUtils.cpp:395) reading blocks from file 0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xab9b5b40 (LWP 12174)]
0xb7d87062 in ?? () from /lib/libc.so.6

And the backtrace:

Code:

#0  0xb7d87062 in ?? () from /lib/libc.so.6
#1  0xb7d852b5 in ?? () from /lib/libc.so.6
#2  0xb4bf23b0 in BlockDataManager_LevelDB::BitcoinQtBlockFiles::readRawBlocksFromFile(BlockFileAccessor&, unsigned int const&, unsigned long long, unsigned long long, std::function<void (BinaryDataRef const&, std::pair<unsigned int, unsigned long long> const&, unsigned int)> const&) const (this=0x8cba778, bfa=..., f=@0xab9b4edc: 0, blockFileOffset=0, stopBefore=2097361271, blockDataCallback=...) at BlockUtils.cpp:506
#3  0xb4bf1bd9 in BlockDataManager_LevelDB::BitcoinQtBlockFiles::readRawBlocks(LMDBBlockDatabase*, std::pair<unsigned int, unsigned long long>, std::pair<unsigned int, unsigned long long>, std::function<void (BinaryData const&, std::pair<unsigned int, unsigned long long> const&, unsigned int)> const&) (this=0x8cba778, db=0x83829d0, startAt=..., stopAt=..., blockDataCallback=...) at BlockUtils.cpp:401
#4  0xb4bea795 in BlockDataManager_LevelDB::loadBlockData (this=0x8344ef0, prog=..., stopAt=..., updateDupID=true) at BlockUtils.cpp:1653
#5  0xb4be8403 in BlockDataManager_LevelDB::loadDiskState(std::function<void (BDMPhase, double, unsigned int, unsigned int)> const&, bool) (this=0x8344ef0, progress=..., forceRescan=false) at BlockUtils.cpp:1395
#6  0xb4be70df in BlockDataManager_LevelDB::doInitialSyncOnLoad(std::function<void (BDMPhase, double, unsigned int, unsigned int)> const&) (this=0x8344ef0, progress=...) at BlockUtils.cpp:1224
#7  0xb4c9243f in BlockDataManagerThread::run (this=0x838e7f0) at BDM_mainthread.cpp:289
#8  0xb4c9349f in BlockDataManagerThread::thrun (_self=0x838e7f0) at BDM_mainthread.cpp:446
#9  0xb7e00ffb in start_thread () from /lib/libpthread.so.0
#10 0xb7d48fee in clone () from /lib/libc.so.6

[goatpig]

is it happening every time?

[m.fridge]

Yes, I tried it 3 times. Although only the last time in gdb, where I got the backtrace.

[Roy Badami]

Just to say that 5c0497f fixes the db build/scan issue for me.  Correct balances and no errors in the logs.

I did get one crash though (not during db build/scan):

Code:

Crashed Thread:  5

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x000000011981d001

Code:

Thread 5 Crashed:
0   _CppBlockUtils.so             	0x000000010417b006 BtcUtils::readVarInt(unsigned char const*, unsigned long, unsigned int*) + 22
1   _CppBlockUtils.so             	0x000000010417ac58 BinaryRefReader::get_var_int(unsigned char*) + 40
2   _CppBlockUtils.so             	0x000000010419951e BtcUtils::TxCalcLength(unsigned char const*, unsigned long, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >*, std::__1::vector<unsigned long, std::__1::allocator<unsigned long> >*) + 78
3   _CppBlockUtils.so             	0x00000001041bd443 Tx::unserialize(unsigned char const*, unsigned long) + 51
4   _CppBlockUtils.so             	0x00000001041bd7b0 Tx::unserialize(BinaryRefReader&) + 32
5   _CppBlockUtils.so             	0x000000010419db30 Tx::Tx(BinaryRefReader&) + 144
6   _CppBlockUtils.so             	0x00000001041a71e5 StoredHeader::unserializeFullBlock(BinaryRefReader, bool, bool) + 1237
7   _CppBlockUtils.so             	0x00000001041a8a66 StoredHeader::unserializeFullBlock(BinaryDataRef, bool, bool) + 54
8   _CppBlockUtils.so             	0x00000001041ce52e std::__1::__function::__func<BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool)::$_1, std::__1::allocator<BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool)::$_1>, void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)>::operator()(BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int&&) + 510
9   _CppBlockUtils.so             	0x00000001041d279b BlockDataManager_LevelDB::BitcoinQtBlockFiles::readRawBlocksFromFile(BlockFileAccessor&, unsigned int const&, unsigned long long, unsigned long long, std::__1::function<void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)> const&) const + 1611
10  _CppBlockUtils.so             	0x00000001041cf197 BlockDataManager_LevelDB::BitcoinQtBlockFiles::readHeaders(std::__1::pair<unsigned long, unsigned long long>, std::__1::function<void (BinaryDataRef const&, std::__1::pair<unsigned long, unsigned long long> const&, unsigned int)> const&, bool) const + 423
11  _CppBlockUtils.so             	0x00000001041c6bdb BlockDataManager_LevelDB::loadBlockHeadersStartingAt(ProgressReporter&, std::__1::pair<unsigned long, unsigned long long> const&, bool) + 283
12  _CppBlockUtils.so             	0x00000001041cb25e BlockDataManager_LevelDB::readBlkFileUpdate(BlockDataManager_LevelDB::BlkFileUpdateCallbacks const&) + 110
13  _CppBlockUtils.so             	0x0000000104235956 BlockDataManagerThread::run() + 3798
14  _CppBlockUtils.so             	0x00000001042349c9 BlockDataManagerThread::thrun(void*) + 9
15  libsystem_pthread.dylib       	0x00007fff8af3f899 _pthread_body + 138
16  libsystem_pthread.dylib       	0x00007fff8af3f72a _pthread_start + 137
17  libsystem_pthread.dylib       	0x00007fff8af43fc9 thread_start + 13

[goatpig]

Thanks for debug logs guys, will get to it soon.