Starting preliminary 0.94 testing - "Headless fullnode"

[etotheipi]

So, preliminarily 0.94 has stabilized, but it hasn't seen much testing outside "the lab."  I would like to invite folks who can build, to checkout the ffreeze branch (feature-freeze) and try it out while we continue our internal testing.

https://github.com/etotheipi/BitcoinArmory/tree/ffreeze

++ In 0.94 the Armory home dir databases folder should now be less than 150 MB!  This is what we refer to as "headless fullnode"
-- All database formats have changed. You must rebuild your databases! (and we don't have a clean way to help the user transition from the GUI).


Therefore, to use/test 0.94, please change your datadir, dbdir, or delete the databases folder in the Armory home directory.  Though, you might want to just move the directory for the moment, in case you have to go back.  Luckily, since the new version uses so much less space, it's not a big deal to create another directory for it!

It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore.   It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode).

Along with this, there's a ton of bug fixes and stability enhancements. We're looking forward to the database design finally settling down and stabilizing.

PLEASE checkout the ffreeze branch and help test!  Once we get a little bit more testing we'll do a semi-official testing release with a proper bug bounty!

[1715178254]

1715178254

[1715178254]

1715178254

[1715178254]

1715178254

[1715178254]

1715178254

[doug_armory]

FWIW, I've been using 0.94 test builds on OS X for awhile now. They're definitely not perfect but they've been pretty good overall so far. It sure is nice to have a much smaller, faster DB. All hail goatpig. (Of course, my old one is backed up just in case. Make backups, lest ye tempt fate.) If you're a power user, you probably wouldn't want this as your daily driver. If you're curious or just use Armory once in awhile, I think it's okay to build this and use it for non-mission-critical tasks.

[TierNolan]

What does headless full node mean?  It just stores the UTXO data or just recent blocks?

[Edit]

On an unrelated topic, I tried to scan the QR code in the README and it doesn't scan with the bitcoin smartphone app.

It is like an extra "?" is added to the end.

[Carlton Banks]

Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish.

Possibly the source of offence:

Code:

(python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
Killed

Logs are otherwise uneventful

[etotheipi]

Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish.

Possibly the source of offence:

Code:

(python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
Killed

Logs are otherwise uneventful

Did you pull the latest ffreeze?  There was a problem with thread synchronization a couple days ago, and we all experienced the exact same issue.  As of yesterday (when I posted this thread) that was fixed and no one else had the issue.  Did you wipe the database?  Is the version number 0.93.99.1?

[Carlton Banks]

Did you pull the latest ffreeze?  

I did:

Code:

git clone git://github.com/etotheipi/BitcoinArmory.git
cd ~/BitcoinArmory
git checkout ffreeze
make

Did you wipe the database?  

Yes

Is the version number 0.93.99.1?

Yes

[goatpig]

Whole thing crashes when scanning tx's. CPU/Memory usage goes vertical-ish.

Possibly the source of offence:

Code:

(python:3377): Gtk-CRITICAL **: IA__gtk_progress_configure: assertion `value >= min && value <= max' failed
Killed

Logs are otherwise uneventful

How far did it get in block height?

[Carlton Banks]

There is a pattern to it. It's very crash prone from blocks 0 > ~200,000. Then it stayed up until top block, minus one seg fault. Doesn't correspond to any pattern in the wallet file/s I'm trying it out with, or I don't think so anyway (I originally though it might be coinbase tx's that were the problem, but I get this with any wallet)

[Carlton Banks]

Also, SDM mode produces this:

Code:

Traceback (most recent call last):
  File "/usr/lib/armory-testing/ArmoryQt.py", line 7129, in <module>
    form = ArmoryMainWindow(splashScreen=SPLASH)
  File "/usr/lib/armory-testing/ArmoryQt.py", line 259, in __init__
    self.startBitcoindIfNecessary()
  File "/usr/lib/armory-testing/ArmoryQt.py", line 2441, in startBitcoindIfNecessary
    self.setSatoshiPaths()
  File "/usr/lib/armory-testing/ArmoryQt.py", line 2499, in setSatoshiPaths
    TheTDM.setSatoshiDir(self.satoshiHomePath)
AttributeError: 'FakeTDM' object has no attribute 'setSatoshiDir'

and then crashes.

[goatpig]

Also, SDM mode produces this:

Code:

Traceback (most recent call last):
  File "/usr/lib/armory-testing/ArmoryQt.py", line 7129, in <module>
    form = ArmoryMainWindow(splashScreen=SPLASH)
  File "/usr/lib/armory-testing/ArmoryQt.py", line 259, in __init__
    self.startBitcoindIfNecessary()
  File "/usr/lib/armory-testing/ArmoryQt.py", line 2441, in startBitcoindIfNecessary
    self.setSatoshiPaths()
  File "/usr/lib/armory-testing/ArmoryQt.py", line 2499, in setSatoshiPaths
    TheTDM.setSatoshiDir(self.satoshiHomePath)
AttributeError: 'FakeTDM' object has no attribute 'setSatoshiDir'

and then crashes.

That's the torrent manager failing to find your /blocks folder. That suggests it's looking at an empty folder.

[Carlton Banks]

Can't get the SDM code to find satoshi datadir no matter what I do. Works fine without it though.

[etotheipi]

This reminds me that we should probably remove all torrent-related code from Armory for 0.94.  It is no longer needed with Core 0.10+ (headers first), and it was actually designed such that you could remove the torrent code directory and Armory would bypass trying to use it.    We wanted to wait for sufficient people to have upgraded to Core 0.10 and Armory 0.93+, and I think that's likely to be the case now (we couldn't upgrade the Core links in the secure downloader, without inadvertantly leading to 0.92 users installing Core 0.10 which doesn't work).

[Carlton Banks]

Ok, any wallet that needs it's tx hints recording (plus whatever else) is somewhere close to the problem. If I create an empty wallet and rebuild the Db, nice smooth scanning, except for a couple of seg faults

[justusranvier]

It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore.   It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode).

Will it ever be possible to run Armory without requiring filesystem-level access to the block files?

Seems like a change such as this makes such an operating mode more difficult.

I run bitcoind and Armory in separate virtual machines, and finding a way to share those files in a way that gets all the permissions right and satisfies LevelDB's locking requirements is non-trivial.

It would be great to have an option for Armory to maintain its own database using blocks it obtains via its peer connection to the node, rather than via direct filesystem access, without requiring a supernode.

[goatpig]

It's also is a heck of a lot faster to start up Armory the first time once Core is sync'd, since we're not building a 30+ GB database anymore.   It's called "headless fullnode" because it's the same as previous "fullnode" version of Armory, but without maintaining it's own copy of the blockchain (of course, if you run supernode it will maintain your 2x-3x sized DB, but that's not the default Armory mode).

Will it ever be possible to run Armory without requiring filesystem-level access to the block files?

Seems like a change such as this makes such an operating mode more difficult.

I run bitcoind and Armory in separate virtual machines, and finding a way to share those files in a way that gets all the permissions right and satisfies LevelDB's locking requirements is non-trivial.

It would be great to have an option for Armory to maintain its own database using blocks it obtains via its peer connection to the node, rather than via direct filesystem access, without requiring a supernode.

We do plan on implementing blocks over p2p

[goatpig]

Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes?

[Carlton Banks]

Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes?

I don't totally know what you mean, but yes in principle. What's gdb?

[goatpig]

Carlton Banks: would you mind building ffreeze in debug mode, running it in gdb and posting the backtrace when it crashes?

I don't totally know what you mean, but yes in principle. What's gdb?

https://www.gnu.org/software/gdb/

Essentially, linux debug mode. First install gdb. On Debian based systems, sudo apt-get install gdb will do it. I'm not familiar with rpm but I expect there are gdb builds available.

With gdb installed, type these commands in the top code folder:

make clean
make DEBUG=1      <- build with debug symbols
gdb python             <- start python in debug mode
run ./ArmoryQt.py   <- The command line argument goes after run
...                         <- code is now running, wait till it segfaults
backtrace               <- after it segfaults, you receive control back, type in backtrace and paste the result back in this thread

[btchris]

Code:

First-chance exception at 0x000007FEDFD63450 (_CppBlockUtils.pyd) in python.exe: 0xC0000005: Access violation reading location 0x0000000088067A8D.

On Win 7 64-bit, I'm also getting an access violation while scanning txs. Running the latest in the ffreeze branch (578c072). Here's a stack trace (via VS 2013) from a debug build:

Code:

 	_CppBlockUtils.pyd!memcpy() Line 356	Unknown
 	_CppBlockUtils.pyd!BinaryData::copyFrom(const unsigned char * inData=0x0000000088067a8d, unsigned __int64 sz=80) Line 198	C++
 	_CppBlockUtils.pyd!BlockHeader::unserialize(const unsigned char * ptr=0x0000000088067a8d, unsigned int size=80) Line 31	C++
 	_CppBlockUtils.pyd!BlockHeader::unserialize(const BinaryDataRef & str={...}) Line 48	C++
 	_CppBlockUtils.pyd!BlockHeader::unserialize(BinaryRefReader & brr={...}) Line 54	C++
 	_CppBlockUtils.pyd!BlockHeader::BlockHeader(BinaryRefReader & brr={...}) Line 52	C++
 	_CppBlockUtils.pyd!PulledBlock::unserializeFullBlock(BinaryRefReader brr={...}, bool doFrag=true, bool withPrefix=false) Line 230	C++
>	_CppBlockUtils.pyd!GrabThreadData::pullBlockAtIter(PulledBlock & pb={...}, LDBIter & iter={...}, LMDBBlockDatabase * db=0x00000000003baa10, BlockFileAccessor & bfa={...}) Line 490	C++
 	_CppBlockUtils.pyd!GrabThreadData::grabBlocksFromDB(std::shared_ptr<LoadedBlockData> blockData={...}, unsigned int threadId=2) Line 279	C++
 	_CppBlockUtils.pyd!std::_Bind<1,void,void (__cdecl*const)(std::shared_ptr<LoadedBlockData>,unsigned int),std::shared_ptr<LoadedBlockData>,unsigned int>::_Do_call<,0,1>(std::tuple<> _Myfargs={...}, std::_Arg_idx<0,1> __formal={...}) Line 1150	C++
 	_CppBlockUtils.pyd!std::_Bind<1,void,void (__cdecl*const)(std::shared_ptr<LoadedBlockData>,unsigned int),std::shared_ptr<LoadedBlockData>,unsigned int>::operator()<>() Line 1138	C++
 	_CppBlockUtils.pyd!std::_LaunchPad<std::_Bind<1,void,void (__cdecl*const)(std::shared_ptr<LoadedBlockData>,unsigned int),std::shared_ptr<LoadedBlockData>,unsigned int> >::_Run(std::_LaunchPad<std::_Bind<1,void,void (__cdecl*const)(std::shared_ptr<LoadedBlockData>,unsigned int),std::shared_ptr<LoadedBlockData>,unsigned int> > * _Ln=0x0000000007d1d060) Line 196	C++
 	_CppBlockUtils.pyd!std::_LaunchPad<std::_Bind<1,void,void (__cdecl*const)(std::shared_ptr<LoadedBlockData>,unsigned int),std::shared_ptr<LoadedBlockData>,unsigned int> >::_Go() Line 188	C++
 	_CppBlockUtils.pyd!_Call_func(void * _Data=0x000000000f8b18b0) Line 28	C++
 	_CppBlockUtils.pyd!_callthreadstartex() Line 376	C
 	_CppBlockUtils.pyd!_threadstartex(void * ptd=0x000000000f8b18b0) Line 354	C
 	kernel32.dll!BaseThreadInitThunk()	Unknown
 	ntdll.dll!RtlUserThreadStart()	Unknown

I think there's something going on near GrabThreadData::pullBlockAtIter(); here's a locals dump after switching to that frame:

Code:

-		bdr	{ptr_=0x0000000088067a8d "" nBytes_=155478 }	BinaryDataRef
+		    ptr_	0x0000000088067a8d ""	const unsigned char *
		    nBytes_	155478	unsigned __int64
-		pb	{stxMap_={ size=0 } nextBlock_=empty fmp_={current_=shared_ptr {lastSeenCumulated_={...} fetch_=FETCH_NONE (0) filemap_=0xec27b9f8692f2700 <Error reading characters of string.> ...} [1 strong ref] [default] ...} }	PulledBlock &
+		    DBBlock	{dataCopy_={data_={ size=0 } } thisHash_={data_={ size=0 } } numTx_=4294967295 ...}	DBBlock
+		    stxMap_	{ size=0 }	std::map<unsigned short,PulledTx,std::less<unsigned short>,std::allocator<std::pair<unsigned short const ,PulledTx> > >
+		    nextBlock_	empty	std::shared_ptr<PulledBlock>
-		    fmp_	{current_=shared_ptr {lastSeenCumulated_={...} fetch_=FETCH_NONE (0) filemap_=0xec27b9f8692f2700 <Error reading characters of string.> ...} [1 strong ref] [default] ...}	FileMapContainer
-		        current_	shared_ptr {lastSeenCumulated_={...} fetch_=FETCH_NONE (0) filemap_=0xec27b9f8692f2700 <Error reading characters of string.> ...} [1 strong ref] [default]	std::shared_ptr<FileMap>
-		            [ptr]	0x000000005ad12f50 {lastSeenCumulated_={...} fetch_=FETCH_NONE (0) filemap_=0xec27b9f8692f2700 <Error reading characters of string.> ...}	FileMap *
-		                lastSeenCumulated_	{...}	std::atomic<unsigned __int64>
		                fetch_	FETCH_NONE (0)	FILEMAP_FETCH
+		                filemap_	0xec27b9f8692f2700 <Error reading characters of string.>	unsigned char *
		                mapsize_	15458524216405398507	unsigned __int64
		                fnum_	82	unsigned short
+		            [deleter and allocator]	default	std::_Ref_count_base
+		            [Raw View]	0x000000004186a2f8 {...}	std::shared_ptr<FileMap> *
-		        prev_	0x00000000154bf8e0 shared_ptr {lastSeenCumulated_={...} fetch_=FETCH_ACCESSED (2) filemap_=0x0000000021b70040 "ù¾´Ù\x1ef\x2" ...} [941 strong refs] [default]	std::shared_ptr<FileMap> *
-		            [ptr]	0x0000000069977950 {lastSeenCumulated_={...} fetch_=FETCH_ACCESSED (2) filemap_=0x0000000021b70040 "ù¾´Ù\x1ef\x2" ...}	FileMap *
+		                lastSeenCumulated_	{...}	std::atomic<unsigned __int64>
		                fetch_	FETCH_ACCESSED (2)	FILEMAP_FETCH
-		                filemap_	0x0000000021b70040 "ù¾´Ù\x1ef\x2"	unsigned char *
		                mapsize_	134020078	unsigned __int64
		                fnum_	81	unsigned short
+		            [deleter and allocator]	default	std::_Ref_count_base
+		            [Raw View]	0x00000000154bf8e0 {...}	std::shared_ptr<FileMap> *
+		iter	{iter_={db_=0x00000000003baaa0 {env=0x000000000038f000 {dbenv=0x000000000581ed80 {...} threadTxMutex_=...} ...} ...} ...}	LDBIter &
+		db	0x00000000003baa10 {baseDir_="C:\\Users\\Chris\\AppData\\Roaming\\Armory\\databases" genesisBlkHash_=...}	LMDBBlockDatabase *
+		bfa	{blkFiles_=shared_ptr { size=270 } [3 strong refs] [default] blkMaps_={ size=2 } lastSeenCumulative_=...}	BlockFileAccessor &
		fnum	82	unsigned short
+		brr	{bdRef_={ptr_=0x0000000065ebbb90 "R" nBytes_=16 } totalSize_=16 pos_=16 }	BinaryRefReader
		offset	555597	unsigned __int64
		size	155478	unsigned int

First off, fnum, offset, and size look good. I verified that they point to the beginning of block 258860, and that the block file doesn't appear corrupted. At the very least, the merkle root is correct.

Here are two things that pop out at me though. pb.fmp_.current_->filemap_ and pb.fmp_.current_->mapsize_ are clearly invalid, perhaps something is overwriting them? IIUC, bdr.ptr_ is calculated from filemap_ and offset, and so bdr.ptr_ is invalid, and it's what eventually gets passed to memcpy() which causes the access violation.

Another thing that looks strange is that I thought (I could be wrong) that bdr.ptr_ == filemap_ + offset, but in the locals above it's not (or multiple things are getting overwritten?).

I'm not quite sure where to go from here.... whatever happened, it may be too late at this point in the debug to find.

[Carlton Banks]

First install gdb. On Debian based systems, sudo apt-get install gdb will do it. I'm not familiar with rpm but I expect there are gdb builds available.

With gdb installed, type these commands in the top code folder:

make clean
make DEBUG=1      <- build with debug symbols
gdb python             <- start python in debug mode
run ./ArmoryQt.py   <- The command line argument goes after run
...                         <- code is now running, wait till it segfaults
backtrace               <- after it segfaults, you receive control back, type in backtrace and paste the result back in this thread

Ok, in the gdb environment and using a build with the debug level you specified, 93.99.1 scans the wallet without a single complaint. That seems pretty weird. Didn't re-checkout, didn't re-clone, didn't change a thing other than follow the above instructions. It looks like the ffreeze branch is up to the same commit as yesterday anyway, so checking out or re-cloning couldn't cause a difference. Don't understand it at all.

Also tried outside of gdb, also the same. Still waiting for the scan to complete in this trial, but it seems just as stable as when using gdb