Armory - Discussion Thread

[Polyatomic]

Adding checklevel=2 to the conf file worked , I'm trying Armory now is all good too.[Windows 7 64 bit]

[justusranvier]

No compile errors, but this happens when I try to start:

Code:

(INFO) armoryengine.py:589 - Executing popen: free -m
(INFO) armoryengine.py:589 - Executing popen: ['cat', '/proc/cpuinfo']
(INFO) armoryengine.py:768 - 
(INFO) armoryengine.py:769 - 
(INFO) armoryengine.py:770 - 
(INFO) armoryengine.py:771 - ************************************************************
(INFO) armoryengine.py:772 - Invoked: /usr/share/armory/ArmoryQt.py --debug
(INFO) armoryengine.py:773 - ************************************************************
(INFO) armoryengine.py:774 - Loading Armory Engine:
(INFO) armoryengine.py:775 -    Armory Version        : 0.89.95
(INFO) armoryengine.py:776 -    PyBtcWallet  Version  : 1.35
(INFO) armoryengine.py:777 - Detected Operating system: Linux
(INFO) armoryengine.py:778 -    OS Variant            : Gentoo Base System-2.2-
(INFO) armoryengine.py:779 -    User home-directory   : /home/justus
(INFO) armoryengine.py:780 -    Satoshi BTC directory : /home/justus/.bitcoin/
(INFO) armoryengine.py:781 -    Armory home dir       : /home/justus/.armory/
(INFO) armoryengine.py:782 - Detected System Specs    : 
(INFO) armoryengine.py:783 -    Total Available RAM   : 15.52 GB
(INFO) armoryengine.py:784 -    CPU ID string         : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
(INFO) armoryengine.py:785 -    Number of CPU cores   : 8 cores
(INFO) armoryengine.py:786 -    System is 64-bit      : True
(INFO) armoryengine.py:787 -    Preferred Encoding    : UTF-8
(INFO) armoryengine.py:788 - 
(INFO) armoryengine.py:789 - Network Name: Main Network
(INFO) armoryengine.py:790 - Satoshi Port: 8333
(INFO) armoryengine.py:791 - Named options/arguments to armoryengine.py:
(INFO) armoryengine.py:793 -     leveldbDir      : DEFAULT
(INFO) armoryengine.py:793 -     skipVerCheck    : False
(INFO) armoryengine.py:793 -     satoshiPort     : DEFAULT
(INFO) armoryengine.py:793 -     settingsPath    : /home/justus/.armory/ArmorySettings.txt
(INFO) armoryengine.py:793 -     logFile         : /home/justus/.armory/ArmoryQt.py.log.txt
(INFO) armoryengine.py:793 -     nettimeout      : 2
(INFO) armoryengine.py:793 -     doDebug         : True
(INFO) armoryengine.py:793 -     datadir         : DEFAULT
(INFO) armoryengine.py:793 -     netlog          : False
(INFO) armoryengine.py:793 -     keypool         : 100
(INFO) armoryengine.py:793 -     testnet         : False
(INFO) armoryengine.py:793 -     rpcport         : DEFAULT
(INFO) armoryengine.py:793 -     satoshiHome     : DEFAULT
(INFO) armoryengine.py:793 -     forceOnline     : False
(INFO) armoryengine.py:793 -     logDisable      : False
(INFO) armoryengine.py:793 -     offline         : False
(INFO) armoryengine.py:793 -     mtdebug         : False
(INFO) armoryengine.py:793 -     interport       : 8223
(INFO) armoryengine.py:794 - Other arguments:
(INFO) armoryengine.py:797 - ************************************************************
(CRITICAL) armoryengine.py:1005 - C++ block utilities not available.
(CRITICAL) armoryengine.py:1006 -    Make sure that you have the SWIG-compiled modules
(CRITICAL) armoryengine.py:1007 -    in the current directory (or added to the PATH)
(CRITICAL) armoryengine.py:1008 -    Specifically, you need:
(CRITICAL) armoryengine.py:1009 -        CppBlockUtils.py     and
(CRITICAL) armoryengine.py:1011 -        _CppBlockUtils.so
(ERROR) Traceback (most recent call last):
  File "/usr/share/armory/ArmoryQt.py", line 30, in <module>
    from armoryengine import *
  File "/usr/share/armory/armoryengine.py", line 1001, in <module>
    import CppBlockUtils as Cpp
  File "/usr/share/armory/CppBlockUtils.py", line 26, in <module>
    _CppBlockUtils = swig_import_helper()
  File "/usr/share/armory/CppBlockUtils.py", line 22, in swig_import_helper
    _mod = imp.load_module('_CppBlockUtils', fp, pathname, description)
ImportError: /usr/share/armory/_CppBlockUtils.so: undefined symbol: _ZN6snappy21GetUncompressedLengthEPKcmPm

Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/share/armory/armoryengine.py", line 582, in logexcept_override
    sys.__excepthook__(type, value, tback) 
AttributeError: 'NoneType' object has no attribute '__excepthook__'

Original exception was:
Traceback (most recent call last):
  File "/usr/share/armory/ArmoryQt.py", line 30, in <module>
    from armoryengine import *
  File "/usr/share/armory/armoryengine.py", line 1001, in <module>
    import CppBlockUtils as Cpp
  File "/usr/share/armory/CppBlockUtils.py", line 26, in <module>
    _CppBlockUtils = swig_import_helper()
  File "/usr/share/armory/CppBlockUtils.py", line 22, in swig_import_helper
    _mod = imp.load_module('_CppBlockUtils', fp, pathname, description)
ImportError: /usr/share/armory/_CppBlockUtils.so: undefined symbol: _ZN6snappy21GetUncompressedLengthEPKcmPm

[etotheipi]

Did you recompile?  Also, did you install the libleveldb package?

[justusranvier]

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

Also, did you install the libleveldb package?

I have leveldb-1.9.0 installed.

[etotheipi]

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

That's not strictly true, here, since i didn't giveyou an installer.  If you have previously compiled the project and then checkout the new branch and/or pull, it will update the code, but you might have to do a "make clean; make" from the base directory to get it to discard the previous binaries and recompile all of them.  You may know this, but I've frequently forgotten to recompile after pulling an update and been confused by the errors that look like what you just showed.

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb.  While trying to get leveldb working I've seen that topic come up a few times even though I didn't have the problem myself on Ubuntu.  But it might be an issue on other system.

[justusranvier]

Did you recompile?

This is Gentoo, so there is no installation method that's not recompiling.

That's not strictly true, here, since i didn't giveyou an installer.  If you have previously compiled the project and then checkout the new branch and/or pull, it will update the code, but you might have to do a "make clean; make" from the base directory to get it to discard the previous binaries and recompile all of them.  You may know this, but I've frequently forgotten to recompile after pulling an update and been confused by the errors that look like what you just showed.

I'm using an ebuild, so what happens with Gentoo is that when you install a package it extracts the source code to a working directory in /var/tmp/portage, which then gets deleted after installation. There are never any intermediate files left lying around - each compilation takes place in a clean environment.

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb.  While trying to get leveldb working I've seen that topic come up a few times even though I didn't have the problem myself on Ubuntu.  But it might be an issue on other system.

I'll check into that.

[laurentb]

However, it's also possible that when Gentoo installs leveldb it does not install the snappy compression library which is needed for leveldb.

The Gentoo Bitcoin ebuilds force you to install leveldb with snappy disabled.
I haven't been able to gather a real reason why.

Is it possible to compile leveldb locally and embed it? Maybe it could be used as a temporary workaround.

[justusranvier]

Leveldb is compiled without snappy support.

Here's where it gets complicated:

Luke-jr maintains the Bitcoin packages on Gentoo. In accordance with their policies, he's not using the bundled leveldb libraries included with the bitcoin-qt source, so instead forces Portage to build leveldb independently, but identically to the bundled version.

This enforces USE="-snappy" for leveldb.

I can tell the Armory ebuild to require leveldb build with USE="+snappy", but that creates a package conflict since Bitcoin-Qt and Armory now have incompatible dependency requirements.

The upshot is that it will be impossible to install Bitcoin-Qt and Armory at the same time on a Gentoo system. That doesn't affect me personally because I use bitcoind anyway - I just uninstalled bitcoin-qt to resolve the block, but that's not a general solution.

[gmaxwell]

Did you recompile?  Also, did you install the libleveldb package?

You are really going to use a system database library for Bitcoin software?  I wish you luck!

Why are you using snappy?  It hurt performance and increased size when it was tested with Bitcoin.

[etotheipi]

Did you recompile?  Also, did you install the libleveldb package?

You are really going to use a system database library for Bitcoin software?  I wish you luck!

Why are you using snappy?  It hurt performance and increased size when it was tested with Bitcoin.

Oh, duh.  I'm not! (or rather, I didn't mean to)  I committed leveldb as part of the project for that reason, and it should be part of the (poorly made) Makefile.  Perhaps I am building leveldb but actually linking to the system libraries by accident.  I'll check.

Also, I thought I had disabled snappy.  But perhaps I didn't.  I assumed the linker error could happen even when you weren't using it.

EDIT: Okay I misread the documentation, I didn't realize it uses snappy by default.  I thought I had to tell it to use it.  I guess I never got around to testing with and without.  I'll disable it and see if I get a performance improvement.

[laurentb]

Why are you using snappy?  It hurt performance and increased size when it was tested with Bitcoin.

Oh, good to know!
Does that mean it has to be disabled at compile time though? Can't bitcoin just ask not to use it at runtime?

[etotheipi]

Why are you using snappy?  It hurt performance and increased size when it was tested with Bitcoin.

Oh, good to know!
Does that mean it has to be disabled at compile time though? Can't bitcoin just ask not to use it at runtime?

The documentation suggests that most applications will benefit from it, and that it's very lightweight and much faster than most file I/O operations so it shouldn't hurt you in terms of performance.  I guess that's why it's enabled by default.  But Bitcoin is special in that the blockchain data is very dense and hardly compressible.  I had suspected it would be of no benefit here and thus no reason to use it (and meant the baseline to disabled it).  I didn't realize it would actually hurt you!  Yeah, good to know...

[etotheipi]

Quick update:

• With snappy, my last rebuild took 4.5 hours after deleting the .armory/databases directory.  There's a lot of room for improvement on that one, but a lot of the options are kind of heavy changes, and it's not critical for stability.  I'll try now it without snappy and see if it improves.
• Displayed balances are really incorrect when used on mainnet.  Perhaps it has to do with length of history.  But luckily, it's an "aesthetic" error -- I just confirmed that the raw DB entries are 100% correct, the interface is just reading the DB incorrectly.
• You can expect a whole bunch of errors and warnings when you restart, because my code accidentally rescans the last few blocks and then triggers those log entries because things that it's expecting to be unspent are already marked spent, etc.

And most importantly:

• Armory uses 288 MB of RAM after the DB build is complete (on Linux).  That's means that Armory online mode should be usable on even a lot of older hardware!
• There's a compile-time configurable parameter to adjust how much RAM is used when it's building the DB.  Right now, it's set so that it won't use more than 1 GB of RAM while building.   I will make that run-time configurable and do some testing on it RAM-vs-performance.

[etotheipi]

Fixed the wallet balance issue.  What a strange bug!   If you saw strange balances, it should be resolved when you pull, recompile and run.  I finally get all the right balances on all of my wallets, and I have some pretty active wallets! (stretching back 2 years and hundreds of tx).

It appears to work, strictly speaking.  A couple things to note:

(1) There's clearly a very large inefficiency in startup.  I already noticed it rewrites the entire headers DB on each load -- which is actually not that slow, but it's also unnecessary.  Similarly, it reapplies a bunch of block on each load even though it isn't supposed to.  Neither of these cause inaccuracies, but they do slow it down dramatically and represent issues to be to be fixed.
(2) It appears that the new backup stuff was merged before the latest version.  I might merge the latest backup stuff ASAP so that people can test it, then I'll work on fixing #1

Getting closer!  At least I think I have all the issues contained.  Except for that battle with leveldb+windows.  Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project.  It is remarkably unpleasant.

[halfawake]

When I loaded armory this AM, it said it detected a corrupt block in the blockchain.

I closed armory, opened bitcoin 0.8.4 and it offered to repair the chain.  I let this run through and it completed giving me the green checkmark in the bottom right corner.  Then as soon as I went back into armory I got the same error, now I go back into the bitcoin client and it is starting the repair again??

I really need to send coins in my armory wallet out today, what can I do to end this loop?

Unfortunately that is actually a bitcoin-qt problem that cropped up today.  From the mailing list:  prepare for some turbulence.

Crap, so reindexing it doesn't fix it?  Guess I've got to change the checklevel so it doesn't keep doing this tomorrow.  Definitely seems like a bug, though obviously not in Armory.

[payb.tc]

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

Can you run a 64 bit VM on 32 bit hardware? That doesn't seem right.

Anyway, the original question was whether setting up a VM would help alleviate the problem of only have 2GB of memory, as the VM could "have" 64GB. The answer is of course no.

From stackoverflow:

You can't run a 64-bit VM session on a 32-bit processor. However, you can run a 64-bit VM session if you have a 64-bit processor but have installed a 32-bit host OS and your processor supports the right extensions.

It's my understanding that every desktop/laptop CPU under the sun is 64-bit by now, even though many people still run 32-bit OS.  So it is most likely possible to do this. 

Yes, slow as dirt.  Painfully slow.  But it might actually, eventually work, instead of seg-faulting when it hits 94%.

agreed, this is my assumption

i'll post here once i try it out on an unused laptop with lots of hdd space and barely any ram

and i'll pull out a calendar to time how long it takes to launch

[goatpig]

Getting closer!  At least I think I have all the issues contained.  Except for that battle with leveldb+windows.  Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project.  It is remarkably unpleasant.

Which compiler are you using?

[etotheipi]

Getting closer!  At least I think I have all the issues contained.  Except for that battle with leveldb+windows.  Btw, I'm still offering 2 BTC to anyone who helps me get leveldb built in Windows and integrated into my project.  It is remarkably unpleasant.

Which compiler are you using?

https://bitcointalk.org/index.php?topic=56424.msg3095869#msg3095869

[goatpig]

leveldbwin is a google project for msvc 9 and 10 that allows you to complile leveldb as a dll on windows

https://code.google.com/p/leveldbwin/downloads/detail?name=leveldb_1_20_win32_src.zip&can=2&q=

To compile, it requires ATL librairies, which are available in WDK 7.1 or pay versions of visual studio 9 and above.

Once you get the DLL to compile the integration is straight forward.

I personally use vc11 express so I have to get the WDK before I can compile the project and test it. I have however successfully compiled this project a few months ago with vs9.

*edit*

After installing WDK, the project successfully compiled, yielding a static library for snappy and both static and dynamic libraries compile options for leveldb.

[etotheipi]

leveldbwin is a google project for msvc 9 and 10 that allows you to complile leveldb as a dll on windows

https://code.google.com/p/leveldbwin/downloads/detail?name=leveldb_1_20_win32_src.zip&can=2&q=

To compile, it requires ATL librairies, which are available in WDK 7.1 or pay versions of visual studio 9 and above.

Once you get the DLL to compile the integration is straight forward.

I personally use vc11 express so I have to get the WDK before I can compile the project and test it. I have however successfully compiled this project a few months ago with vs9.

*edit*

After installing WDK, the project successfully compiled, yielding a static library for snappy and both static and dynamic libraries compile options for leveldb.

Awesome!  Finally some encouraging news on this front.  I don't know how I missed the leveldbwin project...