Armory - Discussion Thread

[etotheipi]

BACKUP CENTER MERGED:

I have merged the new backup center into the ramreduceleveldb branch.  If you went to the Bitcoin conference and you saw the fragmented-backup and SecurePrint demos, then you already know what's there.  Please test it!   (p.s. new backups now only have two lines of data to copy/print!).    There's still a few quirks that need to be fixed, but it's just about fully functional.

At the moment, it looks like Armory is "usable" in Linux.   The biggest problems remaining are:

• (1) Lack of interface feedback on initial blockchain build.  Just wait 4-5 hours, don't worry if it looks like it's not doing anything.
• (2) Slowness and a screen full of harmless errors and warnings on load -- if you shut down Armory and start it later, it can take a couple minutes to sync with the latest blocks.  Will have to investigate, since it clearly builds the database much faster than that on an initial sync.
• (3) Restored wallets appear to have incorrect balances until you restart.  Luckily, restarting only takes about about 5 sec now

I will track down some of the slowness issues later, since it technically works right now.  At the moment, have a long list of tweaks that have been waiting a few months to go in, and a few quirks to fix in the backup center.  Please keep testing!  (if you can)

[Ente]

Anyone in Linux want to help?  It's the ramreduceleveldb branch.

WTF.
I'm late to the party, and now I'm spilling drinks over everybody too:
I tried to switch to the ramreduceleveldb branch, but am too stupid to do so.

git checkout ramreduceleveldb
git pull origin ramreduceleveldb

This seems to create a local branch, with the data of some other branch, probably "master" or "testing".

git branch -a
  master
* ramreduceleveldb
  remotes/origin/HEAD -> origin/master
  remotes/origin/armoryd
  remotes/origin/backupcenter
  remotes/origin/bitcoind0.8andP2Pool
  remotes/origin/bitsafetest
  remotes/origin/blkchainagain_optimized
  remotes/origin/cmake
  remotes/origin/cryptoppstatic
  remotes/origin/dev
  remotes/origin/ecdsacalc
  remotes/origin/leveldb
  remotes/origin/managesatoshi
  remotes/origin/master
  remotes/origin/migratewallet
  remotes/origin/newwallet
  remotes/origin/osx_managesat
  remotes/origin/pointcompress
  remotes/origin/python_test
  remotes/origin/qtdev
  remotes/origin/ramreduceleveldb
  remotes/origin/rrldbtest
  remotes/origin/sddblspend
  remotes/origin/testing
  remotes/origin/threading

Debian Wheezy here. Probably of no big help anyway, now?

Ente

[maaku]

Code:

# undo the mess you got yourself into
git checkout master
git branch -d ramreduceleveldb

Code:

# checkout the branch
git checkout -b ramreduceleveldb origin/ramreduceleveldb

[Ente]

Code:

# undo the mess you got yourself into
git checkout master
git branch -d ramreduceleveldb

Code:

# checkout the branch
git checkout -b ramreduceleveldb origin/ramreduceleveldb

Thank you a ton!

Ente

[Sc@rF@c3]

RE:Win7 64bit, i7 gen1, 16GB Ram, online watch only
Upgrading to Bitcoin-Qt 0.8.4 gives the following crash error


There was an error starting the underlying Bitcoin engine. This should not normally happen. Usually it occurs when you have been using Bitcoin-Qt prior to using Armory, especially if you have upgraded or downgraded Bitcoin-Qt recently (manually, or through the Armory automatic installation). Output from bitcoind:
StdErr:

I have downgraded to Bitcoin-Qt 0.8.3, working fine again

[kostagr33k]

Hi, hope all is well.

Trying to access my online portion of my offline wallet, and its taking quite some time to load. I am on OSx and have 8gb of ram, but armory is now stuck at using 2.47gb of Real Memory and 4.96 Virtual Mem and using 100% CPU of 1 core. IT is stuck at 15 seconds.

Is there anything that can be done here to make this usable ?


Thank you

kosta

[picobit]

Hi, hope all is well.

Trying to access my online portion of my offline wallet, and its taking quite some time to load. I am on OSx and have 8gb of ram, but armory is now stuck at using 2.47gb of Real Memory and 4.96 Virtual Mem and using 100% CPU of 1 core. IT is stuck at 15 seconds.

Is there anything that can be done here to make this usable ?


Thank you

kosta

I have a similar setup (OSX, 8 GB ram) and have no problems.  It takes forever to load (5 min or so), and once loaded it occationally hangs for 10-20 sec at a time.

If you start it in the terminal, you can see the output while it reads the block files.  That shows progress :-)

[Ente]

Late to the ramreduceleveldb party..

Debian Wheezy. I installed into a fresh folder.
I didn't need any extra packages for this. I installed

git-core build-essential pyqt4-dev-tools swig libqtcore4 libqt4-dev python-qt4 python-dev python-twisted python-psutil

way back, it probably isn't all needed any more.

git clone git://github.com/etotheipi/BitcoinArmory.git

git checkout testing
git pull origin testing
make clean
make

git checkout -b ramreduceleveldb origin/ramreduceleveldb
git pull
make clean
make

"testing" gives me version 0.88.2.
"ramreduceleveldb" gives me version 0.98.97.

I still can not compile without editing:

Change the "python2" in the Makefile to just say "python".

"testing" works as ecpected.
"ramreduceleveldb" shows the gui for a second, then crashes:

-ERROR - 1378977312: (leveldb_wrapper.cpp:1974) No tx in DB with hash: 5e5383b79ec2fb038ea556474cde31d3e621a7e52df2d5ee8470c71f9950cb1d
Segmentation fault

On another note, "testing" and bitcoind / bitcoin-qt work, although I didn't do the "-checklevel=2" switch against the reindexing blockchain bug.


If any of these is of help, I'll gladly dig up more logs or the like. I'll downgrade to "testing" again for now.

Ente

[avialias]

HELP NEEDED !!! FAST FAST...  

(edit) : FIXED - I reinstall Armory again and now its fine... (admin delete this post if needed)

I Cant start Armory because of some Runtime Error !
Actualy it starts, syncs with network, downloads new blocks, and then starts to scan for transaction history.
When it reach 71% scan progress or 1.98Gb of RAM then I get C++ Runtime Error.

I installed Bitcoin-Qt 0.8.4 and sync it with blockchain database.
All installatio was default. default path for BTC-Qt and for Blockchain database.

I downloaded Armory 0.88.1-beta x64https://code.google.com/p/bitcoinarmory/downloads/detail?name=armory_0.88.1-beta_win64.msi&can=2&q=.
But when I start installation, setup program wants to install it to "c:\Program Files (x86)\Armory", why there?
Isnt that x64 version, so it mast go to Program files not in program files (x86)

Where is x64 Armory ?

How to fix problem?

I have Win7 x64
AMD phenom II x64 unlocked 4cores
8Gb of DDR3 RAM.
SSD disk
pagefile set to 1Gb
hiberfile disabled

[etotheipi]

UPDATE:  Plan B

Unfortunately, there was just too many problems with super-node Armory.  LevelDB has been extremely inconsistent in keeping up with the stress of constantly updating address ledgers.  On slower systems, I observed random "missing tx" messages and seg faults on one run, then I would re-run on the same data and it would cruise through that part.    I cannot attest that my code was bug-less (though, there's like 2,000 lines of unit-testing code to test the new database stuff, including re-orgs).  But it appears that I cannot worry about code bugs if it's not going to work even without those bugs.

Super-node Armory is likely to be put on hold for a while.  Now, I know that sounds like it's going to be a serious disruption to my progress...


The good news:

All of the modularization I have put into Armory just paid off.  Tremendously!  In 8 hours I was able to swap out the super-node DB operations with something that resembles the previous version, but without super-node capability.  Super-node code would process all the headers, find the longest chain, write out all the raw blocks to disk, apply all the blocks (updated addr histories, mark TxOuts spent/unspent, etc), and then grab all the transactions relevant your wallets and scan them.

Now, it skips the "apply blocks" section entirely, and does a blockchain scan on the raw blocks in the DB to find all the transactions relevant to your wallets.  It injects them into the wallet and everything operates the same!  In those 8 hours, I now have a mostly-working version of Armory that rescans the blockchain on every load, but has the database backing so that it uses dramatically less RAM.  Also, it will be trivial to save the registered tx data between loads, now, so rescanning will only be necessary on the first load or import/restore wallets.  When I originally tried to do this, there was too many subtle issues with where the transactions were on disk, swapping blkfiles/homedirs, etc.  With Armory managing its own database, it can do this reliably now!

I am in the process of reverting many of the skip-this-operation-because-not-needed-for-super-node operations to the old way, so that it still pops up with all the original warnings about needing rescans etc. This should be done today, meaning it only cost me two days to do a major operational shift!  

If you want to play on the new branch, it has been renamed rrld_planB.  It is a fork of ramreduceleveldb, which I wanted to keep around for when it's time to revisit super-node Armory.   Still some UI feedback issues, and doesn't handle being interrupted on the initial build, but it is usable on Linux for simple use (and testing!).  

Stay tuned!  

[halfawake]

UPDATE:  Plan B

Unfortunately, there was just too many problems with super-node Armory.  LevelDB has been extremely inconsistent in keeping up with the stress of constantly updating address ledgers.  On slower systems, I observed random "missing tx" messages and seg faults on one run, then I would re-run on the same data and it would cruise through that part.    I cannot attest that my code was bug-less (though, there's like 2,000 lines of unit-testing code to test the new database stuff, including re-orgs).  But it appears that I cannot worry about code bugs if it's not going to work even without those bugs.

Super-node Armory is likely to be put on hold for a while.  Now, I know that sounds like it's going to be a serious disruption to my progress...


The good news:

All of the modularization I have put into Armory just paid off.  Tremendously!  In 8 hours I was able to swap out the super-node DB operations with something that resembles the previous version, but without super-node capability.  Super-node code would process all the headers, find the longest chain, write out all the raw blocks to disk, apply all the blocks (updated addr histories, mark TxOuts spent/unspent, etc), and then grab all the transactions relevant your wallets and scan them.

Now, it skips the "apply blocks" section entirely, and does a blockchain scan on the raw blocks in the DB to find all the transactions relevant to your wallets.  It injects them into the wallet and everything operates the same!  In those 8 hours, I now have a mostly-working version of Armory that rescans the blockchain on every load, but has the database backing so that it uses dramatically less RAM.  Also, it will be trivial to save the registered tx data between loads, now, so rescanning will only be necessary on the first load or import/restore wallets.  When I originally tried to do this, there was too many subtle issues with where the transactions were on disk, swapping blkfiles/homedirs, etc.  With Armory managing its own database, it can do this reliably now!

I am in the process of reverting many of the skip-this-operation-because-not-needed-for-super-node operations to the old way, so that it still pops up with all the original warnings about needing rescans etc. This should be done today, meaning it only cost me two days to do a major operational shift!  

If you want to play on the new branch, it has been renamed rrld_planB.  It is a fork of ramreduceleveldb, which I wanted to keep around for when it's time to revisit super-node Armory.   Still some UI feedback issues, and doesn't handle being interrupted on the initial build, but it is usable on Linux for simple use (and testing!).  

Stay tuned!  

Wow, this sounds like great progress.  I totally feel like my Armory donation was money well spent.

[kostagr33k]

Thanks! I ran from CLI, and when it gets to the "15 seconds" where its stuck, it complains about orphan blocks. Quick google showed this bug:

https://github.com/etotheipi/BitcoinArmory/issues/100


I guess I will have to download the chain again. Thanks for the pointer.

Hi, hope all is well.

Trying to access my online portion of my offline wallet, and its taking quite some time to load. I am on OSx and have 8gb of ram, but armory is now stuck at using 2.47gb of Real Memory and 4.96 Virtual Mem and using 100% CPU of 1 core. IT is stuck at 15 seconds.

Is there anything that can be done here to make this usable ?


Thank you

kosta

I have a similar setup (OSX, 8 GB ram) and have no problems.  It takes forever to load (5 min or so), and once loaded it occationally hangs for 10-20 sec at a time.

If you start it in the terminal, you can see the output while it reads the block files.  That shows progress :-)

[Kluge]

  I did a sendmany transaction recently and added individual comments. Before, I never added comments in sendmany txs, because I was pretty sure it wouldn't be displayed usefully, but I was proven wrong. Armory separates comments in the transaction and even adds in a rounded amount next to each comment based on how much was sent to the address the comment's attached to. Surprises me every day. 

[etotheipi]

I did a sendmany transaction recently and added individual comments. Before, I never added comments in sendmany txs, because I was pretty sure it wouldn't be displayed usefully, but I was proven wrong. Armory separates comments in the transaction and even adds in a rounded amount next to each comment based on how much was sent to the address the comment's attached to. Surprises me every day. 

No detail is too small for me   Glad to see people appreciating the time I put into those details!


Still a 2 BTC bounty available! (I'm looking in goatpig's direction).

I've determined that I don't care what version of MSVS is used for compiling the C++ side of Armory.  I've been meaning to upgrade to MSVS 2010 or 2012 anyway... if someone gives me a solution using one of those, you get the 2 BTC.  The "checkout" criteria for the bounty is that the project must successfully compile leveldb (or I must be able to do it myself), and then it must link properly to the code in the ramreduceleveldb branch.  And of course, some documentation on how it was done...

The reason I haven't upgraded to MSVS 2010 or 2012 is because there's a bit of pain doing so (like MSVS 2010 now supports the equiv of <cstdint> in Linux, so it complains about my redefinitions).   Whatever, if someone can take a shot at it, or at least give me a partial solution, you get part of the bounty!

[goatpig]

Still a 2 BTC bounty available! (I'm looking in goatpig's direction).

I've determined that I don't care what version of MSVS is used for compiling the C++ side of Armory.  I've been meaning to upgrade to MSVS 2010 or 2012 anyway... if someone gives me a solution using one of those, you get the 2 BTC.  The "checkout" criteria for the bounty is that the project must successfully compile leveldb (or I must be able to do it myself), and then it must link properly to the code in the ramreduceleveldb branch.  And of course, some documentation on how it was done...

The reason I haven't upgraded to MSVS 2010 or 2012 is because there's a bit of pain doing so (like MSVS 2010 now supports the equiv of <cstdint> in Linux, so it complains about my redefinitions).   Whatever, if someone can take a shot at it, or at least give me a partial solution, you get part of the bounty!

I'll give it a look but I won't be around from Friday to Sunday night, so don't expect speedy delivery =P

I'll attempt to compile the whole project on vc11 express for the following reasons: express is free to download, and free to use to deploy open source projects, makes sense so far, and it's MS dedicated compiler. On top of that, vc11 adds some of the C11 improvements, including atomics, and leveldb relies on std::atomics to compile. This is partly why it's such a pain to compile it with vc10 and vc9, as you have to use WinAPI's Interlocked functions to support atomicity. That and setting up the environment >_>

In case you're having trouble with warnings from a version of msvc to another, consider defining  _CRT_SECURE_NO_WARNINGS

[justusranvier]

(2) How long does it take to build the DB?  NOTE: there's actually two steps to building the DB: one is writing the raw blocks to the DB, the other is actually walking through those blocks and updating the DB with spentness info, and building histories for each address.  The system currently does not give you any real indication that the first step is happening (though it should on the command line, if you run with --debug).  Until I fix that, it may not be obvious that anything is happening... and that first step usually takes like 45 min!

Two hours and 41 minutes:

Code:

2013-09-12 18:37 (INFO) -- armoryengine.py:771 - ************************************************************
2013-09-12 18:37 (INFO) -- armoryengine.py:772 - Invoked: /usr/share/armory/ArmoryQt.py --debug
2013-09-12 18:37 (INFO) -- armoryengine.py:773 - ************************************************************
2013-09-12 18:37 (INFO) -- armoryengine.py:774 - Loading Armory Engine:
2013-09-12 18:37 (INFO) -- armoryengine.py:775 -    Armory Version        : 0.89.97
2013-09-12 18:37 (INFO) -- armoryengine.py:776 -    PyBtcWallet  Version  : 1.35
2013-09-12 18:37 (INFO) -- armoryengine.py:777 - Detected Operating system: Linux
2013-09-12 18:37 (INFO) -- armoryengine.py:778 -    OS Variant            : Gentoo Base System-2.2-
2013-09-12 18:37 (INFO) -- armoryengine.py:779 -    User home-directory   : /home/justus
2013-09-12 18:37 (INFO) -- armoryengine.py:780 -    Satoshi BTC directory : /home/justus/.bitcoin/
2013-09-12 18:37 (INFO) -- armoryengine.py:781 -    Armory home dir       : /home/justus/.armory/
2013-09-12 18:37 (INFO) -- armoryengine.py:782 - Detected System Specs    : 
2013-09-12 18:37 (INFO) -- armoryengine.py:783 -    Total Available RAM   : 15.52 GB
2013-09-12 18:37 (INFO) -- armoryengine.py:784 -    CPU ID string         : Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
2013-09-12 18:37 (INFO) -- armoryengine.py:785 -    Number of CPU cores   : 8 cores
2013-09-12 18:37 (INFO) -- armoryengine.py:786 -    System is 64-bit      : True
2013-09-12 18:37 (INFO) -- armoryengine.py:787 -    Preferred Encoding    : UTF-8
(snip)
2013-09-12 18:37 (INFO) -- ArmoryQt.py:4272 - Dashboard switched to "Scanning" mode
(snip)
2013-09-12 21:18 (INFO) -- ArmoryQt.py:4262 - Dashboard switched to fully-online mode

(3) How much space does the new .armory/databases directory use?  I've seen widely different results, and I think it has to do with seemingly-random times that leveldb decides to do DB compactions.  My first test left me with an 18 GB databases dir.  A more recent had it at 25 GB.  I'm wondering how widely this varies.

Code:

$ du -sh databases/
14G     databases/

[etotheipi]

Whoa, how'd you do that?  Isn't that the ramreduceleveldb branch?  2.7 hours is almost twice as fast as my system did it!  Do you have an SSD?  I ask especially because 14 GB seems too low.  Did you scan the whole blockchain up to like 257k blocks?

[justusranvier]

Isn't that the ramreduceleveldb branch?

rrld_planB branch

Do you have an SSD?

Blockchain is stored on two spinning rust disks in RAID 0. /home is an NFS volume accessed via gigabit ethernet. File server is linux MDRAID 6 running on an Ivy Bridge platform.

I ask especially because 14 GB seems too low.  Did you scan the whole blockchain up to like 257k blocks?

It scanned the entire (mainnet) blockchain. I did explicitly disable snappy support in the bundled leveldb, so that might affected something.

[goatpig]

I have successfully compiled the ramreduceleveldb branch on vc11 x86 (msvs2012 express). The method outlined can be reproduced on vc9/10 (msvs 2008/2010) with no extra changes to either projects (leveldbwin and armoryengine_msvs2005). Keep in mind that to since the compile relies on a library compilation of leveldbwin, you need to compile armory with the same compiler as you used for leveldb. (Don't go around building leveldb with vc9 and then linking it to an armory project built with vc11)

Note: this is a compile of armoryengine_msvs2005 project only, not the SWIG to python part, which is trivial in this case from what I understand of the project (i don't know squat about python and swig). The compile is untested, I simply went as far as yielding a ArmoryEngine_MSVS2005.dll (no main declared in this branch)

Step #1: leveldb

Grab leveldbwin from the google project page: https://code.google.com/p/leveldbwin/

1) Pick the project of your liking in the build folder (msvc9 or msvc10). I used msvc10 and converted the project to msvc11 since this is my compiler.

1.a) To compile you need ATL. If you're using an express version of msvc, you'll have to install WinDDK 7.1 (Driver Development Kit): http://www.microsoft.com/en-us/download/details.aspx?id=11800. Make sure you grab 7.1 and not 8.0 or WinSDK as those do not support ATL anymore. Skip this step if you have a pay version of msvs.

2) In the configuration manager, pick Release (for the static lib compilation). Don't bother compiling test, you don't need it. I used snappy for compilation, so no defines to drop at this point.

3) Right click each project (leveldb and snappy), go to "Config properties -> C/C++ -> Code generation". Make sure Runtime Library is set as Multi Threader (/MT). This defines whether the C librairies used to compile are statically or dynamically linked. My understanding of this project is that you'd rather not distribute msvc runtime binaries so stick to static compile.

4) Rebuild the project, pick up leveldb.lib in ./_exports/Release and shove it where you want to have it for the armory compile (either make a new folder for it or directly link to it from the project)

Step #2: Armory

Here is the list of all the modifications I did to the code and the project to get it to compile:

changes to armory bitcoin for msvc compilation:

# added compiler directives in binarydata.h to not manually define int types on vc10+

# added compiler directives in log.h to not overload logstream and children class >> operator with
both size_t and unsigned int arguments in msvc. They're considered the same and the compiler will
complain about method redefinition.

# added leveldb include folder to msvc project

# excluded FileDataPtr and BlockUtilsTest from msvc project (not in the branch but in the msvc project)

# added leveldb_wrapper.cpp to project

# added StoreBlockObj.cpp to project

# NowTimeInt wasn't defined for win32 (log.h). Defined it by copying the non win32 code and including
<time.h>. This version of time returns the amount of seconds since the unix epoch on my machine
(win7 x64 sp1). It's safe to assume all version of windows will behave in such fashion as this library is
wrapped around a WinAPI call on Windows.

# added new folder to hold static leveldb.lib

# added leveldb.lib to addional dependencies

# changed project from .exe to .dll as there is no main defined (probably cause there is no test code)

# got rid of build events for armoryengine_msvs2005.sln

# make sure you're using Multi Threaded (/MT) for the runtime library.

# rebuild only the armoryengine_msvs2005 project so that it doesn't trigger errors with the custom build events failing to grab guardian or py2exe.

voila

Note: I first tried to dynamically link leveldb to armory, but it failed to export leveldb::DestroyDB properly. The issue disappeared with the static link so I stuck to that for now. On top of that leveldbwin has no x64 build setup, so I'll consider making my own compile of leveldb to allow for dynamic linking and x64 releases.

There are about 100 warnings, 90% of them are related to possible data loss due to assignment without type casting. The rest seems to be msvc complaining about unsafe routines. Both cases are irrelevant.

[etotheipi]

Big news! 

https://bitcointalk.org/index.php?topic=56424.msg671650#msg671650

P.S. - This is still the Armory discussion thread, I just changed the topic temporarily to draw attention to the big news!

@goatpig -- thanks!  you may yet get the bounty.  But I'll be back to you later after the media storm subsides