[Round #5] Final Testing Release Before Armory v0.88

[etotheipi]

This is one of the two major upgrades I need in order for Armory to be usable to users with shorter attention spans.  When this is working properly, you will never need to run Bitcoin-Qt/bitcoind again.  Just start Armory and it will do the rest.  

The other major upgrade is the resource usage and load time.  Those are both getting knocked out with the blockchain management/DB stuff.  That is top priority after this (including not rescanning every time).

Updates in 0.87.99-testing (soon to be 0.88)

• Auto management of bitcoind: This most definitely will break on OSX... gonna deal with that this week.  What needs to be done should be fairly well-explained in the interface.  I will be adding a troubleshooting section to the website to catch the most common issues.
• OSX Package:  Finally a real OSX package.  So far, it doesn't appear to work on every system, but works very well when it does!  Auto-bitcoind is disabled, since bitcoind is not distributed for OSX.  I will be implementing a workaround for that in the near future!
• Notifications of new Bitcoin-Qt/bitcoind versions:  It should work with both self-management and auto-management of bitcoind.
• Windows code-signing certificates:  Just like my GPG keys, I have an offline code-signing certificate just for Windows installers/MSIs

• On-screen Keyboard for passphrase entry: You can now fool keyloggers by using an on-screen keyboard.  If you're ultra-paranoid (and ultra-patient) you can use a keyboard that is scrambled in a cryptographically secure manner.   Comes in "regular" and "insane" flavors.
• Clickable (?) objects: Yes, they finally do something when you click them (used to be just 1.5s mouseover)
• Reduced the number of windows you have to click through to execute an offline transaction.
• Created new Ubuntu-10.04-32bit offline bundle:  this includes the frag/unfrag scripts for Shamir's Secret Sharing
• Unicode Issues:  Fixed all show-stopper unicode issues (I think), and detect and warn about non-critical unicode issues.  

To use 0.87.99 (which will be 0.88 when it's released):

Windows 0.87.99 -- SIGNED
Windows users must first uninstall any previous version, before installing this one.
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99-testing_win64.msi
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99-testing_windows_all.msi

Ubuntu/Debian 0.87.99 -- SIGNED (alternatively, use the "testing" branch)
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99_amd64.deb
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99_i386.deb

Ubuntu Offline Bundles (now in 64-bit!) -- With Detached Signature
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/Armory_Offline_bundle_0.87.97-beta_amd64.tar.gz
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/Armory_Offline_Bundle_0.87.97-beta_i386.tar.gz

The first Mac/OSX Release!  (works offline, too!)
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99a_OSX.dmg
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99a_OSX.dmg.sig

I have GPG-signed the SHA256 hashes (use the .sig above, for the .dmg for OSX ... I have updated it since the hashes were signed)
-- https://dl.dropboxusercontent.com/u/1139081/ArmoryTestingReleases/armory_0.87.99_sha256sum.txt.asc


What I've done:

If you are running Bitcoin-Qt when you start Armory, it will now tell you to stop.  "Let Armory manage the Bitcoin software" is the default.  You will be told to either close Bitcoin-Qt, or change your settings (with a convenient button in the middle of the dashboard).   You can go into your settings and uncheck the box to carry on using Armory exactly as you used to.  Or you can leave it checked and just close Bitcoin-Qt/bitcoind.

If you do not have bitcoind installed or it is in a non-standard location, it will tell you to install it, or change your settings to point to where it is located.  If it's not installed, it will give you a button you can press that will install it for you.  I put a lot of time into this, and a lot of testing.  It works as expected in both Ubuntu and Windows 64-bit.  I need others to try it to tell me for sure.  There's also a button for killing bitcoin-qt/bitcoind if you happen to have it open when Armory expects to run bitcoind, itself.

By the way, I know the initial-sync time-remaining counter is out of whack.  Predicting the correct download time is surprisingly hard!  I will come up with a better way, that doesn't choke when bitcoind pauses/hiccups during download.  I made "8+ hours" the maximum, to avoid it saying "187 years" when this happens...

Please mention everything that doesn't look right.  Most likely, configurations/combinations of options that lead to a dashboard that doesn't say the right thing, or buttons missing that should be there, etc.  Try out the new settings (renamed from "preferences").   One thing I threw on top was added a permanent "--skip-online-check" to the settings.  Try that out, too.


To help test...
To help test, these are operations that should be supported:

• Fresh boot of online operating, never installed or run bitcoind/bitcoin-qt.  Install and run Armory.  Have it install bitcoind/bitcoin-qt for you.
• Bitcoin-Qt already running default settings.  Armory will complain and kill it for you if you want
• Bitcoin-Qt can't be found default settings.  Point to the right directory, or install it for you.
• Change settings to manage bitcoind yourself.  Bitcoind already running, everything is like before (but with a completion bar)
• Change settings to manage bitcoind yourself, but it's not running.  Just like before.
• There is no upgrade button yet, but there will be one in the some-what near future, now that I have all these signed installer links.  I might as well autodetect versions and add an "upgrade" button for both Bitcoin-Qt and/or Armory, as necessary.

[oakpacific]

Thanks, do I need to download the blockchain again to test?

[etotheipi]

Thanks, do I need to download the blockchain again to test?

Not necessary.  Though, if your Bitcoin-Qt home directory (-datadir) is non-standard, it might create one in the standard place and start redownloading.  You can change the settings and restart.  It will pick up where Bitcoin-Qt last left off.

[justusranvier]

I need help testing, especially from the Linux folks, because they're the ones most likely to have non-standard configurations that will break this.

The packaging for Gentoo puts bitcoind in /var/bitcoin and runs it at boot under the bitcoin user. If I fix the permissions such that regular users have read-only access to the block database will Armory be able to work?

[etotheipi]

I need help testing, especially from the Linux folks, because they're the ones most likely to have non-standard configurations that will break this.

The packaging for Gentoo puts bitcoind in /var/bitcoin and runs it at boot under the bitcoin user. If I fix the permissions such that regular users have read-only access to the block database will Armory be able to work?

I was thinking of further branching the code based on distribution for stuff like this, but I'm not sure it's worth my time.  If you're in Gentoo, you are probably used to tweaking settings  OTOH, I guess I can just add /var/bitcoin to the search paths.  It's not like it will really slow anything down for anyone, searching one more directory, once each load...

For the permissions question, I assume you're talking about the .bitcoin/blocks/blk*.dat files?  If so then, yes, it will work.  Armory [theoretically] only needs read-only access, though I never tested it.  If I did it right, it should work fine.

[justusranvier]

I guess I can just add /var/bitcoin to the search paths

Actually I misspoke earlier. It's installed at /var/lib/bitcoin. That's a FHS-compliant location so if any other distributions ever package bitcoind like a native daemon that's probably where they'd put it as well.

It is a bit ugly in that everything gets put under /var/lib/bitcoin/.bitcoin because of embedded assumptions in bitcoind that it's running as a regular user.

[oakpacific]

Thanks, do I need to download the blockchain again to test?

Not necessary.  Though, if your Bitcoin-Qt home directory (-datadir) is non-standard, it might create one in the standard place and start redownloading.  You can change the settings and restart.  It will pick up where Bitcoin-Qt last left off.

Yeah, I put the data directory in a non-standard position, but I always start Armory using the satoshi-datadir option, I guess I would be OK then?

Also do I need to delete the old installation? And if I am super paranoid and want to backup the wallet files do I just copy them somewhere else? Thanks.

[etotheipi]

Thanks, do I need to download the blockchain again to test?

Not necessary.  Though, if your Bitcoin-Qt home directory (-datadir) is non-standard, it might create one in the standard place and start redownloading.  You can change the settings and restart.  It will pick up where Bitcoin-Qt last left off.

Yeah, I put the data directory in a non-standard position, but I always start Armory using the satoshi-datadir option, I guess I would be OK then?

Also do I need to delete the old installation? And if I am super paranoid and want to backup the wallet files do I just copy them somewhere else? Thanks.

You will end up with a fresh new download starting at zero percent in the default location.  You can either start it with the --satoshi-datadir option, or you can start it regularly, and change the options to point at the non-standard location, then restart.  If you do it the second way, it will create the default --satoshi-datadir and start downloading.  I guess you could combine the two:  start with --satoshi-datadir, then change the settings to make it permanent.

I'm not sure a better way to do this.  I need to make the process mostly-transparent to new users:  I don't want to walk them through a confusing process when they don't even know what my program is or what Bitcoin is.  I'd rather it "just work".  And for users that have been using Bitcoin-Qt in the default locations:  well it should "just work" by picking up where they last left off with Bitcoin-Qt.  (this means that for switching from Bitcoin-Qt to Armory, it should be actually quite painless).

[oakpacific]

Thanks, do I need to download the blockchain again to test?

Not necessary.  Though, if your Bitcoin-Qt home directory (-datadir) is non-standard, it might create one in the standard place and start redownloading.  You can change the settings and restart.  It will pick up where Bitcoin-Qt last left off.

Yeah, I put the data directory in a non-standard position, but I always start Armory using the satoshi-datadir option, I guess I would be OK then?

Also do I need to delete the old installation? And if I am super paranoid and want to backup the wallet files do I just copy them somewhere else? Thanks.

You will end up with a fresh new download starting at zero percent in the default location.  You can either start it with the --satoshi-datadir option, or you can start it regularly, and change the options to point at the non-standard location, then restart.  If you do it the second way, it will create the default --satoshi-datadir and start downloading.  I guess you could combine the two:  start with --satoshi-datadir, then change the settings to make it permanent.

I'm not sure a better way to do this.  I need to make the process mostly-transparent to new users:  I don't want to walk them through a confusing process when they don't even know what my program is or what Bitcoin is.  I'd rather it "just work".  And for users that have been using Bitcoin-Qt in the default locations:  well it should "just work" by picking up where they last left off with Bitcoin-Qt.  (this means that for switching from Bitcoin-Qt to Armory, it should be actually quite painless).

Ah, thanks, anyway I would just go testing it and report the result.

[svirus]

Before upgrade everything works.
I use Gentoo.

When bitcoin-qt is not running i got:

Code:

$ python2.7 BitcoinArmory/ArmoryQt.py  --skip-version-check --satoshi-datadir=/media/var/.bitcoin/
(ERROR) ArmoryQt.py:1170 - Failed to setup SDM
Setting netmode: 0
(ERROR) Traceback (most recent call last):
  File "BitcoinArmory/ArmoryQt.py", line 4470, in <module>
    form = ArmoryMainWindow()
  File "BitcoinArmory/ArmoryQt.py", line 466, in __init__
    self.setDashboardDetails()
  File "BitcoinArmory/ArmoryQt.py", line 3853, in setDashboardDetails
    raise ShouldNotGetHereError, 'Bitcoind Never started...?'
ShouldNotGetHereError: Bitcoind Never started...?

Traceback (most recent call last):
  File "BitcoinArmory/ArmoryQt.py", line 4470, in <module>
    form = ArmoryMainWindow()
  File "BitcoinArmory/ArmoryQt.py", line 466, in __init__
    self.setDashboardDetails()
  File "BitcoinArmory/ArmoryQt.py", line 3853, in setDashboardDetails
    raise ShouldNotGetHereError, 'Bitcoind Never started...?'
armoryengine.ShouldNotGetHereError: Bitcoind Never started...?

When bitcoin-qt is running:

Code:

 $ python2.7 BitcoinArmory/ArmoryQt.py --skip-online-check --skip-version-check --satoshi-datadir=/media/var/.bitcoin/^C
(ERROR) ArmoryQt.py:1170 - Failed to setup SDM
Setting netmode: 0
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
SDM       :  BitcoindNeverStarted BDM       :         Uninitialized
[...]

When I unset "Let armory run bitcoin in background" Eveything works.

By the way...
Why it takes 2.1GB RSS memory and scan all data every run ?

[etotheipi]

When's the last time you did a git pull?  I have actually been updating the branch pretty constantly, trying to add the new features.  I see that error all the time, but it usually goes away with a quick update.  Check for the latest version, and then also try changing your settings instead of using the command line (I mean, both should work), but I just want to get more info about what does and does not work).

By the way...
Why it takes 2.1GB RSS memory and scan all data every run ?

After this update, both of those concerns will be fixed.  Right now, it's added a tremendous amount of simplicity and robustness to Armory, doing a rescan and holding a bunch of [now-very-large] lookup tables in RAM.  Not having to store any persistent, corrupt-prone data between loads is a huge benefit.  But now Armory has been ironed out so well, and resource usage is so high, that this will be the first thing I address after this update is done (both RAM usage and not rescanning every load).

[svirus]

Code:

 ~/BitcoinArmory $ git pull
Already up-to-date.

I do make clean and compile one more time and I have  still same error

[etotheipi]

Code:

 ~/BitcoinArmory $ git pull
Already up-to-date.

I do make clean and compile one more time and I have  still same error

Can you email or PM me the whole log file.  I suspect there's some logged messages higher up that indicate what happened.   The specific error you pasted for me is actually way after the root of the error.

[etotheipi]

Windows 64-bit users:  there's now a Windows version available for "testing".

http://dl.dropbox.com/u/1139081/ArmoryTestingReleases/armory_0.87.8-testing_win64.msi

I say "testing" because this is really just a sanity check -- it's still fairly rough around in edges, in terms of accommodating all the different configurations.  If you are not upgraded to 0.8.1 yet, uninstall Bitcoin-Qt and install the new Armory and let it download and install the new one for you (yes, there's cryptographic verification to make sure you're getting the right files! see the top of versions.txt).  Let me know how the process goes! 

There will be issues.  Please help find them!  One issue I'm concerned about (but haven't seen happen recently), is Armory closing and leaving bitcoind.exe running in the background.  That shouldn't happen anymore, but I can't know for sure until people try it.  If it does happen, please tell me what you did.  I don't want to have to add a "kill" button to the dashboard, but I'm ready to if necessary

This has taken way too much time, but I'm fairly certain it's almost done!  Help me test it!

[etotheipi]

Gah!  I see the error, and not sure why it happened.  It looks like it's trying to run a non-existent bitcoind.exe.  Can you point it to the correct location in the settings file? 
C:\Program Files (x86)\Bitcoin\daemon\
Run it in offline mode to be able to open and change the settings.  Then continue testing. 

This is actually a wacky bug, because it's not supposed to try running BITCOIND_PATH unless it checks that it exists.  But somehow it found a non-existent one on your system!  I'll double-check the logic to see how that happened.

[prezbo]

Gah!  I see the error, and not sure why it happened.  It looks like it's trying to run a non-existent bitcoind.exe.  Can you point it to the correct location in the settings file?  
C:\Program Files (x86)\Bitcoin\daemon\
Run it in offline mode to be able to open and change the settings.  Then continue testing.  

This is actually a wacky bug, because it's not supposed to try running BITCOIND_PATH unless it checks that it exists.  But somehow it found a non-existent one on your system!  I'll double-check the logic to see how that happened.

Nope, the bitcoind path is correct. It's not working even after explicitly setting it in the config. I'll keep on trying

edit: ok, bitcoind itself crashes on start with

Code:

************************
EXCEPTION: 22DbRunRecoveryException
DbEnv::open: DB_RUNRECOVERY: Fatal error, run database recovery
c:\Program Files (x86)\Bitcoin\bitcoind.exe in AppInit()

terminate called after throwing an instance of 'DbRunRecoveryException'
  what():  DbEnv::open: DB_RUNRECOVERY: Fatal error, run database recovery

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

[etotheipi]

Gah!  I see the error, and not sure why it happened.  It looks like it's trying to run a non-existent bitcoind.exe.  Can you point it to the correct location in the settings file? 
C:\Program Files (x86)\Bitcoin\daemon\
Run it in offline mode to be able to open and change the settings.  Then continue testing. 

This is actually a wacky bug, because it's not supposed to try running BITCOIND_PATH unless it checks that it exists.  But somehow it found a non-existent one on your system!  I'll double-check the logic to see how that happened.

Nope, the bitcoind path is correct. It's not working even after explicitly setting it in the config. I'll keep on trying

The log file shows this:

Code:

Using: C:\Program Files (x86)\Bitcoin\bitcoind.exe

Does that file exist for you?

[prezbo]

Gah!  I see the error, and not sure why it happened.  It looks like it's trying to run a non-existent bitcoind.exe.  Can you point it to the correct location in the settings file?  
C:\Program Files (x86)\Bitcoin\daemon\
Run it in offline mode to be able to open and change the settings.  Then continue testing.  

This is actually a wacky bug, because it's not supposed to try running BITCOIND_PATH unless it checks that it exists.  But somehow it found a non-existent one on your system!  I'll double-check the logic to see how that happened.

Nope, the bitcoind path is correct. It's not working even after explicitly setting it in the config. I'll keep on trying

The log file shows this:

Code:

Using: C:\Program Files (x86)\Bitcoin\bitcoind.exe

Does that file exist for you?

Yes, seems to be a problem with bitcoind:

Code:

************************
EXCEPTION: 22DbRunRecoveryException
DbEnv::open: DB_RUNRECOVERY: Fatal error, run database recovery
c:\Program Files (x86)\Bitcoin\bitcoind.exe in AppInit()

terminate called after throwing an instance of 'DbRunRecoveryException'
  what():  DbEnv::open: DB_RUNRECOVERY: Fatal error, run database recovery

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

edit: ok, there are two bitcoind.exe files, one in the main bitcoin directory and another inside daemon/. Only the one in the daemon/ folder works, but armory crashes with the same error

Code:

2013-03-29 18:26 (INFO) -- ArmoryQt.py:1153 - Bitcoin-Qt/bitcoind is Available: 0
2013-03-29 18:26 (INFO) -- armoryengine.pyc:10283 - Found bitcoind in the following places:
2013-03-29 18:26 (INFO) -- armoryengine.pyc:10285 -    c:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe
2013-03-29 18:26 (INFO) -- armoryengine.pyc:10285 -    C:\Program Files (x86)\Bitcoin\bitcoind.exe
2013-03-29 18:26 (INFO) -- armoryengine.pyc:10285 -    C:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe
2013-03-29 18:26 (INFO) -- armoryengine.pyc:10287 - Using: c:\Program Files (x86)\Bitcoin\daemon\bitcoind.exe
2013-03-29 18:26 (ERROR) -- ArmoryQt.py:1192 - Failed to setup SDM
2013-03-29 18:26 (ERROR) -- Traceback (most recent call last):
  File "ArmoryQt.py", line 4559, in <module>
  File "ArmoryQt.py", line 471, in __init__
  File "ArmoryQt.py", line 3912, in setDashboardDetails
ShouldNotGetHereError: Bitcoind Never started...?

[etotheipi]

Crap, I guess I need to catch that condition!  At least I have access to that output so I can interpret the error message.

Also, having the bitcoind.exe in that directory is not normal.  It should be in the daemon directory.  I wonder if that's an ancient version of bitcoind.exe...?

[prezbo]

Crap, I guess I need to catch that condition!  At least I have access to that output so I can interpret the error message.

Also, having the bitcoind.exe in that directory is not normal.  It should be in the daemon directory.  I wonder if that's an ancient version of bitcoind.exe...?

Quite possibly. That would explain why it's not working with the new blockchain structure, I guess.