Armory - Discussion Thread

[etotheipi]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

[prezbo]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

latest (0.84.5-alpha) 64 bit, win 7

[etotheipi]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

latest (0.84.5-alpha) 64 bit, win 7

Can you export a log file and email it to me?  etotheipi at gmail dot com.

If Armory isn't seeing new blocks found by Bitcoin-Qt, then there will definitely be something indicative of the problem in the log file.

[prezbo]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

latest (0.84.5-alpha) 64 bit, win 7

Can you export a log file and email it to me?  etotheipi at gmail dot com.

If Armory isn't seeing new blocks found by Bitcoin-Qt, then there will definitely be something indicative of the problem in the log file.

done, hope it helps.

[picobit]

The  BlockDataManager thread runs in its own little loop, and only executes non-Qt-related code in its own little bubble... via Queue.Queue objects.  The main thread dumps "work" for it onto the BDM.inputQueue, and it dumps the result to the BDM.outputQueue -- it does not use a callback structure because my computer almost imploded when I attempted Qt/Qapp calls from the BDM thread.

Thanks for getting your hands dirty with this.  I just got back from an extended vacation and don't have a lot of time to deal with this.  If you have some time, do you think you could construct a very simple example worthy of posting in a bug report (or maybe we discover a workaround)?  I'm thinking -- super simple C++ code that does nothing (basically a single main.cpp file), make an importable swig module with it using the "-threads" option, then creating a blank app that only calls getSaveFileName().  I will get to it eventually, if you don't.  But I may not have time until this weekend...

I did some more specific searching for this problem, and I still found nothing.  I suspect it's not a widely triggered bug:  only SWIG+threading+PyQt+OSX.

I'll give it a try - but I am not sure I get time to do it this week.  I am, however, quite motivated to see Armory keep working on OSX, it has become my favourite client.  Not that I can put my finger on what makes it particularly attractive - I guess you have just made an overall good design.

[prezbo]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

Ah, that was my fault. I was running the latest test version of bitcoinqt, and it apparently won't work with that (presumably because it uses ultraprune?). Reverted to 0.7.1 and it works fine now. Sorry for any inconvenience.

[etotheipi]

I've been using armory for a short while now and I'm very impressed by how far it has come. One thing bothers me though, armory won't update blockcount, even when blockchain is synchronized in bitcoinqt. For every new block I have to restart armory (and wait forever for it to scan the blockchain again). Is this a bug or is it like this by design?

That is definitely not by design.  What version and OS?

Ah, that was my fault. I was running the latest test version of bitcoinqt, and it apparently won't work with that (presumably because it uses ultraprune?). Reverted to 0.7.1 and it works fine now. Sorry for any inconvenience.

Crap.  I thought it was going to work with the latest Bitcoin-Qt.  I guess I should actually download it and make sure it works myself (yeah yeah, more testing...)

[picobit]

I did some more specific searching for this problem, and I still found nothing.  I suspect it's not a widely triggered bug:  only SWIG+threading+PyQt+OSX.

I found a workaround!   

Not a solution, but something looking like an acceptable workaround.  I added options=QFileDialog.DontUseNativeDialog to the QFileDialog.getSaveFileName function call (ArmoryQt.py line 1505).  Then it opens a not too petty Qt-style file save dialog instead of the Mac OS X style file save dialog.  And it works!

Of course it should only be done on Mac OS X, but that should be easy to implement.

It is the only place in the code that save dialogs are created, so it fixes both the offline transactions and the export transactions problems.  It really looks like a bug in the Qt library.

[etotheipi]

I did some more specific searching for this problem, and I still found nothing.  I suspect it's not a widely triggered bug:  only SWIG+threading+PyQt+OSX.

I found a workaround!   

Not a solution, but something looking like an acceptable workaround.  I added options=QFileDialog.DontUseNativeDialog to the QFileDialog.getSaveFileName function call (ArmoryQt.py line 1505).  Then it opens a not too petty Qt-style file save dialog instead of the Mac OS X style file save dialog.  And it works!

Of course it should only be done on Mac OS X, but that should be easy to implement.

It is the only place in the code that save dialogs are created, so it fixes both the offline transactions and the export transactions problems.  It really looks like a bug in the Qt library.

Ooooh, cool.  That's exactly what I needed!  I guess that's a good reason alone to make changes to 0.84.5 before renaming it to beta.  And, it's very easy to branch based on OS.

Thanks picobit!  You're a lifesaver!

[HorseRider]

etotheipi,

Thanks for your great work to let us have this easy offline wallet feature. When will you consider develop a light weighted client, which is without the need of satoshi client and download the blockchain file? My online virtual machine's harddisk is going to run out of space very soon, and I have to wait for the block update every time I use Armory.

[etotheipi]

etotheipi,

Thanks for your great work to let us have this easy offline wallet feature. When will you consider develop a light weighted client, which is without the need of satoshi client and download the blockchain file? My online virtual machine's harddisk is going to run out of space very soon, and I have to wait for the block update every time I use Armory.

Horserider,

There is a couple different ways I am considering going.  I would like to do all of them, but I'm not sure what I'll have time for.  One thing will be to switch from RAM-heavy implementation to HDD-heavy (Armory duplicates blockchain data)... that doesn't solve your problem, but it simplifies it for a lot of other people (mostly people with less RAM).  It will also speed load times and allow you to connect to arbitrary [trusted] nodes.

The other thing is that I wanted to pioneer the blockchain pruning idea I posted about a while ago.  Since Armory is focused on security, I'm not really all that excited about making a pure lite-client and the associated risks with that.  But if I can get my pruning idea implemented, it would be extremely lightweight, and have 99% of the same security properties of a full client.  Not to mention, implementing and proving that concept would be quite a boon for Bitcon...

But I don't see either of these happening in the next month or two.  But I definitely want to do something that brings down the system requirements, considerably.  It will be on my mind after I finish the new wallet stuff...

[HorseRider]

etotheipi,

Thanks for your great work to let us have this easy offline wallet feature. When will you consider develop a light weighted client, which is without the need of satoshi client and download the blockchain file? My online virtual machine's harddisk is going to run out of space very soon, and I have to wait for the block update every time I use Armory.

Horserider,

There is a couple different ways I am considering going.  I would like to do all of them, but I'm not sure what I'll have time for.  One thing will be to switch from RAM-heavy implementation to HDD-heavy (Armory duplicates blockchain data)... that doesn't solve your problem, but it simplifies it for a lot of other people (mostly people with less RAM).  It will also speed load times and allow you to connect to arbitrary [trusted] nodes.

The other thing is that I wanted to pioneer the blockchain pruning idea I posted about a while ago.  Since Armory is focused on security, I'm not really all that excited about making a pure lite-client and the associated risks with that.  But if I can get my pruning idea implemented, it would be extremely lightweight, and have 99% of the same security properties of a full client.  Not to mention, implementing and proving that concept would be quite a boon for Bitcon...

But I don't see either of these happening in the next month or two.  But I definitely want to do something that brings down the system requirements, considerably.  It will be on my mind after I finish the new wallet stuff...

Sounds very great, we can wait,  thanks!

[etotheipi]

I did some more specific searching for this problem, and I still found nothing.  I suspect it's not a widely triggered bug:  only SWIG+threading+PyQt+OSX.

I found a workaround!   

Not a solution, but something looking like an acceptable workaround.  I added options=QFileDialog.DontUseNativeDialog to the QFileDialog.getSaveFileName function call (ArmoryQt.py line 1505).  Then it opens a not too petty Qt-style file save dialog instead of the Mac OS X style file save dialog.  And it works!

Of course it should only be done on Mac OS X, but that should be easy to implement.

It is the only place in the code that save dialogs are created, so it fixes both the offline transactions and the export transactions problems.  It really looks like a bug in the Qt library.

Done!  I just committed and pushed the updated code to the threading branch.  Anyone testing on OSX please pull and recompile and try again.  Hopefully that resolves the problem.

The good news is that I have had zero problems/bug reports about 0.84.5 except for this OSX problem.   The bad news is, I just realized that I have made some changes to master since I started the threading branch, and one of those changes is really important (the Bitcoin-Qt devs may start considering some Armory transactions non-std if this change isn't pulled).  This also means I'll have to do a tad bit more testing before release.  But so far, it's been going really well!

[61197da925c1]

hi two things i noticed after trying armory.

#1. my computer freezes sometimes, i have a quad core cpu and 4 gb of ram, this shouldn't happen.
#2. i cannot delete wallets without deleting the .dat file
#3. have you considered making it easier to delete wallets for one time use purposes?
#4. what is the difference between bitcoin-qt encryption and armory encryption, seems the same?

[etotheipi]

hi two things i noticed after trying armory.

#1. my computer freezes sometimes, i have a quad core cpu and 4 gb of ram, this shouldn't happen.
#2. i cannot delete wallets without deleting the .dat file
#3. have you considered making it easier to delete wallets for one time use purposes?
#4. what is the difference between bitcoin-qt encryption and armory encryption, seems the same?

(1) You're right, computer freezes should not happen.  The whole computer freezes?  Or just Armory?  For how long?  If you do experience it, can you please email me a copy of the log file?  (File --> Export Log File) (etotheipi at gmail dot com)
(2) There are no .dat files in Armory.  What are you deleting?  And please don't delete any files in the .armory directory -- that's a good way to lose stuff you didn't want to lose -- everything you need to be able to do should be accessible from Armory itself.  At least make paper backups of your wallets before you delete anything...
(3) Wallets are very delete-able -- just open the wallet properties and click on "Delete/Remove Wallet" on the right-hand side.
(4) Bitcoin-Qt encrypts your wallet by hashing your passphrase 25,000+ times, and using the result as an AES256 encryption key (this is called key-stretching).  This slows down an attacker trying to brute-force your password quite a bit.  However, hashing and AES can be implemented very efficiently on GPUs, so any Bitcoin miner could use his mining rig to brute force an encryption passphrase fairly efficiently.  On the other hand, Armory key-stretching uses a process that not only requires a lot of computation, but also requires multiple megabytes of cache/RAM.  GPUs are super fast because they have thousands of threads working in parallel, but each thread usually only has a tiny amount of fast local/shared memory.  If each thread needs 8MB of cache, it just won't be able to do the computation at all --- or the GPU must run with dramatically-reduced parallelization so that it can allocate that much memory to each thread.  This is what is meant by "GPU-resistant key stretching."

[prezbo]

(4) Bitcoin-Qt encrypts your wallet by hashing your passphrase 25,000+ times, and using the result as an AES256 encryption key (this is called key-stretching).  This slows down an attacker trying to brute-force your password quite a bit.  However, hashing and AES can be implemented very efficiently on GPUs, so any Bitcoin miner could use his mining rig to brute force an encryption passphrase fairly efficiently.  On the other hand, Armory key-stretching uses a process that not only requires a lot of computation, but also requires multiple megabytes of cache/RAM.  GPUs are super fast because they have thousands of threads working in parallel, but each thread usually only has a tiny amount of fast local/shared memory.  If each thread needs 8MB of cache, it just won't be able to do the computation at all --- or the GPU must run with dramatically-reduced parallelization so that it can allocate that much memory to each thread.  This is what is meant by "GPU-resistant key stretching."

Isn't the whole point of hashing that it is intrinsically sequential, therefore cannot be efficently computed in parallel? It's basically just a variation on the Blum-Blum-Shub pseudo random generator. At least that is something Shamir wrote in his paper about time-lock puzzles.

[picobit]

Isn't the whole point of hashing that it is intrinsically sequential, therefore cannot be efficently computed in parallel? It's basically just a variation on the Blum-Blum-Shub pseudo random generator. At least that is something Shamir wrote in his paper about time-lock puzzles.

Yes, hashing a single file is sequential.  But attempting to brute-force a wallet password requires hashing a dictionary of likely passwords (typically computer generated based on vast experience with leaked passwords, so the dictionary will contain the apparently hard-to-guess passwords we typically come up with).  And hashing a dictionary is of course "embarrassingly parallel" as the computer science guys call it.

[picobit]

Another Mac OS X bug 

I exercised the newest Armory from the threading branch, almost everything works fine, although it does tend to exhibit brief "freezes" when you expect a window to open or close, and nothing happens for 5-10 seconds (while one core of the CPU runs at 100%).

There is however a rare but important function that fails on Mac OS X: Printing a paper backup.  Again the problem is that Qt pops up a native print dialog, and it is dead.  However, in this case there is no non-native dialog to fall back upon, apparently the native print dialog is always used on Windows and Mac OS.

In this case I was actually able to open a second print dialog by pressing the Print button again while the dead first dialog was open.  That was followed by a segfault a second later.

It suspiciously looks like a Qt bug, which would make it really hard for you to work around it.

Personally, I can live with this bug.  I can print any paper backup from the offline computer (running Linux).  But it is annoying, and I will continue to investigate.

[prezbo]

Isn't the whole point of hashing that it is intrinsically sequential, therefore cannot be efficently computed in parallel? It's basically just a variation on the Blum-Blum-Shub pseudo random generator. At least that is something Shamir wrote in his paper about time-lock puzzles.

Yes, hashing a single file is sequential.  But attempting to brute-force a wallet password requires hashing a dictionary of likely passwords (typically computer generated based on vast experience with leaked passwords, so the dictionary will contain the apparently hard-to-guess passwords we typically come up with).  And hashing a dictionary is of course "embarrassingly parallel" as the computer science guys call it.

Ah, I see. I was thinking about it all wrong, instead of computing one key stretch in parallel you can compute several of them in parallel, each on it's own "core". Thanks for elaborating.

[etotheipi]

Another Mac OS X bug  

I exercised the newest Armory from the threading branch, almost everything works fine, although it does tend to exhibit brief "freezes" when you expect a window to open or close, and nothing happens for 5-10 seconds (while one core of the CPU runs at 100%).

There is however a rare but important function that fails on Mac OS X: Printing a paper backup.  Again the problem is that Qt pops up a native print dialog, and it is dead.  However, in this case there is no non-native dialog to fall back upon, apparently the native print dialog is always used on Windows and Mac OS.

In this case I was actually able to open a second print dialog by pressing the Print button again while the dead first dialog was open.  That was followed by a segfault a second later.

It suspiciously looks like a Qt bug, which would make it really hard for you to work around it.

Personally, I can live with this bug.  I can print any paper backup from the offline computer (running Linux).  But it is annoying, and I will continue to investigate.

This is really unfortunate.  I'm not sure why threading is causing such problems, since it doesn't seem to have anything to do with the multi-threading I implemented.  I definitely cannot setup a print dialog myself, I most definitely need Qt to help with that.  I wonder if there's any other dialogs that cause this problem...


Unless I hear something compelling, I think I've made my last change for beta.  It was an update to signature encoding that the Bitcoin-Qt devs would like to make more rigorous...  and I pulled in the terminal-signing-script for offline computers ("python extras/cli_sign_script.py wallet.file some.unsigned.tx").  

It seems I've already had 300 downloads of the version I posted a couple days ago!  People must be using it, and it can't be that bad if I haven't gotten any real bug reports yet  (besides the OSX issues).  If you've been using 0.84.5, please post here and let me know that you have been, and any comments about it, with regards to my plan to make it beta!   I am looking for that boost of confidence I need to pull the trigger on it!

P.S. -- While looking at the new wallet format I want to create, I figured out how I'm going to integrate unicode support.  It won't be part of beta, but it will definitely be part of the new wallet release.  Sorry to all you non-US that use all those weird characters in your tx comments and filenames   That support is coming!