Armory - Discussion Thread

[Red Emerald]

Here's something that's on my mind that I hope to get cleared up.

I currently have a cold-storage set up thanks to Armory. Something that only just now came to my attention is the fact that the online watch-only wallet is managing to generate addresses without knowing their private keys. How does it do this? Does the watch-only copy derive a new seed that is only capable of generating public keys in the series? What's the process behind this? Or does the watch-only copy come with a healthy bulk of pre-generated addresses?

Armory wallets are deterministic.  http://bitcoinarmory.com/index.php/using-offline-wallets-in-armory

[chrisrico]

Ok, not to continue hijacking the threat, but I stayed up all night coding again and I vastly improved the serial communication.

Now, Armory tries to listen on the serial device as soon as it's started (rather than only in the offline tx dialog), and it encapsulates its messages to and from the server with protocol buffers. This allowed me to expand the types of messages available (as well as add more robust handling). Currently, you can import a watching only copy of a wallet directly from the offline device. I plan on extending the "Create new wallet" dialog to allow something similar - initiate the creation of a new wallet on the offline device, and send the watching only copy back. Also needed is to make the connection off by default, and enabled in the advanced or expert options, as well as setting the serial device and baud rate.

Also, I tested it on the Pi and a rate of 115200 works even better than 9600... Imagine that! I didn't run into any apparent slowness until trying to transfer a wallet over the wire...

The diff now has quite a few more changes, but it's still surprising to me how little code it took. I'm really liking python, and etotheipi, your code base is a pleasure to work with!

[chrisrico]

etotheipi, I think the "parseNewBlockData did not get enough data..." error is coming from this loop.

Code:

      while(bsb.streamPull())
      {
         while(bsb.reader().getSizeRemaining() > 8)
         {
            ...
   
            BinaryRefReader brr(bsb.reader().getCurrPtr(), nextBlkSize);
            parseNewBlockData(brr, fnum-1, bsb.getFileByteLocation(), nextBlkSize);
            blocksReadSoFar_++;
            bytesReadSoFar_ += nextBlkSize;
            bsb.reader().advance(nextBlkSize);
         }
      }

I made this small change and it drastically reduces the output, my freshly imported wallet still shows the right balance, but I'm sure it's not a real fix, since it looks like there's a bunch of data left unread...

Code:

            if(!parseNewBlockData(brr, fnum-1, bsb.getFileByteLocation(), nextBlkSize))
            {
               cout << "Next block size: " << nextBlkSize << endl;
               cout << "Buffer size remaining: " << bsb.reader().getSizeRemaining() << endl;
               cout << "Reader size remaining: " << brr.getSizeRemaining() << endl;
               break;
            }

And here's some sample output with that change:

Code:

***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 2231
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 3349
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00001.dat
/home/bitcoin/.bitcoin/blocks/blk00001.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 9362
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 20808
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00002.dat
/home/bitcoin/.bitcoin/blocks/blk00002.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 16391
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 28286
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00003.dat
/home/bitcoin/.bitcoin/blocks/blk00003.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 25310
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 25309
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00004.dat
/home/bitcoin/.bitcoin/blocks/blk00004.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 3530
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 6970
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00005.dat
/home/bitcoin/.bitcoin/blocks/blk00005.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 15835
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 24656
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00006.dat
/home/bitcoin/.bitcoin/blocks/blk00006.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 27895
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 29726
Reader size remaining: 0
Attempting to read blockchain from file: /home/bitcoin/.bitcoin/blocks/blk00007.dat
/home/bitcoin/.bitcoin/blocks/blk00007.dat is 128 MB
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 14370
Reader size remaining: 0
***ERROR:  parseNewBlockData did not get enough data...
***ERROR:  parseNewBlockData did not get enough data...
Next block size: 0
Buffer size remaining: 41116
Reader size remaining: 0

[etotheipi]

tried to find the answer but couldn't. Is there a way to verify a message signed in armory with the bitcoinqt and vice versa?

Sorry BRules,

There is not available, yet.  I implemented message signing about the same time Bitcoin-Qt, and I was pretty much done by the time they identified how they were doing it.  I didn't have the crypto functions immediately available, yet, and I designed my message signing interface around "Signature blocks" which the Bitcoin-Qt devs didn't seem all that interested in.  When I upgrade to new wallets, I will grudgingly convert to their version, though I hope I can convince them to do something like signature blocks, since their mechanism seems really clunky and error-prone...

Ok, not to continue hijacking the threat, but I stayed up all night coding again and I vastly improved the serial communication.

Now, Armory tries to listen on the serial device as soon as it's started (rather than only in the offline tx dialog), and it encapsulates its messages to and from the server with protocol buffers. This allowed me to expand the types of messages available (as well as add more robust handling). Currently, you can import a watching only copy of a wallet directly from the offline device. I plan on extending the "Create new wallet" dialog to allow something similar - initiate the creation of a new wallet on the offline device, and send the watching only copy back. Also needed is to make the connection off by default, and enabled in the advanced or expert options, as well as setting the serial device and baud rate.

Cool!  Just don't get too crazy with it... we still want the interface to minimal in terms of complexity and functionality -- less room for security holes.   But, you are doing something I've wanted to do for some time.  I look forward to playing with this myself (especially because I already have serial-usb cables and a null-modem connector.

The diff now has quite a few more changes, but it's still surprising to me how little code it took. I'm really liking python, and etotheipi, your code base is a pleasure to work with!

I will never get tired of hearing that  All this stuff you've been coding, it sounds like you're ready to become an Armory apprentice!  You're already debugging my next upgrade challenge (the 0.8 blockchain stuff)...

[Ente]

All this stuff you've been coding, it sounds like you're ready to become an Armory apprentice!  You're already debugging my next upgrade challenge (the 0.8 blockchain stuff)...

https://www.youtube.com/watch?v=QMJ4JdOlRsQ#t=3m09s
Congrats, chrisrico!
Nice stuff you are doin' there! :-)

Ente

[etotheipi]

etotheipi, I think the "parseNewBlockData did not get enough data..." error is coming from this loop.

Code:

      while(bsb.streamPull())
      {
         while(bsb.reader().getSizeRemaining() > 8)
         {
            ...
   
            BinaryRefReader brr(bsb.reader().getCurrPtr(), nextBlkSize);
            parseNewBlockData(brr, fnum-1, bsb.getFileByteLocation(), nextBlkSize);
            blocksReadSoFar_++;
            bytesReadSoFar_ += nextBlkSize;
            bsb.reader().advance(nextBlkSize);
         }
      }

@chrisrico

I just talked to sipa on IRC and now I know what's going on.  They not only changed the filenames and location, but they changed the write operations too to reduce fragmentation.  Your observation makes so much more sense, now...

bitcoind/-qt now will now pre-allocate the blk files using 16MB of zeros.  When the 16MB is filled, it appends another 16MB, etc.  Understandably, this wreaks havoc on my block utilities which pay attention only to file size to be notified of new blocks, and hence why you never saw any updates.  Sipa pointed out that since each block starts with the magic bytes, simply watching the next four bytes after the next block is a good way to catch new blocks.  It means rewriting the loop slightly, and the "readBlkFileUpdate()" function.  So much for a smooth transition to 0.8! 

Sipa never told me about this because he assumed I watched for block messages on the network instead of watching the files.  Oh well.

[chrisrico]

That makes a lot of sense now.

Sipa never told me about this because he assumed I watched for block messages on the network instead of watching the files.  Oh well.

Why not listen for blocks on the network?

[etotheipi]

That makes a lot of sense now.

Sipa never told me about this because he assumed I watched for block messages on the network instead of watching the files.  Oh well.

Why not listen for blocks on the network?

Two reasons:

(1) Armory doesn't hold raw block information.  It only stores pointers to where the data is in the blk000X.dat files, using std::map<Hash, FileDataPtr>.  So, it doesn't make sense to get it from the network when I have to eventually find it on disk, anyway.
(2) Doing it this way, allows me to build blockchain utilities/tools using armoryengine.py without having to use any of the networking code.  Once you bring networking into the mix, the scripts get dramatically more complicated, because you're now operating in a python-twisted event loop.  The way it is, you can create super-simple "watcher" scripts without any concept of networking:

Pseudo-code:

Code:

loadWallets()
registerWallets()
loadBlockchain()

while(True):
   newBlks = TheBDM.readBlkFileUpdate()

   if newBlks>0:
      doCoolStuff()

   sleep(1)

[etotheipi]

FYI:

I decided that my queue of bug fixes and polishing requests was big enough to be worth taking a temporary vacation from my RAM reduction, and update master.  And by updating "master," I mean branching it into the "dev" branch and fixing a bunch of bugs there.  That will be 0.86.1.  For those wondering about my branching behavior -- I tend to work on branches appropriately named for the hardcore upgrade I'm doing.  However, if I'm just fixing bugs or doing minor enhancements, I stick to the "dev" branch.  Sorry if it makes it difficult to follow the latest testing code!

I will be adding more stuff to the dev branch in the next week.  Mainly handling bug reports and more polishing.  I also need to fix single-threaded mode... I intended to be able to say "TheBDM.setBlocking(True)" and have scripts be able to operate as if everything was a single thread, but it turns out I missed a few things, and I broke some of my code samples.

So version 0.86.1 has:

• -Fixed addresses disappearing and reappearing.  The "alreadyUsed" variable is no longer erased from recent addresses on a restart.
• -Full sorting of the transaction ledger is restored (and optimized!).
• -New warning dialog for offline transactions, reminding users to verify tx details before signing (I have run into a few people who didn't realize they should manually verify tx before signing -- it's not secure just because the keys are offline!)
• New right-click menu on main ledger.

Switch to dev branch right now if you want to get the new stuff!

I'm not sure how I'm going to handle the Bitcoin-Qt 0.8 update... it's going to be kind of complicated to accommodate pre-0.8 and post-0.8 in the same code base (because I have to modify the under-the-hood, optimized C++ loops).  I might just have to compile separate versions  (and maybe I should add a warning message to the upcoming release that the user should check the website for Armory updates if they plan to user Bitcoin-Qt 0.8 )


By the way, I'll be attending the East Coast Bitcoin Summit in Philadelphia tomorrow.  It sounds like it's shaping up to be a big event, so if you live on the east coast (US), then you should join us.  I received a late invitation so I won't be presenting anything, but the organizer did invite me to be on a Q&A panel at the end.  Looking forward to it

[etotheipi]

Someone just asked me about how to run Armory over Tor.  I realized that not only do I have no idea,  but I've heard users claim they have done it,  and I never paid attention.  Can someone please explain how to do it?  I will add the description to the webpage once I see a consensus.

[N.Z.]

Unless you implement SOCKS support into Armory, you won`t be able to run Armory itself over Tor. But maybe you were asked how to connect to Bitcoin net over Tor. In that case you will have to start bitcoind or bitcoin-qt with "-proxy=127.0.0.1:9050" or whatever host Tor is running. Also maybe you should start Armory with "--skip-online-check" parameter.

P.S. I`m a bit confused with your branches...where is the latest Coin control branch for now? And please don`t forget to add labels to CC`s selection window

[etotheipi]

Unless you implement SOCKS support into Armory, you won`t be able to run Armory itself over Tor. But maybe you were asked how to connect to Bitcoin net over Tor. In that case you will have to start bitcoind or bitcoin-qt with "-proxy=127.0.0.1:9050" or whatever host Tor is running. Also maybe you should start Armory with "--skip-online-check" parameter.

Sorry, I meant:  Bitcoin-Qt is running over Tor, but Armory cannot connect to it while it is in this state.  How is this resolved?  Do you have to change the ports in Armory?  skip-online-check is something else that needs to be done... forgot about that...

P.S. I`m a bit confused with your branches...where is the latest Coin control branch for now? And please don`t forget to add labels to CC`s selection window

Coin-control has been merged into master already.  So just getting the current master branch will have coin-control (or downloading from the website, 0.86-beta).  I recommend switching to the "dev" branch, though, which has a bunch of bug fixes and needs testing (0.86.1). 

I guess I should clean up all these one-time-use branches I created in the past...

If you missed it the first time, I did put labels in the mouse-over text for each address in the coin-cointrol window.  I intended to add a column, I just forgot it in 0.86.1. Would it be acceptable to add a very narrow comment field, that only shows you the first 16 characters of the comment, and then you can mouseover for the full comment?  I like the compactness of the CC window, but it's not critical.

[N.Z.]

How is this resolved?

Oh, that. Add "-listen" to bitcoin-qt startup script, it is turned off when "-proxy" is enabled.

Would it be acceptable to add a very narrow comment field, that only shows you the first 16 characters of the comment, and then you can mouseover for the full comment?

Yes, that would be very nice

[wachtwoord]

I have a feature request: the password I have to encrypt the wallet is quite long and I use it really infrequently (I have not put time into finding out how to use the offline functionality yet so this is my main protection). Sometimes I would like to enter the passphrase and Armory should tell me whether the passpahrase is correct. Right now this can only be done when making a transaction or when changing the passphrase. Could this functionality be added?

[etotheipi]

I have a feature request: the password I have to encrypt the wallet is quite long and I use it really infrequently (I have not put time into finding out how to use the offline functionality yet so this is my main protection). Sometimes I would like to enter the passphrase and Armory should tell me whether the passpahrase is correct. Right now this can only be done when making a transaction or when changing the passphrase. Could this functionality be added?

One thing you can do is click on "Print paper backup" or "Backup Individual Keys".   That will ask you for your passphrase and tell you if you entered it correctly.

[wachtwoord]

Great, thanks! 

[etotheipi]

P.S. I`m a bit confused with your branches...where is the latest Coin control branch for now? And please don`t forget to add labels to CC`s selection window

Just updated the "dev" branch with comments in the coin-control dialog, and the dialog auto-sizes now, too.  Please try it out.

[N.Z.]

etotheipi, it looks great, thanks! Exactly what was needed

I got one more feature request I don`t know how exactly should that look like, but it would be great not just put "[[ Change received ]]" in comment to change address but point out the source address (or comment of this address) of that change.

[etotheipi]

etotheipi, it looks great, thanks! Exactly what was needed

I got one more feature request I don`t know how exactly should that look like, but it would be great not just put "[[ Change received ]]" in comment to change address but point out the source address (or comment of this address) of that change.

Hmm...  that's a bit of extra logic, but it can be done.  I presume you'd like to see this on the wallet properties address list, too?  It's a bit easier if I do it only on the coin-control dialog, because I don't have to worry about the efficiency/speed of the search operation (if you have thousands of addresses, it will execute this search on every refresh and could slow down wallets with a a lot of addresses).


So I think I have enough little extra features and bug fixes to be worth an official-but-minor release, 0.86.1... or 0.86.2...?  Maybe I should finally figure out how to do automatically-incrementing build numbers so that there will no longer be any conflict between different builds of the same 0.XX.Y.  So far, I've just been manually incrementing the version number, but I should be using that fourth version number ... such as incrementing it every time I start up Armory for testing.   Actually, I'm going to do that! 

Anyways, if there's any other little bugs or minor features you'd like in the 0.86.X release, let me know.  I'll probably send out a request for testing to my testers email list, and try to do a real release on Tuesday. Most importantly, I want that disappearing-address bug to go away.

[N.Z.]

I presume you'd like to see this on the wallet properties address list, too?

Well it`s up to you. Though I can`t imagine when this can be useful except coincontrol, I can be wrong of course. But anyway it`s easier if you just add comment once to change address (like [['comment of source address' change]] or similar) and not to search connections every time. In that case you just need to look into syntax of change of change ... of change comment.