Armory - Discussion Thread

[etotheipi]

Geez, the moment I hit the post button on this post, the whole thing started imploding.   See next post for details...



***Armory RAM-Reduction is Nearly Complete!  (Linux & OSX only)***

Tested and working on an Ubuntu VM with 1GB of RAM!

-------

Linux & OSX users:  Please please please pretty please help me test it!   Especially on systems with 1GB of RAM (I wonder if 512MB would work...?  I don't see why not...)  While people are testing it in *nix, I will be working on the code branch to make the same thing happen in Windows.  I don't think it should take me too long, but who knows what kind of dragons I'm going to run into...

Armory looks mostly the same, and the only difference is that it will ask you if you are willing to wait, before it executes a full blockchain rescan from disk.  Armory is very smart about it, too -- if you remove an address or wallet then re-import it, no rescan will be required.  And the C++ tools can predict when a rescan is or is not required and will ask you only if necessary.  The only downside is that you can't cancel out of the scan once you've started.  But at least you get a pretty blue "Please Wait..." window

Usually, if you hit cancel before doing a rescan, Armory will either abort the operation or tell you that balances will be incorrect until you restart.  There shouldn't be any behavior that isn't explained.   But a LOT of stuff changed under the hood, and I'm sure I didn't catch everything in my own testing.   So please help me test!

I'm declaring this to be the alpha version of Armory 0.70-beta.    While none of my testing wallets have gotten corrupted, it's always a good idea to backup before trying the new version.  In order to test it in Ubuntu, follow the regular directions on the Building Armory from Source page,  but switch to the mmap branch first.  Modified instructions are below:

• $ sudo apt-get install git-core build-essential libcrypto++-dev swig libqtcore4 libqt4-dev python-qt4 python-dev python-twisted
• $ git clone git://github.com/etotheipi/BitcoinArmory.git
• $ cd BitcoinArmory/cppForSwig
• $ git checkout mmap
• $ make swig
• $ cd ..
• $ python ArmoryQt.py

BTW:  I have not tested 0.70 in offline mode yet.  I'm actually quite sure I broke something minor, but you can still use 0.61 or less on the offline computer even if you are using 0.70 on the online computer.   I really wanted to get the testing release out as soon as is reasonable, and offline systems are not the target of this release![/s]

[etotheipi]

I can't believe I f***ed that up so badly!  I was sooo focused on getting some logic right for 1/2 the overall problem, that I completely botched the other half of the program along the way!  But not badly enough to notice it in testing that first half, that the second half was totally wrong!

Armory 0.70 is not ready.  It's more than not ready, I need to un-overhaul some of the blockchain logic, as I broke instant rescans for users with lots of RAM, and apparently 32-bit systems have a high risk of seg-faulting due to rescans (due to weirdness with MMAP on large files on 32-bit systems).

Sorry about that!  Hopefully it won't take me more than a day or two!   (this is what I get for solving half the problem 2 months ago, then forgetting all the context I needed when I returned...)


P.S. -- My original mmap branch never even did a remap, which was the correct solution before I recklessly changed it.  Even though I'm going back, I'm wondering why the botched solution failed on 32-bit systems and maybe someone can help:

-- MMAP'ing 1.1 GB blockchain file always succeeds on first load.  
-- Remapping 1.100001 GB blockchain file always fails.  I attempted to unmap the original and close the file, before re-opening and remapping.

I bet it has something to do with virtual address spaces... there's enough virtual RAM to map a single 1 GB file, but not two? (perhaps I'm not actually unmapping the first, and it's trying to open a second)    Does this mean that even if the blk0001.dat file splits at 2 GB and I can map it okay, that it will fail when I try to mmap two such files at the same time?   I thought, as long as each file was less than 2^32 bytes in size, that mmap would work. If so, I might want to abandon all mmap-based solutions and do a lower-level overhaul (or just switch entirely to another library for blockchain management)

[jim618]

Hi Alan,

I figure we can only juggle so many things at one times in our brains so the more you focus on one area, the less you see elsewhere.  All part of the joy of being human.

:-)

[Red Emerald]

I'll test it in Mac and some linux VMs with varying RAM for you once it is ready.

Hopefully I'll figure out these mac compile issues, too.

[da2ce7]

I thought, as long as each file was less than 2^32 bytes in size, that mmap would work. If so, I might want to abandon all mmap-based solutions and do a lower-level overhaul (or just switch entirely to another library for blockchain management)

mmap is the right solution to the problem that you have... it just is botched up on 32bit oses.  I would consider asking "do we really need to support 32bit operating systems for a full bitcoin client?"  I would suggest that running in non-full-node mode for a 32bit os makes sense and is a reasonable trade-off.

[etotheipi]

I thought, as long as each file was less than 2^32 bytes in size, that mmap would work. If so, I might want to abandon all mmap-based solutions and do a lower-level overhaul (or just switch entirely to another library for blockchain management)

mmap is the right solution to the problem that you have... it just is botched up on 32bit oses.  I would consider asking "do we really need to support 32bit operating systems for a full bitcoin client?"  I would suggest that running in non-full-node mode for a 32bit os makes sense and is a reasonable trade-off.

Unfortunately, there's a lot of users still using 32-bit OSes.   And it's absolutely possible to support them without mmap.  But if mmap'ing won't work, I've artificially cut off a LOT of users.  Part of the reason for doing RAM reduction was so that 32-bit OSes could use it... (of course, using less RAM, in general, is a good reason)

It's easy to look at your own 6 computers, and they're all 64-bit systems, and forget that a large proportion of the population will not have that, especially the non-tech-savvy users.  And that is a crowd I want to appeal to, eventually.  But I do need RAM-reduction and independent networking first...

[da2ce7]

Unfortunately, there's a lot of users still using 32-bit OSes.   And it's absolutely possible to support them without mmap.  But if mmap'ing won't work, I've artificially cut off a LOT of users.  Part of the reason for doing RAM reduction was so that 32-bit OSes could use it... (of course, using less RAM, in general, is a good reason)

It's easy to look at your own 6 computers, and they're all 64-bit systems, and forget that a large proportion of the population will not have that, especially the non-tech-savvy users.  And that is a crowd I want to appeal to, eventually.  But I do need RAM-reduction and independent networking first...

Well the arguement could be made that non-tech-savvy users shouldn't in-the-long-run be running full nodes. They rarther should be running nodes like bitcoinj.   If the bitcoin block chain keeps on getting larger at this rate, it will be not that long untill your really need higher-end hardware to run a full node.

[etotheipi]

Unfortunately, there's a lot of users still using 32-bit OSes.   And it's absolutely possible to support them without mmap.  But if mmap'ing won't work, I've artificially cut off a LOT of users.  Part of the reason for doing RAM reduction was so that 32-bit OSes could use it... (of course, using less RAM, in general, is a good reason)

It's easy to look at your own 6 computers, and they're all 64-bit systems, and forget that a large proportion of the population will not have that, especially the non-tech-savvy users.  And that is a crowd I want to appeal to, eventually.  But I do need RAM-reduction and independent networking first...

Well the arguement could be made that non-tech-savvy users shouldn't in-the-long-run be running full nodes. They rarther should be running nodes like bitcoinj.   If the bitcoin block chain keeps on getting larger at this rate, it will be not that long untill your really need higher-end hardware to run a full node.

Oh good point.  I missed the distinction between full and lite nodes in your argument.  I do have long term plans to make Armory lite, or at least have the option, but it's a big change and I want to make sure it's done securely.  But that's totally not a discussion to be having right now...

Worst case, 32-bit OSes break when the blockchain file splits at 2GB.  I'll figure out what to do at that time...

[Red Emerald]

Got another feature request.  I'm not sure what to call it, but it should work something like p2pool's patron_sendmany.  I regularly want to send amounts to multiple wallets.

For example, every couple weeks I want to send X BTC to the donation address, Y BTC to an exchange deposit address, and Z BTC that gets divided up by varying percents to be sent to my offline wallet, my GLBSE address, and any number of other places (and then a fee for the miners).

This would be nice if you could send to an exact address or a wallet you own.  It would be like being able to save a transaction for rebroadcast, but with possible different amounts that are quick and easy to set.  When I click "make a transaction like this one" (or whatever you decide to call it), it would prompt for X, Y, and Z and the percents, but they would all default to be whatever they were last time.


Also, I found a small bug.  When building a transaction, I pasted the value that I wanted to send into the text field.  I hit send, but the fee wasn't high enough.  I did the math to subtract the new fee, hit command + A to select all in the text box, and hit paste with my new value.  I then hit send and it told me the value wasn't formatted properly.  I looked at the amount and I think that because the text field only shows at most 8 characters, the select all didn't get the amount and it was pasted as 3.3.12345678.  I couldn't tell without selecting the text field and moving the selection to the left.  I think making that text area larger would be helpful.


2 of my transactions have told me "The transaction that you just executed, does not appear to have been accepted by the bitcoin by the bitcoin network."  The first had the default 0.0005 fee, and the other is the transaction I mentioned in the bug above.  I simply clicked the "Copy Raw Tx" button and sent it through http://bitsend.rowit.co.uk/.  I should have at least 10 mintues (although an hour might be better) just to see if it did actually get broadcast.  I'll do that next time.  Also, I don't think there needs to be a comma in that error message.


I'm excited to try out the mmap branch once it's re-released for alpha testing.

[bitpop]

Theres a site that allows you to do that, i just forget what it is

[Red Emerald]

Theres a site that allows you to do that, i just forget what it is

You mean to create send_many transactions easily? That would be nice.  At first I thought you meant like http://bitsend.rowit.co.uk/, which I mention in the post lol

[bitpop]

Oh no, it gives you a way to send an amount calculated like you wanted by percentage, etc.

[bitpop]

There was also something called a bucket but he shut down

[etotheipi]

FINALLY!!!  

***Armory RAM-Reduction is Nearly Complete!  (0.70-beta)-alpha (Linux & OSX only)***

Tested and working on an Ubuntu VM with 1GB of RAM!  Not only is it working, it is working beautifully. If I don't have a lot of other programs open, the mmap'd data stays mostly in cache and doing a full rescan takes 2-4 seconds!

Unfortunately, the blockchain is getting big, so it's taking almost a full minute to load Armory (on the 1GB system).  But it's better than nothing!  And, given this fact, I'll be implementing something in the near future that loads the GUI immediately, and then does the scan.

-------

Linux & OSX users:  Please please please pretty please help me test it!   Especially on low-RAM systems.  While people are testing it in *nix, I will be working on the code branch to make the same thing happen in Windows.  I don't think it should take me too long, but who knows what kind of dragons I'm going to run into...

Armory looks mostly the same, and the only difference is that it will ask you if you are willing to wait, before it executes a full blockchain rescan from disk.  

Armory is very smart about it, too -- if you remove an address or wallet then re-import it, no rescan will be required.  And it only rescans exactly as many blocks as needed.  It can predict when a rescan is required and will ask you for confirmation only if necessary.  The only downside is that you can't cancel out of the scan once you've started.  But at least you get a pretty blue "Please Wait..." window

One annoying thing is that it can't predict when the rescan would happen super-fast due to caching, meaning you'll get a box warning you it will take a long time, then the process finishes immediately.  I guess there's worse problems to have

Usually, if you hit cancel before doing a rescan, Armory will either abort the operation or tell you that balances will be incorrect until you restart.  There shouldn't be any behavior that isn't explained.   But a LOT of stuff changed under the hood, and I'm sure I didn't catch everything in my own testing.   So please help me test!

I'm declaring this to be the alpha version of Armory 0.70-beta.    While none of my testing wallets have gotten corrupted, it's always a good idea to backup before trying the new version.  In order to test it in Ubuntu, follow the regular directions on the Building Armory from Source page,  but switch to the mmap branch first.  Modified instructions are below:

• $ sudo apt-get install git-core build-essential libcrypto++-dev swig libqtcore4 libqt4-dev python-qt4 python-dev python-twisted
• $ git clone git://github.com/etotheipi/BitcoinArmory.git
• $ cd BitcoinArmory/cppForSwig
• $ git checkout mmap
• $ make swig
• $ cd ..
• $ python ArmoryQt.py

BTW:  I have not tested 0.70 in offline mode yet.  I'm actually quite sure I broke something minor, but you can still use 0.61 or less on the offline computer even if you are using 0.70 on the online computer.   I really wanted to get the testing release out as soon as is reasonable, and offline systems are not the target of this release!

[Red Emerald]

Linux & OSX users:  Please please please pretty please help me test it!

I'm cloning the code now. I'll report back soon.

Wow. It seems to have compiled and started without issue!  I do have 4 GB of RAM though.

It would help testing and maybe even be useful in production to have a command to set the maximum amount of memory that Armory will use.  Something like "$ python BitcoinArmory.py --max-mem 1G"

Of course, for some reason bitcoind isn't downloading new blocks That's totally unrelated to the Armory code though.  This version does seem to be much more responsive though.

[etotheipi]

It would help testing and maybe even be useful in production to have a command to set the maximum amount of memory that Armory will use.  Something like "$ python BitcoinArmory.py --max-mem 1G"

Red Emerald,

It should be completely transparent to the user.  I believe mmap() will consume as much RAM as is available, and scale it back (at the expense of speed on the next rescan) if other programs starting adding memory pressure.

Actually... that something that should be worth testing.  My 1GB system does seem to "fill up" quickly, but the "python" process does seem to get smaller when I load more programs.  I'm going to have to experiment more

[take5]

So whenever I open Armory my two most recent transactions which occurred a few days ago still have yet to be marked as confirmed even though the blockchain is fully up to date. But yesterday when I tried the original 0.70-beta release everything looked fine with xxx confirmations for my transactions, and now after installing the most recent version it's back showing them as unconfirmed. Otherwise though everything is working great!

On my 4 GB system python stops at 1.0GB and doesn't change if that's any help.  I'll try to open some more programs and see what happens.

EDIT: Yeah python definitely shrinks I couldn't occupy more than about 85% of my memory.

[guruvan]

Ran the new code on a 32bit ubuntu precise, it does have 4GB though. Compiled right up, and Um....Ran great...started up in a little longer than the 0.5.3x satoshi client had been on the same box, and performed well.

Made a wallet, sent some money around, got some. I haven't tested it out too much other than that, but it seemed plenty stable. I'll run the same on a smaller machine tomorrow - 2GB.

Clicked on the little blue "?" and got nothing, but that was fine with me. I question everything, and expect no answers

[Red Emerald]

So when I double click a generated transaction (confirmed, and unconfirmed) to find out more info, I get this error.

This is a non-standard transaction, which cannot be interpretted by this program.  DO NOT ASSUME that you own these Bitcoins, even if you see your address in any part of the transaction.  Only an expert can tell you if and how these coins can be redeemed!  

If you would like more information, please copy the information on the next window into an email and send it to alan.reiner@gmail.com.

And this in the terminal.

Code:

nknown TxOut type
***Non-std transaction!
Traceback (most recent call last):
  File "ArmoryQt.py", line 1592, in dblClickLedger
    DlgDispTxInfo( pytx, self.walletMap[wltID], self, self, txtime=txtime).exec_()
  File "/Users/bwstitt/src/BitcoinArmory-mmap/qtdialogs.py", line 6201, in __init__
    self.txOutModel = TxOutDispModel(self.pytx,  self.main, idxGray=indicesMakeGray)
  File "/Users/bwstitt/src/BitcoinArmory-mmap/armorymodels.py", line 541, in __init__
    self.wltIDList.append(main.getWalletForAddr160(recip160))
  File "ArmoryQt.py", line 763, in getWalletForAddr160
    if wlt.hasAddr(addr160):
  File "/Users/bwstitt/src/BitcoinArmory-mmap/armoryengine.py", line 5672, in hasAddr
    return self.addrMap.has_key(addrStr_to_hash160(addrData))
  File "/Users/bwstitt/src/BitcoinArmory-mmap/armoryengine.py", line 607, in addrStr_to_hash160
    return base58_to_binary(binStr)[1:-4]
  File "/Users/bwstitt/src/BitcoinArmory-mmap/armoryengine.py", line 580, in base58_to_binary
    n += BASE58CHARS.index(ch)
ValueError: substring not found

It's a p2pool generation which I believe does have a non-standard output in it.

http://blockexplorer.com/block/000000000000015edbc82d3f7ab76d75157eed6d8f68d6c72e0b1e0291c6acd0

The window with the transaction info doesn't open.

Also, the transaction shows as confirmed even though it is only 50 blocks deep.  It seems like all transactions show as confirmed after 6, generated or not.

Side question.  Would you prefer these bugs posted at github?

[torusJKL]

I have compiled 0.70-beta and imported a Bitcoin-QT 0.5.x wallet.
It has imported all addresses.
But the odd thin is that every transaction is shown twice. As a result my spendable amount is now 2 times what it should be.