Did you "make swig" after pulling the 0.81 code?   A bunch of stuff changed under the hood with 0.80+, including the way that blockchain scanning is done.  This error looks like you're using the latest python code, but the underlying C++ utilities are still the 0.77-compiled versions.

This looks like it might be related to the bug that I referenced in my previous post.  Though, the error I would expect to see would be "The value you have specified is valid, but the required transaction fee exceeds your remaining balance..." (because you specified bal-0.0005 and fee of 0.0005, but Armory needed to increase the fee to 0.001 and could not do so.

What version are you using?  Have you upgraded to 0.82.1?  If so, can you send me the log file?  Usually when a button does nothing, it's because an error was triggered and the function that was called just exits silently without finishing the job.  This will show up in the log file.  You can manually copy the log file from your ARMORY_HOME_DIR or you can use the main window "Export Log File".  Feel free to sanitize it before posting or emailing it to me.  

If not, then wait until tonight:  I will release 0.82.2 which has a bug fix that might resolve this.  If it doesn't, at least there will be a log file.


I haven't actually released the bug fix for that yet.  I got distracted last night, and will do it tonight instead. It will be 0.82.2.  All the SelectCoins errors should go away (hopefully, apetersson's issue, too!).


EDIT: Website should be back up.  There was a payment processing error that is resolved now.  Whoops.

FYI, I found the bug causing this.  It was pretty obvious once the debugger got there.  It only triggers when you try to create a transaction using inputs that are smaller than the transaction fee (and a few other conditions). 

I just fixed it, and also put in dooglus' fix for the blockchain loading (which I now understand to be re-opening the file after scanning... I misread it the first time).  I have a little bit of testing to do, then will re-release it as 0.82.2.  Since the changes are really trivial and amount to only a couple lines of code (since the last testing release), I'll only request testing for a day, and then make an "official" release tomorrow barring anything stupid.

Given that I've been out of town and thinking about other things, is there anything I've forgotten?  There's still some periodic crashes after running for a while.  But beyond that, I'm not aware of anything critical I need to address in the near future (besides another SelectCoins bug that is even more rare).  Please let me know if I missed your concern!

I'm not 100% sure I understand that picture.  It looks like you are saving the pointer-tree of each state, though there's only ever one copy of each piece of underlying data.  Multiple trees will be maintained, but leaf nodes will point to global copies of TxOut data.   Is that correct?

In that case, saving the state of the tree at a previous time means storing a whole lot of pointer data.  If I consider a basic binary search tree and use 8-byte pointers (assuming it's all in memory), then each node in your binary tree is going to store at least two pointers (prtLeft & ptrRight) and probably a few more bytes for random stuff like isLeaf and isRedOrBlack, etc.  So I assume 20 bytes per tree node.  For a tree of 2 million unspent TxOuts, then that's about 40 MB of storage for each state you want to store (each block).  And that's going to go up linearly with the size of the tree. 


(NOTE: I'm discussing this as if it's a single tree full of TxOuts, even though my proposal is about a main tree full of addresses containing unspent TxOuts, and subtrees of TxOuts.  However, the concepts are still valid, it's just easier to make my points as if it's a single tree of TxOuts).

On the other hand, if you're using a trie, you don't have to save the state of the entire trie since its structure is purely deterministic for a given set of addresses.  You only have to save the nodes that were removed that might have to be re-added later in the event of a rollback.  You look at the OutPoints of the TxIns in each tx of the block that need to be reversed, and re-add them to the trie.  Then remove the ones that were added by those transactions.  This is all standard O(1) trie operations.

Thus, the data that needs to be stored in order to rollback a block in the trie (besides the blockdata itself) is proportional to the transaction volume on the network.  And transaction volume is capped by network rules.  Currently, each block could be up to 1MB.  If you filled a block full of transactions that were entirely TxIns with nothing else, it would result in removing about 7200 TxOuts from the tree.  That means that that absolute upper limit for storage per block you want to be able to roll back is about 250 kB, regardless of tree size.

Okay.  Then at worst, it runs at the same speed as the current Armory utilities in Linux, and twice as fast once it's been cached.  I'll have to do the same tests on a native Windows system.

I figured, maybe the system remembers what files you access frequently and pre-caches them on load.  I guess not...

Yes!  This is in my plans.  I was hoping to implement multi-sig and then create an android app that lets you use your phone as a two-factor authentication device.  However, a lot of other priorities have come up, and the wallet file needs to be upgraded, so I haven't been able to pursue this just yet.  But I will!  It was intended to be one of the core Armory features!


Oh I must've misread the patch you posted.  And I must've misunderstood how the file reading/opening works.  I thought that even if the file changed size, that once it's open for reading it would be limited to that size until you explicitly reload it.  And because I wasn't sure about that, I stored/used the file sizes when it opened to try to avoid exactly what you said it's doing.  But clearly I botched something...

I just got back from vacation, and I need to get my life back into order.  But I'll be digging in hard, soon!

OneEyed, thanks for the feedback on this.   Admittedly, I hadn't considered all the other perspectives on the changes I proposed.  I know what I want out of Armory, but that is frequently at odds with others (who use it in different ways than me).  


That's a hell of a "can't you?" statement.  It could take me months to implement the networking protocol, and in the end, Armory would end up worse off (with a buggy, security-untested networking engine).  However, this is one steps towards that Armory-full-node goal, which may include cannabilizing another networking engine (maybe directly from bitcoind) to integrate into Armory (which is still a hardcore modification of the software).  So, there is a transient period where both processes would need it.  On the other hand:

• RAM usage is going to become nil.  Relying entirely on a disk-based DB engine means that it will work well on systems with lot of RAM that can do lots of caching, but it won't need it.
  
• You will be able to connect to any (trusted) full node, just not localhost and not just bitcoind/-qt.  That means you could have one bitcoind on your network, and all your systems can connect to it.  
• This paves the way for making a lite-version of Armory which doesn't need the full blockchain at all.  i.e. you can run your full bitcoind/-qt and Armory will only get the wallet-relevant information it needs.  It wouldn't actually copy the entire blockchain.  This isn't ready yet, but it's most definitely on my path to verison 1.0.  The end result would be a lot more stability and flexibility, with pretty much the same disk usage as right now.
• Although I don't really like the idea much, it would be possible for people to start services supplying the trusted full node for you.
• This paves the way for pruned-full-node and/or pruned-lite-node Armory, and implementing my own version of Ultimate Blockchain compression.  If this eventually gets me to a system like that, then everyone wins.

There's two things you missed here:
(1) Users would no longer be reading the entire blockchain on start. They'd be reading a tiny database that would load instantly.  This scan time would only be relevant on first load (ever) and re-scans when importing addresses/wallets.
(2) At least in Linux, the caching seems to hold between boots -- not perfectly, but I just tested it on my laptop and it scanned in 30s after a reboot.  The only way I can get it up to two minutes is to do an explicit cache dump (gmaxwell pointed out that "echo 3 > /proc/sys/vm/drop_caches" does exactly that).  

Alright, I'll spend some time playing with Armory in a VM with 1024x600 resolution.  I'm sure the issues will pop out.

I just got home from my vacation, and though my work-work is still going to be kind of intense, I will finally have some time to do real development again!  The first thing will be to dig into the SelectCoins problem with the watching-only wallet I have.  I will also inject Dooglus' blockchain utilities fix.  I don't think it's going to fix the periodic crashes (once Armory is running), but it should make sure the scan works.  Then I will release 0.82.2.

FYI -- while I had downtime on vacation, I started playing around with LevelDB.  I think I decided that changing the engine to maintain its own blockchain using leveldb is not only going to be extremely useful, but not terribly difficult either (the last round of blockchain engine updates properly modularized the blockchain management from the rest of the program). 

--First and foremost, all these crashing issues can go away: I'm pretty sure they're related to "hacking" my way through the blk000X.dat files which are being maintained by another process out of my control (that's not to say I did it right or that it can't work, but it seems like an unnecessary situation). 
--This also will allow users to connected to any full node, as long as it is 100% trusted -- it doesn't have to be localhost.  The downside is, if it is localhost, you'll be maintaining two copies of the blockchain.
--Start times will be near instantaneous -- as long as I'm storing blockchain data, I can easily store between-load information
--The new utilities will be designed with blockchain pruning in mind (my pruning method in particular, but any method that cares about separating out unspent TxOuts and possibly removing everything else).  I don't plan to implement pruning yet, but I've already figured out how I can organize the data such that it will be minimal engine modifications to implement lightweight nodes and pruned nodes.
--Faster!  The benchmarks I ran showed that a full scan can be done in like 15s even on a non-SSD.  Granted, that's when much of the file is cached by the OS already -- but it's still faster than what I've got right now which also should benefit from OS caching.

I know I keep pushing back beta -- but it's for a good reason:  I think Beta is going to get a lot of attention, and I don't want 5 minute load-times and 700 MB RAM usage to be the focal point of the discussions/reviews.  I think I can get all this implemented in a couple weeks, then a couple more weeks of testing.  Then a few of Armory's achilles heels will be gone!

You are the second person to report this problem.  It's strange that the SelectCoins code has been in use for 8 months and there's never been a hint of a problem with it, but suddenly multiple people run into this! 

The other person who reported this sent me their wallet so I can debug it.  I will dig into it tonight, and hopefully have a fix for 0.82.2, which will be an official release this week. 

The other user was using a wallet that had tons of inputs, and attempting to move the entire balance in one tx, which would require a large tx fee.  As such, I recommend you compute your total balance minus the requested tx fee, and put that in directly and change the fee directly to the recommended fee.  There may be an issue with the fee calculation when there's dozens of inputs, but if you give it the correct answer to start it may not have an issue.

Let me know.  This seems like a serious issue and I will be attending to it right away!

You can't resize the window?  Are there windows that can't be made small enough?  For a while, this was an issue and I thought I had fixed it.  But I guess not. 

Let me know what windows are problematic, I'll be sure to make the resizing & layouts more flexible.

Well that's why I started looking into this.  My original blockchain utilities design in Armory were more of an educational adventure to see how fast I could get the scanning from scratch (no external libraries), with lots of pointers, etc.  It was educational, and I was happy with the results.  But I had the same realization that even the index will eventually not fit in RAM, and thus there is no way to make the library scalable without using some kind of disk-based engine.  To that end, I had no interest re-inventing disk-based DB operations, etc.  When I heard the devs talking about LevelDB, I looked into and quickly realized why they chose it.  It is very simple, portable... and apparently fast!

However, gmaxwell just told me how to clear my disk cache and then retest.  It was dramatically slower for sure.  Probably by a factor of 5-10x.  However, I never ran the same test using my other blockchain utilities, so I don't know the actual comparison.  Given that modern OS'es are good at doing that caching, the LevelDB version is clearly a very good solution.

I'm currently investigating how I can transition Armory to maintain it's own blockchain data now with LevelDB

I'm sure you know I've spent a lot of time doing blockchain scanning, and attempting to optimize it in various ways in Armory.  In the end, the fastest I could achieve was about 20-25s on a full blk0001.dat (2GB) on Linux-64bit using memory-mapping.  I couldn't achieve the same thing in Windows (mmap'ing seems to be garbage there).   I just wanted to share my experience from playing with leveldb, how awesomely optimized it is.

First, I iterated through blk0001.dat and stored all transactions in a leveldb database using Key:TxHash-->Value:RawTx.   I plugged 4 addresses into a set<BinaryData> object.  I then iterated (out of order) through the entire database, pulling each transaction into memory from the DB, and then searching for TxOuts matching any of those 4 addresses.  This took 8.0 seconds! (but was only the TxOuts).  This included putting the TxOuts into a map<OutPoint,uint64_t> (OutPoint, nBTC), so that I could then scan the entire DB again and search for TxIns. 

At the end of both scans, I had the balance and unspent TxOut list from those four addresses.  It took approximately 12-13 seconds!  And this is with a non-SSD drive:  it seems like it shouldn't even be possible to go that quickly... I had to quadruple-check everything to verify that I wasn't missing something or accidentally supplying the answer.  Perhaps there was some OS-level caching of the DB files, though I would've expected such caching to improve my custom blockchain utilities performance. 

I don't intend to discourage or insult your efforts (in fact, I think you said you were getting similar speeds).  Plus, I think that writing such a parser/scanner is both useful and educational.  I just bring it up because I would guess that this is about as good as anyone can do with a non-SSD drive.  And it's especially impressive considering I did no optimizations.  I'm especially curious how leveldb works under the hood that it could be so efficient.  Maybe there's something to learn from it.

Dooglus,

You are very resourceful, thanks for helping debug the code while I'm on vacation!  I still don't fully understand the issue (and why that line of code fixed it), but it's not a bad thing to reopen the file just before scanning.  As it was before, blk0002.dat was being opened before blk0001.dat was scanned.  Your line reopens the file just before it is scanned (and I could see why that's a good thing).  The first thing I do when I get back is I'll put that fix into 0.82.1 and commit it.  And then I'll release 0.82.1 because it's about time...

Seriously, thanks for helping me with this!  I've been stressed that issues like this are cropping up and I've had no time to deal with it!

I have not done this myself, but I suspect it may have to do with the formatting of the keys.  For one, Armory displays private and public keys with a space between every quad of hex characters, which is done for ease of copying/typing by hand, but not necessarily handled by other applications (actually, it's probably not).  Second, for public keys, they are usually serialized slightly differently.  Armory displays this:


But most applications would use the concatenated version with "04" prefix byte  (and without the spaces between):


This wasn't done to be confusing:  it was done because this display is much cleaner and I didn't expect anyone to really use it for anything (and if they did, I expected they might already know what to do with it).  But as I write this, I realize I should add a checkbox for "Raw Public Key" which gives you exactly what other apps would expect.  

I think it's an interesting idea, and as you suggested, I don't want to derail the thread.  However, I think that the amount of history to maintain is not a critical question.  Standard reorgs on the main chain are rarely more than 1 block.  We save enough info for 10 blocks, and if a reorg happens further back than that (and the client is willing to continue), then the node can just request the missing information from peers.  It's all verifiable information, and if you are trying to catch up to the longest chain there should be plenty of peers who can supply it.

Yeah, I actually realized that and was editing my previous post to say this:

It's not actually trivial to reverse blocks in any particular pure-pruned scheme, since adding blocks involves deleting UTXOs.  So reversing the block means you have to re-add UTXOs that are not defined by the block you are reversing (it only references the OutPoint by hash:index, not the whole UTXO).  So you have to either save them or request them from another node.  Perhaps the solution is to keep a circular buffer of the last N UTXOs that were removed, as long as N is enough to cover the last, say 50 blocks.  Any time you use map.delete when updating the tree, you use a buffer.add to save it (which will also discard the oldest element in the buffer).  Then when you need to reverse a tx, you know the OutPoints that need to be re-added, and if N is large enough, you'll have it in the buffer.  Worst case, you have to fetch some data from another node, which isn't terrible.

Perhaps that's another reason to use insert-order-invariant tree structure.  If you have to reverse some transactions, you don't have to worry how you undo them.  Undoing a block/tx is just as easy as adding it in the first place.

I guess it didn't crash, either...?   Sorry I wasn't clear earlier... that breakpoint should never hit if the code is working.  I was hoping it would break and then the current state would be relevant to the problem.

But it looks like you got the crash somewhere relevant before.  Unfortunately, I think I'm just going to have to try to recreate it myself, later.  I'll try your method of putting in an infinite loop and see if I can get it to crash in gdb and then I can dig through it myself.  Unfortuantely, it's going to be a week before I'll get a chance to look at it.  But I'm sure you'll be a master of triggering the error by then and can help make sure that I can reproduce it at that time

Thanks again for taking the time to look at this...

I bet this is another error-in-the-error-logging-code issue.  I think it may have been fixed in the latest 0.82.1 version I posted.  Were you using it? (posted about 4 days ago).  It's been a while, but I seem to remember seeing a similar error and then fixing it.  If you are using 0.82+, can you try to send the transaction, then when it fails, go to the main menu and File-->Export Log File.  Only the last 50 or so lines will be necessary to see what happened.  If you think it might be sensitive, just extract the error message and PM/email it to me.   

But again, I think this may be fixed already in the previous post.  This is why I kept 0.82+ in testing so long, because I touched every part of the code to put in logging and inevitably broke stuff.... 

Oh, if I had stopped typing so fast into google, I probably would've notice the autocompletion answer.  Interesting that I always thought it "brandia" instead of "brandais".  That's what I get for never going to class... (though I did stay up all night debugging the insert function for a Patricia tree).

Speaking of that, when I do a search for the correct name, I get a lot of links to the exact class I took when I attended UIUC:
http://www.cs.uiuc.edu/class/fa05/cs225/cs225/_notes/_section/cs225ta4/Documents/bsttries.pdf

It's not the most exhaustive introduction to the trie strcutures, but if you are already familiar with the concepts, you can get the gist of it.  And in fact, I was really proposing the Patricia/Brandais hybrid tree.  A pure "Brandais" tree uses linked lists but is not level-compressed.

The linked list may also make it easier to combine children to produce the "hash" of a particular node:  You only need to concatenate the non-null children hashes, which will frequently be very few elements (and the "skip string").  And in those cases, you just hash consecutively through the linked list.  And the dense nodes near the top can be cached for when they need to be recalculated.  This will also reduce the amount of data that needs to be transmitted to communicate a branch of the tree to a node (though, it's probably still more than I originally estimated). 

Unfortunately, my understanding of the correct path forward once these structures need to move from RAM to disk is beyond me.