Armory closing at 99% scanning transaction history

[oxxymoronn]

Hey guys,

I'm having an issue I was hoping someone could help with.

For some reason... today when I try to load Armory it hits 99% scanning transaction history and then crashes.
I've tried about 10 times now and realize that I need some extra help.

Can someone please point me in the right direction?

Thanks

[oxxymoronn]

Problem signature:
  Problem Event Name:   APPCRASH
  Application Name:   ArmoryQt.exe
  Application Version:   0.0.0.0
  Application Timestamp:   49180193
  Fault Module Name:   _CppBlockUtils.pyd
  Fault Module Version:   0.0.0.0
  Fault Module Timestamp:   5294de6a
  Exception Code:   c0000005
  Exception Offset:   00001a11
  OS Version:   6.1.7600.2.0.0.768.3
  Locale ID:   1033
  Additional Information 1:   0a9e
  Additional Information 2:   0a9e372d3b4ad19135b953a78882e789
  Additional Information 3:   0a9e
  Additional Information 4:   0a9e372d3b4ad19135b953a78882e789

[notyabizness]

-INFO  - 1392041012: (..\BlockUtils.cpp:3690) Reading all headers and building chain...
-INFO  - 1392041032: (..\BlockUtils.cpp:3695) Total number of blk*.dat files: 115
-INFO  - 1392041032: (..\BlockUtils.cpp:3696) Total number of blocks found:   285111
-INFO  - 1392041032: (..\BlockUtils.cpp:3708) Getting latest blocks from blk*.dat files
-INFO  - 1392041032: (..\BlockUtils.cpp:3709) Total blockchain bytes: 15,322,661,233
-INFO  - 1392041032: (..\BlockUtils.cpp:3715) Parsing blockchain file: C:\Bitcoin\blocks/blk00114.dat
-INFO  - 1392041032: (..\BlockUtils.cpp:3812) C:\Bitcoin\blocks/blk00114.dat is 33,554,432 bytes
-INFO  - 1392041096: (..\BlockUtils.cpp:3729) Processed 6 raw blocks DB (64 seconds)
-INFO  - 1392041099: (..\BlockUtils.cpp:3758) Starting scan from block height: 0
-ERROR - 1392042812: (..\StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!

Anyone know how to reload the "full StoredTx" ?

[goatpig]

-INFO  - 1392041012: (..\BlockUtils.cpp:3690) Reading all headers and building chain...
-INFO  - 1392041032: (..\BlockUtils.cpp:3695) Total number of blk*.dat files: 115
-INFO  - 1392041032: (..\BlockUtils.cpp:3696) Total number of blocks found:   285111
-INFO  - 1392041032: (..\BlockUtils.cpp:3708) Getting latest blocks from blk*.dat files
-INFO  - 1392041032: (..\BlockUtils.cpp:3709) Total blockchain bytes: 15,322,661,233
-INFO  - 1392041032: (..\BlockUtils.cpp:3715) Parsing blockchain file: C:\Bitcoin\blocks/blk00114.dat
-INFO  - 1392041032: (..\BlockUtils.cpp:3812) C:\Bitcoin\blocks/blk00114.dat is 33,554,432 bytes
-INFO  - 1392041096: (..\BlockUtils.cpp:3729) Processed 6 raw blocks DB (64 seconds)
-INFO  - 1392041099: (..\BlockUtils.cpp:3758) Starting scan from block height: 0
-ERROR - 1392042812: (..\StoredBlockObj.cpp:1063) Cannot get tx copy, because don't have full StoredTx!

Anyone know how to reload the "full StoredTx" ?

Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it

[notyabizness]

Don't mind doing that.  My question, is , HOW?  I can't find any commands to do that....

[Automatic]

Don't mind doing that.  My question, is , HOW?  I can't find any commands to do that....

%appdata%\Armory
%appdata%\Bitcoin

[notyabizness]

Google is my friend.  Thanks for the pointer in the right direction...

[albon]

try deleting the DB folder in roaming/armory

worked for me

[lyth0s]

I just had this issue, I went to Help -> Rebuild and rescan databases, so far its working. Ill update once it's complete

Edit: Crashed again, ignore my previous advice

[picobit]

I just had this issue, I went to Help -> Rebuild and rescan databases, so far its working. Ill update once it's complete

Edit: Crashed again, ignore my previous advice

It may be due to a bad block in the Bitcoin-Qt/bitcoind database - the reference client seems to handle that better than Armory.  It should of course not place malformed blocks in its database, but it often does, and then ignores it.  But Armory may barf at them.

Try deleting the blocks of the original client.  If you don't want to wait ages downloading it again, google for torrents containing a recent snapshot of the blockchain, it is far faster to download.  If you do use a snapshot, you first need to start bitcoin-qt/bitcoind with a specific command line option (Google again, I cannot remember it), and let it rebuild its own database.  After this is done, you can delete the huge file you got with torrent, and then start Armory.  Delete the torrent before starting Armory, that way you never have three copies of the blockchain on the disk simultaneously, two is bad enough

[bto4630]

Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it

Why doesn't this worst case scenario excite me at all? Thing is like 30 gig.

I have to say that this bug has been there from the very beginning and somehow it's still not mitigated in any way. Reference bitcoin client has no issues with it, but somehow armory cannot handle it. Why? I call it bs.

It's very sad how otherwise functional client suffers from such usability problems.

[btchris]

TL;DR: Please try running Memtest86+, this worked for me!

About two or three weeks ago, I was having the worst problems getting past the Sync step. I spent probably about a week or two re-downloading the Bitcoin blockchain (more than once), rebuilding the Armory DB (several times), and rescanning, and it would always fail during the scanning phase with that same "Cannot get tx copy, because don't have full StoredTx!" error often at different points (at different %'s completed).

I became convinced there must be a bug somewhere in Armory. On one occasion in the debug log, it mentioned an additional error message "Invalid txIndex at height # index #". This gave me a place to start looking in the LevelDB database to see if there actually was a corruption. After quite a bit of digging, I did indeed find the issue: in the DB, each transaction in a block is given an ID starting at 0 (that's not the blockchain TxID). There was a transaction in the block whose sequential ID number was greater than the total number of transactions in that block (which is recorded separately in the DB). But how did this happen??

Looking at the transactions that should have been in this block, there was also one missing, so it looked like the corrupted transaction matched up with the missing transaction except for it's sequential ID. Looking closer, the ID was almost correct: it had just a single-bit error in its high nibble, flipping a 0 to a 1 and making it too large.

Well that's strange... maybe my hard drive was failing? Usually HD failures doen't cause single-bit errors, but rather entirely unreadable sectors or relatively large corruptions though.... Maybe a failing RAM stick?

Well, very long story short, I did indeed have some bad RAM which Memtest86+ found pretty quickly. After replacement, I had no trouble rebuilding and rescanning.

A big thanks to Armory for helping to find this bad RAM! If it weren't for all of the double and triple checking in the Armory code, this could have gone undetected for months, and who knows what I could have lost!! Thanks!!!

[goatpig]

Botched block in the DB. Usually rebuilding the Armory DB is enough. Worst case scenario you'll have to delete your copy of the blockchain and redownload it

Why doesn't this worst case scenario excite me at all? Thing is like 30 gig.

I have to say that this bug has been there from the very beginning and somehow it's still not mitigated in any way. Reference bitcoin client has no issues with it, but somehow armory cannot handle it. Why? I call it bs.

It's very sad how otherwise functional client suffers from such usability problems.

Bitcoin Core checks for blocks THEN writes them to disk, and doesn't bother to verify they were written correctly, rather relying on its LevelDB database. On some occasions, that results in botched blocks. And yes, Bitcoin Core will fail to read these botched blocks if asked to.

Armory depends on the raw files for its blockchain data, this is when those botched blocks get in the way. You also can't access Bitcoin Core's DB since LevelDB is set to lock it at process level.

We could hack around that but that's too demanding, will impact stability, and reflects an underlying issues with your copy of the blockchain, which frankly you should fix. The long term solution which to be introduce after supernode and the new wallet format will be blocks over P2P.

A big thanks to Armory for helping to find this bad RAM! If it weren't for all of the double and triple checking in the Armory code, this could have gone undetected for months, and who knows what I could have lost!! Thanks!!!

Well, there are redundancy checks in place for the crypto data, and more on their way with the new wallet format, so while unstable hardware will give you a bad experience, it won't outright lose you coins.

[bto4630]

Bitcoin Core checks for blocks THEN writes them to disk, and doesn't bother to verify they were written correctly, rather relying on its LevelDB database. On some occasions, that results in botched blocks. And yes, Bitcoin Core will fail to read these botched blocks if asked to.

Armory depends on the raw files for its blockchain data, this is when those botched blocks get in the way. You also can't access Bitcoin Core's DB since LevelDB is set to lock it at process level.

We could hack around that but that's too demanding, will impact stability, and reflects an underlying issues with your copy of the blockchain, which frankly you should fix. The long term solution which to be introduce after supernode and the new wallet format will be blocks over P2P.

See, this is your problem right here. You see it as a reason, but it's not - it is an excuse. The botched blocks are a problem, but you are not looking for solution, so you are not finding one. Core client works with these blocks just fine. I don't hear core devs saying "oh, we got a leveldb problem, here's a great idea - lets make you reload your whole blockchain"

Think of it in perspective: resyncing blockchain requires several days. Many of us have lives to live, and not babysit applications. Also, for some of us bitcoin is not an academic proof-of-concept, but a tool, meaning: sometimes you need to pay someone, not in several days when you get a chance to fix leveldb, but right now. Is it usable? How about that it also will result in 30-50gb of internet traffic? You have your funds stored in armory wallet, and this problem happens every couple months. Is that usable? Not on day-to-day basis. Many cable internet customers in US are limited to 200-400gb of traffic per month. So, for some of us, fixing this re-occurring problem requires spending a quarter of monthly internet limit. Is it usable? Maybe for those of us who were not thinking and put their funds in armory hoping this someday gets fixed. What is the value of armory after all these issues, why would anybody be even using it? You still need to deal with maintaining core, this doesn't change. But in addition to this pin in the ass, now you also must maintain it in a way that allows armory to swallow its data. Security? I would rather be using paper wallet, at least I can import it into another client when I need access to funds. Offline transactions? I concur, this is a unique feature, but is it worth going through all this hassle to just use this one feature? I strongly doubt so. Don't get me wrong - i tried using armory, I tried hard, and in the end it's just not worth it.

The bottom line here is: if your application is unusable, then it will not be used. You need to find a way to make it working without regular glitches, and preferably without requirements for major downloads, or it will wither.

Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory

[goatpig]

Our priority at Armory is security. We do understand the need for better user experience but not at the cost of our utmost priority. If you reread my post you'll see we have a long term solution, but it will come in due time, because other features have to get there first to allow this solution to implemented. It isn't stand alone solution that we could implement over night or we would have done so obviously.

You're talking about using paper wallets, but they too are inconvenient, until the point when BIP32 is widely implemented. There are also a lot more clients out there that choose user experience over security. We have a role to fulfill before we can get the user experience up to acceptable levels. By far Armory is not considered the easiest nor the most welcoming wallet, but some people think this is an acceptable trade off for the added features, and here we are today. What you are trully experiencing is a software niche that hasn't just matured yet.

Obviously we would like to get rid of all these quirks. As a matter of fact my work for the past 4 months has been towards implementing long term scalability and stability fixes. The problem you describe cannot be dubbed as a "leveldb" error. This is a symptom of your local copy of the blockchain. Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

Let's put aside the consequence that has on the network, and focus on Armory. Armory requires a full node to function. Implementing a long term solution to this issue would require a shift in paradigm. That can't be addressed lightly nor quickly.

Also consider that most of the codebase is still etotheipi's work: about 100k lines of code over 2 years. He had to make some choices to ensure Armory functions overall. You can't just spend 6 month to fix scanning issues that can be effectively decentralized. Consider that the earlier versions of Armory didn't even have a DB: you had to fully scan the blockchain at every load. Now that 5 developers work full time on Armory, we have the resources and talent required to implement long lasting solutions to all the bugs and quirks, while adding new features.

Have you thought of maybe packaging offline transaction feature as a separate product? It does look like the only unique and useful feature that the other apps don't do better than armory

You can copy the signed Tx and broadcast it with another service, sure.

[picobit]

Simply put, while the you appear to be running a full node, you actually aren't, as you are missing block data.

That does not sound right.  Bitcoin-QT / bitcoind should be verifying its own data, so if block data is missing that verification should fail.  Is the problem not rather that there is extra (defective) data in the local copy of the blockchain which is being ignored by bitcoin-QT / bitcoind but not by Armory?

[goatpig]

We ignore the bad data. The issue occurs when the header is present, but the block data is missing or damaged. By this I mean there is a single instance of the incomplete block data, or that it is outright missing. There are a few instances of damaged blocks that have a second, proper copy available further down the blk file, but there are also cases where the only instance of a given block data is damaged.

I dont think Bitcoin Core verifies more than the headers.

[bto4630]

Our priority at Armory is security

Security is not a product of just having no known exploits. Sledgehammer has no exploits, but it's ridiculously cumbersome to store bitcoins in it. If your product is as good at being a wallet as sledgehammer, you have a problem. Usability has as much to do with security as access control. When using your product ends in denial of service, it's security failure.

[goatpig]

Our priority at Armory is security

Security is not a product of just having no known exploits. Sledgehammer has no exploits, but it's ridiculously cumbersome to store bitcoins in it. If your product is as good at being a wallet as sledgehammer, you have a problem. Usability has as much to do with security as access control. When using your product ends in denial of service, it's security failure.

My quote also implies that we can't just implement a random fix and call it a day. It needs to integrate with the model properly and be thoroughly tested. There has been quite an effort deployed to overhaul the backend, which should hit the next release. It will offer a lot more stability, speed and scalability. Scalability we can leverage to implement other sources for the blockchain data and hybrid modes, in a way we think is robust.

The DB version of Armory was introduced in 0.90, and it should be overhauled for 0.93. You are using a proof of concept version and will get to enjoy the rock solid implementation in 0.93. This is the speed at which we deploy releases, and keep in mind 0.90 was essentially a single man effort. Generally the full blockchain approach was the cheapest solution to implement, and  these kind of choices are easy to make when you are putting a proof of concept together. 0.91 was mainly a 3 job, and the whole team of 5 worked on 0.92. Now that we are functioning at full capacity, a lot is getting improved, and much faster.

Also, before we go after this particular issue you suffered from, we had to address the scalability issue that affected a lot more of our users. In contrast, your issue is actually fixable at the cost of a fresh blockchain download, which we did speed up with an integrated bootstrap downloader in 0.91. Keep in mind we identified this issue while deploying 0.90 test releases. This is isn't ideal of course, just side stepping the issue, but cheap enough that we offered this solution while working on something a lot stronger.

[jjdub7]

Hiya, I'm having the same issue with 0.92 trying to build the DBs and ending up with the C++ Runtime process crapping out at ever-increasing-but-never-quite-completing percents of the blockchain data.  Is there any process you could run that could verify the bootstrap chain against a p2p-downloaded version of the blockset via the core client?  I don't have a data limit from my ISP and this is kind of a side endeavor to try out multisig, so I'll try and fiddle around with my blockchain copy and see what I can manage.

And also, any updates on 0.93?  How close in the implementation and release process are you guys to being out of beta?  Keep up the good work - from what I've heard, Armory is awesome, so keep up the good work.