Wallet unlock issue

[TierNolan]

I have a paper backup, so this is not a critical problem.

I generated around 10000 keys with a new wallet and sent money to a random selection of the addresses.  I computed them with a separate program.

The transactions sent money to high numbered addresses with a large number of unused addresses in between.

Armory detected the incoming transactions, but I had to force generate lots of addresses.

I tried to send money from the wallet.  After 10 minutes, Armory still hadn't unlocked it.

I killed the process and tried again.  This time, I tried to create a paper wallet.

When I pressed unlock, it didn't lock-up the process, but it didn't give the paper wallet print screen.

I then restarted again to generate a clean log file.

This log file is when I tried to send a transaction.  It doesn't seem to have anything in the log file.

I started Armory and then tried to send a transaction, once syncing was complete.

Code:

2013-08-09 23:16 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 2.9
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 5.9
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 8.8
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 11.8
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 14.7
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 17.6
2013-08-09 23:17 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 20.6
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 23.5
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 26.5
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 29.4
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 32.4
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 35.3
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 38.2
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 41.2
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 44.1
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 47.1
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 50.0
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 52.9
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 55.9
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 58.8
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 61.8
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 64.7
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 67.6
2013-08-09 23:18 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 70.6
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 73.5
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 76.5
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 79.4
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 82.4
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 85.3
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 88.2
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 91.2
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 94.1
2013-08-09 23:19 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 97.1
2013-08-09 23:19 (INFO) -- ArmoryQt.py:4166 - Dashboard switched to fully-online mode
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull1
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3483 - Switching Armory functional mode to "Online"
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull2
2013-08-09 23:19 (INFO) -- ArmoryQt.py:4166 - Dashboard switched to fully-online mode
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull1
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3483 - Switching Armory functional mode to "Online"
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull2
2013-08-09 23:19 (WARNING) -- ArmoryQt.py:1758 - Memory pool file was corrupt.  Deleted. (no further action is needed)
2013-08-09 23:19 (INFO) -- ArmoryQt.py:1697 - Syncing wallet: <snip>
2013-08-09 23:19 (INFO) -- ArmoryQt.py:1697 - Syncing wallet: <snip>
2013-08-09 23:19 (INFO) -- ArmoryQt.py:1697 - Syncing wallet: <snip>
2013-08-09 23:19 (INFO) -- ArmoryQt.py:1697 - Syncing wallet: <snip>
2013-08-09 23:19 (INFO) -- ArmoryQt.py:1709 - Current block number: 251198
2013-08-09 23:19 (INFO) -- ArmoryQt.py:4166 - Dashboard switched to fully-online mode
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull1
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3483 - Switching Armory functional mode to "Online"
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull2
2013-08-09 23:19 (INFO) -- ArmoryQt.py:4166 - Dashboard switched to fully-online mode
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull1
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3483 - Switching Armory functional mode to "Online"
2013-08-09 23:19 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:OnlineFull2
2013-08-09 23:20 (INFO) -- qtdialogs.pyc:5700 - Change address behavior: NewAddr

<Process killed by me after 5 mins>

Next, I restarted and tried to print a paper backup.  Doubling clicking on the wallet took a while (I assume to generate all the addresses).  This time, when I tried to generate the paper backup, the process locked up.

Is it possible that it simply takes ages to unlock, even if you have the right passphrase?  It was able to tell I had the wrong passphrase very quickly, when I tested it.

Something I noticed is that all wallet files appear to be updated when you start Armory.  This means that the wallet file is potentially corrupted on every start.

An alternative system would be to have a directory for each wallet.  In the directory would be an unchanging wallet and then a 2nd file with anything that changes.  I assume the main thing that changes is the number of addresses to generate?

[etotheipi]

There is a wild inefficiency with unlocking the wallet when the addresses were originally generated encrypted.  That was going to be fixed with the new wallets, but I haven't gotten there yet.

I know you don't want to let your keys hit the hard-drive unencrypted.  But if you can tolerate it, I recommend you remove encryption from your wallet, generate 10,000 keys, then encrypt it again.  I would've had a bunch of wallet code restructuring to get around this, and so far, only one other person in the last 18 months has reported trying this.

The issue comes down to this:  when you generate new addresses and the wallet is encrypted, it simply extends the public keys and marks that the private keys need to be generated on the next unlock.  But for reasons I can't remember, it does it all out of order, regenerating the same path of private keys (and checking the public keys) on for each address, and so it executes in O(N²) time instead of O(N).  N is the number of private keys to be computed.

For that reason, if you generate, say, 500 addresses at a time without removing encryption (unlocking between each batch), you can probably get around it.  The concept to fix it in code is simple, but implementing that in the code was a PITA, so I skipped it.  And it will be resolved when I finish the new wallet format (in 10 years, after I finish all my other priorities).

[TierNolan]

For that reason, if you generate, say, 500 addresses at a time without removing encryption (unlocking between each batch), you can probably get around it.  The concept to fix it in code is simple, but implementing that in the code was a PITA, so I skipped it.  And it will be resolved when I finish the new wallet format (in 10 years, after I finish all my other priorities).

So it actually stores the private keys in the wallet?

I tried creating a paper back while it was syncing to the network and got this.

Code:

2013-08-10 01:21 (INFO) -- ArmoryQt.py:1101 - Latest versions:
2013-08-10 01:21 (INFO) -- ArmoryQt.py:1102 -    Satoshi: 0.8.1
2013-08-10 01:21 (INFO) -- ArmoryQt.py:1103 -     Armory: 0.88
2013-08-10 01:21 (INFO) -- ArmoryQt.py:1112 - You are running the latest version!
2013-08-10 01:21 (INFO) -- ArmoryQt.py:4266 - Checking Satoshi version:
2013-08-10 01:21 (INFO) -- ArmoryQt.py:4267 -    Current:  801000
2013-08-10 01:21 (INFO) -- ArmoryQt.py:4268 -     Latest:  801000
2013-08-10 01:21 (INFO) -- ArmoryQt.py:4176 - Dashboard switched to "Scanning" mode
2013-08-10 01:21 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:User, State:ScanWithWallets
2013-08-10 01:21 (INFO) -- ArmoryQt.py:3541 - Switching Armory state text to Mgmt:Auto, State:NewUserInfo
2013-08-10 01:21 (INFO) -- ArmoryQt.py:3483 - Switching Armory functional mode to "Scanning"
2013-08-10 01:22 (ERROR) -- Traceback (most recent call last):
  File "qtdialogs.pyc", line 281, in acceptPassphrase
  File "armoryengine.pyc", line 9083, in unlock
  File "armoryengine.pyc", line 2660, in unlock
  File "armoryengine.pyc", line 2691, in unlock
KeyDataError: Stored public key does not match priv key!

2013-08-10 01:22 (INFO) -- armoryengine.pyc:11880 - Reading blockchain, pct complete: 2.9
2013-08-10 01:22 (ERROR) -- Traceback (most recent call last):
  File "qtdialogs.pyc", line 281, in acceptPassphrase
  File "armoryengine.pyc", line 9083, in unlock
  File "armoryengine.pyc", line 2660, in unlock
  File "armoryengine.pyc", line 2691, in unlock
KeyDataError: Stored public key does not match priv key!

2013-08-10 01:23 (ERROR) -- Traceback (most recent call last):
  File "qtdialogs.pyc", line 281, in acceptPassphrase
  File "armoryengine.pyc", line 9083, in unlock
  File "armoryengine.pyc", line 2660, in unlock
  File "armoryengine.pyc", line 2691, in unlock
KeyDataError: Stored public key does not match priv key!

2013-08-10 01:23 (ERROR) -- Traceback (most recent call last):
  File "qtdialogs.pyc", line 281, in acceptPassphrase
  File "armoryengine.pyc", line 9083, in unlock
  File "armoryengine.pyc", line 2660, in unlock
  File "armoryengine.pyc", line 2691, in unlock
KeyDataError: Stored public key does not match priv key!

This suggests actual corruption?

Anyway, your suggestion would be to restore from paper backup and make sure to only generate 500 at a time?  It already "knows" that there are 10k public keys.

[etotheipi]

Code:

2013-08-10 01:23 (ERROR) -- Traceback (most recent call last):
  File "qtdialogs.pyc", line 281, in acceptPassphrase
  File "armoryengine.pyc", line 9083, in unlock
  File "armoryengine.pyc", line 2660, in unlock
  File "armoryengine.pyc", line 2691, in unlock
KeyDataError: Stored public key does not match priv key!

This suggests actual corruption?

Anyway, your suggestion would be to restore from paper backup and make sure to only generate 500 at a time?  It already "knows" that there are 10k public keys.

Oh yeah, that's real wallet corruption!  I have code in there to attempt to avoid that (it updates two files sequentially, writing flags in between so it can determine where it failed writing and restore the good copy.  But I guess it's not 100% reliable).  I've only ever had a couple reports of corrupted wallets, so it can't be that bad, but I don't know if it helps...

Personally, to avoid a lengthy extended scan, I would delete the wallet, restart in offline mode, restore from paper backup, then generate 1000 addresses at a time.  Try viewing the paper backup after each batch of 1000 to unlock.  If 1000 takes too long, scale it down to 500.  Yeah, it's annoying.  

There's only two other ways:

(1) Temporarily remove wallet encryption before you generate the addresses.   It will write the unencrypted keys to disk, though.  Then re-encrypt when you're done.
(2) If you want to mess with the code, you could change the wallet-lock timeout from 10 seconds to something much much longer.  Then when you unlock the wallet, it should stay unlocked and generating the keys will be as if it's unencrypted.  No guarantees, but that might just work.  If you want to try it, change the "wlt.defaultKeyLifetime" for all wallets in armoryengine.py:6855.  Change it from 10 to something like 20*MINUTE.  You'll know it was effective if you unlock your wallet to view your paper backup, and then try again 5 minutes later and it doesn't ask you for your passphrase.

If you try #2, let me know if it works.  I think it will, but I've never tried it.

[TierNolan]

Oh yeah, that's real wallet corruption!  I have code in there to attempt to avoid that (it updates two files sequentially, writing flags in between so it can determine where it failed writing and restore the good copy.  But I guess it's not 100% reliable).  I've only ever had a couple reports of corrupted wallets, so it can't be that bad, but I don't know if it helps...

It probably happened because I will killing the process.

Personally, to avoid a lengthy extended scan, I would delete the wallet, restart in offline mode, restore from paper backup, then generate 1000 addresses at a time.  Try viewing the paper backup after each batch of 1000 to unlock.  If 1000 takes too long, scale it down to 500.  Yeah, it's annoying.  

I tried with 500 and it took 2 minutes.  At O(N^2), that would have taken 13 hours to compute the 10000, do in theory, I could have run it overnight.

(2) If you want to mess with the code, you could change the wallet-lock timeout from 10 seconds to something much much longer.  Then when you unlock the wallet, it should stay unlocked and generating the keys will be as if it's unencrypted.  No guarantees, but that might just work.  If you want to try it, change the "wlt.defaultKeyLifetime" for all wallets in armoryengine.py:6855.  Change it from 10 to something like 20*MINUTE.  You'll know it was effective if you unlock your wallet to view your paper backup, and then try again 5 minutes later and it doesn't ask you for your passphrase.

If you try #2, let me know if it works.  I think it will, but I've never tried it.

It didn't work.  Does the Armory.exe file actually use the python, or is it compiled into the exe?

Is the timeout built into the wallet file, so I would need to generate a new wallet after making the change?

A weird thing happened.  I decrypted the wallet to add the addresses.  I then tried to create a paper backup, and it asked for a pass-phrase.  The only one and blank didn't work.

It worked once I re-encrypted.