Armory - Discussion Thread

[Carlton Banks]

Then repeat this mantra: " VMWare does not turn my PC into the computing equivalent of Doctor Who's more-capacious-on-the-inside space ship telephone booth "

haven't tried it, but when you're just talking RAM, a virtual pc should be able to have 64 GB of ram, with the vm actually having it as a file somewhere, no?

i'm aware that it would be incredibly slow

I think this depends on your virtualisation software, certainly any version 8 or less VMWare software doesn't have a hypervisor swap-file scheme like this, but perhaps the newer releases do.

[chrisrico]

Then repeat this mantra: " VMWare does not turn my PC into the computing equivalent of Doctor Who's more-capacious-on-the-inside space ship telephone booth "

haven't tried it, but when you're just talking RAM, a virtual pc should be able to have 64 GB of ram, with the vm actually having it as a file somewhere, no?

i'm aware that it would be incredibly slow

When your computer needs more memory than it actually has, Windows uses a file on your hard drive as virtual memory (Unix uses a swap partition). As you state, it's incredibly slow. So no, using a VM doesn't benefit you at all.

[etotheipi]

Then repeat this mantra: " VMWare does not turn my PC into the computing equivalent of Doctor Who's more-capacious-on-the-inside space ship telephone booth "

haven't tried it, but when you're just talking RAM, a virtual pc should be able to have 64 GB of ram, with the vm actually having it as a file somewhere, no?

i'm aware that it would be incredibly slow

When your computer needs more memory than it actually has, Windows uses a file on your hard drive as virtual memory (Unix uses a swap partition). As you state, it's incredibly slow. So no, using a VM doesn't benefit you at all.

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

[chrisrico]

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

Can you run a 64 bit VM on 32 bit hardware? That doesn't seem right.

Anyway, the original question was whether setting up a VM would help alleviate the problem of only have 2GB of memory, as the VM could "have" 64GB. The answer is of course no.

[etotheipi]

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

Can you run a 64 bit VM on 32 bit hardware? That doesn't seem right.

Anyway, the original question was whether setting up a VM would help alleviate the problem of only have 2GB of memory, as the VM could "have" 64GB. The answer is of course no.

From stackoverflow:

You can't run a 64-bit VM session on a 32-bit processor. However, you can run a 64-bit VM session if you have a 64-bit processor but have installed a 32-bit host OS and your processor supports the right extensions.

It's my understanding that every desktop/laptop CPU under the sun is 64-bit by now, even though many people still run 32-bit OS.  So it is most likely possible to do this. 

Yes, slow as dirt.  Painfully slow.  But it might actually, eventually work, instead of seg-faulting when it hits 94%.

[cypherdoc]

Bug:

Alienware laptop, 32 bit OS, Ubuntu 12.10, 16 GB ram, I7 1st gen.

Armory  0.88.1 launches fine but self crashes after about 5 minutes.

[Roy Badami]

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

Can you run a 64 bit VM on 32 bit hardware? That doesn't seem right.

Anyway, the original question was whether setting up a VM would help alleviate the problem of only have 2GB of memory, as the VM could "have" 64GB. The answer is of course no.

From stackoverflow:

You can't run a 64-bit VM session on a 32-bit processor. However, you can run a 64-bit VM session if you have a 64-bit processor but have installed a 32-bit host OS and your processor supports the right extensions.

It's my understanding that every desktop/laptop CPU under the sun is 64-bit by now, even though many people still run 32-bit OS.  So it is most likely possible to do this. 

Yes, slow as dirt.  Painfully slow.  But it might actually, eventually work, instead of seg-faulting when it hits 94%.

There's a bit more to it than that.  To run 64-bit VMs your CPU also needs to support Intel VT (Virtualization Technology) or the AMD equivalent, and VT has to be enabled.  Most CPUs do support VT, but I believe Intel doesn't include it on some of their low-end CPUs.

But having a VT-capable CPU isn't enough - VT needs to be enabled, and this is normally done by the BIOS.  Often VT isn't enabled by default, but there is a BIOS setting to enable it.  But if your computer manufacturer chose not to provide a means of enabling VT then you may be out of luck, even if your CPU has the requisite capabilities.

roy

[etotheipi]

fsck yeah!


Balances are now pulled from the DB without rescanning!

I still have plenty left to do (like testing zero-conf tx that never touch the DB), and stop/start/restart/crash DB operations.  And of course, figure out that stupid LevelDB+Windows disaster...

(for instance, I just noticed that when you a receive a tx, it only updates the wallet, not the main ledger, and if a new block comes it that doesn't include it [yet] it disappears)

[etotheipi]

Bug:

Alienware laptop, 32 bit OS, Ubuntu 12.10, 16 GB ram, I7 1st gen.

Armory  0.88.1 launches fine but self crashes after about 5 minutes.

(Cypher sent me more details over email)  That crash is happening in a location I've never seen before (isMineBulkFilter).  Can you export a log file and email that to me?  The only thing that comes to mind that would cause that is:

(1) Corrupted wallet, leading to bad address string which is crashing the code that identifies when tx belong to your wallet
(2) Malformed/corrupt blockdata in your blk*.dat files (does this happen during scanning or after scanning is done?)

I'll tell you more when I see the log file.

[cypherdoc]

Bug:

Alienware laptop, 32 bit OS, Ubuntu 12.10, 16 GB ram, I7 1st gen.

Armory  0.88.1 launches fine but self crashes after about 5 minutes.

(Cypher sent me more details over email)  That crash is happening in a location I've never seen before (isMineBulkFilter).  Can you export a log file and email that to me?  The only thing that comes to mind that would cause that is:

(1) Corrupted wallet, leading to bad address string which is crashing the code that identifies when tx belong to your wallet
(2) Malformed/corrupt blockdata in your blk*.dat files (does this happen during scanning or after scanning is done?)

I'll tell you more when I see the log file.

happening after successfully scanning chain.  sent you an email.

[halfawake]

fsck yeah!


Balances are now pulled from the DB without rescanning!

I still have plenty left to do (like testing zero-conf tx that never touch the DB), and stop/start/restart/crash DB operations.  And of course, figure out that stupid LevelDB+Windows disaster...

(for instance, I just noticed that when you a receive a tx, it only updates the wallet, not the main ledger, and if a new block comes it that doesn't include it [yet] it disappears)

You're pulling the balances from the DB now without rescanning?  That's great news, sounds like it'd be much more performance efficient than the current technique.  Good work.

[etotheipi]

fsck yeah!
...
Balances are now pulled from the DB without rescanning!

I still have plenty left to do (like testing zero-conf tx that never touch the DB), and stop/start/restart/crash DB operations.  And of course, figure out that stupid LevelDB+Windows disaster...

(for instance, I just noticed that when you a receive a tx, it only updates the wallet, not the main ledger, and if a new block comes it that doesn't include it [yet] it disappears)

You're pulling the balances from the DB now without rescanning?  That's great news, sounds like it'd be much more performance efficient than the current technique.  Good work.

Indeed, the new design constructs a master database that hold and tracks all balances of all addresses, though it will be upgraded in the future to hold a much lighter subset of that after this version is stable.  The goal, of course, is to avoid the constant rescanning and the memory usage of the current version which stores just about nothing on disk except your wallet.

[etotheipi]

Anyone in Linux want to help?  It's the ramreduceleveldb branch.   I know there's a variety of loose ends with the current implementation, but it wouldn't be a bad idea to get a few people trying it out and, especially get the database built so they'll be ready to test the more-stable version.  But there's still a ton of stuff that appears to work, and so I'd like people dig into the parts of the application that I haven't gotten to yet.  Like I just realized I've done minimal testing of importing and sweeping.  

I know that zero-confirmation transactions are a mess.   They disappear when included in a block, and only reappear on a restart (but that's pretty darned fast, now).   And the timestamps on the C++ logging (new!) are broken and spit out garbage where it would print the current time.  There may also be some infrastructure left in for "The blockchain needs to be rescanned, do you want to go offline for a few minutes to do this?", even though the "rescan" is instantaneous now.  

If you do help run/test, please take note of the following:

(1) What extra packages did you have to install?  I think libleveldb is one, which will install boost.  Looking for the minimal apt-get install packages that will fetch everything else needed to upgrade.
(2) How long does it take to build the DB?  NOTE: there's actually two steps to building the DB: one is writing the raw blocks to the DB, the other is actually walking through those blocks and updating the DB with spentness info, and building histories for each address.  The system currently does not give you any real indication that the first step is happening (though it should on the command line, if you run with --debug).  Until I fix that, it may not be obvious that anything is happening... and that first step usually takes like 45 min!  
(3) How much space does the new .armory/databases directory use?  I've seen widely different results, and I think it has to do with seemingly-random times that leveldb decides to do DB compactions.  My first test left me with an 18 GB databases dir.  A more recent had it at 25 GB.  I'm wondering how widely this varies.
(4) This is an excellent opportunity to try out the new backup center.  This version already has all the backup stuff merged, so you can try out fragmented backups and SecurePrint.
(5) Just take lots of notes.  Feel free to only use it on testnet, or use it on mainnet but run it with "--datadir=testdir" which will leave your current wallets and settings untouched.


Almost there!

EDIT:  I think I completely fixed the zero-conf transactions in the ramreduceleveldb.

[etotheipi]

Update:

the new version of Armory seems to run flawlessly on testnet.  I accidentally deleted my mainnet database when trying to delete the testnet database, so I have to rebuild before I can do any testing on mainnet.   But so far, I don't see anything that doesn't work.  But there's also so many little nooks and crannies, there's no way I can test everything exhaustively.  Help me out!

Reminder: poor interface-user communication:  there's actually two steps to building the DB: one is writing the raw blocks to the DB, the other is actually walking through those blocks and updating the DB with spentness info.  The system currently does not give you any real indication that the first step is happening (though it should on the command line, if you run with --debug).  Until I fix that, it may not be obvious that anything is happening... and that first step usually takes like 45 min!  After that, you should get a fairly accurate progress bar in the GUI for the next 3-4 hours of updating the DB.

[justusranvier]

I built the ramreduceleveldb branch, and I started it on mainnet with --debug, and I still don't see any indication that something is happening. It's completed the initial blockchain scan, and there is no ~/.armory/databases directory yet.

[etotheipi]

I built the ramreduceleveldb branch, and I started it on mainnet with --debug, and I still don't see any indication that something is happening. It's completed the initial blockchain scan, and there is no ~/.armory/databases directory yet.

The log file you sent indicates you're still on version 0.88.2.  But if you were on the ramreduceleveldb branch, it would say 0.89.95.

After battling multiple issues destroying mainnet DB (all user error), I did notice that a bunch of my wallet balances were totally wrong on mainnet, but not testnet (frequently reporting "total received", not "current balance").   I confirmed that the DB actually has the correct data, and the GUI just happens to be reading it incorrectly.  Not sure why I'm not seeing the problem on testnet, but at least it'll be easier than tracking down a DB error...

[etotheipi]

ARGH.  I somehow reverted one armorymodels.py to the pre-leveldb version.   Just about to commit the fixed version to the ramreduceleveldb branch.

[superbit]

When I loaded armory this AM, it said it detected a corrupt block in the blockchain.

I closed armory, opened bitcoin 0.8.4 and it offered to repair the chain.  I let this run through and it completed giving me the green checkmark in the bottom right corner.  Then as soon as I went back into armory I got the same error, now I go back into the bitcoin client and it is starting the repair again??

I really need to send coins in my armory wallet out today, what can I do to end this loop?

[etotheipi]

When I loaded armory this AM, it said it detected a corrupt block in the blockchain.

I closed armory, opened bitcoin 0.8.4 and it offered to repair the chain.  I let this run through and it completed giving me the green checkmark in the bottom right corner.  Then as soon as I went back into armory I got the same error, now I go back into the bitcoin client and it is starting the repair again??

I really need to send coins in my armory wallet out today, what can I do to end this loop?

Unfortunately that is actually a bitcoin-qt problem that cropped up today.  From the mailing list:  prepare for some turbulence.

All Bitcoin-qt / Bitcoind nodes will currently fail to come back up
after a restart, reporting

": *** coin database inconsistencies found"
and
"Do you want to rebuild the block database now?"

Reindexing _will not_ correct the problem.  In Bitcoin-qt you should
say no to this reindex prompt as it will not help for this problem and
will only waste your time.

To workaround:

Please specify the command-line or configuration file argument
-checklevel=2  to Bitcoind or Bitcoin-qt.

You'll have to turn off auto-bitcoind in the Armory preferences and run it yourself using the checklevel command.  I did that today and it worked fine.

[superbit]

When I loaded armory this AM, it said it detected a corrupt block in the blockchain.

I closed armory, opened bitcoin 0.8.4 and it offered to repair the chain.  I let this run through and it completed giving me the green checkmark in the bottom right corner.  Then as soon as I went back into armory I got the same error, now I go back into the bitcoin client and it is starting the repair again??

I really need to send coins in my armory wallet out today, what can I do to end this loop?

Unfortunately that is actually a bitcoin-qt problem that cropped up today.  From the mailing list:  prepare for some turbulence.

All Bitcoin-qt / Bitcoind nodes will currently fail to come back up
after a restart, reporting

": *** coin database inconsistencies found"
and
"Do you want to rebuild the block database now?"

Reindexing _will not_ correct the problem.  In Bitcoin-qt you should
say no to this reindex prompt as it will not help for this problem and
will only waste your time.

To workaround:

Please specify the command-line or configuration file argument
-checklevel=2  to Bitcoind or Bitcoin-qt.

You'll have to turn off auto-bitcoind in the Armory preferences and run it yourself using the checklevel command.  I did that today and it worked fine.

Please explain what I need to do in the check level command or how this is done?