Update:

the new version of Armory seems to run flawlessly on testnet.  I accidentally deleted my mainnet database when trying to delete the testnet database, so I have to rebuild before I can do any testing on mainnet.   But so far, I don't see anything that doesn't work.  But there's also so many little nooks and crannies, there's no way I can test everything exhaustively.  Help me out!

Reminder: poor interface-user communication:  there's actually two steps to building the DB: one is writing the raw blocks to the DB, the other is actually walking through those blocks and updating the DB with spentness info.  The system currently does not give you any real indication that the first step is happening (though it should on the command line, if you run with --debug).  Until I fix that, it may not be obvious that anything is happening... and that first step usually takes like 45 min!  After that, you should get a fairly accurate progress bar in the GUI for the next 3-4 hours of updating the DB.

Anyone in Linux want to help?  It's the ramreduceleveldb branch.   I know there's a variety of loose ends with the current implementation, but it wouldn't be a bad idea to get a few people trying it out and, especially get the database built so they'll be ready to test the more-stable version.  But there's still a ton of stuff that appears to work, and so I'd like people dig into the parts of the application that I haven't gotten to yet.  Like I just realized I've done minimal testing of importing and sweeping.  

I know that zero-confirmation transactions are a mess.   They disappear when included in a block, and only reappear on a restart (but that's pretty darned fast, now).   And the timestamps on the C++ logging (new!) are broken and spit out garbage where it would print the current time.  There may also be some infrastructure left in for "The blockchain needs to be rescanned, do you want to go offline for a few minutes to do this?", even though the "rescan" is instantaneous now.  

If you do help run/test, please take note of the following:

(1) What extra packages did you have to install?  I think libleveldb is one, which will install boost.  Looking for the minimal apt-get install packages that will fetch everything else needed to upgrade.
(2) How long does it take to build the DB?  NOTE: there's actually two steps to building the DB: one is writing the raw blocks to the DB, the other is actually walking through those blocks and updating the DB with spentness info, and building histories for each address.  The system currently does not give you any real indication that the first step is happening (though it should on the command line, if you run with --debug).  Until I fix that, it may not be obvious that anything is happening... and that first step usually takes like 45 min!  
(3) How much space does the new .armory/databases directory use?  I've seen widely different results, and I think it has to do with seemingly-random times that leveldb decides to do DB compactions.  My first test left me with an 18 GB databases dir.  A more recent had it at 25 GB.  I'm wondering how widely this varies.
(4) This is an excellent opportunity to try out the new backup center.  This version already has all the backup stuff merged, so you can try out fragmented backups and SecurePrint.
(5) Just take lots of notes.  Feel free to only use it on testnet, or use it on mainnet but run it with "--datadir=testdir" which will leave your current wallets and settings untouched.


Almost there!

EDIT:  I think I completely fixed the zero-conf transactions in the ramreduceleveldb.

Indeed, the new design constructs a master database that hold and tracks all balances of all addresses, though it will be upgraded in the future to hold a much lighter subset of that after this version is stable.  The goal, of course, is to avoid the constant rescanning and the memory usage of the current version which stores just about nothing on disk except your wallet.

(Cypher sent me more details over email)  That crash is happening in a location I've never seen before (isMineBulkFilter).  Can you export a log file and email that to me?  The only thing that comes to mind that would cause that is:

(1) Corrupted wallet, leading to bad address string which is crashing the code that identifies when tx belong to your wallet
(2) Malformed/corrupt blockdata in your blk*.dat files (does this happen during scanning or after scanning is done?)

I'll tell you more when I see the log file.

fsck yeah!


Balances are now pulled from the DB without rescanning!

I still have plenty left to do (like testing zero-conf tx that never touch the DB), and stop/start/restart/crash DB operations.  And of course, figure out that stupid LevelDB+Windows disaster...

(for instance, I just noticed that when you a receive a tx, it only updates the wallet, not the main ledger, and if a new block comes it that doesn't include it [yet] it disappears)

In case you don't follow slashdot, there was an article today from John Gilmore about obstructionism by NSA folks he's seen over the years:

http://www.mail-archive.com/cryptography@metzdowd.com/msg12325.html

It's mostly about simple obstruction of implementing standardized security systems.  But it wouldn't surprise me if the same folks also attempted some more subtle things, like trying to inject pre-selected "random" constants into crypto systems.  But I'm not all that familiar with the processes they use or how easy that would be.

From stackoverflow:


It's my understanding that every desktop/laptop CPU under the sun is 64-bit by now, even though many people still run 32-bit OS.  So it is most likely possible to do this. 

Yes, slow as dirt.  Painfully slow.  But it might actually, eventually work, instead of seg-faulting when it hits 94%.

I'm not 100% convinced of that.  The issue is not slowness (well, it is an issue), but that these systems may be running out of RAM.  For instance, it just can't be run on 32-bit because of the lack of address space.  But if you virtualize a system that runs 64-bit OS with 16 GB of RAM (using 14 GB of disk), it may be possible to actually run it even though it takes 3 hours to sync. 

I'd be interested to see someone try it, though I would bet 2:1 that it still doesn't work at all.  But I can see why it might work.

People like to take the idea "hash collisions are theoretically possible!" and pretend like they could actually happen and that something in life should accommodate that possibility.  If you have a solid hash function (which SHA256 is) and you come across a collision, then either:

(1) SHA256 is broken
(2) You hashed two things that were identical

End of story.  There's about as many different SHA256 hash values as there are atoms in the universe.  From the perspective of a human, a proper hash function that outputs more than 128 bits do not have collisions.  They won't even happen in the future due to increasing computational speed -- Bruce Schneier showed that the thermodynamic lower-bound of energy to find such a collision is many billion times more energy than the sun contains. 

That is actually the plan.  Or rather, when I talk about "super-node-lite-node split" that's what I'm talking about.  The lite version will be the same thing as the super version, but its calls to the DB engine will instead go over a socket to request it from another process or system.

It's a pretty big jump from where I currently am, but it's already part of my long-term plans

P.S. - ACK, I totally forgot something!  This only works in Linux!  I've taken some small stabs at getting it to work in Windows, but leveldb is a complete b**** in Windows.  Luckily, the core devs have done a lot of LevelDB stuff in Windows already, which I/we may be able to cannibalize, but there's no question it's going to be painful.

So when I talk about "testing this weekend", I'm most definitely talking about Linux testing.  If you're not on Linux, I won't have anything for you yet.

I will offer a 2 BTC bounty to anyone who can get this new stuff integrated and compiling in MSVS 2008 (it must be 2008, for now).  If it's too difficult, it may be possible to do it mingw, though Armory project itself would have to be ported to mingw and I'm not too keen on that.  But at the moment, any solution is a sufficient one.  It is probably possible to compile the leveldb & boost stuff into a .dll and access it from the existing MSVS project with a lot less pain (I recommend that because the core devs got it working using mingw but I really need to run the Armory C++ utilities in MSVS for debugging).

If you want to try this, you must follow the directions on the Building Armory from Source page, and get it working with the master branch.  You should be able to compile the C++ utilities in MSVS 2008 and run Armory off the compiled product.  Then switch to the ramreduceleveldb branch and... have fun.  If you can compile BlockUtils.cpp on the ramreduceleveldb branch in MSVS, then I can figure out the rest from there.

PLEASE document your process if you try it.  I've gotta understand how to integrate leveldb into my project.  Knowing that the core devs had gotten it working in Windows was sufficient for me to know I can get it working in Windows, but I forgot I still have to go through that process!

Well, it's not done yet.  But those last two tests were the most exciting, because they demonstrate bootstrapping the old wallet system with the database data.  i.e. I actually left most of the thoroughly-tested, thoroughly-complex wallet infrastructure in place, and simply seed it with data from the database and it works smoothly.  That means that most of the GUI will work without modification (except for changing some function names).

I'm going to start trying to run the GUI with the new DB stuff and see what's still missing or what needs to be tweaked.  Before I'm totally done with the full upgrade, I might ask people to run it on mainnet but with a different home dir, to let me know how the initial DB build goes.  I've had mixed results with it -- the last test I did took 6 hours to build, but I've had a lot that took 3 hours.  I could use some extra data points on that one (by the way, I have one more optimization to put in place that should cut down that time even more, but at the moment I consider it "usable").

I'll let you know when I have something.  

Just another update, because I know people are anxious, and I'm making substantial progress.  I'll probably also need people to help with some testing this weekend


I don't have all the reliability features in there.  Meaning, this should probably work 98% of the time, but it really needs to be more like 99.999%.  Things like DB/Armory crashes mid-build or while on a block that is eventually orphaned, etc, are not well-tested.  But I gotta start somewhere...

P.S. - Did I mention how much I love GoogleTest?  it's made my life sooooo much easier...

If the malware is advanced/integrated enough to record mouseclicks, screenshots, and implement OCR to identify the characters being clicked, it's advanced enough to pull your private keys out of RAM when you unlock your wallet.  At that point, the only thing that will protect you is to use an offline computer.

This gives you a little extra protection against "simple" keyloggers, but nothing more.  I haven't looked too much at that website, but it sounds like snake oil.  Your keystrokes have to make it to the application decrypted at some point, so you're probably removing only the simplest of keyloggers with that 3rd party tool anyway.  WHich is what this scrambled keyboard does.

By the way, Armory has anti-keylogging option for entering your password.

https://bitcointalk.org/index.php?topic=56424.msg1761230#msg1761230

It simply allows you to use the mouse to click your password, and it randomizes the key layout so that something recording mouse clicks doesn't know what's being clicked.  It's not a silver bullet against anything, but it's built-in and not any worse than using the keyboard.

Are you using the latest version of Armory (0.88.1)?  It has options in File->Preferences, but not in the previous versions.  If you are running a previous version, you have to use " --satoshi-datadir=/path/to/blocksdir" as a command-line argument when you run Armory (change the target of the shortcut on your desktop, or try running from cmd window).  You can use the same cmd arguments with the newer version, too, but the preferences window now gives you the option.

Kind of.  At the moment there's a very minimal set of operations supported, and it will crash whenever there's a Bitcion-Qt/bitcoind blockfile split (just like the GUI does).  If you don't mind restarting and rescanning every couple days, and only the need to super-limited functionality it has -- yes, it's usable.

Right now it basically can only get you new addresses from wallets, and if it's in online mode, it can "listtransactions" and "getledger".  The listtransactions function makes an attempt to replicate what Bitcoin-Qt produces in response to the same command, but I didn't like what it produces.  It seems... sloppy.  So I implement "getledger" and "getledger_simple" which returns basically the same thing that you see on the "Transactions" tab of the GUI.  If you add up all the items returned by getledger, you will get your current balance without, and all items returned are non-overlapping.  The same cannot be said for "listtransactions."

I had started implementing more functionality, but decided it wasn't worth it until I got the new database stuff done which will make it stable.  A lot of the use cases for it require it not to crash every few days, and my efforts to fix the crash-on-blockfile-split were futile (I even made what I thought was a stressful unit-test for it, which passed, but it still segfaults when run on the network).  But I'm confident that will go away with the new DB stuff.

Yup.  And at least you get something useful out of all that extra storage space:  you can get the entire history for any address (including multisig TxOuts) nearly instantaneously.  This will make armoryd.py quite a bit more powerful than bitcoind, since it will ultimately implement everything bitcoind does plus all the arbitrary addr/script lookups.  It also means that you can import addresses and wallets without having to rescan.

And with that, it may make sense to implement a basic python shell into the expert interface, that allows users to issue commands/requests directly from a "debug" window.  For instance, you'd run "Server Version" GUI (not armoryd.py), and open the "Armory CLI" window, and type in "getAddressHistory('<addr>')", "importAddress('<addr>', '<wallet>')", "findAllMultisig('<addr>')", etc.  I could make the same interface available through RPC, but that will take considerably longer because of how security-sensitive that is.

This super-node version also enables a very specific functionality that is not otherwise possible:  encrypted watching-only wallets.  Without a full address index, and without rescanning the blockchain every load, there's no way to hide the addresses in your watching-only wallet.  My new wallet design (which was side-tracked by this RAM-reduction effort) actually has a way to encrypt the public keys in a WO wallet, but it's not very useful if your database is storing only the transactions related to your wallets.  There's things you could do to improve the situation, but it won't be perfect unless you do a rescan every time you unlock your wallet and hold everything in RAM.  Or save the whole tx history as this super-node version is doing.  

So if you want absolute security and privacy, you can run this "server version" which will use a lot of HDD space, but it will run smoothly once the DB is built and you can hide the public keys in your wallet.  A lot of people have requested this functionality, because of concerns that a stolen laptop could lead to worst things than losing money.  For instance: the thief may not get your $100k worth of Bitcoin since he only has your WO wallet, but he can see that you have $100k worth of Bitcoin and can probably find your address somewhere else on the computer.   Many of my more paranoid/rich users would like to avoid that.

Profiling Update: my initial DB build is up to 221,000 blocks after 1.5 hours, and using 295 MB RAM (15 MB more than at 180k blocks).  So it looks like it may take about 3-4 hours to rebuild the whole DB and about 300 MB RAM.  Those are some solid, usable numbers!

Not only that, but it's very likely that I can build the initial database while Bitcoin-Qt is downloading the initial transaction history, meaning that Armory's DB building can be "hidden" and the whole new-user-startup may not be any worse than running vanilla Bitcoin-Qt.  I will not complicate it with that parallelization right now, but those numbers are very encouraging for a future upgrade.

Okay, enough rambling, more coding ...

Alright!  Back to where I was 10 days ago...



The difference is that now the DB operations are much better optimized for handling excessively-reused addresses like 1VayNert and all the SatoshiDice buckets (at the expense of a ton of extra DB complexity).  I'm in the middle of profiling the new stuff, but the dramatic improvement is immediately obvious:  My previous design was bottlenecked in RAM and speed by the address being updated with the most historical transactions.  Now processing time is fairly independent of that, and only dependent on blockchain size in MB.

So far, the DB itself is a bit bigger than I expected (about 1.7x the .bitcoin/blocks directory).  But that's because this is running in "super-node" mode which tracks the entire history of all addresses/scripts, which could be more appropriately named "Armory - Server Version".  But it was easier to implement than the "regular" version which needs extra logic to reliably track only a specific set of addresses (i.e. your wallets).  When this version is done, I will implement the "regular" mode which use less space and even less RAM, and may be able to work with a remote bitcoind/bitcoin-qt instance.  For now, you'll need a lot of disk space to run the new version, but RAM should no longer be a constraint.

And in case you're wondering about the RAM: it looks good (finally!).  As I type this, my testing run is already deep into SatoshiDice territory and it's been using only 276-279 MB of RAM the whole time.  And this should be the most intense part of the whole process.  It might go up a little bit once I create wallets, etc, but will also go down when it's done batching and updating the DB with thousands of tx per second.

Of course plenty of work left to do, but progress is being made!

If you want to examine the tx directly, you can load the tx into armoryengine and do something like this:


You'll get a higher-level overview of what's in the tx from the first pprint command, and then a hex breakout of the raw tx from the second command (if you feel like digging into the raw tx bytemap).  It's not pleasant, but if broadcasting is failing, it's probably invalid and you need to figure out why.