Does Armory take a long time to synch up for you when you start it?

[TonyT]

Does Armory take a long time to synch up for you when you start it?  In my system [1], it takes a minimum of five minutes of disk thrashing at 100% and sometimes up to 30 minutes (if other programs are running).  In the latter case, the system nearly freezes up and is not usable during that time.

I am thinking of either getting an SSD hard drive to help fix this problem, or maybe abandoning bitcoin altogether, except for keeping a few coins in an online wallet.  IMO the purpose of Bitcoin is to help preserve anonymity while you make purchases, which is defeated by an online wallet.

TonyT

[1]   i5 -quad core, 32 bit OS Windows 8.1, 4 GB RAM, a 3 year old WD hard drive that gets Transfer Rate Average : 112.0 MB/sec (decent for a mechanical hard drive), internet is a Ethernet land line that gets 1.5 MBps (acceptable); latest copy of Armory running Bitcoin Core (latest version)

[etotheipi]

This is likely to get much better soon with the new version of Core (headers-first) announced recently.  Much of the startup time is not Armory's fault, it's the fact that Bitcoin core has a terrible system for "catching up" with the blockchain.  It's easy to get stuck behind one very slow peer and take 20 minutes to download a few megabytes.  There isn't anything Armory can do about that.

But the new headers-first Core should solve those problems, and allow you to synchronize and catch-up as fast at your full download speed.  The only time you would wait for Armory is when you need to rescan, but that should be rare these days, and will be even better in the next major release.

[TonyT]

This is likely to get much better soon with the new version of Core (headers-first) announced recently.  Much of the startup time is not Armory's fault, it's the fact that Bitcoin core has a terrible system for "catching up" with the blockchain.  It's easy to get stuck behind one very slow peer and take 20 minutes to download a few megabytes.  There isn't anything Armory can do about that.

But the new headers-first Core should solve those problems, and allow you to synchronize and catch-up as fast at your full download speed.  The only time you would wait for Armory is when you need to rescan, but that should be rare these days, and will be even better in the next major release.

I hope so!  I was about to give up on bitcoin because of this issue.

I see you are a developer, and not to get too far off topic (I also code, but not as an employee), I see Armory is: " C++ code... Python code", which makes sense, as C++ is very popular with those that like granularity and Python is supposedly good for I/O (I use C# myself for both).  What is the 'typical' Armor programmer resume?  How many years of experience doing peer-to-peer stuff do you need?  What are good books on peer-to-peer (I once built a simple p-to-p using C# using 'delegates')?  I find it's hard to find online tutorials for peer-to-peer, and just now I see I'm not the only one who thinks this way (http://stackoverflow.com/questions/23738/why-is-peer-to-peer-programming-a-hard-topic-to-obtain-good-research-for )

TonyT

[etotheipi]

This is likely to get much better soon with the new version of Core (headers-first) announced recently.  Much of the startup time is not Armory's fault, it's the fact that Bitcoin core has a terrible system for "catching up" with the blockchain.  It's easy to get stuck behind one very slow peer and take 20 minutes to download a few megabytes.  There isn't anything Armory can do about that.

But the new headers-first Core should solve those problems, and allow you to synchronize and catch-up as fast at your full download speed.  The only time you would wait for Armory is when you need to rescan, but that should be rare these days, and will be even better in the next major release.

I hope so!  I was about to give up on bitcoin because of this issue.

I see you are a developer, and not to get too far off topic (I also code, but not as an employee), I see Armory is: " C++ code... Python code", which makes sense, as C++ is very popular with those that like granularity and Python is supposedly good for I/O (I use C# myself for both).  What is the 'typical' Armor programmer resume?  How many years of experience doing peer-to-peer stuff do you need?  What are good books on peer-to-peer (I once built a simple p-to-p using C# using 'delegates')?  I find it's hard to find online tutorials for peer-to-peer, and just now I see I'm not the only one who thinks this way (http://stackoverflow.com/questions/23738/why-is-peer-to-peer-programming-a-hard-topic-to-obtain-good-research-for )

TonyT

Armory uses Bitcoin Core in the background so that we don't have to worry about the P2P stuff.  I mean, we have to understand Bitcoin network/protocol security, but we specifically use Core because Core is developed by a ton of people who've made it their job to harden all that stuff in all dimensions, better than we ever could.  As far as we're concerned, it's our obligation to use Core as our gateway to the network, if we want to be as secure as possible.

Armory uses a mix of Python & C++.  C++ is much more rigorous, low-level, and optimizable.  Just about any heavy-lifting is done by C++ (like scanning the 25 GB blockchain and computing balances).  But for application development, C++ isn't  fluid enough for us.  Python is extremely flexible and makes it easy to create complex interfaces, handle corner cases, threading, exceptions, and has a billion libraries available that save you time instead of reimplementing it yourself.  C++ development can be slow, and the syntax is rigorous and inflexible (for instance, it's very nice to be able to create python functions with a dozen args, set defaults, and then only specify the ones you want on a given call, you can't do that in C++).

We use a tool called SWIG which allows you to compile C++ code into a DLL and creates a wrapper to make it accessible via python.  It's actually quite magical, because python and C++ are kind of polar opposites, with a huge diversity of strengths and weaknesses.  Being able to combine them with SWIG means that you can avoid all the weaknesses and write new code on whichever side of the python/C++ fence has the strengths you need.

[TonyT]

Very interesting, thanks.  Now here's another question:  when Armory loads up it always takes the same amount of time even if it has already been loaded and exited from!  That is, if you load Armory, let it do "Initialization of Database Engine", "Build Database", "Scan Transactions", then exit Armory, then one second later reload Armory, it takes the same amount of time, like 10 minutes, before it goes online.  This is a feature or bug?  It should not take so long the second time it is loaded since the data downloaded is already in place?

BTW, below is something of interest that was mentioned earlier (new version of Core):  I see that the chief scientist Gavin Andresen made his blog post Oct 6, 2014 yet it seems the "headers first" approach has in fact "won the endorsement" of the "Powers That Be" in less than three weeks--very interesting.  I'd like to know why they picked this approach, perhaps it is the easiest to code (but I see Pieter Wuille was pushing for this, and he seems to be their main man, so that's probably why)?  Regardless, how long before they come out with a stable version of Core using Headers First?  It might be several months?  Maybe a year?  Time will tell but I bet this is not a trivial change to make in Core?  

TonyT


A Scalability Roadmap
by Gavin Andresen, Chief Scientist, October 6, 2014

https://bitcoinfoundation.org/2014/10/a-scalability-roadmap/

Pieter Wuille has been hard at work on a “headers first” approach – downloading the longest chain of 80-byte block headers, which is only 25 megabytes of data. The headers are sufficient to know whether or not you have the best chain, and once your node has the headers it can “back fill” by requesting complete blocks from multiple peers in any order it wants, similar to how BitTorrent downloads chunks of a large file from many different peers at once.

You might be surprised that old blocks aren’t needed to validate new transactions. Pieter Wuille re-architected Bitcoin Core a few releases ago so that all of the data needed to validate transactions is kept in a “UTXO” (unspent transaction output) database. The amount of  historical data needed that absolutely must be stored depends on the plausible depth of a blockchain reorganization. The longest reorganization ever experienced on the main network was 24 blocks during the infamous March 11, 2013 chain fork.

[TonyT]

I'm starting to wonder if you are a concern troll?

What is your value to this thread Holliday?  Seems you are projecting your own inadequacies on me.  I am a noob trying to get more information; what are you, a vigilante moderator?  If you are more experienced with bitcoin, let's hear your solution ("but I'm starting to wonder if the OP even cares about solutions").  It's not easy to code P2P (though I've done it, by contrast, by your own admission in another thread, you don't know much about technology), so your suggestion of "do it yourself" is just plain stupid.  What is the purpose of your several posts in this thread?

[doug_armory]

We use a tool called SWIG which allows you to compile C++ code into a DLL and creates a wrapper to make it accessible via python.  It's actually quite magical, because python and C++ are kind of polar opposites, with a huge diversity of strengths and weaknesses.  Being able to combine them with SWIG means that you can avoid all the weaknesses and write new code on whichever side of the python/C++ fence has the strengths you need.

We also use SIP, which is like SWIG, to handle a small amount of OS X-specific Objective-C code. (I don't think this code has been released yet but you can see it on the dev branch on Github.) The Objective-C code isn't required. It's just stuff that was added to handle a couple of small features where the OS X build required Objective-C to enable the features.

[TonyT]

We use a tool called SWIG which allows you to compile C++ code into a DLL and creates a wrapper to make it accessible via python.  It's actually quite magical, because python and C++ are kind of polar opposites, with a huge diversity of strengths and weaknesses.  Being able to combine them with SWIG means that you can avoid all the weaknesses and write new code on whichever side of the python/C++ fence has the strengths you need.

We also use SIP, which is like SWIG, to handle a small amount of OS X-specific Objective-C code. (I don't think this code has been released yet but you can see it on the dev branch on Github.) The Objective-C code isn't required. It's just stuff that was added to handle a couple of small features where the OS X build required Objective-C to enable the features.

Thanks Doug that's very interesting.  I see indeed Objective-C is used in OS X (did not know that) and it has automatic garbage collection, which I don't think C++ or Python has.  I did not notice any memory leaks in the 44 hours it took to install and initialize Armory on my PC, so you guys did a good job making the code robust (whenever I used to code in C++ years ago, in a big project there always seemed to be some memory leak or orphaned pointers, that's why I switched to C# which supports automatic garbage collection).

If you read this, I'm curious whether all the Armory developers have ever met each other face to face?  I bet not.

[doug_armory]

We use a tool called SWIG which allows you to compile C++ code into a DLL and creates a wrapper to make it accessible via python.  It's actually quite magical, because python and C++ are kind of polar opposites, with a huge diversity of strengths and weaknesses.  Being able to combine them with SWIG means that you can avoid all the weaknesses and write new code on whichever side of the python/C++ fence has the strengths you need.

We also use SIP, which is like SWIG, to handle a small amount of OS X-specific Objective-C code. (I don't think this code has been released yet but you can see it on the dev branch on Github.) The Objective-C code isn't required. It's just stuff that was added to handle a couple of small features where the OS X build required Objective-C to enable the features.

Thanks Doug that's very interesting.  I see indeed Objective-C is used in OS X (did not know that) and it has automatic garbage collection, which I don't think C++ or Python has.  I did not notice any memory leaks in the 44 hours it took to install and initialize Armory on my PC, so you guys did a good job making the code robust (whenever I used to code in C++ years ago, in a big project there always seemed to be some memory leak or orphaned pointers, that's why I switched to C# which supports automatic garbage collection).

To be fair, the Objective-C code was added by me when I got annoyed that the OS X build isn't quite at feature parity with the other builds. I found two things that require Objective-C in order to achieve parity. So, I went through the process of adding all this stuff. (It took awhile, but hey, I'm now much more familiar with the Qt build process, and other code won't take nearly as long.) Automativ garbage collection and such really doesn't apply to general Armory use on OS X.

As for there not being any memory leaks, it's a combination of thorough testing and luck. Don't worry, the next release will have lots of new code to pore over and search for memory leaks.

If you read this, I'm curious whether all the Armory developers have ever met each other face to face?  I bet not.

Indeed we have. We were at the Inside Bitcoins conference in New York last April. We're trying to figure out where we'll meet next.

[picobit]

Very interesting, thanks.  Now here's another question:  when Armory loads up it always takes the same amount of time even if it has already been loaded and exited from!  That is, if you load Armory, let it do "Initialization of Database Engine", "Build Database", "Scan Transactions", then exit Armory, then one second later reload Armory, it takes the same amount of time, like 10 minutes, before it goes online.  This is a feature or bug?  It should not take so long the second time it is loaded since the data downloaded is already in place?

That does not seem right.  It takes around half a minute to a minute for me if I just stop Armory and let it restart.  Perhaps you have a very slow harddisk or very little RAM, perhaps there is something else wrong.  But something clearly is wrong.  Have you tried looking at the log while it starts, maybe that can be an indication of what is wrong.

[TonyT]

That does not seem right.  It takes around half a minute to a minute for me if I just stop Armory and let it restart.  Perhaps you have a very slow harddisk or very little RAM, perhaps there is something else wrong.  But something clearly is wrong.  Have you tried looking at the log while it starts, maybe that can be an indication of what is wrong.

Where is the log file?  If it's in the menu or somewhere obvious I will find it.  I doubt my HD or memory is at fault, though I am pretty sure that if I switched to an SSD (as I might for other reasons) no doubt the entire process will be sped up considerably.

[picobit]

Where is the log file? 

It is in the same place as your wallet file.  And yes, I can guess your next question.

The position of the files depend on the OS of your computer.  In Linux it will be in the hidden .armory folder in your home directory.  On a Mac it is in Library/Application Support/Armory (Library is hidden, go to it in the finder by pressing Alt while cliking on the Finder's "Go" menu).  On a Windows machine, it is somewhere.  I don't know where, but try googling for "armory folder windows" or something like that.

[TonyT]

Where is the log file? 

It is in the same place as your wallet file.  And yes, I can guess your next question.

The position of the files depend on the OS of your computer.  In Linux it will be in the hidden .armory folder in your home directory.  On a Mac it is in Library/Application Support/Armory (Library is hidden, go to it in the finder by pressing Alt while cliking on the Finder's "Go" menu).  On a Windows machine, it is somewhere.  I don't know where, but try googling for "armory folder windows" or something like that.

Thanks.  FYI you can "export" the log file in the latest version of Armory under the File | Export menu.  I checked it out and nothing is unusual. There are several interesting entries, on occasion it will say :2014-10-21 01:31 (INFO) -- BDM.pyc:419 - Reading blockchain, pct complete: 99.9
2014-10-21 01:31 (INFO) -- ArmoryQt.py:5997 - Dashboard switched to auto-BadConnection  <--bad connection, which means no internet, as sometimes happens for a few minutes here (I'm outside the USA), but it seems to happen at 99.9 percent complete, which leads me to think there is perhaps a "magic node" that is the last one checked for some reason, and maybe this is a bottleneck?  If an Armory developer is reading this thread they probably could confirm that, for what it's worth to them.

I think the main culprit is the 'bad node' problem, as is well known, if a node that is a peer is slow, it becomes a bottleneck.  Maybe if you exit and restart Armory you have to go through this bottleneck?

BTW, if you are a privacy freak, it might freak you out to know that when you send an Automatic Bug Report to the Armory team, says a popup screen (Help | Submit Bug...) , the log file is automatically included.  And it contains 'path' information ...C:\Users\TonyT\AppData\Roaming\Armory\ArmorySettings.txt that could in theory identify help you, since I assume the IP address is also included in the bug report.  Not a big deal for most people however.  Perhaps in future settings, just to put these privacy freaks at ease, you could replace via the program the "TonyT" in the path above with USERNAME.

TonyT

[grue]

duplicate of https://bitcointalk.org/index.php?topic=824307.0, locking