Extremely long engine initialization, v 0.95.1

[ToF]

Hello everyone,

I used Armory for the last time several months ago, a few days ago I launched it again, version 0.93.x which is now upgraded to the latest 0.95.1.

The problem is, it has been initializing the engine for days now! I do realize there is a lot of new data to download, but it's crawling and I have very fast connection. I do see new block files are being downloaded so it's not stuck. Just very very slow.

Any idea what could be causing this? The OS is Windows 7 64bit with 8GB RAM and Core i7 (albeit an older laptop version).

Thanks!

Tomas

[Carlton Banks]

You are limited by 2 main factors:

• Download speed
• Processor max clock speed

1st issue should be ok if you're using Bitcoin Core 0.10 minimum, I'm assuming you are. 2nd issue shouldn't be so terrible on an i7 laptop, but maybe the reality is different. I don't think additional processing cores help, the best results are achieved with desktop machines with the PSU wattage that can safely support > 3 GHz. The processor clockspeed matters because you've got a whole multitude of cryptographic proofs to compute for all that blockchain data you're downloading.

[goatpig]

Post your logs.

[ToF]

It seems I can't attach the logfile? I cannot see the button for it, perhaps because I have too few posts?

Also, the bitcoind process consumes around 5-8% of the CPU now. I had a problem when the BTC database was on my main SSD drive, the laptop was extremely stressed and I thought it was because of the CPU usage but apparently the SSD writing was the issue.

I moved both the bitcoin and armory directories to NAS with approximately 100MB/s write speed and since then the laptop can breathe again. The CPU usage is very low now as I said, though the bitcoind process takes almost a GB of RAM.

I will post the log if someone tells me how...

Tomas

[achow101]

Just copy and paste the contents of the file into a post. If the post is too big, put it on https://pastebin.com and post the link to the paste here.

[ToF]

OK, here is the pastebin link:

http://pastebin.com/gbXSA9nN

I had to remove the older records as it was over 512KB but all of the current version info should be there.

[goatpig]

There is no trace in your log file indicating that you have properly pointed Armory to your new Bitcoin data folder. There are indications that the auto managed instance of bitcoind is failing to start under these new settings.

I would suggest you change your setup to the following to clarify your custom setup:

1) In Armory, go to File -> Settings and turn off auto bitcoind management (in the first section).

2) Create a shortcut of ArmoryQt.exe. Right click it, pick properties. In the properties dialog, go to the General tab, and in the target input box, append the following:

--satoshi-datadir="X:\Other\BTC\Bitcoin" --datadir="X:\Other\BTC\Armory"

3) Create a shortcut in the same fashion for BitcoinQt.exe (you should find it here: C:\Program Files (x86)\Bitcoin\bin\). Append the following to its target:

-datadir="X:\Other\BTC\Bitcoin"

With that done, start BitcoinQt from your new shortcut, then Armory from its new shortcut. Report here afterwards. Keep in mind that armory command line arguments take 2 hyphens (--) and Bitcoin Core takes 1 (-)

You have been using the default auto bitcoind setup so far, and that setup is really only functional with all settings at their default value. I would suggest you stray away from that arrangement if you wish to use custom paths, at least for this version.

[ToF]

OK, I will change those settings. However, the new bitcoind path is apparently working as the blocks files are saved into the new directory etc. And beside, a few days ago I was running with the default settings and paths, and had the same issue, so I am quite not sure if it will help at all.

Tomas

[goatpig]

However, the new bitcoind path is apparently working as the blocks files are saved into the new directory etc.

Code:

-WARN  - 1448972888: (..\BlockUtils.cpp:1140) Scanning from 386224 to 386224

I was referring to Armory not seeing that at all.

[ToF]

Well, this is under File->Settings->Bitcoin Home Dir: X:\Other\BTC\Bitcoin\

[goatpig]

The log file does not show any evidence of that, so something is failing either way. Try these settings and let me know.

[ToF]

It seems the bitcoin-qt.exe I had was version 0.85.1 from 2013, I thought Armory updated it but I was wrong. I have downloaded the newest Bitcoin Core and will see how that works. Also, the new Bitcoin Core is installed into a different folder for some reason.

I will report back any progress.

[ToF]

Update: Since I last posted I am still trying to sync Bitcoin Core. I wanted to start from scratch but was able to download only approximately 15 GB in a day and then it got almost stuck, so I used the old blocks database, which was over 90GB so that it would not have to download everything again.

I am able to download like 1GB *or less* per day now (I have 200Mbps+ Internet connection)! It's useless really. What is going on with Bitcoin these days? There seem to be almost no available nodes! In the debug window of Bitcoin Core I can see how the app struggles to find some that work.

I do realize it's not an Armory issue...

[Carlton Banks]

It's probably not a Bitcoin issue either.

The blocks need 2 major time consuming treatments:

• Downloading
• Processing

Your laptop is almost certainly choking on the processing part, not the downloading part. Bitcoin needs to d/l the blocks, but it also needs to check that the Bitcoin peers you d/l'ed them from didn't give you bad data. The amount of time it takes to process 1 block goes up noticeably as you get above around 300,000 blocks. That's because the blocks started to become very full from thereon in, there are alot more transactions to check in those blocks.

[goatpig]

Code:

2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1263 -    Total Available RAM   : 7.89 GB
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1264 -    CPU ID string         : Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1265 -    Number of CPU cores   : 8 cores
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1266 -    System is 64-bit      : True
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1267 -    Preferred Encoding    : cp1250
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1268 -    Machine Arch          : amd64
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1269 -    Available HDD (ARM)   : 87 GB
2016-11-16 03:17 (INFO) -- ArmoryUtils.pyc:1270 -    Available HDD (BTC)   : 87 GB

That's a decent CPU. The drive size suggests SSD. I'm guessing the machine isn't the issue. Could be you are getting connected a lot of weak/bad nodes.

You should consider actively managing your connections. I'd ban any node that:

- has >500 ms ping
- is not Core
- has a version number <10.0

[ToF]

Well, the bitcoind process is currently consuming 1-3% of CPU time. It does occupy the NAS quite a lot though, and my NAS is capable of 100MB/s+ writing.

[ToF]

That's a decent CPU. The drive size suggests SSD. I'm guessing the machine isn't the issue. Could be you are getting connected a lot of weak/bad nodes.

You should consider actively managing your connections. I'd ban any node that:

- has >500 ms ping
- is not Core
- has a version number <10.0

The drive in the laptop is 480GB SSD but I moved the Bitcoin as well as Armory databases to my Synology NAS which has multiple 3TB HDDs connected in RAID6. When the DBs were on the SSD, the laptop was basically unusable.

Bad nodes is all I get to be honest. Or no nodes. Right now there is even no response listed for most of them. From time to time I do get a good node and transfer even 1GB in a few hours, then it disappears and I am left with the bad ones again.

[goatpig]

Then you should consider finding a list of strong well known nodes (explorers, miners, high speed ones) and add them to your bitcoin.conf by IP just to finish bootstrapping.

[ToF]

Then you should consider finding a list of strong well known nodes (explorers, miners, high speed ones) and add them to your bitcoin.conf by IP just to finish bootstrapping.

How do I find those? I tried Googling it before but most of the time it was just old info.

In my opinion, this is a major Bitcoin issue though. I do realize most users will never use Bitcoin Core but still, there if there is lack of reliable nodes, it has to cause problems. A year or so ago this was working smoothly.

[goatpig]

I believe at this point you should take the discussion to Core specialists. I'm not too sure what is the bast place to get that info, I only know of the dev channels personally. If anything, a post in D&TD or Technical Support will put you on the right path:

https://bitcointalk.org/index.php?board=6.0

https://bitcointalk.org/index.php?board=4.0