bare ArmoryDB?

[Ente]

I try to install ArmoryDB on a headless machine, to connect with a 'lite' Armory from another machine to it.
Is there a recommended way to install only ArmoryDB, without ArmoryQT?

I ask because I don't use a GUI nor ArmoryQT on that machine, and would like to skip the *qt dependencies.

Afaik all dependencies are

git-core build-essential swig python-dev python-twisted python-psutil pyqt4-dev-tools libqtcore4 libqt4-dev python-qt4

where I guess the bold ones aren't necessary for ArmoryDB?

Cheers,

Ente

[1715412372]

1715412372

[1715412372]

1715412372

[1715412372]

1715412372

[1715412372]

1715412372

[achow101]

AFAIK you don't need anything related to python if you just want to run ArmoryDB. That is entirely c++ code.

[goatpig]

DB is full cpp code. There is no install process to just install the DB, but you can only build that if you wish so, by running

Code:

make server

[Ente]

DB is full cpp code. There is no install process to just install the DB, but you can only build that if you wish so, by running

Code:

make server

Sounds like exactly what I need! Great!
For everyone trying this, you need to enter the cppForSwig folder to compile the server.

I will come back and add the whole install procedure for the server and client side. It doesn't seem too complicated, but the info is a bit spread out.

Cheers,

Ente

[goatpig]

The DB does not use shared libraries, you should be able to just move the binary around and get it to run.

[Ente]

It's been some time, now I'm on my setup again to have bitcoin core and armorydb running on a separate machine from armoryqt.

So just running armorydb and pointing armoryqt to the machine doesn't work, I am asked to setup some http webserver daemon.
I couldn't find any details of what exactly connects where. There's an example "nginx_example.conf" on github.

What exactly does that do? To me it looks like it does nothing else than hosting localhost:9001 on network:80 with a few fastcgi options?
If that's all, why not just let armorydb listen on all interfaces?

I already have apache running on that machine.

Any pointers..? Would be greatly appreciated. I can't wait for this, the greatest new feature in armory in a long time!

Ente

[goatpig]

What exactly does that do? To me it looks like it does nothing else than hosting localhost:9001 on network:80 with a few fastcgi options?

FCGI is a webstack interface that carries CGI over a socket. The DB speaks FCGI, the daemon receives HTTP, converts to FCGI and passes it to the DB. The daemon is explosed to the LAN/WAN, the DB sits behind the daemon. Among other things, this how PHP and MySQL interface with HTTP daemons.

If that's all, why not just let armorydb listen on all interfaces?

Daemons are well established and dedicated tools that deploy well understood services with widely documented features and settings. I would have to reproduce that code in ArmoryDB to do what you are asking for.

It is not only more work for me, it is also undesirable. The DB is meant to operate in a trusted environment. HTTP daemons are built to be exposed to the public and tunnel packets to services behind them on the trusted environment. That's how the web works. I don't presume to reinvent nor sidestep current proper practices.

[Ente]

Thank you for the tl;dr of what's going on in fcgi in general and armorydb too.
Sorry, Igoatpig,  didn't want to hint that you should do some nonsense like making armorydb directly accessible from outside, I asked to understand the role of a webserver in between in this case. Honestly, never heard of fcgi before, and only know cgi from web
addresses.
I think my apache is working right now, with

Code:

ProxyPassMatch ^.*$ fcgi://127.0.0.1:9001

and a virtualhost on 9001.
I can connect to the virtualhost with a browser, as a crude test.

However, armoryqt doesn't seem to do anything, "preparing databases", scanning history at "0%" and "1 sec".
In other cases, where armorydb and armoryqt were on the same machine, this indicated that armorydb wasn't running.
How can I make sure armoryqt is really reaching armorydb?
In the logs I only see the ip and port as parameters handed over at the start of armoryqt, but no other message like "connected" or similar.

I wonder how many people actually switched to a separated setup already?

Thank you again for your patience,

Ente

[goatpig]

Thank you for the tl;dr of what's going on in fcgi in general and armorydb too.
Sorry, Igoatpig,  didn't want to hint that you should do some nonsense like making armorydb directly accessible from outside, I asked to understand the role of a webserver in between in this case. Honestly, never heard of fcgi before, and only know cgi from web
addresses.

Sorry if I came out harsh, wasn't my intention.

Maybe think of fcgi as an ancestor of websockets? CGI goes over stdout and stdin. The idea is that the daemon isn't meant to deal with custom requests. It can just serve files. So protocols were developed for daemons to tunnel certain requests to services sitting behind them for processing. That's dynamic content and PHP in a nutshell.

FCGI took the same data as CGI, but over sockets instead, and allowed for resident services, i.e. you don't need to spawn a service per request, but can streamline all requests over sockets, and let the one service sort things out.

However, armoryqt doesn't seem to do anything, "preparing databases", scanning history at "0%" and "1 sec".
In other cases, where armorydb and armoryqt were on the same machine, this indicated that armorydb wasn't running.
How can I make sure armoryqt is really reaching armorydb?
In the logs I only see the ip and port as parameters handed over at the start of armoryqt, but no other message like "connected" or similar.

Assuming you correctly proxied the DB service with a daemon, opened the proper ports and pointed the clients to the right ip and port, then you have a bug. I'd suggest you get into the 0.96 testing phase and help me identify and fix the issue.

[Ente]

Hello again,

again trying to get Armory, now v0.96, to work with an external ArmoryDB. My client is a Debian VM within Qubes OS.
I set up ArmoryDB and an apache frontend.

The full output of starting with --debug is at https://pastebin.com/qpkQEHw7

When starting with no reachable ArmoryDB (wrong port for example), it hangs after

(INFO) ArmoryQt.py:1814 - startBitcoindIfNecessary

with no GUI open yet (with the splash window visible).

With the ArmoryDB address properly set in the ArmoryQT command, the GUI opens, asks for creating a walet, and then hangs at "preparing databases".
I get the following additional output:

(WARNING) ArmoryQt.py:1804 - DB is already running
(INFO) ArmoryQt.py:1900 - Setting netmode: 1
-ERROR - 1494009752: (DataObject.h:286) exhausted entries in Arguments object
-ERROR - 1494009752: (SwigClient.cpp:61) exhausted entries in Arguments object
(INFO) ArmoryQt.py:1900 - Setting netmode: 0
(INFO) ArmoryQt.py:1882 - loadBlockchainIfNecessary
(INFO) ArmoryQt.py:4619 - Dashboard switched to "Scanning" mode
(INFO) ArmoryQt.py:4619 - Dashboard switched to "Scanning" mode

I am unsure how to proceed from here on. Is there a way to get even more debug output, for example?
Or maybe someone has an external, internet-reachable ArmoryDB I could connect to to doublecheck?

By now I would already have called the paid support phoneline :-)


Cheers and thank you for bearing with me that long already,

Ente

[goatpig]

Hmm I think it needs a wallet basically. Post the dbLog.txt on the server side.

[Ente]

Hmm I think it needs a wallet basically. Post the dbLog.txt on the server side.

..no change (on the client side at least) with having a wallet.

dbLog.txt (after restarting ArmoryDB):

-INFO  - 1494024143: (main.cpp:23) Running on 4 threads
-INFO  - 1494024143: (main.cpp:24) Ram usage level: 4
-INFO  - 1494024143: (BlockUtils.cpp:1338) blkfile dir: /home/user/.bitcoin/blocks
-INFO  - 1494024143: (BlockUtils.cpp:1339) lmdb dir: /home/user/.armory/databases
-INFO  - 1494024143: (lmdb_wrapper.cpp:388) Opening databases...
-INFO  - 1494024143: (BlockUtils.cpp:1521) Executing: doInitialSyncOnLoad
-INFO  - 1494024143: (BitcoinP2P.cpp:783) Connected to Bitcoin node
-INFO  - 1494024143: (DatabaseBuilder.cpp:169) Reading headers from db
-INFO  - 1494024146: (DatabaseBuilder.cpp:208) Found 465098 headers in db
-INFO  - 1494024147: (DatabaseBuilder.cpp:51) updating HEADERS db
-INFO  - 1494024147: (DatabaseBuilder.cpp:477) Found next block after skipping 511156bytes
-INFO  - 1494024148: (DatabaseBuilder.cpp:268) parsed block file #860
-DEBUG - 1494024148: (Blockchain.cpp:242) Organizing chain
-INFO  - 1494024148: (DatabaseBuilder.cpp:56) updated HEADERS db in 0.373228s
-INFO  - 1494024148: (BlockUtils.cpp:1636) Enabling zero-conf tracking
-INFO  - 1494024174: (BDM_Server.cpp:797) registered bdv: aabcae440dae406841c4

..the last line shows up once I run ArmoryQT on the client side.

Ente

[goatpig]

Is Core running behind your DB? There's a bug I need to investigate after moving all things p2p/RPC to the DB that will halt the client if no node is found on the server side.

Edit: actually, try to get the RPC working on your node. Make sure bitcoin.conf has server=1. Log shows it found the node, but didn't connect to the RPC.

[Ente]

Is Core running behind your DB? There's a bug I need to investigate after moving all things p2p/RPC to the DB that will halt the client if no node is found on the server side.

Edit: actually, try to get the RPC working on your node. Make sure bitcoin.conf has server=1. Log shows it found the node, but didn't connect to the RPC.

Bitcoind is running and synced.
You are right, I didn't have server=1 in my conf!
Did that, restarted both services.
I now have indeed a new

-INFO  - 1494025031: (BitcoinP2P.cpp:783) Connected to Bitcoin node

line in dbLog.txt.

It still says the same "Preparing Databases" in ArmoryQT, I'll leave it running a bit.

edit:

-INFO  - 1494025025: (BlockUtils.cpp:1636) Enabling zero-conf tracking
-INFO  - 1494025031: (BitcoinP2P.cpp:783) Connected to Bitcoin node
-INFO  - 1494025276: (BDM_Server.cpp:797) registered bdv: f8b94537d7b3804b9c52
-INFO  - 1494025357: (BlockchainScanner.cpp:52) no history to scan
-INFO  - 1494025477: (BDM_Server.cpp:826) unregistered bdv: f8b94537d7b3804b9c52

Hmm, it unregistered by itself? ArmoryQT is still open and unchanged..

Ente

[goatpig]

Hmm, it unregistered by itself? ArmoryQT is still open and unchanged..

Garbage collected on the server side. Means the client callback loop is dying for some reason and the server considers the client has disconnected ungracefully, which also explains why the lobby is stuck to initializing in the GUI.

-INFO  - 1494025031: (BitcoinP2P.cpp:783) Connected to Bitcoin node

That line was already there, I'm looking for the one that says "RPC connection established".

If you do not have this line yet, means the RPC connection is still borked. The fact that the client gets to register itself means the networking layer is fine though, and what you are experiencing is the notification issue with a half initialized server to node layer.

I got a decent idea where to look at to fix this, so I'll try to get it for 0.96.1. On your end, you should try to get the RPC working.

[Ente]

No matter what, I can't get Armory to connect to Bitcin Core via RPC.

I threw everything out of my bitcoin.conf and only have

disablewallet=1
server=1
rpcallowip=127.0.0.1
rpcbind=127.0.0.1:8332

in it now. I had it with manual rpcuser/rpcpassword credentials before, and without any rpc-options as well.
Everything looks fine to me, the auth cookie is there, both Core and Armory run under the same user, no errors anywhere, and Armory can connect (read blockfiles) to Core just fine.

Still, no luck with RPC.
Could anyone post their bitcoin.conf please?
Does anyone use Bitcoin Core without wallet functionality (not compiled in) and has Armory working with it?

Thank you,

Ente

[goatpig]

Does anyone use Bitcoin Core without wallet functionality (not compiled in) and has Armory working with it?

I do, works fine with the RPC.

This is my bitcoin.conf

Code:

disablewallet=1
dbcache=4096
addnode=127.0.0.1
#rpcuser=generated_by_armory
#rpcpassword=test
maxconnections=50
#minrelaytxfee=.001
#limitfreerelay=0
server=1

I let it run with the cookie file.

[Ente]

Thanks, goatpig.
I added "addnode=127.0.0.1" as the only obvious relevant difference I can see here, no luck.

Is there a minimum bitcoin core version armory needs? I still run

"version": 130100,
  "protocolversion": 70014

could that be it?

Ente

[goatpig]

The code was extensively ran against 0.13.1, so I don't think that's it.

Your best bet is to set some print statements in the RPC init code to get an idea what's going on:

https://github.com/goatpig/BitcoinArmory/blob/master/cppForSwig/nodeRPC.cpp#L34

[Ente]

I found my problem with RPC. I did most of my testing on the client (ArmoryQT) side, and forgot to actually keep the server side (ArmoryDB) up-to-date as well. After updating to v0.96, it RPC-connects to bitcoin-core as well. D'oh!

I couldn't find a "make server" option any more. So I had to install all dependencies (including QT stuff) and compile all Armory components, including ArmoryQT. Is there a better way to do this, when only ArmoryDB is needed?

Now I am troubleshooting why both my nginx- as well as my apache fcgi middleman doesn't seem to work.

Does anyone use ArmoryDB behind an apache?

<VirtualHost 192.168.0.1:9001>
   ServerName armorydb.localhost
   ServerAdmin webmaster@localhost
   DocumentRoot /var/www/armorydb
   ErrorLog ${APACHE_LOG_DIR}/error.log
   CustomLog ${APACHE_LOG_DIR}/access.log combined
   #Include conf-available/serve-cgi-bin.conf
   ProxyPassMatch ^.*$ fcgi://127.0.0.1:9001
   <Directory "/var/www/armorydb">
      Order allow,deny
      Allow from all
      AllowOverride FileInfo All
      # New directive needed in Apache 2.4.3:
      Require all granted
   </Directory>
</VirtualHost>

I probably am missing important fcgi parameters as stated in the nginx example config https://github.com/goatpig/BitcoinArmory/blob/master/nginx_example.conf
especially

fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;

At least I am one step further now!

edit: it works with nginx! yeah! :-)
I'll try to get it to work with apache as well. Else I'll have to move to nginx..

Ente

[goatpig]

I couldn't find a "make server" option any more. So I had to install all dependencies (including QT stuff) and compile all Armory components, including ArmoryQT. Is there a better way to do this, when only ArmoryDB is needed?

Sorry, didn't port these back when moving to autotools. Will do for 0.96.1

edit: it works with nginx! yeah! :-)

So you have no issues with nginx?

[Ente]

I couldn't find a "make server" option any more. So I had to install all dependencies (including QT stuff) and compile all Armory components, including ArmoryQT. Is there a better way to do this, when only ArmoryDB is needed?

Sorry, didn't port these back when moving to autotools. Will do for 0.96.1

Ah, good, that explains it!

edit: it works with nginx! yeah! :-)

So you have no issues with nginx?

It seems to work just fine. I used the example config with just the interface, address and ports edited.
ArmoryQT took a while to rescan the blockchain for the new wallet, after that loads up lightning fast.
My only issue is that I'd prefer to only have apache installed :-)
I guess I'll just leave nginx installed for this anyway, and book it as "security by separation".

I am very happy to have bitcoin-core, blockchain, armorydb and database on a server and a lightweight, always-syncronized notebook.
Thank you again, goatpig!

Ente

[goatpig]

My only issue is that I'd prefer to only have apache installed :-)
I guess I'll just leave nginx installed for this anyway, and book it as "security by separation".

It should work. Ima run that stack and try to figure out what's going on.