[ANN] BitcoinArmory-Daemon - armory on web servers

[goatpig]

This is why I prevent Armory from opening twice.  Wallet operations are not threadsafe. Not at all!  

However, one benefit of python-twisted is that it is guaranteed to run as a single-thread, so as long as there is only one instance of Armory, it's perfectly safe.

For your purposes, not using python-twisted, need to do something else.  Actually, I would recommend using python-twisted to create a server/daemon process to handle (serially) all requests for wallet interaction.  Make sure everything goes through that interface.  The wallet design has been remarkably robust (I even test it with mid-write interrupts and it recovers fine).  But as you can see, two simultaneous operations can break it pretty easily.  

Wallet operations are not threadsafe. Not at all!  
can someone please explain why armory wallets are not thread safe and why armory runs as a singleton? please explain in detail.

In one line: easier to implement.

Up until a few months ago, etotheipi was working on Armory alone. Cost of implementation is a very high priority in this case. Now we're 5, so things are changing, notably Armory's backend model.

[1714121456]

1714121456

[1714121456]

1714121456

[gokudev]

The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

getbalance:
Returns a decimal value in BTC for the total remaining balance in the wallet.

getnewaddress:
Returns the next address in the wallet as a string.

getreceivedbyaddress:
Returns a decimal value in BTC for the amount received by the address.

sendtoaddress:
Returns an unsigned transaction as a string. Implementation of signing and broadcasting is left to the client.



Features that may be included in the future:
User authentication
SSL
More API methods as required (the current set fills my needs). listtransactions is one method I am looking to include in the future.
Please suggest anything else you personally may find useful, as I have designed this only to fill my specific needs.



Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 

[etotheipi]

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 

At the moment, "getledger" and "getledgersimple" are the biggest divergences of them, though we attempted to implement "listtransactions" the same way Bitcoin Core does.  I really don't like what Core does with "listtransactions":  it provides multiple outputs with overlapping entries, which require a bit of extra logic to adapt to your application.

On the other hand "getledger" provides you exactly what you would see on the Armory GUI ledger -- it's a mutually-exclusive list of entries, each one corresponding to a single transaction, and theoretically all the values produced from it can be summed to give you your balance. 

We will also soon have an interface for:

- Multiple wallets (context switching)
- Create unsigned transaction (to take to offline computer)
- Sign unsigned transaction (to run from offline computer)

If you'd like to recommend more stuff you'd like to see, post in the following thread:
https://bitcointalk.org/index.php?topic=588227.0

[gokudev]

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 

At the moment, "getledger" and "getledgersimple" are the biggest divergences of them, though we attempted to implement "listtransactions" the same way Bitcoin Core does.  I really don't like what Core does with "listtransactions":  it provides multiple outputs with overlapping entries, which require a bit of extra logic to adapt to your application.

On the other hand "getledger" provides you exactly what you would see on the Armory GUI ledger -- it's a mutually-exclusive list of entries, each one corresponding to a single transaction, and theoretically all the values produced from it can be summed to give you your balance. 

We will also soon have an interface for:

- Multiple wallets (context switching)
- Create unsigned transaction (to take to offline computer)
- Sign unsigned transaction (to run from offline computer)

If you'd like to recommend more stuff you'd like to see, post in the following thread:
https://bitcointalk.org/index.php?topic=588227.0

ok so at the moment, can i sign/unsign transactions using bitcoin daemon since they already have a json api ? or will that not work with armory wallets?

[etotheipi]

Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

[gokudev]

Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?

[etotheipi]

Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?

I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.

[po0kie]

My current envoirement with ArmoryD

If its an help for someone

I have "X" Online servers (at the moment I am trying to know how much Users I can support on each)
And "X" "Warm-Cold" Wallets server instances. (Each online server instance has an offline Server instance)

Process to broadcast:

It starts with: Web requests to Middle-Api-Server to make a broadcast from AddressX to AddressZ

The Middle API Server communicates to Online Watching Server and checks the balance.
If balance is OK, I create a encrypted Queue of another Middle-Backend with an Unsigned Transaction.

Then another Process Server Checks the Encrypted Queue for unsigned Tx's and sends it to the Warm-Cold Server to sign.
This middle Server stores the SignedTx back to the Encrypted Queue and then the Online Server Queue get a HexTx from ArmoryD (Online server)
and broadcasts this rawtransaction over BitcoinD RPC.

Its a little bit confusing maybe as 5 Servers are needed for this process.
(1 web server, 1 Api server, 1 Online Watching Server, 1 Offline Server, 1 Queue process Server For Signing)
and then increasing on Servers depending on Users.


The testings right now are 100.000 Users per Wallet
- You need to tweak the ArmoryD a little bit for this porpose on some places
   1. The most functions are on Wallet scope - I added functions to support the RPC on Address Scope.
   2. The Offline server Unlock function is very slow on 100.000 Addresses, so I modified the Unlock function to just unlock the PrivKey which I need to sign.

Its just a little info for somebody to get started with armory and Services on a website.

I would preffer Wallet per User, but until armoryd does not support multi Wallet its just impossible - at least I didnt found a suitable solution.


In a few days I can tell if the wallet supports 100.000 Addresses / Users with a avg. of 2-3 Transactions per User. (would be 300.000 Transactions per Wallet / Server)


Best regards

[gokudev]

Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?

I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.

  You don't have to operate through RPC, you can just program your hooks directly in.

Well the thing is, our server making the calls to the daemon is in .net(C#) It is much easier and cleaner to simply make json rpc calls to the coin daemon. What are the advantages of programming the hooks directly?


Also looking at armoryd.py i notice many new jsronrpc methods not found in the original bitcoin API calls list (https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list)  these methods have a prefix of 'jsonrpc' so im thinking they are newly added to the api call list but wiki does not show them.

[gokudev]

Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?

I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.

I need to figure out how to use bitcoind and armory offline wallets together via json rpc or some other method.
I think this is the post I am looking for  https://bitcointalk.org/index.php?topic=448284.0
this is able to convert a signedtransaction from bitcoind into something armory can sign in offline mode. correct me if i am wrong?

[CircusPeanut]

I need to figure out how to use bitcoind and armory offline wallets together via json rpc or some other method.
I think this is the post I am looking for  https://bitcointalk.org/index.php?topic=448284.0
this is able to convert a signedtransaction from bitcoind into something armory can sign in offline mode. correct me if i am wrong?

Assuming you mean that  you want to convert an "unsigned transaction" for armory to sign. Yes, you can do this with a bit of coding in armoryd.

Take whatever raw transaction hex from bitcoind and use the unserialize method on the appropriate object to get an Armory object. You probably want PyTx or PyTxDistProposal. Note that in the devel branch (and others) PyTxDistProposal is renamed UnsignedTransaction. So whatever the code base you're using look for the appropriate object and call unserialize on it.

[gokudev]

OK armory would need to be modified and recompiled. Can Armorybe modified to create wallets for a coin type ? So I want multiple wallets maybe one wallet per coin type. 

[CircusPeanut]

OK armory would need to be modified and recompiled. Can Armorybe modified to create wallets for a coin type ? So I want multiple wallets maybe one wallet per coin type.  

No need to recompile since armoryd is written in Python.

Modifying Armory to run with different crypto currencies requires a lot of work.

[gokudev]

Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt ? 
Indulge me what makes you say its a lot of work ?

[CircusPeanut]

Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt ? 
Indulge me what makes you say its a lot of work ?

I'm basing that assessment on this thread in github https://github.com/etotheipi/BitcoinArmory/issues/32

[gokudev]

Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt?
Indulge me what makes you say its a lot of work?

I'm basing that assessment on this thread in github https://github.com/etotheipi/BitcoinArmory/issues/32

Wondering if etotheipi can chime in, why does the hashing function need changing for ltc if I am just using armory wallets for storing coins. or are wallets and hashing function connected together?

Can armory not use the coin daemon (litecoin daemon via rpc calls ) and manage offline wallets that way. I can modify armory to convert the unsigned tx to work with armory's offline signing and then all should be good?

[gokudev]

I am a bit confused, Why does armory run bitcoind in the background and what is the use of armory daemon?

[etotheipi]

I am a bit confused, Why does armory run bitcoind in the background and what is the use of armory daemon?

If you don't know the answer to that question, you should probably spend some time with the Armory GUI to understand the vast functionality provided by Armory.   Especially the offline wallets.  Most people are interested in armoryd because of the ease with which you can create watching-only wallets to put on your webserver, and keep the private keys on an offline computer.

Armory uses Bitcoin Core solely for communicating with the Bitcoin network, peer finding, and blockchain validation.   Armory provides 100% new wallet infrastructure to provide deterministic wallets (only needeing to be backed up one time), plethora of backup features, the ability to manage multiple wallets at once, and the ability to manage wallets created offline but still monitor the wallet activity and distribute payment addresses from an online computer.
 
Armory uses the filtered datastream provided from Bitcoin Core, since no other software can reliably and securely replicate the validation/verification behavior of Bitcoin Core (if you use something else, you risk getting forked).   Armory doesn't even try to do blockchain validation -- that's Core's job.  Therefore you need to run both, and Armory watches the blk*.dat files from Core and the tx messages that are sent over P2P.

[po0kie]

Hi!

I noticed that some transactions are not getting to mainbranch, and for this reason the Tx never gets confirmed.
At this moment I dont know why it happens. Over 25.000 transactions on Testnet and maybe 50 cases on which happens the mainbranch issue.

Is there a way to detect the "scrap" items on armory core?
At this moment the one and only working solution I got is to restart the watching only wallet service.
After restarting armory service the TxId just dissapears and I mark them as "Scrap" in database and reissue the Tx creating a new one.


Is there a way to ask Armory core if the transaction is really a "scrap" item?
And how can it happen that the Tx does not get into the Mainbranch? (all tests are done over testnet so far)

Best regards

[TakashiMakat]

How can I install txjsonrpc on the BitcoinArmory directory?