Bitcoin Forum

Bitcoin => Armory => Topic started by: mav on July 10, 2012, 05:09:49 AM



Title: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on July 10, 2012, 05:09:49 AM
Armory Daemon is now available on github:

https://github.com/thedawnrider/BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon)

More info on page 2 of this thread - https://bitcointalk.org/index.php?topic=92496.msg1197291#msg1197291 (https://bitcointalk.org/index.php?topic=92496.msg1197291#msg1197291)

-------------------------

I thought I'd start this thread as a place to collate information about using Armory client on a webserver, since the only info I found so far was hidden on page 16 of the Armory Discussion topic.

The info below is about getting an Armory address on a webserver and it comes from https://bitcointalk.org/index.php?topic=56424.msg765880#msg765880 (https://bitcointalk.org/index.php?topic=56424.msg765880#msg765880)

Anyone who has information or questions about using Armory on a webserver, please post it here so webdevs can make the most of Armory, which currently seems to be focusing on GUI interaction rather than API (understandably).




I put some thought into how Armory could be used for a webserver application, and I realized that with just a couple lines of python, you can generate addresses no problem on your webserver.  I added a couple convenience functions and committed them to the master branch of Armory. Here's what you do:

(1) Load Armory on your computer.  
(2) Create a regular wallet.  
(3) In Wallet Properties, "Create Watching-Only Copy"
(4) Upload watchonly wallet to webserver
(5) Setup webserver to load wallet and call wlt.getNextUnusedAddress() for each request
(6) Sit at home with the full wallet and watch/confirm/send transactions while the webserver does the work!

Demonstration code to see how to access/manipulate the wallet from a python script:

Code:
from armoryengine import *

wlt = PyBtcWallet().readWalletFile('/home/server/web_server.watchonly.wallet')
wlt.setAddrPoolSize(1000)

# Quick printing of addr info, and shows how to access various props of the addr
def pprint(addr):
   print 'The new address: '
   print '   Hash160:  ', binary_to_hex(addr.getAddr160(), BIGENDIAN)
   print '   Base58:   ', addr.getAddrStr()
   print '   Index:    ', addr.chainIndex
   print '   Have Pub: ', addr.hasPubKey()
   print '   Have Priv:', addr.hasPrivKey()
  
print 'Get the next unused address...'
addr = wlt.getNextUnusedAddress()
pprint(addr)

print 'Generating 10 more addresses:'
for i in range(10):
   addr = wlt.getNextUnusedAddress()
   print '\t', addr.chainIndex, '\t', addr.getAddrStr()

print 'Get addr #15'
a160 = wlt.getAddress160ByChainIndex(15)
print binary_to_hex(a160, BIGENDIAN)
addr = wlt.addrMap[a160]
pprint(addr)

print 'This wallet has %d addresses computed' % wlt.getHighestComputedIndex()
print 'Addresses used: %d ' % wlt.getHighestUsedIndex()

Here's the output on a watching-only wallet (this is actually the output of the second time I ran it):

Code:
Get the next unused address...
The new address:
   Hash160:   728b0b8cc930cb4756328e743b7b2e7dde93a35f
   Base58:    mpEeSrEr3EiyCMRskT5VrELkY8X9ZwT3BV
   Index:     11
   Have Pub:  True
   Have Priv: False

Generating 10 more addresses:
12 mrhzBHQcn7jmhvpbcfW6ebiAFEm9qZF6AL
13 moPM3KPygygsWzvUHfeDsmPChL5xWycc5y
14 n4ZtG47TNMifGJ57msGAs8ZD1HeVqSqRyx
15 mw3cxXKCaKitV8w1wBRrpjQNWu4od5Nd5H
16 n1UGPkUgUXiwsotzfpAezpy6NkRg1fGMv5
17 muFPi3pAw7Rbb9aGc9UJPX9m6JT1VGHf9J
18 n3TRNc86Vmi2EygvjeMsAXXAYhPygR2sQK
19 mikSy5FjeSPVBSaXDs8ihMKrMgZsoohsjz
20 mwb8h2PJvGGGQTA7QBgvPVrWowQZi43ZD2
21 mpBFZ1H6AcRqHUZ7ETz1Xh1PqaPiMvdQDM

Get addr #15
c42ea65330263fd9c02fe5ca2e3a1c44dca356aa
The new address:
   Hash160:   c42ea65330263fd9c02fe5ca2e3a1c44dca356aa
   Base58:    mw3cxXKCaKitV8w1wBRrpjQNWu4od5Nd5H
   Index:     15
   Have Pub:  True
   Have Priv: False

This wallet has 1021 addresses computed
Addresses used: 21

Your webserver can run blissfully ignorant about what is actually going on with the wallet, it just gives out a new address on every request.  You can sit at home on your computer with Armory running and the full wallet there, with full control.  But someone who compromises the webserver will not get anything!

If you want the webserver to track incoming transactions, it will require loading the blockchain and checking for new blocks periodically.  It's not a lot of code, but I'll save it for later if you are interested.

One other thing:  Armory uses the crypto++ library, which is not particularly fast at ECDSA operations (and my key generation algorithm is kinda slow, which is why I'll be upgrading to a new one, soon).   My [decent] system can generate about 200 addr/sec.  So this script takes a few sec to run.  But all keys are stored in the wallet, so they don't have to be recomputed when you restart the script.


Title: Re: Armory on web servers
Post by: mav on July 12, 2012, 08:15:57 AM
Here is some more info which I found from testing today.

I wanted to cut the armory source down to remove dependency on qt, and thus run just a simple script to access the watchonly wallet on my web server without installing the whole of armory.

My goal in this post is to get new addresses from a watch-only wallet using python, on a server without qt installed.

Steps:

From the source at https://github.com/etotheipi/BitcoinArmory (https://github.com/etotheipi/BitcoinArmory) I took only the following file:

  • armoryengine.py

There were two more files I needed, which are not in the github repo (presumably compiled on install), so I got them from my armory install, located in /usr/share/armory and added them to the folder containing the above three files.

  • CppBlockUtils.py
  • _CppBlockUtils.so

Note there is no dependency on qt for any of these files.

I copied a watchonly wallet from armory into the folder containing the specially selected files above, and created a test.py file as below. It worked! The test script read the wallet file and created / printed the details of the next address.

Code:
from armoryengine import *
wlt = PyBtcWallet().readWalletFile('test.watchonly.wallet')
addr = wlt.getNextUnusedAddress()

print 'The new address: '
print '   Hash160:  ', binary_to_hex(addr.getAddr160(), BIGENDIAN)
print '   Base58:   ', addr.getAddrStr()
print '   Index:    ', addr.chainIndex
print '   Have Pub: ', addr.hasPubKey()
print '   Have Priv:', addr.hasPrivKey()

This allows the web server to run a light-weight non-gui version of the watch-only capabilities of armory, which utilises the watch-only wallet.

Note that this code does not seem to allow interaction with the bitcoin network, only with the wallet. Thus, getting the balance of an armory address from the watchonly wallet is not possible with this code (which is my next task to figure out). Any suggestions how to go about it are most welcome. I imagine this functionality should be possible from the satoshi bitcoin client, but I have yet to figure out how to get the balance of an address not in the current (satoshi) wallet. I'm also looking into how armory connects to bitcoind and how it gets balances for addresses.

Any suggestions or comments are most welcome, this is all new to me.


Title: Re: Armory on web servers
Post by: davout on July 12, 2012, 09:20:42 AM
I would totally donate a few coins for an Armory port to Ruby


Title: Re: Armory on web servers
Post by: mav on July 12, 2012, 10:06:20 AM

Note that this code does not seem to allow interaction with the bitcoin network, only with the wallet. Thus, getting the balance of an armory address from the watchonly wallet is not possible with this code (which is my next task to figure out). Any suggestions how to go about it are most welcome. I imagine this functionality should be possible from the satoshi bitcoin client, but I have yet to figure out how to get the balance of an address not in the current (satoshi) wallet. I'm also looking into how armory connects to bitcoind and how it gets balances for addresses.


I found an answer for these questions and have posted some code about how to get the balance of an address.

Calling BDM_LoadBlockchainFile is very slow (tens of seconds on my machine) since I'm guessing it loads and parses the entire blockchain. I'm going to look into how to incorporate that command (probably with some others to make it useful) and create a kind of 'bitcoind' for armory, which will be able to access the watch-only armory wallet, and also be able to get balances for that wallet. It seems from the comments in the armory source code that any address balance can be easily fetched, not just addresses in the wallet (can't find the comment sorry).

Code:
from armoryengine import *

wlt = PyBtcWallet().readWalletFile('armory_2hdJwWB5h_.watchonly.wallet')

addr = wlt.getNextUnusedAddress()
print 'The new address: '
print '   Hash160:  ', binary_to_hex(addr.getAddr160(), BIGENDIAN)
print '   Base58:   ', addr.getAddrStr()
print '   Index:    ', addr.chainIndex
print '   Have Pub: ', addr.hasPubKey()
print '   Have Priv:', addr.hasPrivKey()

BDM_LoadBlockchainFile()

print wlt.getAddrBalance(addr.getAddr160())  #This prints the address balance from the address above

As for a Ruby port, sorry I'm not going to be much help cause I don't know any ruby. Perhaps someone else will step up and investigate. Possibly if a bitcoind style implementation of armory can be developed, with a json api, perhaps your ruby could access armory through that interface rather than as a direct language port. Time will tell.


Title: Re: Armory on web servers
Post by: etotheipi on July 13, 2012, 04:58:00 AM
Hah!  I didn't even notice this thread.  I'll offer lots of advice for you...

Most importantly, checkout extras/sample_armory_code.py.   That has a lot of details about how to load the blockchain and access address balances.  Constructing transactions isn't so obvious, but it's do-able.

I'm kinda swamped this month, but I'm subscribed now and will provide tips on using the armoryengine.py where I can.

Btw, I did intend for armoryengine.py + _CppBlockUtils.so/.dll/.pyd to be a nice standalone, full-featured library.  I'm pretty sure that all the ArmoryQt.py and qtdialogs.py code only uses armoryengine.py, but does not expand it (thus, anything that Armory does, you can do with just those files).  There's no JSON interface yet, but I think I can add it...

 
P.S. -- The dependencies for just armoryengine+cppBlockUtils is:

Code:
build-essential 
swig
python-dev
python-twisted

All of it is in the Ubuntu repos.  Doing this in Windows is far from easy, but I'm pretty sure the original instructions posted 6 months ago still work.


Title: Re: Armory on web servers
Post by: mav on July 13, 2012, 11:38:58 AM
Anticipating that servers often create multiple threads to handle concurrent requests, I wanted to get a feel for how armoryengine would respond to attempts at fetching many addresses at the same time. I ran the following test code and it borked my watch-only wallet file, to the point where I couldn't even get an address out of it without threading.

Code:
from armoryengine import *
from threading import Thread

def get_addr():
    wlt = PyBtcWallet().readWalletFile('test.watchonly.wallet')
    addr = wlt.getNextUnusedAddress()
    print addr.getAddrStr()

for a in range(10):
    Thread(target=get_addr, args=()).start()

Whenever I call readWalletFile on a borked wallet file, with or without threading, I get the following error:

Code:
***ERROR:  Requested wallet is for a different blockchain!
           Wallet is for:   

.../armoryengine.py", line 6907, in unpackHeader
    print '           Wallet is for:   ', BLOCKCHAINS[self.magicBytes]


The purpose of this post is just to gather the info, I will look more into how to handle this in the near future. Any suggestions from wise heads are welcome.


Title: Re: Armory on web servers
Post by: etotheipi on July 13, 2012, 07:40:28 PM
This is why I prevent Armory from opening twice.  Wallet operations are not threadsafe. Not at all!  

However, one benefit of python-twisted is that it is guaranteed to run as a single-thread, so as long as there is only one instance of Armory, it's perfectly safe.

For your purposes, not using python-twisted, need to do something else.  Actually, I would recommend using python-twisted to create a server/daemon process to handle (serially) all requests for wallet interaction.  Make sure everything goes through that interface.  The wallet design has been remarkably robust (I even test it with mid-write interrupts and it recovers fine).  But as you can see, two simultaneous operations can break it pretty easily.  



Title: Re: Armory on web servers
Post by: mav on July 16, 2012, 02:21:14 AM
Indeed, twisted seems to be the way to go. I've written a twisted web server which allows for threaded interaction with the armory wallet and blockchain.

I'm looking for feedback about what functionality to incorporate (and in what format) from people who may want to use this.

Currently I've got a basic web server with the following url structure (only basic calls for my needs so far but I plan to extend it further)

URLDESCRIPTIONARGSOPTIONAL defaultCONSIDERATIONS
/wallet/getAddressgets the next unused address from the watch-only wallet
/blockchain/getAddressBalancegets the balance of an address (not necessarily in your wallet)address=XbalType=Spendable | unconfirmed | ultimateUnsure to use balType from armory client or minconf from satoshi client

My doubt with the url design is whether to emulate as closely as possible the internal class structure of armoryengine or whether to design an entirely new abstraction with wallet and blockchain etc (like above table) which is independent of the wallet type (satoshi / armory etc). My thoughts right now are to replicate the internal structure of armory, since this is a wrapper for armory and not a generic application. But then I think perhaps a generic web interface would be good to have...

Any particular feature requests will be given priority over general coverage (once I've satisfied my own needs), so make them in this thread.

Suggestions about how the implementation are most welcome.




Also, I am unsure about the difference between spendable, unconfirmed and ultimate in address balances. Can someone clear this up for me? It didn't behave how I expected. eg:

Into one address I had 1.2 btc transferred with 8 confirmations and 2.1 btc with 1 confirmation, and I got this result:
spendable - 3.3
unconfirmed - 0
ultimate - 3.3


Title: Re: Armory on web servers
Post by: etotheipi on July 16, 2012, 02:33:53 AM

Also, I am unsure about the difference between spendable, unconfirmed and ultimate in address balances. Can someone clear this up for me? It didn't behave how I expected. eg:

Into one address I had 1.2 btc transferred with 8 confirmations and 2.1 btc with 1 confirmation, and I got this result:
spendable - 3.3
unconfirmed - 0
ultimate - 3.3


(1) Ultimate/Full includes every, confirmed and not confirmed
(2) Spendable includes everything (SentToSelf || >1 confirmation)
(3) Unconfirmed is (NotSentToSelf && <6 confirmations)

EDIT:  I realize now it would've been clearer to say avoid "Spendable" and just say "Confirmed".  Though maybe not: you send coins to yourself in Armory and they show with 0 confirmations, yet it still shows up as confirmed.  That doesn't fit well with folks conditioned to believe that nothing is confirmed until 1+.  Ehhh... whatever.  I'm leaving it.


Title: Re: Armory on web servers
Post by: Clark on August 12, 2012, 09:37:02 PM
I have been looking for this information on the Armory website (like a developer API section), but I'm sure it will be up there at some point.

Thanks for this! It will be highly useful.


Title: Re: Armory on web servers
Post by: etotheipi on August 12, 2012, 10:22:12 PM
I have been looking for this information on the Armory website (like a developer API section), but I'm sure it will be up there at some point.

Thanks for this! It will be highly useful.

That kind of documentation has not made it into my priority list, yet.  However, threads like this are specifically for users to share ideas about how they successfully leveraged armoryengine for their purposes.  And while I don't have time to make such documentation (yet), I will respond to such threads and help folks accomplish their goals. 

Also, if you checkout the git project, there is a file called "extras/sample_armory_code.py" which has a lot of example code.  It's not as good as actual documentation, but working example code is quite useful.


Title: Re: Armory on web servers
Post by: Herbert on August 21, 2012, 08:30:08 PM
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!


Title: Re: Armory on web servers
Post by: etotheipi on August 22, 2012, 07:12:53 PM
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!

Exactly.

I recommend using an offline computer to make the full wallet, then make a watching-only copy.  Put it on your webserver, and also import it into Armory on your primary system.  Then your webserver will distribute addresses, and you can see the full transaction history from your desktop.  If someone gets the watching-only wallet... well all they can do is watch, but they can't take any funds!

I don't recommend generating addresses from the watching-only wallet, while the webserver is doing the same.  There won't be any fireworks, but you might end up re-using an address by accident.  If you want to also be able to distribute addresses from your primary system, I recommend making a second wallet on the offline computer just for that purpose.  You can watch them both from Armory, but only distribute addresses from the one the server isn't using.  Then you can also execute offline transactions from either wallet using the online computer.


Title: Re: Armory on web servers
Post by: mav on August 23, 2012, 03:45:55 AM
So i can basically create as many receiving addresses on the webserver as I want, but without any risk in case the server gets compromised? Awesome!

This functionality is very simple to achieve and is outlined in the first post thanks to etotheipi providing the example. This speaks volumes about how nicely armory is designed, it's very easy to work with. I have started on an api interface which currently is able to get the next address in the watch-only wallet.

I have had difficulty in achieving my next two goals with armory which is to get the balance of an address, and to make a transaction for signing by the offline wallet. Unlike generating the next address, both of these require the BDM to be loaded (ie the blockchain to be buffered), which I have been able to achieve. However, implementing the functionality I desire has not yet happened, I have had too many other things which have taken priority, and I have not looked closely enough at the source to realise these goals.

If anyone is able to make this extra functionality work I would love to hear how you did it. I have written a very basic foundation which allows interface to armory wallets, however it's so basic that I have not yet uploaded it for public consumption. The project deserves a more complete implementation than what I have started. I don't seem to recall anything being in the example file about generating transactions.

Delay in upload is also due to considering whether to implement a satoshi-clone api or to create an api specific to armory. Any suggestions on this are also welcome.


Title: Re: Armory on web servers
Post by: etotheipi on August 23, 2012, 12:40:09 PM
I mentioned in my original post about this that this part is not trivial.  It's because selecting coins for a transaction is not a one-shot process -- you have to declare what your desired output amount is and your desired fee, then Armory will create the best transaction it can based on those parameters.  If the transaction has properties that require a bigger fee, then you need to re-select coins with the new fee.  Theoretically, this should be part of a while-loop or something (in case the new selection requires an even-bigger fee), but I designed the SelectCoins algorithm to use a selection that leaves some margin (if possible) so that a higher fee can be included without changing the coin selection.

The functions for reference are:
validateInputsGetTxDP() at qtdialogs.py:4867 (https://github.com/etotheipi/BitcoinArmory/blob/master/qtdialogs.py#L4867),
createTxAndBroadcast() at qtdialogs.py:4811 (https://github.com/etotheipi/BitcoinArmory/blob/master/qtdialogs.py#L4811)
(Maybe)broadcastTransaction() in ArmoryQt.py:1889 (https://github.com/etotheipi/BitcoinArmory/blob/master/ArmoryQt.py#L1887)).  
PySelctCoins() in armoryengine.py:5049 (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine.py#L5049).  

There's a lot of code here, because it is also validating the data was entered properly on the Send-Tx dialog.  But what you need is there.  I'll try to summarize it here.

PySelectCoins arguments:
  • unspentTxOutInfo: a list of unspent TxOuts, retrieved via wallet.getTxOutList("Spendable") (qtdialogs.py:4968)
  • targetOutVal: total amount to send to (all) recipients(s), in satoshis  (i.e. 10.0*ONE_BTC for 10BTC)
  • minFee: you can always set this to 0 the first time, and re-call it with higher if the output suggests more
  • numRand (optional): number of randomized SelectCoins solutions to try (more about this in the appendix, below)
  • margin (optional): how much margin you'd like to build into the selection, for the reasons stated above

Actually executing the transaction -- you must get all the spendable coins, make a coin selection, then calculate the change and generate the recipient list (you will need to wallet.getNextUnusedAddress() generate a change address, or you can do something else).  Shuffle the list.   Then create the "PyTxDistProposal" -- this is an intermediate step where the transaction is constructed as a BIP 0010 (https://en.bitcoin.it/wiki/BIP_0010) packet, in case it needs to be signed by an offline computer.  In the case that you have the full wallet -- this TxDP will be signed immediately, then converted to a standard transaction and broadcast.

And finally, here's the code sample you've been waiting for:  Create one transaction from a wallet, to send 10.0 BTC to one recipient

Code:
       # Get unspent TxOutList and select the coins
      totalSend, fee = long(10.0 * ONE_BTC), 0
      spendBal = wallet.getBalance('Spendable')
      utxoList = wallet.getTxOutList('Spendable')
      utxoSelect = PySelectCoins(utxoList, totalSend, fee)

      # Check the minimum fee for the coin selection
      minFeeRec = calcMinSuggestedFees(utxoSelect, totalSend, fee)[1]
      if fee<minFeeRec:
           if totalSend + minFeeRec > spendBal:
               raise NotEnoughCoinsError, "You can't afford the fee!"
           utxoSelect = PySelectCoins(utxoList, totalSend, minFeeRec)
           fee = minFeeRec

      if len(utxoSelect)==0:
         raise CoinSelectError, "Somehow, coin selection failed.  This shouldn't happen"

      # We have a good coinselection, now compute change and create recip list
      totalSelected = sum([u.getValue() for u in utxoSelect])
      totalChange = totalSelected - (totalSend  + fee)

      outputPairs = []
      outputPairs.append( [addr160_recipient, totalSend] )
      if totalChange > 0:
         outputPairs.append( [wallet.getNextUnusedAddress().getAddr160(), totalChange] )

      # Shuffle the list, create the TxDP
      random.shuffle(outputPairs)
      txdp = PyTxDistProposal().createFromTxOutSelection(utxoSelect, outputPairs)
      
      # Sign the TxDP
      wallet.signTxDistProposal(txdp)
      finalTx = txdp.prepareFinalTx()
      wallet.setComment(finalTx.getHash(), "This was the example tx created by the python script")
      finalTx.pprint()
      print "Hexified Tx: ", binary_to_hex(finalTx.serialize())
      
      # Broadcast the transaction
      ...  (will fill this part in later)
      

I suppose I could encapsulate much of this code in a more friendly way.  On the other hand, you can pretty much copy this code and make the appropriate adjustments, and you'll get a signed transaction.  You can modify it for multiple recipients, handling fees differently (but you should trust the calcMinSuggestedFee[1] value), and saving off copies of the transaction before sending.

I will include more about actually broadcasting the transaction, in a bit.  The problem is that you need to start the networking engine in order to actually broadcast it.  On the other hand, you have a PyTx object (finalTx), so you can call binary_to_hex(finalTx.serialize()) to get the raw hex version of it, which you might be able to broadcast in other ways.  (like "sendrawtransaction" RPC call with bitcoind)

I know you'll have questions.  Fire away!



Appendix: The way PySelectCoins works:  a bunch of deterministic coin-selections (solutions to the pseudo-knapsack problem) are generated based on a variety of standard coin pools (the types and sizes of unspent TxOuts for your wallet).  A few random solutions are thrown on top to generate some variety.  Usually, the deterministic algorithms find the best answer, but for strange coin-pool make-ups, some semi-randomized selections will produce decent answers.  In all, we get about a couple dozen possible solutions, and then each one is evaluated for "fitness" ... how big is the fee?  how many addresses are combine?  how obvious is it which output is the recipient and which one is change?  The relative weightings of each of these is defined in armoryengine.py:5000 -- the WEIGHTS variable defines how much relative weighting to use for each factor.  There's six of them, though it really could've been reduced to 3 important ones:  tx-fee/allow-free, input-anonymity, and output-anonymity.  

Code:
WEIGHTS[IDX_ALLOWFREE]  =  100000
WEIGHTS[IDX_NOZEROCONF] = 1000000  
WEIGHTS[IDX_PRIORITY]   =      50
WEIGHTS[IDX_NUMADDR]    =  100000
WEIGHTS[IDX_TXSIZE]     =     100
WEIGHTS[IDX_OUTANONYM]  =      30

You can manually modify these weightings, or find a way to inject a new weights matrix into the PyEvalCoinSelect (I apologize, I didn't make that easy, you might be better to just manually change armoryengine.py, for now).  What do these mean?  The default behavior is to avoid using zero-conf coins at all costs.  Any solution with no zero-conf coins will be scored higher than any solution with zero-conf.  The next two biggest weights are "ALLOWFREE" and "NUMADDR".  This means that equal weighting is given to "please give me a transaction that can be free" and "please use as few addresses as possible on the input side".   "TXSIZE" and "PRIORITY" are next, but probably shouldn't even be there, because they are already tied into the ALLOWFREE and NOZEROCONF scores.  

Finally, OUTANONYM is the last critieria.  Given a set of coin-selects that are basically all the same for the other 5 criteria, it will select the solution that has better output-anonymity.  i.e. -- if the outputs are 10.0 and 1.3482023, it's fairly obvious to someone observing the blockchain which one is the recipient, which is the change-back-to-self (1.3482023).  However, if a solution is found that gives 11.34 and 5.11 -- it's not so obvious anymore, and that solution will get a higher score.  It gets extra credit if it's deceptive:  if the recipient is getting 1.384 and it finds a solution that has a change of 8.1:  then most would observers would make the wrong conclusion.

The point of this description was not only to describe the weightings, but suggest you change them for your own usage.  I had planned to make this customizable in Armory, I just hadn't gotten around to it.  If all you care about is input-anonymity and output-anonymity, I suggest changing NUMADDR and OUTANONYM to high numbers, and change the other 4 to something tiny.  You might pay a fee when you otherwise wouldn't have to, but it will be more anonymous.

DISCLAIMER:  I make no guarantees about transaction anonymity at all.  Use of this feature is completely at your own risk.  This is implemented solely for the purposes of declaring a rough ordering for an under-defined set of solutions.  Setting any particular weight to 1010 and others to 0 does not mean you are guaranteed a "good" solution -- there may only be one solution.


Title: Re: Armory on web servers
Post by: runeks on August 25, 2012, 12:54:01 AM
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.


Title: Re: Armory on web servers
Post by: etotheipi on August 25, 2012, 03:16:19 AM
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.

This seems like a good idea to post on the Improving Offline Wallets Thread (https://bitcointalk.org/index.php?topic=68482).

I like the hardware idea, except I don't understand where the private key & signature comes from?  It's not stored on the online system, is it? 

Why do you need any signing?  Why not just use that device to facilitate passing ASCII-only between the two systems?  Any non-ASCII buffers are rejected, and the offline computer will do nothing except collect the data and wait for the user to act on the received data.  Really, it would just behave like a serial port.

Damnit!  I wish I could make the serial port idea work...


Title: Re: Armory on web servers
Post by: runeks on August 25, 2012, 04:03:55 AM
Another possible web server setup that is possible with Armory (that I've actually thought of setting up myself), is having the web server connect via USB to a small Linux ARM device (just something cheap) that has a USB OTG port (allowing the device to act as a USB client). This USB client device stores the private keys and only responds to request over USB. With this setup, it's possible to simply send transaction requests to the web server that are signed with a certain private key, let the web server forward the request to the device, and then let the device - connected only via USB and not connected to the Internet - verify that request with the corresponding public key, and only return a signed transaction over USB if the signature on the request can be verified with the public key on the device.

This should - if implemented correctly - make the wallet unhackable. The only thing an attacker can do - even if he gains full access to the web server - is send request over the USB connection to the device that hold the keys, but without the proper private key the device will ignore the requests.

This could be useful for use cases such as an exchange needing to transfer coins from its cold storage wallet to it's online wallet. The exchange owner simply has to make sure that the private key used to sign requests is inaccessible to attackers.
Another use case could be a company exchanging fiat currency for bitcoins. Here the bank is in possession of the private key. It promises to sign all incoming wire transfers with this private key. If the wire transfer - in a comments-field of some sort - includes a receiving address, it's possible to let someone wire a certain amount of fiat currency to the business owners account, and let the business send out coins to that person - completely securely - given that the customer includes the send-to address in the wire transfer. If the bank can keep the private key secure, this - again - prevents an attacker from being able to do anything were he to gain full control of the web server.

I like the hardware idea, except I don't understand where the private key & signature comes from?  It's not stored on the online system, is it?
The private key (let's call it the master private key) would basically belong to the person who needs this setup (web server plus key-storage box (the latter having the private keys for the wallet (lets call these wallet private keys))) to produce and send out valid transactions, spending coins from the wallet(s) on the key-storing box. The key-storing box would then have the corresponding "master" public key, and it will only sign a transaction sent to it over the USB connection if it is signed using the master private key.

My initial use case was, as mentioned, me selling bitcoins for fiat currency. I have a site that informs the user to send an amount of fiat currency to my bank account, and include the address - to which the bitcoins are to be sent - in the "comments" field, when sending the wire transfer via his or her e-banking site. The bank I would use to provide my account would have the master private key. Whenever a new bank transfer to my account takes place, the bank sends me an email containing information on the transfer - including the destination bitcoin-address that the buyer included in the "comments" field - signed with the master private key. Then it's simply a matter of having the online web server read new incoming mails from the bank, and pass on the content of the mails to the key-storage box. The key-storage box checks that the wire transfer message is signed with the master private key, and if it is, constructs a transaction that sends an amount of bitcoins to the address specified in the wire transfer message. It passes this to the web server which publishes it. All automated, and - as far as I can see - unhackable, provided the software on the key storage box is trusted.
The only thing that struck me while writing this, is that we need some way of telling the storage box the amount to send. This can't really be included in the signed wire transfer message, as the customer shouldn't be in charge of this. It seems this would be an unsafe operation. Ie. a customer could place an order, hack into the web server, and change the amount being sent to his address, since the storage box can't know the bitcoin exchange rate, since it isn't online. I guess we'd need Mt. Gox to sign a piece of data containing the current time and exchange rate, so the box can know how many coins to send.

Quote
Why do you need any signing?  Why not just use that device to facilitate passing ASCII-only between the two systems?  Any non-ASCII buffers are rejected, and the offline computer will do nothing except collect the data and wait for the user to act on the received data.  Really, it would just behave like a serial port.
Are you imagining three devices? I only imagine two: the web server and the device. The device stores the keys, and does the signing - provided it receives a properly signed message that tells it where to send the coins.

Quote
Damnit!  I wish I could make the serial port idea work...
You and me both! I think I'll be looking into getting the Raspberry Pi to act as a USB client. It would then be necessary to use the USB protocol, which is a lot more complicated than just serial, but would not - as far as I know - allow login to the offline machine.

EDIT: I just discovered that the Raspberry Pi Model A (as opposed to Model B that is out now) will support acting as a USB device/client, because it uses a different communication-chip than Model B. Model A doesn't have an Ethernet port, and only has a single USB port, but I figure this might even be an advantage, since after the software has been installed on the device (using, for example, some USB-to-Ethernet type device to update packages), the Ethernet port will have to be removed because the USB connection needs to be used to communicate with the online computer. Perhaps one will be less tempted - and feel safer - if the device doesn't even have a port to plug in an Ethernet cable.
Additionally, it costs only $25. That's really not much for the ultimate in safe Bitcoin wallets. That being said, Model A isn't out now. And USB client mode won't necessarily work out-of-the-box. This is a software issue, as far as I can gather. Not OS level, I think, but firmware level - so it might not be that easy to implement. But still, looks like a device is coming to the market that fulfils the requirements of an offline hardware wallet. Off the top of my head, I can't see what else one would need other than a Raspberry Pi Model A, and perhaps a USB cable. But most people have USB cables lying around. Sure beats that $40 2xUSB-to-serial-plus-null-modem solution.


Title: Re: Armory on web servers
Post by: ErebusBat on August 26, 2012, 01:34:31 AM
I would like to point out that you do not have to use USB. Ethernet would actually work fine for this. Just have to cards in the webserver; one for Internet and the other for the key-box.

Then you can just set static addresses (or even run a dhcp server on the keybof, but that increases your attack surface).

Anyway you achieve the same goal. There is only one connection to the keybox and it is not on the interwebz.


Title: Re: Armory on web servers
Post by: runeks on August 30, 2012, 07:29:45 PM
That's true. The thing is though, that I imagine IP connectivity opens up a lot more potential attack surfaces than a USB connection. I mean, any program on the key box listening for incoming connections will suddenly be an attack surface. And it's really hard to control which programs do this; a lot will. With the device acting as a USB client this won't be an issue. You can't - not as far as I know at least - "listen" for connections via USB, and so, there will only be one central point - one attack surface - that we need to secure, which is the USB driver and the program that communicates over this connection. If this is secure, the key box should be secure.


Title: Re: Armory on web servers
Post by: ErebusBat on August 30, 2012, 07:33:40 PM
That's true. The thing is though, that I imagine IP connectivity opens up a lot more potential attack surfaces than a USB connection. I mean, any program on the key box listening for incoming connections will suddenly be an attack surface. And it's really hard to control which programs do this; a lot will. With the device acting as a USB client this won't be an issue. You can't - not as far as I know at least - "listen" for connections via USB, and so, there will only be one central point - one attack surface - that we need to secure, which is the USB driver and the program that communicates over this connection. If this is secure, the key box should be secure.

I see you point, but a slim install and a firewall would be all you need.  A lockdown of 'only allow connection to port XYZ from host ABC' would be sufficnt.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on September 17, 2012, 04:52:21 AM
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon)

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

getbalance:
Returns a decimal value in BTC for the total remaining balance in the wallet.

getnewaddress:
Returns the next address in the wallet as a string.

getreceivedbyaddress:
Returns a decimal value in BTC for the amount received by the address.

sendtoaddress:
Returns an unsigned transaction as a string. Implementation of signing and broadcasting is left to the client.



Features that may be included in the future:
User authentication
SSL
More API methods as required (the current set fills my needs). listtransactions is one method I am looking to include in the future.
Please suggest anything else you personally may find useful, as I have designed this only to fill my specific needs.



Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on September 17, 2012, 05:09:43 AM
Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?

-  There is full re-org/double-spend handling, which has been tested (https://bitcointalk.org/index.php?topic=46370.0) at the library level.   But it hasn't been tested at the GUI/interface level, because I never set up a way to test block/tx injection over the networking interface.  I see re-orgs happen all the time -- you'll see output to the console that says "Invalidating old chain tx -- Marking new chain tx valid".  For web-servers, this should be all you need. 

- Armory is pretty dumb when it comes to... lots of network-stuff.  It was written with the assumption that Bitcoin-Qt is going to feed it trustworthy, reasonable data.  I have noticed, under strange testing conditions, if a conflicting tx happens to make it in, it will show up in the ledger but it actually won't affect your balance at all.  Again, this is not all that relevant for web-servers, but it's worth noting:  Armory receives the conflicting tx, and immediately detects it's relevant, and adds it to the ledger on the GUI.  Then Armory tries to add it to it's pool of tx for each address, etc, but finds those outputs have already been spent.  So it gets dropped, and the underlying UTXO list is untouched.  It rarely ever matters, because Bitcoin-Qt/bitcoind can't pass it conflicting tx...

So I need to fix the auto-display-in-ledger bug.  But balances and spendability should maintain correctness in this situation.  Please let me know if they appear not to.

As for storing data between loads:  some people complain about the load times, etc, but it comes with a tremendous benefit of robustness, because there's no way for Armory to ever be "out of sync" with the blockchain.  One issue with Bitcoin-Qt is that it can mark outputs as spent, or believe that some transaction succeeded that wasn't accepted and get stuck in a state that isn't consistent with the rest of the network.  Armory avoids all this by having no memory.  At the expense of load times.

This functionality will eventually have to be added to do this (especially with the blockchain size getting so big), but right now it's not there, and it will probably require a lot of testing.  If the server is going to process lots and lots of transactions separately, you might consider taking my examples above for creating transactions, and add that memory/tracking around it (maintain the tracking yourself).  Maybe that's too much work...  I don't know.




Btw, I haven't looked at your stuff yet, but this sounds fantastic!  Thanks so much for doing this, and the JSON-RPC interface is something I've wanted to do for a while.  Perhaps we'll find a way to merge functionality together...


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: Herbert on September 17, 2012, 12:09:52 PM
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon)

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

This looks very interesting. Combining this with bitcoinmonitor.net url callbacks (using the API) will provide you a complete solution to accept payments without the need to put your hot wallet online anywhere and without the hassle to set up/maintain a long list of pre-generated addresses with your bitcoinmonitor.net agent. Awesome!  8)
As soon as I find some time I will set this scenario up for one of my sites.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on September 18, 2012, 06:06:06 AM
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon)

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

This looks very interesting. Combining this with bitcoinmonitor.net url callbacks (using the API) will provide you a complete solution to accept payments without the need to put your hot wallet online anywhere and without the hassle to set up/maintain a long list of pre-generated addresses with your bitcoinmonitor.net agent. Awesome!  8)
As soon as I find some time I will set this scenario up for one of my sites.

The api for everything except getnewaddress depends on loading the blockchain. If you stripped it right down to just that one call, then yes it would be great when combined with bitcoinmonitor.net, and as you say, it's not even required to have bitcoin running. But for any functionality beyond getting new addresses from the wallet, you will still need to have bitcoin running so there's access to the blockchain.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: Morranr on September 22, 2012, 05:33:29 AM
Just thought to check this thread again, and that daemon looks like it could be extremely useful.  Thanks for working on this and putting it out for the public to use!

I was pointed to this thread by etothepi when I sent an email asking about generating addresses for my server to use, and ended up deciding that the best implementation for my needs was to generate chunks of addresses at a time with the method described in the first post, and putting them into a redis list (acting as a queue) which would get refilled when it dropped below a certain amount of addresses.  I'll probably stick with my current implementation as it is more than sufficient for what I need right now, but I will certainly keep an eye on this daemon and might decide to switch to it later on.  :)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: unclescrooge on October 09, 2012, 03:02:56 PM
Hi mav,

You daemon is very very useful. However I haven't been able to make any function beside getNextUnusedAddress work. getbalance() return 0 even though there's one adress with 1 btc on it, listransactions doesn't return anything (but seems like the function isn't developped yet in your code) and getreceivedbyaddress raises an unknown error.

Did you manage to make it work?

Any way I'm not a python programmer but I'll try to debug that and post what I found, if any :)

unclescrooge


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on October 16, 2012, 10:04:21 AM
Hi mav,

You daemon is very very useful. However I haven't been able to make any function beside getNextUnusedAddress work. getbalance() return 0 even though there's one adress with 1 btc on it, listransactions doesn't return anything (but seems like the function isn't developped yet in your code) and getreceivedbyaddress raises an unknown error.

Did you manage to make it work?

Any way I'm not a python programmer but I'll try to debug that and post what I found, if any :)

unclescrooge

Sorry for the slow response, I am subscribed to this thread but very rarely come on this forum any more so have not been updated on the new posts. I recently started at a new job which has sadly diverted my attention away from bitcoin dev.

I will be looking at this code on the weekend. I made some changes a while ago but never pushed them to git. By the end of the weekend check back on this thread, there should be updates. It's been a while since I've looked at this daemon, and certainly have not been using it in production on any of my systems. But from memory I also could only get getNextUnusedAddress to work properly, although the other calls did something slightly related to the required task, just not something completely useful yet.

The end of the weekend - I am sure some updates will be ready.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: unclescrooge on October 16, 2012, 10:16:33 AM
Thanks for the response mav. Although I never touched python in my life nor am I a programmer, I added some codes from ArmoryQt and manage to get getbalance and listtransactions working (for this last only a few items are returned and some are wrong so I need to work more on that).

Here is yur modified code so far, ugly as hell lol:
Code:
################################################################################
#
# Copyright (C) 2012, Ian Coleman
# Distributed under the GNU Affero General Public License (AGPL v3)
# See http://www.gnu.org/licenses/agpl.html
#
################################################################################

from twisted.web import server
from twisted.internet import reactor
from txjsonrpc.web import jsonrpc

import decimal
import os

RPC_PORT = 7070
STANDARD_FEE = 0.0005 # BTC
       
class Wallet_Json_Rpc_Server(jsonrpc.JSONRPC):

    def __init__(self, wallet):
        self.wallet = wallet

    def jsonrpc_getnewaddress(self):
        addr = self.wallet.getNextUnusedAddress()
        return addr.getAddrStr()

    def jsonrpc_getbalance(self):
        int_balance = self.wallet.getBalance()
        decimal_balance = decimal.Decimal(int_balance) / decimal.Decimal(ONE_BTC)
        return float(decimal_balance)

    def jsonrpc_getreceivedbyaddress(self, address):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot get received amount when offline')
        # Only gets correct amount for addresses in the wallet, otherwise 0
        addr160 = addrStr_to_hash160(address)
        txs = self.wallet.getAddrTxLedger(addr160)
        balance = sum([x.getValue() for x in txs if x.getValue() > 0])
        decimal_balance = decimal.Decimal(balance) / decimal.Decimal(ONE_BTC)
        float_balance = float(decimal_balance)
        return float_balance

    def jsonrpc_sendtoaddress(self, bitcoinaddress, amount):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot create transactions when offline')
        return self.create_unsigned_transaction(bitcoinaddress, amount)

    def jsonrpc_listtransactions(self, p_count=10, p_from=0):
        #TODO this needs more work
        result2 = []
        txs = self.wallet.getTxLedger('blk')
        #txs = json.dumps(txs)
        #id_le_pairs = [[le] for le in txs]
        for x in txs:
            #result.append('{')
            account = ''
            txHashBin = x.getTxHash()#hex_to_binary(x.getTxHash())   
            cppTx = TheBDM.getTxByHash(txHashBin)
            pytx = PyTx().unserialize(cppTx.serialize())
            for txout in pytx.outputs:
                scrType = getTxOutScriptType(txout.binScript)
                if not scrType in (TXOUT_SCRIPT_STANDARD, TXOUT_SCRIPT_COINBASE):
                  continue
                address = hash160_to_addrStr(TxOutScriptExtractAddr160(txout.binScript))
            if x.getValue() < 0:
            category = 'send'
            else:
            category = 'receive'
            amount = x.getValue()
            confirmations = TheBDM.getTopBlockHeader().getBlockHeight() - x.getBlockNum()+1
            blockhash = ''
            blockindex = ''#x.getBlockNum()
            txid = str(binary_to_hex(x.getTxHash()))
            time = ''#x.getTxTime()
            result = {'account':account,'address':address,'category':category,'amount':amount,'confirmations':confirmations,'blockhash':blockhash,'blockindex':blockindex,'txid':txid,'time:':time}
            #result.append('}')
            result2.append(result)
            #print addr
            #dumpObj(x)   
        #result = json.dumps(result)
        return result2

       
    # https://bitcointalk.org/index.php?topic=92496.msg1126310#msg1126310
    def create_unsigned_transaction(self, bitcoinaddress_str, amount_to_send_btc):
        # Get unspent TxOutList and select the coins
        addr160_recipient = addrStr_to_hash160(bitcoinaddress_str)
        totalSend, fee = long(amount_to_send_btc * ONE_BTC), (STANDARD_FEE * ONE_BTC)
        spendBal = self.wallet.getBalance('Spendable')
        utxoList = self.wallet.getTxOutList('Spendable')
        utxoSelect = PySelectCoins(utxoList, totalSend, fee)

        minFeeRec = calcMinSuggestedFees(utxoSelect, totalSend, fee)[1]
        if fee<minFeeRec:
            if totalSend + minFeeRec > spendBal:
                raise NotEnoughCoinsError, "You can't afford the fee!"
            utxoSelect = PySelectCoins(utxoList, totalSend, minFeeRec)
            fee = minFeeRec

        if len(utxoSelect)==0:
            raise CoinSelectError, "Somehow, coin selection failed.  This shouldn't happen"

        totalSelected = sum([u.getValue() for u in utxoSelect])
        totalChange = totalSelected - (totalSend  + fee)

        outputPairs = []
        outputPairs.append( [addr160_recipient, totalSend] )
        if totalChange > 0:
            outputPairs.append( [self.wallet.getNextUnusedAddress().getAddr160(), totalChange] )

        random.shuffle(outputPairs)
        txdp = PyTxDistProposal().createFromTxOutSelection(utxoSelect, outputPairs)

        return txdp.serializeAscii()
       
    def jsonrpc_createCombinedLedger(self, wltIDList=None, withZeroConf=True):
      """
      Create a ledger to display on the main screen, that consists of ledger
      entries of any SUBSET of available wallets.
      """

      start = RightNow()


      self.combinedLedger = []
      totalFunds  = 0
      spendFunds  = 0
      unconfFunds = 0
      currBlk = 0xffffffff
      if TheBDM.isInitialized():
         currBlk = TheBDM.getTopBlockHeader().getBlockHeight()

      wlt = self.waldlet
      totalFunds += wlt.getBalance('Total')
      spendFunds += wlt.getBalance('Spendable')
      unconfFunds += wlt.getBalance('Unconfirmed')


      self.combinedLedger.sort(key=lambda x: x[LEDGERCOLS.UnixTime], reverse=True)
      self.ledgerSize = len(self.combinedLedger)
     
      print totalFunds
      return

class Armory_Daemon():

    def __init__(self):

        print "Reading wallet file"
        self.wallet = self.find_wallet()
        self.loadBlockchain()
       
        use_blockchain = not CLI_OPTIONS.offline
       
        print "Initialising server"
        reactor.listenTCP(RPC_PORT, server.Site(Wallet_Json_Rpc_Server(self.wallet)))

        self.NetworkingFactory = ArmoryClientFactory( \
                                func_loseConnect=self.showOfflineMsg, \
                                func_madeConnect=self.showOnlineMsg, \
                                func_newTx=self.newTxFunc)
                               
        reactor.connectTCP('127.0.0.1', BITCOIN_PORT, self.NetworkingFactory)
        reactor.callLater(5, self.Heartbeat)
        self.start()

    def start(self):
        print "Server started"
        reactor.run()
           
    def newTxFunc(self, pytxObj):
        # Cut down version from ArmoryQt.py
        TheBDM.addNewZeroConfTx(pytxObj.serialize(), long(RightNow()), True)
        TheBDM.rescanWalletZeroConf(self.wallet.cppWallet)

        # TODO set up a 'subscribe' feature so these notifications can be
        # pushed out to interested parties.

        # From here down is display purposes only, copied from ArmoryQt.py
        message = "New TX"
        le = self.wallet.cppWallet.calcLedgerEntryForTxStr(pytxObj.serialize())
        if not le.isSentToSelf():
            txref = TheBDM.getTxByHash(le.getTxHash())
            nOut = txref.getNumTxOut()
            recips = [txref.getTxOut(i).getRecipientAddr() for i in range(nOut)]
            values = [txref.getTxOut(i).getValue()         for i in range(nOut)]
            idxMine  = filter(lambda i:     self.wallet.hasAddr(recips[i]), range(nOut))
            idxOther = filter(lambda i: not self.wallet.hasAddr(recips[i]), range(nOut))
            mine  = [(recips[i],values[i]) for i in idxMine]
            other = [(recips[i],values[i]) for i in idxOther]

            # Collected everything we need to display, now construct it and do it
            if le.getValue()>0:
               # Received!
               message = 'Bitcoins Received!'
               totalStr = coin2str( sum([mine[i][1] for i in range(len(mine))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(mine)==1:
                  message += '\nAddress:\t%s' % hash160_to_addrStr(mine[0][0])
                  addrComment = self.wallet.getComment(mine[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  message += '\n<Received with Multiple Addresses>'
            elif le.getValue()<0:
               # Sent!
               message = 'Bitcoins Sent!'
               totalStr = coin2str( sum([other[i][1] for i in range(len(other))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(other)==1:
                  message += 'Sent To:\t%s' % hash160_to_addrStr(other[0][0])
                  addrComment = self.wallet.getComment(other[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  dispLines.append('<Sent to Multiple Addresses>')
        else:
            amt = self.determineSentToSelfAmt(le, self.wallet)[0]
            message = 'Wallet "%s" just sent %s BTC to itself!' % \
               (self.wallet.labelName, coin2str(amt,maxZeros=1).strip())
               
        print message

    def determineSentToSelfAmt(self, le, wlt):
      """
      NOTE:  this method works ONLY because we always generate a new address
             whenever creating a change-output, which means it must have a
             higher chainIndex than all other addresses.  If you did something
             creative with this tx, this may not actually work.
      """
      amt = 0
      if TheBDM.isInitialized() and le.isSentToSelf():
         txref = TheBDM.getTxByHash(le.getTxHash())
         if not txref.isInitialized():
            return (0, 0)
         if txref.getNumTxOut()==1:
            return (txref.getTxOut(0).getValue(), -1)
         maxChainIndex = -5
         txOutChangeVal = 0
         txOutIndex = -1
         valSum = 0
         for i in range(txref.getNumTxOut()):
            valSum += txref.getTxOut(i).getValue()
            addr160 = txref.getTxOut(i).getRecipientAddr()
            addr    = wlt.getAddrByHash160(addr160)
            if addr and addr.chainIndex > maxChainIndex:
               maxChainIndex = addr.chainIndex
               txOutChangeVal = txref.getTxOut(i).getValue()
               txOutIndex = i

         amt = valSum - txOutChangeVal
      return (amt, txOutIndex)

    def showOfflineMsg(self):
        print "Offline - not tracking blockchain"

    def showOnlineMsg(self):
        print "Online - tracking blockchain"

    def find_wallet(self):
        fnames = os.listdir(os.getcwd())
        for fname in fnames:
            is_wallet = fname[-7:] == ".wallet"
            is_watchonly = fname.find("watchonly") > -1
            is_backup = fname.find("backup") > -1
            if(is_wallet and is_watchonly and not is_backup):
                wallet = PyBtcWallet().readWalletFile(fname)
                # Register all wallets with TheBDM
                TheBDM.registerWallet( wallet.cppWallet )
                print "Using wallet file %s" % fname
                return wallet
        raise ValueError('Unable to locate a watch-only wallet in %s' % os.getcwd())
           
    def loadBlockchain(self):

      print "Loading blockchain"
      BDM_LoadBlockchainFile()
      self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
   
      # Now that theb blockchain is loaded, let's populate the wallet info
      if TheBDM.isInitialized():
         mempoolfile = os.path.join(ARMORY_HOME_DIR,'mempool.bin')
         self.checkMemoryPoolCorruption(mempoolfile)
         TheBDM.enableZeroConf(mempoolfile)
   
        # self.statusBar().showMessage('Syncing wallets with blockchain...')
         print "Syncing wallets with blockchain..."
         print "Syncing wallet: ", self.wallet.uniqueIDB58
         self.wallet.setBlockchainSyncFlag(BLOCKCHAIN_READONLY)
         self.wallet.syncWithBlockchain()
             
               
         #self.createCombinedLedger()
         #self.ledgerSize = len(self.combinedLedger)
         #self.statusBar().showMessage('Blockchain loaded, wallets sync\'d!', 10000)

    def checkMemoryPoolCorruption(self, mempoolname):
      if not os.path.exists(mempoolname):
         return

      memfile = open(mempoolname, 'r')
      memdata = memfile.read()
      memfile.close()

      binunpacker = BinaryUnpacker(memdata)
      try:
         while binunpacker.getRemainingSize() > 0:
            binunpacker.get(UINT64)
            PyTx().unserialize(binunpacker)
      except:
         os.remove(mempoolname);
         LOGWARN('Memory pool file was corrupt.  Deleted. (no further action is needed)') 
               
    def Heartbeat(self, nextBeatSec=2):
       """
       This method is invoked when the app is initialized, and will
       run every 2 seconds, or whatever is specified in the nextBeatSec
       argument.
       """
       # Check for new blocks in the blk0001.dat file
       if TheBDM.isInitialized():
          newBlks = TheBDM.readBlkFileUpdate()
          self.topTimestamp   = TheBDM.getTopBlockHeader().getTimestamp()
          if newBlks>0:
             self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
             LOGINFO('New Block! : %d', self.latestBlockNum)
             didAffectUs = False
             prevLedgerSize = len(self.wallet.getTxLedger())
             self.wallet.syncWithBlockchain()
             TheBDM.rescanWalletZeroConf(self.cppWallet)

       self.wallet.checkWalletLockTimeout()

       reactor.callLater(nextBeatSec, self.Heartbeat)
       
## {{{ http://code.activestate.com/recipes/137951/ (r1)
def printDict(di, format="%-25s %s"):
    for (key, val) in di.items():
        print format % (str(key)+':', val)

def delchars(str, chars):
    """Returns a string for which all occurrences of characters in
    chars have been removed."""

    # Translate demands a mapping string of 256 characters;
    # whip up a string that will leave all characters unmolested.
    identity = ''.join([chr(x) for x in range(256)])

    return str.translate(identity, chars)
## end of http://code.activestate.com/recipes/137951/ }}}


if __name__ == "__main__":
    from armoryengine import *
    rpc_server = Armory_Daemon()


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: unclescrooge on October 19, 2012, 10:15:48 AM
Ok so I updated the previous code which didn't work at all.

Now I manage to get working  "getbalance()", "listtransactions()" (only a few items that I think important are returned though), "sendtoaddress()" (create a file containing the unsigned transaction in the current directory and return a random string).

Do not use in production server without testing though:

Code:
################################################################################
#
# Copyright (C) 2012, Ian Coleman
# Distributed under the GNU Affero General Public License (AGPL v3)
# See http://www.gnu.org/licenses/agpl.html
#
################################################################################

from twisted.web import server
from twisted.internet import reactor
from txjsonrpc.web import jsonrpc

import decimal
import os

RPC_PORT = 7070
STANDARD_FEE = 0.0005 # BTC
       
class Wallet_Json_Rpc_Server(jsonrpc.JSONRPC):

    def __init__(self, wallet):
        self.wallet = wallet

    def jsonrpc_getnewaddress(self):
        addr = self.wallet.getNextUnusedAddress()
        return addr.getAddrStr()

    def jsonrpc_getbalance(self):
        int_balance = self.wallet.getBalance()
        decimal_balance = decimal.Decimal(int_balance) / decimal.Decimal(ONE_BTC)
        return float(decimal_balance)

    def jsonrpc_getreceivedbyaddress(self, address):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot get received amount when offline')
        # Only gets correct amount for addresses in the wallet, otherwise 0
        addr160 = addrStr_to_hash160(address)
        txs = self.wallet.getAddrTxLedger(addr160)
        balance = sum([x.getValue() for x in txs if x.getValue() > 0])
        decimal_balance = decimal.Decimal(balance) / decimal.Decimal(ONE_BTC)
        float_balance = float(decimal_balance)
        return float_balance

    def jsonrpc_sendtoaddress(self, bitcoinaddress, amount):
        if CLI_OPTIONS.offline:
            raise ValueError('Cannot create transactions when offline')
        return self.create_unsigned_transaction(bitcoinaddress, amount)

    def jsonrpc_listtransactions(self, p_count=10, p_from=0):
        #TODO this needs more work
        result2 = []
        txs = self.wallet.getTxLedger('blk')
        #txs = json.dumps(txs)
        #id_le_pairs = [[le] for le in txs]
        for x in txs:
            #result.append('{')
            account = ''
            txHashBin = x.getTxHash()#hex_to_binary(x.getTxHash())   
            cppTx = TheBDM.getTxByHash(txHashBin)
            pytx = PyTx().unserialize(cppTx.serialize())
            for txout in pytx.outputs:
                scrType = getTxOutScriptType(txout.binScript)
                if not scrType in (TXOUT_SCRIPT_STANDARD, TXOUT_SCRIPT_COINBASE):
                  continue
                address = hash160_to_addrStr(TxOutScriptExtractAddr160(txout.binScript))
                #wltID = self.wallet.getWalletForAddr160(a160)
                if self.wallet.hasAddr(address) == False:
                  continue
                else:
                break
            if x.getValue() < 0:
            category = 'send'
            else:
            category = 'receive'
            amount = float(decimal.Decimal(x.getValue()) / decimal.Decimal(ONE_BTC))
            confirmations = TheBDM.getTopBlockHeader().getBlockHeight() - x.getBlockNum()+1
            blockhash = ''
            blockindex = ''#x.getBlockNum()
            txid = str(binary_to_hex(x.getTxHash()))
            time = ''#x.getTxTime()
            result = {'account':account,'address':address,'category':category,'amount':amount,'confirmations':confirmations,'blockhash':blockhash,'blockindex':blockindex,'txid':txid,'time:':time}
            #result.append('}')
            result2.append(result)
            #print addr
            #dumpObj(x)   
        #result = json.dumps(result)
        return result2

       
    # https://bitcointalk.org/index.php?topic=92496.msg1126310#msg1126310
    def create_unsigned_transaction(self, bitcoinaddress_str, amount_to_send_btc):
        # Get unspent TxOutList and select the coins
        addr160_recipient = addrStr_to_hash160(bitcoinaddress_str)
        totalSend, fee = long(amount_to_send_btc * ONE_BTC), (STANDARD_FEE * ONE_BTC)
        spendBal = self.wallet.getBalance('Spendable')
        utxoList = self.wallet.getTxOutList('Spendable')
        utxoSelect = PySelectCoins(utxoList, totalSend, fee)

        minFeeRec = calcMinSuggestedFees(utxoSelect, totalSend, fee)[1]
        if fee<minFeeRec:
            if totalSend + minFeeRec > spendBal:
                raise NotEnoughCoinsError, "You can't afford the fee!"
            utxoSelect = PySelectCoins(utxoList, totalSend, minFeeRec)
            fee = minFeeRec

        if len(utxoSelect)==0:
            raise CoinSelectError, "Somehow, coin selection failed.  This shouldn't happen"

        totalSelected = sum([u.getValue() for u in utxoSelect])
        totalChange = totalSelected - (totalSend  + fee)

        outputPairs = []
        outputPairs.append( [addr160_recipient, totalSend] )
        if totalChange > 0:
            outputPairs.append( [self.wallet.getNextUnusedAddress().getAddr160(), totalChange] )

        random.shuffle(outputPairs)
        txdp = PyTxDistProposal().createFromTxOutSelection(utxoSelect, outputPairs)
        outfilename = str(int(time.time())) + '.unsigned.tx'
        output = open(outfilename, 'w')
        output.write(txdp.serializeAscii())
        output.close()
        result2 = []
        result = {"Code":""}
        result2.append(result)     
        return "success"+str(int(time.time()))
       
    def jsonrpc_createCombinedLedger(self, wltIDList=None, withZeroConf=True):
      """
      Create a ledger to display on the main screen, that consists of ledger
      entries of any SUBSET of available wallets.
      """

      start = RightNow()


      self.combinedLedger = []
      totalFunds  = 0
      spendFunds  = 0
      unconfFunds = 0
      currBlk = 0xffffffff
      if TheBDM.isInitialized():
         currBlk = TheBDM.getTopBlockHeader().getBlockHeight()

      wlt = self.waldlet
      totalFunds += wlt.getBalance('Total')
      spendFunds += wlt.getBalance('Spendable')
      unconfFunds += wlt.getBalance('Unconfirmed')


      self.combinedLedger.sort(key=lambda x: x[LEDGERCOLS.UnixTime], reverse=True)
      self.ledgerSize = len(self.combinedLedger)
     
      print totalFunds
      return

class Armory_Daemon():

    def __init__(self):

        print "Reading wallet file"
        self.wallet = self.find_wallet()
        self.loadBlockchain()
       
        use_blockchain = not CLI_OPTIONS.offline
       
        print "Initialising server"
        reactor.listenTCP(RPC_PORT, server.Site(Wallet_Json_Rpc_Server(self.wallet)))

        self.NetworkingFactory = ArmoryClientFactory( \
                                func_loseConnect=self.showOfflineMsg, \
                                func_madeConnect=self.showOnlineMsg, \
                                func_newTx=self.newTxFunc)
                               
        reactor.connectTCP('127.0.0.1', BITCOIN_PORT, self.NetworkingFactory)
        reactor.callLater(5, self.Heartbeat)
        self.start()

    def start(self):
        print "Server started"
        reactor.run()
       
    def newTxFunc(self, pytxObj):
        # Cut down version from ArmoryQt.py
        TheBDM.addNewZeroConfTx(pytxObj.serialize(), long(RightNow()), True)
        TheBDM.rescanWalletZeroConf(self.wallet.cppWallet)

        # TODO set up a 'subscribe' feature so these notifications can be
        # pushed out to interested parties.

        # From here down is display purposes only, copied from ArmoryQt.py
        message = "New TX"
        le = self.wallet.cppWallet.calcLedgerEntryForTxStr(pytxObj.serialize())
        if not le.isSentToSelf():
            txref = TheBDM.getTxByHash(le.getTxHash())
            nOut = txref.getNumTxOut()
            recips = [txref.getTxOut(i).getRecipientAddr() for i in range(nOut)]
            values = [txref.getTxOut(i).getValue()         for i in range(nOut)]
            idxMine  = filter(lambda i:     self.wallet.hasAddr(recips[i]), range(nOut))
            idxOther = filter(lambda i: not self.wallet.hasAddr(recips[i]), range(nOut))
            mine  = [(recips[i],values[i]) for i in idxMine]
            other = [(recips[i],values[i]) for i in idxOther]

            # Collected everything we need to display, now construct it and do it
            if le.getValue()>0:
               # Received!
               message = 'Bitcoins Received!'
               totalStr = coin2str( sum([mine[i][1] for i in range(len(mine))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(mine)==1:
                  message += '\nAddress:\t%s' % hash160_to_addrStr(mine[0][0])
                  addrComment = self.wallet.getComment(mine[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  message += '\n<Received with Multiple Addresses>'
            elif le.getValue()<0:
               # Sent!
               message = 'Bitcoins Sent!'
               totalStr = coin2str( sum([other[i][1] for i in range(len(other))]), maxZeros=1)
               message += '\nAmount: \t%s BTC' % totalStr.strip()
               if len(other)==1:
                  message += 'Sent To:\t%s' % hash160_to_addrStr(other[0][0])
                  addrComment = self.wallet.getComment(other[0][0])
                  #if addrComment:
                     #message += '\n%s...' % addrComment[:24]
               else:
                  dispLines.append('<Sent to Multiple Addresses>')
        else:
            amt = self.determineSentToSelfAmt(le, self.wallet)[0]
            message = 'Wallet "%s" just sent %s BTC to itself!' % \
               (self.wallet.labelName, coin2str(amt,maxZeros=1).strip())
               
        print message

    def determineSentToSelfAmt(self, le, wlt):
      """
      NOTE:  this method works ONLY because we always generate a new address
             whenever creating a change-output, which means it must have a
             higher chainIndex than all other addresses.  If you did something
             creative with this tx, this may not actually work.
      """
      amt = 0
      if TheBDM.isInitialized() and le.isSentToSelf():
         txref = TheBDM.getTxByHash(le.getTxHash())
         if not txref.isInitialized():
            return (0, 0)
         if txref.getNumTxOut()==1:
            return (txref.getTxOut(0).getValue(), -1)
         maxChainIndex = -5
         txOutChangeVal = 0
         txOutIndex = -1
         valSum = 0
         for i in range(txref.getNumTxOut()):
            valSum += txref.getTxOut(i).getValue()
            addr160 = txref.getTxOut(i).getRecipientAddr()
            addr    = wlt.getAddrByHash160(addr160)
            if addr and addr.chainIndex > maxChainIndex:
               maxChainIndex = addr.chainIndex
               txOutChangeVal = txref.getTxOut(i).getValue()
               txOutIndex = i

         amt = valSum - txOutChangeVal
      return (amt, txOutIndex)

    def showOfflineMsg(self):
        print "Offline - not tracking blockchain"

    def showOnlineMsg(self):
        print "Online - tracking blockchain"

    def find_wallet(self):
        fnames = os.listdir(os.getcwd())
        for fname in fnames:
            is_wallet = fname[-7:] == ".wallet"
            is_watchonly = fname.find("watchonly") > -1
            is_backup = fname.find("backup") > -1
            if(is_wallet and is_watchonly and not is_backup):
                wallet = PyBtcWallet().readWalletFile(fname)
                # Register all wallets with TheBDM
                TheBDM.registerWallet( wallet.cppWallet )
                print "Using wallet file %s" % fname
                return wallet
        raise ValueError('Unable to locate a watch-only wallet in %s' % os.getcwd())
           
    def loadBlockchain(self):

      print "Loading blockchain"
      BDM_LoadBlockchainFile()
      self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
   
      # Now that theb blockchain is loaded, let's populate the wallet info
      if TheBDM.isInitialized():
         mempoolfile = os.path.join(ARMORY_HOME_DIR,'mempool.bin')
         self.checkMemoryPoolCorruption(mempoolfile)
         TheBDM.enableZeroConf(mempoolfile)
   
        # self.statusBar().showMessage('Syncing wallets with blockchain...')
         print "Syncing wallets with blockchain..."
         print "Syncing wallet: ", self.wallet.uniqueIDB58
         self.wallet.setBlockchainSyncFlag(BLOCKCHAIN_READONLY)
         self.wallet.syncWithBlockchain()
             
               
         #self.createCombinedLedger()
         #self.ledgerSize = len(self.combinedLedger)
         #self.statusBar().showMessage('Blockchain loaded, wallets sync\'d!', 10000)

    def checkMemoryPoolCorruption(self, mempoolname):
      if not os.path.exists(mempoolname):
         return

      memfile = open(mempoolname, 'r')
      memdata = memfile.read()
      memfile.close()

      binunpacker = BinaryUnpacker(memdata)
      try:
         while binunpacker.getRemainingSize() > 0:
            binunpacker.get(UINT64)
            PyTx().unserialize(binunpacker)
      except:
         os.remove(mempoolname);
         LOGWARN('Memory pool file was corrupt.  Deleted. (no further action is needed)') 
               
    def Heartbeat(self, nextBeatSec=2):
       """
       This method is invoked when the app is initialized, and will
       run every 2 seconds, or whatever is specified in the nextBeatSec
       argument.
       """
       # Check for new blocks in the blk0001.dat file
       if TheBDM.isInitialized():
          newBlks = TheBDM.readBlkFileUpdate()
          self.topTimestamp   = TheBDM.getTopBlockHeader().getTimestamp()
          if newBlks>0:
             self.latestBlockNum = TheBDM.getTopBlockHeader().getBlockHeight()
             didAffectUs = False
             prevLedgerSize = len(self.wallet.getTxLedger())
             self.wallet.syncWithBlockchain()
             TheBDM.rescanWalletZeroConf(self.wallet.cppWallet)

       self.wallet.checkWalletLockTimeout()

       reactor.callLater(nextBeatSec, self.Heartbeat)
 


if __name__ == "__main__":
    from armoryengine import *
    rpc_server = Armory_Daemon()


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on October 21, 2012, 04:25:07 AM
Ok so I updated the previous code which didn't work at all.
...

Massive kudos for making this happen unclescrooge. Armory takes a fair bit of poking around, so it seems you've done well with this.

I added the code and have written some basic tests, unfortunately time is unexpectedly short for me this weekend so later in the week I will try adding the missing fields from listtransactions (and properly confirm that the existing fields are correct).

See the latest on github for the changes. I appreciate the efforts you have put in unclescrooge. It's nice to know people are looking at this, gives me motivation to continue on it. I was planning to use this in a project I was working on, but that has been put on the backburner and as such armory-daemon has languished.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: unclescrooge on October 22, 2012, 08:46:03 AM
Ok so I updated the previous code which didn't work at all.
...

Massive kudos for making this happen unclescrooge. Armory takes a fair bit of poking around, so it seems you've done well with this.

I added the code and have written some basic tests, unfortunately time is unexpectedly short for me this weekend so later in the week I will try adding the missing fields from listtransactions (and properly confirm that the existing fields are correct).

See the latest on github for the changes. I appreciate the efforts you have put in unclescrooge. It's nice to know people are looking at this, gives me motivation to continue on it. I was planning to use this in a project I was working on, but that has been put on the backburner and as such armory-daemon has languished.

Well thank you, you did the most part, and I had an interest in this as I'm launching a new bitcoin service using this script.

I hope you tested it before pushing to git, I did but the more testing the better. Thanks for pushing it to github anyway :)

Anyway I'll keep updating the code if I find so way to improve it with. But for now I have to focus on a way to generate an offline TX and push it to an offsite computer to be signed and broadcast immediately. Because if you don't broadcast your unsigned tx immediately, the next generated one should be incorrect :/


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on October 22, 2012, 09:50:42 AM
Anyway I'll keep updating the code if I find so way to improve it with. But for now I have to focus on a way to generate an offline TX and push it to an offsite computer to be signed and broadcast immediately. Because if you don't broadcast your unsigned tx immediately, the next generated one should be incorrect :/

This is an interesting point you make. On one hand it seems fairly certain you are right about this, but maybe etotheipi can confirm whether the wallet can handle the creation of multiple unsigned transactions at a time? It seems like a fairly likely situation to happen under most normal use cases, even with the gui client. If the daemon software can avoid having to implement a strict 'sequential' series of transaction where the server must create-a-tx-then-announce-before-creating-the-next-tx, it would make life much much easier.

TBH I always assumed transactions could be grouped and processed offline as a batch. If armory does require one-at-a-time, that would become a very limiting factor for even a moderately popular web-based service.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: unclescrooge on October 22, 2012, 11:10:37 AM
Anyway I'll keep updating the code if I find so way to improve it with. But for now I have to focus on a way to generate an offline TX and push it to an offsite computer to be signed and broadcast immediately. Because if you don't broadcast your unsigned tx immediately, the next generated one should be incorrect :/

This is an interesting point you make. On one hand it seems fairly certain you are right about this, but maybe etotheipi can confirm whether the wallet can handle the creation of multiple unsigned transactions at a time? It seems like a fairly likely situation to happen under most normal use cases, even with the gui client. If the daemon software can avoid having to implement a strict 'sequential' series of transaction where the server must create-a-tx-then-announce-before-creating-the-next-tx, it would make life much much easier.

TBH I always assumed transactions could be grouped and processed offline as a batch. If armory does require one-at-a-time, that would become a very limiting factor for even a moderately popular web-based service.

I tested several send transactions in a row (with same parameters), and for each transactions the inputs and outputs are the same. Which is logical since armory look at the existing adresses balances to construct a transaction, if a transaction is not broadcasted it can't be taken into account.

I don't know if etotheipi is gonna change this behavior in the future, but for now you have to broadcast an unsigned transaction before creating the other one, no other way around it.

What I'm thinking to get around this limitation is to have a computer, completely behind a firewall, which pull from production servers all unsigned txs when they are created, signed and broadcast them. Then it send a signal to the productions servers (delete the unsigned tx file?) to go ahead and create the next unsigned tx if needed.

That's not as secure as offline signing, especially if signing is automated, but I think it's still relatively secure compared to existing solution with bitcoind.


Title: Re: Armory on web servers
Post by: arsenische on November 07, 2012, 09:54:05 AM
There won't be any fireworks, but you might end up re-using an address by accident.

I am sorry, I am misunderstanding something. I like the idea of deterministic address generation without private keys. But does Armory deterministically separate change addresses from normal ones?

Let's say I don't want to use offline transactions. I just want to generate addresses with watching-only wallet (being run on the web server machine) and then spend the received bitcoins from the full wallet (being run on my desktop computer).

I like this idea since it doesn't require any synchronization between wallets.

But would it be safe, or would Armory generate change addresses that may conflict with addresses generated on server? Is there any chance to make this simple scheme work?


Title: Re: Armory on web servers
Post by: arsenische on November 07, 2012, 02:05:18 PM
But would it be safe, or would Armory generate change addresses that may conflict with addresses generated on server?

I made a simple experiment and noticed that change addresses come from the same sequence. Is there a way around this feature?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on November 07, 2012, 04:09:07 PM
You are right, there is no alternate chain for change addresses.  It is possible to end up with accidentally re-using an address once, especially if your server is very active in giving out addresses.

However, in version 0.82 I added a new feature to the Expert interface (in the GUI) that lets you specify the change address to use.  It allows you to send change back to the first input address (may be better than the alternative), or you can supply your own address.  You can't specify another wallet yet, but I want to allow that, eventually.

However, the new wallet format that will come after beta, is based on endless discussion with Pieter Wiulle (bitcoin-qt dev), and that will result in wallets, by default, having two different subchains for each wallet -- the second chain will be used solely for change.  The new wallet format is a ways off (hopefully Beta isn't!), but it is coming.  For now, it sounds like switching your desktop to Expert usermode is your solution.



Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: arsenische on November 07, 2012, 07:00:04 PM
You are right, there is no alternate chain for change addresses.  It is possible to end up with accidentally re-using an address once, especially if your server is very active in giving out addresses.

However, in version 0.82 I added a new feature to the Expert interface (in the GUI) that lets you specify the change address to use.  It allows you to send change back to the first input address (may be better than the alternative), or you can supply your own address.  You can't specify another wallet yet, but I want to allow that, eventually.

However, the new wallet format that will come after beta, is based on endless discussion with Pieter Wiulle (bitcoin-qt dev), and that will result in wallets, by default, having two different subchains for each wallet -- the second chain will be used solely for change.  The new wallet format is a ways off (hopefully Beta isn't!), but it is coming.  For now, it sounds like switching your desktop to Expert usermode is your solution.



Thanks! Great software!


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on January 06, 2013, 03:50:14 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)

Any ideas why these errors are being thrown? This is happening in TheBDM.run()

The daemon still runs and correctly responds to getnewaddress however I haven't got a suitable wallet to test the other calls, namely:
getbalance
getreceivedbyaddress
listtransactions
sendtoaddress

Please see https://github.com/thedawnrider/BitcoinArmory-Daemon/blob/master/armory-daemon.py (https://github.com/thedawnrider/BitcoinArmory-Daemon/blob/master/armory-daemon.py) for the source


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on January 06, 2013, 04:18:51 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)

Any ideas why these errors are being thrown? This is happening in TheBDM.run()

The daemon still runs and correctly responds to getnewaddress however I haven't got a suitable wallet to test the other calls, namely:
getbalance
getreceivedbyaddress
listtransactions
sendtoaddress

Please see https://github.com/thedawnrider/BitcoinArmory-Daemon/blob/master/armory-daemon.py (https://github.com/thedawnrider/BitcoinArmory-Daemon/blob/master/armory-daemon.py) for the source

Oh boy.  I forgot about this thread, and should've warned you that I converted Armory to a multi-threaded application.  As such, some of the interfaces changed a bit.  There is an example of blockchain loading in the extras/sample_armory_code.py file, but I'm not sure all the samples are up-to-date for the new multi-threaded code.  On the upside, just about any function that doesn't return something (like blockchain scanning) you can use wait=False and it will run in the background allowing you to do other operations while you wait. 

You'll see what I mean if you look at the blocking vs. non-blocking methods, shown in sample_armory_code.py:

Code:
################################################################################
if run_LoadBlockchain_Block:
   start = RightNow()
   TheBDM.setBlocking(True)
   TheBDM.setOnlineMode(True)
   # The setOnlineMode should block until blockchain loading is complete
   print 'Loading blockchain took %0.1f sec' % (RightNow() - start)

   topBlock = TheBDM.getTopBlockHeight()
   print '\n\nCurrent Top Block is:', topBlock
   TheBDM.getTopBlockHeader().pprint()

As above, if you set blocking to True, it should behave much like the original... the main thread always waits for a response from the BDM before continuing.  On the other hand, if blocking=False (default):

Code:
################################################################################
if run_LoadBlockchain_Async:
   start = RightNow()
   TheBDM.setBlocking(False)
   TheBDM.setOnlineMode(True)
   print 'Waiting for blockchain loading to finish',
   while not TheBDM.getBDMState()=='BlockchainReady':
      print '.',
      sleep(2)
      # do other stuff while waiting...
   print 'Loading blockchain took %0.1f sec' % (RightNow() - start)

   topBlock = TheBDM.getTopBlockHeight()
   print '\n\nCurrent Top Block is:', topBlock
   TheBDM.getTopBlockHeader().pprint()

In this case, the blockchain scan happens in the background and the main thread continues running immediately, giving you an opportunity to do other calculations or operations in the main thread.  If you use the setBlocking=False, you should ALWAYS check theBDMState() before making a call like getTopBlockHeader() which you expect to return data -- anything that returns data is blocking by default.  Use a conditional like this:

Code:
if TheBDM.getBDMState()=='BlockchainReady':
   # The blockchain is loaded and TheBDM is sitting idle waiting for something to do
elif TheBDM.getBDMState()=='Scanning':
   # Currently scanning the blockchain, go do something else
elif TheBDM.getBDMState()=='Offline':
   # Blockchain is not loaded and was not requested to be loaded (usually because of --offline)
elif TheBDM.getBDMState()=='Uninitialized':
   # The BDM expects it will be online at some point, but the blockchain isn't loaded yet

"Uninitialized" and "Offline" are pretty much the same, so I usually use "if TheBDM.getBDMState() in ('Uninitialized','Offline'): ..."

Anyways, I'm sure there's other things that are new, but obviously ArmoryQt.py is functioning correctly, so there's lots of sample code in there.  Really, 95% of it is the same, just some new/renamed calls and slightly different arguments.  For this application, you probably want setBlocking(True), since you might just be adding complication to an app that doesn't actually need the multi-threading.  If you run the latest Armory GUI, you'll see why I made the change.  MUCH better user experience...

Sorry about that!  Please pester me about more errors, and I'll help you recover from this under-the-hood overhaul...
(and sorry in advance, there's probably another one coming... upgrading to completely new wallets and unicode support...)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: pvz on January 06, 2013, 04:28:54 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)
MAV thanks!

I have a fully operational bitcoind daemon (Debian 6) and also get the same errors.

Does the CppBlockUtils.py and/or _CppBlockUtils.so can stay on the same version or do they need a more accurate version also?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on January 06, 2013, 04:37:27 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)
MAV thanks!

I have a fully operational bitcoind daemon (Debian 6) and also get the same errors.

Does the CppBlockUtils.py and/or _CppBlockUtils.so can stay on the same version or do they need a more accurate version also?

That's definitely going to have to be recompiled. 

On that note: mav, did you change any of the C++ code?  Or were you just using the stock armory code, and just putting a wrapper around it?  If you just wrapped it, it may be better to just fork the BitcoinArmory repo and add your .py files to it.  Then updates are just a "make" away, and goes pretty smoothly on all linux distros (few dependencies, and no version requirements on them).


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on January 06, 2013, 05:13:01 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)
MAV thanks!

I have a fully operational bitcoind daemon (Debian 6) and also get the same errors.

Does the CppBlockUtils.py and/or _CppBlockUtils.so can stay on the same version or do they need a more accurate version also?

That's definitely going to have to be recompiled. 

On that note: mav, did you change any of the C++ code?  Or were you just using the stock armory code, and just putting a wrapper around it?  If you just wrapped it, it may be better to just fork the BitcoinArmory repo and add your .py files to it.  Then updates are just a "make" away, and goes pretty smoothly on all linux distros (few dependencies, and no version requirements on them).

Aaaaack been a long time since I looked at this and forgot to copy the new _CppBlockUtils.so to the repo. I have not changed any c++ code, this is just a wrapper. I am going to put this on hold until the changes are made. Then I will properly re-assess the situation and put this in a fork of BitcoinArmory rather than as a standalone daemon with the .so binary included.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on January 06, 2013, 05:24:21 AM
I have recently pushed changes to the daemon to bring it up to date with the latest armoryengine.py

Unfortunately I am unable to load the blockchain, and am getting these errors.

Code:
(ERROR) armoryengine.py:11362 - Error processing BDM input
(ERROR) armoryengine.py:11363 - Received inputTuple: StartScanRequested [8, 28474661, False]
(ERROR) armoryengine.py:11364 - Error processing ID (28474661)
MAV thanks!

I have a fully operational bitcoind daemon (Debian 6) and also get the same errors.

Does the CppBlockUtils.py and/or _CppBlockUtils.so can stay on the same version or do they need a more accurate version also?

That's definitely going to have to be recompiled. 

On that note: mav, did you change any of the C++ code?  Or were you just using the stock armory code, and just putting a wrapper around it?  If you just wrapped it, it may be better to just fork the BitcoinArmory repo and add your .py files to it.  Then updates are just a "make" away, and goes pretty smoothly on all linux distros (few dependencies, and no version requirements on them).

Aaaaack been a long time since I looked at this and forgot to copy the new _CppBlockUtils.so to the repo. I have not changed any c++ code, this is just a wrapper. I am going to put this on hold until the changes are made. Then I will properly re-assess the situation and put this in a fork of BitcoinArmory rather than as a standalone daemon with the .so binary included.

Actually, this may not be so bad.  I was busy preaching about all the changes and how it was all worth the effort given the usability improvement for Armory GUI ... but actually you may be relatively unaffected as long as you set blocking=True.  In general, the GUI is advancing, not the library, so you don't have to upgrade like this often.

I just browsed your code and didn't see a whole lot that has to be changed, except you need to replace "TheBDM.LoadBlockchain()" with "TheBDM.setBlocking(True);  TheBDM.setOnlineMode(True)".  There might be a couple minor things that you have to update, and please post here with any more issues and I bet we can clear them up quickly.  But overall you don't appear to be using many things that changed.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: pvz on January 06, 2013, 07:03:11 AM
I just browsed your code and didn't see a whole lot that has to be changed, except you need to replace "TheBDM.LoadBlockchain()" with "TheBDM.setBlocking(True);  TheBDM.setOnlineMode(True)".  There might be a couple minor things that you have to update, and please post here with any more issues and I bet we can clear them up quickly.  But overall you don't appear to be using many things that changed.

With this change I get the following error:
Code:
(ERROR) armoryengine.py:11363 - Received inputTuple: GoOnlineRequested [13, 31616721, True]

If I set:
Code:
TheBDM.setBlocking(True);  TheBDM.setOnlineMode(FALSE)

There are no errors, also armory-daemon does not get started.
Well...I think it does not get started because I do not receive this message:
Code:
Loading blockchain
Initialising server
Server started

The process seems like to hang. Most of the time a ^C helps me out, now I have to shut down the terminal.

I expect the online mode needs a TheBDM.setOnlineMode(true) but I can't get to the code triggering the error.

By the way, I am not able to compile the CppBlockUtils.py or _CppBlockUtils.so code.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on January 06, 2013, 03:08:45 PM
By the way, I am not able to compile the CppBlockUtils.py or _CppBlockUtils.so code.

Then how are you running with the new code?  The .so most definitely needs to be recompiled to use the new stuff.  I would expect seg faults, not hanging, but computers are weird....

I recommend checking out the current Armory (https://github.com/etotheipi/BitcoinArmory) branch to a different directory, type "make", and then copy the CppBlockUtils.py and _CppBlockUtils.so over to the BitcoinArmory-Daemon directory.  Then it should work.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: pvz on January 06, 2013, 04:37:57 PM
The .so most definitely needs to be recompiled to use the new stuff.
You are right, this was the problem.
Thanks!

Status on my Debian 6 server:
- operational bitcoind daemon
- download BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon.git (https://github.com/thedawnrider/BitcoinArmory-Daemon.git))
  with latest armoryengine.py by mav
  (added Armory and Armory-Daemon dependencies)
- Replaced some code in armory-daemon.py (line 217 https://bitcointalk.org/index.php?topic=92496.msg1437378#msg1437378 (https://bitcointalk.org/index.php?topic=92496.msg1437378#msg1437378) by etotheipi)
- download and make BitcoinArmory (https://github.com/etotheipi/BitcoinArmory.git (https://github.com/etotheipi/BitcoinArmory.git))
- replace CppBlockUtils.py and _CppBlockUtils.so on Armory-Daemon from BitcoinArmory
-> start Armory-Daemon ($ python armory-daemon.py)

Now starting to explore how the armory-daemon works... ;)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mav on January 09, 2013, 10:33:33 AM
Great news, web-armory people! The daemon is being included with armory itself so will be up to date and more fully featured than I ever could have achieved.

see https://bitcointalk.org/index.php?topic=56424.msg1443054#msg1443054 (https://bitcointalk.org/index.php?topic=56424.msg1443054#msg1443054) for details.

I will keep this thread open to discuss the particulars of using armory on webservers, as the daemon is not strictly only for use on webservers (may be used for, say, running your uniquely common yet complex task easily, or remotely controlling your armory application).


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CashBitango on March 06, 2014, 01:12:53 PM
Hi

I guess this is really basic, but I still need some help getting this script running.
So I'm running this on a linux machine, I've installed python, and I'm trying to run the script.

$ ./armoryd.py
from: can't read /var/mail/twisted.internet
from: can't read /var/mail/twisted.web
from: can't read /var/mail/txjsonrpc.web
from: can't read /var/mail/txjsonrpc.auth
from: can't read /var/mail/twisted.cred.checkers
from: can't read /var/mail/armoryengine
./armoryd.py: line 51: import: command not found
./armoryd.py: line 52: import: command not found
./armoryd.py: line 53: import: command not found
./armoryd.py: line 54: import: command not found
./armoryd.py: line 55: import: command not found
./armoryd.py: line 56: import: command not found
./armoryd.py: line 59: import: command not found
from: can't read /var/mail/jsonrpc
./armoryd.py: line 61: syntax error near unexpected token `('
./armoryd.py: line 61: `class UniversalEncoder(json.JSONEncoder):'
$

So then I installed twisted:
$ sudo apt-get install python-twisted

But I still get the same message.
Trying to run it on Windows I get the same import problem.

Any suggestions?

Question2
When I get the script running and can use it to create offline transactions etc, can I have it interact with another program in another language I've written?
If so, can I do it over localhost, and send RPC or something, getting replies in json?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on March 06, 2014, 04:06:34 PM
You are trying to run it as a bash script.  You need to run it as a python script:

"python armoryd.py".

Also, as someone else noted, you'll need to download the txjsonrpc project and put it in your base directory (there should be a txjsonrpc dir next to armoryd.py).  Or actually get txjsonrpc installed -- it's an extra dependency not needed by regular Armory but needed for armoryd.py.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on March 06, 2014, 04:35:57 PM

Question2
When I get the script running and can use it to create offline transactions etc, can I have it interact with another program in another language I've written?
If so, can I do it over localhost, and send RPC or something, getting replies in json?


I haven't had a need to use JSON across languages yet, but I understand that is one of it's features.

From http://en.wikipedia.org/wiki/JSON

Quote
Although JSON was originally based on a non-strict subset of the JavaScript scripting language (specifically, Standard ECMA-262 3rd Edition—December 1999[6]) and is commonly used with that language, it is a language-independent data format. Code for parsing and generating JSON data is readily available for a large variety of programming languages. JSON's Web site provides a comprehensive listing of existing JSON libraries, organized by language.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CashBitango on March 08, 2014, 02:30:40 PM
Hi

Thanks for the help thus far, but I still have problems. Sorry for this massive post, but I think it makes it easier to debug.

I'm trying to get this thing working on both windows and linux, but encounter difficulties on both.

1) Windows:
I put the  txjsonrpc into the suggested folder, I used this one: https://github.com/oubiwann/txjsonrpc
I installed twisted, and that required zope, so I installed that too.
Then I installed psutils-1.2.1 and confirmed that it's working.
I'm running this thing in Python 2.7.6, opening the file from the shell, then hitting run (F5)

So now I get this error:
----------------------
(CRITICAL) armoryengine.py:1043 - C++ block utilities not available.
(CRITICAL) armoryengine.py:1044 -    Make sure that you have the SWIG-compiled modules
(CRITICAL) armoryengine.py:1045 -    in the current directory (or added to the PATH)
(CRITICAL) armoryengine.py:1046 -    Specifically, you need:
(CRITICAL) armoryengine.py:1047 -        CppBlockUtils.py     and
(CRITICAL) armoryengine.py:1051 -        _CppBlockUtils.pyd

Traceback (most recent call last):
  File "X:\BitcoinArmory-master\armoryd.py", line 49, in <module>
    from armoryengine import *
  File "X:\BitcoinArmory-master\armoryengine.py", line 1039, in <module>
    import CppBlockUtils as Cpp
ImportError: No module named CppBlockUtils
----------------------

Any suggestions?
I'm using the latest source from https://github.com/etotheipi/BitcoinArmory, downloaded as zip.

2) Ubuntu 12.04
I put the txjsonrpc into the suggested folder, I used this one: https://github.com/oubiwann/txjsonrpc
I tried installing psutils-1.2.1:
----------------------
$ python setup.py  install
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite'
  warnings.warn(msg)
running install
running build
running build_py
creating build
creating build/lib.linux-i686-2.7
creating build/lib.linux-i686-2.7/psutil
copying psutil/_pslinux.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_psbsd.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_compat.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_psosx.py -> build/lib.linux-i686-2.7/psutil
copying psutil/__init__.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_pssunos.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_error.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_psmswindows.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_psposix.py -> build/lib.linux-i686-2.7/psutil
copying psutil/_common.py -> build/lib.linux-i686-2.7/psutil
copying psutil/error.py -> build/lib.linux-i686-2.7/psutil
running build_ext
building '_psutil_linux' extension
creating build/temp.linux-i686-2.7
creating build/temp.linux-i686-2.7/psutil
gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c psutil/_psutil_linux.c -o build/temp.linux-i686-2.7/psutil/_psutil_linux.o
psutil/_psutil_linux.c:13:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: command 'gcc' failed with exit status 1
----------------------
Since install failed, I put psutils into the same folder that I'm trying to run the armoryd.py from, then try to run it:
----------------------
$ python armoryd.py
Traceback (most recent call last)
  File "armoryd.py", line 49, in <module>
    from armoryengine import *
  File "/home/linuxuser/Documents/arm/armoryengine.py", line 47, in <module>
    import psutil
  File "/home/linuxuser/Documents/arm/psutil/__init__.py", line 88, in <module>
    import psutil._pslinux as _psplatform
  File "/home/linuxuser/Documents/arm/psutil/_pslinux.py", line 20, in <module>
    import _psutil_linux
ImportError: No module named _psutil_linux
$
----------------------

More suggestions?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CashBitango on March 08, 2014, 02:52:57 PM

Question2
When I get the script running and can use it to create offline transactions etc, can I have it interact with another program in another language I've written?
If so, can I do it over localhost, and send RPC or something, getting replies in json?


I haven't had a need to use JSON across languages yet, but I understand that is one of it's features.

From http://en.wikipedia.org/wiki/JSON

Quote
Although JSON was originally based on a non-strict subset of the JavaScript scripting language (specifically, Standard ECMA-262 3rd Edition—December 1999[6]) and is commonly used with that language, it is a language-independent data format. Code for parsing and generating JSON data is readily available for a large variety of programming languages. JSON's Web site provides a comprehensive listing of existing JSON libraries, organized by language.

OK, I see I wasn't expressing myself clear.

The question wasn't about JSON per se, it was about how to interact with armory. I have written software in Java, and I need it to interact with Armory, to create an offline transaction.
I want my program to be able to connect for example to localhost:8999, send command jsonrpc_sendmany([array of bitcoin addresses : amount to send])
Then I'd like to get in return the offline transaction, unsigned, into my program, and my program stores file to disk/USB or whatever. Then I want to take that offline transaction to an offline computer and sign it before broadcasting it from an online computer.

Think it can be done?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on March 08, 2014, 05:27:53 PM
...
1) Windows:
...
So now I get this error:
----------------------
(CRITICAL) armoryengine.py:1043 - C++ block utilities not available.
(CRITICAL) armoryengine.py:1044 -    Make sure that you have the SWIG-compiled modules
(CRITICAL) armoryengine.py:1045 -    in the current directory (or added to the PATH)
(CRITICAL) armoryengine.py:1046 -    Specifically, you need:
(CRITICAL) armoryengine.py:1047 -        CppBlockUtils.py     and
(CRITICAL) armoryengine.py:1051 -        _CppBlockUtils.pyd

Traceback (most recent call last):
  File "X:\BitcoinArmory-master\armoryd.py", line 49, in <module>
    from armoryengine import *
  File "X:\BitcoinArmory-master\armoryengine.py", line 1039, in <module>
    import CppBlockUtils as Cpp
ImportError: No module named CppBlockUtils
----------------------

Any suggestions?


You are missing _CppBlockUtils.pyd and CppBlockUtils.py. You can either get them from your armory installation or build cppForSwig to produce these files.

More detailed instructions are here:
https://bitcoinarmory.com/download/building-armory-from-source/

Re: Question 2...I don't have any experience building on linux so I can't help trouble shoot that.

....I have written software in Java, and I need it to interact with Armory, to create an offline transaction.
I want my program to be able to connect for example to localhost:8999, send command jsonrpc_sendmany([array of bitcoin addresses : amount to send])
Then I'd like to get in return the offline transaction, unsigned, into my program, and my program stores file to disk/USB or whatever. Then I want to take that offline transaction to an offline computer and sign it before broadcasting it from an online computer.

Think it can be done?

Yes that can definitely be done. Send your command from the Java program and the result of any JSON method you call will return a text representation of the returned value or values (as a name/value mapping).


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on March 08, 2014, 07:22:22 PM
Getting armoryd working on windows will be much more challenging than linux (which is five commands (https://bitcoinarmory.com/download/building-armory-from-source/) plus getting txjsonrpc).  I'm not sure we've even tried armoryd.py in Windows without having the whole build environment setup.   As CircusPeanut said, you might only need to get the _CppBlockUtils.pyd and .py from a preinstalled version of Armory, but there might be other .dlls you need as well.  Maybe just copy those two files plus all the .dlls from the installed Armory dir, into your checkout directory.  Or use Linux... which is dramatically easier to setup.


We should have a function in there that gets you an unsigned transaction for a list of recipients.  To make the signing process easier, you can look at extras/cli_sign_txdp.py which is a full-featured, command-line interface for signing offline transactions.  The two arguments for it are wallet file and a unsigned.tx files from Armory. 

Be aware that the master branch may not have all the armoryd.py features implemented.  The testing branch will have all the updated features, but of course is the unstable version.  Hopefully we'll have that ironed out in a couple weeks.




Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: halfawake on March 18, 2014, 05:52:06 PM
I just now noticed this thread.  This is an amazing feature, given all the hacks / thefts that I've seen, this kind of webserver daemon seems like it'd be the holy grail of bitcoin daemons since it'd be effectively unhackable.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CashBitango on March 24, 2014, 07:07:32 PM
Hi guys

Thanks for all your support in trying to get Armory working on server, but it's still a bit difficult for me.

I need it to work easily in different environments... No hard setup or installs.

I have an idea though. If you guys tell me what classes and methods I should be looking at in the source, I might be able to write my own version in Java. So I'll look at your code, figure out what it does to produce the unsigned transaction, and then I'll write some code that replicates the output from the same input.

If I get it working, I'll send you the source files so you can let others use them too, if you want that.
I'm writing the tests as well of course, so that anyone can see that they get expected result.

I'm imagining something like creating a class and method in Java:

public String getArmoryUnsignedTransaction(String[] unspentInput, String[] addressesTo, long amountSatoshi){
           //Do some magic
           return theWholeTransactionAsASingleString;
}

Would you be able to point me to some folders, classes and methods in the source where I can see what's going on when creating the unsigned transaction?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on March 24, 2014, 07:25:44 PM
Here is a link to the latest code:
https://github.com/etotheipi/BitcoinArmory/blob/0.91-dev/armoryd.py


Title: Re: Armory on web servers
Post by: gokudev on April 03, 2014, 07:00:38 PM
Quote
Damnit!  I wish I could make the serial port idea work...
You and me both! I think I'll be looking into getting the Raspberry Pi to act as a USB client. It would then be necessary to use the USB protocol, which is a lot more complicated than just serial, but would not - as far as I know - allow login to the offline machine.

EDIT: I just discovered that the Raspberry Pi Model A (as opposed to Model B that is out now) will support acting as a USB device/client, because it uses a different communication-chip than Model B. Model A doesn't have an Ethernet port, and only has a single USB port, but I figure this might even be an advantage, since after the software has been installed on the device (using, for example, some USB-to-Ethernet type device to update packages), the Ethernet port will have to be removed because the USB connection needs to be used to communicate with the online computer. Perhaps one will be less tempted - and feel safer - if the device doesn't even have a port to plug in an Ethernet cable.
Additionally, it costs only $25. That's really not much for the ultimate in safe Bitcoin wallets. That being said, Model A isn't out now. And USB client mode won't necessarily work out-of-the-box. This is a software issue, as far as I can gather. Not OS level, I think, but firmware level - so it might not be that easy to implement. But still, looks like a device is coming to the market that fulfils the requirements of an offline hardware wallet. Off the top of my head, I can't see what else one would need other than a Raspberry Pi Model A, and perhaps a USB cable. But most people have USB cables lying around. Sure beats that $40 2xUSB-to-serial-plus-null-modem solution.
[/quote]

Wonder where things are at with serial ports. I own a rasp pi model A and I like the idea of using it for running armory in offfline mode. The rasp pi model A can work with serial ports no problem using a shield. Has there been any development on armory for making serial ports work? I gather we have to send the 'unsigned transaction' over from online armory to an offline pc over a serial connection?


Title: Re: Armory on web servers
Post by: gokudev on April 04, 2014, 04:20:15 AM
This is why I prevent Armory from opening twice.  Wallet operations are not threadsafe. Not at all!  

However, one benefit of python-twisted is that it is guaranteed to run as a single-thread, so as long as there is only one instance of Armory, it's perfectly safe.

For your purposes, not using python-twisted, need to do something else.  Actually, I would recommend using python-twisted to create a server/daemon process to handle (serially) all requests for wallet interaction.  Make sure everything goes through that interface.  The wallet design has been remarkably robust (I even test it with mid-write interrupts and it recovers fine).  But as you can see, two simultaneous operations can break it pretty easily.  


Wallet operations are not threadsafe. Not at all!  
can someone please explain why armory wallets are not thread safe and why armory runs as a singleton? please explain in detail.


Title: Re: Armory on web servers
Post by: goatpig on April 04, 2014, 12:08:19 PM
This is why I prevent Armory from opening twice.  Wallet operations are not threadsafe. Not at all!  

However, one benefit of python-twisted is that it is guaranteed to run as a single-thread, so as long as there is only one instance of Armory, it's perfectly safe.

For your purposes, not using python-twisted, need to do something else.  Actually, I would recommend using python-twisted to create a server/daemon process to handle (serially) all requests for wallet interaction.  Make sure everything goes through that interface.  The wallet design has been remarkably robust (I even test it with mid-write interrupts and it recovers fine).  But as you can see, two simultaneous operations can break it pretty easily.  


Wallet operations are not threadsafe. Not at all!  
can someone please explain why armory wallets are not thread safe and why armory runs as a singleton? please explain in detail.

In one line: easier to implement.

Up until a few months ago, etotheipi was working on Armory alone. Cost of implementation is a very high priority in this case. Now we're 5, so things are changing, notably Armory's backend model.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 08, 2014, 08:10:45 PM
The simple Armory Daemon is now on github. It aims to be as close to a direct replacement for the Satoshi interface as possible.

https://github.com/thedawnrider/BitcoinArmory-Daemon (https://github.com/thedawnrider/BitcoinArmory-Daemon)

Available json-rpc calls are based on the Satoshi rpc interface and are as follows:

getbalance
getnewaddress
getreceivedbyaddress
sendtoaddress

getbalance:
Returns a decimal value in BTC for the total remaining balance in the wallet.

getnewaddress:
Returns the next address in the wallet as a string.

getreceivedbyaddress:
Returns a decimal value in BTC for the amount received by the address.

sendtoaddress:
Returns an unsigned transaction as a string. Implementation of signing and broadcasting is left to the client.



Features that may be included in the future:
User authentication
SSL
More API methods as required (the current set fills my needs). listtransactions is one method I am looking to include in the future.
Please suggest anything else you personally may find useful, as I have designed this only to fill my specific needs.



Clarifications required:

- Are incoming transactions which eventually do not become part of the blockchain dealt with correctly? I am unsure how / unable to test this.

- What happens if multiple transactions are made which would result in the balance being less than zero? Presumably when it comes to broadcasting the last of the signed transactions there will be an error from the Armory client which is performing the signing/broadcasting. Should tracking of the likely future balance in the daemon be enforced even though the transactions have not yet been broadcast and maybe never will be? How should this be managed, if at all?

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on May 08, 2014, 08:15:57 PM

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 

At the moment, "getledger" and "getledgersimple" are the biggest divergences of them, though we attempted to implement "listtransactions" the same way Bitcoin Core does.  I really don't like what Core does with "listtransactions":  it provides multiple outputs with overlapping entries, which require a bit of extra logic to adapt to your application.

On the other hand "getledger" provides you exactly what you would see on the Armory GUI ledger -- it's a mutually-exclusive list of entries, each one corresponding to a single transaction, and theoretically all the values produced from it can be summed to give you your balance. 

We will also soon have an interface for:

- Multiple wallets (context switching)
- Create unsigned transaction (to take to offline computer)
- Sign unsigned transaction (to run from offline computer)

If you'd like to recommend more stuff you'd like to see, post in the following thread:
https://bitcointalk.org/index.php?topic=588227.0


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 08, 2014, 08:32:53 PM

What commands are specific to the armory daemon that are not in the bitcoindaemon?
 

At the moment, "getledger" and "getledgersimple" are the biggest divergences of them, though we attempted to implement "listtransactions" the same way Bitcoin Core does.  I really don't like what Core does with "listtransactions":  it provides multiple outputs with overlapping entries, which require a bit of extra logic to adapt to your application.

On the other hand "getledger" provides you exactly what you would see on the Armory GUI ledger -- it's a mutually-exclusive list of entries, each one corresponding to a single transaction, and theoretically all the values produced from it can be summed to give you your balance. 

We will also soon have an interface for:

- Multiple wallets (context switching)
- Create unsigned transaction (to take to offline computer)
- Sign unsigned transaction (to run from offline computer)

If you'd like to recommend more stuff you'd like to see, post in the following thread:
https://bitcointalk.org/index.php?topic=588227.0

ok so at the moment, can i sign/unsign transactions using bitcoin daemon since they already have a json api ? or will that not work with armory wallets?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on May 08, 2014, 08:38:55 PM
Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 08, 2014, 08:56:55 PM
Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on May 09, 2014, 01:34:41 AM
Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?


I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: po0kie on May 10, 2014, 11:57:45 AM
My current envoirement with ArmoryD

If its an help for someone

I have "X" Online servers (at the moment I am trying to know how much Users I can support on each)
And "X" "Warm-Cold" Wallets server instances. (Each online server instance has an offline Server instance)

Process to broadcast:

It starts with: Web requests to Middle-Api-Server to make a broadcast from AddressX to AddressZ

The Middle API Server communicates to Online Watching Server and checks the balance.
If balance is OK, I create a encrypted Queue of another Middle-Backend with an Unsigned Transaction.

Then another Process Server Checks the Encrypted Queue for unsigned Tx's and sends it to the Warm-Cold Server to sign.
This middle Server stores the SignedTx back to the Encrypted Queue and then the Online Server Queue get a HexTx from ArmoryD (Online server)
and broadcasts this rawtransaction over BitcoinD RPC.

Its a little bit confusing maybe as 5 Servers are needed for this process.
(1 web server, 1 Api server, 1 Online Watching Server, 1 Offline Server, 1 Queue process Server For Signing)
and then increasing on Servers depending on Users.


The testings right now are 100.000 Users per Wallet
- You need to tweak the ArmoryD a little bit for this porpose on some places
   1. The most functions are on Wallet scope - I added functions to support the RPC on Address Scope.
   2. The Offline server Unlock function is very slow on 100.000 Addresses, so I modified the Unlock function to just unlock the PrivKey which I need to sign.

Its just a little info for somebody to get started with armory and Services on a website.

I would preffer Wallet per User, but until armoryd does not support multi Wallet its just impossible - at least I didnt found a suitable solution.


In a few days I can tell if the wallet supports 100.000 Addresses / Users with a avg. of 2-3 Transactions per User. (would be 300.000 Transactions per Wallet / Server)


Best regards






Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 22, 2014, 07:58:35 PM
Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?


I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.

Quote
  You don't have to operate through RPC, you can just program your hooks directly in.
Well the thing is, our server making the calls to the daemon is in .net(C#) It is much easier and cleaner to simply make json rpc calls to the coin daemon. What are the advantages of programming the hooks directly?


Also looking at armoryd.py i notice many new jsronrpc methods not found in the original bitcoin API calls list (https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list)  these methods have a prefix of 'jsonrpc' so im thinking they are newly added to the api call list but wiki does not show them.





Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 22, 2014, 08:14:37 PM
Armory unsigned transactions are different.  It's because they store all the extra data needed for the offline computer to verify the transaction without the blockchain.  It can be a lot of data, but required to maximize security. 

On the other hand, you might be able to go the other way, but you'd have to build some helper functions to convert a raw unsigned transaction from bitcoind, to collect all the blockchain data that is needed for Armory to verify the tx before signing.  Armory assumes all transactions will always be signed by a dumb device with no blockchain -- there is no alternative form.

This is not standardized in any way.  Maybe one day it will be.

Ok I am actually trying to build an exchange and the idea is to have cold warm and hot wallets. The cold would be using offline armory and the hot would be using online armory. The warm wallet will be built using either armory daemon or bitcoind.  I didnt know armory daemon was not complete. So like you explained, I can use bitcoind and then before sending the unsigned transaction to cold storage, I would need add extra data to the transaction.

What about signing a transaction, can that be done using bitcoind or do i need to implement a command specific to  armory's requirements?


I'm not sure what you're asking with the last question.  If you are signing with bitcoind (can it do offline signing?) then you can just broadcast it with any network-connected app once you get it back online.  Armory's default is to pass in this decorated-unsigned-tx, and return the same decorated-signed-tx to the online computer.  That Armory instance will "prepareFinalTx().serialize()" which converts the decorated-tx to a byte string appropriate to be sent over the wire.

Btw, we do have many users leveraging armoryd as their backend, it just requires a bit of work to fill in the gaps.  One of the nice things about armoryd.py is that it's pretty easy to adapt it to whatever you need to do:  there's a clear place to put code everytime a new tx comes in, everytime a new wallet-relevant tx comes in, and every new block.  You don't have to operate through RPC, you can just program your hooks directly in.

I need to figure out how to use bitcoind and armory offline wallets together via json rpc or some other method.
I think this is the post I am looking for  https://bitcointalk.org/index.php?topic=448284.0
this is able to convert a signedtransaction from bitcoind into something armory can sign in offline mode. correct me if i am wrong?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on May 22, 2014, 09:36:10 PM

I need to figure out how to use bitcoind and armory offline wallets together via json rpc or some other method.
I think this is the post I am looking for  https://bitcointalk.org/index.php?topic=448284.0
this is able to convert a signedtransaction from bitcoind into something armory can sign in offline mode. correct me if i am wrong?


Assuming you mean that  you want to convert an "unsigned transaction" for armory to sign. Yes, you can do this with a bit of coding in armoryd.

Take whatever raw transaction hex from bitcoind and use the unserialize method on the appropriate object to get an Armory object. You probably want PyTx or PyTxDistProposal. Note that in the devel branch (and others) PyTxDistProposal is renamed UnsignedTransaction. So whatever the code base you're using look for the appropriate object and call unserialize on it.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 22, 2014, 09:43:13 PM
OK armory would need to be modified and recompiled. Can Armorybe modified to create wallets for a coin type ? So I want multiple wallets maybe one wallet per coin type. 


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on May 22, 2014, 09:45:20 PM
OK armory would need to be modified and recompiled. Can Armorybe modified to create wallets for a coin type ? So I want multiple wallets maybe one wallet per coin type.  

No need to recompile since armoryd is written in Python.

Modifying Armory to run with different crypto currencies requires a lot of work.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 22, 2014, 09:54:28 PM
Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt ? 
Indulge me what makes you say its a lot of work ?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on May 22, 2014, 10:24:03 PM
Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt ? 
Indulge me what makes you say its a lot of work ?

I'm basing that assessment on this thread in github https://github.com/etotheipi/BitcoinArmory/issues/32


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 23, 2014, 03:16:44 AM
Its the work I'm afraid  I will have to do if I'm to use armory for our exchange. Etotheipi mentioned some where it requiring some work. The transaction format is the same across coins only the hashing algorithm changes from sha256 to scrypt?
Indulge me what makes you say its a lot of work?

I'm basing that assessment on this thread in github https://github.com/etotheipi/BitcoinArmory/issues/32

Wondering if etotheipi can chime in, why does the hashing function need changing for ltc if I am just using armory wallets for storing coins. or are wallets and hashing function connected together?

Can armory not use the coin daemon (litecoin daemon via rpc calls ) and manage offline wallets that way. I can modify armory to convert the unsigned tx to work with armory's offline signing and then all should be good?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: gokudev on May 23, 2014, 03:53:58 PM
I am a bit confused, Why does armory run bitcoind in the background and what is the use of armory daemon?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: etotheipi on May 25, 2014, 09:17:51 PM
I am a bit confused, Why does armory run bitcoind in the background and what is the use of armory daemon?

If you don't know the answer to that question, you should probably spend some time with the Armory GUI to understand the vast functionality provided by Armory.   Especially the offline wallets.  Most people are interested in armoryd because of the ease with which you can create watching-only wallets to put on your webserver, and keep the private keys on an offline computer.

Armory uses Bitcoin Core solely for communicating with the Bitcoin network, peer finding, and blockchain validation.   Armory provides 100% new wallet infrastructure to provide deterministic wallets (only needeing to be backed up one time), plethora of backup features, the ability to manage multiple wallets at once, and the ability to manage wallets created offline but still monitor the wallet activity and distribute payment addresses from an online computer.
 
Armory uses the filtered datastream provided from Bitcoin Core, since no other software can reliably and securely replicate the validation/verification behavior of Bitcoin Core (if you use something else, you risk getting forked).   Armory doesn't even try to do blockchain validation -- that's Core's job.  Therefore you need to run both, and Armory watches the blk*.dat files from Core and the tx messages that are sent over P2P.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: po0kie on May 27, 2014, 08:37:34 AM
Hi!

I noticed that some transactions are not getting to mainbranch, and for this reason the Tx never gets confirmed.
At this moment I dont know why it happens. Over 25.000 transactions on Testnet and maybe 50 cases on which happens the mainbranch issue.

Is there a way to detect the "scrap" items on armory core?
At this moment the one and only working solution I got is to restart the watching only wallet service.
After restarting armory service the TxId just dissapears and I mark them as "Scrap" in database and reissue the Tx creating a new one.


Is there a way to ask Armory core if the transaction is really a "scrap" item?
And how can it happen that the Tx does not get into the Mainbranch? (all tests are done over testnet so far)

Best regards


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: TakashiMakat on June 05, 2014, 12:20:56 PM
How can I install txjsonrpc on the BitcoinArmory directory?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: doug_armory on June 05, 2014, 02:27:04 PM
How can I install txjsonrpc on the BitcoinArmory directory?

Hi. As part of my ongoing cleanup of armoryd, I grabbed a copy of txjsonrpc and am now distributing it along with Armory. If anybody can't wait for the next version of Armory and wants to do what I did, https://github.com/etotheipi/BitcoinArmory/commit/5a75164e90e9898cddf66d13199744e8d94b873b has the details.

Speaking of armoryd, I've added quite a bit of functionality over the past couple of weeks. armoryd is still rough around the edges but is quickly becoming a pretty useful tool. Feel free to check it out, with the caveat that there's still plenty to do and that any version you download now definitely won't be the version that goes into Armory 0.92. (In fact, I'll be checking in some more lockbox functionality in the next hour or two.)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: TakashiMakat on June 05, 2014, 03:01:34 PM
I cloned the repo to BitcoinArmory directory. But I still get

Code:
root@cloud-server-07:/BitcoinArmory# python armoryd.py
Traceback (most recent call last):
  File "armoryd.py", line 55, in <module>
    from txjsonrpc.auth import wrapResource
ImportError: No module named txjsonrpc.auth

I'm guessing I need to install it but I could not figure out how to do that.
 


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: doug_armory on June 05, 2014, 03:04:56 PM
I cloned the repo to BitcoinArmory directory. But I still get

Code:
root@cloud-server-07:/BitcoinArmory# python armoryd.py
Traceback (most recent call last):
  File "armoryd.py", line 55, in <module>
    from txjsonrpc.auth import wrapResource
ImportError: No module named txjsonrpc.auth

I'm guessing I need to install it but I could not figure out how to do that.
 

You're probably cloning the master branch, which doesn't have any of the new armoryd code. You need to clone the devel branch to get all the new armoryd code. Just be aware that it is a work-in-progress. Various things may be incomplete or completely broken.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: TakashiMakat on June 06, 2014, 01:54:45 PM
I cloned the repo to BitcoinArmory directory. But I still get

Code:
root@cloud-server-07:/BitcoinArmory# python armoryd.py
Traceback (most recent call last):
  File "armoryd.py", line 55, in <module>
    from txjsonrpc.auth import wrapResource
ImportError: No module named txjsonrpc.auth

I'm guessing I need to install it but I could not figure out how to do that.
 

You're probably cloning the master branch, which doesn't have any of the new armoryd code. You need to clone the devel branch to get all the new armoryd code. Just be aware that it is a work-in-progress. Various things may be incomplete or completely broken.

All I need is to be able to get the balance. So I think the version on the master branch will be fine?

The only problem I have right now is to get armoryd.py to find txjsonrpc.auth module.

Traceback (most recent call last):
  File "armoryd.py", line 55, in <module>
    from txjsonrpc.auth import wrapResource
ImportError: No module named txjsonrpc.auth


I'm probably going to run into some other issues after I solve it but still...


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: TakashiMakat on June 06, 2014, 01:59:19 PM
Ok I fixed the txjsonrpc problem.

1) sudo apt-get install python-setuptools
2) sudo easy_install txJSON-RPC

The second command must be run IN the armory directory. Now the error I get is:

root@cloud-server-07:/BitcoinArmory# python armoryd.py
(ERROR) Traceback (most recent call last):
  File "armoryd.py", line 1158, in <module>
    rpc_server = Armory_Daemon()
  File "armoryd.py", line 852, in __init__
    self.lastChecked
AttributeError: 'Armory_Daemon' object has no attribute 'lastChecked'

Traceback (most recent call last):
  File "armoryd.py", line 1158, in <module>
    rpc_server = Armory_Daemon()
  File "armoryd.py", line 852, in __init__
    self.lastChecked
AttributeError: 'Armory_Daemon' object has no attribute 'lastChecked'


Which seems to be more Armory specific.


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: doug_armory on June 06, 2014, 03:33:54 PM
Ok I fixed the txjsonrpc problem.

As stated before, that library will be included with the next version of Armory.

Quote
The second command must be run IN the armory directory. Now the error I get is:

root@cloud-server-07:/BitcoinArmory# python armoryd.py
(ERROR) Traceback (most recent call last):
  File "armoryd.py", line 1158, in <module>
    rpc_server = Armory_Daemon()
  File "armoryd.py", line 852, in __init__
    self.lastChecked
AttributeError: 'Armory_Daemon' object has no attribute 'lastChecked'

Traceback (most recent call last):
  File "armoryd.py", line 1158, in <module>
    rpc_server = Armory_Daemon()
  File "armoryd.py", line 852, in __init__
    self.lastChecked
AttributeError: 'Armory_Daemon' object has no attribute 'lastChecked'


Which seems to be more Armory specific.

It is. To be honest, armoryd basically won't work out of the box if you download the latest version. I had to fix at least two or three bugs before armoryd even ran at all, and I think other team members fixed a bug or two. I don't know what happened. It just fell out of sync with the rest of Armory, I guess.

That being said, the latest version works great. There are a few rough edges and a couple of bugs that need to fixed before it gets released, but trust me, it'll be a much smoother experience. :)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mtdox on September 22, 2014, 02:34:01 PM
Hi everyone!

I made a thread over here (click me) (http://"https://bitcointalk.org/index.php?topic=792107.0") about getreceivedbyaddress.

In short, does getreceivedbyaddress only include confirmed transactions or also unconfirmed ones?

Looking through the source gives me the impression that it counts both confirmed and unconfirmed transactions:
Code:
   def jsonrpc_getreceivedbyaddress(self, address):
      """
      DESCRIPTION:
      Get the number of coins received by a Base58 address associated with
      the currently loaded wallet.
      PARAMETERS:
      address - The Base58 address associated with the current wallet.
      RETURN:
      The balance received from the incoming address (BTC).
      """

      if CLI_OPTIONS.offline:
         raise ValueError('Cannot get received amount when offline')
      # Only gets correct amount for addresses in the wallet, otherwise 0
      addr160 = addrStr_to_hash160(address, False)[1]

      txs = self.curWlt.getAddrTxLedger(addr160)
      balance = sum([x.getValue() for x in txs if x.getValue() > 0])
      return AmountToJSON(balance)

Also looking through Transaction.py and PyBtcWallet.py did not give me any clues on how one would add an extra check to the lambda ( something like x.getConfirmations() > minconfs ) to only include confirmed transactions.

Any advice / inputs?


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: CircusPeanut on September 22, 2014, 02:48:15 PM
Thanks to Doug we actually have a new version of that method with an optional minconf argument  on the "dev" branch:

Code:
   #############################################################################
   @catchErrsForJSON
   def jsonrpc_getreceivedbyaddress(self, address, minconf=0):
      """
      DESCRIPTION:
      Get the number of coins received by a Base58 address associated with
      the currently loaded wallet.
      PARAMETERS:
      address - The Base58 address associated with the current wallet.
      minconf - (Default=0) The minimum number of confirmations required for a
                TX to be included in the final balance.
      RETURN:
      The balance received from the incoming address (BTC).
      """

      if CLI_OPTIONS.offline:
         raise ValueError('Cannot get received amount when offline')
      # Only gets correct amount for addresses in the wallet, otherwise 0
      addr160 = addrStr_to_hash160(address, False)[1]

      # Iterate through the address in our wallet. Include coins only if values
      # are positive and the number of confirmations is high enough.
      txs = self.curWlt.getAddrTxLedger(addr160)
      balance = 0
      for curTX in txs:
         nconf = (TheBDM.getTopBlockHeader().getBlockHeight() - \
                  curTX.getBlockNum()) + 1
         if (nconf >= minconf) and (curTX.getValue() > 0):
            balance += curTX.getValue()

      return AmountToJSON(balance)


Title: Re: [ANN] BitcoinArmory-Daemon - armory on web servers
Post by: mtdox on September 22, 2014, 03:11:24 PM
Awesome, thanks to everyone involved!