Title: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: rbdrbd on February 04, 2014, 08:25:32 AM Quick question:
I was wondering if it was possible to sign an unsigned transaction (hex encoded) essentially produced by bitcoind's createrawtransaction (i.e. long hex string starting with 010000000) in offline armory, assuming that my offline armory wallet has the necessary private keys of course. I tried producing one using and pasting it into armory's Sign and/or Broadcast Transaction text box, but it says "Unrecognized", along with "Transaction data is invalid and cannot be shown". The txn is valid (if signed in bitcoind it will broadcast fine, and it decodes properly with https://blockchain.info/decode-tx). I've noticed the format that armory appears to produce for its raw transactions has header and footer lines, as well as _TXDIST/_TXINPUT, and the hex encoded transaction is wrapped at a certain line width. Do I need to somehow encode the raw bitcoin transaction to match this form for armory to be able to work with it? Thanks! EDIT: So I figured it out and wrote a script to do this, which will take a bitcoind hex raw txn and convert to an armory style txn, which should be good for signing and/or broadcasting (if already signed) I would think. I couldn't get the system path hack to fully work... it still complained about not being able to import a module in the jsonrpc folder. That being the case, you could just throw this script in /usr/lib/armory on the online armory box (with a full or watch-only wallet) and run it. Starts armory up, loads the blockchain, and spits out the armory-formatted txn. Any suggestions to the script are welcome (and feel free to add this to the armory extras folder): Code: import sys Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: fthoughts on April 25, 2014, 03:33:42 AM this just saved me so hard.
thanks bootloads rbdrbd, keep doin yer thing, great dev work Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: gokudev on May 22, 2014, 08:22:05 PM wondering why you need to go in online mode and sync with blockchain?
Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: fthoughts on May 22, 2014, 09:06:58 PM cause you have to let armory sort out whether the input you selected is a valid (unspent) one or not
Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: gokudev on May 22, 2014, 09:18:47 PM And this can't be done with when creating raw unsigned transaction with the online bitcoin daemon ?
Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: etotheipi on May 24, 2014, 05:05:45 PM You don't have to go online to sign transactions created by Armory, because Armory bundles in extra information needed for the offline computer to be able to securely review and sign the transaction. Just a raw unsigned transaction is not enough, and Bitcoin Core doesn't have any way to provide that information (without modifying it). You should create the unsigned transaction (TxDP) with Armory online, and then it can be signed with Armory without going online.
That's the entire purpose of this thread: https://bitcointalk.org/index.php?topic=181734.0. Requesting to piggyback on the next hard fork to make it possible to supply all the secure threshold of data without having to bundle potentially MB of supporting transactions. However, even if that was fixed, you still need the previous TxOut scripts in order to execute signing, which have to be retrieved from the blockchain -- again something else that Armory's TxDP structure bundles with it, but a raw unsigned tx from Bitcoin Core does not have. Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: fthoughts on May 24, 2014, 06:41:41 PM "However, even if that was fixed, you still need the previous TxOut scripts in order to execute signing, which have to be retrieved from the blockchain -- again something else that Armory's TxDP structure bundles with it, but a raw unsigned tx from Bitcoin Core does not have."
I think you could make an additional query to the blockchain from bitcoin-core to find this info, AFAIK that is what we are doing now, but the answer to the above question is basically Armory has metadata that has to be fetched while online first, if going from bitcoin-core tx -> Armory tx etothepi, could you let us know what exactly that info is? Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: etotheipi on May 24, 2014, 10:19:23 PM "However, even if that was fixed, you still need the previous TxOut scripts in order to execute signing, which have to be retrieved from the blockchain -- again something else that Armory's TxDP structure bundles with it, but a raw unsigned tx from Bitcoin Core does not have." I think you could make an additional query to the blockchain from bitcoin-core to find this info, AFAIK that is what we are doing now, but the answer to the above question is basically Armory has metadata that has to be fetched while online first, if going from bitcoin-core tx -> Armory tx etothepi, could you let us know what exactly that info is? https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki With multi-sig coming, that whole message format changed, and BIP 10 is obsolete. But at least right now, everything you need to know is there. It's "draft" because I was the only one doing offline tx and no one wanted to contribute to making a standard out of it. The new version for multi-sig also natively handles offline tx (it's just a 1-of-1 "multi"-sig). I'll document it once design flux settles down. Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: PhantomPhreak on July 09, 2014, 01:59:15 AM "However, even if that was fixed, you still need the previous TxOut scripts in order to execute signing, which have to be retrieved from the blockchain -- again something else that Armory's TxDP structure bundles with it, but a raw unsigned tx from Bitcoin Core does not have." I think you could make an additional query to the blockchain from bitcoin-core to find this info, AFAIK that is what we are doing now, but the answer to the above question is basically Armory has metadata that has to be fetched while online first, if going from bitcoin-core tx -> Armory tx etothepi, could you let us know what exactly that info is? https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki With multi-sig coming, that whole message format changed, and BIP 10 is obsolete. But at least right now, everything you need to know is there. It's "draft" because I was the only one doing offline tx and no one wanted to contribute to making a standard out of it. The new version for multi-sig also natively handles offline tx (it's just a 1-of-1 "multi"-sig). I'll document it once design flux settles down. Hi. I don't mean to necro-bump! I'm trying to implement bip-0010 with no signing and no multi-sig. Here's what I have now (Python 3): Code: txdp = [] And here is some example output: Code: -----BEGIN-TRANSACTION-ENeepxUG------------------------------------------------- When I paste this into Armory, running offline with --debug, I don't get any error messages except the following: Code: X Error: BadMatch (invalid parameter attributes) 8 The signing window reads only "Unrecognized!". Thoughts on what I'm doing wrong? (It must be something obvious.) Any help would be greatly appreciated! Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: fthoughts on July 09, 2014, 09:17:56 PM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray.
For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: PhantomPhreak on July 10, 2014, 04:11:35 AM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray. For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: I'm looking at https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki (https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki), and, yes, at the code itself. These lines, in particular, are what I'm working off of: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py). Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: rbdrbd on July 11, 2014, 02:39:02 PM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray. For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: I'm looking at https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki (https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki), and, yes, at the code itself. These lines, in particular, are what I'm working off of: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py). I'm actually xnova (another one of the Counterparty devs). Any assistance here would be very much welcome. Getting this working would allow us to add Armory offline signing support to counterpartyd and Counterwallet, which would be great for our users and probably add dozens of more new armory users, at the very least. Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: etotheipi on July 11, 2014, 03:01:19 PM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray. For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: I'm looking at https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki (https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki), and, yes, at the code itself. These lines, in particular, are what I'm working off of: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py). I'm actually xnova (another one of the Counterparty devs). Any assistance here would be very much welcome. Getting this working would allow us to add Armory offline signing support to counterpartyd and Counterwallet, which would be great for our users and probably add dozens of more new armory users, at the very least. Guys, I apologize for not being responsive here. I'm up to my eyeballs in a variety of other things. After 0.92, I will have some time to discuss this further, though I would then encourage you to switch to the new format we are using which handles offline transactions and multi-sig alike (you don't have to use the multi-sig stuff, it's simply a lot cleaner and more flexible). Until then, everything that you need should be in serializeAscii() method: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py#L1246 From a glance, I'm not sure that you should be using .decode('utf-8') when doing the serializations. This is binary data, not unicode. Construct all pieces as raw binary and convert to hex. There shouldn't be any utf-8 calls in it. In the next release, we switch to UnsignedTransaction (https://github.com/etotheipi/BitcoinArmory/blob/devel/armoryengine/Transaction.py#L1934), which handles more general transaction types and securely signing with offline-multi-sig devices. But it will become the new standard for Armory offline transactions and multi-sig. And serialization is simpler: create one giant binary string and convert it all to base64. Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: PhantomPhreak on July 11, 2014, 03:23:57 PM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray. For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: I'm looking at https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki (https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki), and, yes, at the code itself. These lines, in particular, are what I'm working off of: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py). I'm actually xnova (another one of the Counterparty devs). Any assistance here would be very much welcome. Getting this working would allow us to add Armory offline signing support to counterpartyd and Counterwallet, which would be great for our users and probably add dozens of more new armory users, at the very least. binascii.hexlify() returns ASCII-encoded hex only, at least in Python 3: Code: Python 3.4.1 (default, May 19 2014, 17:23:49) Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: PhantomPhreak on July 11, 2014, 03:28:21 PM Hm, I'm curious, how or where are you getting the data to reconstruct the Armory tx? If you haven't looked at their code, I think this approach is perilous, since Armory's custom format will surely throw your code in disarray. For example, I'm not sure if the bytes from utxo's are appended like this: Code: for coin in inputs: I'm looking at https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki (https://github.com/bitcoin/bips/blob/master/bip-0010.mediawiki), and, yes, at the code itself. These lines, in particular, are what I'm working off of: https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py (https://github.com/etotheipi/BitcoinArmory/blob/master/armoryengine/Transaction.py). I'm actually xnova (another one of the Counterparty devs). Any assistance here would be very much welcome. Getting this working would allow us to add Armory offline signing support to counterpartyd and Counterwallet, which would be great for our users and probably add dozens of more new armory users, at the very least. Guys, I apologize for not being responsive here. I'm up to my eyeballs in a variety of other things. After 0.92, I will have some time to discuss this further, though I would then encourage you to switch to the new format we are using which handles offline transactions and multi-sig alike (you don't have to use the multi-sig stuff, it's simply a lot cleaner and more flexible). [snip] In the next release, we switch to UnsignedTransaction (https://github.com/etotheipi/BitcoinArmory/blob/devel/armoryengine/Transaction.py#L1934), which handles more general transaction types and securely signing with offline-multi-sig devices. But it will become the new standard for Armory offline transactions and multi-sig. And serialization is simpler: create one giant binary string and convert it all to base64. Is the new format stable now? For how long will the current format be supported? Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: etotheipi on July 11, 2014, 03:34:08 PM Is the new format stable now? For how long will the current format be supported? Yes, we've done 5-way simulfunding and 7-of-7 multi-sig spending with this format and we plan to move forward with our release in its current form. We expect that in the future, a variety of multi-sig services will standardize these formats to allow interoperability, but many such services are scrambling to get anything functional and standardizing would inhibit the design process. So far, we see no reason that we will have to change the format at all, until that standardization occurs -- in which case everyone will be changing. (much like how Armory and Electrum had their own deterministic wallet algorithms, then BIP32 was standardized and now everyone is moving torwards that). From a glance, I'm not sure that you should be using .decode('utf-8') when doing the serializations. This is binary data, not unicode. Construct all pieces as raw binary and convert to hex. There shouldn't be any utf-8 calls in it. binascii.hexlify() returns ASCII-encoded hex only, at least in Python 3: Code: Python 3.4.1 (default, May 19 2014, 17:23:49) Why do you need the last line? Wouldn't you end up with some of the binary chars being converted to ASCII/unicode characters? My apologies for being naive about python3 ... I'm not actually that familiar with it, beyond the fact that if I were to rewrite Armory from scratch, I'd use python3 from the start. Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: PhantomPhreak on July 11, 2014, 04:09:12 PM binascii.hexlify() returns ASCII-encoded hex only, at least in Python 3: Code: Python 3.4.1 (default, May 19 2014, 17:23:49) Why do you need the last line? Wouldn't you end up with some of the binary chars being converted to ASCII/unicode characters? My apologies for being naive about python3 ... I'm not actually that familiar with it, beyond the fact that if I were to rewrite Armory from scratch, I'd use python3 from the start. Going back and forth between Python 2 and Python 3 is a mindfuck. ;) binascii.hexlify() returns ASCII-encoded binary of the hexadecimal string (I don't know why). When you try to print that directly, Python prints the binary as binary (EDIT: implicitly converting to ASCII literal characters, wherever it can, just for display). Applying this to the code I wrote... With: Code: _TXDIST_f9beb4d9_ENeepxUG_0164 Without: Code: _TXDIST_b'f9beb4d9'_ENeepxUG_b'0164' Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: xnova on July 12, 2014, 10:14:36 PM Alan,
Will the new version of armory still support the BIP10 format for signing transactions offline, or is that going to be totally removed? Also, with this new format, why use binary? Why not have a JSON-formatted object of some kind thats produced (even if it's then encoded to base64 as a final step). A format like that could be much more easily read and verified via a standard JSON schema (http://json-schema.org/), instead of custom byte-packing code. That will make it especially easier for external tool creators/integrators like us to support this new format, especially, and make it easier for developers in general to debug issues surrounding it. I think another one of the issues we see with this new format is that adding any support for it (at least at this time) involves delving into the guts of numerous armory objects which have their own esoteric methods and functioning. It just feels that could be changing a lot between now and the beta release, given how tightly coupled to the implementation it appears to be, at first glance. (If that's not the case at all and the format is pretty well stabilized, that would be great though.) Title: Re: SOLVED - MADE SCRIPT: signing raw transactions from bitcoind with armory Post by: etotheipi on July 12, 2014, 10:46:46 PM Alan, Will the new version of armory still support the BIP10 format for signing transactions offline, or is that going to be totally removed? Also, with this new format, why use binary? Why not have a JSON-formatted object of some kind thats produced (even if it's then encoded to base64 as a final step). A format like that could be much more easily read and verified via a standard JSON schema (http://json-schema.org/), instead of custom byte-packing code. That will make it especially easier for external tool creators/integrators like us to support this new format, especially, and make it easier for developers in general to debug issues surrounding it. I think another one of the issues we see with this new format is that adding any support for it (at least at this time) involves delving into the guts of numerous armory objects which have their own esoteric methods and functioning. It just feels that could be changing a lot between now and the beta release, given how tightly coupled to the implementation it appears to be, at first glance. (If that's not the case at all and the format is pretty well stabilized, that would be great though.) Unfortunately, we removed BIP 10 because it was not used by anyone else, and the new format completely encapsulated everything we needed for both regular offline and multisig. Hence, if you upgrade your online computer to 0.92+ you'll have to do the same for the offline computer. It's the first time in 2-3 years that we've broken compatibility with older offline systems, but we felt we didn't have a choice. Everything is far simpler for us this way. As for the binary formats: most of what's moving between systems is binary data. And all systems have the standard set of get/put VAR_INT, VAR_STR, etc, the same as serializing and unserializing other Bitcoin objects. We had pondered protobuf, but we ran into some issues getting it working cross-platform. Maybe in the future we will. As I mentioned above: we've done 5-way simulfunding and 7-of-7 lockbox spends with the new format, and have no issues. Our barrier to release is stabilizin the GUI -- we see no reason we will have to further modify those formats. Also, a lot of things in that new format are forward-thinking, like space for payment protocol or other web-of-trust mechanisms (authMethod & authData), and wallet-defined strings to help lite-wallets find their own addresses (wltLocators). Those can be left blank right now (as Armory does). The rest of it is just raw scripts, raw public keys, raw transactions, some metadata & labels, etc. The only thing that might be unusual is the way we compute lockbox IDs, but it's not complex. I do admit it may have to be updated in the future -- usually something this complex you can't get everything right the first time. But at least for now, this works for 100% of Armory's simulfunding and lockbox activities, securely works with offlien devices, and even has a bit of foresight built-in to handle lite wallets and authentication of addresses. I expect this will be sufficient for a very long time, maybe even until the community determines how they want to standardize this, and then we'll upgrade (as well everyone else) |