I mentioned in my original post about this that this part is not trivial.  It's because selecting coins for a transaction is not a one-shot process -- you have to declare what your desired output amount is and your desired fee, then Armory will create the best transaction it can based on those parameters.  If the transaction has properties that require a bigger fee, then you need to re-select coins with the new fee.  Theoretically, this should be part of a while-loop or something (in case the new selection requires an even-bigger fee), but I designed the SelectCoins algorithm to use a selection that leaves some margin (if possible) so that a higher fee can be included without changing the coin selection.

The functions for reference are:
validateInputsGetTxDP() at qtdialogs.py:4867,
createTxAndBroadcast() at qtdialogs.py:4811
(Maybe)broadcastTransaction() in ArmoryQt.py:1889).  
PySelctCoins() in armoryengine.py:5049.  

There's a lot of code here, because it is also validating the data was entered properly on the Send-Tx dialog.  But what you need is there.  I'll try to summarize it here.

PySelectCoins arguments:

• unspentTxOutInfo: a list of unspent TxOuts, retrieved via wallet.getTxOutList("Spendable") (qtdialogs.py:4968)
• targetOutVal: total amount to send to (all) recipients(s), in satoshis  (i.e. 10.0*ONE_BTC for 10BTC)
• minFee: you can always set this to 0 the first time, and re-call it with higher if the output suggests more
• numRand (optional): number of randomized SelectCoins solutions to try (more about this in the appendix, below)
• margin (optional): how much margin you'd like to build into the selection, for the reasons stated above

Actually executing the transaction -- you must get all the spendable coins, make a coin selection, then calculate the change and generate the recipient list (you will need to wallet.getNextUnusedAddress() generate a change address, or you can do something else).  Shuffle the list.   Then create the "PyTxDistProposal" -- this is an intermediate step where the transaction is constructed as a BIP 0010 packet, in case it needs to be signed by an offline computer.  In the case that you have the full wallet -- this TxDP will be signed immediately, then converted to a standard transaction and broadcast.

And finally, here's the code sample you've been waiting for:  Create one transaction from a wallet, to send 10.0 BTC to one recipient


I suppose I could encapsulate much of this code in a more friendly way.  On the other hand, you can pretty much copy this code and make the appropriate adjustments, and you'll get a signed transaction.  You can modify it for multiple recipients, handling fees differently (but you should trust the calcMinSuggestedFee[1] value), and saving off copies of the transaction before sending.

I will include more about actually broadcasting the transaction, in a bit.  The problem is that you need to start the networking engine in order to actually broadcast it.  On the other hand, you have a PyTx object (finalTx), so you can call binary_to_hex(finalTx.serialize()) to get the raw hex version of it, which you might be able to broadcast in other ways.  (like "sendrawtransaction" RPC call with bitcoind)

I know you'll have questions.  Fire away!

---

Appendix: The way PySelectCoins works:  a bunch of deterministic coin-selections (solutions to the pseudo-knapsack problem) are generated based on a variety of standard coin pools (the types and sizes of unspent TxOuts for your wallet).  A few random solutions are thrown on top to generate some variety.  Usually, the deterministic algorithms find the best answer, but for strange coin-pool make-ups, some semi-randomized selections will produce decent answers.  In all, we get about a couple dozen possible solutions, and then each one is evaluated for "fitness" ... how big is the fee?  how many addresses are combine?  how obvious is it which output is the recipient and which one is change?  The relative weightings of each of these is defined in armoryengine.py:5000 -- the WEIGHTS variable defines how much relative weighting to use for each factor.  There's six of them, though it really could've been reduced to 3 important ones:  tx-fee/allow-free, input-anonymity, and output-anonymity.  


You can manually modify these weightings, or find a way to inject a new weights matrix into the PyEvalCoinSelect (I apologize, I didn't make that easy, you might be better to just manually change armoryengine.py, for now).  What do these mean?  The default behavior is to avoid using zero-conf coins at all costs.  Any solution with no zero-conf coins will be scored higher than any solution with zero-conf.  The next two biggest weights are "ALLOWFREE" and "NUMADDR".  This means that equal weighting is given to "please give me a transaction that can be free" and "please use as few addresses as possible on the input side".   "TXSIZE" and "PRIORITY" are next, but probably shouldn't even be there, because they are already tied into the ALLOWFREE and NOZEROCONF scores.  

Finally, OUTANONYM is the last critieria.  Given a set of coin-selects that are basically all the same for the other 5 criteria, it will select the solution that has better output-anonymity.  i.e. -- if the outputs are 10.0 and 1.3482023, it's fairly obvious to someone observing the blockchain which one is the recipient, which is the change-back-to-self (1.3482023).  However, if a solution is found that gives 11.34 and 5.11 -- it's not so obvious anymore, and that solution will get a higher score.  It gets extra credit if it's deceptive:  if the recipient is getting 1.384 and it finds a solution that has a change of 8.1:  then most would observers would make the wrong conclusion.

The point of this description was not only to describe the weightings, but suggest you change them for your own usage.  I had planned to make this customizable in Armory, I just hadn't gotten around to it.  If all you care about is input-anonymity and output-anonymity, I suggest changing NUMADDR and OUTANONYM to high numbers, and change the other 4 to something tiny.  You might pay a fee when you otherwise wouldn't have to, but it will be more anonymous.

DISCLAIMER:  I make no guarantees about transaction anonymity at all.  Use of this feature is completely at your own risk.  This is implemented solely for the purposes of declaring a rough ordering for an under-defined set of solutions.  Setting any particular weight to 10¹⁰ and others to 0 does not mean you are guaranteed a "good" solution -- there may only be one solution.

For the record:  I'm still against brain-wallets.  But there is such intense, passionate interest in them by so many users, that it's difficult to ignore.  I absolutely insist users always have a printed backup, but some people are absolutely intent on taking their own bitcoins to their grave (literally, in some cases).  Even though I use a different alphabet for Armory wallet seeds, someone figured it out and wrote a tool for converting any passphrase into a wallet-recovery string.  I can't do anything about that, except offer my disapproval.

So while I disapprove of the practice, at least this proposal by Peter, sets up a system whereby the popular clients will not accept brain-seeds unless they have this 1/2^N property:  which means the user is forced to add a certain amount of entropy (by searching random suffix strings) to their otherwise-crappy passphrase.  I think this goes a long way towards mitigating the issues we don't like about brainwallets.  It might be just enough convenience that users will do it (add entropy), but not inconvenient enough that they are motivated to evade the mechanism (such as modifying client source code).

Btw, for anyone having connection issues with (standard) bitcoind/-qt, I think I've found a bug in the connection code.  It will only fix startup-connection issues, but I'm hoping all the networking issues are related and will all go away with this (I can't replicate them, so I can't test it).  I have compiled a windows installer with the bug-fix:

http://dl.dropbox.com/u/1139081/ArmoryTestingReleases/armory_0.82.5-alpha_netfix.msi

If you are in Linux, you can check out the "dev" branch and recompile. 

Unfortunately, I can't comment on how Armory behaves behind proxies and/or Tor.  However, if it stops loading at the same block every time, my guess is it's reading an old blk0001.dat file, and actually has nothing to do with the proxy.  Do you keep your bitcoind/-qt datadir somewhere other than the default location?  If I had to guess, I'd say you used to have bitcoin-qt in the default location, and later moved it out. 

If so, use the " --satoshi-datadir=/path/to/bitcoinqt/homedir" to tell Armory where to find it.  This is because Armory requires the blk000X.dat files created by the currently-running version of bitcoin-qt. 

The transaction showed up in Armory because it saw the zero-confirmation tx forwarded by bitcoind.  But it will never see it enter the blockchain, because it's stuck on block 117,519, and obviously your tx is going to enter the blockchain much later than that.

It seemed there was a lot of dissent in this thread about that advice.  I wanted to make sure that dissenters understood the basis of that advice, and either confirm it or make further recommendations about how to address the concerns I put forward.  These are legitimate concerns, and I think it's critical that it's done 100% right, for this first swing into the open political scene ("right", from a legal and accountability standpoint).  I'd rather some Bitcoiners be inconvenienced, than have Mark accused of shady campaign financing practices.  Obviously we'd like to avoid both, if possible

Guys,

I just want to repeat the advice that I gave Mark via email, because it appears that there is a disconnect about returning funds.  I believe it is absolutely critical that:

(1) Funds be returned to exactly one of the addresses it was received from
(2) A large warning is posted on the donation page, notifying that that is the policy and no exceptions will be made

This is in Mark's best interest.  There are two serious issues with returning to different addresses:

(1) With addresses being semi-anonymous, returning to a different address looks suspicious.  It could have been Mark claiming to return the funds to someone, but actually siphoning them off to a secret address he created for himself.  I'm not, in any way, accusing Mark of doing this.  But there's no way to prove that he didn't do that.  The only way for an oversight committee to know for sure that the coins were actually returned is if it goes back to the same address

(2) There's all sorts of money-laundering issues with returning to a different address.  I know it doesn't make much sense for someone to do it, but theoretically someone wants to pay their drug dealer $10k BTC, so they make an invalid donation to Mark's campaign, then has Mark return it to his drug dealer's address.  Mark was now unknowingly an accomplice.  While the possibility of that happening is pretty low, I think there's actually a greater risk of someone setting up something like this in order to blackmail and/or discredit Mark.  

It's just not worth the risk.

If someone donates from an online service, then I still recommend it be returned to the sender, and Mark can send that person the TxID of the return transaction.  Then the user/donor can go sort it out with the online service, on their own time.  They will recover the funds, it just might take some work. EDIT:  Mark's campaign can even provide a signed message to be given to the service that says "Transaction with ID: ... is a return transaction for the following user: ...".   This will make it much easier for the user to recover their funds.

Exactly.

I recommend using an offline computer to make the full wallet, then make a watching-only copy.  Put it on your webserver, and also import it into Armory on your primary system.  Then your webserver will distribute addresses, and you can see the full transaction history from your desktop.  If someone gets the watching-only wallet... well all they can do is watch, but they can't take any funds!

I don't recommend generating addresses from the watching-only wallet, while the webserver is doing the same.  There won't be any fireworks, but you might end up re-using an address by accident.  If you want to also be able to distribute addresses from your primary system, I recommend making a second wallet on the offline computer just for that purpose.  You can watch them both from Armory, but only distribute addresses from the one the server isn't using.  Then you can also execute offline transactions from either wallet using the online computer.

As a client developer, I want to discourage "simple" seed generation.  A user can always go to the "Recover Paper Wallet" dialog, and plug in his own sequence of characters, which may be something like sha256("Bob"), assuming no one else would ever do the same thing.  You'd like to blame the user when his coins are stolen, but the application itself will get blamed for security issues, and the user is not likely to admit they used such a simple wallet key.  This is why I wanted to make it difficult for users to enter custom strings into the wallet-recovery dialog in Armory.

What this proposal does, is it allows me to pare down the space of valid keys to 1/2^N.  This is beneficial for not only key-strengthening, but forcing the user to add some entropy to their key.  If Armory will only accept keys that follow the pattern Peter described, then sha256("Bob") is not going to be accepted.   The user will be required to add N bits of real entropy to it to make it valid.   I'd almost be okay with a program that helps them do that.  Because while sha256("Bob") will be an easy key to break, sha256("Bob_g4982fX") is much better but still memorizable by a human.  So for users that want brain wallets, you're forcing them to accept extra entropy to their wallet key, while also slowing down someone who is trying to brute force it. 

I posted that calculation a long time ago.  I checked again a month ago, and it was still valid:

https://bitcointalk.org/index.php?topic=29675.0

Summary:  Block 125,552 had a hash that would've been valid at at a difficulty of 36,000,000,000.

However, this has nothing to do with "breaking" sha256, or anything anything of the sort.  It's just luck.  Your statement would be like saying that "one time I rolled 10 dice and they all came up 1, so I must've come close to breaking the dice-rolling game."  SHA256 is no more or less broken because of this.

I had spent a little bit of extra time looking at it, but not a lot.  I had been meaning to reply to this thread, with some updates on my philosophy on this topic:

(1) Physical cables connecting the two computers can make people really unconfortable.  I feel really uncomfortable, even knowing that nothing can happen
(2) There are some extra lockdown procedures to execute to make sure that your version of linux doesn't allow logins over the serial connection.  I was already aware of this, but it dawned on me that this could actually have a net negative impact if I convince people to use this system and a certain number of them don't set it up properly.  I'd rather use USB keys than risk impatient people just hooking up the cable and carrying forward because they don't realize that issue.

As such, I had started to think that maybe the serial cables wasn't the best idea, except as a "niche" technique for the ultra-motivated.  I imagine there are plenty of people capable of doing the serial cable version, but many of them may not want to even see the cable connecting the two computers where bits could theoretically be flowing.

However, I have not thought about any particular way I'd like to resolve this.  I think the ultimate solution is what you alluded to:  a handheld device with a small screen and a few buttons.  Outside a dedicated chip, the wallet itself would be write-only, and and the chip would simply be a blackbox with transaction data going in one side, and signatures coming out the other.  However, I'm not a hardware guy, so I wouldn't know the first thing about how to pursue this idea.  And I have ideas... I just can't make them realizable.

The end of this story is this:  I'm all for developing a serial-cable solution.  But I'm also dedicated to developing Armory for the general user, and I'm not sure serial cables are fitting in this case.  So I will be happy to contribute and even compile official instructions for how to do serial cables (and maybe put in a small interface for pulling tx data from a serial port), but I can't make it my personal priority right now.

Absent a dedicated device/dongle, I am still interested in other ways that might work.  The webcam+QR idea is in the right direction, but there's a variety of reasons why it's not good for the general user (I've mentioned it before).

I need more details to understand how this works.  You just say, "this isn't a requirement" and "done deterministically using the redblack deletion rules."  Those rules get you a different tree.  Every level of the redblack tree might've unrolled/reorganized, all the way up to the root.  Without saving the entire structure of the tree after every insertion (which would be an ungodly amount of data), I don't know how you can expect to reverse those operations on a reorg. 

You keep mentioning how much data you need to download, but that's not what I'm worried about.  I want to know what, and how much, data has to be saved before every insertion and deletion (or block of such operations), to guarantee that your tree at time T can be rolled back to time T-1.



I'm not sure what you mean?  The hashes have no relevance to the structure of that tree:  only the (addr,txhash,idx) values matter.  You can parallel-insert or delete all the necessary leaves to get the correct structure, then walk up the tree from each leaf, recomputing the hashes for the branch nodes as you go.  Each sub-branch can be done completely independently, with only one process at the end to merge them into a single root hash.  What am I neglecting?

You're focusing a lot on access speed.  Getting the access speed question out of the way is important one, but I think dwelling on it is unnecessary.  We've established that even for billions of elements, the trie and BST will have the same, absolute order of magnitude access time.  The trie has better theoretical order of growth with respect to number of elements (O(1)), but it's absolute constant access time will be comparable to the BST's O(logN) for N < 10^10.  I'm already satisfied with the conclusion:  "both are fast enough."

So then my concern remains with regards to order-independence.  Two particular concerns pop out that I'm sure you have figured out, but I want to make sure:

(1) How can you guarantee that node-deletion from the BST will result in the same underlying tree structure as you started with?  Any particular node addition could trickle rebalancing/rotation operations from the bottom of the BST to the top.  Is node-deletion guaranteed to undo those rotations?  If not, what do you need after every operation to make sure you know how to remove the node, if it becomes necessary?
(2)  If a lite-node requests all the UTXOs for a given address, how do you (the supplier) transmit that information?  You can't just give it a sequential list of UTXOs, and you don't know just from looking at the address tree what order they were inserted into the sub-branch.  How do you communicate this information to the lite-node so that it can verify the UTXO list?

Of course, I have to supplement those questions with the fact that tries don't even have to think about this.  (1) Delete a node from the trie!  If two tries have the same elements, they have the same structure.  (2) Send the UTXO list in any order:  if two tries (or trie-branches) have the same elements, they have the same structure!

I don't want to derail this thread too far in the trie-vs-BST direction.  I just want to make sure there's good answers for these questions.  I really do appreciate that someone has put in time to try to make a proof-of-concept.  Even if I don't agree with the particular optimization, there's a lot of value/education in actually trying to put the theory into practice, and the lessons you learn will help us out in future -- or you will simply finish it for us and then we will gratefully leverage your accomplishment!

So, thanks for pioneering this effort!

P.S. -- You argued in a PM, against the point that a trie can be parallel-updated/maintained.  But I didn't understand the argument.  I can design a system where the top-level branch node (branching on first byte), split up between X threads/HDDs/servers.  Each server handles 256/X sub-branches.  When a new block comes in, all additions and deletions induced by that block are distributed to the appropriate server.  The main thread waits for all the servers to finish -- which happens by reporting back the updated "roots" of their sub-trees -- then the main thread combines those into the final, master root.  This seems like a valid (major!) benefit.  

Temporarily putting aside the debate about insert-order-dependence and tries-vs-BSTs, why not make all nodes "leaf" nodes?  The traditional BST doesn't actually distinguish -- it just has a left and right pointer and if they're both NULL, it's a leaf.  What I'm suggesting is that every node contains an element of the set, and while searching for that element you might stop before getting to the bottom of sub-branch, because the branch node itself is an element.   It's "value" is the hash of some concatenation, such as sha256(elementHash || LeftPtrValue || RightPtrValue).  It seems like it might save some space, though I can't figure out off the top of my head whether it's significant.  Or does that break some property of the tree you've shown here?

Unfortunately, I am not too adept with the networking side of things, and what's going on under-the-hood with proxies, VPN, etc.  I've been quite happy that python-twisted and urllib2 do a good job handling all that transparently, but it means I need some time to familiarize myself with how non-standard setups work in order to make it more flexible.

However, if I'm not mistaken, the real problem is Armory going into offline mode when it actually shouldn't.  You need a way to force it to think it's online, even though it can't access google.com to verify (it's got the bitcoind/-qt connection, so it doesn't matter).  For that reason, I will add to the next version a --force-online flag that allows you to over-ride that behavior.

For now, ArmoryQt.py:890 is the line where it determines whether you should be in online mode based on the previous method.  You can comment that out and replace it with "self.isOnline = True" and try rerunning it.  Please try that and tell me whether it works and/or causes any strange behavior!

Red Emerald,

I just committed a Makefile modification to the dev branch.  It's basically just the current master with a bug fix and the updated Makefile.  Can you do a fresh checkout and confirm it works on your system, too?  Just want to make sure I did it right.  It works on the OSX VM I have.


I will try it, too.  I just don't have high expectations.  I also had to reinstall my Win64 VM and can't get py2exe working.  I've spent hours on it and even my existing config that works on another machine isn't helping me figure it out.  Not looking forward to doing it again on OSX...

I definitely can't bank on the user having the stuff installed already.  It seems like it would be... "irresponsible"... to have the install script actually install brew and all the brew packages.  Perhaps there's a way to make a completely self-contained directory tree (that has python and all dependencies pre-compiled and included).  I don't see why not, it just seems like a mess.

Maybe I should try py2app... but so far I've only heard gripes about things not working right, so I imagine I won't be figuring it out, either (I'd need some serious tinkering time available). 

So, perhaps someone here could donate some time to try this.  I bet it would be even easier with:  Portable Python.

More specifically, each new block is appended to the blkXXXX.dat files as they are received.  Their format is pretty simple:

Well, I got Armory working on OSX!  I need to play around with it a little bit, but it looks like it mostly works (except for some of the text labels being cut off on the bottom... might consider some OSX-specific settings for the GUI).  I haven't actually moved any coins around with it yet, but creating wallets worked, and I was even able to change the filter (didn't freeze... though I didn't have any tx in it...)

Red Emerald:  thanks so much for the great instructions!  I did some digging around and found about 30 different copies of python and various subdirs that were all candidates for linking/include options and environment variables.  I even tried some of the others thinking I knew better than you (I wanted to try to get everything running from the same python directory tree), but nothing worked except exactly what you posted.  Thanks!




I'm a little bothered by the over-protectiveness of Apple (signing keys, app store reviews, etc), but I guess that's one reason many end users like them, but it really makes me not want to develop for them. 

I found a couple links for creating dmg files.  Perhaps I can copy in all the necessary site-packages directories and try to override the PYTHONPATH variable is part of running when OSX is detected.  Then it looks like I just create the .dmg and the user can expand it and drag it into the applications folder...?

http://stackoverflow.com/questions/116657/how-do-you-create-an-osx-application-dmg-from-a-python-package
http://stackoverflow.com/questions/96882/how-do-i-create-a-nice-looking-dmg-for-mac-os-x-using-command-line-tools

It looks like those two links together might give me what I need to create an automatic-dmg-creator for Armory.  I just have to experiment a bit...

(Disclosure:  I am biased, since I wrote Armory)  This is one very good reason to be using deterministic wallets like Armory uses (and other alternative clients).  You create an encrypted wallet, and before you use it the first time, you create paper backups (which are unencrypted, but you can encrypt the information manually before printing, if you want).  Put it in a safe-deposit box or any other means you consider secure.

Now you can generate an infinite number of addresses, and never have to worry about permanent loss. Ever.  If the wallet is corrupted or HDD fails, just get a new computer, and restore the wallet from the paper backup.  All money is recovered.

And of course, Armory additionally lets you maintain the private keys offline but still use a watching-only wallet to generate/distribute addresses and monitor payments, but without risk of compromise.  If you need to move the coins, use a USB key to get the offline computer to sign the transaction.  Of course, the offline computer should have limited access, but you can still keep the keys encrypted on it in case someone does get access to it.  I designed this feature with businesses in mind (and regular users, but businesses really require this level of robustness and security)

Done!  Will commit it in the next release.  I haven't really tested it, but I just commented out all the code that relies on bsddb, and it shouldn't make a difference because the only module that calls any of that code is the Satoshi-wallet import code that is disabled anyway.

I have been doing one command line call per day, between random other things.  So far, brew has been good to me.  I will try to finish the instructions you posted and maybe I'll finally have Armory working on OSX!

Anyone have advice for how I can distribute it?  The OSX equivalent of py2exe does not look... reliable.  I'll try it, but I would prefer to avoid the complexities and run it as a native python script, if possible.  Perhaps bundle up all the site-packages into the install directory so that it can be standalone without having to install extra dependencies...? 

I'm guessing that packaging and distribution will be the biggest hurdle to getting Armory released on OSX.  Perhaps someone with more OSX experience can provide guidance...