Sweep/import private key feature request

[etotheipi]

This is a good discussion, and thanks for letting me know how I can add $500 income to my first release of Armory.  I hope that bounty is still valid   But I'm also interested to converge on a good way to "responsibly" implement these features...

Regardless of the Armory software itself, I have proven with the PyBtcEngine/ArmoryEngine that it's possible to scan the entire blockchain for a new address within a reasonable amount of time:

• If you're holding the entire blockchain in memory, the code can find all transactions in less than one second. 
• If it's not in memory, then I can load the blockchain from disk, index it, and scan for the address in less than 15s.

(the top-layer is Python, but I have a super-optimized C++ layer doing all the blockchain operations).

It might be possible to wrap the C++ engine and link it as a dll, purely for doing these types of scans.  It sidesteps a lot of the issues expressed in this thread (in the medium term, when it's still feasible to hold the entire blockchain), as you don't have to trust anyone more than you trust the blockchain itself in order to find and sweep funds for a given address/key. 

As for the Armory software itself, I already have the ability to import external private keys, and appropriate warnings have been displayed on the import dialog.  You can always add a new wallet just for importing untrusted private keys, though I recognize that it can still be dangerous for folks who don't quite understand the under-the-hood stuff.   The default behavior is to import the key as a permanent address in your wallet, but after reading this thread, I might make "sweeping" be the default behavior.  I might also move this feature to the "advanced" mode.  Recommendations are welcome.

[casascius]

• If it's not in memory, then I can load the blockchain from disk, index it, and scan for the address in less than 15s.

Somehow, I suspect that this scan will be highly dependent on the blockchain file not being fragmented on disk, or being on flash memory.  If overly fragmented, the hard drive simply may not be able to deliver the whole file in 15 seconds regardless of how well the code performs.

I think an index is desirable regardless, because at best, an optimized blockchain scan scales O(n) and an index scales O(log n)... as the block chain gets bigger, the index scan will be faster than the full scan by increasing orders of magnitude.

[etotheipi]

• If it's not in memory, then I can load the blockchain from disk, index it, and scan for the address in less than 15s.

Somehow, I suspect that this scan will be highly dependent on the blockchain file not being fragmented, or being on flash memory.

I think an index is desirable regardless, because at best, an optimized blockchain scan scales O(n) and an index scales O(log n)... as the block chain gets bigger, the index scan will be faster than the full scan by increasing orders of magnitude.

I don't disagree with you, and my goal for the first release of Armory is feature-based proof-of-concept, with no concern for how heavy the client is.  In the future, I will be converting to index-based approaches, as you suggested.  I actually already have a method for indexing all addresses, I just don't have it integrated as the "go-to" source of information for addresses.  That will be investigated for the next release.

But it doesn't change the fact that is still works right now, and even it's 30s-60s for a scan, it's a still "workable" as a client feature for users that want their money.  If someone wants to use the library in their own client for this purpose, I can help them get it integrated/linked.

[netrin]

Isn't a radix hash index O(1), or constant with respect to chain size? The performance look-up penalty is the number of instances of a single address, not the total number of addresses nor blocks.

[etotheipi]

Isn't a radix hash index O(1), or constant with respect to chain size? The performance look-up penalty is the number of instances of a single address, not the number of addresses.

It depends what kind of tree structure you're using.   If you keep a tree of every address as the key, and a list of block numbers as the values, then a trie/patricia tree will have constant access time (actually, a function of the length of the addresses).  But, it requires prefix searching and C++ doesn't have this kind of tree available in the STL.

So I have to settle for map<key,val> which is a binary tree ~ O(log(n)).  But in the grand scheme of things O(log(n)) is much "closer" to O(1) than O(n).  In this application, it's probably indistinguishable.

[DeathAndTaxes]

The default behavior is to import the key as a permanent address in your wallet, but after reading this thread, I might make "sweeping" be the default behavior.  I might also move this feature to the "advanced" mode.  Recommendations are welcome.

Yeah the way I see it there are three good choices.

Default IMPORT (option in menu to change to SWEEP).

Default SWEEP (option in menu to change to IMPORT).

Default prompt the user (w/ option "check to always xxxx").

I guess it really depends on what people are using it for.  I would recommend any wallet explain to the user what is happening (maybe w/ a help icon for more detailed info).

Something like (sweep version as summing no auto-sweep)

"You are about to sweep xxxx BTC from private key xxxxxx into your wallet.  This will not import the key.  If you receive funds at the address asssociated with this private key you will need to sweep the funds again. ( check box) no longer warn me about sweeping funds.   

[netrin]

"You are about to sweep xxxx BTC from private key xxxxxx into your wallet.  This will not import the key.  If you receive funds at the address asssociated with this private key you will need to sweep the funds again. ( check box) no longer warn me about sweeping funds.  

Why not the third option, where "SWEEP" means, "Import and immediately transfer coins"? I think "SWEEP" with second class keys is the worst of all worlds: confusing to newbies, disempowering to geeks, and complicated to maintain.

Thus with import implied, the only question is: "Do you want to transfer funds to a new trusted address?" Yes (recommended) or No (I know what I'm doing).

[etotheipi]

"You are about to sweep xxxx BTC from private key xxxxxx into your wallet.  This will not import the key.  If you receive funds at the address asssociated with this private key you will need to sweep the funds again. ( check box) no longer warn me about sweeping funds.  

Why not the third option, where "SWEEP" means, "Import and immediately transfer coins"? I think "SWEEP" with second class keys is the worst of all worlds: confusing to newbies, disempowering to geeks, and complicated to maintain.

Thus with import implied, the only question is: "Do you want to transfer funds to a new trusted address?" Yes (recommended) or No (I know what I'm doing).

My only concern with this is that it does leave open an attack vector for the person who gave you the key.  By importing that key into your wallet permanently, it's funds look like they belong 100% to you.  The person who gave you that key, can then send you money to that key to make you believe you received it, and then transfer the money back to themselves once the exchange is complete.

You could argue for an "auto-sweep" function/option, but I am not fond of any kind of automated transaction... anything that moves money when the user isn't looking, especially when it might require a tx fee, is asking for people to stop trusting your software.  Not to mention, I would have to create a new class of keys/wallets in my software to handle "kind of trusted" private keys, and it would confuse users who now have to understand this new class of information.

I prefer the option of sweep once, throw away the key.  If the user is importing an untrusted key to be used once, then they should only use it once to sweep, and not leave themselves open to such games as above.  Otherwise, it leaves the door open for not-even-so-creative attackers to plant a key in your wallet that they own, and then start pulling your funds out from under you when you use it for other transactions.  By throwing it away, we avoid the mess entirely.

[DeathAndTaxes]

My only concern with this is that it does leave open an attack vector for the person who gave you the key.  By importing that key into your wallet permanently, it's funds look like they belong 100% to you.  The person who gave you that key, can then send you money to that key to make you believe you received it, and then transfer the money back to themselves once the exchange is complete.

+1.  My thoughts exactly.  I don't really see the case where someone would want to import and sweep. Sweep is a mechanism used to secure funds from a compromised (or potentially compromised) key. If you can't guarantee the security of the key it shouldn't be imported.  If you can guarantee the security of the key why do you need to create an unnecessary transaction to sweep the value to another equally secure key in the same wallet?

It is a good idea for everyone to get into the mindset that things we don't trust completely don't belong in the wallet.  Period.

IMPORT = FULL TRUST
Example you generated an vanity address and want to integrate it into an existing wallet.  No need to transfer funds (if any) as you implicitly trust this key. Import is essentially saying "Yes I am absolutely sure there is no chance this key has been compromised and I am confident risking future coins into perpetuity with that assertion."

SWEEP = UNTRUSTED.
Example would be someone gives you a funded private key.  It has value right now but it might not in the future.  Sweeping ensures the transaction can't be reversed but the same risk that necessitated the "sweep" remains, thus the key can NEVER be trusted. There is no reason to risk future attack by integrating a compromised (even potentially compromised) key into your wallet.    Sweep is essentially saying "Hell no I don't trust this key I have no idea where it has been.  Still I would like those coins please."

While there may be edge cases I think they are rare and not worth the confusion it would create.  Two cases is already potentially confusing.  The SWEEP vs IMPORT having a clear line helps to reinforce the broader concept that your wallet (and keys inside it) should remain private and secure.  You shouldn't put insecure stuff in there and if you feel your wallet is compromised you should stop using those keys.  The risk isn't just in the here and now but all future value in those keys into perpetuity.

[netrin]

Why not the third option, where "SWEEP" means, "Import and immediately transfer coins"? I think "SWEEP" with second class keys is the worst of all worlds: confusing to newbies, disempowering to geeks, and complicated to maintain.

Thus with import implied, the only question is: "Do you want to transfer funds to a new trusted address?" Yes (recommended) or No (I know what I'm doing).

My only concern with this is that it does leave open an attack vector for the person who gave you the key.  By importing that key into your wallet permanently, it's funds look like they belong 100% to you.  The person who gave you that key, can then send you money to that key to make you believe you received it, and then transfer the money back to themselves once the exchange is complete.

Then that can and must be handled at the user interface level. Not twisting up the data model.

Flag the address in the wallet as untrusted/imported and mark it with skull and cross bones in the GUI or: "100 BTC (20 untrusted)" in the HIGHLY UNLIKELY case of this occurring. You could also perpetually import coins sent to imported addresses to new addresses, but second class keys in the data model is a backassward design.

You could argue for an "auto-sweep" function/option, but I am not fond of any kind of automated transaction... anything that moves money when the user isn't looking, especially when it might require a tx fee, is asking for people to stop trusting your software.

It can and should be optional, recommended or not. It is far worse to have strange behavior ALWAYS, rather than strange behavior in the unlikely and potential malicious case of someone planting a trojan-address in your wallet.

Not to mention, I would have to create a new class of keys/wallets in my software to handle "kind of trusted" private keys, and it would confuse users who now have to understand this new class of information.

How is this 'new class' of flagged key different from your 'second class' swept key?

I prefer the option of sweep once, throw away the key.

That's draconian and at best it could be optional though I am sure no one would elect to throw a key away. Why should anyone ever throw a key away? On this point Satoshi would agree with me.

[DeathAndTaxes]

That's draconian and at best it could be optional though I am sure no one would elect to throw a key away. Why should anyone ever throw a key away? On this point Satoshi would agree with me.

I sure as hell would (see post above).  

You either trust a private key or you don't.  By trust I mean you are 100% absolutely certain that nobody else on this planet has ever seen the private key.

If you trust a key why do you need to sweep it?
If you don't trust a key why are you putting something you don't trust in your wallet?

[etotheipi]

How is this 'new class' of flagged key different from your 'second class' swept key?

I prefer the option of sweep once, throw away the key.

That's draconian and at best it could be optional though I am sure no one would elect to throw a key away. Why should anyone ever throw a key away? On this point Satoshi would agree with me.

Netrin, I sympathize with your arguments, but I can't concede to them.  As an application developer where I will be implicitly responsible the for well-being of a lot of peoples' money, I just simply cannot do it.  It's not to say that it shouldn't ever be done: maybe another developer would have the patience and creativity to design such a system and add more value than risk, but that's not me.  When something goes wrong, my fault or the users', I'll still be dealing with it.  And I will deal with it by putting in the 90% solution as we just discussed... so far I haven't been compelled by this discussion otherwise.

But it's not a lost cause for the argument you made.  Yes, throwing away the key is strange.  The default for "sweep" will be to for them to put in the private key, the client sweeps it, and then it's never seen again.    But that's not really the end of the story... the user still has the private key.   If they disagree with the fact that they don't have the option to sweep and import... then just run the dialog again and make it a permanent part of your wallet.  Done.   

So my client effectively "supports" that option and advanced/smart users who want this behavior can get it.  I just refuse to directly support/encourage it, because as has been described above, it opens up much more risk to the user than the value it adds.

[netrin]

That's draconian and at best it could be optional though I am sure no one would elect to throw a key away. Why should anyone ever throw a key away? On this point Satoshi would agree with me.

I sure as hell would (see post above).

Look either you trust a key or you don't.  By trust I mean you are 100% absolutely certain that nobody else on this planet has seen the private key but you.

If you trust a key why do you need to sweep it?
If you don't trust a key why are you putting something you don't trust in your wallet?


A import and sweep is essentially saying I don't trust the key (think if I don't sweep funds might be taken/reversed) BUT I would like to trust this key for future funds.

I misunderstood your position before your re-edit (EDIT: and Etotheipi's re-clarification). I suppose sweep and purge is at least consistent with respect to the data model. As is pure import.

Call me a pack-rat then, I abhor the idea of throwing a published address into the aether. But, my use cases are admittedly obscure. If I want to keep an untrusted key, I could just import it into a untrusted wallet. Everyone is happy. OK, I'm on board for exclusively both IMPORT and SWEEP (and forget).

[btc_artist]

For private keys that are known to be secure, I want the option to import them as first-class citizens in my wallet (think a securely-generated vanity address I want to be using).  For private keys given to me by someone else, I want to transfer the funds to a new key in my wallet, but I want to keep a list of those private keys that have been transferred from, so that if any more funds ever shows up in any of them, I can also transfer those funds to my wallet.

I think part of the inherent problem is that the Satoshi client shows transactions and your total balance. This is fundamentally wrong. It should show all your key pairs and the funds that are in each. When you send a payment, you choose one or more keys to send the funds from. If it was done this way, each key/address pair could be marked as generated in the client or imported.

If I have a bunch of USD in my wallet in my pants, the actual physical selection of bills I have is important to some degree. What if I want to tip someone with a small bill? What if I need to make a small purchase where large bills are not accepted? What if one of the bills might be counterfeit and worthless? All of this is important information and if shown in the client, mitigates many of the problems of how to deal with imported/swept private keys.

[paraipan]

yep agree with DeathAndTaxes, maybe we should think that any priv key is more like a prepaid card. The company or vendor that made the card know the priv code and you only have to scratch the protecting film to redeem the value.
In theory you could save the card in case they send any amount of value in the future but we know that isn't the reality. Every time i refill my mobile credit, that card is trashed or guarded few days just in case. Same thing could be done with "untrusted" priv keys, save them in a separate cache file that you can purge regularly.

[netrin]

For private keys given to me by someone else, I want to transfer the funds to a new key in my wallet, but I want to keep a list of those private keys that have been transferred from, so that if any more funds ever shows up in any of them, I can also transfer those funds to my wallet.

I think part of the inherent problem is that the Satoshi client shows transactions and your total balance. This is fundamentally wrong. It should show all your key pairs and the funds that are in each.

This is my way of thinking as well. But we must admit that we are not the typical user. I have little understanding of the average user's mental model (of any technology, whether email or electricity) and I imagine they have none at all but it is hopeless to try to educate them at this phase. Shouldn't the 'coin control' patch give you what you want?

I'm actually pretty cool with the idea of a 'trusted wallet' and an 'untrusted wallet'. I would like a service that does pull/sweep untrusted keys periodically. In fact, I'd like a service that randomly scrambles all of my coins while I am sleeping. These features are possible but perhaps none of them are on topic with respect to 'sweep/import private keys'.

[btc_artist]

Call me a pack-rat then, I abhor the idea of throwing a published address into the aether. But, my use cases are admittedly obscure. If I want to keep an untrusted key, I could just import it into a untrusted wallet. Everyone is happy. OK, I'm on board for exclusively both IMPORT and SWEEP (and forget).

I agree mostly.  Why not have:

1. IMPORT and keep the key as a first-class citizen in your wallet.
2. SWEEP key and TRANSFER any funds to your wallet (or another arbitrary address). There would be a separate interface (completely independent from the wallet) in the program showing a list of all keys that have been swept, and showing any balance in each of them. This would not be included in the wallets balance, unless you SWEEP them again.

[DeathAndTaxes]

I think part of the inherent problem is that the Satoshi client shows transactions and your total balance. This is fundamentally wrong. It should show all your key pairs and the funds that are in each. When you send a payment, you choose one or more keys to send the funds from. If it was done this way, each key/address pair could be marked as generated in the client or imported.

If I have a bunch of USD in my wallet in my pants, the actual physical selection of bills I have is important to some degree. What if I want to tip someone with a small bill? What if I need to make a small purchase where large bills are not accepted? What if one of the bills might be counterfeit and worthless? All of this is important information and if shown in the client, mitigates many of the problems of how to deal with imported/swept private keys.

Well "bills" are an incomplete abstraction of wealth.  They only come in certain sizes, you need to deal w/ counterfeits, you can accidentally give the wrong one, you can make mistakes when making change, etc.

None of those concerns are relevent w/ Bitcoin.  For 99% of users the per address value of their wealth distribution isn't material.  They only want to know three things
1) how much wealth do I have?
2) did that transaction clear?
3) are my funds safe?

The abstraction of addresses into a wallet was intentional and good IMHO.  There may be isolated cases where knowing you have 1.28928392729873894 BTC in address a, 0.1827789347389 BTC in address b ...  (hundreds if not more addresses later) .... etc  is valuable but for most people it is just noise.  It doesn't answer the three questions above.

[DeathAndTaxes]

I agree mostly.  Why not have:

1. IMPORT and keep the key as a first-class citizen in your wallet.
2. SWEEP key and TRANSFER any funds to your wallet (or another arbitrary address). There would be a separate interface (completely independent from the wallet) in the program showing a list of all keys that have been swept, and showing any balance in each of them. This would not be included in the wallets balance, unless you SWEEP them again.

I would have problem with that but it seems excessive.  I mean it would be like when I use a prepaid card for a cellphone my phone showing my current balance, transactions, and expiration and then having a separate section which shows me all the prepaid codes and their current value ($0.00).

Can you think of any instance where you would receive an insecure funded private key and you anticipate someone sending you funds there in the future? Why? Why not just use a new "disposable" funded private key? 

I really think most users have no reason to look at a list of spent 0 BTC insecure private keys anymore than they keep track of spent giftcard or phone cards.

[btc_artist]

The abstraction of addresses into a wallet was intentional and good IMHO.  There may be isolated cases where knowing you have 1.28928392729873894 BTC in address a, 0.1827789347389 BTC in address b ...  (hundreds if not more addresses later) .... etc  is valuable but for most people it is just noise.  It doesn't answer the three questions above.

The abstraction was definitely intentional, but I'm not sure it was good. It works well for casual users (the majority), but for one has fundamental privacy concerns.  So for some (most?) it would be noise, and for others it would be essential.  I don't see the problem with having the funds to address breakdown in an advanced tab that casual users can ignore. 

I agree mostly.  Why not have:

1. IMPORT and keep the key as a first-class citizen in your wallet.
2. SWEEP key and TRANSFER any funds to your wallet (or another arbitrary address). There would be a separate interface (completely independent from the wallet) in the program showing a list of all keys that have been swept, and showing any balance in each of them. This would not be included in the wallets balance, unless you SWEEP them again.

I would have problem with that but it seems excessive.  I mean it would be like when I use a prepaid card for a cellphone my phone showing my current balance, transactions, and expiration and then having a separate section which shows me all the prepaid codes and their current value ($0.00).

Can you think of any instance where you would receive an insecure funded private key and you anticipate someone sending you funds there in the future? Why? Why not just use a new "disposable" funded private key? 

I really think most users have no reason to look at a list of spent 0 BTC insecure private keys anymore than they keep track of spent giftcard or phone cards.

I assume a redeemed phone card can never have funds placed in it again.  If I were to publish an address for payment, then sweep the private key to get the funds, who is to say that someone will never send funds to that (published) address again?