Title: backup vs keypool Post by: grv on December 29, 2013, 09:58:35 AM is this a design flaw or do i just misunderstand something?
example 1: 1. send 1 btc to address in wallet A 2. backup wallet A 3. run bitcoin client with wallet A 4. make 101 send transactions, each at 0.0001 (this uses up all 100 original addresses in keypool?) 5. wallet gets destroyed (for example harddisk failure right after transaction #101) 6. restore backup of wallet A 7. will this result in 0 balance and unrecoverable funds because of disparity in keypool addresses example 2: 1. send 1 btc to address in wallet A 2. backup wallet A 3. run bitcoin client with wallet A 4. make 201 send transactions, each at 0.0001 (this uses up all 100 original addresses in keypool, replacing them with completely new set of addresses?) 5. wallet gets destroyed (for example harddisk failure right after transaction #201) 6. restore backup of wallet A 7. will this result in 0 balance and unrecoverable funds because of disparity in keypool addresses Title: Re: backup vs keypool Post by: grau on December 29, 2013, 10:21:56 AM It is poor design as you describe.
You can loose money if bitcoin-qt's wallet is not regularly backed up. Deterministic key generation schemes that overcome this problem were designed long ago and exist in many alternate implementations e.g. Armory, Electrum .... and BOP. Title: Re: backup vs keypool Post by: grv on December 29, 2013, 10:32:19 AM It is poor design as you describe. You can loose money if bitcoin-qt's wallet is not regularly backed up. Deterministic key generation schemes that overcome this problem were designed long ago and exist in many alternate implementations e.g. Armory, Electrum .... and BOP. i guess even regular backup wouldn't help in this case: even if you make an offsite backup right before running the client, if your harddrive fails while the client is running and you already made 200 transactions during that session. are you out of luck? (doesnt have to be hdd failure, any problem which would cause the wallet.dat to be destroyed or corrupted, like power outage, etc). Title: Re: backup vs keypool Post by: grau on December 29, 2013, 10:34:54 AM It is poor design as you describe. You can loose money if bitcoin-qt's wallet is not regularly backed up. Deterministic key generation schemes that overcome this problem were designed long ago and exist in many alternate implementations e.g. Armory, Electrum .... and BOP. i guess even regular backup wouldn't help in this case: even if you make an offsite backup right before running the client, if your harddrive fails while the client is running and you already made 200 transactions during that session. are you out of luck? (doesnt have to be hdd failure, any problem which would cause the wallet.dat to be destroyed or corrupted, like power outage, etc). Title: Re: backup vs keypool Post by: justusranvier on December 29, 2013, 10:51:57 AM is this a design flaw or do i just misunderstand something? It's a design flaw, but actually represents an improvement over the original design.IIRC, the first version of the client didn't even have a keypool at all. Title: Re: backup vs keypool Post by: gmaxwell on December 29, 2013, 01:03:53 PM There are tradeoffs here.
All of these deterministic wallets are based on schemes I originally promoted in one form or another, so I'm certainly no enemy to them. I mounted an unsuccessful effort in early 2011 to get buy in for changing. A lot of people were— and some still are— suspicious about deterministic schemes. At the same time, key management is an essential part of the safe use of cryptographic systems. Losing control of an old system or an old backup— say, after it ends up on ebay— should not result in you losing funds, and certainly not all your funds. The use of non-deterministic keys automatically has some good key management properties. The problem is that it doesn't mesh well with backups. Swinging 100% the other way to deterministic keys does well with backups, but so far the implementations I've seen have completely neglected key management. They effectively put all the eggs in one basket, which isn't good— especially systems which use the type-2 homomorphism exclusively as compromise of a single key potentially compromises all of them. As with most things in engineering a middle path is probably best.... as also with most things in engineering, the first try is seldom the best. In Bitcoin-Qt you can adjust the keypool size to better match your usage and backup patterns. This is— in my opinion— too much to ask of many users, so I'm not defending it as a good setup, but it's perhaps not quite as bad as you're thinking. In my opinion an ideal system would support multiple independent master keys, precomputing the next one, and doing controlled transactions triggered by backups, according to user configurable retention policies... e.g. An example policy: once a month it adds a new master key which is initially unused, then after at least one month and four backups have passed it switches to using it (so the key has made it into at least four backups), funds which remain on keys which are over a year old are periodically swept onto new keys. I think an ideal system would would also include things like an emergency key compromise panic buttons which would generate new keys, walk you through creating a new set of backups, and then migrate all the funds. Quote from: Grau Deterministic key generation schemes that overcome this problem were designed long ago and exist in many alternate implementations e.g. Armory, Electrum .... and BOP. Yes, invented long ago (https://bitcointalk.org/index.php?topic=19137.0) by the Bitcoin core development team, and eagerly rushed to market by alternative implementations — potentially heedless of security considerations, design fragility, quality of specification, full spectrum of use cases, performance, implications of key management, compatibility, etc.[...] You seem to recognize why there is a market I'm super happy that people went out and built new and interesting things— and I intentionally decided to allow the patentablity window on the homorphic derivation scheme to close, because I concluded that paranoia over the patent would to more harm than the good that could be done by defensive licensing. Your own software implements BIP32 (https://en.bitcoin.it/wiki/BIP_0032) as well, now— and while I cannot say you rushed... but please, if you're going to brag about some feature you have that Bitcoin-QT lacks, please don't pick one that Bitcoin-QT lacks because instead we were spending time building the feature for everyone else— including you— too. Your patches are welcome to Bitcoin-QT as well. Or does the collaboration only flow in one direction? :) Title: Re: backup vs keypool Post by: grau on December 29, 2013, 05:29:34 PM Your own software implements BIP32 (https://en.bitcoin.it/wiki/BIP_0032) as well, now— and while I cannot say you rushed... but please, if you're going to brag about some feature you have that Bitcoin-QT lacks, please don't pick one that Bitcoin-QT lacks because instead we were spending time building the feature for everyone else— including you— too. Your patches are welcome to Bitcoin-QT as well. Or does the collaboration only flow in one direction? :) If I brag, then that I put BIP32 it into commercial use quite a few months ago. Or was that not its purpose? I remember asking sipa, a day before the San Jose conference in May on IRC, not to change the spec again until I finish my product presentation with it :). So yes, I rushed because it is/was much of a need. I do not think I ever intentionally missed to credit this elegant design those who made it. Seven months passed since then and this masterpiece of your design did not made it to bitcoin-qt. I believe that this is not because of resource constraints, since the core team has more and better qualified resources than I do, but because bitcoin-qt is a work of a genius, a proof of concept code and a hairball written in an ancient language in a monolithic style that runs an economy of billions. Sure I can move quicker without that ballast and responsibility. This is why people create start-ups, what BOP is, because working from a clean sheet with new technologies and without inherited liability is just more productive and fun. It is not like I would not contribute to the success of the project Bitcoin. I open sourced my work more than a year ago and keep extending it: https://github.com/bitsofproof/supernode Maybe I will come up with something truly amazing comparable to BIP32, I wish for, but how could promise that. I am a quant developer and investment banker, not a cryptographer, not even a mathematician. Did I not stop using C++ more than 10 years ago (for good reasons), I would likely add more to the bitcoin-qt code. As I open sourced the first version of BOP I was asked: I wonder how the author plans to keep up with all the development done in bitcoind though... they have many competent people improving the code and adding functionality. Just to keep up would be an enormous effort. But anyway, writing such a thing was certainly an enormous effort already! Now, BOP has products that go in usability for business beyond that of bitcoin-qt and I expect the gap to widen further. It would be great if the core team would stop bloating that monolithic code base as if it would be able to address the problems of everyone with one solution. We rather need refactoring that isolates consensus protocol, creates a trusted communication channel inside the company (instead of that badly designed RPC) and removes the wallet from bitcoind. End user use mobile wallets and companies need a border router with extensible interfaces and a frozen consensus engine and lots of custom functions that create business value. If there was a push for refactoring in above sense and if there were funds to work on it, then I am on board. In the meanwhile allow me to be proud of my hard work. I do not need to step on bitcoin-qt to stand out :P Title: Re: backup vs keypool Post by: grau on December 29, 2013, 06:00:44 PM @gmaxwell Your own software implements BIP32 (https://en.bitcoin.it/wiki/BIP_0032) as well, now— and while I cannot say you rushed... but please, if you're going to brag about some feature you have that Bitcoin-QT lacks, please don't pick one that Bitcoin-QT lacks because instead we were spending time building the feature for everyone else— including you— too. Your patches are welcome to Bitcoin-QT as well. Or does the collaboration only flow in one direction? :) Just for the records: I first implemented BIP32 on 27. February 2013 and posted it here: https://bitcointalk.org/index.php?topic=147452.0 The code since then moved to: https://github.com/bitsofproof/supernode/blob/master/api/src/main/java/com/bitsofproof/supernode/common/ExtendedKey.java I kept it updated with every change of the spec and passed all test vectors as soon as they were published. Still it was not considered a reference implementation, not even worth of a review. This is how that collaboration you ask me looks from my angle: https://bitcointalk.org/index.php?topic=19137.msg2551084#msg2551084 Title: Re: backup vs keypool Post by: TierNolan on December 30, 2013, 01:35:20 AM In Bitcoin-Qt you can adjust the keypool size to better match your usage and backup patterns. This is— in my opinion— too much to ask of many users, so I'm not defending it as a good setup, but it's perhaps not quite as bad as you're thinking. A simple change would be that the client wouldn't silently extend the key pool. If you try to send money and the keypool is exhausted, the client informs you that the transaction cannot be processed. The user would have to manually extend they keypool along with a warning to make a new backup immediately. Deterministic wallets which don't support public generation would protect against a fundamental break of elliptic curve maths. That system doesn't seem to be in use anywhere. Title: Re: backup vs keypool Post by: kjj on December 30, 2013, 05:44:28 AM In Bitcoin-Qt you can adjust the keypool size to better match your usage and backup patterns. This is— in my opinion— too much to ask of many users, so I'm not defending it as a good setup, but it's perhaps not quite as bad as you're thinking. A simple change would be that the client wouldn't silently extend the key pool. If you try to send money and the keypool is exhausted, the client informs you that the transaction cannot be processed. The user would have to manually extend they keypool along with a warning to make a new backup immediately. Deterministic wallets which don't support public generation would protect against a fundamental break of elliptic curve maths. That system doesn't seem to be in use anywhere. I much prefer linking the keypool extension to the backup process. As in, the backupwallet RPC command extends the keypool, then does the backup, and there would be no other way to generate new keys. But it probably won't happen because we are moving away from unrelated keys. (FYI, I'm one of those people that dislikes HD keys, but I fully support changing to them because they are safer for 99% of the people to use.) Title: Re: backup vs keypool Post by: grau on December 30, 2013, 06:43:00 AM I much prefer linking the keypool extension to the backup process. As in, the backupwallet RPC command extends the keypool, then does the backup, and there would be no other way to generate new keys. this. I keep wondering why an obvious source of loss for the majority of user is being ignored for years. This is like the past opposition to wallet encryption or the current to https://bitcointalk.org/index.php?topic=181734.0 Title: Re: backup vs keypool Post by: kjj on December 30, 2013, 07:00:27 AM This is like the past opposition to wallet encryption or the current to https://bitcointalk.org/index.php?topic=181734.0 Meh. I think I found the source of opposition to wallet encryption: Armory paper backups are explicitly unencrypted. The vast majority of people using the paper backups is because they forgot their wallet password. I'm just using this as an illustration because I read it recently. There are also countless threads around from people using the satoshi client asking how to recover their encrypted wallets with forgotten keys. The reason we aren't spending much time on fixing the problems with the current wallet is that we don't intend to keep our current wallet for very much longer. Title: Re: backup vs keypool Post by: grau on December 30, 2013, 08:23:32 AM The reason we aren't spending much time on fixing the problems with the current wallet is that we don't intend to keep our current wallet for very much longer. Implementing your suggested workaround would not take much time. I think the problem is that development is not prioritized following end-user or business needs. Decisions are rather technocratic or a the team core member simply commits time on the solution of a problem of his choice. The Bitcoin Foundation could have been coordinating this but failed miserably (also) in this respect. It is really time to come up with some governance of the project Bitcoin, including but not limited to the narrow scope of twiddling the code of bitcoin-qt. Take this as my new year's wish. Title: Re: backup vs keypool Post by: Mysticsam_3579 on December 31, 2013, 12:42:33 AM Cant a solution be that:
1. You think how many transactions you will make in your lifetime. 2. You start the Bitcoin-qt client with "-keypool=(the number of transactions you think you will make)" 3. Take a cup of coffee. 4. Make a backup of your wallet file. Or 1. Change the standard size of keypool from 100 to 100.000. 2. Make a backup of your wallet file. Title: Re: backup vs keypool Post by: grv on December 31, 2013, 06:54:19 PM In Bitcoin-Qt you can adjust the keypool size to better match your usage and backup patterns. This is— in my opinion— too much to ask of many users, so I'm not defending it as a good setup, but it's perhaps not quite as bad as you're thinking. A simple change would be that the client wouldn't silently extend the key pool. If you try to send money and the keypool is exhausted, the client informs you that the transaction cannot be processed. The user would have to manually extend they keypool along with a warning to make a new backup immediately. i have to agree, at the very least users should be informed/warned about a hidden client action which has potentially disastrous outcome. if i hadn't been messing around with possibilities and outliers i wouldn't even know about this flaw. users should be informed about actions like these, i would also inform users that address has been changed to a new one, after sending partial amount. maybe offer possibility to transfer it back to previous address with explanation of what it means to security/anonymity. |