[PULL] private key and wallet export/import

[netrin]

160 000 blocks with 100 addresses of 33 byte each is half a gigabyte. Surely 'not indexing' would have been an optimization unnecessary for beta software?

... or was it believed that a malicious node could flood everyone's hard drive with minimal cost? ...or?

[casascius]

160 000 blocks with 100 addresses of 33 byte each is half a gigabyte. Surely 'not indexing' would have been an optimization unnecessary for beta software?

... or was it believed that a malicious node could flood everyone's hard drive with minimal cost? ...or?

I dunno, but I have heard developer resistance to this index, I have been promoting it for a while.  In the past, developers seemed to have the sentiment that it served no purpose other than private key imports, which at the time, they considered an obscure thing only advanced users would do (part of what motivated me to pump out physical bitcoins).

So, as you can imagine, I'm pleased to see a bandwagon chanting that we need the index.  It really isn't that complex, it just needs to be done.

Such an index need not contain all 33 bytes of a key - if it indexed only 4 bytes, it would use a fraction of the space, at the expense that collisions would require extra blocks to be needlessly checked on occasion, so for example we might read 8 blocks when we only really needed to read 2.  A negligible penalty, this would be more than acceptable, because it would still eliminate having to check well over 99% of the block chain that does not contain addresses with that prefix whenever someone imports or sweeps a private key or rescans their wallet.

In fact, I have believed that there should be a facility to reindex the block chain.  Building this second index could be part of that.  Right now, there is:

- blk0001.dat - the block chain itself (not a database file, but just a blob of binary data)
- blkindex.dat - the index we have now (keyed on transaction ID's) - which COULD be rebuilt entirely from blk0001.dat with a -reindex switch, except that there presently exists no implementation to do this.

Adding this index would just create a new .dat file that would point to blk0001.dat, which could be generated from scratch with a hypothetical -reindex switch.

[Red Emerald]

Should talk about a new pull request for a firstbits index be in a new thread?  It seems like there is enough support for it here.

[netrin]

Firstbits are incredibly useful, but I think it should see more side adoption. There have already been a few minor tweaks to the algorithm (like collision handling within the same block).

As for the index, 4 bytes, 5, whatever, it should be possible to calculate the optimal radix-prefix periodically. I just can't imagine the index is a significant portion of the blockchain (before merkle tree pruning). But it's absence has philosophical consequences.

I think private key import is cool, but my consistent drum beating (from the side-lines) is for simplification, transparency, and modularization. I believe bitcoin could have been a lot further along if the core did not attempt to protect users, the wallet was plain text, the local blockchain and wallet were completely independent, and generally more hacker-friendly ... is the opaque nature of the bitcoin client a matter of different philosophies (monolithic GUI vs. Unix utilities), or the evolutionary consequence of man-decades of work and discussion over performance, scalability, and usability?

[casascius]

I think private key import is cool, but my consistent drum beating (from the side-lines) is for simplification, transparency, and modularization. I believe bitcoin could have been a lot further along if the core did not attempt to protect users, the wallet was plain text, the local blockchain and wallet were completely independent, and generally more hacker-friendly ... is the opaque nature of the bitcoin client a matter of different philosophies (monolithic GUI vs. Unix utilities), or the evolutionary consequence of man-decades of work and discussion over performance, scalability, and usability?

I agree with several of the points you have made here:

I do think the wallet encryption is a waste of effort.  But a side benefit is that the design of bitcoind has been pushed toward supporting addresses it thinks are "mine" without needing to know the actual private key.  That has some good use cases.

I do think the local blockchain and wallet should be completely independent.  However, this index, if added, would put bitcoind within reach of offering features where someone can implement their own wallet on top of bitcoind.  The built-in wallet, in such a case, could be completely ignored.  If this index gets built, bitcoind could easily support a "give me all unspent tx's for these addresses" RPC command, which would make a third-party wallet app on top of bitcoind much more feasible.  Another RPC command to dump a transaction to the P2P network, and a way to get notifications/callbacks of incoming transactions/blocks would seal the deal.

tl;dr: many of the built-in features are far from the best implementation, but given this index and a few other trivial RPC commands, those built-in features can be ignored, replaced, and left for deprecation.

[netrin]

Independence of concerns would go a long way to accelerate development and experimentation. Personally, I'd like to host the bitcoind south of the polar circle, pay someone for the service, or more trivially have a single standard bitcoind on a shared host (/var). A lot of the web clients and mobile apps seem to be recreating a wheel that shouldn't have needed to be recreated. If the blockchain and index is too big, slow, or bandwidth too expensive, then we should outsource the responsibility.

[btc_artist]

Independence of concerns would go a long way to accelerate development and experimentation. Personally, I'd like to host the bitcoind south of the polar circle, pay someone for the service, or more trivially have a single standard bitcoind on a shared host (/var). A lot of the web clients and mobile apps seem to be recreating a wheel that shouldn't have needed to be recreated. If the blockchain and index is too big, slow, or bandwidth too expensive, then we should outsource the responsibility.

This.  The daemon should have nothing to do with wallets.  It should relay transactions, download new blocks, and maintain a local copy of the block chain along with needed indexes.

[SgtSpike]

Firstbits are incredibly useful, but I think it should see more side adoption. There have already been a few minor tweaks to the algorithm (like collision handling within the same block).

As for the index, 4 bytes, 5, whatever, it should be possible to calculate the optimal radix-prefix periodically. I just can't imagine the index is a significant portion of the blockchain (before merkle tree pruning). But it's absence has philosophical consequences.

I think private key import is cool, but my consistent drum beating (from the side-lines) is for simplification, transparency, and modularization. I believe bitcoin could have been a lot further along if the core did not attempt to protect users, the wallet was plain text, the local blockchain and wallet were completely independent, and generally more hacker-friendly ... is the opaque nature of the bitcoin client a matter of different philosophies (monolithic GUI vs. Unix utilities), or the evolutionary consequence of man-decades of work and discussion over performance, scalability, and usability?

I agree regarding Firstbits.  I think it'd be great to see it in a 3rd-party client, but I don't think it should be placed in the standard client yet.  Although, just FYI, the algorithm has stayed exactly the same since Firstbits was released.  It's just, no one except myself knew how it worked with collisions in the same block when that question arose.

[netrin]

I do think the wallet encryption is a waste of effort.  But a side benefit is that the design of bitcoind has been pushed toward supporting addresses it thinks are "mine" without needing to know the actual private key.  That has some good use cases.

That notion of 'mine' can and should be part of the wallet (or in that tier) not twisted up in the public global data model.

I do think the local blockchain and wallet should be completely independent.  However, this index, if added, would put bitcoind within reach of offering features where someone can implement their own wallet on top of bitcoind.  The built-in wallet, in such a case, could be completely ignored.  If this index gets built, bitcoind could easily support a "give me all unspent tx's for these addresses" RPC command, which would make a third-party wallet app on top of bitcoind much more feasible.  Another RPC command to dump a transaction to the P2P network, and a way to get notifications/callbacks of incoming transactions/blocks would seal the deal.

tl;dr: many of the built-in features are far from the best implementation, but given this index and a few other trivial RPC commands, those built-in features can be ignored, replaced, and left for deprecation.

Is there a thread discussing modularization of the Satoshi client? I think a lot of people have simply focused their attention away to alternatives: libbitcoin, node.js bitcoinjs, and various web clients.

I agree regarding Firstbits.  I think it'd be great to see it in a 3rd-party client, but I don't think it should be placed in the standard client yet.  Although, just FYI, the algorithm has stayed exactly the same since Firstbits was released.  It's just, no one except myself knew how it worked with collisions in the same block when that question arose.

I'd like to see a discussion started though. I think awareness of firstbits needs to increase. I introduced a few developers who seemed surprised but immediately interested. As for collisions in the block, I thought they were first ordered within the block and then later assumed to be simultaneous within the block. There was a slight word-change to the algorithm, at least enough that someone had to make a change (maybe puik or genjix?).

[SgtSpike]

I'd like to see a discussion started though. I think awareness of firstbits needs to increase. I introduced a few developers who seemed surprised but immediately interested. As for collisions in the block, I thought they were first ordered within the block and then later assumed to be simultaneous within the block. There was a slight word-change to the algorithm, at least enough that someone had to make a change (maybe puik or genjix?).

They've always been ordered in the same order as they appear within a block's raw data.  So if one address appears before the other address within the same block's raw data (and they're both brand new addresses with the same firstbits), then that first address will get the shorter firstbits, and the second address will have to add at least one character to get its unique firstbits.

What are puik and genjix working on related to firstbits?  If they are using a different type of assumption for collisions within a block, their results won't match up with those at firstbits.com.

[DeathAndTaxes]

Independence of concerns would go a long way to accelerate development and experimentation. Personally, I'd like to host the bitcoind south of the polar circle, pay someone for the service, or more trivially have a single standard bitcoind on a shared host (/var). A lot of the web clients and mobile apps seem to be recreating a wheel that shouldn't have needed to be recreated. If the blockchain and index is too big, slow, or bandwidth too expensive, then we should outsource the responsibility.

This.  The daemon should have nothing to do with wallets.  It should relay transactions, download new blocks, and maintain a local copy of the block chain along with needed indexes.

+1

I have always thought this also.  It seems to me the daemon is too tightly coupled with the Satoshi wallet and that requires either significant code surgery, sticking with limits of the existing wallet, or a lot of wheel inventing.

What if the daemon would removed into a new project.  Wallets could communicate the w/ daemon via encrypted JSON or socket messages. (i.e. check balance of this address, submit this transaction, what is current block count).

It would also abstract the differences between light clients, and full clients.  A light client is communicating w/ a daemon running on another computer (public, private, whatever).  A full client is communicating w/ a copy of the daemon running locally.

As an example on how this tight coupling generates a significant barrier for new projects:

I did some proof of concept work on a smartcard to use it as a secure private keys storage device. Due to the limits of smartcard resources the card would only contain the private key generation code and signing aspects.  This would require a custom wallet format which contains only the address and an identifier to reconstruct the private key in the smartcard.  I got the card portion working (albeit slowly in my proof of concept) and talking w/ a non-compliant  public wallet handler.  However the amount of work needed to couple this w/ the Satoshi client, make it compliant w/ protocol was a significant barrier.

[Red Emerald]

Independence of concerns would go a long way to accelerate development and experimentation. Personally, I'd like to host the bitcoind south of the polar circle, pay someone for the service, or more trivially have a single standard bitcoind on a shared host (/var). A lot of the web clients and mobile apps seem to be recreating a wheel that shouldn't have needed to be recreated. If the blockchain and index is too big, slow, or bandwidth too expensive, then we should outsource the responsibility.

This.  The daemon should have nothing to do with wallets.  It should relay transactions, download new blocks, and maintain a local copy of the block chain along with needed indexes.

+1

I have always thought this also.  It seems to me the daemon is too tightly coupled with the Satoshi wallet and that requires either significant code surgery, sticking with limits of the existing wallet, or a lot of wheel inventing.

What if the daemon would removed into a new project.  Wallets could communicate the w/ daemon via encrypted JSON or socket messages. (i.e. check balance of this address, submit this transaction, what is current block count).

It would also abstract the differences between light clients, and full clients.  A light client is communicating w/ a daemon running on another computer (public, private, whatever).  A full client is communicating w/ a copy of the daemon running locally.

I completely agree that the wallet and the nodes need to be separated.  This is pretty much what Electrum is doing.

https://bitcointalk.org/index.php?topic=50936.0

[DeathAndTaxes]

I completely agree that the wallet and the nodes need to be separated.  This is pretty much what Electrum is doing.

https://bitcointalk.org/index.php?topic=50936.0

I never thought about using the "server" portion of Electrum running locally to support the custom wallet.  Hmm.

I like Electrum's approach still there is no real difference between a remote server and a local daemon.  The issue is still duplication of work.  If some fix needs to be made to the daemon then electrum needs to make the same change also. 

[netrin]

They've always been ordered in the same order as they appear within a block's raw data.  So if one address appears before the other address within the same block's raw data (and they're both brand new addresses with the same firstbits), then that first address will get the shorter firstbits, and the second address will have to add at least one character to get its unique firstbits.

What are puik and genjix working on related to firstbits?  If they are using a different type of assumption for collisions within a block, their results won't match up with those at firstbits.com.

Genjix's libbitcoin will contain firstbits. He went on a tangent removing the '1' prefix, which has subtle but incompatible implications. I believe he's returned to your implementation. FreeMoney and Puik had a short conversation about order within blocks. It should be possible to find real-world test cases.

One little possible wrinkle though. We have been ordering two transactions that would otherwise have the same firstbits if they were not in the same block according to their appearance in the block data (first to appear being first). We are investigating switching to giving them both the longer firstbits. An example:

Block 1:
1asd1234567...

Block 2:
1asd2fkkkgrt...
1asd2sqp434...

Currently 1asd2fkkkgrt... would have FB of 1asd2 and 1asd2sqp434... would have FB of 1asd2s.

It seems like it might be better to not need the extra rule of tiebreaking with order of appearance and instead to give each the FB as if the similar address came before. In this case giving 1asd2fkkkgrt...  a FB of 1asd2f. It seems more elegant to use a rule of "String required to differentiate an address from all addresses in the SAME and previous blocks" and have nothing about order in the block.

[SgtSpike]

They've always been ordered in the same order as they appear within a block's raw data.  So if one address appears before the other address within the same block's raw data (and they're both brand new addresses with the same firstbits), then that first address will get the shorter firstbits, and the second address will have to add at least one character to get its unique firstbits.

What are puik and genjix working on related to firstbits?  If they are using a different type of assumption for collisions within a block, their results won't match up with those at firstbits.com.

Genjix's libbitcoin will contain firstbits. He went on a tangent removing the '1' prefix, which has subtle but incompatible implications. I believe he's returned to your implementation. FreeMoney and Puik had a short conversation about order within blocks. It should be possible to find real-world test cases.

One little possible wrinkle though. We have been ordering two transactions that would otherwise have the same firstbits if they were not in the same block according to their appearance in the block data (first to appear being first). We are investigating switching to giving them both the longer firstbits. An example:

Block 1:
1asd1234567...

Block 2:
1asd2fkkkgrt...
1asd2sqp434...

Currently 1asd2fkkkgrt... would have FB of 1asd2 and 1asd2sqp434... would have FB of 1asd2s.

It seems like it might be better to not need the extra rule of tiebreaking with order of appearance and instead to give each the FB as if the similar address came before. In this case giving 1asd2fkkkgrt...  a FB of 1asd2f. It seems more elegant to use a rule of "String required to differentiate an address from all addresses in the SAME and previous blocks" and have nothing about order in the block.

Ah, right.  Yes, what FreeMoney said is correct about the current implementation, and we have not changed it.  It still would result in 1asd2 and 1asd2s.  We talked about changing it to make both addresses longer in such a case, but discarded the idea in favor of maintaining consistency.  We didn't want it to retroactively change any firstbits that could have already been handed out, as that would be dangerous.

[etotheipi]

I'd like to see a discussion started though. I think awareness of firstbits needs to increase. I introduced a few developers who seemed surprised but immediately interested.

I'm one of those developers!  I'll be releasing a client in the near future, and I can pretty easily support Firstbits.  And with the full-RAM version of the client (that holds the entire blockchain in RAM), I can resolve a firstbits address in 0.5 seconds without contacting any server... pretty slick

I went to the firstbits site but didn't see the technical information I needed to be able to implement it.  Any links?

Btw, my client will have a completely decoupled blockchain interface and wallet.  In fact, you don't even need to wait for Armory to be released, because you can already use the full-set of blockchain tools that I have, written in C++.  https://github.com/etotheipi/BitcoinArmory/tree/master/cppForSwig  It's a ton of code, but it can all be boiled down to a small interface of a couple-dozen calls.   It can do a full blockchain scan from cold boot in about 20s (collect all tx for any address directly from disk).  PM me if you have any interest in leveraging it.

[SgtSpike]

I'd like to see a discussion started though. I think awareness of firstbits needs to increase. I introduced a few developers who seemed surprised but immediately interested.

I'm one of those developers!  I'll be releasing a client in the near future, and I can pretty easily support Firstbits.  And with the full-RAM version of the client (that holds the entire blockchain in RAM), I can resolve a firstbits address in 0.5 seconds without contacting any server... pretty slick

I went to the firstbits site but didn't see the technical information I needed to be able to implement it.  Any links?

Btw, my client will have a completely decoupled blockchain interface and wallet.  In fact, you don't even need to wait for Armory to be released, because you can already use the full-set of blockchain tools that I have, written in C++.  https://github.com/etotheipi/BitcoinArmory/tree/master/cppForSwig  It's a ton of code, but it can all be boiled down to a small interface of a couple-dozen calls.   It can do a full blockchain scan from cold boot in about 20s (collect all tx for any address directly from disk).  PM me if you have any interest in leveraging it.

I'll try to remember to PM you what you need.

[netrin]

... resolve a firstbits address in 0.5 seconds without contacting any server... completely decoupled blockchain interface and wallet... small interface of a couple-dozen calls.   It can do a full blockchain scan from cold boot in about 20s (collect all tx for any address directly from disk).

Sounds excellent. Looking forward to playing with it. I'm curious, does Armory support plaintext i/o for command line pipes? Have you and Genjix (libbitcoin) shared C++ code or are you both reinventing wheels?

[etotheipi]

... resolve a firstbits address in 0.5 seconds without contacting any server... completely decoupled blockchain interface and wallet... small interface of a couple-dozen calls.   It can do a full blockchain scan from cold boot in about 20s (collect all tx for any address directly from disk).

Sounds excellent. Looking forward to playing with it. I'm curious, does Armory support plaintext i/o for command line pipes? Have you and Genjix (libbitcoin) shared C++ code or are you both reinventing wheels?

There's a lot of stuff it could do, but not implemented yet.  However, accessing it via Python CLI gets you just about anything you want in the blockchain.  It could easily be ported to support piping from regular CLI, but loading the blockchain every call would be slow.  However, my current focus is on full-fledged client with all its innovative features.  But I don't want to get too derailed on that... (PM me if you want more info about how you might use it). 

The point was that for importing new addresses, it helps to be able to do a full blockchain scan for its balance in reasonable time without contacting a separate service.  If one is going to implement an "import private key" function, you should be able to tell the user, in a reasonable amount of time, what is that balance of that key.  In this case, a separate client that uses none of my code, could still make a dll/system call to my library, which will load the entire blockchain from disk and scan for a full list of TxOuts, within 20s. 

On that note, I would argue I didn't exactly reinvent the wheel.... the Satoshi client is a wheel that spins at 1000 RPM permanently attached to a satoshi axle.  My code is a wheel that spins at 10,000 RPM and can be used with any axle.  I'm not sure it's possible to do (single-threaded) blockchain scanning any faster (after all, I am an expert on C++ data structures).

[netrin]

However, accessing it via Python CLI gets you just about anything you want in the blockchain.

That should be enough for my experiments.

On that note, I would argue I didn't exactly reinvent the wheel.... the Satoshi client is a wheel that spins at 1000 RPM permanently attached to a satoshi axle.  My code is a wheel that spins at 10,000 RPM and can be used with any axle.  I'm not sure it's possible to do (single-threaded) blockchain scanning any faster (after all, I am an expert on C++ data structures).

I did not mean to imply that your modular blockchain wasn't an improvement over Satoshi's client. I was just curious whether you and Genjix collaborated or shared any code. I believe you are both doing some similar things. Each implementation might test the other, or if either approach for a particular component is superior, why not steal like artists?

Genjix' libbitcoin license is also Affero GPL, while FellowTraveler is suggesting a new hybrid Lesser-AGPL, which might be more appropriate.