Not necessary, I have no problem converting hex to binary on-the-fly.  I just gotta dig into pywallet code (eventually) and figure out how to get what I want out of it.

This tool doesn't have to backup to the HDD, it's just that there's no other place to put it, at the moment.  I wanted this for my own use, to have a human-readable list of keys -- for backup or for easy input into other scripts/programs I want to write to do things with the keys without having to understand the wallet.dat format.   

As I expected, I'm not the first person to extract keys from a wallet, but I did want to recreate the wallet without the transaction history.  Sounds like pywallet will get me there.

So, I have created this script which pulls out every public and private key in your wallet and stores them into flat files.  The "keylistpub.txt" will contain every public key in the wallet file whether it's yours or not, and "keylistpair.txt" contains a list of just the keys for which there is an associated private key in the wallet, and includes both.  This script should avoid duplicate keys in the output files.

http://dl.dropbox.com/u/1139081/extractKeys.tar.gz

Keep in mind, this utility is going to create a new file on your computer with your private keys.   Handle with care! 

Now, I want to create something that converts the flat file back into a wallet.  This will allow someone to extract keys from multiple wallets, combine them into a single file, and the create a merged wallet.  The output file format looks like this:

(don't get too excited, I have mangled the private keys)

-- The first string is the address, which should be fairly obvious. 
-- The PubKey is two 32-byte numbers (x,y), which correspond to a point on the secp256k1 elliptic curve (the ECDSA curve used by the bitcoin network)
-- The PrivKey is literally a random 32-byte number, which gives the public key when you multiply the generator point by this number.  Yes, a private key is just a random number.  As such, there is no way to identify whether a string of digits is a private key, without having a public key to compare to.  Or rather, every 256-bit number is a private key, so a "private key" is only meaningful in the context of a public key point (x,y).   (all hex numbers are encoded in BigEndian)


Anyone want to help converting pub/priv keypairs into a wallet file?  I believe it can be done with the bsddb package in Python, but I haven't gotten it to work, myself, yet.

So I've decided I was partially wrong, and also had a change of heart on this topic. 

(1) I don't I have the drive to work on a full-client implementation.  I am still not too familiar with the networking part, but some of the nuances of the algorithms are biting me after I thought I had finished them (I gotta make some more elaborate unit-tests...).  I can't imagine getting 100% of the client implemented perfectly, at least not without a big team.
(2) I am starting to question how "safe" it is for others to implement this... ever.   As Gavin said, what does it take to avoid a new client forking the blockchain?  Well, maybe we can't.

But, we can side-step both of these problems with a bit of extra work from the community.  I believe that the best thing to do is to separate out the minimal amount of code from the reference client to create a "node" or "engine", and isolate that from the GUI.  Then, spend the effort to make this reference BTC engine accessible in every language out there, so that the official implementation can be used as a  drop-in backend for any budding project.  It could be accessible as a background process via localhost sockets, or shared-library/dll which can be integrated into your target language (such as wrapping it with SWIG to get it into python). 

This solution appeases everyone's concerns, here.  The community can decide on an "interface" or set of interfaces that can be implemented for "correct" access to the official BTC engine, and then developers can work on getting this interface implemented in their language and document it.   New developers won't have an excuse not to use it, if it's easily accessible in their favorite language (unless they really want to change the underlying mechanics).  Then, this guarantees that there is a single, official engine that follows the community's standards, and most people will use it.

-Eto

I'm only searching the wallet.dat file.  I wanted to get my keys into a flat file for fun, maybe come up with a way to convert between the wallet.dat and wallet.flat.txt, which would make manual key management and recovery easier.

I figured out that the other keys, I believe, are addresses to which I've sent coins before... the ones that show up under the "Sent" addresses tab in the client. 

However, it does seem that there are multiple instances of your public keys in the file.  One of my public keys (for which I have the private key) shows up 10+ times.  Others show up two or three times.  I guess the wallet file holds transaction information, in addition to the keys themselves.

Is it expected that the private key would come way after the public key?  I'm doing a brute force search of my wallet file, using my crypto library to check whether each sequence of 64 consecutive bytes is on the secp256k1 elliptic curve (meaning it's a public key), then searching for a 32-byte sequence that gives you this public key when you interpret it as the private key.  What I am finding is that (1) this is really slow, (2) not every public key has an associated private key in the wallet file and (3) the private keys I'm finding are located about 200-300KB past where the public key was found.

This works under the assumption that both keys are encoded in big-endian and the public keys are encoded as [0x04 [X] [Y]] and private keys are encoded as [0x0420 [secretInt]].  But it doesn't work well enough to be confident that I'm extracting everything--why am I finding so many public keys that don't have matching private keys?   Could keys be stored in different endiannesses?  are some of them not preceded by '0x0420'?

I'd like to see the keys stored in a flat file that makes key recovery a million times easier.  The keypairs can be encoded as constant-length binary strings, just like the headers are serialized.  The private keys can be encrypted individually in this flat file, without encrypting the entire file.  Instead, you just convert the 256-bit private-keys to a 256-bit encrypted-private-key before writing it to file.  Then your wallet file still works for tracking transactions (since the public keys/addresses are still in plaintext), but the private key will be useless without transforming it back to the unencrypted bitstring.

Five or six posts up I included the .svg file that contains all the images so far.  It's kind of a mess, but it's all there. 

And please let me know if you find any problems with it, I'll update it myself. 
-Eto

How about if I just change the title to say "Bitcoin Main-Network Address Operations"?  I don't feel like all the extra detail is justified, it'll just muck up the diagram to accomodate cases that most people will never encounter.

P.S. - I just updated the image to include one \x00 byte, and changed the title as I just described.  Let me know what you think.

Ah hah.  Little details that I missed...

Okay.  So I need to add a '\x00' byte to the 24-byte binary address to make it 25-byte... and it never mattered because it's big endian so the integer calculation works out the same...?  So when would this byte not be '\x00'?  And is this the same \x00 that you say is responsible for the '1' at the beginning of the address?  There could be multiple '\x00' bytes? 

Damn these small details!   I'll update the diagrams once I understand it, myself.

-Eto

First of all, here is a link to the raw .svg file I am using:  

https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/BTC_Illustrations.svg

This is typically how I prepare images for presentations I make at work.  I use Inkscape to create a bunch of diagrams, then select each one and "File-->Export Bitmap".  With this file out there, it should be available for others to download, modify and re-post somewhere else if I get hit by a bus tomorrow.   By the way, Inkscape is amazing.  Once you learn some of the hotkeys, alignments, grouping, fill&stroke, etc, it's very pleasant and efficient to use.  The learning curve isn't terribly steep, and they have interactive tutorials which are actually Inkscape canvases, with inkscape objects for you to play with.

Second, here's another diagram.  It's not nearly as complicated as OP_CHECKSIG, but once I had it mapped out like this, I was able to do a ton of address magic in my code and get it right on the first try.  Not having to guess endianness at every step is a big help!  Remembering the extra bytes that have to be added here and there helps, too.



The full size image is here:  https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/PubKeyToAddrSmall.png

Gavin, I am very interested to help out with this.  I think unit-tests are great, but I wasn't sure how to do it with networking.  It sounds like you already have some ideas.  I have started a test-suite in python (for python) for what you mentioned above, that involves endianness checks, serialization/unserialization of different object types, example transactions, blocks and addresses to verify consistency, and a ECDSA signature verification test.  I'd like to add some scripting-engine tests... my understanding is that even if non-standard scripts aren't generally accepted, the client has to be able to evaluate them in case they show up in the blockchain.

As for people not following the standardization documents:  I don't think it really matters.  Some people are going to want to do their own thing, and not let "standards documents" get in their way.  But at least when they try to release their client and there is a cold reception for it... we have somewhere to point them if they want to get serious.  There won't be any "Why does my client have to do this?" or "I didn't know I needed to accommodate [some complicated thing requiring a lot of code rework]".  And hopefully a little less of the community saying "well, just look at the C++ code, it's all in there."   I think that C++ code is a terrible way to document something that really needs to be standardized...

Additionally, I think Mark's concerns for me are more easily communicated: "you do realize that you have to do all this stuff, right?"  Instead of him just telling me there's a lot to do and I have no way to gauge before jumping in.

@Mike:  I guess we'll have to respectfully disagree, and not because of reasonable disagreement, but my naivety on the matter won't be solved through debate/discussion.  I have to try it and find out.  The worst thing that can happen is I will become an "expert" on the topic and know better where to put my effort in the future (which has already started, now that I've got most of the scripting and ECDSA code implemented).  My effort isn't well-spent on the C++ code, because there's too many uncertainties hitting my brain at once, I don't understand anything more or do anything productive.  When I understand the algorithms, I'll consider battling the C++. 

Btw, I have been working 20-25 hours per week on this, for less than 2 weeks.  At that rate, 2 more months I think is reasonable.   Even if you are right and it doesn't work, I find it fascinating, educational and rewarding to learn the finer details of the system, and I don't think there's a more efficient way to get a deep understanding of it. 

To get back on topic: I will look into the wiki page linked before, and see how I can contribute to it.  I think, regardless of our discussion on the virtues of alternative clients, the information should be organized and available to those who want to try it.

By the way,  I've learned from working with other people's algorithm designs at work, the only way to actually, really understand the complexities of a complicated algorithm, is to try implementing it yourself.   Without doing it, you just can't realize the subtleties/pitfalls/nuances of the system without actually battling them yourself.  I would argue that having people dive right into an existing codebase (especially a deep C++ project) is dangerous for a security-sensitive piece of software.

Prime example:  if I hadn't gone through the effort to actually implement scripting on my own, I never would've realized the complexities of OP_CHECKSIG and created this:  http://forum.bitcoin.org/index.php?topic=29416.new;topicseen#new. 

The point here is that you complain about wasted effort, re-doing work.  But I argue that not only will new clients good for BTC as a whole (if done right), it breeds expert developers who can further the technology... which is exactly what I am trying to do on these forums!    Perhaps one day I will take my expertise and contribute it to the Satoshi client.  But battling algorithm details with complex C++ jargon simultaneously is a recipe for breaking things.

Perhaps my request is better made by starting a wiki page for what I want... seed it with the fields that I think should be there, even if they're wrong, and let the community hit equilibrium on what a "full node" means.  Any recommendations for where to start this?  Perhaps the page linked earlier would be a perfect starting point.

-Eto

You forget the part where it's in very archaic C++.  I work in C++ for a living, yet I still have a very tough time following it, much less developing for it, correctly.  My goal is to produce a python implementation, because python code can be extremely simple, functional, and readable all at the same time, with one-tenth the lines of code (part of the reason it's readable).  It's lack of speed is the only downside, which is not really a concern for the BTC network.  Binary I/O, serialization/unserialization is as simple as it gets, networking/sockets are much more pleasant, exception handling is a breeze, and the ability to pull in external libraries quickly and painlessly are all fantastic features for exploring design ideas.

I'd much rather spend the time upfront developing such a client, instead of spending 80% of my time in the C++ worrying about syntax, templates, polymorphism/inheritance, casting issues, linker errors, etc.

-Eto

 

Some examples:
   --I want to be able to rewrite wallets to be more-straightforward and easily-recoverable than the current BerkleyDB format.  I've battled corrupted wallets quite frequently and I'm disappointed I can't reconstruct them easily (and that the DB is easily corruptable).  The database of keys is really add-only, so I don't see why we need a complex DB engine to track them.
   --I want to redesign the file formats to better accommodate wallet recovery, memory pool organization, transaction filtering, reduced memory footprint, etc
   --I want to have custom account types that support various CONOPs, such as tracking, locking, uploading, recovering android/iPhone accounts, and sync'ing between computers.
   --I want to be able to separate out private keys from public/keys/addresses to allow tracking of balances while being able to keep the private keys safe on external media
   --I want to be able to sign certificates containing desired transactions from an offline computer, and transfer the certificate to my client online and have it broadcast (if you have $10,000 worth of BTC, you would probably be willing to buy a $200 Eee PC just to engage in this type of security measures)

I think there's a lot of good reasons someone would want to change the under-the-hood stuff, instead of just re-wrapping the reference client with a new GUI.

-Eto

I don't think I'm underestimating at all -- I know it's a massive amount of work.  that's why I think we need a community-maintained "standards" document (wiki) that will allow developers to see what it will take to get there.  In return, such clients can conform to the "rules of the land" and make sure that the network stays "pleasant."

I don't think Bitcoin has much of a future without the ability for devoted developers to develop their own, full clients.  I think having only a single "real" client will stifle the development of the Bitcoin in general.  The ability for individuals, groups and companies to develop their own implementations, tailored for a variety of different use-cases is going to help BTC succeed.  Some people want to just spend and receive coins without even seeing address strings or wallet details.  Others will be power users, and want more fine-grained control over their wallet(s), keys, encryptions, and super-security techniques.  And there's a million shades of gray in between.  I don't think all this can be achieve with one client.
 
I have the intent to do one myself, even if it takes me 2 months to get the details right.  And I think we should promote it.  And I think the best way to promote it is to have a thoroughly-hashed-out description of these details, with as many unit-tests as possible for developers to apply to their code.  The alternative is for developers to create clients but not know they weren't supposed to forward transactions of unknown validity, then open up the door for people to start flooding the network.

-Eto

Hi all, I'm mostly active in the parent discussion group, but I am working on an alternative client in python, so I should be spending more time here.  In recent days, I have realized that alternative clients could seriously disrupt BTC network unless their is very high standards for what it does.   What I'm concerned about is someone making a really great GUI (like I plan to), that has a subtle vulnerabilities that the base group of developers have tried so hard to avoid (which I don't plan to do).  The last thing we want is to have 30% of users switch to a client that has a scripting bug in it that evaluates certain scripts to true that should be false, then the block chain starts getting messy with hundreds of invalid blocks and "reversed" transactions. 

So, of course, I would ask if there is an existing description like what I ask for.  If not, or if it's not complete, I would like to start one here.  I want to create a living, master checklist for developers to use, that they need to conform to before declaring their client "correct."  This would include flood defense techniques, community-agreed transaction fee schedules, scripting unit-tests, network broadcast/forwarding rules, etc.  I recognize a lot of this is in the C++ source code, but that's not the best way to document the standard...

I will start here with a short a short list of questions, and I'd like to get feedback concerning what is needed for a new client to be considered safe & "correct."


-- Node connection protocols:  node-discovery techniques, node-cycling, suggested IP-rules, backwards-compatibility handling
-- Scripting: 
    - Would like to see a list of OP_KEYWORDs to be supported by the scripting engine (are they all the non-"Currently Disabled" words on the scirpting wiki?)
    - Would like to see/create a set of reasonably-complex unit-test scripts that can be used to check the scripting engine, including most required keywords
         - Tests would look like (scriptBinaryTest1, True), (scriptBinaryTest2, True), (scriptBinaryTest3, False), etc,
-- Reduced-memory nodes
    - If a node is not intended to hold the entire blockchain, how should its network-protocols be adjusted?
    - What classes of nodes can we define, and what are the associated capabilities:
       -- A node that only holds it's own transaction list should only connect for getting information, not forwarding transactions or blocks (or maybe forwarding blocks, but not transactions?)
       -- A reduced-footprint node should identify itself in some way on the network -- and other nodes would be aware and pick it's peers diversely... how?
-- Flood-disconnect conditions: 
    -What conditions should be applied before disconnecting a node (besides the bug currently disconnecting those downloading the block chain)
    -What conditions determine whether a broadcast transaction is forwarded to other nodes/peers?  Blocks forwarded?
         - Does a node have to be able to check validity of all blocks/transactions to forward them?
         - If a node is a reduced-memory-footprint node (not holding the entire blockchain), should it forward transaction even if it can't verify they are valid?
-- Transaction Fees:
    - What is the current acceptable fee schedule?
    - How much flexibility will we allow for developers to implement their own fee schedule?
    - If a transaction fee is not included where we think it should be, should it be forwarded to other nodes?
-- Other:
    -What are the timeout rules for peer nodes?
    -What are the timestamping rules for creating new blocks (I know there's a median calculation based on a bunch of peers)
    -Would like to see a list of checks required before declaring a transaction valid.  A developer could easily remember to check the merkle tree root, but not that sum(TxIn)==sum(TxOut)


I know I'm forgetting a ton of stuff, but that's why this is a discussion and not a monologue

-Eto

That's high, but it's not ludicrous given the amount of hashing that has been done in the last year.  At the current global hashrate, it seems it would take 150 days (on average) to find another hash just as good.  The blockchain has been in generation for over a year, but with much lower rates for most of that time.

Well okay, let's be more precise:  I summed up all the difficulties up to block 136,496.  The total value is 10,939,043,020.8.  Take the log-base-2 and add 32 for the difficulty=1 size:  you get 65.35.  So the network has executed on somewhere around 2^65.35 hashes to produce the 136,496 blocks with their associated difficulties.  This isn't terribly far from the block in question, requiring on average 2^67.07 hashes.   It is only 3.3 times higher (1.72 bits) higher than the total difficulty sum.  

This is well within the scope of the exponential distribution of block generation.  This is like having enough computational power on the network for the difficulty to be at 35 billion, and then the network solving a block at that difficulty in 3 minutes.  Three-minute blocks happen all the time.

Since I finally figured out how to read the block chain, I decided it would be fun to find the lowest hash produced, yet.  The hash for a block doesn't have to be AT that difficulty, it just has to beat it, and I figured there's gotta be some blocks with major overkill in terms target hash, just by luck.  Well here it is, block 125,552:

http://blockexplorer.com/block/00000000000000001e8d6829a8a21adc5d38d0a473b144b6765798e61f98bd1d

If I did the difficulty calculation correctly (no guarantees), I believe this block would've been valid even at difficulty 35,987,768,035  (current difficulty is 1,564,057).  Can someone verify that?

EDIT: Okay I didn't realize this thread was started a year ago... I thought Gavin was proposing a NEW test network, a third one on top of the production and test networks already there.  Please ignore this message.

As someone interested in developing a new client soon, can you tell me what is the difference between this new network, and the previous test-network?  I mean, I thought that's what the test network was for... right?

In other words, what are the use cases for the two networks?
-Eto