Wallet Import Format

[TT]

As for ECDSA key generation, the following OpenSSL commands work, which I got from that bitcoin-off-the-grid link casascius gave:

#To generate the key and save it to the file ecKey.pem:
openssl ecparam -genkey -name secp256k1 -out ecKey.pem

#To pull out a 32-byte private key as hex:
openssl ec -text -noout -in ecKey.pem | head -5 | tail -3 | fmt -120 | sed 's/[: ]//g'

#To get ripemd160(sha256(public key)), which is what bitcoin uses for addresses, as hex:
openssl ec -in ecKey.pem -pubout -outform DER | tail -c 65 | openssl dgst -sha256 -binary | openssl dgst -rmd160 -binary | xxd -p -c 80

There's surely a more efficient method for performing these steps, but they seem to do the trick for now. Perhaps
later I'll document usage of the OpenSSL API from within a single process, to avoid the overhead of starting additional processes.

Once you have the 32-private key and the 20-byte hash of the public key, apply the base58Check steps to them.

-TT

[1714007338]

1714007338

[TT]

OK, I have the following raw data:

Private key: 24ed089647b7f330588c491309e527c44cbf5e04444540782d6b88f8c44b3105 length: 64
Public key hash: 211f0c809a1a14f46af53ae59aa32d02aaf72724 length: 40
------------------------
Wallet import format: 5J6YocBZpn5j9hcPWv1wPEGtfXvHP8g2ZPSSTrjgr9PxUhALeYM length: 51
Address: 1428VeCoiJR81vVjdtXe9sb5G15qjYyLwo length: 34

PrevTx - Raw Data
-----------------
0100000001e1877fe168c04e1f91a170c37320d8d6e6dbac94cb1edf56eab2d075f548cb9300000 0008c493046022100b0ac6689455d95fb81f0012f38b9285d44ae75f64b4c82ea9d1e96c2541392 7c022100bcf31e15dde4d83b567f848cf6b4a708a23f0a71a206d858bfaea0285fca350f014104c 6420d1b499b277a1f4e284cb4bc4cc327539adfc24bc6fd212577af5665395886660c9777484448 745868e8e5c5159d34c929706941e941f9de2fa6a18817f4ffffffff023000c901000000001976a 9145d9536d605d7ddf4f51f57006d1dddc38bb3f79c88ac80969800000000001976a914211f0c80 9a1a14f46af53ae59aa32d02aaf7272488ac00000000

PrevTx - Human Readable
-----------------------
    Hash: e52fa8a693eaff60f644b5006798215e0798e54a2dfc82aaec045c99cf70e4fb
    Data format version: 1
      Input 0 - 1LL8GeU5AxAhG7NuopgSrfeKCnzz46AaGM
          Previous out: 93cb48f575d0b2ea56df1ecb94acdbe6d6d82073c370a1911f4ec068e17f87e1#0
          scriptSig: 493046022100b0ac6689455d95fb81f0012f38b9285d44ae75f64b4c82ea9d1e96c25413927c022 100bcf31e15dde4d83b567f848cf6b4a708a23f0a71a206d858bfaea0285fca350f014104c6420d 1b499b277a1f4e284cb4bc4cc327539adfc24bc6fd212577af5665395886660c977748444874586 8e8e5c5159d34c929706941e941f9de2fa6a18817f4
          sequence: 0xffffffff
      Output 0 - 19XpbRe7XRT2c9FKGP6jTwcbjVwyyGBKiS
          Value: 0.29950000
          scriptPubKey: 76a9145d9536d605d7ddf4f51f57006d1dddc38bb3f79c88ac
      Output 1 - 1428VeCoiJR81vVjdtXe9sb5G15qjYyLwo
          Value: 0.10000000
          scriptPubKey: 76a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac


NewTx - Raw Data
----------------
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 0008b48304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03 022100d0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a30141043ef 1593aa79bab3c6a21f4f82f348b12e68d107f95f577e610466aca7d0f2e4ebcfd9a9bafcad72076 18ec4e5e78358dbac3dff4a432b833bf8a9cee85834ac0ffffffff0180969800000000001976a91 48073e789954e05c5938c5cc493308f9021539bb588ac00000000

NewTx - Human Readable
----------------------
    Hash: 460fcfa566eaf7906cf7768f22d624c4f2e8dc1ba00474b497ad7bbacd696f14
    Data format version: 1
      Input 0 - 1428VeCoiJR81vVjdtXe9sb5G15qjYyLwo
          Previous out: e52fa8a693eaff60f644b5006798215e0798e54a2dfc82aaec045c99cf70e4fb#1
          scriptSig: 48304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e0302210 0d0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a30141043ef1593a a79bab3c6a21f4f82f348b12e68d107f95f577e610466aca7d0f2e4ebcfd9a9bafcad7207618ec4 e5e78358dbac3dff4a432b833bf8a9cee85834ac0
          sequence: 0xffffffff
      Output 0 - 1CiCLjhX1291hwjxZxBNCq1k9Ptkb4YNgR
          Value: 0.10000000
          scriptPubKey: 76a9148073e789954e05c5938c5cc493308f9021539bb588ac

Can you show me exactly what needs to be hashed and signed to generate the signature?

-TT

[TT]

Specifically, I couldn't find the hash type and wasn't sure what to enter for the hash type code. where is the hashtype byte? and what 4 bytes do I need to enter for the code? and do these 4 bytes need to be big-endian? or little-endian?

-TT

[etotheipi]

Sorry to break the news to you:  but you're asking about OP_CHECKSIG which is really complicated.  But, see my post here for a description of how it works (about halfway down the page).  It tells you exactly how to construct the string to be signed.

And everything should be in little-endian unless otherwise stated, including the hashcode.  Just make sure the final hash is big-endian before signing.  Yeah, it's ridiculous...that's why I made this diagram!

-Eto

[TT]

Right, I saw your diagram. I think I get all of it except for the hashTypeCode.

So NewTx broken down looks like:

NewTx
-----
01000000 <---version (big endian)
01           <----tx_in count

fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe5 <-PrevTx hash (big endian)
01000000    <------ index (big endian)

8b      <----- scriptSig length (139 bytes = 2 opcodes + 72 for sig + 65 for pubKey)
48      <---- push the next 72 bytes onto stack OpCode
304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a301 <---- 72 bytes (sig)
41      <----- push the next 65 bytes onto stack OpCode
043ef1593aa79bab3c6a21f4f82f348b12e68d107f95f577e610466aca7d0f2e4ebcfd9a9bafcad 7207618ec4e5e78358dbac3dff4a432b833bf8a9cee85834ac0 <---- 65 bytes (pubKey)

ffffffff     <------sequence
01        <----- tx_out count
8096980000000000  <------- value (big-endian)
19     <--- pk_script length (25 bytes)
76a9148073e789954e05c5938c5cc493308f9021539bb588ac <---- pk_script
00000000   <----- lock time

.....
so where exactly is the hashTypeCode?

-TT

[TT]

The signature is:

304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2 <--- 70 bytes

a301  <---- two additional bytes. what hashTypeCode does this correspond to and
how do we expand it to 4 bytes?

Or am I missing something?

[etotheipi]

so where exactly is the hashTypeCode?

It's the very last byte of the 72-byte signature (bolded below):

304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a301

Expand that 0x01 to four bytes, little-endian, and attach it to the modified, serialized txCopy before signing.

[TT]

OK,


So we have:

01000000       <-----NewTx up to before sigScript
01
fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe5
01000000

76a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac <----- pkScript for PrevTx

00000000  <---- lock time for NewTx

00000001  <---- expanded hashTypeCode (little-endian)


.....
so all together, now:
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 00076a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac0000000000000001

and so this is what gets signed?

I tried signing sha256^2 of it using the private key
24ed089647b7f330588c491309e527c44cbf5e04444540782d6b88f8c44b3105
and got:

304302200a4637d31ac9f9edceeeb3b1095e79e48a49b4048f9177030659f8ae560521f9021f0bf 8515fc5ea6a51c1a8d9c809f7cc62d9914a1c5152947dccd5155c3b52e7

but the signature should be:
304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2

also, what's with the extra a3 byte before the hashTypeCode?

-TT

[etotheipi]

OK,


So we have:

01000000       <-----NewTx up to before sigScript
01
fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe5
01000000

76a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac <----- pkScript for PrevTx

00000000  <---- lock time for NewTx

00000001  <---- expanded hashTypeCode (little-endian)


.....
so all together, now:
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 00076a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac0000000000000001

and so this is what gets signed?

I tried signing it using the private key
24ed089647b7f330588c491309e527c44cbf5e04444540782d6b88f8c44b3105
and got:

304302200a4637d31ac9f9edceeeb3b1095e79e48a49b4048f9177030659f8ae560521f9021f0bf 8515fc5ea6a51c1a8d9c809f7cc62d9914a1c5152947dccd5155c3b52e7

but the signature should be:
304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2

also, what's with the extra a3 byte before the hashTypeCode?

-TT

I'm at work, and don't have my python/btc tools here, but from manual inspection that a3 appears to be part of the signature.  The signature is represented as an r and s value, and in this case the s-value is actually 33 bytes, not 32.  I haven't counted it all up, but I could see how that would lead you to believe there's an extra byte (if you look at the s-value, you'll notice it's actually 32-bytes, but has an extra 0x00 leading byte to remove sign ambiguity).

As for signing:  don't forget that the process for generating a signature involves a random number.  Every signature you generate on the exact same data will produce a completely different-looking signature.  The only way to know for sure whether you did it right is if you used the same random number both times, or (more likely) execute your verification algorithm to check if the signature is valid. 

[TT]

Oh, right...I forgot about that random number.

So it's impossible to generate an exact tx that is identical...you can only verify that the signature is valid...

Would you happen to know how I can verify an ECDSA signature using openssl from the command line?

-TT

[TT]

OK, I've managed to get the OpenSSL library working in C...I can generate an ECDSA signature and verify it.

However, it is telling me that this particular signature for this particular example is invalid.

Can we go over the steps?

NewTx - Raw Data
----------------
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 000

8b48304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022 100d0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a30141043ef159 3aa79bab3c6a21f4f82f348b12e68d107f95f577e610466aca7d0f2e4ebcfd9a9bafcad7207618e c4e5e78358dbac3dff4a432b833bf8a9cee85834ac0

ffffffff0180969800000000001976a91 48073e789954e05c5938c5cc493308f9021539bb588ac00000000

NewTx - scriptSig removed
------------------------------
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 000
ffffffff0180969800000000001976a91 48073e789954e05c5938c5cc493308f9021539bb588ac00000000

PrevTx - scriptPubKey
------------------------
76a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac

NewTx scriptSig replaced by PrevTx scriptPubKey and hashTypeCode appended
------------------------------------------------------
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 000
76a914211f0c809a1a14f46af53ae59aa32d02aaf7272488ac
ffffffff0180969800000000001976a91 48073e789954e05c5938c5cc493308f9021539bb588ac00000000
00000001

NewTx sig
------------
304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03022100d 0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a3

key.pem
------------
-----BEGIN EC PARAMETERS-----
BgUrgQQACg==
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
MHQCAQEEICTtCJZHt/MwWIxJEwnlJ8RMv14EREVAeC1riPjESzEFoAcGBSuBBAAK
oUQDQgAEPvFZOqebqzxqIfT4LzSLEuaNEH+V9XfmEEZqyn0PLk68/Zqbr8rXIHYY
7E5eeDWNusPf9KQyuDO/ipzuhYNKwA==
-----END EC PRIVATE KEY-----

Now in principle, as long as the signature was in fact generated with the key above, sig should be valid for sha256^2(NewTx with scriptSig replaced by PrevTx scriptPubKey and hashTypeCode appended), correct?

-TT

[TT]

You can check that the private key I'm using does in fact match the bitcoin address for the scriptPubKey here:

https://bitcointools.appspot.com/?k=5J6YocBZpn5j9hcPWv1wPEGtfXvHP8g2ZPSSTrjgr9PxUhALeYM

I've tried verifying the hash in both big endian and little endian and neither worked. What am I still missing?

[etotheipi]

There's a lot of data there, so it's tough for me to parse all of it by eye.  But it looks like you're appending a big-endian hashcode instead of little-endian.  Also, try switching the endianness after the sha256^2 operation.  You might have to do some guess-and-check for this part.

Also don't forget to add the 1-byte hashcode to the end of the signature you generated.  It may not be relevant for OpenSSL, but if you're signing transactions to be put in the blockchain, you'll need that 0x01 byte at the end of the pubkey script.

[TT]

I tried appending 00000001 and 01000000,
I tried verifying the data in both endiannesses.
None of the combinations work.

-TT

[TT]

It's easier to see the data when you compare it to:
https://bitcointalk.org/index.php?topic=50330.msg620292#msg620292

[etotheipi]

I don't know if it helps at all, but here's my python code for verifying signatures (which is the same for signing up until step 10).  Default output for all numbers/values is little-endian.  The "binary_switchEndian" call at the end converts the hash to big-endian just before signing.

You can double-check your process against this, as this method is works reliably on real data from the blockchain.

Code:

def checkSig(self, binSig, binPubKey, txOutScript, txInTx, txInIndex, stack, lastOpCodeSep=None):
      # 1. Pop key and sig from the stack 
      binPubKey = stack.pop()
      binSig    = stack.pop()

      # 2. Subscript is from latest OP_CODESEPARATOR until end... if DNE, use whole script
      subscript = txOutScript
      if lastOpCodeSep:
         subscript = subscript[lastOpCodeSep:]
      
      # 3. Signature is deleted from subscript
      #    I'm not sure why this line is necessary - maybe for non-standard scripts?
      lengthInBinary = int_to_binary(len(binSig))
      subscript = subscript.replace( lengthInBinary + binSig, "")

      # 4. Hashtype is popped and stored
      hashtype = binary_to_int(binSig[-1])
      justSig = binSig[:-1]

       # 5. Make a copy of the transaction -- we will be hashing a modified version
      txCopy = PyTx().unserialize( txInTx.serialize() )

      # 6. Remove all OP_CODESEPARATORs
      subscript.replace( int_to_binary(OP_CODESEPARATOR), '')

      # 7. All the TxIn scripts in the copy are blanked (set to empty string)
      for txin in txCopy.inputs:
         txin.binScript = ''

      # 8. Script for the current input in the copy is set to subscript
      txCopy.inputs[txInIndex].binScript = subscript

      # 9. Prepare the signature and public key
      senderAddr = PyBtcAddress().createFromPublicKey(binPubKey)
      binHashCode = int_to_binary(hashtype, widthBytes=4)
      toHash = txCopy.serialize() + binHashCode

      hashToVerify = hash256(toHash)
      hashToVerify = binary_switchEndian(hashToVerify)

      # 10. Apply ECDSA signature verification
      return senderAddr.verifyDERSignature(hashToVerify, justSig)

[TT]

Thanks,

I appreciate all the help.

I'm really close.

[TT]

Hmmm...still missing something.

Would it be possible to tell me exactly what the data needs to look like before the hash + verify are performed for my specific example? Unfortunately, the example given at https://en.bitcoin.it/wiki/OP_CHECKSIG is for a transaction involving generated bitcoin.

PrevTx - Raw Data
-----------------
0100000001e1877fe168c04e1f91a170c37320d8d6e6dbac94cb1edf56eab2d075f548cb9300000 0008c493046022100b0ac6689455d95fb81f0012f38b9285d44ae75f64b4c82ea9d1e96c2541392 7c022100bcf31e15dde4d83b567f848cf6b4a708a23f0a71a206d858bfaea0285fca350f014104c 6420d1b499b277a1f4e284cb4bc4cc327539adfc24bc6fd212577af5665395886660c9777484448 745868e8e5c5159d34c929706941e941f9de2fa6a18817f4ffffffff023000c901000000001976a 9145d9536d605d7ddf4f51f57006d1dddc38bb3f79c88ac80969800000000001976a914211f0c80 9a1a14f46af53ae59aa32d02aaf7272488ac00000000

NewTx - Raw Data
----------------
0100000001fbe470cf995c04ecaa82fc2d4ae598075e21986700b544f660ffea93a6a82fe501000 0008b48304502205ea291ce55ecc95f346f6be2c198993dcb1a72cc4eddf520f173ed9ac85a1e03 022100d0ae6c394d014de8fecb44d034904a0c6142e6335a394aa4629d7a839aaaa2a30141043ef 1593aa79bab3c6a21f4f82f348b12e68d107f95f577e610466aca7d0f2e4ebcfd9a9bafcad72076 18ec4e5e78358dbac3dff4a432b833bf8a9cee85834ac0ffffffff0180969800000000001976a91 48073e789954e05c5938c5cc493308f9021539bb588ac00000000

String to verify = ?

[TT]

Also, I was wondering in your diagram what SCRIPT_PART4 refers to? is that a nonstandard transaction type? or am I really completely lost? all the scripts I've seen so far end with OP_CHECKSIG

[etotheipi]

Also, I was wondering in your diagram what SCRIPT_PART4 refers to? is that a nonstandard transaction type? or am I really completely lost? all the scripts I've seen so far end with OP_CHECKSIG

My diagram is showing an arbitrarily complex script, which would definitely not be standard.  In fact, we were just having a conversation on IRC about how we're not even sure if OP_CODESEPARATOR scripts can actually be useful in any application (I'm sure there is, we just couldn't think of it)!   For your purposes, you can ignore everything in subscript except for that chunk of standard TxOut script:

Code:

OP_DUP OP_HASH160 <ADDR> OP_EQUALVERIFY OP_CHECKSIG

If that's all I put in the diagram, then steps 2-4 would do nothing since they have no effect on standard scripts.  I figured that was too boring, so that's why I added the other stuff.

I am really busy right now, but if you haven't solved your problem by Friday I'll dig in a bit to see if I can help.  Good luck!