[SOLVED] Can someone explain the details of how OP_CHECKSIG operates?

[0x6763]

This is one of my final major roadblocks I need to get past for my bitcoin implementation, and I'm having a hard time understanding everything that's going on in Satoshi's code, and haven't found any other implementations of bitcoin scripting, yet.  A couple weeks ago on IRC, ArtForz explained some of the scripting, but we didn't get into much detail about OP_CHECKSIG.  It was mentioned, however, that it creates a new transaction containing a subset of inputs/outputs, and that new transaction is then hashed.

Can/will anyone please explain this in further detail?

I created a link to a page solely about OP_CHECKSIG on the wiki, if anyone wants to add an explanation there: https://en.bitcoin.it/w/index.php?title=OP_CHECKSIG&action=edit&redlink=1

Thanks!

[dirtyfilthy]

https://en.bitcoin.it/wiki/OP_CHECKSIG

Hope this helps, good luck with your implementation, look forward to seeing it.

[Hal]

I made some edits. It might be nice to explain what the various SIGHASH values are intended to be used for, and whether they will work at this time.

[0x6763]

Thanks dirtyfilthy and Hal!  I believe I now have it mostly figured out.  The opcodes, including OP_CHECKSIG, are running as expected, but my signature verification function is returning false, causing OP_CHECKSIG to return 0.

dirtyfilthy, I'm curious about your code for verifying signatures (and though not related to this current problem, I'd love to see your code for generating new key pairs, and signing data, too). Would you please share that?  This will help me narrow down where the problem is (I think either my signature verification function is incorrect or the temporary transactions I'm creating are incorrect, or both).

If I'm still having problems after I make sure my signature verification function is correct, I'll post hex dumps of the transaction I'm trying to verify as well as the temporary transaction created for signature verification to see if anyone can point out where I went wrong.

Thanks again!

[dirtyfilthy]

public class KeyTools {
   
   // encodes a public key into the form [0x04][32 bytes 'x' value][32 bytes 'y' value]
   
   public static byte[] encodePublicKey(ECPublicKey key){
      net.dirtyfilthy.bouncycastle.math.ec.ECPoint p=EC5Util.convertPoint(key.getParams(), key.getW(), false);
      return p.getEncoded();
   }
   
   // decodes a raw public key in the form [0x04][32 bytes 'x' value][32 bytes 'y' value]
   
   public static ECPublicKey decodePublicKey(byte[] encoded){
      ECNamedCurveParameterSpec params = ECNamedCurveTable.getParameterSpec("secp256k1");
      KeyFactory fact;
      try {
         fact = KeyFactory.getInstance("ECDSA", "DFBC");
      } catch (NoSuchAlgorithmException e) {
         throw new RuntimeException(e);
      } catch (NoSuchProviderException e) {
         throw new RuntimeException(e);
      }
      ECCurve curve = params.getCurve();
      java.security.spec.EllipticCurve ellipticCurve = EC5Util.convertCurve(curve, params.getSeed());
       java.security.spec.ECPoint point=ECPointUtil.decodePoint(ellipticCurve, encoded);
       java.security.spec.ECParameterSpec params2=EC5Util.convertSpec(ellipticCurve, params);
       java.security.spec.ECPublicKeySpec keySpec = new java.security.spec.ECPublicKeySpec(point,params2);
       try {
         return (ECPublicKey) fact.generatePublic(keySpec);
      } catch (InvalidKeySpecException e) {
         throw new RuntimeException(e);
      }
      
   }
   

   public static byte[] signData(ECPrivateKey key,byte data[]){
      Signature s;
      try {
         s = Signature.getInstance("ECDSA", "DFBC");
         s.initSign(key);
         s.update(data);
         return s.sign();
      } catch (NoSuchAlgorithmException e) {
         throw new RuntimeException(e);
      } catch (NoSuchProviderException e) {
         throw new RuntimeException(e);
      } catch (InvalidKeyException e) {
         throw new RuntimeException(e);
      } catch (SignatureException e) {
         throw new RuntimeException(e);
      }
      
   }
   

   public static KeyPair generateKeyPair() {
      ECParameterSpec ecSpec = ECNamedCurveTable.getParameterSpec("secp256k1");
      KeyPairGenerator generator;
      try {
         generator = KeyPairGenerator.getInstance("ECDSA", "DFBC");
         generator.initialize(ecSpec, new SecureRandom());
         KeyPair pair = generator.generateKeyPair();
         ContentValues initialValues = new ContentValues();
         return pair;
      } catch (NoSuchAlgorithmException e) {
         throw new RuntimeException(e);
      } catch (NoSuchProviderException e) {
         throw new RuntimeException(e);
      } catch (InvalidAlgorithmParameterException e) {
         throw new RuntimeException(e);
      }
      
   }

   public static boolean verifySignedData(ECPublicKey key,byte data[], byte sig[]){
      Signature s; 
      try {
      s= Signature.getInstance("ECDSA", "DFBC");
      s.initVerify(key);
      s.update(data);
      return s.verify(sig);
      } catch (NoSuchAlgorithmException e) {
         throw new RuntimeException(e);
      } catch (NoSuchProviderException e) {
         throw new RuntimeException(e);
      } catch (InvalidKeyException e) {
         throw new RuntimeException(e);
      } catch (SignatureException e) {
         throw new RuntimeException(e);
      }
   }

}

I renamed the packages from org.bouncycastle to net.dirtyfilthy.bouncycastle so as not to conflict with the (incomplete) android bouncycastle libraries. Also renamed the provider string from "BC" to "DFBC" for the same reason. Not sure if your developing for android or just for the standard javavm, in which case you should rename these to their usual names.

[0x6763]

Thanks again dirtyfilthy!  Amazingly, it turns out I'm generating keys, signing and verifying data the same way as you are.

So this means I'm misunderstanding something when creating the transaction copy or the sub-script to be used in the transaction copy.

I'm working with transaction 0f898c5494eaa468d12cf0630a0c0b238cc1149c1c53bbc592d16497094e95ff:
http://blk.bitcoinwatch.com/b?h=103958
http://blockexplorer.com/t/AgktCstTqJ

Code:

Here's a parsed version of this transaction (plain hexdump version below):

Tx:
01 00 00 00 - version
03 - TxIn count

TxIn:
  Outpoint:
    30 f3 70 1f 9b c4 64 55 2f 70 49 57 91 04 08 17
    ce 77 7a d5 ed e1 6e 52 9f cd 0c 0e 94 91 56 94 - prev tx hash
    00 00 00 00                                     - prev TxOut index

  8c                                                - scriptSig length
  49 30 46 02 21 00 f5 74 6b 0b 25 4f 5a 37 e7 52
  51 45 9c 7a 23 b6 df cb 86 8a c7 46 7e dd 9a 6f
  dd 1d 96 98 71 be 02 21 00 88 94 8a ea 29 b6 91
  61 ca 34 1c 49 c0 26 86 a8 1d 8c bb 73 94 0f 91
  7f a0 ed 71 54 68 6d 3e 5b 01 41 04 47 d4 90 56
  1f 39 6c 8a 9e fc 14 48 6b c1 98 88 4b a1 83 79
  bc ac 2e 0b e2 d8 52 51 34 ab 74 2f 30 1a 9a ca
  36 60 6e 5d 29 aa 23 8a 9e 29 93 00 31 50 42 3d
  f6 92 45 63 64 2d 4a fe 9b f4 fe 28               - scriptSig
  ff ff ff ff                                       - sequence

TxIn:
  Outpoint:
    72 14 2b f7 68 6c e9 2c 6d e5 b7 33 65 bf b9 d5
    9b b6 0c 2c 80 98 2d 59 58 c1 e6 a3 b0 8e a6 89 - prev tx hash
    00 00 00 00                                     - prev TxOut index
  4a                                                - scriptSig length
  49 30 46 02 21 00 bc e4 3a d3 ac bc 79 b0 24 7e
  54 c8 c9 1e ac 1c f9 03 75 05 00 0e 01 d1 fd 81
  18 54 d8 5b c2 1a 02 21 00 99 2a 6f 6f 2f eb 6f
  62 d3 70 6f 3b 9a aa b8 8d 9f 11 32 95 6a 1d ff
  a9 26 cd 55 6e d5 53 60 df 01                     - scriptSig
  ff ff ff ff                                       - sequence

TxIn:
  Outpoint:
    d2 81 28 bb b6 20 7c 1c 3d 0a 63 0c c6 19 dc 7e
    7b ea 56 ac 19 a1 da b1 27 c6 2c 78 fa 1b 63 2c - prev tx hash
    00 00 00 00                                     - prev TxOut index
  49                                                - scriptSig length
  48 30 45 02 20 20 97 57 36 81 61 53 77 08 fd 29
  d8 9b b1 e9 d6 48 00 79 49 ec fd ed 78 9b 51 a9
  63 24 cb 65 18 02 21 00 cd 0f 7c 30 21 39 16 48
  2b 6e 16 6d 8a 4f 2b 98 1f 77 7e b1 84 cd 8a 49
  5f 1b 3d 36 90 fb bf 2d 01                        - scriptSig
  ff ff ff ff                                       - sequence

01                                                  - TxOut count

TxOut:
  00 a6 f7 5f 02 00 00 00                           - value
  19                                                - scriptPubKey length
  76 a9 14 9e 35 d9 3c 77 92 bd ca ad 56 97 dd eb
  f0 43 53 d9 a5 e1 96 88 ac                        - scriptPubKey
00 00 00 00                                         - lock time


Plain hexdump:
01 00 00 00 03 30 f3 70  1f 9b c4 64 55 2f 70 49   .....0.p...dU/pI
57 91 04 08 17 ce 77 7a  d5 ed e1 6e 52 9f cd 0c   W.....wz...nR...
0e 94 91 56 94 00 00 00  00 8c 49 30 46 02 21 00   ...V......I0F.!.
f5 74 6b 0b 25 4f 5a 37  e7 52 51 45 9c 7a 23 b6   .tk.%OZ7.RQE.z#.
df cb 86 8a c7 46 7e dd  9a 6f dd 1d 96 98 71 be   .....F~..o....q.
02 21 00 88 94 8a ea 29  b6 91 61 ca 34 1c 49 c0   .!.....)..a.4.I.
26 86 a8 1d 8c bb 73 94  0f 91 7f a0 ed 71 54 68   &.....s......qTh
6d 3e 5b 01 41 04 47 d4  90 56 1f 39 6c 8a 9e fc   m>[.A.G..V.9l...
14 48 6b c1 98 88 4b a1  83 79 bc ac 2e 0b e2 d8   .Hk...K..y......
52 51 34 ab 74 2f 30 1a  9a ca 36 60 6e 5d 29 aa   RQ4.t/0...6`n]).
23 8a 9e 29 93 00 31 50  42 3d f6 92 45 63 64 2d   #..)..1PB=..Ecd-
4a fe 9b f4 fe 28 ff ff  ff ff 72 14 2b f7 68 6c   J....(....r.+.hl
e9 2c 6d e5 b7 33 65 bf  b9 d5 9b b6 0c 2c 80 98   .,m..3e......,..
2d 59 58 c1 e6 a3 b0 8e  a6 89 00 00 00 00 4a 49   -YX...........JI
30 46 02 21 00 bc e4 3a  d3 ac bc 79 b0 24 7e 54   0F.!...:...y.$~T
c8 c9 1e ac 1c f9 03 75  05 00 0e 01 d1 fd 81 18   .......u........
54 d8 5b c2 1a 02 21 00  99 2a 6f 6f 2f eb 6f 62   T.[...!..*oo/.ob
d3 70 6f 3b 9a aa b8 8d  9f 11 32 95 6a 1d ff a9   .po;......2.j...
26 cd 55 6e d5 53 60 df  01 ff ff ff ff d2 81 28   &.Un.S`........(
bb b6 20 7c 1c 3d 0a 63  0c c6 19 dc 7e 7b ea 56   .. |.=.c....~{.V
ac 19 a1 da b1 27 c6 2c  78 fa 1b 63 2c 00 00 00   .....'.,x..c,...
00 49 48 30 45 02 20 20  97 57 36 81 61 53 77 08   .IH0E.  .W6.aSw.
fd 29 d8 9b b1 e9 d6 48  00 79 49 ec fd ed 78 9b   .).....H.yI...x.
51 a9 63 24 cb 65 18 02  21 00 cd 0f 7c 30 21 39   Q.c$.e..!...|0!9
16 48 2b 6e 16 6d 8a 4f  2b 98 1f 77 7e b1 84 cd   .H+n.m.O+..w~...
8a 49 5f 1b 3d 36 90 fb  bf 2d 01 ff ff ff ff 01   .I_.=6...-......
00 a6 f7 5f 02 00 00 00  19 76 a9 14 9e 35 d9 3c   ..._.....v...5.<
77 92 bd ca ad 56 97 dd  eb f0 43 53 d9 a5 e1 96   w....V....CS....
88 ac 00 00 00 00                                  ......

I'm attempting to verify the transaction input that points to the only output of transaction 30f3701f9bc464552f70495791040817ce777ad5ede16e529fcd0c0e94915694:
http://blk.bitcoinwatch.com/b?h=103640
http://blockexplorer.com/t/6d3E7ZWBDe

Code:

Parsed version of the transaction copy when trying to verify the input (plain hexdump version below):

Tx:
01 00 00 00                                          - version
03                                                   - TxIn count

TxIn:
  Outpoint:
    30 f3 70 1f 9b c4 64 55 2f 70 49 57 91 04 08 17
    ce 77 7a d5 ed e1 6e 52 9f cd 0c 0e 94 91 56 94  - prev TxOut hash
    00 00 00 00                                      - index

  5b                                                 - scriptSig length
  41 04 47 d4 90 56 1f 39 6c 8a 9e fc 14 48 6b c1
  98 88 4b a1 83 79 bc ac 2e 0b e2 d8 52 51 34 ab
  74 2f 30 1a 9a ca 36 60 6e 5d 29 aa 23 8a 9e 29
  93 00 31 50 42 3d f6 92 45 63 64 2d 4a fe 9b f4
  fe 28 76 a9 14 02 bf 4b 28 89 c6 ad a8 19 0c 25
  2e 70 bd e1 a1 90 9f 96 17 88 ac                   - scriptSig (includes scriptPubkey
                                                                  from prev TxOut)
  ff ff ff ff                                        - sequence

TxIn:
  Outpoint:
    72 14 2b f7 68 6c e9 2c 6d e5 b7 33 65 bf b9 d5
    9b b6 0c 2c 80 98 2d 59 58 c1 e6 a3 b0 8e a6 89  - prev TxOut hash
    00 00 00 00                                      - index
  00                                                 - scriptSig length
  ff ff ff ff                                        - sequence

TxIn:
  Outpoint:
    d2 81 28 bb b6 20 7c 1c 3d 0a 63 0c c6 19 dc 7e
    7b ea 56 ac 19 a1 da b1 27 c6 2c 78 fa 1b 63 2c  - prev TxOut hash
    00 00 00 00                                      - index
  00                                                 - scriptSig length
  ff ff ff ff                                        - sequence

TxOut Count
  01                                                 - TxOut count
  00 a6 f7 5f 02 00 00 00                            - Transaction Value
  19                                                 - scriptPubKey length
  76 a9 14 9e 35 d9 3c 77 92 bd ca ad 56 97 dd eb
  f0 43 53 d9 a5 e1 96 88 ac                         - scriptPubKey

00 00 00 00                                          - sequence


Plain hexdump:
01 00 00 00 03 30 f3 70  1f 9b c4 64 55 2f 70 49   .....0.p...dU/pI
57 91 04 08 17 ce 77 7a  d5 ed e1 6e 52 9f cd 0c   W.....wz...nR...
0e 94 91 56 94 00 00 00  00 5b 41 04 47 d4 90 56   ...V.....[A.G..V
1f 39 6c 8a 9e fc 14 48  6b c1 98 88 4b a1 83 79   .9l....Hk...K..y
bc ac 2e 0b e2 d8 52 51  34 ab 74 2f 30 1a 9a ca   ......RQ4.t/0...
36 60 6e 5d 29 aa 23 8a  9e 29 93 00 31 50 42 3d   6`n]).#..)..1PB=
f6 92 45 63 64 2d 4a fe  9b f4 fe 28 76 a9 14 02   ..Ecd-J....(v...
bf 4b 28 89 c6 ad a8 19  0c 25 2e 70 bd e1 a1 90   .K(......%.p....
9f 96 17 88 ac ff ff ff  ff 72 14 2b f7 68 6c e9   .........r.+.hl.
2c 6d e5 b7 33 65 bf b9  d5 9b b6 0c 2c 80 98 2d   ,m..3e......,..-
59 58 c1 e6 a3 b0 8e a6  89 00 00 00 00 00 ff ff   YX..............
ff ff d2 81 28 bb b6 20  7c 1c 3d 0a 63 0c c6 19   ....(.. |.=.c...
dc 7e 7b ea 56 ac 19 a1  da b1 27 c6 2c 78 fa 1b   .~{.V.....'.,x..
63 2c 00 00 00 00 00 ff  ff ff ff 01 00 a6 f7 5f   c,............._
02 00 00 00 19 76 a9 14  9e 35 d9 3c 77 92 bd ca   .....v...5.<w...
ad 56 97 dd eb f0 43 53  d9 a5 e1 96 88 ac 00 00   .V....CS........
00 00                                              ..

This is just a plain JVM-based client written in Clojure (http://clojure.org), not intended for Android use at this point.

P.S. dirtyfilthy, it looks like you've got an unnecessary line of code in your generateKeyPair() function:

Code:

ContentValues initialValues = new ContentValues();

EDIT: Changed byte-order of transaction hashes.

[Mike Hearn]

Yes, this took a while for me to get right as well.

I'm afraid you just have to pay REALLY careful attention to endian-ness and data element size. Note that at one point the hash type marker is serialized as a uint32 (four bytes) and at another point it's serialized as one byte.

In the end I did this by instrumenting the official client and making it dump out what it was hashing. After fixing some bugs I got them to match and was able to successfully send/receive coins with my own Java code.

I really wish I could release this code to help you! Unfortunately for reasons I won't delve into here, I can't do so (at the moment). Good luck!

[Hal]

I don't see how that hex dump corresponds to the transaction. You have outpoint hashes starting with 30 f3, 72 14, and d2 81, none of which are in the transaction.

[0x6763]

I don't see how that hex dump corresponds to the transaction. You have outpoint hashes starting with 30 f3, 72 14, and d2 81, none of which are in the transaction.

This is a difference between the byte order as pulled from blk0001.dat and as shown on sites like blockexplorer and blk.bitcoinwatch.com.  I know when I lookup a block or transaction hash in the blkindex.dat file, I have to lookup the reverse of what is shown on those websites, and I although I'm setting my ChannelBuffer's to little endian, when reading bytes rather than numbers, it should be reading them left-to-right.  To top it off, however, my packet sniff's have also shown these hashes to be in the opposite order from what is found on blockexplorer and blk.bitcoinwatch.com.

Code:

This is the reverse byte order of --------------------------------> This
945691940e0ccd9f526ee1edd57a77ce170804915749702f5564c49b1f70f330 -- 30f3701f9bc464552f70495791040817ce777ad5ede16e529fcd0c0e94915694

I wrote the hashes on my previous post to match the order found at those sites, but perhaps I should edit my post.  At least they'd be consistent in my posts, then.  No reason to make my posts confusing because some sites are doing things backwards.

EDIT: I changed the byte-order of those hashes.

[Hal]

Wow, that's bizarre that the byte order of the hashes is reversed on the wire like that. Have you tried reproducing the published transaction hash by hashing the raw transaction? Does that work? What do you get, 0f89... as it is on the wire, or the reverse, ff95... as blockexplorer has it? And do you need to byte reverse the embedded hashes in the tx in order to get the right answer?

[0x6763]

Wow, that's bizarre that the byte order of the hashes is reversed on the wire like that. Have you tried reproducing the published transaction hash by hashing the raw transaction? Does that work? What do you get, 0f89... as it is on the wire, or the reverse, ff95... as blockexplorer has it? And do you need to byte reverse the embedded hashes in the tx in order to get the right answer?

Yeah, I can take a raw transaction from blk0001.dat or from the wire and hash it, and I get the same bytes that are shown on those websites, but in reverse order.  From what I just found out in IRC, bitcoin internally treats these hashes as little endian BigNumbers, and then when their hex code is printed out, it's in reverse order from the actual data.

[Hal]

Okay, I've confirmed that hashes OK using sha256sum and gets your value, 0f898c5494eaa468d12cf0630a0c0b238cc1149c1c53bbc592d16497094e95ff. So the question remains to verify the sig.

You will have to add 01 00 00 00 to the end before you hash, for the hashtype.

I'm not sure about your scriptsig shortening. You removed the sig ok, but there seems to be more stuff added at the end, after the pubkey.

[0x6763]

Okay, I've confirmed that hashes OK using sha256sum and gets your value, 0f898c5494eaa468d12cf0630a0c0b238cc1149c1c53bbc592d16497094e95ff. So the question remains to verify the sig.

You will have to add 01 00 00 00 to the end before you hash, for the hashtype.

I'm not sure about your scriptsig shortening. You removed the sig ok, but there seems to be more stuff added at the end, after the pubkey.

Yeah, I'm adding those 4 bytes before hashing.

As for the extra stuff after the pubkey, what I'm (possibly incorrectly) doing there is using the combined scripts of scriptSig (from TxIn) and scriptPubKey (from prev TxOut), then removing the signature and OP_CODESEPARATOR (which there are none in any examples I've tried so far).  So that extra stuff is the scriptPubKey...it's possible it's not supposed to be there, but even when I remove that, it still doesn't verify anyway, so I don't know yet if that's a problem, too.

Here's the original scriptSig from TxIn:

Code:

49 30 46 02 21 00 f5 74 6b 0b 25 4f 5a 37 e7 52
51 45 9c 7a 23 b6 df cb 86 8a c7 46 7e dd 9a 6f
dd 1d 96 98 71 be 02 21 00 88 94 8a ea 29 b6 91
61 ca 34 1c 49 c0 26 86 a8 1d 8c bb 73 94 0f 91
7f a0 ed 71 54 68 6d 3e 5b 01 41 04 47 d4 90 56
1f 39 6c 8a 9e fc 14 48 6b c1 98 88 4b a1 83 79
bc ac 2e 0b e2 d8 52 51 34 ab 74 2f 30 1a 9a ca
36 60 6e 5d 29 aa 23 8a 9e 29 93 00 31 50 42 3d
f6 92 45 63 64 2d 4a fe 9b f4 fe 28


Signature size:
 49
Signature:
 30 46 02 21 00 f5 74 6b 0b 25 4f 5a 37 e7 52 51
 45 9c 7a 23 b6 df cb 86 8a c7 46 7e dd 9a 6f dd
 1d 96 98 71 be 02 21 00 88 94 8a ea 29 b6 91 61
 ca 34 1c 49 c0 26 86 a8 1d 8c bb 73 94 0f 91 7f
 a0 ed 71 54 68 6d 3e 5b 01


Broken down further:

Public key size:
 41
Public key:
 04
 X:
  47 d4 90 56 1f 39 6c 8a 9e fc 14 48 6b c1 98 88
  4b a1 83 79 bc ac 2e 0b e2 d8 52 51 34 ab 74 2f
 Y:
  30 1a 9a ca 36 60 6e 5d 29 aa 23 8a 9e 29 93 00
  31 50 42 3d f6 92 45 63 64 2d 4a fe 9b f4 fe 28

and here's the concatenated scripts with the signature removed:

Code:

41 04 47 d4 90 56 1f 39 6c 8a 9e fc 14 48 6b c1
98 88 4b a1 83 79 bc ac 2e 0b e2 d8 52 51 34 ab
74 2f 30 1a 9a ca 36 60 6e 5d 29 aa 23 8a 9e 29
93 00 31 50 42 3d f6 92 45 63 64 2d 4a fe 9b f4
fe 28 76 a9 14 02 bf 4b 28 89 c6 ad a8 19 0c 25
2e 70 bd e1 a1 90 9f 96 17 88 ac


Broken down further:

Public key size:
41
Public key:
04
 X:
  47 d4 90 56 1f 39 6c 8a 9e fc 14 48 6b c1 98 88
  4b a1 83 79 bc ac 2e 0b e2 d8 52 51 34 ab 74 2f
 Y:
  30 1a 9a ca 36 60 6e 5d 29 aa 23 8a 9e 29 93 00
  31 50 42 3d f6 92 45 63 64 2d 4a fe 9b f4 fe 28
scriptPubKey:
 76 - OP_DUP
 a9 - OP_HASH160
 14 - number of bytes to push
 02 bf 4b 28 89 c6 ad a8 19 0c 25 2e 70 bd e1 a1 90 9f 96 17 - push these bytes onto stack
 88 - OP_EQUALVERIFY
 ac - OP_CHECKSIG

Is scriptPubKey supposed to be in here, too, when created the new temporary transaction to be hashed?  So far any docs I've found make it sound like it's there or just don't specify what script or combination of scripts it's supposed to deal with when verifying a transaction.

[Mike Hearn]

The tx is hashed in a form where all the inputs are cleared except the one you are calculating, and that one is set to the connected outputs scriptPubKey.

See below for an example.

Code:

public void signInputs(@NotNull SigHash hashType, @NotNull Wallet wallet) throws ScriptException {
        assert inputs.size() > 0;
        assert outputs.size() > 0;

        // I don't currently have an easy way to test other modes work, as the official client does not use them.
        assert hashType == SigHash.ALL;

        // The transaction is signed with the input scripts empty except for the input we are signing. In the case
        // where addInput has been used to set up a new transaction, they are already all empty. The input being signed
        // has to have the connected OUTPUT program in it when the hash is calculated!
        //
        // Note that each input may be claiming an output sent to a different key. So we have to look at the outputs
        // to figure out which key to sign with.

        byte[][] signatures = new byte[inputs.size()][];
        ECKey[] signingKeys = new ECKey[inputs.size()];
        for (int i = 0; i < inputs.size(); i++) {
            TransactionInput input = inputs.get(i);
            assert input.scriptBytes.length == 0 : "Attempting to sign a non-fresh transaction";
            // Set the input to the script of its output.
            input.scriptBytes = input.outpoint.getConnectedPubKeyScript();
            // Find the signing key we'll need to use.
            byte[] connectedPubKeyHash = input.outpoint.getConnectedPubKeyHash();
            ECKey key = wallet.findKeyFromPubHash(connectedPubKeyHash);
            // This assert should never fire. If it does, it means the wallet is inconsistent.
            assert key != null : "Transaction exists in wallet that we cannot redeem: " + Utils.bytesToHexString(connectedPubKeyHash);
            // Keep the key around for the script creation step below.
            signingKeys[i] = key;
            // The anyoneCanPay feature isn't used at the moment.
            boolean anyoneCanPay = false;
            byte[] hash = hashTransactionForSignature(hashType, anyoneCanPay);
            Utils.LOG("  signInputs hash=" + Utils.bytesToHexString(hash));
            // Set the script to empty again for the next input.
            input.scriptBytes = TransactionInput.EMPTY_ARRAY;

            // Now sign for the output so we can redeem it. We use the keypair to sign the hash,
            // and then put the resulting signature in the script along with the public key (below).
            try {
                ByteArrayOutputStream bos = new ByteArrayOutputStream();
                bos.write(key.sign(hash));
                bos.write((hashType.ordinal() + 1) | (anyoneCanPay ? 0x80 : 0)) ;
                signatures[i] = bos.toByteArray();
            } catch (IOException e) {
                throw new RuntimeException(e);  // Cannot happen.
            }
        }

        // Now we have calculated each signature, go through and create the scripts. Reminder: the script consists of
        // a signature (over a hash of the transaction) and the complete public key needed to sign for the connected
        // output.
        for (int i = 0; i < inputs.size(); i++) {
            TransactionInput input = inputs.get(i);
            assert input.scriptBytes.length == 0;
            ECKey key = signingKeys[i];
            input.scriptBytes = Script.createInputScript(signatures[i], key.getPubKey());
        }

        // Every input is now complete.
    }

[Hal]

What you are doing looks pretty good to me now that I understand it better. I tried hashing your data and got 90377525e05bd71ce8ba413a84fdaea299766732f165fab28a69d30c83337f9b, don't know if that matches yours. The only other thing I can think of would be to try reversing the hash on input to the ECDSA functions.

[0x6763]

What you are doing looks pretty good to me now that I understand it better. I tried hashing your data and got 90377525e05bd71ce8ba413a84fdaea299766732f165fab28a69d30c83337f9b, don't know if that matches yours. The only other thing I can think of would be to try reversing the hash on input to the ECDSA functions.

That's the same hash I'm getting...I'm trying different things but not getting it to verify.  I still need to look a little closer at [mike]'s code posted above...

[0x6763]

I looked at [mike]'s example, but I couldn't find anything that I'm doing wrong, yet.

I'm starting to wonder if maybe we (JVM-based developers) should be using org.bouncycastle.crypto.signers.ECDSASigner for signing and verification instead of java.security.Signature.  I'd like to try it, but I need to figure out what parameters I need to give it...

[0x6763]

For now I'm going to take [mike]'s approach at making the official client print the transactions that are about to be hashed to the log file.

This is in the SignatureHash() function in script.cpp around line 930, right at the very end of the function.  I added the PrintHex line, which prints a hex version of ss to the debug.log file in ~/.bitcoin/

Code:

uint256 SignatureHash(CScript scriptCode, const CTransaction& txTo, unsigned int nIn, int nHashType)

    // other code here
    ...

    // Serialize and hash
    CDataStream ss(SER_GETHASH);
    ss.reserve(10000);
    ss << txTmp << nHashType;
    PrintHex(ss.begin(), ss.end(), "ss: %s\n"); // I added this line here
    return Hash(ss.begin(), ss.end());
}

As soon as it got the next block, I got a few examples to work with.  I'm going to dig through those sometime, and I'll post back here (probably tomorrow) with what I find out.

[Hal]

Okay, I did that too, and the secret is that the scriptSig is entirely removed and replaced with the scriptPubKey from the source (old) transaction. I'm still a little confused about how this works in the source.

VerifyScript() doesn't actually concatenate the two scripts. It runs scriptSig, and that leaves stuff on the stack, then it runs the old scriptPubKey with the stack left by scriptSig. The only connection is the stack. So when we run the scriptPubKey, which holds the OP_CHECKSIG, the "current script" is just that, the old scriptPubKey. This is the script which gets OP_CODESEPARATOR stripped and then put in place of the scriptSig, for hashing.

What I don't understand is this line from the OP_CHECKSIG code:

                    // Drop the signature, since there's no way for a signature to sign itself
                    scriptCode.FindAndDelete(CScript(vchSig));

This is apparently the basis for the advice to "remove the signature" for hashing. But it doesn't make sense, because there is no signature in scriptPubKey, which is where this opcode is found. This line doesn't seem to do anything, and advice to remove a nonexistent signature is misleading.

[theymos]

But it doesn't make sense, because there is no signature in scriptPubKey, which is where this opcode is found. This line doesn't seem to do anything, and advice to remove a nonexistent signature is misleading.

It's legal to put a signature in the scriptPubKey -- Bitcoin just doesn't do so.