Constructing raw P2WSH by hand: witnessScript does not match scriptPubKey

[dkjfh]

I've been trying to create and spend from a P2WSH transaction, for learning purposes, and have been unable to construct a correct spending transaction.

Here are all my steps, starting from a fresh wallet :
1. Get wallet address, public key, and private key, mine some blocks to it:

Code:

ADRESS=bitcoin-cli getnewaddress
PUBKEY=parse(bitcoin-cli getaddressinfo $ADRESS) //gets result.pubkey from command result
PRIVKEY=bitcoin-cli dumpprivkey $ADDRESS
bitcoin-cli dumpprivkey $ADDRESS

2. Grab an unspent output from "bitcoin-cli listunspent" and create a P2WSH output transaction:

Code:

UTX= //grab here
TX=bitcoin-tx -create in=$UTX:0 outscript=49.99:"0x21 0x$PUBKEY OP_CHECKSIGVERIFY":W

3. Sign and broadcast:

Code:

bitcoin-cli signrawtransactionwithkey "$TX" '["$PRIVKEY"]'
NEWTX=bitcoin-cli sendrawtransaction [/* the hex from the output of the above command */]

4. Create a transaction spending from NEWTX (still sending to myself):

Code:

SECONDTX=bitcoin-tx -create in=$NEWTX:0 outscript=49.98:"0x21 0x$PUBKEY OP_CHECKSIGVERIFY":W

5. Attempt to sign it:

Code:

SERIALISEDSCRIPT=//serialse it into 21$PUBKEYad - verify with bitcoin-cli decodescript
bitcoin-cli signrawtransactionwithkey $SECONDTX '["$PRIVKEY"]' '[{"txid": "$NEWTX", "vout": 0, "scriptPubKey": "/* the scriptpubkey from NEWTX */", "witnessScript":"$SERIALISEDSCRIPT", "amount": 49.99}]'

At this point I'm getting the error

Code:

redeemScript/witnessScript does not match scriptPubKey

.

I cannot figure out why it wouldn't match. My serialisation would be the obvious culprit, but decodescript confirms it is correct. I have found https://bitcointalk.org/index.php?topic=5236818.0, but that bug was apparently fixed in Bitcoin Core 0.20, and I am on the latest release (0.20.1).

Any pointers would be greatly appreciated.

Here's the standard support template, if necessary:
Bitcoin Client Software and Version Number: 0.20.1
Operating System: Linux 64-bit
System Hardware Specs: N/A
Description of Problem: See above
Any Related Addresses: N/A (regtest network)
Any Related Transaction IDs: N/A
Screenshot of the problem: N/A
Log Files from the Bitcoin Client: N/A

[achow101]

Can you post the actual witnessScript and the funding transaction (the one that sent money to the address)?

[dkjfh]

The witnessScript I'm using is just a push of a pubkey followed by op_checksigverify. Would you like me to post the verbose log of what I did, including the actual values for the public keys, private keys and addresses?

I'm not sure what you mean by the funding transaction here - the address was initially funded by coinbase transactions, from running the generatetoaddress command (this is all on regtest). Then I created a transaction spending from one of the coinbases, with a P2WSH output - this was successful, and my process is also described in the post here (that's TX when being built, and the txid is saved as NEWTX once it's broadcast - granted my naming isn't the best for this). Then I'm trying to spend the P2WSH output, which is where I'm encountering the issue.

[achow101]

Would you like me to post the verbose log of what I did, including the actual values for the public keys, private keys and addresses?

Yes. Private keys aren't necessary, only if you want to share them.

I tried doing exactly the same thing you did, but did not run into your error (although I ran into another one which I'll explain below). The likely culprit is that you have specified the scriptPubKey incorrectly.

The error I ran into is one that you will run into too. Specifically, it's that Bitcoin Core cannot sign this transaction because you are using OP_CHECKSIGVERIFY. It looks like you are going for the typical Pay to Pubkey script, but the Pay to Pubkey script uses OP_CHECKSIG, not OP_CHECKSIGVERIFY. Unfortunately Bitcoin Core does not have a general signer (note that this is a hard problem) and thus cannot sign your transaction because it does not understand a script that uses OP_CHECKSIGVERIFY.

[dkjfh]

So I tried to do it again and somehow got a different error this time:

Code:

"error": "Unable to sign input, invalid stack size (possibly missing key)"

Is this what you meant? I've included a full log below just in case if not (if that's the expected error, then feel free to skip the second half of my post).

Could you clarify why it wouldn't be able to sign my transaction? Would it be because it's non-standard? I'm not sure how that would affect things. From reading the spec, in particular BIP141 and the Script definitions, my impression is that signing a transaction is a simple matter of serialising part of it and calculating the ECDSA signature, which shouldn't depend on the actual script being used (taken from https://en.bitcoin.it/wiki/BIP_0143). Then, as https://en.bitcoin.it/wiki/BIP_0141 defines it, constructing the witness script would simply involve concatenating the resulting signature with whatever is passed as the input to the command. Am I misunderstanding something here?

I did give it a try using OP_CHECKSIG for a canonical pay-to-pubkey script, but bitcoin-tx recognised it as a pubkey script and created a P2WPK output, which is not what I want. Does this mean that there's no way to use Bitcoin Core to sign P2WSH outputs, or am I misunderstanding what you said about OP_CHECKSIGVERIFY? If so, how would one even sign and use such a transaction?

I will also admit that my original attempt used a much more complex witnessScript, which would succeed either on a 2-of-2 multisig, or on a single sig after a checksequenceverify timelock expired. In my attempts to simplify it yesterday night, I must have botched something in the process. Now that I've gotten the witness script to at least match and the error is different, I'll give it another try with my original script, and if I get anything interesting I'll add it here.

EDIT: After re-running with my full script, I am still getting a script mismatch. This makes me believe I am making some mistake in the serialisation of the full script. Is there a canonical tool that I can use to "compile" Script? I have found some javascript library which does the job, and was using a combination of that and just reviewing it by hand, but I must be missing some subtlety, possibly in the encoding of the sequence number or similar. I am slightly surprised that there is a decodescript RPC but no encodescript or similar; what would be a good way to make sure I canonically serialise it? I suppose I could dig inside bitcoin core and find a way to call the code it's using directly, but surely there's a more end-user facing interface somewhere?

Anyway, as promised, here's the full log of what I'm doing - in case that error wasn't what you expected, or if you by any chance spot something I'm doing wrong. I'll be fully verbose to make sure every step of what I'm doing is clear, so apologies I include some things which seem trivial or unnecessary (and also for the large size of the upcoming post).

Code:

$ ./bitcoin-cli createwallet "testing"
$ ./bitcoin-cli -rpcwallet="testing" getnewaddress
bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr

$ ./bitcoin-cli -rpcwallet="testing" getaddressinfo bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr
#snip full output object...
    "pubkey": "02b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c",
#snip

$ ./bitcoin-cli -rpcwallet="testing" dumpprivkey bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr
cTiHMJ3hwBb3Rtq5XzQB7569AJ7PfRsCfLr1anJnKWGMwYuB6L6S

$ ./bitcoin-cli generatetoaddress 200 bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr
$ ./bitcoin-cli -rpcwallet="testing" listunspent
#pick a random transaction, snip the rest...
{
    "txid": "98e151b3525d96e7e138199c8562ad1ad5e84e9686543b593dedae8f828715f0",
    "vout": 0,
    "address": "bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr",
    "label": "",
    "scriptPubKey": "00145d1cd270f53287f2e51d81c522da2634d10498f0",
    "amount": 25.00000000,
    "confirmations": 139,
    "spendable": true,
    "solvable": true,
    "desc": "wpkh([462dc519/0'/0'/0']02b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c)#dg08u9pr",
    "safe": true
},

#...snip the rest

At this point I have a wallet with an address, corresponding pub/priv keys, a balance and a funding transaction to spend from. So I proceed to create the P2WSH transaction:

Code:

$ ./bitcoin-tx -create -json in=98e151b3525d96e7e138199c8562ad1ad5e84e9686543b593dedae8f828715f0:0 outscript=24.99:"0x21 0x02b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c OP_CHECKSIGVERIFY":W
{
    "txid": "d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756",
    "hash": "d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756",
    "version": 2,
    "size": 94,
    "vsize": 94,
    "weight": 376,
    "locktime": 0,
    "vin": [
#snip
    ],
    "vout": [
        {
            "value": 24.99000000,
            "n": 0,
            "scriptPubKey": {
                "asm": "0 5067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c",
                "hex": "00205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c",
                "reqSigs": 1,
                "type": "witness_v0_scripthash",
                "addresses": [
                    "bc1q2pnsyrafcgwtzvsssr5zca9pzdnr3zufvlm03kjs5tyu59raxekqw9ak09"
                ]
            }
        }
    ],
    "hex": "0200000001f01587828faeed3d593b5486964ee8d51aad62859c1938e1e7965d52b351e1980000000000ffffffff01c0b6f394000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c00000000"
}

$ ./bitcoin-cli signrawtransactionwithkey "0200000001f01587828faeed3d593b5486964ee8d51aad62859c1938e1e7965d52b351e1980000000000ffffffff01c0b6f394000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c00000000" '["cTiHMJ3hwBb3Rtq5XzQB7569AJ7PfRsCfLr1anJnKWGMwYuB6L6S"]'
{
  "hex": "02000000000101f01587828faeed3d593b5486964ee8d51aad62859c1938e1e7965d52b351e1980000000000ffffffff01c0b6f394000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c0247304402207435d05eaa88fe005fd30409ac0b4da6a1f606b8aab81b774b18249c23e8c52a022013f8c3098faeeca01e6a4ac6141a1dda55b2d6536516ba283a908c030aa86da2012102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c00000000",
  "complete": true
}

$ ./bitcoin-cli sendrawtransaction "02000000000101f01587828faeed3d593b5486964ee8d51aad62859c1938e1e7965d52b351e1980000000000ffffffff01c0b6f394000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c0247304402207435d05eaa88fe005fd30409ac0b4da6a1f606b8aab81b774b18249c23e8c52a022013f8c3098faeeca01e6a4ac6141a1dda55b2d6536516ba283a908c030aa86da2012102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c00000000"
d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756

$ ./bitcoin-cli -rpcwallet="testing" generatetoaddress 1 bcrt1qt5wdyu84x2rl9egas8zj9k3xxngsfx8s6qgejr

Now we have out P2WSH transaction confirmed:

Code:

$ ./bitcoin-cli -rpcwallet="testing" gettransaction d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756
#snip...
    "blockhash": "3b3bd76667bc3b7ed3e689e76653ab231625f6a9cbbc8efbce5829faaee159b4",
#snip
$ ./bitcoin-cli getrawtransaction d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756 true 3b3bd76667bc3b7ed3e689e76653ab231625f6a9cbbc8efbce5829faaee159b4
#snip...
  "vout": [
    {
      "value": 24.99000000,
      "n": 0,
      "scriptPubKey": {
        "asm": "0 5067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c",
        "hex": "00205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c",
        "reqSigs": 1,
        "type": "witness_v0_scripthash",
        "addresses": [
          "bcrt1q2pnsyrafcgwtzvsssr5zca9pzdnr3zufvlm03kjs5tyu59raxekq55plqs"
        ]
      }
    }
#snip

So I try to spend from it:

Code:

$ ./bitcoin-tx -create in=d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756:0 outscript=24.98:"0x21 0x02b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182c OP_CHECKSIGVERIFY":W
02000000015637f6e035ffd7572e05248e4d328d8814eeffa457c5ecbb4175837d36a663d80000000000ffffffff018074e494000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c00000000

At this point I also serialise my script:

Code:

0x2102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182cad

And now trying to sign the spending transaction:

Code:

./bitcoin-cli signrawtransactionwithkey 02000000015637f6e035ffd7572e05248e4d328d8814eeffa457c5ecbb4175837d36a663d80000000000ffffffff018074e494000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c00000000 '["cTiHMJ3hwBb3Rtq5XzQB7569AJ7PfRsCfLr1anJnKWGMwYuB6L6S"]' '[{"txid": "d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756", "vout":0, "scriptPubKey":"00205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c","witnessScript":"2102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182cad", "amount":24.98}]'

{
  "hex": "020000000001015637f6e035ffd7572e05248e4d328d8814eeffa457c5ecbb4175837d36a663d80000000000ffffffff018074e494000000002200205067020fa9c21cb1321080e82c74a11366388b8967f6f8da50a2c9ca147d366c01232102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182cad00000000",
  "complete": false,
  "errors": [
    {
      "txid": "d863a6367d837541bbecc557a4ffee14888d324d8e24052e57d7ff35e0f63756",
      "vout": 0,
      "witness": [
        "2102b3f7623cf09a02088eb66737e2d3d308ea21a1d4fa105fc5a1d1ac775c63182cad"
      ],
      "scriptSig": "",
      "sequence": 4294967295,
      "error": "Unable to sign input, invalid stack size (possibly missing key)"
    }
  ]
}

Hey, at least the witnesscript matches, unlike yesterday.

[achow101]

So I tried to do it again and somehow got a different error this time:

Code:

"error": "Unable to sign input, invalid stack size (possibly missing key)"

Is this what you meant?

Yes, that is the error.

Could you clarify why it wouldn't be able to sign my transaction? Would it be because it's non-standard? I'm not sure how that would affect things. From reading the spec, in particular BIP141 and the Script definitions, my impression is that signing a transaction is a simple matter of serialising part of it and calculating the ECDSA signature, which shouldn't depend on the actual script being used (taken from https://en.bitcoin.it/wiki/BIP_0143). Then, as https://en.bitcoin.it/wiki/BIP_0141 defines it, constructing the witness script would simply involve concatenating the resulting signature with whatever is passed as the input to the command. Am I misunderstanding something here?

It is because the script is non-standard.

Generating the signature is easy, it's constructing the witness that is hard. While for this particular script constructing the witness is easy, the only way you know that is because you have analyzed the script and determined how to make the witness. For other scripts it's not so easy. As a script gets more complicated, you need to reason about what it does in order to figure out how to create the final witness. Since it is hard to do, Bitcoin Core just does template matching - it will only sign and make a witness for scripts that match some predetermined templates. The script that you have chosen does not match any of those templates.

I did give it a try using OP_CHECKSIG for a canonical pay-to-pubkey script, but bitcoin-tx recognised it as a pubkey script and created a P2WPK output, which is not what I want.

That's a bug and has been fixed for 0.21.

Does this mean that there's no way to use Bitcoin Core to sign P2WSH outputs, or am I misunderstanding what you said about OP_CHECKSIGVERIFY? If so, how would one even sign and use such a transaction?

Yes, there is no way to use Bitcoin Core to sign for arbitrary P2WSH and P2SH outputs.

There is a project which does allow for a general witness creator. It's called Miniscript and it is able to work on arbitrary scripts because it limits the scripts to certain opcodes and requires certain opcode ordering in order for it to be able to understand what a script does. Once this is implemented in Bitcoin Core, it will be possible to create and sign for arbitrary scripts so long as they conform to Miniscript. However we are still a ways out from this getting into Core.

I will also admit that my original attempt used a much more complex witnessScript, which would succeed either on a 2-of-2 multisig, or on a single sig after a checksequenceverify timelock expired. In my attempts to simplify it yesterday night, I must have botched something in the process. Now that I've gotten the witness script to at least match and the error is different, I'll give it another try with my original script, and if I get anything interesting I'll add it here.

This makes me believe I am making some mistake in the serialisation of the full script. Is there a canonical tool that I can use to "compile" Script?

You can try using btcdeb

[dkjfh]

Ah, I see. Thank you for the detailed reply!
I suppose "concatenate any signatures to the witness script" is just so common that I assumed it would be somewhat standard, but you're absolutely right, it definitely doesn't have to be.

If I wanted to press on, is there a way to generate a signature (and leave me to create the final witness manually), or at that point am I left with basically reimplementing it myself or hooking directly into the Bitcoin Core code? Since the "concatenate the signature with my script" is the only step that I'm missing for signing the thing automatically.

You can try using btcdeb

Thanks, I'll give that a try! Looks more promising than doing it by hand.

[achow101]

If I wanted to press on, is there a way to generate a signature (and leave me to create the final witness manually), or at that point am I left with basically reimplementing it myself or hooking directly into the Bitcoin Core code? Since the "concatenate the signature with my script" is the only step that I'm missing for signing the thing automatically.

With Bitcoin Core's tooling, there is no way. You will have to do it manually or find other tools to make the signature for you.

[dkjfh]

Alright, makes sense. Thanks again for your help! I sent a tip to express my gratitude. I was stuck for some time on these issues and you really helped me figure out what was going on and some details I was missing, so I'm very grateful.