scriptPubKey not verified upon locking, but when trying to unlock it

[pgmforever]

I've been playing with multisig transactions today and I wrote a transaction that would lock the funds with the following scriptPubKey: (not using P2SH !)

<2> pubkey1 pubkey2 pubkey3 <3> OP_CHECKMULTISIG

When I wrote the actual hex for that, I used wrong opcodes for pushing to the stack, basically for pushing "2" I used a "0102" command, just like I saw pushing to the stack works with larger values encountered in signatures and pubkeys (0x47/48 for sigs, 0x21 (33 in decimal) for pubkeys, etc). I did not know there are special opcodes with meaning "push this exact value to the stack".

But guess what, my transaction was broadcast successfully...

Now when I tried to spend those funds, I kept getting: mandatory-script-verify-flag-failed (Data push larger than necessary) (code 16)

So I spent couple hours debugging the raw hex digit by digit trying to figure out what the hell was wrong. But of course there was nothing wrong with this spending hex, the problem was already irreversible (funds lost forever in the first transaction). I got to the bottom of it by printing the whole "trouble script" in bitcoind and recompiling, that's how I saw it has nothing to do with my spending scriptSig but with the previous scriptPubKey.

Now I have 2 questions about all this:

1) why it worked in the first place, doesn't the scriptPubKey go via the same validation rules as the scriptSig ?

2) why would bitcoind even care about the scriptPubKey of the UTXO i'm trying to spend, i thought it was supposed to just first run <scriptSig> then keep the stack and run <scriptPubKey> (which it may already consider valid...?)

[1714817476]

1714817476

[1714817476]

1714817476

[1714817476]

1714817476

[pebwindkraft]

are you on testnet or regtest? I'd be interested to see the example, I have from time to time similiar "experience"...
It is clear, that bitcoin demands for the shortest representation of data. So you have to use (from the wiki https://en.bitcoin.it/wiki/Script):

0x01-0x4b   (special) ... The next opcode bytes is data to be pushed onto the stack

for data, which has less than 75 OpCodes.
I'd just like to play with the examples on regtest ...

[pgmforever]

are you on testnet or regtest? I'd be interested to see the example, I have from time to time similiar "experience"...
It is clear, that bitcoin demands for the shortest representation of data. So you have to use (from the wiki https://en.bitcoin.it/wiki/Script):

0x01-0x4b   (special) ... The next opcode bytes is data to be pushed onto the stack

for data, which has less than 75 OpCodes.
I'd just like to play with the examples on regtest ...

I am on testnet, this is the transaction from which I cannot spend the #0 output: https://live.blockcypher.com/btc-testnet/tx/fb2c25f1dd9ebd0e9722eea47810210cd2da9e11326c7b41acd53824e2fb5473/

And yeah, I got it that bitcoin wants to represent data as shortest as possible, but when I wrote it i didn't know about the 0x52 onwards opcodes... so why it was accepted? :-s

[achow101]

1) why it worked in the first place, doesn't the scriptPubKey go via the same validation rules as the scriptSig ?

Because the scriptPubKey is only half of a script. It cannot be executed. If it were, it would always fail because a scriptPubKey is just part of a script. Same with the scriptSig; it is just part of a script. On their own, neither can be executed and get a successful result.

2) why would bitcoind even care about the scriptPubKey of the UTXO i'm trying to spend, i thought it was supposed to just first run <scriptSig> then keep the stack and run <scriptPubKey> (which it may already consider valid...?)

The scriptPubKey defines the spending conditions, i.e. what needs to go into the scriptSig. It has to run the scriptPubKey to ensure that the scriptSig is correct.

You can think of the scriptSig being the first half of a script and the scriptPubKey being the second half. On their own, they mean nothing and when executed on their own, they will fail. But concatenate them together, and you get a full script which can be executed. That is conceptually what script validation does. Before script validation, neither the scriptPubKey nor the scriptSig are considered valid.

The scriptPubKey in an output can be anything, it is not and cannot be validated when it is created. There have been multiple instances of people putting random garbage in scriptPubKeys. Those transactions are totally valid and in the blockchain. It is just that the outputs are not spendable because attempting to spend them would result in an invalid script.

[pgmforever]

That is hardly an explanation, we are talking about syntactic correctness here, not semantic.

A scriptPubKey can be preliminarily validated against this kind of mistakes that I described above, I was not expecting it to be run on its own because I know they only make sense concatenated together. However, errors like these can be caught up without actually running the script...

[achow101]

That is hardly an explanation, we are talking about syntactic correctness here, not semantic.

A scriptPubKey can be preliminarily validated against this kind of mistakes that I described above, I was not expecting it to be run on its own because I know they only make sense concatenated together. However, errors like these can be caught up without actually running the script...

It is up to the transaction creator to ensure that their script is constructed correctly. There are tools available that let you do this. It is not the Bitcoin network's job to be your debugger. Further, any script is syntactically correct, even if it is "wrong". There really is no syntax.

[amaclin1]

scriptPubKey not verified upon locking, but when trying to unlock it

- There are no checkings on testnet for standardness of scriptPubkey. Only on mainnet.
- I am sure that this output is spendable (if you know the private keys)

[pebwindkraft]

@amaclin: if the tests for standardness on testnet are not implemented, why is the error "mandatory-script-verify-flag-failed" happening? Am I missing s.th.?

Looking at TX_OUT[0] pk_script:
01022103341B4D862E7E1C3D7FE7D0ADE7E97DFAC122760B1EF4FFB21B7A9A27DD525B6C2102EE1 C79ACEE56F0E2CAFCC1069A6E1E1E9D059FF9A60B60904E97A455D5B27F672102839D639C109F09 0104EAB7C3EEF07D6BF25952A5A88C65BC5B185F0EA2A3FAF20103AE

    01: Push next byte as data onto stack
        02 (2-of-x multisig?)
    21: OP_Data33
        03341B4D862E7E1C:3D7FE7D0ADE7E97D
        FAC122760B1EF4FF:B21B7A9A27DD525B
        6C
    21: OP_Data33
        02EE1C79ACEE56F0:E2CAFCC1069A6E1E
        1E9D059FF9A60B60:904E97A455D5B27F
        67
    21: OP_Data33
        02839D639C109F09:0104EAB7C3EEF07D
        6BF25952A5A88C65:BC5B185F0EA2A3FA
        F2
    01: Push next byte as data onto stack
        03 (2-of-3 multisig)
    AE: OP_CHECKMULTISIG

pk1=mqBRdBj2aen8aM3t8vtuSdiQLh2kSDZZfc
pk2=mwyjowbdKK9QHK6nGHTS8HoNEsULxCbNce
pk3=mqxmeXoLf3ANX6fY1epLZEHcnNvr1rYPMA

So if pgmforever assembles his tx with two priv keys of the mentioned pubkeys (the sequence of sigs must be same as the pubkeys), it should be ok? I had the impression, it fails with the mentioned error. 

@pgmforever: maybe we can see the spending tx?
It should be good to go on regtest network as well...

[amaclin1]

@amaclin: if the tests for standardness on testnet are not implemented, why is the error
"mandatory-script-verify-flag-failed" happening?

Invalid signature?   Bug in code?
I am too lazy to check. And I have a right for mistake.
I think that consensus code permits such scripts for redeeming

pk1=mqBRdBj2aen8aM3t8vtuSdiQLh2kSDZZfc
pk2=mwyjowbdKK9QHK6nGHTS8HoNEsULxCbNce
pk3=mqxmeXoLf3ANX6fY1epLZEHcnNvr1rYPMA

These are addresses. One have to know private keys (2-of-3) to redeem

[amaclin1]

Okay, let me reproduce.

Keys:

Code:

  const MyKey32 priv1 ( MyKey32::brain ( "correct horse battery staple" ) );
  const MyKey32 priv2 ( MyKey32::brain ( "password" ) );
  const MyKey32 priv3 ( MyKey32::brain ( "a" ) );

  const MyByteArray pub1 ( priv1.getPublicKeyCompressed ( ) );
  const MyByteArray pub2 ( priv2.getPublicKeyCompressed ( ) );
  const MyByteArray pub3 ( priv3.getPublicKeyCompressed ( ) );

  const MyByteArray scriptPub ( MyByteArray ( )
                                  .putInt8 ( 1 ).putInt8 ( 2 )
                                  .putPush ( pub1 ).putPush ( pub2 ).putPush ( pub3 )
                                  .putInt8 ( 1 ).putInt8 ( 3 )
                                  .putInt8 ( OP_CHECKMULTISIG ) );

Funding:

Code:

23:40:07
sendrawtransaction 0100000000010101b68c67976fbc7e869aba1a4ff363c7e3278f2b449bf86692e4756395bef3b30000000017160014590d1a4921cf47406b2cbfb4985d264071ade7d9ffffffff01bedc0000000000006b0102210378d430274f8c5ec1321338151e9f27f4c676a008bdf8638d07c0b6be9ab35c712102b568858a407a8721923b89df9963d30013639ac690cce5f555529b77b83cbfc721034da006f958beba78ec54443df4a3f52237253f7ae8cbdb17dccf3feaa57f31260103ae02463043021f43dd03b3a8edd14f83fcd6012e8b2ba0ce3b32559a72e4df07fc940d6a8bf2022023fa8ff92c56cba9a1661e5bd420512f846b7de2edc1281ba4131868a451df6d0121032165dd9c0e73b7273254fe40f21a89ba9b993fc352413cd8419a71e377237d8c00000000

Result:
https://testnet.smartbit.com.au/tx/a44eef6028d395f7e9e7d1cf9a212c68ce715fa01744efdc3e7a4fd2906947d1

Spending:

Code:

23:40:10
sendrawtransaction 0100000001d1476990d24f7a3edcef4417a05f71ce682c219acfd1e7e9f795d32860ef4ea4000000008f0046304302201d3f0960f8897b67ee394d0f7c7467a62c9234b5c6025bedd0f29f540038cfe2021f0df7b647773323332864db02e1424db75453a4d1abf4d751a819625f4f3afa0146304302204897f6262220ff49488bac74b4861645ebf07acdaae09f03fbd1b66985c30a19021f3e0d8b758ffab1d0831ebb8e2ba7a30174bac7f04c0a471bdcc5b1819b296b01ffffffff014ad80000000000001976a91406599750ae954e00bc73d19a1f4d898659d6c01388ac00000000

Result:

23:40:10
64: non-mandatory-script-verify-flag (Data push larger than necessary) (code -26)


Solution: you should patch the client code and mine block yourself
NOTE: the error message is : non-mandatory... Do you see the word "NON"? Do you know what it means?

[pebwindkraft]

23:40:10
64: non-mandatory-script-verify-flag (Data push larger than necessary) (code -26)

Solution: you should patch the client code and mine block yourself
NOTE: the error message is : non-mandatory... Do you see the word "NON"? Do you know what it means?

you're the man! 

[pgmforever]

Hmm guys hold on i'm falling behind 

so why I was getting mandatory-flag failure while you got non-mandatory?

And how come verifications are done differently on testnet?

Tbh i can really give up the addresses so if you want i can post the private keys here. I thought about mining the block myself but other than that I guess there is not much that can be done since every other node that i may relay the tx to will reject it right?

[amaclin1]

Hmm guys hold on i'm falling behind 
so why I was getting mandatory-flag failure while you got non-mandatory?

May be wrong signature? Can you provide the spending transaction in hex?

And how come verifications are done differently on testnet?

Client it test mode allows much more flexibility than mainnet. For testing purposes. Surprise?

Tbh i can really give up the addresses so if you want i can post the private keys here.

I do not need them. One can try to spend my example above.

I thought about mining the block myself but other than that I guess there is not much that can
be done since every other node that i may relay the tx to will reject it right?

Wrong. Since transaction is in block the verifying process is different. The clients check only
the mandatory things.

[pgmforever]

May be wrong signature? Can you provide the spending transaction in hex?

unfortunately i don't have the old hex because as soon as i figured the problem i corrected it then made another pay-to-multisig which i spent successfully. but the error the client would give me is exactly the one i quoted, if there was a wrong signature it would say so, wouldnt it?

Client it test mode allows much more flexibility than mainnet. For testing purposes. Surprise?

Makes sense for greater flexibility yet i would expect the behaviour to be somewhat similar so that when changes are deployed to mainnet they won't behave differently. But yeah, I got your point.

I do not need them. One can try to spend my example above.

you're right i was able to spend it... -- https://live.blockcypher.com/btc-testnet/tx/b4c013e72bdb1f5503f67aeca697a3da2b21bf8c9cf0e90d74a97f98b6679a79/
i didn't check the hex, it is spending the utxo generated with the above code indeed? seems so weird because you basically used the same "wrong" opcodes :\

Wrong. Since transaction is in block the verifying process is different. The clients check only
the mandatory things.

You got my question wrong, i was saying that mining the block by myself should be the only option because otherwise relaying only the tx to other nodes will get it rejected

[amaclin1]

you're right i was able to spend it... -- https://live.blockcypher.com/btc-testnet/tx/b4c013e72bdb1f5503f67aeca697a3da2b21bf8c9cf0e90d74a97f98b6679a79/
i didn't check the hex, it is spending the utxo generated with the above code indeed? seems so weird because you basically used the same "wrong" opcodes :\

You haven't spent it   You have pushed transaction to block-explorer. Transaction is not confirmed.
blockcypher's engine have another policy for verifying transactions than Bitcoin Core client

You got my question wrong, i was saying that mining the block by myself should be the only option
because otherwise relaying only the tx to other nodes will get it rejected

Not nessesary.
Nodes have a right to relay non-standard transactions which do not pass non-mandatory rules.
It is a small chance to relay the non-standard transaction to a miner.

[amaclin1]

if there was a wrong signature it would say so, wouldnt it?

Easy to check! Let me change one bit in my original transaction and try to food the cleint!

13:10:10
sendrawtransaction 0100000001d1476990d24f7a3edcef4417a05f71ce682c219acfd1e7e9f795d32860ef4ea400000 0008f0046304302202d3f09...

13:10:10
16: mandatory-script-verify-flag-failed (Data push larger than necessary) (code -26)


So, you did at least two mistakes: (1) wrong opcodes were used (2) wrong signing  

but the error the client would give me is exactly the one i quoted

Handling errors - is not the best of Core Programmers. They are too smart to do errors. 

[pgmforever]

if there was a wrong signature it would say so, wouldnt it?

Easy to check! Let me change one bit in my original transaction and try to food the cleint!

13:10:10
sendrawtransaction 0100000001d1476990d24f7a3edcef4417a05f71ce682c219acfd1e7e9f795d32860ef4ea400000 0008f0046304302202d3f09...

13:10:10
16: mandatory-script-verify-flag-failed (Data push larger than necessary) (code -26)


So, you did at least two mistakes: (1) wrong opcodes were used (2) wrong signing  

but the error the client would give me is exactly the one i quoted

Handling errors - is not the best of Core Programmers. They are too smart to do errors. 

Haha this is really funny. I wasn't expecting it 

How do you sign a multisignature input? Have the same hash signed with different private keys right?

[amaclin1]

How do you sign a multisignature input? Have the same hash signed with different private keys right?

Yes. But remember - the scriptPubkey is used "as is" while creating hash (digest)
If you have 01-02-pub-pub-pub-01-03-chkmsig - you must use the same byte array.

And it is better to use p2sh-version for multisignature purposes.

[pgmforever]

I managed to spend from a raw multisig and since you mentioned it I tried p2sh version so if I have something like this for the scriptPubKey:

Code:

a9[hash160] + 14[push 20 bytes] + ripemd(sha256(52[push 2] + 21[push next 33 bytes] + pub1 + 21 + pub2 + 21 + pub3 + 53[push 3] + ae[checkmultisig])) + 87[op_equals] 

will the scriptSig be something like this? (assuming *** is whats inside ripemd(sha256(...)) above, also assuming all sigs are 71 bytes long)

Code:

00[push "0" for buggy checkmultisig] + 47[push next 71 bytes] + sig1 + 47 + sig2 + 4c[next byte is number of bytes to be pushed] + 69[105 decimal script size] + *** 

Looks good? having issues with it :-s

[amaclin1]

Looks good? having issues with it :-s

It would be better if you provide your testnet private keys, code, raw transactions, etc

[pgmforever]

Looks good? having issues with it :-s

It would be better if you provide your testnet private keys, code, raw transactions, etc

yeah, that sounds better indeed. will post tomorrow the hexes together with private keys and some code ive used to generate txes

[pgmforever]

So there was no need to post everything after all because I was able to track down the issue.

If anyone else hits this problem, the thing is that when spending p2sh-locked utxo, one needs to put into the input's scriptPubKey (when signing) the full redeem script, not the 23 bytes long (a9 + 14 + hash + 87) scriptPubKey that was used to lock that output in the first place (like it's the case with normal p2pkh transactions).

Anyways thanks a lot for all the help