Total confusion about Scriptpubkey, Standard and non standard Transactions

[Felicity_Tide]

So lately, I have been finding it difficult to understand the relationship between ScriptPubKey and standard transactions. When I try to relate the ideal meanings of both terms, there seem to be some similarities.

ScriptPubKey is a script that contains steps on how Bitcoin can be spent. In other words, we can say that it helps to control how one can spend Bitcoin. Standard transactions from my perspective are transactions that meet all the necessary criteria, including confirmation and validation.

There seem to be several(types) attributes tied to both of them, which are:
1. P2PK
2. P2PKH
3. P2SH
4. P2MS
5. P2TR
6. P2WKH
7. P2WSH

I am quite familiar with few of the types listed above, such as P2PK, which means paying(pay) to public key. P2PKH, which means paying(pay) to public key hash(same as paying to an address. P2SH, which means paying(pay) to script hash. P2WKH, which means paying(pay) to witness key hash, and P2WSH, which means paying(pay) to witness script hash.

Aside from that, I also observed that there is a phrase "non standard transactions", which sounds like the opposite of "standard transactions". I tried to find a straightforward article to give me a better explanation of its meaning which I couldn't get, but had to reason from an ideal perspective which is: non standard transactions are transactions that might not meet the necessary criteria.

There are speculations that such transactions end up in the block, which I don't understand because the phrase says "non standard".

My questions:
1. Is there any proper definition for ScriptPubKey, standard and non standard transactions?.
2. Do ScriptPubKey and standard transactions mean the same thing?.
3. How do P2TR and P2MS work?.
4. Is it possible for a transaction to be non standard and still be valid?.
5. Can you explain the last paragraph above?.

This thread is basically to seek answers to my questions and confusion, which means, only the reply section is likely to be %100 accuracy.

I am 100% open to correction as I still see myself as a learner. Pardon any of my error and share your personal opinion. You might want to also DOYR after reading this.


Mod, please move the thread if it doesn't fit or contain enough technical matters

[vjudeu]

Is there any proper definition for ScriptPubKey, standard and non standard transactions?

https://github.com/bitcoin/bitcoin/blob/master/src/policy/policy.cpp#L70

Code:

bool IsStandard(const CScript& scriptPubKey, const std::optional<unsigned>& max_datacarrier_bytes, TxoutType& whichType)

Do ScriptPubKey and standard transactions mean the same thing?

No. When you have "scriptPubKey", then it is just a field, expressed as "const CScript& scriptPubKey", which contains raw bytes of the Script. But "IsStandard()" is a function, taking "scriptPubKey" as one of arguments, and it can return "bool" saying "true" if a transaction is standard, or "false" if it is not.

How do P2TR and P2MS work?

https://github.com/bitcoin/bitcoin/blob/master/src/script/solver.cpp#L140

Code:

TxoutType Solver(const CScript& scriptPubKey, std::vector<std::vector<unsigned char>>& vSolutionsRet)

Is it possible for a transaction to be non standard and still be valid?

Of course. Non-standard and invalid are two different things.

Can you explain the last paragraph above?

Non-standard transaction, which is valid: https://mempool.space/testnet4/tx/09096e1e6fb31f33f3de40b9a1d76908e565e9646e5aed4e0813e0a2a799fc4c

[Findingnemo]

Aside from that, I also observed that there is a phrase "non standard transactions", which sounds like the opposite of "standard transactions". I tried to find a straightforward article to give me a better explanation of its meaning which I couldn't get, but had to reason from an ideal perspective which is: non standard transactions are transactions that might not meet the necessary criteria.

5. Can you explain the last paragraph above?.

Non standard transaction is something that doesn't follow Bitcoin's protocol rules which are

Code:

Check syntactic correctness
Make sure neither in or out lists are empty
Size in bytes <= MAX_BLOCK_SIZE
Each output value, as well as the total, must be in legal money range
Make sure none of the inputs have hash=0, n=-1 (coinbase transactions)
Check that nLockTime <= INT_MAX[1], size in bytes >= 100[2], and sig opcount <= 2[3]
Reject "nonstandard" transactions: scriptSig doing anything other than pushing numbers on the stack, or scriptPubkey not matching the two usual forms[4]
Reject if we already have matching tx in the pool, or in a block in the main branch
For each input, if the referenced output exists in any other tx in the pool, reject this transaction.[5]
For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction is missing for any input, this will be an orphan transaction. Add to the orphan transactions, if a matching transaction is not in there already.
For each input, if the referenced output transaction is coinbase (i.e. only 1 input, with hash=0, n=-1), it must have at least COINBASE_MATURITY (100) confirmations; else reject this transaction
For each input, if the referenced output does not exist (e.g. never existed or has already been spent), reject this transaction[6]
Using the referenced output transactions to get input values, check that each input value, as well as the sum, are in legal money range
Reject if the sum of input values < sum of output values
Reject if transaction fee (defined as sum of input values minus sum of output values) would be too low to get into an empty block
Verify the scriptPubKey accepts for each input; reject if any are bad
Add to transaction pool[7]
"Add to wallet if mine"
Relay transaction to peers
For each orphan transaction that uses this one as one of its inputs, run all these steps (including this one) recursively on that orphan

Any transaction that doesn't meet these criteria can be classified as a non-standard transaction and it was supposed to be rejected by the nodes and so the miners but not necessary, there are examples that nonstandard transactions were mined.

Recently there was one pool that accepted a non-standard transaction and was successfully mined but it came with a bounty.
Here's the user's/pool's post: https://bitcointalk.org/index.php?topic=5192454.msg55842508#msg55842508

As the name suggests it's highly rare though.

[ABCbits]

Aside from that, I also observed that there is a phrase "non standard transactions", which sounds like the opposite of "standard transactions". I tried to find a straightforward article to give me a better explanation of its meaning which I couldn't get, but had to reason from an ideal perspective which is: non standard transactions are transactions that might not meet the necessary criteria.

5. Can you explain the last paragraph above?.

Non standard transaction is something that doesn't follow Bitcoin's protocol rules which are

Code:

Check syntactic correctness
Make sure neither in or out lists are empty
Size in bytes <= MAX_BLOCK_SIZE
Each output value, as well as the total, must be in legal money range
Make sure none of the inputs have hash=0, n=-1 (coinbase transactions)
Check that nLockTime <= INT_MAX[1], size in bytes >= 100[2], and sig opcount <= 2[3]
Reject "nonstandard" transactions: scriptSig doing anything other than pushing numbers on the stack, or scriptPubkey not matching the two usual forms[4]
Reject if we already have matching tx in the pool, or in a block in the main branch
For each input, if the referenced output exists in any other tx in the pool, reject this transaction.[5]
For each input, look in the main branch and the transaction pool to find the referenced output transaction. If the output transaction is missing for any input, this will be an orphan transaction. Add to the orphan transactions, if a matching transaction is not in there already.
For each input, if the referenced output transaction is coinbase (i.e. only 1 input, with hash=0, n=-1), it must have at least COINBASE_MATURITY (100) confirmations; else reject this transaction
For each input, if the referenced output does not exist (e.g. never existed or has already been spent), reject this transaction[6]
Using the referenced output transactions to get input values, check that each input value, as well as the sum, are in legal money range
Reject if the sum of input values < sum of output values
Reject if transaction fee (defined as sum of input values minus sum of output values) would be too low to get into an empty block
Verify the scriptPubKey accepts for each input; reject if any are bad
Add to transaction pool[7]
"Add to wallet if mine"
Relay transaction to peers
For each orphan transaction that uses this one as one of its inputs, run all these steps (including this one) recursively on that orphan

Do you write it yourself or find it elsewhere? While i don't remember all rules, i recall transaction considered as non-standard if the size is higher than 100 vKB or 400000 weight unit.

[vjudeu]

Do you write it yourself or find it elsewhere?

Let's see: https://en.bitcoin.it/wiki/Protocol_rules#%22tx%22_messages

[Felicity_Tide]

https://github.com/bitcoin/bitcoin/blob/master/src/policy/policy.cpp#L70

Code:

bool IsStandard(const CScript& scriptPubKey, const std::optional<unsigned>& max_datacarrier_bytes, TxoutType& whichType)

No. When you have "scriptPubKey", then it is just a field, expressed as "const CScript& scriptPubKey", which contains raw bytes of the Script. But "IsStandard()" is a function, taking "scriptPubKey" as one of arguments, and it can return "bool" saying "true" if a transaction is standard, or "false" if it is not.

NOTED

Code:

if (whichType == TxoutType::NONSTANDARD) {
        return false;
    } else if (whichType == TxoutType::MULTISIG) {
        ... ... ...
    } else if (whichType == TxoutType::NULL_DATA) {
        if (!max_datacarrier_bytes || scriptPubKey.size() > *max_datacarrier_bytes) {
            return false;
}

With the basic knowledge I have in C++, I was able to remember some syntax here, so reading wasn't that difficult. But the last line of code Is a little bit confusing

Code:

else if (whichType == TxoutType::NULL_DATA

Does the NULL_DATA represents txoutput(data) with zero value entry?
.

How do P2TR and P2MS work?

https://github.com/bitcoin/bitcoin/blob/master/src/script/solver.cpp#L140

Code:

TxoutType Solver(const CScript& scriptPubKey, std::vector<std::vector<unsigned char>>& vSolutionsRet)

This part was a little bit complex to understand, but with list of switch statements written above the code, I will assume that they were all threated in the main code.

Code:

std::string GetTxnOutputType(TxoutType t)
{
    switch (t) {
    case TxoutType::NONSTANDARD: return "nonstandard";
    case TxoutType::PUBKEY: return "pubkey";
}

Non standard transaction is something that doesn't follow Bitcoin's protocol rules which are

Code:

~

Any transaction that doesn't meet these criteria can be classified as a non-standard transaction and it was supposed to be rejected by the nodes and so the miners but not necessary, there are examples that nonstandard transactions were mined.

As the name suggests it's highly rare though.

Noted. Have just bookmarked the rules for future referencing.

While i don't remember all rules, i recall transaction considered as non-standard if the size is higher than 100 vKB or 400000 weight unit.

Noted. Should probably add this to the rules also.

[nc50lc]

Non standard transaction is something that doesn't follow Bitcoin's protocol rules which are
-snip-
As the name suggests it's highly rare though.

The term "non-standard" do not apply to transactions that have broken the Bitcoin "protocol".
Those that broken the protocol have no chance to be included to a block because if ever included by a miner, the block itself will be invalid.

The provided example (the quote of my post) is something that didn't comply with the majority of the nodes' "policy" instead.
So the miner that included the transaction to his block still managed to add it to the blockchain since it is still valid in the protocol.

The policy related to that "Uncompressed SegWit" incident is in (SCRIPT_VERIFY_WITNESS_PUBKEYTYPE):

• github.com/bitcoin/bitcoin/blob/5ee6b76c69d51158c13f6ad9ea1b264372e58d4d/src/policy/policy.h#L112
• github.com/bitcoin/bitcoin/blob/5ee6b76c69d51158c13f6ad9ea1b264372e58d4d/src/script/interpreter.h#L124-L126

[vjudeu]

Does the NULL_DATA represents txoutput(data) with zero value entry?

https://github.com/bitcoin/bitcoin/blob/master/src/script/solver.h#L22

Code:

enum class TxoutType {
    NONSTANDARD,
    // 'standard' transaction types:
    PUBKEY,
    PUBKEYHASH,
    SCRIPTHASH,
    MULTISIG,
    NULL_DATA, //!< unspendable OP_RETURN script that carries data
    WITNESS_V0_SCRIPTHASH,
    WITNESS_V0_KEYHASH,
    WITNESS_V1_TAPROOT,
    WITNESS_UNKNOWN, //!< Only for Witness versions not already defined above
};

This part was a little bit complex to understand

https://github.com/bitcoin/bitcoin/blob/master/src/script/solver.cpp#L164

Code:

        if (witnessversion == 1 && witnessprogram.size() == WITNESS_V1_TAPROOT_SIZE) {
            vSolutionsRet.push_back(std::move(witnessprogram));
            return TxoutType::WITNESS_V1_TAPROOT;
        }

This is Taproot, also known as P2TR.

https://github.com/bitcoin/bitcoin/blob/master/src/script/solver.cpp#L196

Code:

    int required;
    std::vector<std::vector<unsigned char>> keys;
    if (MatchMultisig(scriptPubKey, required, keys)) {
        vSolutionsRet.push_back({static_cast<unsigned char>(required)}); // safe as required is in range 1..20
        vSolutionsRet.insert(vSolutionsRet.end(), keys.begin(), keys.end());
        vSolutionsRet.push_back({static_cast<unsigned char>(keys.size())}); // safe as size is in range 1..20
        return TxoutType::MULTISIG;
    }

And this is bare multisig, also known as P2MS.

[pooya87]

A non-standard transaction is a valid transaction that we prefer not to have included in a block. It can have different reasons as well.

For example it is perfectly valid to create a P2WPKH output using an uncompressed public key but it is considered non-standard and rejected. The reason is because uncompressed pubkeys are bigger and using them would be inefficient usage of block space. So they are "discouraged" by being counted as non-standard.
They weren't made invalid because after all an uncompressed public key is still a valid public key.

Another example would be the limits on OP_RETURN size which is to prevent abuse so that people don't turn bitcoin blockchain into cloud storage. So the size of such outputs are limited to 80 bytes.

Another example is limits on other fields such as transaction version (ver >3 or <1 are non-standard) that is to preserve them for possible future usage. It is easier to just add the (if ver == x) line in the code if there has never been any tx version = x than it is to add exception cases that skip the previous txs with version = x.

[NotATether]

The scriptpubkey field contains what you might consider to be a scripting language, and the opcodes that are inside take one parameter, the "stack", which is either in the scriptsig or in the witness data depending on whether the transaction format is segwit or legacy. The stack has a bunch of values on it which the code in the scriptpubkey "pops" (in stack nomenclature, removing the top value) and it can also "push" (add values to the top) its own data to it.

The purpose of the scriptpubkey is to keep the bitcoins in place until someone unlocks them by putting a stack that makes the script return a "1" (just a single 1 on the stack) inside a transaction - along with the UTXO being spent, of course. If it returns anything else then it is considered a failure and the transaction is invalid.

[Felicity_Tide]

A non-standard transaction is a valid transaction that we prefer not to have included in a block. It can have different reasons as well.

For example it is perfectly valid to create a P2WPKH output using an uncompressed public key but it is considered non-standard and rejected. The reason is because uncompressed pubkeys are bigger and using them would be inefficient usage of block space. So they are "discouraged" by being counted as non-standard.
They weren't made invalid because after all an uncompressed public key is still a valid public key.

The choice of not including a non-standard transaction in the block, was it a general community agreement?. Because I read an article which am not too sure of the authenticity, that says : Non-standard transactions are submitted to a miner directly rather than dropping them in the mempool waiting for approval. The article specified that this transactions are hard to get approval especially when they are seated at the mempool. I am not really sure of how certain transactions are given to miners directly because it sounds strange to me.
 Please clarify me on this, so I don't learn something wrong.

The scriptpubkey field contains what you might consider to be a scripting language, and the opcodes that are inside take one parameter, the "stack", which is either in the scriptsig or in the witness data depending on whether the transaction format is segwit or legacy. The stack has a bunch of values on it which the code in the scriptpubkey "pops" (in stack nomenclature, removing the top value) and it can also "push" (add values to the top) its own data to it.

The purpose of the scriptpubkey is to keep the bitcoins in place until someone unlocks them by putting a stack that makes the script return a "1" (just a single 1 on the stack) inside a transaction - along with the UTXO being spent, of course. If it returns anything else then it is considered a failure and the transaction is invalid.

Noted
So, for more clarity, does this mean that the ScriptPubKey acts as a locking script, while the other two (ScriptSig and ScriptWitness) act as unlocking scripts for SegWit and Legacy individually?.

[pooya87]

The choice of not including a non-standard transaction in the block, was it a general community agreement?. Because I read an article which am not too sure of the authenticity, that says : Non-standard transactions are submitted to a miner directly rather than dropping them in the mempool waiting for approval. The article specified that this transactions are hard to get approval especially when they are seated at the mempool. I am not really sure of how certain transactions are given to miners directly because it sounds strange to me.
 Please clarify me on this, so I don't learn something wrong.

When you want to spend coins, you create a transaction then broadcast it to the network by sending it to a handful other full nodes. The Bitcoin Network is a web of these interconnected nodes. Your transaction would propagate throughout the network as each node relays it to other nodes, eventually reaching majority of node including the nodes used by miners. Then the miner can select some of transactions from its own node and mine the block, hence confirming your tx.

Each full node has its own mempool that stores the valid transactions they receive. They criteria to enter their mempool is consensus rules and they can also choose to enforce standard rules. Since the majority of full nodes are running bitcoin core, majority of the network is enforcing all standard rules by default.

This means when you broadcast a non-standard transaction, majority of full nodes reject it so it won't propagate and the chances of it reaching a miner is small.

This is why the only way to get your non-standard tx confirmed is to first find a miner that accepts non-standard transactions (miners run the same software and usually reject such txs as well) and then directly send your tx to their node.