Sigscript and hashing question

[smemo92]

Hi, I understand that in transaction input structure there is the SigScript that contains the signature of the transaction and the public key of who generates the transaction so that everyone can verify the signature. But I've also read that the signature is calculated from the transaction hash, but if the signature is a part of the transaction, how I can sign an hash that is generated from something that contains the signature itself. I imagine that there is some steps that misses in my argument.

[DannyHamilton]

See here:
https://en.bitcoin.it/w/images/en/7/70/Bitcoin_OpCheckSig_InDetail.png

[smemo92]

thank you. So the hash isn't calculated with the signature and the public key in SigScript field, but there is the Pkscript of the utxo it is related to. After calculating hash signature is inserted in the related field. So when someone receive the transaction that he have to verify, he create a new transaction copy that has in sigscript field the Pkscript part of utxo and then check the signature..
It is right?

[Peter R]

thank you. So the hash isn't calculated with the signature and the public key in SigScript field, but there is the Pkscript of the utxo it is related to. After calculating hash signature is inserted in the related field. So when someone receive the transaction that he have to verify, he create a new transaction copy that has in sigscript field the Pkscript part of utxo and then check the signature..
It is right?

I found this to be a useful reference when I was learning about byte-level details of bitcoin TXs: http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html

It sounds like you're on the right path.  Remember also that many TXs will contain multiple inputs.  Each input is signed individually.  Very roughly, if I wanted to sign input #2 on a TX that you passed to me (e.g., as part of a coinjoin TX), I would place the scriptPubKey (the rule for spending this output) for input #2 in the "script sig" slot, nil the script sigs for every other input, append the 4-byte hash code to the TX, and then hash the resulting string of bytes twice with SHA256 to get the 32-byte digest that needs to be signed.  Note that because of the way that only the input being signed has its scriptPubKey populated when hashing (the other inputs have empty script sigs), the 32-byte digest that actually gets signed is different for each input in the transaction.  

To continue, I would sign the resulting digest with my private key, DER encode this signature, append the sighash specifier byte, and append the pubkey.  This signature/pubkey pair then replaces the scriptPubKey in the "script sig" slot for that input.  If all the other inputs in this TX are signed (properly), then the network should accept the TX when I try to push it.  

The tables in the link I posted above should help you wrap your head around this.  

[smemo92]

thank you. So the hash isn't calculated with the signature and the public key in SigScript field, but there is the Pkscript of the utxo it is related to. After calculating hash signature is inserted in the related field. So when someone receive the transaction that he have to verify, he create a new transaction copy that has in sigscript field the Pkscript part of utxo and then check the signature..
It is right?

I found this to be a useful reference when I was learning about byte-level details of bitcoin TXs: http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html

It sounds like you're on the right path.  Remember also that many TXs will contain multiple inputs.  Each input is signed individually.  Very roughly, if I wanted to sign input #2 on a TX that you passed to me (e.g., as part of a coinjoin TX), I would place the scriptPubKey (the rule for spending this output) for input #2 in the "script sig" slot, nil the script sigs for every other input, append the 4-byte hash code to the TX, and then hash the resulting string of bytes twice with SHA256 to get the 32-byte digest that needs to be signed.  Note that because of the way that only the input being signed has its scriptPubKey populated when hashing (the other inputs have empty script sigs), the 32-byte digest that actually gets signed is different for each input in the transaction.  

To continue, I would sign the resulting digest with my private key, DER encode this signature, append the sighash specifier byte, and append the pubkey.  This signature/pubkey pair then replaces the scriptPubKey in the "script sig" slot for that input.  If all the other inputs in this TX are signed (properly), then the network should accept the TX when I try to push it.  

The tables in the link I posted above should help you wrap your head around this.  

thank you very much for your help, i'm doing a university work and so I need to learn about byte-level details. I've also read that every input is signed individually but this makes me confused about how previous output hash field works. This means that the hash in this field refers to an individual input hash and not to a transaction hash? I've used blockexplorer to see some blocks and transactions and I've understood that this field refers to a transaction hash and index field denotes which input is considered.

Another possibility is that to verify transaction and to sign it we calculate individual hash for inputs but then is calculated a general hash for the entire transaction so that we can refer to? In this way transaction's hash would be used only to refer that and not also to sign it.

I hope you can clarify this question.

[DannyHamilton]

The hash used in the "previous output hash field" is the "general hash for the entire transaction" of the previous transaction which created the output that is now being spent.  The index is then used to indicate which of the, potentially multiple, outputs from that transaction are being spent.

When signing the new transaction that you are creating, a separate signature is generated for each input in your new transaction using the appropriate private key associated with the address that received that previous output to prove that you are authorized to spend that output as an input. What is actually signed is a hash of a modified copy of your new transaction where all the inputs and outputs exist, but all the other signatures have been removed (as seen in some of the other links provided in this thread).

[smemo92]

Ok, thanks. So I imagine that the entire transaction hash is calculated with all inputs sigscript field filled with their own signature.
I have a last question, I've searched where utxo are stored but I've not found a clear answer. Are they stored in a database? O are they organized in a particular structure? I'm only quite sure they are indexed by their transaction hash but it's not clear how are organized and updated to the network after that some are spent.

[Peter R]

Ok, thanks. So I imagine that the entire transaction hash is calculated with all inputs sigscript field filled with their own signature.
I have a last question, I've searched where utxo are stored but I've not found a clear answer. Are they stored in a database? O are they organized in a particular structure? I'm only quite sure they are indexed by their transaction hash but it's not clear how are organized and updated to the network after that some are spent.

Haha.  This reminds me of how I used to be confused about "where is the Merkle Tree for each block stored?"  

The majority of nodes use the reference client (bitcoin core) and the UTXO set is stored for fast referencing in a local database.

But note that there is no specially-stored UTXO set at the protocol/blockchain level--all there is is the blockchain.  It's probably easiest to understand if you imagine bringing a new node online.  The node starts with the hard-coded genesis block, then parses Block 1, Block 2, … Block N.  Each block it updates its UTXO database based on the new information in the higher block.  So, no one tells it what the UTXO set is.  The node figures this out for itself based on the protocol rules and the information stored in the blockchain (that it downloads from peers or from a bootstrap service).  Furthermore, the UTXO set in your node may be slightly different than the UTXO set in my node because I may know about transactions or new blocks that you don't know about yet, and vice versa.  

 

[smemo92]

Ok thanks again
I take advantage of your kindness to ask you if I've understood how merkle tree/root works. I've understood how merkle root is calculated and that it is in the block header, that's easy. Then I read that if I have a SPV client, I only have block header and so only merkle root, so in order to verify if a transaction is included in a block I need some merkle branch ("some intermediate hash of the tree") in order to calculate merkle root and see if it corresponds to merkle root of the header. Is it right? I read that merkle branch are provided by full node, so SPV client needs to ask for these branch, but inside a block I see only a list of transactions with their own hash, so merkle branch (so "intermediate" hash of the tree) where are stored? Are they dinamically calculated by full node and sent to SPV? (I don't think so, it wouldn't be a good idea)

I'm sorry if this question is not related to topic's title

[Peter R]

Ok thanks again
I take advantage of your kindness to ask you if I've understood how merkle tree/root works. I've understood how merkle root is calculated and that it is in the block header, that's easy. Then I read that if I have a SPV client, I only have block header and so only merkle root, so in order to verify if a transaction is included in a block I need some merkle branch ("some intermediate hash of the tree") in order to calculate merkle root and see if it corresponds to merkle root of the header. Is it right?

Yes, that's correct.  Here's a TX with a fictitious Merkle branch.  The SPV client would calculate the root hash from the TX and the shown interior hashes, and compare that to Merkle root in the blockheader.

I read that merkle branch are provided by full node, so SPV client needs to ask for these branch, but inside a block I see only a list of transactions with their own hash, so merkle branch (so "intermediate" hash of the tree) where are stored? Are they dinamically calculated by full node and sent to SPV? (I don't think so, it wouldn't be a good idea)

I'm sorry if this question is not related to topic's title

SPV clients get the Merkle branch for a particular TX from full nodes. Here's some more info:



Full nodes need to construct the Merkle tree each time they verify a new block anyways.  The Merkle root they calculate for themselves must match the Merkle root in the blockheader in order for the block to be considered valid.  Although I've never checked, I would assume that, after constructing the tree,  the reference client stores it in a database so that it can efficiently respond to requests from SPV clients.  But I'm not sure what the protocol looks like for obtaining a Merkle branch from a full node.  I'd be interested to learn too.  

[smemo92]

Full nodes need to construct the Merkle tree each time they verify a new block anyways.  The Merkle root they calculate for themselves must match the Merkle root in the blockheader in order for the block to be considered valid.  Although I've never checked, I would assume that, after constructing the tree,  the reference client stores it in a database so that it can efficiently respond to requests from SPV clients.  But I'm not sure what the protocol looks like for obtaining a Merkle branch from a full node.  I'd be interested to learn too.  

Ok this is what I was searching. It is an idea but it seems a right one.