Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: etotheipi on July 16, 2011, 02:48:41 PM



Title: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 16, 2011, 02:48:41 PM
As part of the effort I invested to really understand the BTC protocol, I made some SVG images to help me piece everything together.  I got most of the difficult details I needed from the BTC forums, so here is me giving back :)

The first image is a breakdown of an entire transaction.  I could not put the full-size image into this post, because it had to be saved at 800 DPI in order to capture all the small text describing the scripts.  Click on it for the full-sized image.

https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/TxBinaryMapSmall.png (https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/TxBinaryMap.png)

I have a couple more images coming, showing the process for OP_CHECKSIG, but they need to be cleaned up before I can post them here.  Please let me know if you find any errors so I can fix them.  

Enjoy!
-Eto

P.S. - I would like the full-sized images to be stored somewhere permanently on the forums instead of in my dropbox folder, but I can't put them in a post without flooding your browser screen.  Any recommendations?


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: wumpus on July 16, 2011, 03:03:15 PM
Cool! Put them on the wiki!


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: etotheipi on July 16, 2011, 07:21:43 PM
Alright!  After spending a dozen hours figuring out the OP_CHECKSIG procedure, and working out all the endianness issues and other details in my code, I decided no one else should have to go through this as confused as I was!  So, I hereby present the illustrated guide to OP_CHECKSIG (with SIGHASH_ALL only).   Again, click on it for the full-detail image.

https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/OpCheckSigDiagramSmall.png (https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/OpCheckSigDiagram.png)

Please let me know if you spot any errors.  I would like to make this diagram as accurate as possible.

Btw, this was a lot of work!  So, if you get some benefit out of this diagram, please consider donating 0.1 BTC to me!  My address is in my signature. (and I would love to show my girlfriend that it's possible to get compensated for all the time I spend on the computer! :))

Enjoy!
-Eto


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: vector76 on July 16, 2011, 08:32:58 PM
Well done!  0.1 BTC sent


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: patvarilly on July 16, 2011, 10:56:32 PM
Nice work!  The only glitch I noticed is that the value field in a TxOut is 64 *bits* long, i.e. 8 bytes.  Most other field lengths seem to be in bytes.

I hadn't delved into the details of OP_CHECKSIG before, and hadn't realized it was this complicated.  Thanks for sorting this all out for us!


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: etotheipi on July 16, 2011, 11:09:50 PM
Ack, nice catch!  I have updated the image in Dropbox.   Maybe I'll keep it in Dropbox for a while, until I get a little more feedback like this...

I had no idea OP_CHECKSIG was so complicated either... until I started trying to figure out what I needed to hash to verify the signature.  That was like 2-3 days ago, and only now did I get it!

-Eto



Title: Re: On-the-wire byte map (knowledge donation!)
Post by: WakiMiko on July 19, 2011, 12:20:30 AM
thanks a lot for those!


Title: Re: On-the-wire byte map (knowledge donation!)
Post by: MountainMan on July 19, 2011, 08:00:53 AM
Thank you sir. Documenting that is a big plus!


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 20, 2011, 09:15:39 PM
First of all, here is a link to the raw .svg file I am using:  

https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/BTC_Illustrations.svg (https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/BTC_Illustrations.svg)

This is typically how I prepare images for presentations I make at work.  I use Inkscape to create a bunch of diagrams, then select each one and "File-->Export Bitmap".  With this file out there, it should be available for others to download, modify and re-post somewhere else if I get hit by a bus tomorrow.   By the way, Inkscape is amazing.  Once you learn some of the hotkeys, alignments, grouping, fill&stroke, etc, it's very pleasant and efficient to use.  The learning curve isn't terribly steep, and they have interactive tutorials which are actually Inkscape canvases, with inkscape objects for you to play with.

Second, here's another diagram.  It's not nearly as complicated as OP_CHECKSIG, but once I had it mapped out like this, I was able to do a ton of address magic in my code and get it right on the first try.  Not having to guess endianness at every step is a big help!  Remembering the extra bytes that have to be added here and there helps, too.

https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/PubKeyToAddrSmall.png

The full size image is here:  https://dl.dropboxusercontent.com/u/1139081/BitcoinImg/PubKeyToAddrSmall.png


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: samr7 on July 20, 2011, 10:27:39 PM

This is good.  But I'll call you on the length of the pre-base58 address: it's 25 bytes long, not 24, but the upper byte is always 0 for regular bitcoin addresses.  Also, the base58 encoding procedure will prepend a 1 for each preceding \x00 byte in the pre-encoded address, not just a single 1.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 20, 2011, 10:36:00 PM
Ah hah.  Little details that I missed...

Okay.  So I need to add a '\x00' byte to the 24-byte binary address to make it 25-byte... and it never mattered because it's big endian so the integer calculation works out the same...?  So when would this byte not be '\x00'?  And is this the same \x00 that you say is responsible for the '1' at the beginning of the address?  There could be multiple '\x00' bytes? 

Damn these small details!   I'll update the diagrams once I understand it, myself.

-Eto



Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: samr7 on July 20, 2011, 11:09:52 PM
Okay.  So I need to add a '\x00' byte to the 24-byte binary address to make it 25-byte... and it never mattered because it's big endian so the integer calculation works out the same...?  So when would this byte not be '\x00'?

That's right!

The only interesting cases for the first byte being nonzero are testnet addresses and namecoin addresses.

Quote
And is this the same \x00 that you say is responsible for the '1' at the beginning of the address?  There could be multiple '\x00' bytes? 

Yes to both!  And, as you mentioned, preceding zeros are ignored by the binary-to-base58-string conversion step.  Without this rule, it wouldn't be possible to have an address that started with 11 or 111, and it would be possible to have an address shorter than 25 characters (which is impossible in the current scheme).


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 21, 2011, 12:29:35 AM
How about if I just change the title to say "Bitcoin Main-Network Address Operations"?  I don't feel like all the extra detail is justified, it'll just muck up the diagram to accomodate cases that most people will never encounter.

P.S. - I just updated the image to include one \x00 byte, and changed the title as I just described.  Let me know what you think.



Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: genjix on July 21, 2011, 10:02:25 AM
Hey thanks.

Definitely going to use this in my project: http://forum.bitcoin.org/index.php?topic=30646
(check it out ;)

Maybe you have some code samples to share?


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: genjix on July 21, 2011, 01:44:11 PM
You should consider giving us the sources to these pics too. I'm going to be going through all the OP_CHECKSIG code soon with a fine comb and I'll probably have corrections to add to your images.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 21, 2011, 01:49:15 PM
Five or six posts up I included the .svg file that contains all the images so far.  It's kind of a mess, but it's all there. 

And please let me know if you find any problems with it, I'll update it myself. 
-Eto


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: prof7bit on July 26, 2011, 12:12:21 PM
This is extremely valuable information! I will need it once I come to the point of verifying transactions myself. The OP_CHECKSIG mechanism is the single one thing about the entire protocol that raises the most and the biggest question marks when reading the protocol specification. Compared to that the rest of the protocol is a child's play.

Having this independently documented by different people (who all have their own style of visualizing and explaining the same thing) is a great help. I believe especially this topic (OP_CHECKSIG & friends) deserves more explanaition and this thread and the excellent diagrams are a very valuable contribution.

I hope more of those people wo have already completely understood it will attempt to document and pass on their knowledge and maybe also try to explain in their own words not only what the code does but also why certain steps are needed or done exactly the way they are done or find even better ways to visualize what is going on. I personally will certainly try to do this once I have a working implementation of OP_CHECKSIG (I'm still busy with boring implementtion of network protocol and persistance layer but at one point I will need to start working on transaction verification).


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 26, 2011, 10:58:06 PM
I have produced a few hour-long seminars at work, with the goal of adding extremely detailed visualizations to describe the mechanisms of cryptography.  So, for me, these visualizations were quite easy as I have become extremely efficient with Inkscape.   The hard part was understanding the algorithms to begin with!  Once I finally got my OP_CHECKSIG code working, I knew that I could leverage my Inkscape skills to benefit others (and myself, as these images are great for my own reference).

If you feel that there are more algorithms/concepts that could benefit from such visualization, I will be happy to produce them, as long as I understand it.  Perhaps others could help by injecting these images into the wikis at appropriate places.  Of course, if you believe there is an error in one of them, please let me know.

On that note, I would like to make a few variants of the OP_CHECKSIG diagram, one for each hashtype.  Unfortunately, I don't fully understand them myself.  I need SIGHASH_NONE, SIGHASH_SINGLE, SIGHASH_ANYONECANPAY.  I have read the descriptions, but still don't understand them well enough to complete the visualization.  Perhaps someone that does understand can explain them in the context of the my diagram for SIGHASH_ALL.

-Eto


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: genjix on July 27, 2011, 03:54:07 AM
i wouldnt worry about them yet. they arent used anywhere in the blockchain and they are for future transaction types. they control which parents of the tx arent hashed.

together with them you can make replaceable transactions using sequence numbers for example.

Anyway if anyone wants some raw code to do OP_CHECKSIG then libbitcoin has a unit test under tests/ec-key.cpp (make ec-key && ./bin/tests/ec-key)... Will upload tomorrow togther with my script system's working OP_CHECKSIG once I've cleaned it all up internally:

Code:
#include <iostream>
#include <iomanip>
#include <bitcoin/util/serializer.hpp>
#include <bitcoin/util/elliptic_curve_key.hpp>
#include <bitcoin/util/sha256.hpp>
#include <bitcoin/util/assert.hpp>
#include <bitcoin/util/logger.hpp>
#include <bitcoin/types.hpp>
#include <openssl/ecdsa.h>
#include <openssl/obj_mac.h>
using libbitcoin::elliptic_curve_key;
using libbitcoin::serializer;
using libbitcoin::hash_digest;
using libbitcoin::data_chunk;
using libbitcoin::log_info;
using libbitcoin::log_fatal;

int main()
{
    serializer ss;
    // blk number 170, tx 1, input 0
    // version = 1
    ss.write_4_bytes(1);
    // 1 inputs
    ss.write_var_uint(1);

    // input 0
    // prevout hash
    ss.write_hash(hash_digest{0x04, 0x37, 0xcd, 0x7f, 0x85, 0x25, 0xce, 0xed, 0x23, 0x24, 0x35, 0x9c, 0x2d, 0x0b, 0xa2, 0x60, 0x06, 0xd9, 0x2d, 0x85, 0x6a, 0x9c, 0x20, 0xfa, 0x02, 0x41, 0x10, 0x6e, 0xe5, 0xa5, 0x97, 0xc9});
    // prevout index
    ss.write_4_bytes(0);

    // input script after running OP_CHECKSIG for this tx is a single
    // OP_CHECKSIG opcode
    data_chunk raw_data;
    raw_data = {0x04, 0x11, 0xdb, 0x93, 0xe1, 0xdc, 0xdb, 0x8a, 0x01, 0x6b, 0x49, 0x84, 0x0f, 0x8c, 0x53, 0xbc, 0x1e, 0xb6, 0x8a, 0x38, 0x2e, 0x97, 0xb1, 0x48, 0x2e, 0xca, 0xd7, 0xb1, 0x48, 0xa6, 0x90, 0x9a, 0x5c, 0xb2, 0xe0, 0xea, 0xdd, 0xfb, 0x84, 0xcc, 0xf9, 0x74, 0x44, 0x64, 0xf8, 0x2e, 0x16, 0x0b, 0xfa, 0x9b, 0x8b, 0x64, 0xf9, 0xd4, 0xc0, 0x3f, 0x99, 0x9b, 0x86, 0x43, 0xf6, 0x56, 0xb4, 0x12, 0xa3};
    data_chunk raw_script;
    raw_script = data_chunk();
    raw_script.push_back(raw_data.size());
    libbitcoin::extend_data(raw_script, raw_data);
    raw_script.push_back(172);
    ss.write_var_uint(raw_script.size());
    ss.write_data(raw_script);
    // sequence
    ss.write_4_bytes(0xffffffff);

    // 2 outputs for this tx
    ss.write_var_uint(2);

    // output 0
    ss.write_8_bytes(1000000000);
    // script for output 0
    raw_data = {0x04, 0xae, 0x1a, 0x62, 0xfe, 0x09, 0xc5, 0xf5, 0x1b, 0x13, 0x90, 0x5f, 0x07, 0xf0, 0x6b, 0x99, 0xa2, 0xf7, 0x15, 0x9b, 0x22, 0x25, 0xf3, 0x74, 0xcd, 0x37, 0x8d, 0x71, 0x30, 0x2f, 0xa2, 0x84, 0x14, 0xe7, 0xaa, 0xb3, 0x73, 0x97, 0xf5, 0x54, 0xa7, 0xdf, 0x5f, 0x14, 0x2c, 0x21, 0xc1, 0xb7, 0x30, 0x3b, 0x8a, 0x06, 0x26, 0xf1, 0xba, 0xde, 0xd5, 0xc7, 0x2a, 0x70, 0x4f, 0x7e, 0x6c, 0xd8, 0x4c};
    // when data < 75, we can just write it's length as a single byte ('special'
    // opcodes)
    raw_script = data_chunk();
    raw_script.push_back(raw_data.size());
    libbitcoin::extend_data(raw_script, raw_data);
    // OP_CHECKSIG
    raw_script.push_back(172);
    // now actually write the script
    ss.write_var_uint(raw_script.size());
    ss.write_data(raw_script);

    // output 0
    ss.write_8_bytes(4000000000);
    // script for output 0
    raw_data = {0x04, 0x11, 0xdb, 0x93, 0xe1, 0xdc, 0xdb, 0x8a, 0x01, 0x6b, 0x49, 0x84, 0x0f, 0x8c, 0x53, 0xbc, 0x1e, 0xb6, 0x8a, 0x38, 0x2e, 0x97, 0xb1, 0x48, 0x2e, 0xca, 0xd7, 0xb1, 0x48, 0xa6, 0x90, 0x9a, 0x5c, 0xb2, 0xe0, 0xea, 0xdd, 0xfb, 0x84, 0xcc, 0xf9, 0x74, 0x44, 0x64, 0xf8, 0x2e, 0x16, 0x0b, 0xfa, 0x9b, 0x8b, 0x64, 0xf9, 0xd4, 0xc0, 0x3f, 0x99, 0x9b, 0x86, 0x43, 0xf6, 0x56, 0xb4, 0x12, 0xa3};
    // when data < 75, we can just write it's length as a single byte ('special'
    raw_script.push_back(raw_data.size());
    libbitcoin::extend_data(raw_script, raw_data);
    // OP_CHECKSIG
    raw_script.push_back(172);
    // now actually write the script
    ss.write_var_uint(raw_script.size());
    ss.write_data(raw_script);

    // End of 2 outputs

    // locktime
    ss.write_4_bytes(0);

    // write hash_type_code
    ss.write_4_bytes(1);

    // Dump hex to screen
    log_info() << "hashing:";
    {
        auto log_obj = log_info();
        log_obj << std::hex;
        for (int val: ss.get_data())
            log_obj << std::setfill('0') << std::setw(2) << val << ' ';
    }
    log_info();

    data_chunk raw_tx = {0x01, 0x00, 0x00, 0x00, 0x01, 0xc9, 0x97, 0xa5, 0xe5, 0x6e, 0x10, 0x41, 0x02, 0xfa, 0x20, 0x9c, 0x6a, 0x85, 0x2d, 0xd9, 0x06, 0x60, 0xa2, 0x0b, 0x2d, 0x9c, 0x35, 0x24, 0x23, 0xed, 0xce, 0x25, 0x85, 0x7f, 0xcd, 0x37, 0x04, 0x00, 0x00, 0x00, 0x00, 0x43, 0x41, 0x04, 0x11, 0xdb, 0x93, 0xe1, 0xdc, 0xdb, 0x8a, 0x01, 0x6b, 0x49, 0x84, 0x0f, 0x8c, 0x53, 0xbc, 0x1e, 0xb6, 0x8a, 0x38, 0x2e, 0x97, 0xb1, 0x48, 0x2e, 0xca, 0xd7, 0xb1, 0x48, 0xa6, 0x90, 0x9a, 0x5c, 0xb2, 0xe0, 0xea, 0xdd, 0xfb, 0x84, 0xcc, 0xf9, 0x74, 0x44, 0x64, 0xf8, 0x2e, 0x16, 0x0b, 0xfa, 0x9b, 0x8b, 0x64, 0xf9, 0xd4, 0xc0, 0x3f, 0x99, 0x9b, 0x86, 0x43, 0xf6, 0x56, 0xb4, 0x12, 0xa3, 0xac, 0xff, 0xff, 0xff, 0xff, 0x02, 0x00, 0xca, 0x9a, 0x3b, 0x00, 0x00, 0x00, 0x00, 0x43, 0x41, 0x04, 0xae, 0x1a, 0x62, 0xfe, 0x09, 0xc5, 0xf5, 0x1b, 0x13, 0x90, 0x5f, 0x07, 0xf0, 0x6b, 0x99, 0xa2, 0xf7, 0x15, 0x9b, 0x22, 0x25, 0xf3, 0x74, 0xcd, 0x37, 0x8d, 0x71, 0x30, 0x2f, 0xa2, 0x84, 0x14, 0xe7, 0xaa, 0xb3, 0x73, 0x97, 0xf5, 0x54, 0xa7, 0xdf, 0x5f, 0x14, 0x2c, 0x21, 0xc1, 0xb7, 0x30, 0x3b, 0x8a, 0x06, 0x26, 0xf1, 0xba, 0xde, 0xd5, 0xc7, 0x2a, 0x70, 0x4f, 0x7e, 0x6c, 0xd8, 0x4c, 0xac, 0x00, 0x28, 0x6b, 0xee, 0x00, 0x00, 0x00, 0x00, 0x43, 0x41, 0x04, 0x11, 0xdb, 0x93, 0xe1, 0xdc, 0xdb, 0x8a, 0x01, 0x6b, 0x49, 0x84, 0x0f, 0x8c, 0x53, 0xbc, 0x1e, 0xb6, 0x8a, 0x38, 0x2e, 0x97, 0xb1, 0x48, 0x2e, 0xca, 0xd7, 0xb1, 0x48, 0xa6, 0x90, 0x9a, 0x5c, 0xb2, 0xe0, 0xea, 0xdd, 0xfb, 0x84, 0xcc, 0xf9, 0x74, 0x44, 0x64, 0xf8, 0x2e, 0x16, 0x0b, 0xfa, 0x9b, 0x8b, 0x64, 0xf9, 0xd4, 0xc0, 0x3f, 0x99, 0x9b, 0x86, 0x43, 0xf6, 0x56, 0xb4, 0x12, 0xa3, 0xac, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00};
    BITCOIN_ASSERT(raw_tx == ss.get_data());

    hash_digest tx_hash = libbitcoin::generate_sha256_hash(ss.get_data());

    data_chunk pubkey{0x04, 0x11, 0xdb, 0x93, 0xe1, 0xdc, 0xdb, 0x8a, 0x01, 0x6b, 0x49, 0x84, 0x0f, 0x8c, 0x53, 0xbc, 0x1e, 0xb6, 0x8a, 0x38, 0x2e, 0x97, 0xb1, 0x48, 0x2e, 0xca, 0xd7, 0xb1, 0x48, 0xa6, 0x90, 0x9a, 0x5c, 0xb2, 0xe0, 0xea, 0xdd, 0xfb, 0x84, 0xcc, 0xf9, 0x74, 0x44, 0x64, 0xf8, 0x2e, 0x16, 0x0b, 0xfa, 0x9b, 0x8b, 0x64, 0xf9, 0xd4, 0xc0, 0x3f, 0x99, 0x9b, 0x86, 0x43, 0xf6, 0x56, 0xb4, 0x12, 0xa3};
    // Leave out last byte since that's the hash_type_code (SIGHASH_ALL in this
    // case)
    data_chunk signature{0x30, 0x44, 0x02, 0x20, 0x4e, 0x45, 0xe1, 0x69, 0x32, 0xb8, 0xaf, 0x51, 0x49, 0x61, 0xa1, 0xd3, 0xa1, 0xa2, 0x5f, 0xdf, 0x3f, 0x4f, 0x77, 0x32, 0xe9, 0xd6, 0x24, 0xc6, 0xc6, 0x15, 0x48, 0xab, 0x5f, 0xb8, 0xcd, 0x41, 0x02, 0x20, 0x18, 0x15, 0x22, 0xec, 0x8e, 0xca, 0x07, 0xde, 0x48, 0x60, 0xa4, 0xac, 0xdd, 0x12, 0x90, 0x9d, 0x83, 0x1c, 0xc5, 0x6c, 0xbb, 0xac, 0x46, 0x22, 0x08, 0x22, 0x21, 0xa8, 0x76, 0x8d, 0x1d, 0x09};
    BITCOIN_ASSERT(signature.size() == 70);

    elliptic_curve_key key;
    if (!key.set_public_key(pubkey))
    {
        log_fatal() << "unable to set EC public key";
        return -1;
    }

    log_info() << "checksig returns: " << (key.verify(tx_hash, signature) ? "true" : "false");
    return 0;
}


BTW that's the first spent tx in bitcoin from block 170,

http://blockexplorer.com/block/00000000d1145790a8694403d4063f323d499e655c83426834d4ce2f8dd4a2ee
http://blockexplorer.com/tx/f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 27, 2011, 04:03:32 AM
For my OP_CHECKSIG code I tested using the following two transactions:

Tx2 uses an output from Tx1.  I wish I could give credit to whoever gave this to me... but I can't remember....

Code:
tx1raw= '\x01\x00\x00\x00\x02\x0f{\x7f\xb8mL\xf6F\x05\x8eA\xd3\xb0\x07\x18?\xdfysn\xd1\x9b*th\xab\xc5\xbd\x04\xb1n\x91\x00\x00\x00\x00\x8cI0F\x02!\x00\xb2\xee9\xd2\xfc\xc2\xe5TJW\xc3\x0f{NI\xcf\xb8""fm\x03O\xb9\x0e"4\x8e\x17\xe2\x8e\x0f\x02!\x00\xdb\x91\xc3\x19\x9c\xc7\xb4\x1dMz\xfc\xe0\xcc\xb4\xce\xb4$\xb9GmQ\xc0aBX=\xafS\xce\n\x9bf\x01A\x04\xc3"\x15\xa9\t0\x11\xbd<A(:\xce=\x00,f`w\xb2J`[<\xfc\x8fq\x01\x9a\x0fC\xdff\xf3\x89\xf3\xd9\xa6!\x88\xa4\x94\xb8i\xdc~_\x9d\xff\xc9\x8av\xd3\x08\x8a!\xe9\xb78\xec\x9e\xba\x98\xcb\xff\xff\xff\xff\x97\x00A%R\x8f{^\xd34e\xca\xaa\xe0!\xc0\xb8\x15\xf3\xe6\xa3pvA\xd5\xa0\xbc\xa4?\xc1II\x01\x00\x00\x00\x8aG0D\x02 3\xd0,.\x89o\x1a\x12RH\x8dSL\xfb\x08\xab\xf3\xe7\xea\x90\xab\xa7\xbaoW\xab\xf1\x89\xce\xf1\xd87\x02 \x05f\x8duP\x13\xb0\xe5\x9a*\xf5\x14_\x10\xef\xe6.\xa7\x16\xd33&\x8b\x0bZ>\xfb\xd8-\x149\xbe\x01A\x04\xc3"\x15\xa9\t0\x11\xbd<A(:\xce=\x00,f`w\xb2J`[<\xfc\x8fq\x01\x9a\x0fC\xdff\xf3\x89\xf3\xd9\xa6!\x88\xa4\x94\xb8i\xdc~_\x9d\xff\xc9\x8av\xd3\x08\x8a!\xe9\xb78\xec\x9e\xba\x98\xcb\xff\xff\xff\xff\x01\x00\xc2\xeb\x0b\x00\x00\x00\x00\x19v\xa9\x14\x02\xbfK(\x89\xc6\xad\xa8\x19\x0c%.p\xbd\xe1\xa1\x90\x9f\x96\x17\x88\xac\x00\x00\x00\x00'
tx2raw= "\x01\x00\x00\x00\x030\xf3p\x1f\x9b\xc4dU/pIW\x91\x04\x08\x17\xcewz\xd5\xed\xe1nR\x9f\xcd\x0c\x0e\x94\x91V\x94\x00\x00\x00\x00\x8cI0F\x02!\x00\xf5tk\x0b%OZ7\xe7RQE\x9cz#\xb6\xdf\xcb\x86\x8a\xc7F~\xdd\x9ao\xdd\x1d\x96\x98q\xbe\x02!\x00\x88\x94\x8a\xea)\xb6\x91a\xca4\x1cI\xc0&\x86\xa8\x1d\x8c\xbbs\x94\x0f\x91\x7f\xa0\xedqThm>[\x01A\x04G\xd4\x90V\x1f9l\x8a\x9e\xfc\x14Hk\xc1\x98\x88K\xa1\x83y\xbc\xac.\x0b\xe2\xd8RQ4\xabt/0\x1a\x9a\xca6`n])\xaa#\x8a\x9e)\x93\x001PB=\xf6\x92Ecd-J\xfe\x9b\xf4\xfe(\xff\xff\xff\xffr\x14+\xf7hl\xe9,m\xe5\xb73e\xbf\xb9\xd5\x9b\xb6\x0c,\x80\x98-YX\xc1\xe6\xa3\xb0\x8e\xa6\x89\x00\x00\x00\x00JI0F\x02!\x00\xbc\xe4:\xd3\xac\xbcy\xb0$~T\xc8\xc9\x1e\xac\x1c\xf9\x03u\x05\x00\x0e\x01\xd1\xfd\x81\x18T\xd8[\xc2\x1a\x02!\x00\x99*oo/\xebob\xd3po;\x9a\xaa\xb8\x8d\x9f\x112\x95j\x1d\xff\xa9&\xcdUn\xd5S`\xdf\x01\xff\xff\xff\xff\xd2\x81(\xbb\xb6 |\x1c=\nc\x0c\xc6\x19\xdc~{\xeaV\xac\x19\xa1\xda\xb1'\xc6,x\xfa\x1bc,\x00\x00\x00\x00IH0E\x02  \x97W6\x81aSw\x08\xfd)\xd8\x9b\xb1\xe9\xd6H\x00yI\xec\xfd\xedx\x9bQ\xa9c$\xcbe\x18\x02!\x00\xcd\x0f|0!9\x16H+n\x16m\x8aO+\x98\x1fw~\xb1\x84\xcd\x8aI_\x1b=6\x90\xfb\xbf-\x01\xff\xff\xff\xff\x01\x00\xa6\xf7_\x02\x00\x00\x00\x19v\xa9\x14\x9e5\xd9<w\x92\xbd\xca\xadV\x97\xdd\xeb\xf0CS\xd9\xa5\xe1\x96\x88\xac\x00\x00\x00\x00"



Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: genjix on July 27, 2011, 04:46:57 AM
OK, OP_CHECKSIG pushed to libbitcoin,

https://gitorious.org/libbitcoin/libbitcoin/commit/5f5dbd3f9b307e3d343cc007036995d8f8d24958


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: genjix on July 27, 2011, 04:50:47 AM
Code:
01 00 00 00              version
01                       number of inputs (var_uint)

input 0:
c9 97 a5 e5 6e 10 41 02
fa 20 9c 6a 85 2d d9 06
60 a2 0b 2d 9c 35 24 23

43                       size of script (var_uint)
41                       push 65 bytes to stack
04 11 db 93 e1 dc db 8a
01 6b 49 84 0f 8c 53 bc
1e b6 8a 38 2e 97 b1 48
2e ca d7 b1 48 a6 90 9a
5c b2 e0 ea dd fb 84 cc
f9 74 44 64 f8 2e 16 0b
fa 9b 8b 64 f9 d4 c0 3f
99 9b 86 43 f6 56 b4 12
a3
ac                       OP_CHECKSIG

02                       number of outputs (var_uint)

output 0:
00 ca 9a 3b 00 00 00 00  amount = 10.00000000
43                       size of script (var_uint)
script for output 0:
41                       push 65 bytes to stack
04 ae 1a 62 fe 09 c5 f5
1b 13 90 5f 07 f0 6b 99
a2 f7 15 9b 22 25 f3 74
cd 37 8d 71 30 2f a2 84
14 e7 aa b3 73 97 f5 54
a7 df 5f 14 2c 21 c1 b7
30 3b 8a 06 26 f1 ba de
d5 c7 2a 70 4f 7e 6c d8
4c
ac                       OP_CHECKSIG

output 1:
00 28 6b ee 00 00 00 00  amount = 40.00000000
43                       size of script (var_uint)
script for output 1:
41                       push 65 bytes to stack
04 11 db 93 e1 dc db 8a
01 6b 49 84 0f 8c 53 bc
1e b6 8a 38 2e 97 b1 48
2e ca d7 b1 48 a6 90 9a
5c b2 e0 ea dd fb 84 cc
f9 74 44 64 f8 2e 16 0b
fa 9b 8b 64 f9 d4 c0 3f
99 9b 86 43 f6 56 b4 12
a3                       
ac                       OP_CHECKSIG

00 00 00 00              locktime
01 00 00 00              hash_code_type (added on)

result =
01 00 00 00 01 c9 97 a5 e5 6e 10 41 02 fa 20 9c 6a 85 2d d9 06 60 a2 0b 2d 9c 35
24 23 ed ce 25 85 7f cd 37 04 00 00 00 00 01 ac 02 00 ca 9a 3b 00 00 00 00 43 41
04 ae 1a 62 fe 09 c5 f5 1b 13 90 5f 07 f0 6b 99 a2 f7 15 9b 22 25 f3 74 cd 37 8d
71 30 2f a2 84 14 e7 aa b3 73 97 f5 54 a7 df 5f 14 2c 21 c1 b7 30 3b 8a 06 26 f1
ba de d5 c7 2a 70 4f 7e 6c d8 4c ac 00 28 6b ee 00 00 00 00 43 41 04 11 db 93 e1
dc db 8a 01 6b 49 84 0f 8c 53 bc 1e b6 8a 38 2e 97 b1 48 2e ca d7 b1 48 a6 90 9a
5c b2 e0 ea dd fb 84 cc f9 74 44 64 f8 2e 16 0b fa 9b 8b 64 f9 d4 c0 3f 99 9b 86
43 f6 56 b4 12 a3 ac 00 00 00 00 01 00 00 00


Relevant file,  https://gitorious.org/libbitcoin/libbitcoin/blobs/master/src/script.cpp

See function called script::op_checksig()

Also, https://gitorious.org/libbitcoin/libbitcoin/blobs/master/tests/script-test.cpp


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on August 21, 2011, 01:16:36 AM
I finally got around to updating my code to handle test-network addresses, and so I finally had to figure out how to do the address conversion correctly.  Here are the updated diagram!

http://dl.dropbox.com/u/1139081/BitcoinImg/PubKeyToAddrSmall.png

The full size image is here:  http://dl.dropbox.com/u/1139081/BitcoinImg/PubKeyToAddr.png (http://dl.dropbox.com/u/1139081/BitcoinImg/PubKeyToAddr.png)

As before, you can still access the raw SVG data here (http://dl.dropbox.com/u/1139081/BitcoinImg/BTC_Illustrations.svg).  I used Inkscape to create this, so there's no guarantees the file is usable unless you also use Inkscape.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: samr7 on August 21, 2011, 01:44:13 AM
I finally got around to updating my code to handle test-network addresses, and so I finally had to figure out how to do the address conversion correctly.  Here are the updated diagrams!

+1

Nice work etotheipi!


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on August 23, 2011, 12:49:27 AM
And now I've updated the Transaction bytemap with examples of each TxIn and TxOut type.  Once again, I've learned quite a bit about BTC since making the original illustrations, and so I thought it was a good idea to update it.

http://dl.dropbox.com/u/1139081/BitcoinImg/TxBinaryMapSmall.png

The full size image can be downloaded here:  http://dl.dropbox.com/u/1139081/BitcoinImg/TxBinaryMap.png (http://dl.dropbox.com/u/1139081/BitcoinImg/TxBinaryMap.png)

The only thing I'm not 100% sure about is the script field on a coinbase generation TxIn.  I'm pretty sure I read that mining pools use that field as an extra nonce, so that they can have all the connected workers hashing different things and avoid redundant computations (different script --> different Tx hash --> different merkle tree --> different merkle root).


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: jackjack on August 23, 2011, 01:57:32 AM
In TxOut 0, it's hash160(recipientPubkey) instead of recipientAdress


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on August 23, 2011, 02:34:52 AM
I originally decided not to go into byte-level detail on the TxOut scripts, but I have no idea why.  I've updated it to be a complete byte map now.  Also changed constants like "30" to "0x30" for clarification.  If it's not perfect, it's damned close!  :)


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: jackjack on August 23, 2011, 02:47:29 AM
Indeed, it was close: length(hash160) = 20 = 0x14 (not 0x16) :P
It should be perfect now


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: bcforum on August 23, 2011, 03:31:51 AM

Second, here's another diagram.  It's not nearly as complicated as OP_CHECKSIG, but once I had it mapped out like this, I was able to do a ton of address magic in my code and get it right on the first try.  Not having to guess endianness at every step is a big help!  Remembering the extra bytes that have to be added here and there helps, too.


IXCoin is using the same leading byte as Bitcoin (yet another example of the laziness of the "developer")


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: Nasakioto on August 24, 2011, 10:51:13 PM
IXCoin is using the same leading byte as Bitcoin (yet another example of the laziness of the "developer")


That's being addressed in the upcoming update.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: Red Emerald on November 17, 2011, 06:10:48 AM
Awesome diagrams! I am really getting interested in scripts.  This cleared up some grey areas for me on the "basic" transactions. Thanks!


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: crispy on November 20, 2011, 03:33:03 AM
Thanks for the hard work.  Usually the work of documenting software can teach you better than almost any other way how it actually works.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on November 22, 2011, 06:36:22 AM
Awesome diagrams! I am really getting interested in scripts.  This cleared up some grey areas for me on the "basic" transactions. Thanks!
If you're interested in scripts, you might find the following links interesting.  The first one is literally every non-std script from the testnet.  I was looking for non-standard transactions for testing my scripting code in PyBtcEngine -- and it turns out there's quite a few on the testnet:

https://github.com/etotheipi/PyBtcEngine/blob/master/testnetNonStdScript.txt
https://github.com/etotheipi/PyBtcEngine/blob/master/scriptEvalStackState.txt

The second link is a few multi-sig transactions, with the stack state at every step of script evaluation (as produced by my code once I finally got it working).  It is a variety of scripts not currently considered "standard" but may become standard in the near future.  Seeing the script-evaluation chain for complicated transactions is quite educational (and critical for debugging!).

Thanks for the hard work.  Usually the work of documenting software can teach you better than almost any other way how it actually works.
Thanks crispy.  It was actually the act of implementing all this stuff in code that really taught me about it, and the documentation was my way of giving back for all the knowledge I gained from for the forums.  I wish I had documentation like this when I first started down this development path 4 months ago.

P.S. - if you found any of this useful, please consider donating.  This stuff took a lot of time!



Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: marcus_of_augustus on March 16, 2012, 09:33:06 PM
tagging


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: Realpra on November 26, 2012, 06:06:09 PM
Forgive me for being a noob and doing a necro here, but what is "VI"/"scriptlen", it does not say on the wiki either?


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: MatthewLM on November 26, 2012, 06:41:16 PM
The VI stands for VarInt and is described here: https://en.bitcoin.it/wiki/Protocol_specification#Variable_length_integer

It's an integer representation for the length of the scripts and I implemented the encoding and decoding of them here: https://github.com/MatthewLM/cbitcoin/blob/master/src/CBVarInt.c


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on January 13, 2013, 03:58:26 AM
Speaking of VAR_INTs, smtp just pointed out that my OP_CHECKSIG diagram was not showing the VAR_INTs that represent script length in the TxCopy (with blanked scripts).  While I think that your Tx class in whatever language you're using should handle that detail for you, the graphic was definitely inconsistent.  So I fixed it and re-uploaded it to the wiki, too.

https://en.bitcoin.it/wiki/OP_CHECKSIG
https://bitcointalk.org/index.php?topic=29416.msg370321#msg370321


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: CompNsci on April 11, 2014, 09:57:01 PM
Can we get the .svg files posted for the OP_CHECKSIG diagram somewhere? The png available at the wiki is quite small and difficult to read.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on April 11, 2014, 10:09:25 PM
It's actually on the first page:

https://bitcointalk.org/index.php?topic=29416.msg384034#msg384034

Though please doublec-check that's up to date.  It's been updated a couple times and it's possible I updated a copy of that svg.


Title: Re: On-the-wire byte map & OP_CHECKSIG diagram (knowledge donation!)
Post by: etotheipi on July 16, 2014, 04:31:22 AM

This is good.  But I'll call you on the length of the pre-base58 address: it's 25 bytes long, not 24, but the upper byte is always 0 for regular bitcoin addresses.  Also, the base58 encoding procedure will prepend a 1 for each preceding \x00 byte in the pre-encoded address, not just a single 1.

Whoops, I was trying to update the diagrams and accidentally replied instead of editing.  Sorry for the spam.... but enjoy the pics!