Looking for a reference to specific examples of BTC mining calculation

[gt.townsend]

I have spend two weeks now searching the internet for a worked calculation concerning the formulation of the block header to be hashed, but I can't find anything that addresses the WHOLE puzzle, so I thought I'd ask for help here.  I am working on some code for BTC mining on a CPU. Before you get all up in arms about this, I have a fleet of S21 ASIC miners. I know a CPU can't compare, but this is an academic exercise.

I've got my code to communicate to the pool and submit fake or stale shares just to prove that I am able to log in and start mining properly. I'm at the point where I want to fiddle with nonce, etcetera, to find some real shares. I'm on a Ryzen CPU, so I might possibly have enough horse power if I'm lucky to bump into a share before my block expires. But that's not the point ... I want to prove that I fully understand the process by being able to write code to do this.

I managed to get as far as finding an "almost" fully worked example, and tested my code to confirm that I got the same (good) hash as the author did, however I had to manually feed the value for the Merkle root into my code, since there was not enough information in the resource to know how the root was calculated. I know how is SHOULD be calculated, but without this missing information I cannot confirm my understanding. I do have captured traffic from my S21's to test with, but I've never been able to get a hash value that made sense. i.e. no leading or even trailing zeros on it - it always starts (and ends) with a non-zero hex digit. So I temporarily put my captured traffic aside and spent two weeks searching for a fully worked example to confirm that I was doing everything correctly.

As I said, I got as far as finding a source that gave the 80 byte block to be hashed, along with a detailed explanation of most of the bit fields it contained. It works fine, but the portion of the 80 bytes that contains the Merkel root does not have enough information for me to test that my code builds the root correctly. The worked example shows all the transaction hashes in the Merkel Branch, but I also require coinbase1 and coinbase2 along with Extranonce1 in order to include that in the hashing process to build the Merkel root. The example does not include these elements ... I forget just now, but MAYBE the source included an example Extranonce1 in there someplace, but it doesn't matter without coinbase1 and coinbase2, and those are definitely not present.

Now I found many sources that include an actual worked example of JUST this portion, i.e. everything needed find the doubleHash of (coinbase1 + Extranonce + coinbase2), but these examples do not include the rest of what is needed to actually build the 80 byte block header and hash it. So there is no way to verify that it has been done correctly or that an nonce even exists that would correspond to finding a share.  I am looking for a single, fully worked example showing all the steps that includes what the block is supposed to hash to. This is the only way I can confirm my code. Once I can prove my code works, I should be able to plug in my captured S21 data and calculate the hash which had better have a lot of zeros on one side (or the other side).  

Can anyone point me such an example online or even give me some fields that work so I can get my code working? I need all the components that make up the block header including the nonce, and I also need to know what it is supposed to hash to. Can anyone help?
-gt-

[Cricktor]

You could inspect the code of mining tools like cgminer or bfgminer. Or try to figure out stuff at https://learnmeabitcoin.com/technical/mining/. On this site you can go pretty deep into the details, you can expand interactive tools there and learn a lot.

Look at stuff like candidate block and how the block header is assembled for it, also how Merkle root is computed exactly. Look for early block examples with few transactions only and see if your code does it right. I don't quite remember if the Bitcoinbook explain it detailed enough. I guess, it's time to read another pass through it...

[gt.townsend]

Yes, thanks, Cricktor. Of all the source information I was able to find, this one (which I already knew about)  came the closest to solving my troubles. It filled in a lot of holes, but still lacks a 100% completely worked example start to finish. I'll try the other source you mentioned since I haven't seen it yet. Thanks.
-gt-


Okay, I've been able to get a little further with this now. I found an example that had the coin base hash. Still missing this:   coinbase = coinbase1 + extranonce1 + extranonce2 + coinbase2

However this SHOULD be one of the easiest parts to deal with. If I had the pieces to make coinbase, I could hash it twice and I'd get the double hash of coinbase. But I'm pretty close now, because the example I now found misses these pieces, but DOES include the double hash of coinbase.

I was able to use that to calculate the correct Merkle Root.    So even though I STILL haven't managed to find a complete worked example end-to-end, the only missing link is extremely trivial - just concatenating four things and double hashing them.

Therefore, I'm assuming that my captured mining traffic will now work and generate a reasonable hash. Although it's not over yet, its pretty close.

I found so much misleading information. One source claimed that regardless of what kind of BTC operation it is (transmitting funds, or submitting shares) that you have to hash pairs linearly, not in a tree. Another source insisted that the concatenation of pieces from the merkle branches had to be paired in lexographic order, not in the order they appeared. Finally, an (apparently) correct source indicated NOT to lexographically order, but to build a binary tree hash the nodes as pairwise leaves moving them up a level after each completion until only the hashed root remains. THAT strategy allowed me to reproduce the example's final value from the coinbase and merkel branches.

I even tried using ChatGPT, but he kept changing his mind about which of the three strategies to use to hash the merkel tree. I can't believe there doesn't seem to be a complete, proper, end-to-end example out there on the net, and what's more surprising is all the posted code I've found which clearly uses completely different strategies to do the hashing. Every posting claims to be tested and correct and for BTC mining, but each one gives a different final hash for the same inputs.

-gt-


Well, I'm not out of the woods yet. Having taken data from https://learnmeabitcoin.com/ I now realize that I can't tell how it relates to my captured mining data. For example, when I see the table of TX hashes in the table on the website example, does the endian-ness match how they look in the JSON strings from the miner/pool? There are still too many pieces and unknown details, and too many possible things to reverse or not to reverse, so without more details I STILL can't resolve this. I've decided to just post the pool/miner exchange that led to a accepted packet and hope that SOMEONE who sees this feels like seeing if they can put it all together in a way that gives a block header that hashes to a reasonable hash. I've removed all traffic except that relevant to the successfully mined share. However, the exchanges are probably not in order. Here's what I got:

The miner subscribes:
{"id": 3, "method": "mining.subscribe", "params": ["Antminer S19j Pro+/Fri Apr 14 21:39:16 CST 2023"]}



The pool responds:
{"id":3,"result":[[["mining.notify","00"],["mining.set_difficulty","00"]],"00",7],"error":null}
ExtraNonce1: '00',ExtraNonce2_size: 7,


The miner asks to authorize:
{"id": 5, "method": "mining.authorize", "params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "x"]}

The pool responds:
{"id":5,"result":true,"error":null}

Pool offers job to miner:
{"id":null,"method":"mining.notify","params":["3782626","f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000","01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666","ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000",["298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98","713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb","d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af","cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7","43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a","86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9","fa5d9bd36f07cd05d7c38b37e5dcb18a4ff5a1c5f137c6c03f97acce091b4770","05b0ee676903b07295f97c2833348101368d5f0f0039fde81e35f3067df03278","4fe266926aab2d9e481aa79ab081b2953fb043c873f8111ec171673aafeb3981","96511215a92f1011bdfae46a4e2b86359571a72f1beaa5f795abb7493fbe4206"],"20000000","17028bb1","67c28154",false]}

Miner submits a share:
{"params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "3782626", "ec100000000000", "67c28154", "16384721", "09b66000"], "id": 19, "method": "mining.submit"}

The pool responds:
{"id":19,"result":true,"error":null}
==========================================
{
  "params": [
    "1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018",  // Username (Bitcoin address + worker name)
    "3782626",                                // Job ID
    "ec100000000000",                         // Extranonce2
    "67c28154",                               // Time
    "16384721",                               // Nonce
    "09b66000"                                // Extra nonce (optional, rarely used)
  ],
  "id": 19,                                   // Request ID
  "method": "mining.submit"                   // Method name
}


pool offers job to miner
{
  "id": null,                          // Identifier for the request (null means no specific ID)
  "method": "mining.notify",           // Method being called (notifies miners of a new block)
  "params": [                          // Parameters for the mining job
    "3782626",                         // Job ID (unique identifier for this mining job)
    "f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000", // Previous block hash
    "01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666", // Coinbase transaction (block reward + fees)
    "ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000", // Merkle root (hash of all transactions)
    [
      "298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98", // Transaction hash 1
      "713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb", // Transaction hash 2
      "d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af", // Transaction hash 3
      "cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7", // Transaction hash 4
      "43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a", // Transaction hash 5
      "86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9", // Transaction hash 6
      "fa5d9bd36f07cd05d7c38b37e5dcb18a4ff5a1c5f137c6c03f97acce091b4770", // Transaction hash 7
      "05b0ee676903b07295f97c2833348101368d5f0f0039fde81e35f3067df03278", // Transaction hash 8
      "4fe266926aab2d9e481aa79ab081b2953fb043c873f8111ec171673aafeb3981", // Transaction hash 9
      "96511215a92f1011bdfae46a4e2b86359571a72f1beaa5f795abb7493fbe4206"  // Transaction hash 10
    ],
    "20000000",                        // Block version (format of the block)
    "17028bb1",                        // Network difficulty target (how hard it is to mine)
    "67c28154",                        // Current time (Unix timestamp)
    false                              // Clean flag (false means the block may include unconfirmed transactions)
  ]
}



20000000  # Version
f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac300000000000000000  # Previous Block Hash
[Merkle Root (computed from coinbase & merkle branches)]  # Placeholder
67c28154  # Timestamp
17028bb1  # Bits (Difficulty Target)
16384721  # Nonce




Name                   Usage in Block Construction
Extranonce1             Used in coinbase transaction.
Extranonce2             Used in coinbase transaction.
Username                Embedded in the coinbase transaction.
Job ID                  Included in the share submission.
Previous Block Hash     Used in block header.
Coinbase1               Combined with extranonce1 and extranonce2.
Coinbase2               Combined with extranonce1 and extranonce2.
Merkle Branches   List    Used in block header.
Timestamp (time)   Used in block header.
Bits                    Defines the maximum valid hash for the block.
Nonce                   Used in block header.
Extra Nonce             Not always used in hashing, but included in submission.

[sdfasdf]

Well, I'm not out of the woods yet. Having taken data from https://learnmeabitcoin.com/ I now realize that I can't tell how it relates to my captured mining data. For example, when I see the table of TX hashes in the table on the website example, does the endian-ness match how they look in the JSON strings from the miner/pool? There are still too many pieces and unknown details, and too many possible things to reverse or not to reverse, so without more details I STILL can't resolve this. I've decided to just post the pool/miner exchange that led to a accepted packet and hope that SOMEONE who sees this feels like seeing if they can put it all together in a way that gives a block header that hashes to a reasonable hash. I've removed all traffic except that relevant to the successfully mined share. However, the exchanges are probably not in order. Here's what I got:

The miner subscribes:
{"id": 3, "method": "mining.subscribe", "params": ["Antminer S19j Pro+/Fri Apr 14 21:39:16 CST 2023"]}



The pool responds:
{"id":3,"result":[[["mining.notify","00"],["mining.set_difficulty","00"]],"00",7],"error":null}
ExtraNonce1: '00',ExtraNonce2_size: 7,


The miner asks to authorize:
{"id": 5, "method": "mining.authorize", "params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "x"]}

The pool responds:
{"id":5,"result":true,"error":null}

Pool offers job to miner:
{"id":null,"method":"mining.notify","params":["3782626","f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000","01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666","ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000",["298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98","713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb","d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af","cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7","43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a","86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9","fa5d9bd36f07cd05d7c38b37e5dcb18a4ff5a1c5f137c6c03f97acce091b4770","05b0ee676903b07295f97c2833348101368d5f0f0039fde81e35f3067df03278","4fe266926aab2d9e481aa79ab081b2953fb043c873f8111ec171673aafeb3981","96511215a92f1011bdfae46a4e2b86359571a72f1beaa5f795abb7493fbe4206"],"20000000","17028bb1","67c28154",false]}

Miner submits a share:
{"params": ["1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018", "3782626", "ec100000000000", "67c28154", "16384721", "09b66000"], "id": 19, "method": "mining.submit"}

The pool responds:
{"id":19,"result":true,"error":null}
==========================================
{
  "params": [
    "1AGf9BSutC5ZAnMXi8rvE6nxoaVv4rYfNa.018",  // Username (Bitcoin address + worker name)
    "3782626",                                // Job ID
    "ec100000000000",                         // Extranonce2
    "67c28154",                               // Time
    "16384721",                               // Nonce
    "09b66000"                                // Extra nonce (optional, rarely used)
  ],
  "id": 19,                                   // Request ID
  "method": "mining.submit"                   // Method name
}


pool offers job to miner
{
  "id": null,                          // Identifier for the request (null means no specific ID)
  "method": "mining.notify",           // Method being called (notifies miners of a new block)
  "params": [                          // Parameters for the mining job
    "3782626",                         // Job ID (unique identifier for this mining job)
    "f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac30000000000000000", // Previous block hash
    "01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff5a032f840d1d506f7765726564206279204c75786f72205465636868005f002a971a59fabe6d 6d648fad0e94f875d4783a1d2b146e0fa869052d181b79a88ebc54783aeda6018e1000000000000 00000010666", // Coinbase transaction (block reward + fees)
    "ffffffff05220200000000000017a914bf73ad4cf3a107812bad3deb310611bee49a3c79871a14c 1120000000017a914056adde53ebc396a1b3b678bb0d3a5c116ff430c870000000000000000266a 24aa21a9ed23ed3159409e966bff9642d36a658a385d5b655b0ba0bbba8d8b9317ce9a841500000 000000000002f6a2d434f524501a21cbd3caa4fe89bccd1d716c92ce4533e4d4733f459cc4ca322 d298304ff163b2a360d756c5db8400000000000000002b6a2952534b424c4f434b3a5033e57db45 8230515b003ad6b37755e76755337c12a9d8efd024916006f4f1500000000", // Merkle root (hash of all transactions)
    [
      "298563959d824c769c90df6880153ffe37ce692f40c86030f8da46072883ca98", // Transaction hash 1
      "713f3880adf3ec99241e10773c5994e0173631b23013be3a406423744523b5cb", // Transaction hash 2
      "d2cedcb2598099f5155c70f3964c8b9983d6bacdeae094a0c125eee2826332af", // Transaction hash 3
      "cd0308e9611464d808fc0cc7cd7a5220e7db837d27f8753ddf7ba1cf0e1004c7", // Transaction hash 4
      "43d98cfae83c979cb52376012ac6087c566b48e5b13c1e1fffbad82ab1ef429a", // Transaction hash 5
      "86c27c93418feac302f2fa49165314470c1c98fb989b7dd0712cba3c7ae515c9", // Transaction hash 6
      "fa5d9bd36f07cd05d7c38b37e5dcb18a4ff5a1c5f137c6c03f97acce091b4770", // Transaction hash 7
      "05b0ee676903b07295f97c2833348101368d5f0f0039fde81e35f3067df03278", // Transaction hash 8
      "4fe266926aab2d9e481aa79ab081b2953fb043c873f8111ec171673aafeb3981", // Transaction hash 9
      "96511215a92f1011bdfae46a4e2b86359571a72f1beaa5f795abb7493fbe4206"  // Transaction hash 10
    ],
    "20000000",                        // Block version (format of the block)
    "17028bb1",                        // Network difficulty target (how hard it is to mine)
    "67c28154",                        // Current time (Unix timestamp)
    false                              // Clean flag (false means the block may include unconfirmed transactions)
  ]
}



20000000  # Version
f29143edc62458e69f1a5661f85d2cdff4aeff9000008ac300000000000000000  # Previous Block Hash
[Merkle Root (computed from coinbase & merkle branches)]  # Placeholder
67c28154  # Timestamp
17028bb1  # Bits (Difficulty Target)
16384721  # Nonce




Name                   Usage in Block Construction
Extranonce1             Used in coinbase transaction.
Extranonce2             Used in coinbase transaction.
Username                Embedded in the coinbase transaction.
Job ID                  Included in the share submission.
Previous Block Hash     Used in block header.
Coinbase1               Combined with extranonce1 and extranonce2.
Coinbase2               Combined with extranonce1 and extranonce2.
Merkle Branches   List    Used in block header.
Timestamp (time)   Used in block header.
Bits                    Defines the maximum valid hash for the block.
Nonce                   Used in block header.
Extra Nonce             Not always used in hashing, but included in submission.

"I am also a beginner in writing Bitcoin mining code in Python. I know Python is slow, but I need to write a full Bitcoin mining program in Python. Can you share your code? I need to learn."

"How do I get the correct block template to start mining, and if I successfully find a block, how do I send and receive my reward?"

i have 8 core amd cpu                  # i need understand & write python code
i have Bitmain Antminer S19 Pro # after convert c++

https://github.com/iceland2k14/solominer/blob/main/solo_miner.py  # i know this git . but i cant understand.

Code:

bitcoin Miner
│
├── Network Layer (JSON-RPC/Stratum)
│   ├─ Get block templates # i don't know
│   └─ Submit found blocks # i don't know
│
├── Mining Core                # i know
│   ├─ Header construction# i know
│   ├─ Nonce iteration         # i know
│   └─ Hash validation         # i know
│
└── Transaction Management     # i know
    ├─ Coinbase TX generation   # i don't know
    └─ Merkle root calculation   # i know

@gt.townsend
Can you share your code? I need to learn.

[gt.townsend]

Hello, sdfasdf

I had a look at the URL you sent. I note that the author includes a commented out test "blockheader" and he is hashing it correctly. I know this for two reasons, i), the hash value obtained makes sense, and ii) I can take that blockheader and run it through any online tool and obtain the same hash. If you "uncomment" that line of code, you can confirm that his code really does generate the correct hash for it. However, if you examine the test blockheader, you'll note that it is clear that the version has been endian reversed. If you add a statement to print the blockheader generated from his program from the mining pool, you can see that the version is embeded with the OPPOSITE endian. This is the problem I'm running into - all sorts of code that claims to do one thing but does something else. I'll show you what I added to thecode to print the blockheaders that the program assembles from the pool:

Find this part of the code:

        blockheader = ctx.version + ctx.prevhash + merkle_root + ctx.ntime + ctx.nbits + nonce

        # Test
        #blockheader = "00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce"


and modify it like this:


        blockheader = ctx.version + ctx.prevhash + merkle_root + ctx.ntime + ctx.nbits + nonce
        print(blockheader);

        # Test
        #blockheader = "00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce"


compare the test blockheader to what the printout produces, in my case the print statement printed this:
20000000c26520e21ad7f8a9f277dc76355d111ab8f107cd00008d7d0000000000000000048b5d6 f84057432be23908430f4c7325f1266cc899a3656be49dbbdc33cbaef67c9bbf817028bb1c1778f dc


comparing these together (any of course they won't be the same, but the layout/organization should match):

00000034a701be3e9898775007c26f5f956db38ad858465828a500000000000000000000482eb3d 98241ac8bb925d6419a20d67d1a79e3f217b21805776f9ca537fad3809fe96667fa970217bf573c ce
20000000c26520e21ad7f8a9f277dc76355d111ab8f107cd00008d7d0000000000000000048b5d6 f84057432be23908430f4c7325f1266cc899a3656be49dbbdc33cbaef67c9bbf817028bb1c1778f dc

The font fools you here - these are indeed the SAME length, even though it might not seem to be so.

Anyway, clearly the version in the test block is:
34000000
and has been switched to little endian in the block, while the version printed from the pool is:
20000000
and appears in BIG endian order in the block

That should immediately raise a red flag. As far as I understand, the pool would reconstruct the block with the version flipped as 00000020 ... we can't get the same has at the miner that the pool gets if one flips it and the other doesn't!!!!

[gt.townsend]

Well, here's an update for anyone who cares.   I now FINALLY have all the bits and pieces I need to sort this out. From greg's  "learn me a bit coin" website,  I found an example of everything EXCEPT the coinbase value. What WAS included was everything else, plus the 32 bit double SHA256 hash of coinbase. Tonight I just found out that you can use this URL is you have a coinbasehash, and it will tell you the original coinbase value. [Obviously it cannot hash backwards, but it searches the block chain for the hash, and finds coinbase elsewhere in the block that it came from:

https://blockstream.info/api/tx/<inset coinbase hash here>/hex

for example:

https://blockstream.info/api/tx/18a16d322b235f636ab90e62e79a9f20a0b9c14e8da51e9dc0974f99f82ee444/hex

This gives coinbase=01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0804233fa04e028b12ffffffff0130490b2a010000004341047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95cd9bbe6da7e7ad7f2acb09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac00000000

Now you can break this apart as follows ... ask ChatGPT for the details:
01000000
010000000000000000000000000000000000000000000000000000000000000000000000
ffffffff
0804233fa04e028b12
ffffffff
01
30490b2a01000000
43
41047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95cd9bbe6da7e7ad7f2acb 09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac
00000000

and then we get coinbase1 = 01000000010000000000000000000000000000000000000000000000000000000000000000fffff fff0804233fa04e028b12

extranonce1+extranonce2=ffffffff

coinbase2 = 0130490b2a010000004341047eda6bd04fb27cab6e7c28c99b94977f073e912f25d1ff7165d9c95 cd9bbe6da7e7ad7f2acb09e0ced91705f7616af53bee51a238b7dc527f2be0aa60469d140ac0000 0000

An now, FINALLY we have ALL the bits and pieces required to compute the entire block header.

Everything else required is packed into the example blocks and is easy to find on greg's website. The above was the only missing link.
-gt-

[gt.townsend]

Well, apparently it is not that easy. I STILL cannot produce a decent hash. I've been working with an older version of cgminer that supports BTC mining, and I've written a program to act as a fake pool which serves up the exact same block every time. I've hacked the cgminer code to freeze the timestamp and extranonce2 and nonce to the values in my mined share. I'm having trouble figuring our where to set the nonce, so I'm not done yet, but in the meantime, I've discovered that cgminer uses a liner approach to create the merkle root, NOT a binary tree hash. I've also discovered that there is an endian change, not byte-wise, but a 32-bit endian change that I had no idea existed.

So many more pieces of the puzzle are now solved, but the puzzle is not solved yet ...

[gt.townsend]

In case anyone still cares about this post, I should mention that I FINALLY go the thing 100% solved. After wondering about the extra "sixth field" that the miner was submitting after the usual five fields that are clearly explained in the documentation, I looked more carefully and realized that during the JSON exchange, early after authenticating, the miner negotiated something with the pool called "version rolling" and the pool gave the miner a "version rolling mask".

It turns out I was doing everything correct EXCEPT I did not take this into account. It seems that the miner can negotiate with the pool to alter the "version number" in the block header, limited by the constraints specified in the "version rolling bitmask". There's a little Boolean equation that you use to combine your suggested version, with the original version, and with this bitmask to produce a "new_version" that you use in place of the original. Put in a simplified way, you are allowed to alter the bits in the original version as dictated by the bitmask. The sixth field contains your suggested modification to the bitmask.

This acts like an "extra" extranonce that you can play with, but the advantage is that you don't need to recalculate the coinbase, its hash, and the merkle root. You still have to hash the new blockheader, but that's quite a bit less work!

Anyway, I made that small adjustment and suddenly I calculated a share with a nice long string of leading zeros.