What exactly is the midstate? (trying to decode the json data in perl)

[j16sdiz]

The document say midstate is the hash of first half of data. But I can not repeat this in perl... 

For example, I have got a work like this (from the PMS pool)

Code:

  "data": "000000014e010ed08a87644d4a7d9b9852e8d0c5f5d3fc253917fc2b0000201600000000cec99c8365ef81d58321dbfea0473d01fd513ffa84132a0138d82ead5fd705264d12a5dd1b04864c00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000",
...
  "midstate": "4e3b9b00f856803067ebe0aa4026eac253a77a70374d6e490b50c947d549f74e"

My code:

Code:

use Digest::SHA qw/ sha256_hex /;
$data = '000000014e010ed08a87644d4a7d9b9852e8d0c5f5d3fc253917fc2b0000201600000000cec99c8365ef81d58321dbfea0473d01fd513ffa84132a0138d82ead';   # first 64 bytes
$data_raw =  pack("H*", $data );   # unpack it as a byte string
print "sha_raw = " . sha256_hex( $data_raw  ) . "\n";

$data_swapped =   pack "N*", unpack "V*", $data ;   # endian swap
print "sha_swapped = " . sha256_hex( $data_swapped ) . "\n";

Output:

Code:

sha_raw = bb4c09641e2d71c977b78edfd86c9793251e8e03f7ea211dcf7c71b7e231c7ac
sha_swapped = 8cbe144c95e66e244959719dabc1cfd4812275548ba099d78361fad1bfa7123b

Anybody can help me on this?

[jib]

SHA256 prepares the input data by appending padding and a length field to the end of it, so the total input ends up being a multiple of 64 bytes.

The "data" string you're given has this preparation step done already, which means you can't just feed it into a normal hashing function like sha256_hex and expect the right result (because sha256_hex will do this preparation step again).

Read about how SHA256 works. The midstate is the internal state after processing the first 64-byte block.

(Also, in the line of your code that swaps the data, you meant to use $data_raw rather than $data, right?)

[j16sdiz]

hmm... 
I was doing this because there are no perl module allow me to access the sha256 internals (at least not without saving the state to disk).

Does it means there are no way using the OS-provided crypto function that do complete sha256 hash work?
I think the  "via" code do the extra padding as well, or does it?

[jib]

You can use the crypto functions: Ignore the midstate, take the first 80 bytes (I think) of the data (i.e the data itself but not the padding) and hash that.

Why are you doing this? If you're writing a miner in perl using built-in crypto functions, it will be much slower than existing miners.

[taiki]

Actually I'm pretty sure you can use the built in crypto without any modification, since it is already a multiple of 64 it won't try to pad it.

Python code makes it really simple:

Code:

    def checkwork(block, difficulty_rep):
        s = hashlib.sha256(block.decode('hex'))
        val = int(s.hexdigest(), 16)		
        diff = int(difficulty_rep, 16)
        return val < diff

[lfm]

Actually I'm pretty sure you can use the built in crypto without any modification, since it is already a multiple of 64 it won't try to pad it.

Standard hash routines would add another 64 bytes then. It is not just padding, it is a length value and a terminator bit.

[lfm]

I was doing this because there are no perl module allow me to access the sha256 internals (at least not without saving the state to disk).

Does it means there are no way using the OS-provided crypto function that do complete sha256 hash work?
I think the  "via" code do the extra padding as well, or does it?

ignore the midstate and only use the first 80 bytes of the data if you are using a standard sha256 library. I have used the libgcrypt this way also (it is slow tho). You can also ignore the hash/hash1 fields passed, just use your own. Then return the data with your nonce inserted if you find a candidate hash.

Using standard sha256 libraries you will probably also need to undo the byte swaps on the data field. And reswap the data to return it.

[j16sdiz]

Actually I'm pretty sure you can use the built in crypto without any modification, since it is already a multiple of 64 it won't try to pad it.

Python code makes it really simple:

Code:

    def checkwork(block, difficulty_rep):
        s = hashlib.sha256(block.decode('hex'))
        val = int(s.hexdigest(), 16)		
        diff = int(difficulty_rep, 16)
        return val < diff

what is that "block" object?

I tried to print out some hash from  the "cpuminer" and repeat it in perl with no success:

perl code:

Code:

#!/usr/bin/perl
use Digest::SHA qw/ sha256_hex/;

$raw =  pack 'H*', '00000001258124a1e0837367309ed9433af69c741513067793bf1f490000c0c800000000f2a45d9b1294bf78d27fe1d77558fbedf2b1eb37bb5f1808d7b77e33d809b8fb4d1c26d21b04864c01000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000';

$head = substr($raw, 0, 0x80); # first 80 byte
$head_le = pack "N*", unpack "V*", $head;  # first 80 byte, endian-swapped

print sha256_hex($head), "\n";
print sha256_hex($head_le), "\n";

my output:

Code:

feca1e6a511bf10a01177086ae141115f0d554fc8758c640b81f9d3121d6cbc9
06f33a488390fb281b7d2fc68140a7ad69f8397caa04475c7f65f0b5f3f4922f

output generated with cpu-miner:

Code:

DBG: data:
 00000001258124a1e0837367309ed9433af69c741513067793bf1f490000c0c800000000f2a45d9b1294bf78d27fe1d77558fbedf2b1eb37bb5f1808d7b77e33d809b8fb4d1c26d21b04864c01000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000
DBG: hash1:
 697c6991ecddeb98d9af8f02a249b7e4006c8ae107a0991d4be8a88f4c9fcf1b
DBG: hash:     
 924638a8a492065a924f699ca1c13dd1fb4af43bb1c6d9ce8cab2e9bb6189ea7

patch i used again cpuminer:

Code:

diff --git a/sha256_generic.c b/sha256_generic.c
index 444e913..b9ba930 100644
--- a/sha256_generic.c
+++ b/sha256_generic.c
@@ -250,11 +250,23 @@ bool scanhash_c(const unsigned char *midstate, unsigned char *data,
                n++;
                *nonce = n;

+char *hs;
+hs = bin2hex(data - 64, 128);
+fprintf(stderr, "DBG: data:\n %s\n", hs);
+free(hs);
+
                runhash(hash1, data, midstate);
                runhash(hash, hash1, sha256_init_state);

                stat_ctr++;

+hs = bin2hex(hash1, 32);
+fprintf(stderr, "DBG: hash1:\n %s\n", hs);
+free(hs);
+hs = bin2hex(hash, 32);
+fprintf(stderr, "DBG: hash:\n %s\n", hs);
+free(hs);
+

[secmff]

I found two issues with that perl code. You need the first 80 bytes, not 0x80. Second, calculate back to big-endian to get the same result as the miner:

Code:

#!/usr/bin/perl
use Digest::SHA qw/ sha256 /;

$raw =  pack 'H*', '00000001258124a1e0837367309ed9433af69c741513067793bf1f490000c0c800000000f2a45d9b1294bf78d27fe1d77558fbedf2b1eb37bb5f1808d7b77e33d809b8fb4d1c26d21b04864c01000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000';
$short = substr($raw, 0, 80); # first 80 byte

$short_le = pack "N*", unpack "V*", $short; # to Little Endian

print "short_le:", unpack("H*", $short_le), "\n";

$hash1 = sha256($short_le);
$hash = sha256(sha256($short_le));

$hash1_be = pack "N*", unpack("V*", $hash1); # BE again
$hash_be  = pack "N*", unpack("V*", $hash);
print "hash1:  ", unpack("H*", $hash1_be), "\n";
print "hash:   ", unpack("H*", $hash_be), "\n";