miner client - just to understand mining

[questionstorm]

Hi,

I'm trying to implement a simple miner just to understand the steps
of minimg bitcoins.  The implementation is a bit simplistic because the
goal is to understand mining, not to make a fast miner.

This is the code skeleton:

Code:

#!/usr/bin/env ruby

require 'json'
require 'digest'

loop do
        getwork = JSON.parse `.../bitcoind getwork`

        data = getwork['data']
        target = getwork['target']

        version = data.slice(0, 8)
        prev_hash = data.slice(8, 64)
        merkle_root = data.slice(8 + 64, 64)
        timestamp = data.slice(8 + 64 + 64, 8)
        bits = data.slice(8 + 64 + 64 + 8, 8)

        # little vs. big endians:
        bin =
                [version].pack('H*').reverse +
                [prev_hash].pack('H*').reverse +
                [merkle_root].pack('H*').reverse +
                [timestamp].pack('H*').reverse +
                [bits].pack('H*').reverse

        (0..4294967295).each { |nonce|
                if Digest::SHA256.hexdigest(Digest::SHA256.digest(bin + [nonce.to_s(16)].pack('H*').reverse)) <= target

                        puts "yay!!!"

                         # do a 'submitblock' somehow (what is the hex data parameter to it?)
                         #
                         # maybe: ???
                         #
                         #   system ".../bitcoind submitblock '#{nonce.to_s(16)}'"
                         #
                         # or: ???
                         #   system ".../bitcoind submitblock '#{version + prev_hash + merkle_root + timestamp + bits + nonce.to_s(16)}'"
                         # or:
                         #   SHA256(SHA256(bin+nonce)) ??
                        break
                end
        }
end

I'd like to ask a little help how to do it.  getwork['data'] is a 256 length hexa string (128 bytes),
and I'm not sure what parts of that data have to be hashed, and how to combine it with the
nonce.  In the above code I just use the first 76 bytes (152 hexa digits) of the data, but I'm
afraid it's not what I should do.

As you can see I don't know how to send back when I find a hash.  I suppose I should use the
submitblock request, but what is the proper way to do this?  What is exactly the first paramether
of that call?

What are midstate and hash1 good for?  How do these things speed up mining?

I know that getwork is an ancient way, and there are so-called extensions to the protocol,
long polling, and other scary things.  Is there a brief introductory documentation about all
these things somewhere?

[deepceleron]

https://en.bitcoin.it/wiki/Getwork

https://bitcointalk.org/index.php?topic=51281

http://crypto.stackexchange.com/questions/1862/sha-256-midstate

http://www.javvin.com/protocolRPC.html (reply)

http://stackoverflow.com/questions/6273201/bitcoin-calculate-hash-from-getwork-function-how-to-do-it

{"result":{"data":"00000001a4940ed3fc5812a2205111bbe1b45a50557ef8d78ed74a380000016d00000000f64f077 c8c3847ef63d9b358374c540989aecc23ad7f7bb276b463b6ab22b4dc508ce4821a0575ef000000 0000000080000000000000000000000000000000000000000000000000000000000000000000000 0000000000080020000"

change the last four bytes of "data", which is the nonce (count from 0 to 0xffffffff if you like). Hash twice. If the hash is smaller than target, submit.
don't feel like typing.

[Schleicher]

What are midstate and hash1 good for?  How do these things speed up mining?

SHA256 only hashes 256 bits and the data is bigger than that. You need to hash the first part of the data (result is the midstate) and then hash the rest of the data.
You add the hash1 to the result (as padding bytes) and hash it.
The hash of the first part of the data doesn't change if you change the nonce, so you don't have to do it again.

[questionstorm]

I guess I've finished my miner but I can't check if it works fine. Simply
because it is too slow to find a share or block.  (Targets usually start
with at least 40 "0"s, so it's almost impossible to solve for a miner
that was written in a script language.)  Could someone please send me
a data and a target that is not too difficult and it's easy to verify if my
program does the right steps?  I think it would be great for others too
to see a whole getwork from the beginning up to the final nonce.

[Gavin Andresen]

See https://github.com/bitcoin/bitcoin/tree/master/contrib/pyminer  :

This is a 'getwork' CPU mining client for bitcoin.

It is pure-python, and therefore very, very slow.  The purpose is to
provide a reference implementation of a miner, for study.

[questionstorm]

This is a 'getwork' CPU mining client for bitcoin.

It is pure-python, and therefore very, very slow.  The purpose is to
provide a reference implementation of a miner, for study.

Thanks, that's great, but I have already done my own.  I just want to test it somehow.

[Insu Dra]

try on testnet, the difficulty there is allot lower ?
https://en.bitcoin.it/wiki/Testnet

[mskwik]

I do seem to recall having the same issue when working on mining code myself, now I keep logs of communications to replay for testing if needed.  Here are a couple of getwork requests and valid responses if that helps:

Code:

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91138", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e8e631c12000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91138", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824503257921a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824503257921a07a85e75636781000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91140", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc000000431000000004ef2e77a1afd0bc73e7b8e0becd271e43895372dabb062c78d5a97fb55f48cc05032581d1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "c2d1fff4a613196dd9ae9fd848673eefc1d7d501ae6ddc0ae2e5f820671971b0"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc000000431000000004ef2e77a1afd0bc73e7b8e0becd271e43895372dabb062c78d5a97fb55f48cc05032581d1a07a85e376a8504000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

[runeks]

I guess I've finished my miner but I can't check if it works fine. Simply
because it is too slow to find a share or block.  (Targets usually start
with at least 40 "0"s, so it's almost impossible to solve for a miner
that was written in a script language.)  Could someone please send me
a data and a target that is not too difficult and it's easy to verify if my
program does the right steps?  I think it would be great for others too
to see a whole getwork from the beginning up to the final nonce.

You could try simply plugging in values form already existing blocks, and see if they hash to the correct hash. That's what I did with my Javascript implementation.

I would find a block on Blockexplorer.com, then plug in the data from this block into my script, then set the nonce to, say, 1000 less than the value which yields the correct hash, start it, and wait for it to reach the nonce that is supposed to yield a block below difficulty, and see if it reports success when it reaches the right nonce. My script uses the "getblocktemplate", though, to construct the block header. And it also uses Gavin's bitcointools, to serialize the values correctly (this can be a real bitch).

[questionstorm]

I do seem to recall having the same issue when working on mining code myself, now I keep logs of communications to replay for testing if needed.  Here are a couple of getwork requests and valid responses if that helps:

Code:

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91138", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e8e631c12000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91138", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824503257921a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824503257921a07a85e75636781000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

{"error": null, "jsonrpc": "2.0", "id": 1, "result": {"hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold": true, "identifier": "91140", "data": "00000002fc517a2df2b283474b135215a00604af276318262f5eebc000000431000000004ef2e77a1afd0bc73e7b8e0becd271e43895372dabb062c78d5a97fb55f48cc05032581d1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate": "c2d1fff4a613196dd9ae9fd848673eefc1d7d501ae6ddc0ae2e5f820671971b0"}}
{"method": "getwork", "params": [ "00000002fc517a2df2b283474b135215a00604af276318262f5eebc000000431000000004ef2e77a1afd0bc73e7b8e0becd271e43895372dabb062c78d5a97fb55f48cc05032581d1a07a85e376a8504000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000" ], "id":1}

Thanks, this is exactly what I need.  Test data.

However there's something wrong with my understanding here.  All your targets are

Code:

"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000"

so my program tells that there's no need for heavy calculations, because it's a good solution if nonce = 0 since the hash in this case is  ...  well, something definitely smaller than ffffffffffffffffff...fffff00000000.    (It starts with somithing like 0xb, but doesn't matter, almost all hashes are smaller than this target.)  Am I missing something?  Yes, I am, but what's that?   

[runeks]

You need to reverse the byte order. The hashes you see on sites like blockexplorer.com have reversed the byte order, in order to make the zeros appear first.

In the Python example on this page you can see that the byte order is reversed: https://en.bitcoin.it/wiki/Block_hashing_algorithm

[questionstorm]

You could try simply plugging in values form already existing blocks, and see if they hash to the correct hash. That's what I did with my Javascript implementation.

Yes, that was one of my ideas too.  There I have two problems:  I'm not sure how to convert the time to the format it is in the data field, and there's no target value after a block is finished.  According to the help in blockexplorer the short form of difficulty can be converted into target somehow, but the way how to do it is not clear for me at all.

[runeks]

You could try simply plugging in values form already existing blocks, and see if they hash to the correct hash. That's what I did with my Javascript implementation.

Yes, that was one of my ideas too.  There I have two problems:  I'm not sure how to convert the time to the format it is in the data field, and there's no target value after a block is finished.  According to the help in blockexplorer the short form of difficulty can be converted into target somehow, but the way how to do it is not clear for me at all.

Take a look at the "Raw block" part of blockexplorer.com. Here the time is in decimal format: https://blockexplorer.com/rawblock/000000000000044b71ccd7180e9a3fdeb4d457b76bb6092ad5ebe9692cbf6ef8
So you just convert this to hex and then (perhaps, I don't quite remember) reverse the byte order.

In any case, when working with Bitcoin-related code, and things looks like they should work, but they don't, the first thing that should pop into your head is "byte order".

Here's how my Javascript miner does the conversion from the compact target to the full format target:

Code:

function compact_to_full(compact)
{
	bigendian = switch_endian(compact);
	
	length = parseInt(bigendian.substring(0,2), 16);
	
	full = bigendian.substring(2,8)
	while (full.length < length*2) {
		full = full + "00"
	}
	
	return full
}

So basically, you first switch endianness on the compact target hex string. Then you take the last 3 bytes of the compact target, and add as many zero bytes to the end of this as it takes for the length of the final number (in bytes) to be equal to the first byte in the compact target.

So with 0x1b0404cb as an example. We first get:

Code:

0404cb

and then we add enough zero bytes after this number so the total byte length of this number becomes equal to 0x1b (27):

Code:

0404cb000000000000000000000000000000000000000000000000

And then we pad with zeros so the final target becomes:

Code:

00000000000404cb000000000000000000000000000000000000000000000000

[questionstorm]

I still have trouble with my mining example.

Thanks to mskwik I have example data to work with.  Examining that data I think my script is wrong.
Could you please look at the example below and point me out where things go wrong?

So, the example code from mskwik is:

Code:

{"hash1"=>"00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target"=>"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold"=>true, "identifier"=>"91138", "data"=>"00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate"=>"03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}

In a more human-readable format the essence of this is the data and the target:

Code:

Data:
00000002 fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000 db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824 5032580a 1a07a85e 00000000 000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000

Target:
ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000

We also know that the right solution is:  (nonce = 0x8e631c12)

Code:

Solution:
00000002 fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000 db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824 5032580a 1a07a85e [b]8e631c12[/b]
000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000

Now I'll calculate the hash of this, and it should be less than, or equal to the target.

I do the endian change first (I guess that's the point where I screw up things), and then I
convert it to binary from hexa, because SHA2-256 hashing needs binary input:

Code:

After endian correction:
02000000 00000000000004312f5eebc027631826a00604af4b135215f2b28347fc517a2d 05c4c824392e6b8085e688d086cbf2994e4640722c2750213781c134db9fcfcc 0a583250 5ea8071a 121c638e 020000002d7a51fc4783b2f21552134baf0406a026186327c0eb5e2f3104000000000000cccf9fdb34c181372150272c

Binary string representation:
"\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x041/^\xEB\xC0'c\x18&\xA0\x06\x04\xAFK\x13R\x15\xF2\xB2\x83G\xFCQz-\x05\xC4\xC8.k\x80\x85\xE6\x88\xD0\x86\xCB\xF2\x99NF@r,'P!7\x81\xC14\xDB\x9F\xCF\xCC\nX2P^\xA8\a\x1A\x12\x1Cc\x8E\x02\x00\x00\x00-zQ\xFCG\x83\xB2\xF2\x15R\x13K\xAF\x04\x06\xA0&\x18c'\xC0\xEB^/1\x04\x00\x00\x00\x00\x00\x00\xCC\xCF\x9F\xDB4\xC1\x817!P',"

From this point it's easy.  I calculate the hash of the binary data, and compare it with
the (reversed) target.  Both of them are hexa numbers:

Code:

The hash of the binary data:
"bd702df8e1212ab65f23972dfd3b1b601b132efb61a09e4523acdbaefae80e73"

Target to compare with:
"00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff"

Result: BAD NEWS: the hash is greater than the target :(

So, my program tells me that it can't be the good solution, because bd702df8e... > 00000000f...

Where's the mistake?  Or:  where are the mistakes?

[2112]

I do the endian change first (I guess that's the point where I screw up things),

Among Bitcoiners the words "big endian" mean different thing than anywhere else. In a regular world computer scientists would call it "confused endian", "accidental endian", "crazy endian", "beginner's mistake endian", "consultant padding billable hours endian", "we made a mistake and are ashamed to admit it endian", "deranged endian", "tragicomic endian", etc.

It is a representation where a bignum quantity is first divided into words 4 bytes wide. The words are ordered starting with most significant word at the lowest position. The bytes within the words are ordered with the least significant byte at the lowest position.

Edit: OK, I've found a good name for this representation: mojibake endian. It should go well with Satoshi.

http://en.wikipedia.org/wiki/Mojibake

[mskwik]

Yes, working with the strange endian-ness of the various numbers is one of the "interesting" challenges in working with bitcoin code.  A quick test suggests you should get something like this with that particular test data:

Code:

mskwik@mskwik ~ $ perl
use Digest::SHA;
$hash="00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e"."8e631c12";
while($hash){$xx=substr($hash,0,8);$xx=reverse(pack('H*',$xx));$fixedhash.=$xx;$hash=substr($hash,8);}
$solution=Digest::SHA::sha256_hex(Digest::SHA::sha256($fixedhash));
print $solution."\n";
$fixedsol=reverse(unpack('h*',pack('H*',$solution)));
print $fixedsol."\n";
3246917b692937fbceb2376ba36a8f97794d33784a0472ae8a37061900000000
000000001906378aae72044a78334d79978f6aa36b37b2cefb3729697b914632

[Schleicher]

This whole endianess mess is mostly caused by developers doing pointer voodoo in C++.
Just in case you are still confused:
You have to switch the endianness 4 bytes at a time.
First you have, for example, "0123456789abcdef".
After switching the endianness you get "67452301efcdab89".

[dimitry-js]

Take a look at webworker ( JS bitcoin miner ).

[sdp]

Big endian is intuitively more natural than little endian.  It's analogous to how we write numbers.  Some old computers use big endian internally and so does Java.  It's also used as a standard network endian.  There are routines for converting to and from big endian but you have to roll your own for converting to and from little endian.

[sir.humus]

OK, OK, don't yell at me. I know this topic is really old.
But I was hoping someone here will be able to give me some help.

I tried to implement my own miner (just to understand mining).
I implemented it using getwork (I know it's old, but it was simpler for me)
so to check that it works I used the same example as here:

Code:

{"hash1"=>"00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000", "target"=>"ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000", "submitold"=>true, "identifier"=>"91138", "data"=>"00000002fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c8245032580a1a07a85e00000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000", "midstate"=>"03ad4305c1cad2bf14a99b82f3557b5722a060d6ac207450e939cb9f8143a605"}

Or in a more human-readable format the essence of this is the data and the target:

Code:

Data:
00000002 fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000 db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824 5032580a 1a07a85e 00000000 000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000

Target:
ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000

We also know that the right solution is:  (nonce = 0x8e631c12)

Code:

Solution:
00000002 fc517a2df2b283474b135215a00604af276318262f5eebc00000043100000000 db9fcfcc3781c1342c2750214e46407286cbf29985e688d0392e6b8005c4c824 5032580a 1a07a85e [b]8e631c12[/b]
000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000

The Nounce that I get is:
0x000141D0

And the hash of the binary data that I get is:
"0000075B2D7B45479BDE7D0A073900AE825F69C9FF16084CF83BB45B4F7B63C9"

Target to compare with:
"00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff"

Result: BAD NEWS: the calculated nounce is different and the hash is greater than the target

Where's the mistake?  Or:  where are the mistakes?