Getwork protocol - what are the rules? Examples?

[ThePiachu]

I`ve been trying to find some information on how does the getwork protocol actually look. The Wiki page
https://en.bitcoin.it/wiki/Getwork
doesn't provide nearly as much information as the standard protocol specification. Does someone have some detailed resources on how the protocol should look, preferably with some examples of valid messages?

[1715262677]

1715262677

[1715262677]

1715262677

[1715262677]

1715262677

[1715262677]

1715262677

[1715262677]

1715262677

[1715262677]

1715262677

[DrHaribo]

getwork with data is used to create a new block. getwork without parameters is used to get new work to hash.

Example getwork request:

{"method":"getwork","params":[],"id":1}

Example getwork response:

{"id": "1", "result": {"hash1": "0000000000000000000000000000000000000000000000000000000000000000000000800000000 0000000000000000000000000000000000000000000010000", "data": "00000001c570c4764aadb3f09895619f549000b8b51a789e7f58ea750000709700000000103ca06 4f8c76c390683f8203043e91466a7fcc40e6ebc428fbcc2d89b574a864db8345b1b00b5ac000000 0000000080000000000000000000000000000000000000000000000000000000000000000000000 0000000000080020000", "midstate": "e772fc6964e7b06d8f855a6166353e48b2562de4ad037abc889294cea8ed1070", "target": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffff00000000"}, "error": null}

Note: midstate has been deprecated and will be removed in some future version of bitcoind. You will then be required to calculate your own midstate in a miner. Also deprecated is "hash1" which is totally useless. "data" here is an 80 byte block header (you can find it described in the wiki) plus padding up to 128 bytes so it is ready for SHA-256 hashing as two 64-byte chunks.

Example getwork request sending in a proof of work:

{"method":"getwork","params":["0000000141a0e898cf6554fd344a37b2917a6c7a6561c20733b09c8000009eef00000000d559e21 882efc6f76bbfad4cd13639f4067cd904fe4ecc3351dc9cc5358f1cd54db84e7a1b00b5acba97b6 0400000080000000000000000000000000000000000000000000000000000000000000000000000 0000000000080020000"],"id":1}

Example response to the above:
{"result": false, "id": "1", "error": null}

The data sent in above is an 80 byte block header as described in the wiki, SHA-256-padded up to 128 bytes. It's the same "data" that you got from the server with an earlier getwork request, except you replaced the 4-byte nonce value.

Note: if you make software that fetches work from several servers and send proofs of work back to them, make sure you send a proof of work (block header with good nonce) back to the server from where you got that block header. Some software does this wrong, and such proofs of work are useless when delivered to the wrong server.

[kano]

To be specific, data = 20 x 4 byte numbers:

0x1 (version)
8 numbers (previous hash - reverse order)
8 numbers (transaction merkle root - reverse order)
time
difficulty
nonce

total 80 bytes

The reason for the 128 byte padding is that it matches the internal 2 x 64 byte process of sha256 hashing the 80 bytes
sha256 pads up to a multiple of 64 bytes with 0x80, 0x0 repeated then data size (80 = 80 bytes)

[DrHaribo]

Example response to the above:
{"result": false, "id": "1", "error": null}

Forgot to say what this means. Bitcoind returns true or false indicating whether it was able to use your proof of work to create a new block. Mining pools return true or false to indicate whether the proof of work from the miner is accepted or rejected.

[DeathAndTaxes]

What is/was the deprecated "midstate" used for?

[ThePiachu]

Okay, so what do the different fields contain and how are they calculated? From what I understand:

data - contains previous hash, merkle root, time, difficulty and nonce
are all the time and nonce ranges iterated by requesting more getwork, or are they given as some range?

target - the difficulty of 1?


But what is the id, hash1, midstate, what can the error be, what are the params when sending proof of work, etc? And how does one add long polling and other desired features into getwork request?

[kano]

Be aware, there is no standard document defining the meaning of what a pool returns.
Each piece of pool software does as it pleases to some extent.

[ThePiachu]

Well, since most mining software would probably adhere to Deepbit's standard, how does it define those fields?

[DrHaribo]

What is/was the deprecated "midstate" used for?

The midstate is the SHA-256 hashing state after hashing the first chunk (64) bytes of the block header. After you get a block header from the server, you typically hash the first chunk 1 time, and hash the second chunk 2^32 times going through all the possible nonce values.

http://en.wikipedia.org/wiki/Sha-256

At the link above the midstate is what you have after processing 1 chunk and adding that chunk's hash to the result so far (h0-h7). Because the nonce is in the second chunk, there is no need to hash the first chunk over and over every time you try a new nonce value.

data - contains previous hash, merkle root, time, difficulty and nonce
are all the time and nonce ranges iterated by requesting more getwork, or are they given as some range?

You can find the details about this at https://en.bitcoin.it/wiki/Protocol_specification

There are no ranges given by getwork as used by bitcoind. Some pool servers implement extensions you can read about here: https://en.bitcoin.it/wiki/Getwork

target - the difficulty of 1?

From most pool servers you'd get a target for difficulty 1. From bitcoind you'd get the actual bitcoin target at the time.

But what is the id, hash1, midstate, what can the error be, what are the params when sending proof of work, etc? And how does one add long polling and other desired features into getwork request?

id and error is standard JSON-RPC. You can read about JSON-RPC here: http://json-rpc.org/

Note that some miners are a bit dumb and won't really do much if they get a JSON-RPC error, like telling the user the actual message or even that something went wrong.

midstate I explained above. Long polling is explained on the wiki page on getwork extensions (link above).

params when sending in a proof of work is the 80-byte bitcoin block header + sha-256 padding up to 128-bytes. It should be exactly as the pool server or bitcoind gave it to you except for the nonce. The nonce should be modified in such a way that when hashing the block header and then hashing its hash, you end up with a hash that when interpreted as a number is below the target that the server or bitcoind gave you.

[kano]

To be specific, data = 20 x 4 byte numbers:

0x1 (version)
8 numbers (previous hash - reverse order)
8 numbers (transaction merkle root - reverse order)
time
difficulty
nonce

total 80 bytes

The reason for the 128 byte padding is that it matches the internal 2 x 64 byte process of sha256 hashing the 80 bytes
sha256 pads up to a multiple of 64 bytes with 0x80, 0x0 repeated then data size (80 = 80 bytes)

Correction
It's (80 << 3)
i.e. 0x50 << 3 = 0x280
thus when you reverse it, it looks like 0x80020000

[ThePiachu]

Hmm, I've calculating midstate, but I'm having some problems with it. From what I understand, I should take

00000001c570c4764aadb3f09895619f549000b8b51a789e7f58ea750000709700000000103ca06 4f8c76c390683f8203043e91466a7fcc40e6ebc428fbcc2d8

from the example, and perform SHA-256 on it. It gives me a result of:

12E84A43CBC7689AE9916A30E1AA0F3CA12146CBF886B60103AEC21A5CFAA268

if I SHA-256 a reverse of the first string, I get:

63A2E2A3DD0943D8D7A30FFB296D384044AA08F2C4EE086C2ED7D552BDC1BF8F

While the proper midstate is:

e772fc6964e7b06d8f855a6166353e48b2562de4ad037abc889294cea8ed1070

What am I doing wrong?

[DrHaribo]

Look at http://en.wikipedia.org/wiki/SHA-2, under pre-processing. You will see that running a full sha-256 implementation will add some data to your input. It will add that, hash 2 chunks, and give you the wrong result. Unless you can force your sha-256 implementation to only hash one chunk of data and nothing more, you'll have to write a function that will.

Also you should not take the reverse of the string. For each group of 4 bytes you should reverse the order of the bytes. So "0102030405060708" becomes (0x04030201, 0x08070605, ...). If you are going to send the result as part of a getwork response then you will have to do the same on output. You will find this little-endian mess all over bitcoin code and APIs, sometimes mixed with big-endian to make sure you get confused.

If midstate() is a function that takes an array of integers and returns an array of integers:

Code:

scala> midstate("00000001c570c4764aadb3f09895619f549000b8b51a789e7f58ea750000709700000000103ca064f8c76c390683f8203043e91466a7fcc40e6ebc428fbcc2d8"
.grouped(8).map(x => Integer.reverseBytes(java.lang.Long.parseLong(x,16).toInt)).toArray).map(x => "%08x".format(Integer.reverseBytes(x))).mkString("")
res721: String = e772fc6964e7b06d8f855a6166353e48b2562de4ad037abc889294cea8ed1070

Note the Integer.reverseBytes() on both input and output to reverse the order of the bytes within each integer.

[ThePiachu]

So instead of calling

SHA(00000001c570c4764aadb3f09895619f549000b8b51a789e7f58ea750000709700000000103ca06 4f8c76c390683f8203043e91466a7fcc40e6ebc428fbcc2d8)

I should call

Midstate(8FBCC2D80E6EBC4266A7FCC43043E9140683F820F8C76C39103CA06400000000000070977F58EA7 5B51A789E549000B89895619F4AADB3F0C570C47600000001)

or

Midstate(0100000076C470C5F0B3AD4A9F619598B80090549E781AB575EA587F977000000000000064A03C1 0396CC7F820F8830614E94330C4FCA76642BC6E0ED8C2BC8F)

?

[DrHaribo]

Midstate(0100000076C470C5F0B3AD4A9F619598B80090549E781AB575EA587F977000000000000064A03C1 0396CC7F820F8830614E94330C4FCA76642BC6E0ED8C2BC8F)

Yep, that looks correct.

[ThePiachu]

Very complicated, but I managed to get it running - http://stackoverflow.com/questions/9245235/golang-midstate-sha-256-hash . Seriously, who comes up with these things, and why isn't there some "Bitcoin Bible" yet on how to program and calculate such weird things...

[kano]

Very complicated, but I managed to get it running - http://stackoverflow.com/questions/9245235/golang-midstate-sha-256-hash . Seriously, who comes up with these things, and why isn't there some "Bitcoin Bible" yet on how to program and calculate such weird things...

Read the bitcoind code - it already does it of course ...

[DrHaribo]

why isn't there some "Bitcoin Bible" yet on how to program and calculate such weird things...

Maybe now you can write one.

Yeah, like you say, kano, you basically have to read the source code, because the documentation is a bit slim.

But if you thought this was bad, you should try implementing merged mining.

[optimator]

Sorry for resurrecting an older topic. I'm trying to follow along, but I still don't get it.

I've taken the first 64 bytes of the "data" string and swapped the bytes around. Now I need to create a byte array to use as input for the SHA256 function.

Here's what I'm going to hash: 0100000076C470C5F0B3AD4A9F619598B80090549E781AB575EA587F977000000000000064A03C1 0396CC7F820F8830614E94330C4FCA76642BC6E0ED8C2BC8F

Do I create the byte array by converting each group of 2 characters from hex to decimal? (e.g. byte[] array = {1,0,0,0,118....})

Or do I take a group of (4 or 8 ) characters convert that hex number to an int32 and then convert the int32 into a byte array?

[ThePiachu]

What do you want to hash, and where to use it? Do you want to use the hash in the getwork protocol (calculating midstate or something else), or something else? From what I found, is that Bitcoin hashes different things differently.

[optimator]

Specifically I'd like to learn how to calculate the midstate.

Generally, I'm trying to learn a little of everything. I'm trying to understand the bitcoin protocol and specifically getwork -  it's hard.

Today, I was able to write a recursive method that takes an array of hashes and calculates the merkle root - That felt really good.

I know the midstate is deprecated, but I'd still like to understand the calculation. It seems that I use this string

0100000076C470C5F0B3AD4A9F619598B80090549E781AB575EA587F977000000000000064A03C1 0396CC7F820F8830614E94330C4FCA76642BC6E0ED8C2BC8F

but then I can't understand how to get the midstate. I *think* each group of 8 characters represents an unsigned 32 bit integer. But then I get lost.