Print Page - parsing getwork blockheaders with BitCoinJ

Title: parsing getwork blockheaders with BitCoinJ
Post by: shads on June 24, 2011, 11:59:26 AM

So what are the extra bytes?

I've tried feeding the getwork data field into the BitCoinJ Block class for parsing but I get rubbish results:

This is the JSON result:

Code:

{
    "error": null,
    "id": 2,
    "result": {
        "data": "000000013aef333a9788611a61cee16e0d555b989015425446f0776c00000478000000009d0221ae1fe47f5963e1b868dbe6388070888400ffcce91c7b60e4f157ed70f44e047b5d1a0c2a1200000000000000800000000000000000000000000000000000000000000000000000000000000000000000000000000080020000",
        "hash1": "00000000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000000010000",
        "midstate": "b5160a8f1c6ade7fc3cac93d31f5f34a5d53faa48b04fa549126c7d6cd4fe791",
        "target": "0000000000000000000000000000000000000000000000122a0c000000000000"
    }
}

This is what I get from parsing it with BitCoinJ:

Code:

size: 128
difficulty: 304745498
nonce: 0
time: 1568343118
My system time: 1308916582
version: 16777216
hash: fa336f7e89b73b44ca1ec41b597c03f92c1e9e68a06789d10517ed1fe9cd9c38

The only thing that seems correct is the nonce.

I've tried reversing the bytes but still rubbish. Should I be offsetting perhaps?

Title: Re: Block header should be 81 bytes but getwork:data is 128 bytes
Post by: kjj on June 24, 2011, 01:26:41 PM

The block is right. The parser is wrong. SHA256 works on 64 byte blocks, so messages get padded and salted before hashing.

Title: Re: Block header should be 81 bytes but getwork:data is 128 bytes
Post by: shads on June 24, 2011, 01:45:10 PM

ok, so do you know if they are padded at the beginning or end? Is it just a simple matter of trimming the byte array? The parser is probably not designed to handle the results of a getwork. According to it's constructor:

/** Constructs a block object from the BitCoin wire format. */

Which probably doesn't include the padding I'm guessing.

Title: Re: Block header should be 81 bytes but getwork:data is 128 bytes
Post by: kjj on June 24, 2011, 02:20:34 PM

I couldn't tell without looking at the function, and I despise Java, so... Hopefully someone familiar with bitcoinj will pop in and help. Maybe change the subject of the thread.

Usually the first problem people run into when trying to parse/hash this stuff is the difference between network byte order and host byte order.

Title: Re: Block header should be 81 bytes but getwork:data is 128 bytes
Post by: Mike Hearn on June 24, 2011, 04:27:56 PM

Yes, that constructor isn't intended to take the output of getwork.

Presumably the problem is the following code from FormatHashBuffers:

Code:

    // Byte swap all the input buffer
    for (int i = 0; i < sizeof(tmp)/4; i++)
        ((unsigned int*)&tmp)[i] = ByteReverse(((unsigned int*)&tmp)[i]);

But what are you trying to do? If you want to build a miner I don't think you don't have to parse the output of getwork. The getwork protocol is something else.

Title: Re: parsing getwork blockheaders with BitCoinJ
Post by: shads on June 25, 2011, 12:00:18 AM

I thought it might be a problem with endianess. Never had to deal with it before in my little java bubble.

Most likely use is to parse the returned getwork to verify it before passing it on upstream. Or possibly even to generate work for downstream clients. I know bitcoinj is really focussed on being an end user client atm so not really built with those sort of things in mind but seemed sensible to try an leverage some of the code and it's useful particularly as a learning exercise.

p.s. Mike, the javadoc and comments are great, I've learned more from those than I have from the wiki.

p.p.s. I've search my entire eclipse workspace for references to FormatHashBuffers and even done a text searches for the code snippets but not there. I've just checked out the trunk so if it's part of the project I should have it? -- whoops looked a bit closer and that's not java...

Title: Re: parsing getwork blockheaders with BitCoinJ
Post by: shads on June 25, 2011, 04:05:47 AM

got it... had to brute force it with ever permutation I could think of but finally got an almost result:

Code:

	public static void checkByteSwapped(String data) {
		byte[] bytes = Hex.decode(data);
		byte[] rev = new byte[bytes.length];
		for (int i = 0; i < bytes.length; i += 4) {
			byte[] chunk = Arrays.copyOfRange(bytes, i ,  i  + 4);
			chunk = Utils.reverseBytes(chunk);
			System.arraycopy(chunk, 0, rev, i , 4);
		}
		checkBytes(rev);
	}

produces:

Code:

offset: 0
v1 block: 
   previous block: 00000000000008ee4076724291e1656d84ac61d98bf89dd3656680782aad1a4a
   merkle root: 8de49d9cc1dce6cf0a89b95cd5d683eb787e3f6a02072acf69ddab88b8ae01ff
   time: [1308974260] Sat Jun 25 13:57:40 EST 2011
   difficulty target (nBits): 437004818
   nonce: 0

which matches the last block from blockexplorer. The only part that doesn't match up is the difficulty target. Block explorer says: 1379223.4296725

but at least it's progress...

Title: Re: parsing getwork blockheaders with BitCoinJ
Post by: shads on June 25, 2011, 05:03:57 AM

ok so I've just learned about compact form vs long form... and I seem to be decoding difficulty close but not precise:

Code:

					L.println("************************************************************************");
					L.println("offset: " + i);
					L.println(block.toString());
					String diff = block.getDifficultyTargetAsInteger().toString(16);
					long highestDifficult = 0x1d00ffff;
					BigDecimal highestDifficulty = new BigDecimal(Utils.decodeCompactBits(highestDifficult));
					diff = StringUtils.leftPad(diff, 64, '0');
					L.println("max diff: " + StringUtils.leftPad(Utils.decodeCompactBits(highestDifficult).toString(16), 64, '0'));
					L.println("hex diff: " + diff);
					BigDecimal ratio = highestDifficulty.divide(new BigDecimal(block.getDifficultyTargetAsInteger()), 7, RoundingMode.FLOOR);
					L.println("ratio: " + ratio.toPlainString());
					L.println("************************************************************************");

gives me:

Code:

************************************************************************
offset: 0
v1 block: 
   previous block: 00000000000003fc392bab7a327a019fce4b48c98d5bc4b0d8d20f6e21e4ce1f
   merkle root: 2891334d6466ea274ac50cdb2af50afc889d236689cfe6ace9682b752900985a
   time: [1308977962] Sat Jun 25 14:59:22 EST 2011
   difficulty target (nBits): 437004818
   nonce: 0

max diff: 00000000ffff0000000000000000000000000000000000000000000000000000
hex diff: 0000000000000c2a120000000000000000000000000000000000000000000000
ratio: 1379192.2882280
************************************************************************

which is pretty damn close to blockexplorer: 1379223.4296725

It seems odd that difference is very close to 32 (31.1414). Perhaps I've mashed a bit somewhere.

Title: Re: parsing getwork blockheaders with BitCoinJ
Post by: shads on June 25, 2011, 05:57:40 AM

excuse me continuing to talk to myself...

Just notices that my bitcoind client is returning 1379192.28822808 from getdifficulty so it was decoded properly.

Odd though that blockexplorer is returning a different difficulty?

Title: Re: parsing getwork blockheaders with BitCoinJ
Post by: theymos on June 25, 2011, 06:41:40 AM

BBE's /q/getdifficulty uses Bitcoin's old difficulty calculation, which has some precision problems. You'll notice that the difficulty reported on block pages (http://blockexplorer.com/block/000000000000028b3c88a2744b602f86ddf9a5ed250c661ad83519b6e198238b) is the same as yours.

Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: shads on June 24, 2011, 11:59:26 AM