[BOUNTY] fix mobile WLCj wallet for android

[hexafraction]

I can help u PM me

Cryptosun, there's a link to the code. Run it and see if you can fix the issue.

[cryptosun]

I can help u PM me

Cryptosun, there's a link to the code. Run it and see if you can fix the issue.

Oh Ok bro:)

[rik8119]

I can help u PM me

Cryptosun, there's a link to the code. Run it and see if you can fix the issue.

Oh Ok bro:)

Hi cryptosun,

thanks for your offer. Hope you can fix the issue with bitcoin as parent while merge mining.

Rik

[hexafraction]

I can help u PM me

Cryptosun, there's a link to the code. Run it and see if you can fix the issue.

Oh Ok bro:)

Please keep us informed if you manage to make any progress, where I haven't been able to do so.

[achow101]

I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

[hexafraction]

I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.

[achow101]

So I have done some debugging and I found the problem to be at Line 505 in message.java with the line

Code:

System.arraycopy(payload, cursor, b, 0, length);

which is called at line 43 of TransactionInput.java. It appears that the the length passed in is too long since the cursor + length exceeds the length of payload. Either the length passed in is too large, or the cursor is positioned too far back. I will investigate some more the find out.

Edit: The length of the payload is 10430, the cursor is at 10177, and the length is 34966. 34966 + 10177 =45143 > 10430.

[achow101]

How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

[rik8119]

How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

The Blockexplorer can be found at: 109.73.173.119:81 (the balances are in satoshi not in wlc).

I try to find out more about the merge mining data in the block, can you specify a little bit what information you need?

[hexafraction]

How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

I'm not sure, but https://github.com/WorldLeadCurrency/WLC/blob/master/src/auxpow.cpp might help you. I can't read others' C++ very well, however.

[achow101]

How does the coin format the merge mining data in the block? Also, is there a block explorer I can use for this coin? That would make it much easier to figure this whole thing out.

The Blockexplorer can be found at: 109.73.173.119:81 (the balances are in satoshi not in wlc).

I try to find out more about the merge mining data in the block, can you specify a little bit what information you need?

Thanks. The blockexplorer helps. It gives me the data I need, which is the raw format of the blocks in a human-readable format.

[achow101]

I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.

I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.

[hexafraction]

I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.

I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.

Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?

[achow101]

I think I may have found where the problem is, but I will need to do some testing. This is just from looking at the code. I think it has to do with VarInt and the readVarInt() method in message.java. This method is called to get the length of the array and I think it isn't doing something correctly which causes the length to be too small. I need to check this out with debugging though.

Hexafraction, can you send me the main class that you are using?

readVarInt isn't the culprit, as far as I can tell. It's being called when the array pointer is already pointing in an invalid location. I'll pastebin the main class I'm using shortly.

Edit: http://paste.ubuntu.com/12112739/
It's nothing fancy, but it fetches blocks and gives me a debuggable program. Chain is static so the debugger can easily access it for expression evaluation and conditional breakpoints.

I think the error is definitely in VarInt or readVarInt because at line 259 of Block.java returns number of transactions to be 132 when there is only 1 and in line 590 of Transaction.java it also returns 132 from readVarInt(). The specific ArrayIndexOutOfBounds error is caused by the length being too long, which is also set by ReadVarInt(). I am going to take a look at VarInt to see whether I can find the problem.

Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?

The data for block 11418 is different because it is merge mined, which is probably causing the problem. Actually, it might not be VarInt but the offset passed into it. After a little more analysis, it looks like the offset is not set correctly, which is causing VarInt to return incorrect numbers. I could be wrong.

[rik8119]

Why is reading varints for the last thousands of blocks OK? Is the format different for that specific piece of data? Or are the parser reads misordered, misaligned to the actual structure, or calling for a varint where the format isn't a varint?

The data for block 11418 is different because it is merge mined, which is probably causing the problem. Actually, it might not be VarInt but the offset passed into it. After a little more analysis, it looks like the offset is not set correctly, which is causing VarInt to return incorrect numbers. I could be wrong.

https://github.com/FreicoinAlliance/freicoinj/commit/8c27b562cd814f2a6a026e76f0e11bcc0960ab4d

As you can see in this hash the offset for transactions has to be changed because of the extra property "refHeight" of the transactions of the coin.

Merged mining with freicoin as parent chain is working ok (Blocks 1 - 1405).

When merge mining with bitcoin as parent starts, the offset is suddenly wrong,
so maybe the wallet is expecting also the parent chain to have refHeight in its transactions?

I would like to test it somehow but right now i have no clue how to implement a solution.

Maybe something like:

If (length !="expected value")
    length = "value without refHeight";   

[hexafraction]

I might have an idea on how this could be done. I think there should be extra fields to differentiate the merged chain used, probably a version or something. I don't think we can just check an expected value in a neat way. I'm going to try it shortly, and see if my idea works in practice.

[hexafraction]

I managed to sync successfully up to height 22103, by modifying org.bitcoinj.core.Transaction.

Edit: Now synced to 22194 before crashing. I think I've managed to sync almost all of the chain, however it's rejecting the last block with an invalid difficulty. The C++ client accepts it, and the two clients are in agreement about difficulty.

The parse() method in that class should now be:

Code:

 @Override
    void parse() throws ProtocolException {

        if (parsed)
            return;

        cursor = offset;

        version = readUint32();
        optimalEncodingMessageSize = 4;

        // First come the inputs.
        long numInputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numInputs);
        inputs = new ArrayList<TransactionInput>((int) numInputs);
        for (long i = 0; i < numInputs; i++) {
                TransactionInput input = new TransactionInput(params, this, payload, cursor, parseLazy, parseRetain);
                inputs.add(input);
                long scriptLen = readVarInt(TransactionOutPoint.MESSAGE_LENGTH);
                optimalEncodingMessageSize += TransactionOutPoint.MESSAGE_LENGTH + VarInt.sizeOf(scriptLen) + scriptLen + 4;
                cursor += scriptLen + 4;
        }
        // Now the outputs
        long numOutputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numOutputs);
        outputs = new ArrayList<TransactionOutput>((int) numOutputs);
        for (long i = 0; i < numOutputs; i++) {
            TransactionOutput output = new TransactionOutput(params, this, payload, cursor, parseLazy, parseRetain);
            outputs.add(output);
            long scriptLen = readVarInt(8);
            optimalEncodingMessageSize += 8 + VarInt.sizeOf(scriptLen) + scriptLen;
            cursor += scriptLen;
        }
        lockTime = readUint32();
        optimalEncodingMessageSize += 4;
        if(this.version==2) {
            refHeight = readUint32();
            optimalEncodingMessageSize += 4 + 4;
        }
        length = cursor - offset;
    }

Here's the new error:

Code:

org.bitcoinj.core.VerificationException: Could not verify block f26842324aca22db7c16ddf087277ee99117d0fa29ca5d5f6eaaaf4d42d9d8f0
v1114369 block: 
   previous block: c85bfc960881fa6cfa833feef10e2842fe7b56f56aa962fdd0e2300b59c4da3e
   merkle root: 505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a
   time: [1440012311] Wed Aug 19 15:25:11 EDT 2015
   difficulty target (nBits): 436643261
   nonce: 0
   Merged-mining info: 
      version: v1114369
      time: [Wed Aug 19 15:25:11 EDT 2015] Wed Aug 19 15:25:11 EDT 2015
      difficulty target (nBits): 436643261
      nonce: 0
      Network: org.worldleadcurrency.production
      parent block coin base transaction: 
  679a73c3467721d6fc3ffdcf6865da6b0283d50664b1d2914d67b111bbe072fd: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(3)[a4a705] PUSHDATA(44)[fabe6d6d7f15c02dd05c3eacefde83dd74b0f9b5ed7155c9ec92ef280acdda2bacd1f5c32000000000000000])  (scriptPubKey DUP HASH160 PUSHDATA(20)[3d47986f727402feb3df5134dff583af4ace8074] EQUALVERIFY CHECKSIG)

      coinbase link: 
          hash of parent block header: 000000000000023a97f467afb051ac1480e27bed9788e8fdb9ac2d892614f551
      parent block header: 
v3 block: 
   previous block: 00000000000000000759c455cd9b9402a461885fe76e0261092a5cd7a575038d
   merkle root: cac502a2d3a6865430f959b445ff563b4bcd49f60ab9ad8579d6ab54ff829108
   time: [1440012417] Wed Aug 19 15:26:57 EDT 2015
   difficulty target (nBits): 404020484
   nonce: 3237608803


   with 1 transaction(s):
  505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(4)[bda5061a] PUSHDATA(1)[01] 2)  (scriptPubKey PUSHDATA(33)[02bc41065c462d75844cf77624b712a29064cd21630e9b8c3c68990e4881a9b58e] CHECKSIG)

	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:286)
	at org.bitcoinj.core.Peer.processBlock(Peer.java:872)
	at org.bitcoinj.core.Peer.processMessage(Peer.java:360)
	at org.bitcoinj.core.PeerSocketHandler.receiveBytes(PeerSocketHandler.java:178)
	at org.bitcoinj.net.ConnectionHandler.handleKey(ConnectionHandler.java:217)
	at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:75)
	at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:111)
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
	at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.bitcoinj.core.VerificationException: Hash is higher than target: 13e6b9f88b5a8bd9a36adb475588d6108d8594b8eb37c5122ae70d3c4beface7 vs 6a5bd0000000000000000000000000000000000000000000000
	at org.bitcoinj.core.Block.checkProofOfWork(Block.java:766)
	at org.bitcoinj.core.Block.verifyHeader(Block.java:886)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:397)
	at org.bitcoinj.core.AbstractBlockChain.tryConnectingOrphans(AbstractBlockChain.java:825)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:424)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:275)
	... 9 more

[achow101]

I managed to sync successfully up to height 22103, by modifying org.bitcoinj.core.Transaction.

Edit: Now synced to 22194 before crashing. I think I've managed to sync almost all of the chain, however it's rejecting the last block with an invalid difficulty. The C++ client accepts it, and the two clients are in agreement about difficulty.

The parse() method in that class should now be:

Code:

 @Override
    void parse() throws ProtocolException {

        if (parsed)
            return;

        cursor = offset;

        version = readUint32();
        optimalEncodingMessageSize = 4;

        // First come the inputs.
        long numInputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numInputs);
        inputs = new ArrayList<TransactionInput>((int) numInputs);
        for (long i = 0; i < numInputs; i++) {
                TransactionInput input = new TransactionInput(params, this, payload, cursor, parseLazy, parseRetain);
                inputs.add(input);
                long scriptLen = readVarInt(TransactionOutPoint.MESSAGE_LENGTH);
                optimalEncodingMessageSize += TransactionOutPoint.MESSAGE_LENGTH + VarInt.sizeOf(scriptLen) + scriptLen + 4;
                cursor += scriptLen + 4;
        }
        // Now the outputs
        long numOutputs = readVarInt();
        optimalEncodingMessageSize += VarInt.sizeOf(numOutputs);
        outputs = new ArrayList<TransactionOutput>((int) numOutputs);
        for (long i = 0; i < numOutputs; i++) {
            TransactionOutput output = new TransactionOutput(params, this, payload, cursor, parseLazy, parseRetain);
            outputs.add(output);
            long scriptLen = readVarInt(8);
            optimalEncodingMessageSize += 8 + VarInt.sizeOf(scriptLen) + scriptLen;
            cursor += scriptLen;
        }
        lockTime = readUint32();
        optimalEncodingMessageSize += 4;
        if(this.version==2) {
            refHeight = readUint32();
            optimalEncodingMessageSize += 4 + 4;
        }
        length = cursor - offset;
    }

Here's the new error:

Code:

org.bitcoinj.core.VerificationException: Could not verify block f26842324aca22db7c16ddf087277ee99117d0fa29ca5d5f6eaaaf4d42d9d8f0
v1114369 block: 
   previous block: c85bfc960881fa6cfa833feef10e2842fe7b56f56aa962fdd0e2300b59c4da3e
   merkle root: 505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a
   time: [1440012311] Wed Aug 19 15:25:11 EDT 2015
   difficulty target (nBits): 436643261
   nonce: 0
   Merged-mining info: 
      version: v1114369
      time: [Wed Aug 19 15:25:11 EDT 2015] Wed Aug 19 15:25:11 EDT 2015
      difficulty target (nBits): 436643261
      nonce: 0
      Network: org.worldleadcurrency.production
      parent block coin base transaction: 
  679a73c3467721d6fc3ffdcf6865da6b0283d50664b1d2914d67b111bbe072fd: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(3)[a4a705] PUSHDATA(44)[fabe6d6d7f15c02dd05c3eacefde83dd74b0f9b5ed7155c9ec92ef280acdda2bacd1f5c32000000000000000])  (scriptPubKey DUP HASH160 PUSHDATA(20)[3d47986f727402feb3df5134dff583af4ace8074] EQUALVERIFY CHECKSIG)

      coinbase link: 
          hash of parent block header: 000000000000023a97f467afb051ac1480e27bed9788e8fdb9ac2d892614f551
      parent block header: 
v3 block: 
   previous block: 00000000000000000759c455cd9b9402a461885fe76e0261092a5cd7a575038d
   merkle root: cac502a2d3a6865430f959b445ff563b4bcd49f60ab9ad8579d6ab54ff829108
   time: [1440012417] Wed Aug 19 15:26:57 EDT 2015
   difficulty target (nBits): 404020484
   nonce: 3237608803


   with 1 transaction(s):
  505a544293220d73c2f1b1ca3285906828fc8abcc39aa41a3f4155fc347b3c3a: Unknown confidence level.
     == COINBASE TXN (scriptSig PUSHDATA(4)[bda5061a] PUSHDATA(1)[01] 2)  (scriptPubKey PUSHDATA(33)[02bc41065c462d75844cf77624b712a29064cd21630e9b8c3c68990e4881a9b58e] CHECKSIG)

	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:286)
	at org.bitcoinj.core.Peer.processBlock(Peer.java:872)
	at org.bitcoinj.core.Peer.processMessage(Peer.java:360)
	at org.bitcoinj.core.PeerSocketHandler.receiveBytes(PeerSocketHandler.java:178)
	at org.bitcoinj.net.ConnectionHandler.handleKey(ConnectionHandler.java:217)
	at org.bitcoinj.net.NioClientManager.handleKey(NioClientManager.java:75)
	at org.bitcoinj.net.NioClientManager.run(NioClientManager.java:111)
	at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60)
	at com.google.common.util.concurrent.Callables$3.run(Callables.java:93)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.bitcoinj.core.VerificationException: Hash is higher than target: 13e6b9f88b5a8bd9a36adb475588d6108d8594b8eb37c5122ae70d3c4beface7 vs 6a5bd0000000000000000000000000000000000000000000000
	at org.bitcoinj.core.Block.checkProofOfWork(Block.java:766)
	at org.bitcoinj.core.Block.verifyHeader(Block.java:886)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:397)
	at org.bitcoinj.core.AbstractBlockChain.tryConnectingOrphans(AbstractBlockChain.java:825)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:424)
	at org.bitcoinj.core.AbstractBlockChain.add(AbstractBlockChain.java:275)
	... 9 more

Yup. Getting the same error as well. I think I know what the problem is. For some reason (can't find it yet) it sets all of the headerParsed boolean to true except for the last block. Then, for the last block, it parses the header, and finds that the hash of the header is not less than the target. To show this, if you convert the last block's header (22214) to a decimal integer, you get

Code:

f76c22e3179b613366e6098d5a963df3147a140684377cb04babfa31615e839d -> 111912333864832649963788629443113007904287670404150873962674491627731299500957

and the bits (1a6a5bd) is

Code:

6a5bd0000000000000000000000000000000000000000000000 -> 10681984318408322427192353450286343108761369678138805919416320

For a valid block, subtracting the block integer from the bits integer should result in a positive number (hash is less than difficulty). However, in this block (and many others) the hash is an integer that is GREATER than the bits. Therefore I conclude that there is something wrong with how WLC is merged mining its blocks.

Of course this could all be wrong. Please not that I have not yet taken a look at the c++ code yet, but I am running the client.

Nevermind, the above is semi wrong. I forgot that merged mining checks the parent block hash and not the actual block hash. The problem is that it is not setting the parent block hash as the hash, but is using the normal hash. The problem is specifically at the if block at line 732 in Block.java. I am investigating to see what is causing it to fail.

[rik8119]

Cool, i would have thought that the transactionversion is not set before parsing, but as long as it works  .

The difficulty fail is really unexpected. Why is it only the last block? Totally confusing. The check for the merged mining work is copy-paste from devcoin (which copy-pasted it from namecoin) and no rocket science. As you said it is taking the blockhash from the parent chain as POW. So i would exclude mistakes in the c++ wallet if there are no further hints.

So either the diffucultyfilter is wrong for all the blocks (or all merge miningblocks) but the last,
or the difficultyfilter is correct but fails to validate the most current block.

To me the last option is the most probalbe which throws up the question:
what is special about the current block that hinders the wallet to get the right hash?

-----------------------------------------------------------------
Ok, of course it is set before parsing, but i thought the "new Transaction" in BlockMergeMinedPayload is setting it to Version 2 and that is the problem

[rik8119]

So it seems the merge mining information can not be found mmBlock == null returns true.

I dont know why but the c++ wallet somehow handles the newly generated blocks in another way than namecoin and devcoin.