pushpool's client developing issues

[elmigranto]

Developing pushpool client

Hello,
I'm really interested in jgarzik's pushpool (it's binary protocol in particular) and writing Qt-based client.

Problem is that I can't make it through LOGIN step.
As far as I undestand from forum's threads and (mostly) pushpool's source code, I need to send ubbp_header with BC_OP_LOGIN operation code and message's length in bytes. Message is SHA256 of compressed json updated with userpass following compressed json.

Json looks something like this (protocol is always 1, config is always null):

Code: (json)

{"protocol":1, "user":"username", "config":null}

And I'm doing following (higlighted version):

Code:

    ubbp_header header;
    strcpy((char*)header.magic, PUSHPOOL_UBBP_MAGIC);

    QSocket socket;
    socket.connectToHost("my.pushpool.host", 8336);
    socket.waitForConnected();

    // add compressed json to data
    dataToSend = json.toUtf8();
    dataToSend = (qCompress(dataToSend, 9));

    // Since qCompress() adding length (first 4 bytes in dataToSend) in Big-Endian,
    // but server uses le32toh() to get it, change endianness to Little-Endian:
    dataToSend.remove(0, 4);
    uint32_t len = json.length();
    dataToSend.insert(0, (char*)&len;, sizeof(uint32_t));

    // add compressed json digest
    char pass[] = "test";
    uchar md[SHA256_DIGEST_LENGTH];
    SHA256_CTX ctx;
    SHA256_Init(&ctx;);
    SHA256_Update(&ctx;, dataToSend.data(), dataToSend.size());
    SHA256_Update(&ctx;, pass, strlen(pass));
    SHA256_Final(md, &ctx;);
    dataToSend.append((char*)md, SHA256_DIGEST_LENGTH);

    // Send operation code & message size
    header.op_size = UBBP_OP_SIZE(BC_OP_LOGIN, dataToSend.size());
    socket.write((char*)&header;, sizeof(ubbp_header));
    socket.waitForBytesWritten();

    // Send message
    socket.write(dataToSend);
    socket.waitForBytesWritten();

    // Here's a problem:
    // after message sent, I get
    // QAbstractSocket::error() with 'Remote host closed connection'

My assumptions:

• JSON compressed by qCompress() cannot be uncompressed by zlib (even after my manipulation)
• I'm calculating wrong SHA256
• Something wrong with json (i.e. config must not be null)

After puzzling for almost whole day, I'm writing here.
Guys, please help.

[1713939453]

1713939453

[1713939453]

1713939453

[elmigranto]

By printing out some debug info I've find out that's problem is uncompressing on server-side.

I'm trying to do the following using Qt-built-in zlib (highlighted version):

Code:

    char json[] = "{\"version\":1,\"user\":\"test\"}";

    std::auto_ptr<Bytef> message(new Bytef[             // allocate memory for:
                                   sizeof(ubbp_header)  //  + ubbp_header
                                 + sizeof(uLongf)       //  + uncompressed length
                                 + strlen(json)         //  + message
                                 + SHA256_DIGEST_LENGTH //  + sha256
                                 + 64]);                //  + reserve (if compressed size > uncompressed size)

    uLongf unc_len = strlen(json);
    uLongf comp_len = unc_len + 64;

    Bytef* pHeader  = message.get(); // header
    Bytef* pLen     = pHeader + sizeof(ubbp_header); // length of uncompressed data
    Bytef* pCompDat = pLen + sizeof(uLongf); // compressed data
    Bytef* pSha256; // digest

    if (Z_OK != compress2(pCompDat, &comp_len, (Bytef*)json, unc_len, 9))
    {
        qDebug("Compression failed");
        return false;
    }

    /* calculating message digest */
    char pass[] = "test";
    uchar md[SHA256_DIGEST_LENGTH];
    SHA256_CTX ctx;
    SHA256_Init(&ctx);
    SHA256_Update(&ctx, pLen, comp_len + sizeof(uLongf));
    SHA256_Update(&ctx, pass, strlen(pass));
    SHA256_Final(md, &ctx);

    /* building message */
    uLongf msg_len = sizeof(ubbp_header) + sizeof(uLongf) + comp_len + SHA256_DIGEST_LENGTH;
    header.op_size = UBBP_OP_SIZE(BC_OP_LOGIN, msg_len - sizeof(ubbp_header));

    pSha256 = pCompDat + comp_len;
    memcpy(pHeader, &header, sizeof(ubbp_header));
    memcpy(pLen, &unc_len, sizeof(uLongf));
    memcpy(pSha256, &md, SHA256_DIGEST_LENGTH);

    /* sending message */
    socket.write((char*)message.get(), msg_len);
    socket.waitForBytesWritten();

And still server-side uncompress() (located in cjson_decode()) doesn't returns Z_OK.
Where's mistake?

[elmigranto]

uncompress() actually returns Z_BUF_ERROR indicating the output buffer was too small.
I don't understand how it can be since server allocating uncompressed_length + 1 bytes which obviously enough.

I'm pretty sure that sent uncompressed_length is correct.

[elmigranto]

Turns out it was server-side bug. In cjson_decode() function located in server.c unpacking code contains following lines:

Code:

176	/* decompress buffer (excluding first 32 bits) */
177	comp_p = buf + 4;
178	if (uncompress(obj_unc, &dest_len, comp_p, buflen - 4) != Z_OK)
179		goto out;
180	if (dest_len != unc_len)
181		goto out;
182	memcpy(obj_unc + unc_len, &zero, 1);	/* null terminate */

This call of uncompress() isn't completely correct according to zlib docs. dest_len should contain actual size of buffer for uncompressed data before calling uncompress(), but it's never initialized.

This should be corrected (e.g. dest_len = unc_len + 1).

[ius]

You seem to be right. I have submitted a pull request for Jeff: https://github.com/jgarzik/pushpool/pull/39

(I passed unc_len instead of unc_len + 1, as the last byte is used for null termination and not part of the compressed data)

[elmigranto]

You seem to be right. I have submitted a pull request for Jeff

Could you submit another one (I'm pretty new to CVS, and have sadly too little free time to dig in)?

cjson_encode(), server.c:

Code:

233	/* fill in compressed length */
234	*obj_clen = htole32(comp_len);

It's little mismatched with whole idea. Instead of comp_len, unc_len should be there.

Must be too late in night, when this part was written.

[ius]

Really looks like you're the only one to have ever used the binary protocol - anyway, submitted as well. https://github.com/jgarzik/pushpool/pull/40

Thanks.