[EXPERIMENTAL] Patch to move work generation to pushpoold, faster? + merged mine

[makomk]

IIRC, one of the factors limiting pools has been how slow bitcoin is to answer getwork requests. It turns out that it's reasonably easy - with a few tweaks to the Bitcoin client - to move most of the work-generation out of bitcoind and into the pool server, with the pool server using one modified work item from Bitcoin to answer essentially as many getwork requests as it likes. So that's what I did:

https://github.com/makomk/bitcoin/tree/poolserv-work-gen https://github.com/makomk/pushpool/tree/local-work-gen

The changes are relatively straightforward: there's a new getworkex RPC call that provides a copy of the coinbase transaction and the merkle branch it's on, and when submitting solutions the pool server can send in a modified coinbase tx that's used instead of the original. pushpoold in turn now knows how to use getworkex, insert its own additional nonce into the coinbase, increment that nonce to generate work items,  and convert the submitted shares back into a form that bitcoind will accept. A very quick-and-dirty benchmark suggests it should be able to reach in the ballpark of around 3000 getworks/sec, which is much better than I was seeing before. (I'm planning to implement a way for bitcoin to push new work to the pool server at some point too.)

These changes are also incredibly experimental; I've basically only tested them with poclbm and cgminer on a testnet-in-a-box install so far. I hear that luke-jr has been working on something vaguely similar too, and shadders is developing another approach to poolserver work generation in PoolServerJ as well.

Edit: Exercise for the curious: the Bitcoin patch should in theory be enough to implement the parent blockchain side of merged mining, maybe even more effectively than the existing approach. You can use getworkex to add whatever you like to the coinbase transaction, and it gives you the merkle branch needed to submit the work to the auxiliary blockchains.

[shads]

I hear that luke-jr has been working on something vaguely similar too, and shadders is developing another approach to poolserver work generation in PoolServerJ as well.

Actually I'm implementing a work generator module that can either run embedded inside poolserverj or as a standalone.  The reason for the latter is that psj can get pretty busy with other things as well so you may still want to spread that load across two servers... The standalone will not use rpc.  It will simply stream work to psj in a compact binary format and psj controls the flow by sending stop/start signals.

[makomk]

Actually I'm implementing a work generator module that can either run embedded inside poolserverj or as a standalone.  The reason for the latter is that psj can get pretty busy with other things as well so you may still want to spread that load across two servers... The standalone will not use rpc.  It will simply stream work to psj in a compact binary format and psj controls the flow by sending stop/start signals.

Neat! I couldn't figure out whether that made sense or not; work generation is embarassingly parallel and easy to make threaded, but some of the other bits of a pool server aren't. I wasn't sure how quickly the other bits would become a bottleneck.

Also: I appear to now have merged mining working without using merged-mining-proxy! Just for a single aux chain at the moment and it's rather untested, but still. Uses https://github.com/makomk/pushpool/tree/local-work-gen-mm and a very  slightly patched version of namecoin in
https://github.com/makomk/bitcoin/tree/namecoin-getauxblock-prevhash - each chain gets its own shares table, and in theory shares are recorded as stale for each chain independently of the others. (In practice this isn't quite working right; shares that are stale on the parent chain are treated as stale for them all, and the client is only told if they're stale on the parent.)

Code:

    "auxchains" : [
        {
            "rpc.url" : "http://127.0.0.1:9337/", "rpc.user" : "", "rpc.pass" : "foo",
            "stmt.sharelog":"INSERT INTO shares_nc (rem_host, username, our_result, upstream_result, reason, solution) VALUES (?, ?, ?, ?, ?, ?)"
        }
    ]

[Balthazar]

Looks very nice, and works. Good solution for systems with small amount of RAM. But I don't understand, why you aren't use getmemorypool RPC call instead of your custom RPC call.