[BOUNTY 5BTC] cgminer: add getwork proxy functionality

[zefir]

Developers, miners and geeks,

I need to have a getwork proxy functionality added to cgminer and am willing to pay 5BTC for its development.


Overview
Last year I had 5 computers distributed all over my flat running cgminer to run different types of mining rig. Since each instance operated its own set of pools and work items, I thought it would be a nice feature to have only one of them operating as proxy, i.e. it on one side connects to pools and handles work items and on the other side allow other cgminer instances to connect and get served.

At that time I asked Con how feasible such a feature would be. His response was that it is doable, but since I was the only one requesting it, maybe not worth the work. With the subsequent feature advances in cgminer, I was able to reduce the number of instances to two and gradually refrained from my idea.

Until recently, when it became hot again: I started operating Asicminer BE blades that are interfaced over Ethernet and use getwork as work API. The best way to operate them is to use a Stratum proxy. While generally doable, I miss lots of cgminer features (most relevant: pool backup / recovery, stability, scalability). With that, I realize that the proxy functionality I had in mind before to cascade multiple cgminer instances would also be useful to support BE blades.

I am familiar with cgminer myself and have already a concept of how it could be realized (similar to hotplug_thread()). If I had 5 days of free time in a row, I'd do it myself - but obviously in today's interesting Bitcoin times I don't have it. Therefore I am offering a 5 BTC bounty to the skilled and motivated developer capable to develop that proxy functionality.

Requirements
From a functional point, the requirements should be clear by the title. I need cgminer to be able to

• act as a getwork proxy at a configurable port
• allow getwork based miners to connect (example: other cgminer instances, BE blades)
• keep track of the getwork clients (identified by username)

Initial testing can be accomplished by connecting multiple cgminer instances to the proxy. If required, I can provide wireshark traces of the communication between blades and Stratum proxy; also I will do qualification testing and performance evaluation with a larger scale BE blades setup myself.

Beside the functional requirements, I am asking for

• a cgminer compliant open source license to be used
• the coding style to be kept consistent (wish: Linux kernel coding style)
• sufficient code quality to get integrated into cgminer upstream

Alternative: stand alone proxy
If it makes development and testing easier for you, you are free to derive a getwork proxy component from existing cgminer. That is, this component would only use the pool and work management from cgminer, but not support mining devices itself (which btw. equals a hotplug-enabled cgminer with no active devices). cgminer's main source file is not modular enough to really support such an approach, just noting it here for completeness.

Process
Before you start hacking into your keyboard, please contact me for further details via email (visible from my forum profile).


Thanks,
zefir

[1713854269]

1713854269

[1713854269]

1713854269

[1713854269]

1713854269

[zefir]

200+ views, zero feedback

Is this too challenging, or am I missing something?

[turtle83]

200+ views, zero feedback

Is this too challenging, or am I missing something?

You scoped it well. There is nothing to feedback.

[zefir]

200+ views, zero feedback

Is this too challenging, or am I missing something?

You scoped it well. There is nothing to feedback.

I was hoping for feedback of type: "Yeah, that's something I always wanted to do - and now I can get even some coins for it!"

Maybe after the recent exchange rate drop, 5 coins for 5 days of work is not enough as compensation.

But anyhow, I have one develper already looking at it, hope it can be contributed soon to the community.

[Luke-Jr]

I've got the basic concept working as a new driver for BFGMiner:

Code:

 CPU 0:       |  3.71/ 2.52/ 0.00Mh/s | A:   0 R:0+0(none) HW:0/none
 SGW 0:       | 82.29/89.74/100.6Gh/s | A:2035 R:0+0(none) HW:0/none
 SGW 1:       | 12.05/22.13/22.13Mh/s | A:   1 R:0+0(none) HW:0/none

Each username creates a new SGW virtual device, and hashrate is measured by shares.
In this case, I have my main dev miner running on SGW 0, and a lucky CPU miner on SGW 1.

[zefir]

I've got the basic concept working as a new driver for BFGMiner:

Code:

 CPU 0:       |  3.71/ 2.52/ 0.00Mh/s | A:   0 R:0+0(none) HW:0/none
 SGW 0:       | 82.29/89.74/100.6Gh/s | A:2035 R:0+0(none) HW:0/none
 SGW 1:       | 12.05/22.13/22.13Mh/s | A:   1 R:0+0(none) HW:0/none

Each username creates a new SGW virtual device, and hashrate is measured by shares.
In this case, I have my main dev miner running on SGW 0, and a lucky CPU miner on SGW 1.

Looks good!

So you have an Avalon unit pointing to BFGminer and serving as GW proxy? Then it is basically done

I'd need to check if the AM BE blades' GW implementation is working similarly well, so please let me know when the branch is ready for testing.

Thanks for your time and efforts.

[zefir]

Luke-Jr implemented the getwork server / proxy functionality, which is now upstream in BFGminer master branch. Bounty paid.

Proxy works as specified and was tested with multiple cascading instances of BFGminer. There are some specific issues with the BE blade open, which will be addressed separately (since they require access to a blade).


Thanks a lot!

[hanti]

can i ask if one of those issues with blades are low speed ? im getting like 2.8gh/s from one blade. On stratum proxy at same PC it works ok (max speed)

[zefir]

can i ask if one of those issues with blades are low speed ? im getting like 2.8gh/s from one blade. On stratum proxy at same PC it works ok (max speed)

Yes, exactly.

I am currently looking at it, but it is a PITA to go through protocol traces of JSON-over-HTTP-over-TCP traffic

[zefir]

Update: ASICMINER BE blades now fully supported by bfgminer

The problem with the blades observed earlier was caused by the blades requiring a minimum size of the response packets and was fixed with the latest commits to bfgminer. With that, you can operate your blade pointing to a bfgminer instance ran with

Code:

./bfgminer --http-port <port>

and enjoy all the unbeatable pool management and job scheduling features provided by bfg/cgminer.


The GW server feature is built on top of Libmicrohttpd. On Debian based Linux systems install it with

Code:

sudo apt-get install libmicrohttpd-dev

and rebuild bfgminer from GIT sources.

BE blades (or GW clients generally) are identified by their username. If you have more than one blade and configure individual usernames, each will have its own statistics. If otherwise all have the same name, they will show up as one SGW device with cumulative statistics.


Happy Mining!

[ewibit]

and rebuild bfgminer from GIT sources.

Code:

In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: Fehler: unbekannter Typname: »intptr_t«
/usr/include/microhttpd.h:830:5: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:868:46: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:893:55: Fehler: unbekannter Typname: »va_list«
/usr/include/microhttpd.h:1085:57: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:1087:57: Fehler: unbekannter Typname: »MHD_ContentReaderCallback«
/usr/include/microhttpd.h:1197:54: Fehler: unbekannter Typname: »MHD_PostDataIterator«
make[2]: *** [bfgminer-httpsrv.o] Fehler 1

[zefir]

and rebuild bfgminer from GIT sources.

Code:

In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: Fehler: unbekannter Typname: »intptr_t«
/usr/include/microhttpd.h:830:5: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:868:46: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:893:55: Fehler: unbekannter Typname: »va_list«
/usr/include/microhttpd.h:1085:57: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:1087:57: Fehler: unbekannter Typname: »MHD_ContentReaderCallback«
/usr/include/microhttpd.h:1197:54: Fehler: unbekannter Typname: »MHD_PostDataIterator«
make[2]: *** [bfgminer-httpsrv.o] Fehler 1

Hard to say what's the issue without knowing your system, but the compiler not knowing standard integer types looks more like a build environment issue than a bfgminer related problem.

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.


On a side note: there is lots of activity with bfgminer currently as preparation for the upcoming release. As a result, the HEAD in fact does not compile right now, so better wait until the release.

[ewibit]

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.

thx
yes I have done all pre-reqs

[liyingfei]

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.

thx
yes I have done all pre-reqs

我也遇到了这个问题 ubuntu server ，恳请知道 如何解决？

 CC     bfgminer-httpsrv.o
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: error: unknown type name ‘intptr_t’
/usr/include/microhttpd.h:830:5: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:868:46: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:893:55: error: unknown type name ‘va_list’
/usr/include/microhttpd.h:1085:57: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:1087:57: error: unknown type name ‘MHD_ContentReaderCallback’
/usr/include/microhttpd.h:1197:54: error: unknown type name ‘MHD_PostDataIterator’
make[2]: *** [bfgminer-httpsrv.o] Error 1
make[2]: Leaving directory `/root/bfgminer'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/bfgminer'
make: *** [all] Error 2

[zefir]

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.

thx
yes I have done all pre-reqs

我也遇到了这个问题 ubuntu server ，恳请知道 如何解决？

 CC     bfgminer-httpsrv.o
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: error: unknown type name ‘intptr_t’
/usr/include/microhttpd.h:830:5: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:868:46: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:893:55: error: unknown type name ‘va_list’
/usr/include/microhttpd.h:1085:57: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:1087:57: error: unknown type name ‘MHD_ContentReaderCallback’
/usr/include/microhttpd.h:1197:54: error: unknown type name ‘MHD_PostDataIterator’
make[2]: *** [bfgminer-httpsrv.o] Error 1
make[2]: Leaving directory `/root/bfgminer'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/bfgminer'
make: *** [all] Error 2

I am working with Linux.

Code:

$ lsb_release -irc
Distributor ID:	Ubuntu
Release:	13.04
Codename:	raring

$ uname -a
Linux PC 3.8.0-27-generic #40-Ubuntu SMP Tue Jul 9 00:17:05 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ gcc --version
gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Just tried with a fresh clone of the GIT repository and it worked.

Step-by-Step:
1. ensure you have the required libs installed

Code:

sudo apt-get install libjansson-dev libmicrohttpd-dev libncurses5-dev uthash-dev libcurl4-openssl-dev

2. clone the GIT repository

Code:

git clone https://github.com/luke-jr/bfgminer.git

3. prepare and build

Code:

cd bfgminer
./autogen.sh
./configure --disable-opencl --disable-bitforce --disable-adl --disable-modminer --disable-ztex --disable-avalon --disable-icarus --disable-modminer --disable-x6500
make

If this fails for you, and you have a different system, please post a bug report at https://github.com/luke-jr/bfgminer/issues

[hanti]

it works almost perfect
but i have much more rejected shares than on stratum proxy and blades stats page shows speed like 12600-12700 when on stratum proxy it was 13xxx
and efficiency is 96.5 now, on stratum proxy it was like 99
i think maybe its bcouse 2 blades im getting much rejected shares (stale)
im trying same user on both blades

edit: yes with same user on both blades it works ok no more rejected shares

edit2: yes confirmed

again 2 diferent users = again some rejected (stale) shares

for now it works perfect only if u set same user/pw on all blades (no stale rejected shares)

[zefir]

it works almost perfect
but i have much more rejected shares than on stratum proxy and blades stats page shows speed like 12600-12700 when on stratum proxy it was 13xxx
and efficiency is 96.5 now, on stratum proxy it was like 99
i think maybe its bcouse 2 blades im getting much rejected shares (stale)
im trying same user on both blades

edit: yes with same user on both blades it works ok no more rejected shares

edit2: yes confirmed

again 2 diferent users = again some rejected (stale) shares

for now it works perfect only if u set same user/pw on all blades (no stale rejected shares)

Hm, sounds like an issue I observed when debug log displays work-item queue underruns.

In my case it helped to increase the the work-item queue, you could try adding e.g.

Code:

--queue 10

to the parameter list or put it into the config file.

[hanti]

with --queue 10 had many rejected shares too (3%)
i think its ok now with --no-submit-stale option.
it still shows at bfgminer stats A:1733 R:0+87(2.2%) but at least im not sending that shares to the mine
whats strange with that option stats on blade config panel showing normal stats like total mhz 13k and efficency 99

edit: after like 2h its again same as before 12700-12800 ;p so no diference too

dont know why so many stales on bfgminer but not with stratum proxy

[zefir]

with --queue 10 had many rejected shares too (3%)
i think its ok now with --no-submit-stale option.
it still shows at bfgminer stats A:1733 R:0+87(2.2%) but at least im not sending that shares to the mine
whats strange with that option stats on blade config panel showing normal stats like total mhz 13k and efficency 99

My blades settled at 2.3% after running for a week, and to me it seems perfectly accurate.

Since the blades do not support LP, each time a block is found you get up to 32 stales (since each chip processes its own full nonce range). At 336 MHz, each chip needs 12.8 seconds for a full nonce range, so on average wastes 6.4 seconds per block change. With currently one block found every 440s, the theoretical loss is somewhere at 1.5% already - not too far from the measured stats.

[hanti]

ye from what i can see on bitminter site in "latest shifts" speed is the same like with stratum proxy
only one whats bothering me is blade stats page
with stratum total speed = 13xxx and 99 efficiency but with bfgminer its 12600-12800 with 96 efficiency ;p