Bitcoin Forum

Developers, miners and geeks,

I need to have a getwork proxy functionality added to cgminer and am willing to pay 5BTC for its development.


Overview
Last year I had 5 computers distributed all over my flat running cgminer to run different types of mining rig. Since each instance operated its own set of pools and work items, I thought it would be a nice feature to have only one of them operating as proxy, i.e. it on one side connects to pools and handles work items and on the other side allow other cgminer instances to connect and get served.

At that time I asked Con how feasible such a feature would be. His response was that it is doable, but since I was the only one requesting it, maybe not worth the work. With the subsequent feature advances in cgminer, I was able to reduce the number of instances to two and gradually refrained from my idea.

Until recently, when it became hot again: I started operating Asicminer BE blades that are interfaced over Ethernet and use getwork as work API. The best way to operate them is to use a Stratum proxy. While generally doable, I miss lots of cgminer features (most relevant: pool backup / recovery, stability, scalability). With that, I realize that the proxy functionality I had in mind before to cascade multiple cgminer instances would also be useful to support BE blades.

I am familiar with cgminer myself and have already a concept of how it could be realized (similar to hotplug_thread()). If I had 5 days of free time in a row, I'd do it myself - but obviously in today's interesting Bitcoin times I don't have it. Therefore I am offering a 5 BTC bounty to the skilled and motivated developer capable to develop that proxy functionality.

Requirements
From a functional point, the requirements should be clear by the title. I need cgminer to be able to

• act as a getwork proxy at a configurable port
• allow getwork based miners to connect (example: other cgminer instances, BE blades)
• keep track of the getwork clients (identified by username)

Initial testing can be accomplished by connecting multiple cgminer instances to the proxy. If required, I can provide wireshark traces of the communication between blades and Stratum proxy; also I will do qualification testing and performance evaluation with a larger scale BE blades setup myself.

Beside the functional requirements, I am asking for

• a cgminer compliant open source license to be used
• the coding style to be kept consistent (wish: Linux kernel coding style)
• sufficient code quality to get integrated into cgminer upstream

Alternative: stand alone proxy
If it makes development and testing easier for you, you are free to derive a getwork proxy component from existing cgminer. That is, this component would only use the pool and work management from cgminer, but not support mining devices itself (which btw. equals a hotplug-enabled cgminer with no active devices). cgminer's main source file is not modular enough to really support such an approach, just noting it here for completeness.

Process
Before you start hacking into your keyboard, please contact me for further details via email (visible from my forum profile).


Thanks,
zefir

200+ views, zero feedback ???

Is this too challenging, or am I missing something?

You scoped it well. There is nothing to feedback.

I was hoping for feedback of type: "Yeah, that's something I always wanted to do - and now I can get even some coins for it!" :)

Maybe after the recent exchange rate drop, 5 coins for 5 days of work is not enough as compensation.

But anyhow, I have one develper already looking at it, hope it can be contributed soon to the community.

I've got the basic concept working as a new driver for BFGMiner:
Each username creates a new SGW virtual device, and hashrate is measured by shares.
In this case, I have my main dev miner running on SGW 0, and a lucky CPU miner on SGW 1.

Looks good!

So you have an Avalon unit pointing to BFGminer and serving as GW proxy? Then it is basically done :)

I'd need to check if the AM BE blades' GW implementation is working similarly well, so please let me know when the branch is ready for testing.

Thanks for your time and efforts.

Luke-Jr implemented the getwork server / proxy functionality, which is now upstream in BFGminer master branch. Bounty paid (https://blockchain.info/tx/f6941e4efc315d380e6bb8777174afd412fc529749af039ecc99f9b8462b211e).

Proxy works as specified and was tested with multiple cascading instances of BFGminer. There are some specific issues with the BE blade open, which will be addressed separately (since they require access to a blade).


Thanks a lot!

can i ask if one of those issues with blades are low speed ? im getting like 2.8gh/s from one blade. On stratum proxy at same PC it works ok (max speed)

Yes, exactly.

I am currently looking at it, but it is a PITA to go through protocol traces of JSON-over-HTTP-over-TCP traffic :(

Update: ASICMINER BE blades now fully supported by bfgminer

The problem with the blades observed earlier was caused by the blades requiring a minimum size of the response packets and was fixed with the latest commits to bfgminer. With that, you can operate your blade pointing to a bfgminer instance ran with
and enjoy all the unbeatable pool management and job scheduling features provided by bfg/cgminer.


The GW server feature is built on top of Libmicrohttpd (http://www.gnu.org/software/libmicrohttpd/). On Debian based Linux systems install it with
and rebuild bfgminer from GIT sources.

BE blades (or GW clients generally) are identified by their username. If you have more than one blade and configure individual usernames, each will have its own statistics. If otherwise all have the same name, they will show up as one SGW device with cumulative statistics.


Happy Mining!

In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: Fehler: unbekannter Typname: »intptr_t«
/usr/include/microhttpd.h:830:5: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:868:46: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:893:55: Fehler: unbekannter Typname: »va_list«
/usr/include/microhttpd.h:1085:57: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:1087:57: Fehler: unbekannter Typname: »MHD_ContentReaderCallback«
/usr/include/microhttpd.h:1197:54: Fehler: unbekannter Typname: »MHD_PostDataIterator«
make[2]: *** [bfgminer-httpsrv.o] Fehler 1

Hard to say what's the issue without knowing your system, but the compiler not knowing standard integer types looks more like a build environment issue than a bfgminer related problem.

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.


On a side note: there is lots of activity with bfgminer currently as preparation for the upcoming release. As a result, the HEAD in fact does not compile right now, so better wait until the release.

thx
yes I have done all pre-reqs

我也遇到了这个问题 ubuntu server ，恳请知道 如何解决？

 CC     bfgminer-httpsrv.o
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: error: unknown type name ‘intptr_t’
/usr/include/microhttpd.h:830:5: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:868:46: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:893:55: error: unknown type name ‘va_list’
/usr/include/microhttpd.h:1085:57: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:1087:57: error: unknown type name ‘MHD_ContentReaderCallback’
/usr/include/microhttpd.h:1197:54: error: unknown type name ‘MHD_PostDataIterator’
make[2]: *** [bfgminer-httpsrv.o] Error 1
make[2]: Leaving directory `/root/bfgminer'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/bfgminer'
make: *** [all] Error 2

I am working with Linux.

Just tried with a fresh clone of the GIT repository and it worked.

Step-by-Step:
1. ensure you have the required libs installed

2. clone the GIT repository

3. prepare and build

If this fails for you, and you have a different system, please post a bug report at https://github.com/luke-jr/bfgminer/issues

it works almost perfect ;)
but i have much more rejected shares than on stratum proxy and blades stats page shows speed like 12600-12700 when on stratum proxy it was 13xxx
and efficiency is 96.5 now, on stratum proxy it was like 99 ;)
i think maybe its bcouse 2 blades im getting much rejected shares (stale)
im trying same user on both blades

edit: yes with same user on both blades it works ok no more rejected shares

edit2: yes confirmed

again 2 diferent users = again some rejected (stale) shares

for now it works perfect only if u set same user/pw on all blades (no stale rejected shares)

Hm, sounds like an issue I observed when debug log displays work-item queue underruns.

In my case it helped to increase the the work-item queue, you could try adding e.g. to the parameter list or put it into the config file.

with --queue 10 had many rejected shares too (3%)
i think its ok now with --no-submit-stale option.
it still shows at bfgminer stats A:1733 R:0+87(2.2%) but at least im not sending that shares to the mine
whats strange with that option stats on blade config panel showing normal stats like total mhz 13k and efficency 99

edit: after like 2h its again same as before 12700-12800 ;p so no diference too

dont know why so many stales on bfgminer but not with stratum proxy

My blades settled at 2.3% after running for a week, and to me it seems perfectly accurate.

Since the blades do not support LP, each time a block is found you get up to 32 stales (since each chip processes its own full nonce range). At 336 MHz, each chip needs 12.8 seconds for a full nonce range, so on average wastes 6.4 seconds per block change. With currently one block found every 440s, the theoretical loss is somewhere at 1.5% already - not too far from the measured stats.

ye from what i can see on bitminter site in "latest shifts" speed is the same like with stratum proxy
only one whats bothering me is blade stats page
with stratum total speed = 13xxx and 99 efficiency but with bfgminer its 12600-12800 with 96 efficiency ;p

Ah, that you mean.

Look into the source code: in driver-getwork.c:handle_getwork() you see how results are processed. The blades sometimes send submit request with empty work field (unclear why this happens), which are counted as HW errors and rejected as 'unknown-work'. Then there are those 'H-not-zero' rejects with a wrong nonce, plus the 'stale' ones.

I assume that the stratum-proxy does not reject some of them (e.g. the empty ones) and since the efficiency provided by the blades is the ratio of accepted to requested work items it differs between bfgminer and stratum-proxy (with bfgminer being more accurate).

My long term values for efficiency reported by blades is 96.4% and matches the expected numbers quite well.

ok thx for explanation all clear now ;)

thanks for your post   but 。。。。

root@ubuntu:/home/enki/bfgminer# ./bfgminer -V
bfgminer 3.2.0
root@ubuntu:/home/enki/bfgminer# ./bfgminer --http-port
 [2013-09-10 00:05:35] ./bfgminer: --http-port: unrecognized option

./configure --disable-opencl --disable-bitforce --disable-adl --disable-modminer --disable-ztex --disable-avalon --disable-icarus --disable-modminer --disable-x6500 --with-libmicrohttpd

Must have been added only lately, was enabled by default initially.

Anyway, from numerous users I know that the getwork server is working quite well on Linux systems. There are / were some issues with specific OS / libmicrohttpd version combinations, but once solved, the efforts pay off immediately. To serve ~150 clients over stratum-proxy, I had to use a dual-Xeon machine - now a tiny netbook does the job.

hi i have newest rasbian and pi512
i have configured with microhttpd
but same result
unrecognized option

bfgminer 3.2.90

to reproduce

git clone https://github.com/luke-jr/bfgminer.git
./autogen.sh
 ./configure --disable-opencl --disable-bitforce --disable-adl --disable-modminer --disable-ztex --disable-avalon --disable-icarus --disable-modminer --disable-x6500 --with-microhttpd
make
./bfgminer --http-port 8332
 [2013-09-27 10:13:16] ./bfgminer: --http-port: unrecognized option

thanks

i found an error

should be --with-microhttpd
instead of --with-libmicrohttpd

now it works

but workers are SGW0 1 etc
not like in blade user configuration

Workers are just counted up incrementally based on their username - which is very useful since it allows you to partition your farm arbitrary. In my setup, I have 8 racks of blades operating. I give all blades within a rack the same username, resulting in bfgminer treating them as unit, i.e. I see SGW0-7 and immediately know which rack has problems.

In effect, you have a fine-grained statistics chain: each blade reports individual stats; bfgminer reports stats per rack; pool reports global stats. You could even add more intermediate stats layers by connecting bfgminer instances hierarchically, e.g. one main bfgminer serving over getwork server other bfgminers for your sub-domains.

yeah but i have 5 blades and solomining shitcoins for fun.

blades goes sick often and then i have to restart them, problem is to keep them in proper numbers i have to turn them on in order it is annoying :)

if instead SGW-0 would be SGW-0-workername (from the auth lan blade) it would be much easier for everybody :)

I always met a problem of "Segmentation fault (core dumped)" after running for a while, and the terminal closed automaticly. (ubuntu)

Any ideas?

Crashes happen with any miner now and then. You need to run bfgminer from a looping script to prevent your boards from idling.

I am using this one:

Save this as $HOME/usr/bin/BEB-bfgminer.sh and adapt the related variables for your configuration. Run this script with screen and you have the perfect control over it for local and remote access. To have it started during boot-up, add this line to your /etc/rc.local:

To attach to the miner's screen session, run
to detach from the screen session press <Control-a><d>.

If you happened to forget detaching and want to attach from a different terminal / location, you can force detaching the previous session and take over with

Finally, you can follow the log from local or remote sessions with


Happy mining ;)

Crashes aren't a given, but I'd need at least a backtrace to try to debug it :/

Right, fighting the symptoms is always the second best choice.

Problem is, I never found a crash report or core dump while running bfgminer on my Ubuntu system, but from the startlog I know it restarts every now and then (in the order of once per week). Since this happens without interaction, I conclude those must be crashes - which might not be correct.

ulimit -c unlimited #to enable core dumps :)

Thank you zefir ,this is useful to me  ;D