Bitcoin Forum
July 20, 2019, 09:13:01 PM *
News: Latest Bitcoin Core release: 0.18.0 [Torrent] (New!)
 
   Home   Help Search Login Register More  
Pages: [1] 2 »  All
  Print  
Author Topic: [BOUNTY 5BTC] cgminer: add getwork proxy functionality  (Read 8881 times)
zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
July 06, 2013, 03:59:46 PM
 #1

Developers, miners and geeks,

I need to have a getwork proxy functionality added to cgminer and am willing to pay 5BTC for its development.


Overview
Last year I had 5 computers distributed all over my flat running cgminer to run different types of mining rig. Since each instance operated its own set of pools and work items, I thought it would be a nice feature to have only one of them operating as proxy, i.e. it on one side connects to pools and handles work items and on the other side allow other cgminer instances to connect and get served.

At that time I asked Con how feasible such a feature would be. His response was that it is doable, but since I was the only one requesting it, maybe not worth the work. With the subsequent feature advances in cgminer, I was able to reduce the number of instances to two and gradually refrained from my idea.

Until recently, when it became hot again: I started operating Asicminer BE blades that are interfaced over Ethernet and use getwork as work API. The best way to operate them is to use a Stratum proxy. While generally doable, I miss lots of cgminer features (most relevant: pool backup / recovery, stability, scalability). With that, I realize that the proxy functionality I had in mind before to cascade multiple cgminer instances would also be useful to support BE blades.

I am familiar with cgminer myself and have already a concept of how it could be realized (similar to hotplug_thread()). If I had 5 days of free time in a row, I'd do it myself - but obviously in today's interesting Bitcoin times I don't have it. Therefore I am offering a 5 BTC bounty to the skilled and motivated developer capable to develop that proxy functionality.

Requirements
From a functional point, the requirements should be clear by the title. I need cgminer to be able to
  • act as a getwork proxy at a configurable port
  • allow getwork based miners to connect (example: other cgminer instances, BE blades)
  • keep track of the getwork clients (identified by username)

Initial testing can be accomplished by connecting multiple cgminer instances to the proxy. If required, I can provide wireshark traces of the communication between blades and Stratum proxy; also I will do qualification testing and performance evaluation with a larger scale BE blades setup myself.

Beside the functional requirements, I am asking for
  • a cgminer compliant open source license to be used
  • the coding style to be kept consistent (wish: Linux kernel coding style)
  • sufficient code quality to get integrated into cgminer upstream


Alternative: stand alone proxy

If it makes development and testing easier for you, you are free to derive a getwork proxy component from existing cgminer. That is, this component would only use the pool and work management from cgminer, but not support mining devices itself (which btw. equals a hotplug-enabled cgminer with no active devices). cgminer's main source file is not modular enough to really support such an approach, just noting it here for completeness.

Process
Before you start hacking into your keyboard, please contact me for further details via email (visible from my forum profile).


Thanks,
zefir

1563657181
Hero Member
*
Offline Offline

Posts: 1563657181

View Profile Personal Message (Offline)

Ignore
1563657181
Reply with quote  #2

1563657181
Report to moderator
1563657181
Hero Member
*
Offline Offline

Posts: 1563657181

View Profile Personal Message (Offline)

Ignore
1563657181
Reply with quote  #2

1563657181
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1563657181
Hero Member
*
Offline Offline

Posts: 1563657181

View Profile Personal Message (Offline)

Ignore
1563657181
Reply with quote  #2

1563657181
Report to moderator
1563657181
Hero Member
*
Offline Offline

Posts: 1563657181

View Profile Personal Message (Offline)

Ignore
1563657181
Reply with quote  #2

1563657181
Report to moderator
1563657181
Hero Member
*
Offline Offline

Posts: 1563657181

View Profile Personal Message (Offline)

Ignore
1563657181
Reply with quote  #2

1563657181
Report to moderator
zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
July 08, 2013, 09:39:35 PM
 #2

200+ views, zero feedback Huh

Is this too challenging, or am I missing something?

turtle83
Sr. Member
****
Offline Offline

Activity: 322
Merit: 250


Supersonic


View Profile WWW
July 09, 2013, 09:02:48 AM
 #3

200+ views, zero feedback Huh

Is this too challenging, or am I missing something?

You scoped it well. There is nothing to feedback.

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
July 09, 2013, 12:50:42 PM
 #4

200+ views, zero feedback Huh

Is this too challenging, or am I missing something?

You scoped it well. There is nothing to feedback.

I was hoping for feedback of type: "Yeah, that's something I always wanted to do - and now I can get even some coins for it!" Smiley

Maybe after the recent exchange rate drop, 5 coins for 5 days of work is not enough as compensation.

But anyhow, I have one develper already looking at it, hope it can be contributed soon to the community.

Luke-Jr
Legendary
*
Offline Offline

Activity: 2464
Merit: 1017



View Profile
August 18, 2013, 11:51:31 AM
 #5

I've got the basic concept working as a new driver for BFGMiner:
Code:
 CPU 0:       |  3.71/ 2.52/ 0.00Mh/s | A:   0 R:0+0(none) HW:0/none
 SGW 0:       | 82.29/89.74/100.6Gh/s | A:2035 R:0+0(none) HW:0/none
 SGW 1:       | 12.05/22.13/22.13Mh/s | A:   1 R:0+0(none) HW:0/none
Each username creates a new SGW virtual device, and hashrate is measured by shares.
In this case, I have my main dev miner running on SGW 0, and a lucky CPU miner on SGW 1.

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 18, 2013, 06:53:09 PM
 #6

I've got the basic concept working as a new driver for BFGMiner:
Code:
 CPU 0:       |  3.71/ 2.52/ 0.00Mh/s | A:   0 R:0+0(none) HW:0/none
 SGW 0:       | 82.29/89.74/100.6Gh/s | A:2035 R:0+0(none) HW:0/none
 SGW 1:       | 12.05/22.13/22.13Mh/s | A:   1 R:0+0(none) HW:0/none
Each username creates a new SGW virtual device, and hashrate is measured by shares.
In this case, I have my main dev miner running on SGW 0, and a lucky CPU miner on SGW 1.

Looks good!

So you have an Avalon unit pointing to BFGminer and serving as GW proxy? Then it is basically done Smiley

I'd need to check if the AM BE blades' GW implementation is working similarly well, so please let me know when the branch is ready for testing.

Thanks for your time and efforts.

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 19, 2013, 11:13:41 AM
 #7

Luke-Jr implemented the getwork server / proxy functionality, which is now upstream in BFGminer master branch. Bounty paid.

Proxy works as specified and was tested with multiple cascading instances of BFGminer. There are some specific issues with the BE blade open, which will be addressed separately (since they require access to a blade).


Thanks a lot!

hanti
Full Member
***
Offline Offline

Activity: 126
Merit: 100


View Profile
August 20, 2013, 10:49:18 PM
 #8

can i ask if one of those issues with blades are low speed ? im getting like 2.8gh/s from one blade. On stratum proxy at same PC it works ok (max speed)

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 21, 2013, 06:51:26 AM
 #9

can i ask if one of those issues with blades are low speed ? im getting like 2.8gh/s from one blade. On stratum proxy at same PC it works ok (max speed)

Yes, exactly.

I am currently looking at it, but it is a PITA to go through protocol traces of JSON-over-HTTP-over-TCP traffic Sad

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 22, 2013, 03:30:33 PM
 #10

Update: ASICMINER BE blades now fully supported by bfgminer

The problem with the blades observed earlier was caused by the blades requiring a minimum size of the response packets and was fixed with the latest commits to bfgminer. With that, you can operate your blade pointing to a bfgminer instance ran with
Code:
./bfgminer --http-port <port>
and enjoy all the unbeatable pool management and job scheduling features provided by bfg/cgminer.


The GW server feature is built on top of Libmicrohttpd. On Debian based Linux systems install it with
Code:
sudo apt-get install libmicrohttpd-dev
and rebuild bfgminer from GIT sources.

BE blades (or GW clients generally) are identified by their username. If you have more than one blade and configure individual usernames, each will have its own statistics. If otherwise all have the same name, they will show up as one SGW device with cumulative statistics.


Happy Mining!

ewibit
Legendary
*
Offline Offline

Activity: 2393
Merit: 1002


View Profile
August 23, 2013, 12:30:38 PM
 #11

and rebuild bfgminer from GIT sources.
Code:
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: Fehler: unbekannter Typname: »intptr_t«
/usr/include/microhttpd.h:830:5: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:868:46: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:893:55: Fehler: unbekannter Typname: »va_list«
/usr/include/microhttpd.h:1085:57: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:1087:57: Fehler: unbekannter Typname: »MHD_ContentReaderCallback«
/usr/include/microhttpd.h:1197:54: Fehler: unbekannter Typname: »MHD_PostDataIterator«
make[2]: *** [bfgminer-httpsrv.o] Fehler 1
zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 23, 2013, 04:52:15 PM
 #12

and rebuild bfgminer from GIT sources.
Code:
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: Fehler: unbekannter Typname: »intptr_t«
/usr/include/microhttpd.h:830:5: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:868:46: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:893:55: Fehler: unbekannter Typname: »va_list«
/usr/include/microhttpd.h:1085:57: Fehler: unbekannter Typname: »uint64_t«
/usr/include/microhttpd.h:1087:57: Fehler: unbekannter Typname: »MHD_ContentReaderCallback«
/usr/include/microhttpd.h:1197:54: Fehler: unbekannter Typname: »MHD_PostDataIterator«
make[2]: *** [bfgminer-httpsrv.o] Fehler 1

Hard to say what's the issue without knowing your system, but the compiler not knowing standard integer types looks more like a build environment issue than a bfgminer related problem.

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.


On a side note: there is lots of activity with bfgminer currently as preparation for the upcoming release. As a result, the HEAD in fact does not compile right now, so better wait until the release.

ewibit
Legendary
*
Offline Offline

Activity: 2393
Merit: 1002


View Profile
August 23, 2013, 04:55:46 PM
 #13

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.
thx
yes I have done all pre-reqs
liyingfei
Newbie
*
Offline Offline

Activity: 47
Merit: 0



View Profile WWW
August 29, 2013, 07:21:46 PM
 #14

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.
thx
yes I have done all pre-reqs

我也遇到了这个问题 ubuntu server ,恳请知道 如何解决?

 CC     bfgminer-httpsrv.o
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: error: unknown type name ‘intptr_t’
/usr/include/microhttpd.h:830:5: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:868:46: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:893:55: error: unknown type name ‘va_list’
/usr/include/microhttpd.h:1085:57: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:1087:57: error: unknown type name ‘MHD_ContentReaderCallback’
/usr/include/microhttpd.h:1197:54: error: unknown type name ‘MHD_PostDataIterator’
make[2]: *** [bfgminer-httpsrv.o] Error 1
make[2]: Leaving directory `/root/bfgminer'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/bfgminer'
make: *** [all] Error 2
zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
August 30, 2013, 10:10:39 PM
 #15

Are you aware of the build process required to follow (pre-reqs, autogen, configure, make)? If not, be sure to read the READMEs.
thx
yes I have done all pre-reqs

我也遇到了这个问题 ubuntu server ,恳请知道 如何解决?

 CC     bfgminer-httpsrv.o
In file included from httpsrv.c:15:0:
/usr/include/microhttpd.h:497:3: error: unknown type name ‘intptr_t’
/usr/include/microhttpd.h:830:5: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:868:46: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:893:55: error: unknown type name ‘va_list’
/usr/include/microhttpd.h:1085:57: error: unknown type name ‘uint64_t’
/usr/include/microhttpd.h:1087:57: error: unknown type name ‘MHD_ContentReaderCallback’
/usr/include/microhttpd.h:1197:54: error: unknown type name ‘MHD_PostDataIterator’
make[2]: *** [bfgminer-httpsrv.o] Error 1
make[2]: Leaving directory `/root/bfgminer'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/root/bfgminer'
make: *** [all] Error 2

I am working with Linux.
Code:
$ lsb_release -irc
Distributor ID: Ubuntu
Release: 13.04
Codename: raring

$ uname -a
Linux PC 3.8.0-27-generic #40-Ubuntu SMP Tue Jul 9 00:17:05 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$ gcc --version
gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Copyright (C) 2012 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Just tried with a fresh clone of the GIT repository and it worked.

Step-by-Step:
1. ensure you have the required libs installed
Code:
sudo apt-get install libjansson-dev libmicrohttpd-dev libncurses5-dev uthash-dev libcurl4-openssl-dev

2. clone the GIT repository
Code:
git clone https://github.com/luke-jr/bfgminer.git

3. prepare and build
Code:
cd bfgminer
./autogen.sh
./configure --disable-opencl --disable-bitforce --disable-adl --disable-modminer --disable-ztex --disable-avalon --disable-icarus --disable-modminer --disable-x6500
make

If this fails for you, and you have a different system, please post a bug report at https://github.com/luke-jr/bfgminer/issues

hanti
Full Member
***
Offline Offline

Activity: 126
Merit: 100


View Profile
September 03, 2013, 03:04:52 PM
Last edit: September 03, 2013, 03:23:10 PM by hanti
 #16

it works almost perfect Wink
but i have much more rejected shares than on stratum proxy and blades stats page shows speed like 12600-12700 when on stratum proxy it was 13xxx
and efficiency is 96.5 now, on stratum proxy it was like 99 Wink
i think maybe its bcouse 2 blades im getting much rejected shares (stale)
im trying same user on both blades

edit: yes with same user on both blades it works ok no more rejected shares

edit2: yes confirmed

again 2 diferent users = again some rejected (stale) shares

for now it works perfect only if u set same user/pw on all blades (no stale rejected shares)

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
September 03, 2013, 04:17:12 PM
 #17

it works almost perfect Wink
but i have much more rejected shares than on stratum proxy and blades stats page shows speed like 12600-12700 when on stratum proxy it was 13xxx
and efficiency is 96.5 now, on stratum proxy it was like 99 Wink
i think maybe its bcouse 2 blades im getting much rejected shares (stale)
im trying same user on both blades

edit: yes with same user on both blades it works ok no more rejected shares

edit2: yes confirmed

again 2 diferent users = again some rejected (stale) shares

for now it works perfect only if u set same user/pw on all blades (no stale rejected shares)

Hm, sounds like an issue I observed when debug log displays work-item queue underruns.

In my case it helped to increase the the work-item queue, you could try adding e.g.
Code:
--queue 10
to the parameter list or put it into the config file.

hanti
Full Member
***
Offline Offline

Activity: 126
Merit: 100


View Profile
September 03, 2013, 07:49:17 PM
Last edit: September 03, 2013, 08:12:29 PM by hanti
 #18

with --queue 10 had many rejected shares too (3%)
i think its ok now with --no-submit-stale option.
it still shows at bfgminer stats A:1733 R:0+87(2.2%) but at least im not sending that shares to the mine
whats strange with that option stats on blade config panel showing normal stats like total mhz 13k and efficency 99

edit: after like 2h its again same as before 12700-12800 ;p so no diference too

dont know why so many stales on bfgminer but not with stratum proxy

zefir
Donator
Hero Member
*
Offline Offline

Activity: 919
Merit: 1000



View Profile
September 03, 2013, 08:34:45 PM
 #19

with --queue 10 had many rejected shares too (3%)
i think its ok now with --no-submit-stale option.
it still shows at bfgminer stats A:1733 R:0+87(2.2%) but at least im not sending that shares to the mine
whats strange with that option stats on blade config panel showing normal stats like total mhz 13k and efficency 99

My blades settled at 2.3% after running for a week, and to me it seems perfectly accurate.

Since the blades do not support LP, each time a block is found you get up to 32 stales (since each chip processes its own full nonce range). At 336 MHz, each chip needs 12.8 seconds for a full nonce range, so on average wastes 6.4 seconds per block change. With currently one block found every 440s, the theoretical loss is somewhere at 1.5% already - not too far from the measured stats.

hanti
Full Member
***
Offline Offline

Activity: 126
Merit: 100


View Profile
September 03, 2013, 11:15:28 PM
 #20

ye from what i can see on bitminter site in "latest shifts" speed is the same like with stratum proxy
only one whats bothering me is blade stats page
with stratum total speed = 13xxx and 99 efficiency but with bfgminer its 12600-12800 with 96 efficiency ;p

Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!