(OLD) BFGMiner: modular FPGA/GPU, GBT, Stratum, RPC, Avalon/Lnx/OpnWrt/PPA/W64

[bombo999]

lol ... kano is such a cry baby

[1713940553]

1713940553

[1713940553]

1713940553

[1713940553]

1713940553

[BR0KK]

Unable to test it atm .... working on some of my Singles that have some weird problems (one thinks to be a quad 1.15y and the other one seems to be dead for no reason?) 

[Luke-Jr]

NEW VERSION 2.3.6, APRIL 29 2012


Note this is a "upstream changes from CGMiner only" release.

Human readable changelog

• Important: --submit-stale option no longer exists. Con has replaced it with --no-submit-stale and made it submit stale shares by default now. The reason for this change is that with the new longpoll code in cgminer, it can detect block changes if you have backup pools almost always faster than the primary pool can detect the block change (unless the primary pool found the block). This means the primary pool is still accepting shares for the old block, so it won't consider them stale yet.
• Con has further revised the networking code to be able to cope with multigigahash machines that can overwhelm the one upstream and one downstream connection that was introduced into 2.3.5. Now it tries to get the best of all worlds. It does this by keeping one connection open for up and one for down and reuses that at all times, but, when it detects a backlog of share submission or getworks will occur, it starts recruiting extra connections. Thus it will be as nice as it can be to networks, but get nasty if it needs to.
• Two crashes were fixed: One was that hot-adding of pools was broken in 2.3.5. The second crash would very rarely occur across a longpoll when many pools were set up.
• The double longpoll message from the same pool should be fixed.
• Slight changes to make the screen output even neater.

Full changelog:

• Shorten stale share messages slightly.
• Protect the freeing of current_hash under mutex_lock to prevent racing on it when set_curblock is hit concurrently.
• Change default behaviour to submitting stale, removing the --submit-stale option and adding a --no-submit-stale option.
• Make sure to start the getwork and submit threads when a pool is added on the fly. This fixes a crash when a pool is added to running cgminer and then switched to.
• Faster hardware can easily outstrip the speed we can get work and submit shares when using only one connection per pool.
• Test the queued list to see if any get/submits are already queued and if they are, start recruiting extra connections by generating new threads.
• This allows us to reuse network connections at low loads but recuit new open connections as they're needed, so that cgminer can scale to hardware of any size.

[preparation]

I'd like to request a protection of BFGMiner in case a BFL unit will be disconnected. If you unplug the usb cable the miner crashes as 'BFwrite failed'.

[Luke-Jr]

I'd like to request a protection of BFGMiner in case a BFL unit will be disconnected. If you unplug the usb cable the miner crashes as 'BFwrite failed'.

What would you expect it to do? BFGMiner isn't hotplug-capable (yet). Maybe it could disable the device...

[Inaba]

Yes, please... if one of my units fails or otherwise goes off line, it basically brings the entire cluster down... very annoying. 

Hotplug would be the best, but disabling the device is a second best, since the rest can keep mining at the point.

[preparation]

Yes, please... if one of my units fails or otherwise goes off line, it basically brings the entire cluster down... very annoying. 

Hotplug would be the best, but disabling the device is a second best, since the rest can keep mining at the point.

this is exactly what I wanted to say

[Luke-Jr]

OK, I wrote the code to do it and sent it to Con for CGMiner. Whether he merges it or not, it'll be in the next BFGMiner.

[Hpman]

Hi, I'm using ubuntu 12.04. For preparation i'm using ./configure --enable-ztex but it stops with

Code:

checking for libusb_init in -lusb-1.0... no
configure: error: Could not find usb library - please install libusb

I thought libusb is already installed per default on ubuntu ? Do i need to copy any header file in the bfgminer directory ?
I installed libsusb-dev too but it doesn't help.

Thanks Hpman
PS: BTCMiner works, i guess without libusb it would not.

[Epoch]

Yes, please... if one of my units fails or otherwise goes off line, it basically brings the entire cluster down... very annoying. 

this is exactly what I wanted to say

Guys, as a temporary workaround, you can wrap your cgminer call in a .bat file (Windows), or the Linux equivalent. Something like this (again, Windows code):

Code:

:loop
cgminer <your parameters>
waitfor /t 10 signame
goto loop

If cgminer quits for any reason (such as a Single going offline), the batch file will pause for 10 seconds (adjust this to whatever you want) and then runs cgminer again. If you ever want to stop the infinite loop and get out of the batch file, hit <ctrl-c> and answer 'y' to the prompt that appears (again, Windows).

[Epoch]

Yes, please... if one of my units fails or otherwise goes off line, it basically brings the entire cluster down... very annoying.  

this is exactly what I wanted to say

Guys, as a temporary workaround, you can wrap your cgminer call in a .bat file (Windows), or the Linux equivalent. Something like this (again, Windows code):

Code:

:loop
cgminer <your parameters>
waitfor /t 10 signame
goto loop

If cgminer quits for any reason (such as a Single going offline), the batch file will pause for 10 seconds (adjust this to whatever you want) and then runs cgminer again. If you ever want to stop the infinite loop and get out of the batch file, hit <ctrl-c> and answer 'y' to the prompt that appears (again, Windows).

Epoch, cgminer crashes.  If process is still running, your script will not work.
It will only work when cgminer quits gracefully or calls exit()/abort().

That is not what I am seeing. I have tried this exact scenario on 4 separate Windows 7/64 machines. When I unplug one of the Singles, cgminer simply quits. Perhaps not gracefully, but it does not hang or lock up. It merely quits ... essentially back to the command prompt. My script then gets control and loops back to re-run it; it has worked fine on 4 different machines.

Some of the recent earlier versions of cgminer used to crash/lock-up. But the recent releases (certainly 2.3.6) do not and merely quit to the command prompt.

I realize that if cgminer hangs, this workaround will not work. The real solution is a fix in cgminer. I offer this looping .bat file as something that may work for some of us (Windows 7 at least, perhaps not Linux ... that's something I cannot comment on).

edit: sorry, Luke, I don't mean to go off-topic here with cgminer comments ... I'll stay back on track with bfgminer!

I haven't 'unplugged a Single' running with bfgminer yet to see how it behaves ... I'll try that this evening.

[Inaba]

Same with my experience ... cgminer just stops, it doesn't hang... so the batch file idea would work until it's fixed.

How is Ufasoft handling the hot-plugging? I'd *really* like to see that functionality in cgminer/bfgminer.

[Luke-Jr]

How is Ufasoft handling the hot-plugging? I'd *really* like to see that functionality in cgminer/bfgminer.

It's polling every serial port every few seconds. Too ugly to put in any sane software, IMO. The real problem for CGMiner/BFGMiner is that the initialization has to happen in a certain order right now, so adding devices later isn't possible. Changing that will require some refactoring.

[rjk]

How is Ufasoft handling the hot-plugging? I'd *really* like to see that functionality in cgminer/bfgminer.

It's polling every serial port every few seconds. Too ugly to put in any sane software, IMO. The real problem for CGMiner/BFGMiner is that the initialization has to happen in a certain order right now, so adding devices later isn't possible. Changing that will require some refactoring.

I'd argue that this is very acceptable for a dedicated miner, even if it isn't for a non-dedicated miner, therefore I would like the same thing to be implemented in cgminer/bfgminer. It could be an option, such as -S aggressive or similar.

[Luke-Jr]

How is Ufasoft handling the hot-plugging? I'd *really* like to see that functionality in cgminer/bfgminer.

It's polling every serial port every few seconds. Too ugly to put in any sane software, IMO. The real problem for CGMiner/BFGMiner is that the initialization has to happen in a certain order right now, so adding devices later isn't possible. Changing that will require some refactoring.

I'd argue that this is very acceptable for a dedicated miner, even if it isn't for a non-dedicated miner, therefore I would like the same thing to be implemented in cgminer/bfgminer. It could be an option, such as -S aggressive or similar.

I was considering a "-S all"; but in any case, hotplug support still requires a refactor of CGMiner's initialization.

[Tinua]

Hi Luke-Jr

Like promise in the cgminer-tread, i make the same test with 15 BFL's.

Thats the result:

http://www.fileswap.com/dl/QELC51peid/bfgminer2.jpg.html

With CGminer runs everything now!

Regards
Tinua

[Luke-Jr]

NEW VERSION: 2.4.0 - May 3, 2012

Note this is an "upstream changes from CGMiner only" release.
However, since BFGMiner has had reasonably accurate Icarus hash measurement from the start, my fix in CGMiner/BFGMiner 2.4.0 makes BFGMiner as ideally accurate as reasonably possible.

This version has a fairly significant upgrade to the way networking is done, so there is a minor version number update instead of a micro version, but it has already been heavily tested.

Human readable changelog:

• A whole networking scheduler of sorts was written for this version, designed to scale to any sized workload with the fastest networking possible, while minimising the number of connections in use, reusing them as much as possible.
• The restart feature was added to the API to restart BFGMiner remotely.
• If you're connected to a pool that starts rejecting every single share, BFGMiner will now automatically disable it unless you add the --no-pool-disable option.
• Once a pool stops responding, BFGMiner won't keep trying to open a flood of extra connections.
• Failing BFL won't cause cgminer to stop; it'll just disable the device, which an attempt may be made to re-enable it.
• Hashrates on FPGAs are now reasonably accurate.
• Longpoll messages won't keep going indefinitely while a pool is down.

Full changelog:

• Only show longpoll warning once when it has failed.
• Convert hashes to an unsigned long long as well.
• Detect pools that have issues represented by endless rejected shares and disable them, with a parameter to optionally disable this feature.
• Bugfix: Use a 64-bit type for hashes_done (miner_thread) since it can overflow 32-bit on some FPGAs
• Implement an older header fix for a label existing before the pthread_cleanup macro.
• Limit the number of curls we recruit on communication failures and with delaynet enabled to 5 by maintaining a per-pool curl count, and using a pthread conditional that wakes up when one is returned to the ring buffer.
• Generalise add_pool() functions since they're repeated in add_pool_details.
• Bugfix: Return failure, rather than quit, if BFwrite fails
• Disable failing devices such that the user can attempt to re-enable them
• Bugfix: thread_shutdown shouldn't try to free the device, since it's needed afterward
• API bool's and 1TBS fixes
• Icarus - minimise code delays and name timer variables
• api.c V1.9 add 'restart' + redesign 'quit' so thread exits cleanly
• api.c bug - remove extra ']'s in notify command
• Increase pool watch interval to 30 seconds.
• Reap curls that are unused for over a minute. This allows connections to be closed, thereby allowing the number of curl handles to always be the minimum necessary to not delay networking.
• Use the ringbuffer of curls from the same pool for submit as well as getwork threads. Since the curl handles were already connected to the same pool and are immediately available, share submission will not be delayed by getworks.
• Implement a scaleable networking framework designed to cope with any sized network requirements, yet minimise the number of connections being reopened. Do this by create a ring buffer linked list of curl handles to be used by getwork, recruiting extra handles when none is immediately available.
• There is no need for the submit and getwork curls to be tied to the pool struct.
• Do not recruit extra connection threads if there have been connection errors to the pool in question.
• We should not retry submitting shares indefinitely or we may end up with a huge backlog during network outages, so discard stale shares if we failed to submit them and they've become stale in the interim.

[Hpman]

Hi, I'm using ubuntu 12.04. For preparation i'm using ./configure --enable-ztex but it stops with

Code:

checking for libusb_init in -lusb-1.0... no
configure: error: Could not find usb library - please install libusb

I thought libusb is already installed per default on ubuntu ? Do i need to copy any header file in the bfgminer directory ?
I installed libsusb-dev too but it doesn't help.

Thanks Hpman
PS: BTCMiner works, i guess without libusb it would not.

libusb-dev wasn't enough, I've added  "libusb-1.0-0-dev" too and it works. Now it stops at

Code:

checking for LIBCURL... no
checking for LIBCURL... no
configure: error: Missing required libcurl dev >= 7.10.1

It seems that there is no libcurl.pc existing (for pkg-config)
curl-config --version shows  libcurl 7.22.0

Hpman

[Luke-Jr]

Code:

[quote author=Hpman link=topic=78192.msg881895#msg881895 date=1336121371]checking for LIBCURL... no
checking for LIBCURL... no
configure: error: Missing required libcurl dev >= 7.10.1

It seems that there is no libcurl.pc existing (for pkg-config)
curl-config --version shows  libcurl 7.22.0[/quote]It's part of libcurl4-gnutls-dev for Debian/Ubuntu.

[Luke-Jr]

NEW VERSION - 2.4.1, MAY 6 2012

While Kano continues to try rewriting his own hashrate measurements for CGMiner's Icarus driver, BFGMiner's Icarus driver was already (and still is) more accurate. I did take the opportunity to give it even more specific calibration, though.

As BFL+Icarus owners might have known, BFGMiner has worked with both devices together from the start.

Finally, Kano recently injected RPC API code through the device driver API in CGMiner. While I made an attempt to sanitize it upstream, Con didn't merge it in time for 2.4.1; BFGMiner of course includes this cleanup, since it is important to a sane (and secure!) device driver API.

Human readable changelog:

• --benchmark won't crash
• A pool will only be disabled if it rejects shares for at least 3 minutes in a row. Then it will be checked every longpoll to see if it is accepting shares, and if so, will be re-enabled.
• More accurate hashrate on Icarus
• Ztex quad 1.15y support
• Extra device stats in RPC API

Full changelog

• Icarus: Calibrate hashrate yet even more accurately
• In the unlikely event of finding a block, display the block solved count with the pool it came from for auditing.
• Display the device summary on exit even if a device has been disabled.
• Use correct pool enabled enums in api.c.
• Import Debian packaging configs
• Ensure we test for a pool recovering from idle so long as it's not set to disabled.
• Fix pool number display.
• Give BFGMiner -T message only if curses is in use.
• Reinit_adl is no longer used.
• API 'stats' allow devices to add their own stats also for testing/debug
• API add getwork stats to BFGMiner - accesable from API 'stats'
• Don't initialise variables to zero when in global scope since they're already initialised.
• Get rid of unitialised variable warning when it's false.
• Move a pool to POOL_REJECTING to be disabled only after 3 minutes of continuous rejected shares.
• Some tweaks to reporting and logging.
• API support new pool status
• Add a temporarily disabled state for enabled pools called POOL_REJECTING and use the work from each longpoll to help determine when a rejecting pool has started working again. Switch pools based on the multipool strategy once a pool is re-enabled.
• Removing extra debug
• Fix the benchmark feature by bypassing the new networking code.
• Reset sequential reject counter after a pool is disabled for when it is re-enabled.
• ztex updateFreq was always reporting on fpga 0
• Trying harder to get 1.15y working
• Specifying threads on multi fpga boards extra cgpu
• Missing the add cgpu per extra fpga on 1.15y boards
• API add last share time to each pool
• Don't try to reap curls if benchmarking is enabled.