The Habanero - 650GH/s - OOS

[gateway]

Yeah, this sucks, I guess they are damaged now somehow, no idea, Ive upped the voltage plenty on the 875mhz one and, temps are till under 100C but hashrate are still stuck at 645-650GH
Even tried next mv setting up for the one at 850mhz and it has no effect, just runs hotter ... ugh, its hashrate still at 630-635GH
So, somewhere Im losing 3% on each one =*(

What do the 3 buttons do? more specifically the reconfig one, does that restore firmware defaults or allow firmware installation?

Also, like in my original post, any way to see the status of each 96 cores?
Thanks

Sorry to hear about your issues, as far as a status of each core cgminer does not provide this information, BFGMiner does but I haven't had the time or energy to work on porting the Pepper App to use BFG, and while BFG is a great miner its missing some of the functionality currently that you can do with cgminer for HF/HAB products.

One thing is to check your inflight value per board, now this isnt displayed in the pepper app, I could prob make a build with a debug option at the bottom of each page, this should be around 760ish.. if you have damaged a die this value gets lower.  Mr Teal could maybe elaborate a bit on this if he has time.

Also while these things can be fun to play with and tweak to maximum performance their are times where damage *could* happen.  I have busted a few boards just doing this and killed off a die so just be careful with them.

[GenTarkin]

Yeah, this sucks, I guess they are damaged now somehow, no idea, Ive upped the voltage plenty on the 875mhz one and, temps are till under 100C but hashrate are still stuck at 645-650GH
Even tried next mv setting up for the one at 850mhz and it has no effect, just runs hotter ... ugh, its hashrate still at 630-635GH
So, somewhere Im losing 3% on each one =*(

What do the 3 buttons do? more specifically the reconfig one, does that restore firmware defaults or allow firmware installation?

Also, like in my original post, any way to see the status of each 96 cores?
Thanks

Sorry to hear about your issues, as far as a status of each core cgminer does not provide this information, BFGMiner does but I haven't had the time or energy to work on porting the Pepper App to use BFG, and while BFG is a great miner its missing some of the functionality currently that you can do with cgminer for HF/HAB products.

One thing is to check your inflight value per board, now this isnt displayed in the pepper app, I could prob make a build with a debug option at the bottom of each page, this should be around 760ish.. if you have damaged a die this value gets lower.  Mr Teal could maybe elaborate a bit on this if he has time.

Also while these things can be fun to play with and tweak to maximum performance their are times where damage *could* happen.  I have busted a few boards just doing this and killed off a die so just be careful with them.

I think I saw the inflight field in my miner.php, will look for that. I would like an explanation about this value and how to make heads or tails about the condition of the cores.

Also, I cant fathom how I would damage a board, Ive barely gone over stock (highest Ive put in hf-tool is 950mv and one one hab Ive not gone over stock period) in voltages and the dies have never seen over 104C - cgminer immediately drops them off. Most of the times my temps are under 100C

I will look into bfgminer ... see if it can give me more info =)

[GenTarkin]

Quick update, I looked at miner.php api output and it just shows inflight_target at 760
Also shows "sequence modules" at 2048, if that means anything to figuring this all out =/

[GenTarkin]

Think I just figured it out and how can this be fixed....?
I ran bfgminer and immediately on startup it shows a total of 8 lines showing "disabled" .. like cores or something...  it goes by to fast to see exactly what its saying.
Ugh... why would it be disabling shit? =/

[i3luefire]

Brand...?...Model...? ...Link...?

Thanks,

ZiG

They're from an HP BLC7000 blade server, model # HSTNS-PR16
http://www.ebay.ca/itm/Hewlett-Packard-HP-HSTNS-PR16-488603-001-2450W-Power-Supply-for-HP-C7000-/321369086929?pt=PCA_UPS&hash=item4ad3171bd1

Thanks, buddy...

What is the pinout on that connector? is it easy to mate to?

[GenTarkin]

Welp, after doing some even further digging, Ive confirmed there are 8 cores being disabled upon startup by bfgminer. The resultant hashrate loss is nearly identical to what Im seeing in cgminer.
Some further things noticed & tested:

1. Ive confirmed that at least 1 of those disabled cores is able to do good work, by enabling it in bfgminer & watching it have a share accepted.
2. I couldnt believe it at first, thought I was going crazy, but the 8 cores that are being identical are 4 per hab, the id of the cores being EXACT between the 2 habs... WTF?!
    0dr,0hj,0lb,0ot & 1dr,1hj,1lb,1ot   - as seen in bfgminer

3. I went and hit the reset button on the board and noticed bfgminer re-detect the boards, each time a different core count was detected, ranging from min of 760 to a max of 766!(only 2 disabled)
4. I manually turned on those disabled cores in bfgminer, seeing my hashrate climb to what it should be, at least measured locally.
5. The state in which those 8 cores are upon bfgminer start is "RST"

Any ideas as to what could be going on?! =/
This is confusing the hell outta me....

[QuiveringGibbage]

EDIT: Issue resolved https://bitcointalk.org/index.php?topic=519943.msg7377697#msg7377697

Are you running both boards from that AX1200 PSU ?
If yes there might be your problem.
I'm using 2 PSUs 850W + 750W and power draw from the wall is around 1450W at 775MHZ.

no no, Antec1300w and AX1200i pulling a total of 1440w at the wall.

and this is the cgminer stats at the time of posting.

Code:

 0: HFB JOLOKIA : 825MHz  87C 100% 0.84V  | 534.3G / 621.0Gh/s WU:8676.0/m A:428159 R:  0 HW:119
 1: HFB FRUTESCE: 762MHz  77C 100% 0.84V  | 539.6G / 586.9Gh/s WU:8199.9/m A:418570 R:532 HW: 49

sadly if I push either board any more it's noticeably unstable. 620ghs and 580ghs is what i can get from my boards. without the ability to adjust the individual dies' clock rate, i can't squeeze the last bits of hash out of them. I don't know if anyone's going to put the effort in to read the code and suggest what needs to be fixed so we can fully utilize the HF-Tool.

oh and this is what happens when i say unstable:

Code:

[2014-06-17 21:25:45] Accepted 380808fb Diff 1.17K/532 HFB 0 pool 0
 [2014-06-17 21:25:51] Accepted 65e2e50e Diff 643/532 HFB 1 pool 0
 [2014-06-17 21:25:52] HFB JOLOKIA: No valid hashes for over 5 seconds, shutting down thread
 [2014-06-17 21:25:59] HFB 0 failure, disabling!
 [2014-06-17 21:26:05] HFA 0 HFName usb write err:(-7) LIBUSB_ERROR_TIMEOUT
 [2014-06-17 21:26:05] HFA 0 attempted reset got err:(0) LIBUSB_SUCCESS
 [2014-06-17 21:26:05] HFA : hfa_send_frame: USB Send error, ret 0 amount 0 vs. tx_length 8, retrying
 [2014-06-17 21:26:05] HFA : hfa_send_frame: recovered OK
 [2014-06-17 21:26:05] HFA 0 HFGetHeader usb read err:(-9) LIBUSB_ERROR_PIPE
 [2014-06-17 21:26:05] HFA 0 attempted reset got err:(-5) LIBUSB_ERROR_NOT_FOUND
 [2014-06-17 21:26:05] FAIL: USB get_lock not found (2:11)
 [2014-06-17 21:26:10] Accepted 79a271c1 Diff 539/532 HFB 1 pool 0
 [2014-06-17 21:26:11] HFA: Found old instance by op name JOLOKIA at device 0
 [2014-06-17 21:26:11] Hotplug: Hashfast added HFA 2
 [2014-06-17 21:26:24] Accepted 23b05141 Diff 1.84K/532 HFB 1 pool 0
 [2014-06-17 21:26:25] Accepted 32558e25 Diff 1.3K/532 HFB 0 pool 0

I have no solution to the above other than to use a lower clock rate. I have tried to set the Voltage higher with the HF-Tool but even if i set 930, cgminer still only runs with under 0.90V.

if anyone has any hints, I'd much appreciate it.

Cheers,
QG

EDIT: Issue resolved https://bitcointalk.org/index.php?topic=519943.msg7377697#msg7377697

[MrTeal]

Are you running both boards from that AX1200 PSU ?
If yes there might be your problem.
I'm using 2 PSUs 850W + 750W and power draw from the wall is around 1450W at 775MHZ.

no no, Antec1300w and AX1200i pulling a total of 1440w at the wall.

and this is the cgminer stats at the time of posting.

Code:

 0: HFB JOLOKIA : 825MHz  87C 100% 0.84V  | 534.3G / 621.0Gh/s WU:8676.0/m A:428159 R:  0 HW:119
 1: HFB FRUTESCE: 762MHz  77C 100% 0.84V  | 539.6G / 586.9Gh/s WU:8199.9/m A:418570 R:532 HW: 49

sadly if I push either board any more it's noticeably unstable. 620ghs and 580ghs is what i can get from my boards. without the ability to adjust the individual dies' clock rate, i can't squeeze the last bits of hash out of them. I don't know if anyone's going to put the effort in to read the code and suggest what needs to be fixed so we can fully utilize the HF-Tool.

oh and this is what happens when i say unstable:

Code:

[2014-06-17 21:25:45] Accepted 380808fb Diff 1.17K/532 HFB 0 pool 0
 [2014-06-17 21:25:51] Accepted 65e2e50e Diff 643/532 HFB 1 pool 0
 [2014-06-17 21:25:52] HFB JOLOKIA: No valid hashes for over 5 seconds, shutting down thread
 [2014-06-17 21:25:59] HFB 0 failure, disabling!
 [2014-06-17 21:26:05] HFA 0 HFName usb write err:(-7) LIBUSB_ERROR_TIMEOUT
 [2014-06-17 21:26:05] HFA 0 attempted reset got err:(0) LIBUSB_SUCCESS
 [2014-06-17 21:26:05] HFA : hfa_send_frame: USB Send error, ret 0 amount 0 vs. tx_length 8, retrying
 [2014-06-17 21:26:05] HFA : hfa_send_frame: recovered OK
 [2014-06-17 21:26:05] HFA 0 HFGetHeader usb read err:(-9) LIBUSB_ERROR_PIPE
 [2014-06-17 21:26:05] HFA 0 attempted reset got err:(-5) LIBUSB_ERROR_NOT_FOUND
 [2014-06-17 21:26:05] FAIL: USB get_lock not found (2:11)
 [2014-06-17 21:26:10] Accepted 79a271c1 Diff 539/532 HFB 1 pool 0
 [2014-06-17 21:26:11] HFA: Found old instance by op name JOLOKIA at device 0
 [2014-06-17 21:26:11] Hotplug: Hashfast added HFA 2
 [2014-06-17 21:26:24] Accepted 23b05141 Diff 1.84K/532 HFB 1 pool 0
 [2014-06-17 21:26:25] Accepted 32558e25 Diff 1.3K/532 HFB 0 pool 0

I have no solution to the above other than to use a lower clock rate. I have tried to set the Voltage higher with the HF-Tool but even if i set 930, cgminer still only runs with under 0.90V.

if anyone has any hints, I'd much appreciate it.

Cheers,
QG

The voltage reported by cgminer comes from some on-die sensors, so it will always be lower than the setpoint measured by the VRM, which happens off-chip. That's normal.

Do you still have that piece of MDF right behind the boards, and is there any air flowing over the boards themselves? You can see similar things if the VRMs shut down due to high temperatures.

[gateway]

Welp, after doing some even further digging, Ive confirmed there are 8 cores being disabled upon startup by bfgminer. The resultant hashrate loss is nearly identical to what Im seeing in cgminer.
Some further things noticed & tested:

1. Ive confirmed that at least 1 of those disabled cores is able to do good work, by enabling it in bfgminer & watching it have a share accepted.
2. I couldnt believe it at first, thought I was going crazy, but the 8 cores that are being identical are 4 per hab, the id of the cores being EXACT between the 2 habs... WTF?!
    0dr,0hj,0lb,0ot & 1dr,1hj,1lb,1ot   - as seen in bfgminer

3. I went and hit the reset button on the board and noticed bfgminer re-detect the boards, each time a different core count was detected, ranging from min of 760 to a max of 766!(only 2 disabled)
4. I manually turned on those disabled cores in bfgminer, seeing my hashrate climb to what it should be, at least measured locally.
5. The state in which those 8 cores are upon bfgminer start is "RST"

Any ideas as to what could be going on?! =/
This is confusing the hell outta me....

are you running cgminer with --hfa-noshed ? HF chips can report to have babbling cores that just seem to do random stuff at times, its not clear and we where told it was fixed in .5 of the firmware but well take it with a grain of salt with that statement.

[GenTarkin]

Welp, after doing some even further digging, Ive confirmed there are 8 cores being disabled upon startup by bfgminer. The resultant hashrate loss is nearly identical to what Im seeing in cgminer.
Some further things noticed & tested:

1. Ive confirmed that at least 1 of those disabled cores is able to do good work, by enabling it in bfgminer & watching it have a share accepted.
2. I couldnt believe it at first, thought I was going crazy, but the 8 cores that are being identical are 4 per hab, the id of the cores being EXACT between the 2 habs... WTF?!
    0dr,0hj,0lb,0ot & 1dr,1hj,1lb,1ot   - as seen in bfgminer

3. I went and hit the reset button on the board and noticed bfgminer re-detect the boards, each time a different core count was detected, ranging from min of 760 to a max of 766!(only 2 disabled)
4. I manually turned on those disabled cores in bfgminer, seeing my hashrate climb to what it should be, at least measured locally.
5. The state in which those 8 cores are upon bfgminer start is "RST"

Any ideas as to what could be going on?! =/
This is confusing the hell outta me....

are you running cgminer with --hfa-noshed ? HF chips can report to have babbling cores that just seem to do random stuff at times, its not clear and we where told it was fixed in .5 of the firmware but well take it with a grain of salt with that statement.

Yeah, seems to have no effect =/ ... Whats odd is w/ noshed, the inflight count was 760, w/o noshed, at least this time, the inflight shows 768 =P ROFL! ... complete opposite of what it should be like, but I may verify those results.
If only bfgminer supported setting fan speeds, Id be using it instead of cgminer for the habs, seems to have much more accurate hashrate readout & consistency ... not to mention poolside hashrate measurement is always nice to see, which cgminer doesnt show at all.
Also, I dont know which miner to believe, because bfgminer shows upwards of 4% HW error on one of the habs, cuz its clocked a bit too high, whereas cgminer shows only 1.2% ... so which to believe, not sure, but the loss in hashrate would be consistent w/ the 4% HW  ... but dropping clocks, even tho lowers HW%, doesnt affect the drop in hashrate. Its just allout confusing ... LOL

[Taugeran]

Welp, after doing some even further digging, Ive confirmed there are 8 cores being disabled upon startup by bfgminer. The resultant hashrate loss is nearly identical to what Im seeing in cgminer.
Some further things noticed & tested:

1. Ive confirmed that at least 1 of those disabled cores is able to do good work, by enabling it in bfgminer & watching it have a share accepted.
2. I couldnt believe it at first, thought I was going crazy, but the 8 cores that are being identical are 4 per hab, the id of the cores being EXACT between the 2 habs... WTF?!
    0dr,0hj,0lb,0ot & 1dr,1hj,1lb,1ot   - as seen in bfgminer

3. I went and hit the reset button on the board and noticed bfgminer re-detect the boards, each time a different core count was detected, ranging from min of 760 to a max of 766!(only 2 disabled)
4. I manually turned on those disabled cores in bfgminer, seeing my hashrate climb to what it should be, at least measured locally.
5. The state in which those 8 cores are upon bfgminer start is "RST"

Any ideas as to what could be going on?! =/
This is confusing the hell outta me....

are you running cgminer with --hfa-noshed ? HF chips can report to have babbling cores that just seem to do random stuff at times, its not clear and we where told it was fixed in .5 of the firmware but well take it with a grain of salt with that statement.

Yeah, seems to have no effect =/ ... Whats odd is w/ noshed, the inflight count was 760, w/o noshed, at least this time, the inflight shows 768 =P ROFL! ... complete opposite of what it should be like, but I may verify those results.
If only bfgminer supported setting fan speeds, Id be using it instead of cgminer for the habs, seems to have much more accurate hashrate readout & consistency ... not to mention poolside hashrate measurement is always nice to see, which cgminer doesnt show at all.
Also, I dont know which miner to believe, because bfgminer shows upwards of 4% HW error on one of the habs, cuz its clocked a bit too high, whereas cgminer shows only 1.2% ... so which to believe, not sure, but the loss in hashrate would be consistent w/ the 4% HW  ... but dropping clocks, even tho lowers HW%, doesnt affect the drop in hashrate. Its just allout confusing ... LOL

We're all your runs long enough to let things settle? ~30mins

I remember flipping out very time if have to restart bfgminer on my bitfury rigs it'd occasionally show stupid high error rates for about 4-5 minutes while things settled down to normal 0.5-1.5%

Sorry to bring different hardware into the tread. Just sharing experience

[MrTeal]

Welp, after doing some even further digging, Ive confirmed there are 8 cores being disabled upon startup by bfgminer. The resultant hashrate loss is nearly identical to what Im seeing in cgminer.
Some further things noticed & tested:

1. Ive confirmed that at least 1 of those disabled cores is able to do good work, by enabling it in bfgminer & watching it have a share accepted.
2. I couldnt believe it at first, thought I was going crazy, but the 8 cores that are being identical are 4 per hab, the id of the cores being EXACT between the 2 habs... WTF?!
    0dr,0hj,0lb,0ot & 1dr,1hj,1lb,1ot   - as seen in bfgminer

3. I went and hit the reset button on the board and noticed bfgminer re-detect the boards, each time a different core count was detected, ranging from min of 760 to a max of 766!(only 2 disabled)
4. I manually turned on those disabled cores in bfgminer, seeing my hashrate climb to what it should be, at least measured locally.
5. The state in which those 8 cores are upon bfgminer start is "RST"

Any ideas as to what could be going on?! =/
This is confusing the hell outta me....

are you running cgminer with --hfa-noshed ? HF chips can report to have babbling cores that just seem to do random stuff at times, its not clear and we where told it was fixed in .5 of the firmware but well take it with a grain of salt with that statement.

Yeah, seems to have no effect =/ ... Whats odd is w/ noshed, the inflight count was 760, w/o noshed, at least this time, the inflight shows 768 =P ROFL! ... complete opposite of what it should be like, but I may verify those results.
If only bfgminer supported setting fan speeds, Id be using it instead of cgminer for the habs, seems to have much more accurate hashrate readout & consistency ... not to mention poolside hashrate measurement is always nice to see, which cgminer doesnt show at all.
Also, I dont know which miner to believe, because bfgminer shows upwards of 4% HW error on one of the habs, cuz its clocked a bit too high, whereas cgminer shows only 1.2% ... so which to believe, not sure, but the loss in hashrate would be consistent w/ the 4% HW  ... but dropping clocks, even tho lowers HW%, doesnt affect the drop in hashrate. Its just allout confusing ... LOL

We're all your runs long enough to let things settle? ~30mins

I remember flipping out very time if have to restart bfgminer on my bitfury rigs it'd occasionally show stupid high error rates for about 4-5 minutes while things settled down to normal 0.5-1.5%

Sorry to bring different hardware into the tread. Just sharing experience

I've noticed that with the Habaneros and Chilis as well, and it seems to be pool dependent. Right off the start you might see a few hundred HW errors, and after that it settles down to a normal pace.

[gateway]

Btw JakeTri has his code he converted from the python hf-tool into cgminer pushed into the latest git of cgminer.. so you have commands like

Code:

--hfa-options <arg> Set hashfast options name:clock or clock@voltage (comma separated)

Code:

--hfa-options "rabbit:650,turtle:550@800"

+Would set a device named rabbit to clock speed 650 MHz using default voltage
 +and the one named turtle to 550 MHz using a voltage of 800 mv. Starting the
 +device at a speed where it is most stable will give more reliable hashrates
 +long term and prevent it interacting with other devices, rather than depending
 +on the clockdown feature in cgminer.

see https://github.com/ckolivas/cgminer/commit/ef97e80a14240c95a7d2966abe462b12f7d4bf03

you would need to build the latest git cgminer for this to work.

Also use with caution, we cannot be responsible if you happen to blow up your board

[QuiveringGibbage]

Are those push or pull fans? I would expect them to be the other way round pushing the hot air away from the boards?

Pull, sucking the hot air from radiator and pushing them away from the board.

The voltage reported by cgminer comes from some on-die sensors, so it will always be lower than the setpoint measured by the VRM, which happens off-chip. That's normal.
Do you still have that piece of MDF right behind the boards, and is there any air flowing over the boards themselves? You can see similar things if the VRMs shut down due to high temperatures.

Right, I'll try more air flow over and under the boards. I have ~50cm standoffs which the boards mount on so there's space below the boards for airflow, just need to attached fans to my frame.

Will report back once i've made completed the build.
 
Cheers,
QG

[GenTarkin]

We're all your runs long enough to let things settle? ~30mins

I remember flipping out very time if have to restart bfgminer on my bitfury rigs it'd occasionally show stupid high error rates for about 4-5 minutes while things settled down to normal 0.5-1.5%

Sorry to bring different hardware into the tread. Just sharing experience

Yep, Ive let it settled for sure.

[btchedge]

Important consideration regarding cooling heads!

Through the process of sourcing various cooling heads in our search for the best performance, we learned that many popular cooling heads are NOT flat. Certain heads are just slightly curved so as to better fit the intended CPU form factor they were originally designed for. Using such a head could lead to less than optimal results  with Habaneros since the contact surface is not perfectly flat. I will post some more details soon regarding the heads that test best. In the meantime, make sure you don't haul off and buy a high performance solution whose design will only guarantee lower performance due to it's design.

[GenTarkin]

btw ... for me --hfa-noshed actually disables those 8 cores to begin with in cgminer
Leaving out --hfa-noshed ... activates all 768 cores from start...
I wonder, will the inflight value in the api ... change to reflect if a core gets disabled via firmware ... as its mining?

[QuiveringGibbage]

The voltage reported by cgminer comes from some on-die sensors, so it will always be lower than the setpoint measured by the VRM, which happens off-chip. That's normal.

Do you still have that piece of MDF right behind the boards, and is there any air flowing over the boards themselves? You can see similar things if the VRMs shut down due to high temperatures.

ahh yes, you the man!! and Gateway, I remember him telling me to put more air flow over the boards. just got ahead of my self and didn't attached the fans. Here's the difference; Pic1 Temps. (90c) with fan OFF and Pic2 temps.(75c) with fan ON and I could see the errors appear and disappear as I turn the fan off an on and as the board heated up and cooled.

Pic1 and Pic2


Pic3


More pics here: http://imgur.com/a/SMBJP#21

Cheers,
QG

[GenTarkin]

So, quick update again in my journey of discoving how this core shedding works:
noshed enabled: 8 cores(per hab) get disabled on start of mining.
noshed disabled: ALL cores enabled on start of mining, over 24hr period, one of my habs incremented "shed count" to 2

Theoretically, if ur cores arent complete shit, you will benefit from running w/o noshed!

It seems to be very common(as in Ive had several people report this to me) that using noshed...causes 8 cores to get disabled right from the start, per GN!!! why?! no fucking clue, but they work fine!

[MrTeal]

So, quick update again in my journey of discoving how this core shedding works:
noshed enabled: 8 cores(per hab) get disabled on start of mining.
noshed disabled: ALL cores enabled on start of mining, over 24hr period, one of my habs incremented "shed count" to 2

Theoretically, if ur cores arent complete shit, you will benefit from running w/o noshed!

It seems to be very common(as in Ive had several people report this to me) that using noshed...causes 8 cores to get disabled right from the start, per GN!!! why?! no fucking clue, but they work fine!

That's interesting, and that it appears to be the same four cores on dies 0 and 1. I would say it's likely a FW issue that we'll have to track down.