Hacking BFL Monarchs and servicing them while times are weird.

[lightfoot]

It is insanely priced.

Eh. Most miners are. By the time you get them you won't make much money anyway. So all miners are overpriced if you expect money to fall from the sky (I never got this).

ASICminer tube. You get to fiddle a bunch, and earn back the bitcoins!

Hm. URL? Are people blowing those up and such?

I never really got into hacking things like the blades or block erupters or those chinese ones. No real artistic interest. I did fix someone's gridseed, but that was kind of simple like an erupter.

I wonder if there is a market for Neptune repair. I thought about the X3's, but HashFast can't seem to figure out how to make them and they are catching fire, which is serious problems.

C

[lightfoot]

Regarding that spreadsheet: Thank you! That's interesting.

So what can we think here:

Most of these cortex things are hashing at 6400. Fastest reasonable speed is 6700. So if we assume 128 of them, and 6.7gh each we get 857gh as the max hashing speed at this voltage/clock combo. This is where I would start screwing with the clock speed; the trick was if you clocked faster you got more hashes but sometimes fewer engines as they errored out. Sometimes it was worth it, sometimes not.

Hm. Anyone else able to post their chiliflash values? More would be helpful; I have a sample size of el-one.

[Pt0x]

Where's the picture of the shed with the new addition???

[SLok]

Do you also see messages like "sanity check, device is processing unknown work, device is missing queued job. device flushed (goes by too fast), failed to find work for queue results" in the bfgminer cmd window?

Interesting. When I first powered up I got a nice 700gh. After an hour I shut it down to switch power supplies, fired up, and saw some error messages and about 600gh speeds.

Shut it down, went back to previous supply, started up. Same.

This afternoon I shut it down, let it sit for 20 minutes to cool off, powered it up. 700gh, no errors I can see (I don't stare at it, just check every once in awhile). Solid as a rock.

One odd thing: I did notice the power supply high pitch frequency from the chokes was different. Maybe when the board is cycled hot it does a similar thing to what the old Singles did when they would power up more slowly. But letting it sit for a few mins seems to have restored it. Note it's been running for 6 hours now without a problem.

What are you seeing. Can you post the messages? Do they all do that, can you try shutting off the one that does for 15-20 mins? How does it hash?
C

posted this on bfl forum too, edited to today's findings;
"I'm getting errors on 1 of 2 monarchs, switched usb cables, ports, tried powered usb hub, un-powered, direct into pc. I noticed on powering the monarchs, one has a 1 red led going on/off/on/off etc. near the back usb, the other one with the errors has 2, one with the same on/off sequence, and one that blinks fast about 6 times, then goes off for a sec., then blinks fast again, etc.
errors are:
"sanity check, device is processing unknown work,
device is missing queued job. device flushed (goes by too fast)
failed to find work for queue results", then it hashes a couple of shares, and the same errors pop up again, and this repeats itself over and over, on the same monarch."

Later noticed they occur on bfl0 and bfl1, probably normal, I don't know if those messages in the command window are new for bfgminer 4.7.x and higher?

later

"Yeah, switching psu/usb/cables with the other one brings no difference, what does the fast blinking led next to the once-per-second flashing one on the error one mean? Even when it is only powered up and not hashing, it shows the same fast blinking of a second led. Doesn't look like I'm missing out on much hashrate though, it's running as it is specced, over 790-840GH for 2x 400GH rated units."

Actually a 400GH and a 425GH unit, and labelled so on the small fan's side. BFGminer reports couple of degrees above room temp, more shed temp, running about 22C. Odd thing is I found bfgminer shut down 2 times now. I'm kind of busy and don't have time to sit down with the miners and host and the bfgminer manuals to see what's to adapt, but I guess overclocking goes by fw like the sc's did?
And 5% and 4% hw errors btw, 0% and 0% rejected.

[bcp19]

Do you also see messages like "sanity check, device is processing unknown work, device is missing queued job. device flushed (goes by too fast), failed to find work for queue results" in the bfgminer cmd window?

Interesting. When I first powered up I got a nice 700gh. After an hour I shut it down to switch power supplies, fired up, and saw some error messages and about 600gh speeds.

Shut it down, went back to previous supply, started up. Same.

This afternoon I shut it down, let it sit for 20 minutes to cool off, powered it up. 700gh, no errors I can see (I don't stare at it, just check every once in awhile). Solid as a rock.

One odd thing: I did notice the power supply high pitch frequency from the chokes was different. Maybe when the board is cycled hot it does a similar thing to what the old Singles did when they would power up more slowly. But letting it sit for a few mins seems to have restored it. Note it's been running for 6 hours now without a problem.

What are you seeing. Can you post the messages? Do they all do that, can you try shutting off the one that does for 15-20 mins? How does it hash?
C

posted this on bfl forum too, edited to today's findings;
"I'm getting errors on 1 of 2 monarchs, switched usb cables, ports, tried powered usb hub, un-powered, direct into pc. I noticed on powering the monarchs, one has a 1 red led going on/off/on/off etc. near the back usb, the other one with the errors has 2, one with the same on/off sequence, and one that blinks fast about 6 times, then goes off for a sec., then blinks fast again, etc.
errors are:
"sanity check, device is processing unknown work,
device is missing queued job. device flushed (goes by too fast)
failed to find work for queue results", then it hashes a couple of shares, and the same errors pop up again, and this repeats itself over and over, on the same monarch."

Later noticed they occur on bfl0 and bfl1, probably normal, I don't know if those messages in the command window are new for bfgminer 4.7.x and higher?

later

"Yeah, switching psu/usb/cables with the other one brings no difference, what does the fast blinking led next to the once-per-second flashing one on the error one mean? Even when it is only powered up and not hashing, it shows the same fast blinking of a second led. Doesn't look like I'm missing out on much hashrate though, it's running as it is specced, over 790-840GH for 2x 400GH rated units."

Actually a 400GH and a 425GH unit, and labelled so on the small fan's side. BFGminer reports couple of degrees above room temp, more shed temp, running about 22C. Odd thing is I found bfgminer shut down 2 times now. I'm kind of busy and don't have time to sit down with the miners and host and the bfgminer manuals to see what's to adapt, but I guess overclocking goes by fw like the sc's did?
And 5% and 4% hw errors btw, 0% and 0% rejected.

Just an FYI, the light below the blinking one (which indicates ASIC initialization when solid, hashing when blinking) is related to the FGPA.  In the older code it was off, in the newer code it has the fast blink then off on a ~2 sec cycle.  If the FPGA is erased, it lights dimly.

Have you seen the 4.2.0 custom build BFG on the BFL website?  I know it doesn't seem anywhere near as prone to the sanity checks (unless you stop/start BFG without cycling power on the card)  The FPGA buffers the work done and submits upon BFG restart and you get a mass of work not found.  This buffer send seems to cause BFG to hiccup as I see an occasion 'failed to send queue' which goes away if I then cycle power on the card and restart BFG.

[lightfoot]

Actually a 400GH and a 425GH unit, and labelled so on the small fan's side. BFGminer reports couple of degrees above room temp, more shed temp, running about 22C. Odd thing is I found bfgminer shut down 2 times now. I'm kind of busy and don't have time to sit down with the miners and host and the bfgminer manuals to see what's to adapt, but I guess overclocking goes by fw like the sc's did?
And 5% and 4% hw errors btw, 0% and 0% rejected.

Hm, ok, that's still quite good, so this is not affecting hash rate. I'll take a look at mine tomorrow, still solid as a rock at 702gh. One set of FETs is running at 100C (quite warm, not super hot yet), the other at 80c (eh). I might put a fan on the front, but no errors, error rate 1.9%.

Hm. I just put a little BFL jally fan pointing at the top side FET heat sinks head on, temps dropped from the 90's to the 60's. Note to self: Dropping FET heat can be done with a simple little fan pointing at them head on. Very interesting, I may have some room to play with like this. Drill 2 holes in the mounting bracket for the old jally standoffs and it will be permanent.

I'm going to let it run for 24 hours like this. Then go from there.

C

[lightfoot]

I really should go to bed but I thought I'd share this with you Slok:

I found this super cool little windows app called "Restart on Crash" version 1.3.3.174. It will restart an app if it crashes, if it crashes again it will back off for a few seconds, etc. Keeps my miners running no matter what. No evidence of spyware, I trust it on a mining rig.

Mine like hell!

C

[SLok]

Quick update, found bfgminer off again, odd since it shut down at 6.15AM, same time as yesterday. Internet by home plugs in the shed (over power line) may be the culprit, making a 40 meter utp cable tonight to try tomorrow. The custom bfgminer 4.2 shuts down right away, I restarted everything before trying it.
BCP about the leds, nearest one to the back usb port act similar on both units, one unit has an extra led, it blinks about 6 times very fast, then off for a sec, then blinks about 6 times very fast, then off for a sec., and so on.
I've set several back-up pools, no change, will look into that restart app. too, thanks for the tips lightfoot and bcp.

[Delarock]

Your threads are still my favorite. I've been so far removed from the mining world since I shut my miners off a few months ago, now I have a reason to pay attention again. 

[Biffa]

Quick update, found bfgminer off again, odd since it shut down at 6.15AM, same time as yesterday. Internet by home plugs in the shed (over power line) may be the culprit, making a 40 meter utp cable tonight to try tomorrow. The custom bfgminer 4.2 shuts down right away, I restarted everything before trying it.
BCP about the leds, nearest one to the back usb port act similar on both units, one unit has an extra led, it blinks about 6 times very fast, then off for a sec, then blinks about 6 times very fast, then off for a sec., and so on.
I've set several back-up pools, no change, will look into that restart app. too, thanks for the tips lightfoot and bcp.

Had problems with powerline and miners in my garage. Just ran a cable instead and never had a problem again.

Still use it for other equipment though without issue.

[SLok]

Just tried bfgminer 4.2 custom build from the .exe instead of a batch file, pop-up showed it wouldn't run because it's 64bit, need a 32bit version. It's a hot day, 23C and at least 26 in the shed, temps still not out of line I think?
Just placed a huge fan covering both monarch's water block sides, the messages as seen in the pics stay the same though. BCP are those common on more units, or should I look further? Will still try the utp later, so far I ran them with wifi (5 meters from AP) and by said home plugs.

[lightfoot]

Hm, that's interesting. I think I'll get the BFG code and see if I can find this error. Checked my unit, 698gh, 1% errors, no odd messages. Can you log the messages to a file so we can see them in situ?

In the meantime I can say as a flat fact: Putting a little fan in front of the FETs 4 inches away will reduce my operating temps from 90c to 60c, and will reduce the on-heatsink-FET temps from 100c/80c to 65c/50c.

Moving air across heat sinks makes the difference. Even the smallest amount of air movement drops the temps 30-40c.

C

[lightfoot]

Also what speed is being reported by Eligius? Say the 22 minute numbers, do they match the lower or higher of the 3 BFG numbers?

The pool is kind of the final solution on speed, you're not getting rejects so I'm curious.

[OgNasty]

Some advice...  When I found my bfgminer was shutting down, turning off all the power supplies and then turning them back on fixed any stability issues.  I also included a line in my .bat file so that if bfgminer does shut down, it automatically restarts.

[SLok]

Also what speed is being reported by Eligius? Say the 22 minute numbers, do they match the lower or higher of the 3 BFG numbers?

The pool is kind of the final solution on speed, you're not getting rejects so I'm curious.

22.5 minutes 809.57 Gh/s 254464sh, went for a drink, refreshed the page, and....22.5 minutes 765.59 Gh/s 240640sh, next refresh and the same 22.5 minutes   809.57Gh/s 254464sh again.

I never considered eligius' status page to be accurate, is the 22 min. number any good as a reference? BFGminer low shows 776, mid 807, high 854 and up till 940
BCP both units have the solid on led, that goes on/off when hashing. One has a second led just below the first one, it blinks fast about 6 times (so fast it's impossible to count the actual number), then it goes off for a sec., it does this from powering up, and keeps doing it. There are more leds, just noticed 1 or 2 between the pcb and the copper plate, solid on, I see the reflection on the sledge's base plate under the right chip.

Lightfoot have you tried getting anything by jtag, or is that what you got as said earlier, with chiliflash?

Ognasty, I restart the pc and miners every time I make a change or after a shut down bfgminer, shouldn't be a problem there. The fan on the fets made no change so far, after 6 hours hashing.

edit; no time for the utp now, but I doubt it will change anything.

[lightfoot]

Lightfoot have you tried getting anything by jtag, or is that what you got as said earlier, with chiliflash?

Chiliflash. Basically it's a serial app that sends a ZCX and reads back status. I've never been able to talk to these things with a serial program like PUTTY, never know the baud rate or stop bits. Oh well, someday.

I haven't tried to download the code yet, if the ROMs are protected that won't work. I don't think the Atmel will brick if you just try to read, I'll give it a shot tomorrow after running for a day with no problems.

Do they make a high pitch whine? I don't think that is a problem between BFG and queue. Still, you're getting more hashing than you bought, so be happy. Let's think this through for a bit, if you can get some sort of details on the message it might help. Maybe the two messages are related. Thinking.

Who else is getting those messages, and where are they?

C

[SLok]

No whining sound at all, I'm pretty happy with the units and their speed, just curious what the errors mean and a possible fix, if necessary. About that second led too, what does it indicate?
Running both off a single server psu, I was wondering, could a 60GH SC psu (the BFL ones) cope with the 210-250W draw of a 400GH unit?

[lightfoot]

No whining sound at all, I'm pretty happy with the units and their speed, just curious what the errors mean and a possible fix, if necessary. About that second led too, what does it indicate?
Running both off a single server psu, I was wondering, could a 60GH SC psu (the BFL ones) cope with the 210-250W draw of a 400GH unit?

Well, hm. If fans make no diff and your units run cool we can deduce that at 200gh per chip the FETs are not getting very hot. At 350gh per chip (me) they start to get hot and need a fan. They can run at 100c, but like on the singles, that's hot.

A limiter might be the FETs, if we put a water block on them and the FET drivers it might allow us to draw off a lot of heat as we go to 800 and beyond. So.... Who do we know that can find water blocks of a size that can cover all six power channels from FET to FET?

That's project 1. Get that in place, I'll order one, and we can start cranking.

Slok, your units are more interesting now. If you can run the ZCX command and tell us how many and what speed the cortexes are, we can compare them to mine. Maybe your units run all of them slow. Maybe you have B grade chips and whole bunches are not running. If they are all running slow, we might be able to boost.

C

[Taugeran]

If he throws numbers up I can throw together charts to start a comparison table and averages etc

[SLok]

Slok, your units are more interesting now. If you can run the ZCX command and tell us how many and what speed the cortexes are, we can compare them to mine. Maybe your units run all of them slow. Maybe you have B grade chips and whole bunches are not running. If they are all running slow, we might be able to boost.

C

ok, but how, can't find that chiliflash app.