[Troubleshoot] ASICMINER Cube : HIGH Clock - 1 Card - 1 Bank (8 chips) - x'd

[Bobs Yerunkle]

Howdy,
Has anyone run into the problem described below and found a solution? 

Apart from fixing the issue, it would be great to actually understand what is happening.
Any reference or troubleshooting type urls for this sort of situation would be super and fantastic.
------------------------------------
Item: ASICMINER Cube
Area of Impact: 1 card only

Problem:
Card functions fine at LOW clock however at HIGH clock the second bank of 8 chips are all x'd up (ooooooooxxxxxxxx).

The issue occurs no matter what slot on the backplane the board is installed.
The cube was run with just the 1 card during my testing.

All other boards run at HIGH clock just fine.
------------------------------------

I have seen a similar problem mentioned here: https://bitcointalk.org/index.php?topic=307095.msg3871818#msg3871818

One actually hashed slower in High mode: Low: ~32Gh/s  High: 29Gh/s.  Now it's running at 33.5Gh/s  (now most likely limited by one card that has 8 X's on High)

Is it 8 scattered X, or 8 together? Probably an undervolt condition on that bank's VRM; I can think of a few possible causes most of which I could probably fix for you.

It's 8 all together on the right side.  Anything I can do?  It's hashing at ~34.1Gh/s now btw.

The possible issue mentioned by sidehack was "voltage underrun in the bank's VRM". These words make sense to me but I do not know what steps to take to remedy it. I have searched for forum and interwebs and have not succeeded in finding a solution.

All my attempts to solve the issue have focused on heat dissapation and PSU power

What I have done so far:

• Run cube with only the 1 problem blade with an older 700w PSU
• Run cube with only the 1 problem blade with a newer 860w PSU
• Removing and replacing the heat sink
• Inspecting board for visibly bad compenents or solder bridges
  
• Making a thermal paste sandwich. (Heat sink | paste where chip is | thermal pad |paste where chip is | chip backing.)
  src: https://bitcointalk.org/index.php?topic=310752.msg3732017#msg3732017[\li]
  

Much obliged. 

(Why do I want to understand this issue?....
"Curiosity, I guess. Heck, I'm curious like a cat. I have a couple of friends that call me whiskers." - Harry Caray)
((as portrayed by Will Ferrell [https://www.youtube.com/watch?v=7_0ava_MSSw]))

[1715130457]

1715130457

[1715130457]

1715130457

[1715130457]

1715130457

[1715130457]

1715130457

[1715130457]

1715130457

[sidehack]

Regarding a voltage underrun, the VRMs on Cube Cards are actually fairly clever. There are two VRMs per card, each sourcing voltage to 8 chips. The regulator uses three resistors and a FET to select the high and low voltages. In one configuration, all three resistors are in the series divider for feedback sampling; in high-speed configuration, the FET shorts around one resistor which changes the division ratio for the sampled output voltage and shifts it up from 1.05V to 1.15V
(More information on Cube VRMs here http://www.gekkoscience.com/webuilds/cube_oc/cube_oc.html

What I would recommend is, with the card running, measure the outputs from the VRMs (across C99-103 and across C83-87) and make sure you have about 1.05V; then switch it to high clock and see if both VRMs changed to 1.15V

If one is still reading 1.05V, measure the voltage present on R25 and R26 and see what they read. If they're both reading about the same (several volts), I'd say the FET itself is bad or not properly connected. First I'd resolder the FET (Q3/Q4) and see if that fixed it.
If one is several volts and one is low, you're not getting the signal to turn on the FET and switch the VRM to high-voltage mode. I'm not sure offhand which logic driver controls this signal but it shouldn't be too hard to trace. In the case of no signal or a bad FET, a workaround should be to remove the R9 or R18 resistor (on mine, it was 1800ohm but I've seen some that had 2100 there) and short across its pads. This would force the VRM to run in high-power mode all the time, so it would be providing more power than is required for "low clock" but would be right for "high clock".

[k2_1971]

I have a cube where one of the mini-blades does this also (OOOOOOOOXXXXXXXX). By restarting the cube through the web interface, I can sometimes get that card to display all O's, however one of the chips will read '999' and stay stuck at that value, and the entire cube will only run at about 10-15% efficiency and the hash rate suffers considerably, regardless of clock setting.

Ever seen that behavior before?

[k2_1971]

(More information on Cube VRMs here http://www.gekkoscience.com/webuilds/cube_oc/cube_oc.html

Great write-up by the way. Curious, after doing this modification to OC these to 43 Gh/s, what kind of real-world power draw are you seeing coming from the wall?

[sidehack]

Well I didn't actually get them up to 43. I'd expect to see about 400W though, based on my Blade numbers.

[Bobs Yerunkle]

I have a cube where one of the mini-blades does this also (OOOOOOOOXXXXXXXX). By restarting the cube through the web interface, I can sometimes get that card to display all O's, however one of the chips will read '999' and stay stuck at that value, and the entire cube will only run at about 10-15% efficiency and the hash rate suffers considerably, regardless of clock setting.
Ever seen that behavior before?

Common issues that can cause erratic behavior are loose heat sinks, unseated cards, and insufficient power.

I have not personally seen the behavior you describe but I think I saw something similar in this thread:
[Guide] Dogie's Comprehensive ASICMiner Cube Setup thread (https://bitcointalk.org/index.php?topic=352658.0)
(Print view is easier to digest and search in some cases: https://bitcointalk.org/index.php?action=printpage;topic=352658.0)

[k2_1971]

Well I didn't actually get them up to 43. I'd expect to see about 400W though, based on my Blade numbers.

Sounds about right. I am seeing ~330W per cube while OC'ed to 38 Gh/s.

[Bobs Yerunkle]

Regarding a voltage underrun, the VRMs on Cube Cards are actually fairly clever. There are two VRMs per card, each sourcing voltage to 8 chips. The regulator uses three resistors and a FET to select the high and low voltages. In one configuration, all three resistors are in the series divider for feedback sampling; in high-speed configuration, the FET shorts around one resistor which changes the division ratio for the sampled output voltage and shifts it up from 1.05V to 1.15V
(More information on Cube VRMs here http://www.gekkoscience.com/webuilds/cube_oc/cube_oc.html

What I would recommend is, with the card running, measure the outputs from the VRMs (across C99-103 and across C83-87) and make sure you have about 1.05V; then switch it to high clock and see if both VRMs changed to 1.15V

If one is still reading 1.05V, measure the voltage present on R25 and R26 and see what they read. If they're both reading about the same (several volts), I'd say the FET itself is bad or not properly connected. First I'd resolder the FET (Q3/Q4) and see if that fixed it.
If one is several volts and one is low, you're not getting the signal to turn on the FET and switch the VRM to high-voltage mode. I'm not sure offhand which logic driver controls this signal but it shouldn't be too hard to trace. In the case of no signal or a bad FET, a workaround should be to remove the R9 or R18 resistor (on mine, it was 1800ohm but I've seen some that had 2100 there) and short across its pads. This would force the VRM to run in high-power mode all the time, so it would be providing more power than is required for "low clock" but would be right for "high clock".

Thanks for the expounding on the voltage under run and the Cube OC write up. Great stuff. 

Testing was done with one card on a 700W PSU.

Both banks appear to be switching the voltage appropriately when the clock is changed.
For both banks:
Low clock ~1.058V
High clock ~1.180V

Is .008V(.78%) or .03V(~2.6%) variance a concern?
I didn't check the other blades for comparison (which was silly ).

I will have to take another look tomorrow.

For now I am low clockin' it.
Low clockin' is cooler any how.

Capacitors to check VRM voltage at:

[sidehack]

I would think those voltages would be good enough. The next consideration is, perhaps the high-clock oscillator output isn't getting to the chips. The problem with that though, is I believe the two oscillators are wired in parallel. This works because each has an enable line, which when disabled gives a high impedance on the output so it's more-or-less isolated from the signals generated by the other when enabled. They share the logic buffer and signal paths to the chips. If it were not a clock-dependent thing, I'd consider that either the address decoder was missing a bit, or the oscillator's logic buffer was missing an output or two.

As far as I know, the only things that change when switching to high clock are the VRM output and the select lines to the oscillators. Weird.


I do have on hand a few Cube cards in need of repair, the symptom of which is half the chips are X'd on high clock. I'll raise the priority for diagnosing and repairing them, then report back my results.

[Bobs Yerunkle]

Measured voltages on 2 cards that function fine on High.

Card 1
C099-103 | Low 1.063V | High 1.186V
C083-087 | Low 1.062V | High 1.186V

Card 1
C099-103 | Low 1.057V | C099-103 High 1.178V
C083-087 | Low 1.062V | C083-087 High 1.185V

The 'problem' card's voltages are comparable to these so at least that part of the voltage delivery is fine.

Removed and re-soldered the 14.318 MHz oscillator just for kicks.
In retrospect that doesn't make any sense since one oscillator is for both banks. 
Good practice though. The smell of flux. The shine of solder. Good times.

Concerned it really was just a power issue I put the Cube back on the Seasonic 860w PSU and High clock behaves as expected for the chips that are not all x'd up, but the issue with the dead bank is still present.
The 700w OCZ700GXSSLI, which technically should have enough oomph, is old and apparently does not quite chop the ketchup... no matter how hard I wish it.

The chips on the mystery bank have come back alive a couple times but it does not seem to be consistently reproducible.
Sometimes a reboot will do it. Other times powering off and powering on the PSU. Most of the time none of that works, times such as right now.

I am tempted to Fonzi the cube but if that worked I'd have to get back into voodoo.

[TheWoodser]

I have the same problem as described by the OP on one of the blades in my Cube.

Thanks for all the help so far sidehack!

Woodser

[Bobs Yerunkle]

I nudged a loose plug last night and killed the power to a couple machines.
It wasn't a clean power off but an on/off/on/off sort of event.
One of those machines was the Cube.

Now the all chips are o-faced at HIGH clock.  
Cube has been chugging along for 11 hours.

Not the solution I was hoping for but for now it is better than x'd chips...at least until the next shutdown/restart.

[Bobs Yerunkle]

After almost two days the Cube self reset itself and was displaying the 8 x'd chips.
A couple hours later it reset itself again and the x'd chips where o'd.

[TheWoodser]

Wow...Not sure I understand the fix here.....Sounds like you don't either.

Thanks for posting your experience.

Woodser

[Bobs Yerunkle]

No solution so far.
I am only documenting any changes.
 
I am back to being 8 x'd in the chips.