Hacking KNC Titan / Jupiter / Neptune miners back to life. Why not?

[lightfoot]

Answer, mixed.

Working on the second board is kind of odd. On this one pins 4 and 6 are once again connected and read low resistance. However putting 2.5 volts from a regulated supply on the lines brings the current draw overall to 2.5 amps. Odd. That is enough to play "find the heat". And the heat is found, it's in the Titan die.

Crap.

Then again maybe not so much crap: Something is powering up. I think I'm going to sleep, and will fiddle with this more tomorrow. Maybe what I can do is bypass these two power supply lines by using my cut ribbon cable and explicitly supplying them with 2.5 volts from the big supply. Maybe something will come up.

[lightfoot]

Man this poor little guy *really* has a headache.

So I have it sort of "alive"

DC/DC   Voltage (V)   Current (A)   Power (W)   Temperature (°C)
0   0.8136   2.0703   1.684   35.200
1   0.8121   2.1641   1.757   34.600
2   0.8168   1.9062   1.557   32.400
3   0.8173   2.0547   1.679   34.000
4   0.8149   0.5908   0.481   35.400
5   0.8163   1.9297   1.575   36.100

But although it is registering temps, it is not powering up the full hashing. Also I noted that pin 8 while not shorted will shut down the connection to the controller if connected, so it's disconnected right now. So we have:

2 power supplies off.
Pins 4 and 6 hot wired to a 2.5 volt supply
Pin 8 disconnected

And it thinks it is at 127c every once in awhile, then down to 30c. Oh well, no one is perfect. Still, no hashing.

So what's up?

Not sure, but I think I have a thought popping up. What if they had two i2c connections on each unit. One would handle the house crud, the EEPROM, LM75, and the 8 power supplies. The other goes to the chip via a common set of lines or something.

Well, I found pin 8: It's a common to the four dies on one of those pair of lines that go to each corner which can be disconnected by the um... 0 ohm resistors. So in theory I could power up the heat tool, disconnect each line, and find the one that is high. With that I can get pin 8 to communicate again, leaving the mystery of where the other line to each chip goes and is. Is it a supply? Clock? or SCL?

Enquiring minds want to know....

[hawkfish007]

Man this poor little guy *really* has a headache.

So I have it sort of "alive"

DC/DC   Voltage (V)   Current (A)   Power (W)   Temperature (°C)
0   0.8136   2.0703   1.684   35.200
1   0.8121   2.1641   1.757   34.600

I have 3 cubes with similar situation, Die #1  low amp, low power and low temp but other three hashes at -.0586 V and 325 MHz, so I keep die 1 off in all three cubes.

DC/DC   Voltage (V)   Current (A)   Power (W)   Temperature (°C)
0           0.0143           0                   0.000   41.600
1           0.0060           0                   0.000   37.500
2           0.7566           41.2500           31.210   71.900
3           0.7625           40.3750           30.786   74.400
4           0.7729           41.6250           32.172   71.000
5           0.7694           42.3125           32.555   66.600
6           0.7583           40.5625           30.759   65.400
7           0.7567           40.9375           30.977      65.900

If you can figure out what causes this, I am sure you would have plenty of customers.

[lightfoot]

It may actually be damage to the chip die itself: One of the things I found in the hulk is that one corner has a short to ground when both of the signal pins are lifted, the other three don't. That's not something that can be fixed, but thank the universe that KNC built these things to run with a die or two down.

Letting it cool down from the latest rework, I disconnected all 8 lines and pin 8 is now proper (infinite) resistance to ground. Now to see which die was running with an 8k short-out.

Never dull.

[lightfoot]

Ok. Hooked up pin 8 though, that doesn't shut down the controller anymore. Good.

However unit still doesn't hash. Voltages are up, temps are stable, but nothing when I fire up for hashing. Either the short on pin 4,6 is still a problem, or something else is up. Will try hooking up the second die to see if I can get the one that was 800k resistance to ground working, maybe that one will have something.

[Searing]

Man this poor little guy *really* has a headache.

So I have it sort of "alive"

DC/DC   Voltage (V)   Current (A)   Power (W)   Temperature (°C)
0   0.8136   2.0703   1.684   35.200
1   0.8121   2.1641   1.757   34.600

I have 3 cubes with similar situation, Die #1  low amp, low power and low temp but other three hashes at -.0586 V and 325 MHz, so I keep die 1 off in all three cubes.

DC/DC   Voltage (V)   Current (A)   Power (W)   Temperature (°C)
0           0.0143           0                   0.000   41.600
1           0.0060           0                   0.000   37.500
2           0.7566           41.2500           31.210   71.900
3           0.7625           40.3750           30.786   74.400
4           0.7729           41.6250           32.172   71.000
5           0.7694           42.3125           32.555   66.600
6           0.7583           40.5625           30.759   65.400
7           0.7567           40.9375           30.977      65.900

If you can figure out what causes this, I am sure you would have plenty of customers.

ditto...I think my 2 dead dies show some of the above issues....looks familiar at least for one

I don't know electronics (I like reading your stuff...no idea what it means exactly it is like listening in on NASA for ASIC Nerds)

But anyway a LOT of 1st BATCH (first couple months) of 2014 Titans have DEAD DIES could the above be a symptom of such
ie....that is why there were so many that sucked when they went out the gate...bad quality control perhaps that you (or others)
can now MAYBE tweak?

just saying...finding a common error in all the above NOT a bad ASIC issue would be a lot better then the KNC argument of the chips were iffy

anyway have exhausted my attempt at contributing to this will go back and just lurk on the 'tech speak' etc

[lightfoot]

There is one odd thing: The heat sink compound on a 2 die down titan that came in was very odd. It looked almost as if it had been slathered on one side of the chip more than the other. This would cause pressure on the chip top and I have seen other types of miners where asymmetrical pressure cracks a chip and causes all sorts of weird stuff to happen.

I still need to check the tiny chips on the board, I think they interface between the SPI bus and the hashing chip but I will say that fixing a Titan that shorts out the power supply is do-able, fixing a dead die on a Titan is not very possible, and fixing a Titan board that blows out the SPI interface is a nightmare bitch on wheels.

Question: From the command line is it possible to screen over to see the cgminer process running? I'd like to find out the error rates.

[lightfoot]

Having Die 1 showing no voltage might actually be easier of a repair than the ones that show voltage but no hashing. Your issue is most likely shorted low-side FETs holding the power supply for that core to ground. Let me know if you want to send one in for a review.

C

[Tigggger]

Question: From the command line is it possible to screen over to see the cgminer process running? I'd like to find out the error rates.

Yes, you need to login with username 'pi' instead of 'admin', then screen -r will bring it up.

[lightfoot]

Question: From the command line is it possible to screen over to see the cgminer process running? I'd like to find out the error rates.

Yes, you need to login with username 'pi' instead of 'admin', then screen -r will bring it up.

Bingo! Thanks for that, with screen up and running I don't have to lobotomize bfgminer (though it's nice to do if you want logs).

Talking to SPI directly and playing with WAAS tells me all sorts of things. Like the fact that on this Titan with two working engines and two non-working (voltage shows up, small current draws)

Die 0 is insane. Powers up, can query, but can't set the nonces. Mumbles sometimes on the SPI, corrupting results from engine 2.

Die 1 is dead as a doornail. 4 amps of nothing.

Die 2 and 3 are great. Leaving die 0 off gives a solid 40mh.

With die 0 on, 1,2,3 off:

 [2016-02-12 22:36:00] KnC 1-0: Found TITAN die with 571 cores
 [2016-02-12 22:36:00] KnC 1-1: No KnC chip found
 [2016-02-12 22:36:00] KnC 1-2: No KnC chip found
 [2016-02-12 22:36:00] KnC 1-3: No KnC chip found

...

[2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x01, get 0x00
)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x01, get 0x00
)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x02, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x02, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x03, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x03, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x05, get 0x00)

...

[2016-02-12 22:36:13] KnC spi:
 [2016-02-12 22:36:13] 00000000   82 59 81 00 ff ff f2 04  00 00 00 5e f2 33 e3 a6 |.Y.........^.3..|
 [2016-02-12 22:36:13] 00000010   3e 2d 18 36 1c b2 4d 90  06 9d 90 51 d8 85 92 0e |>-.6..M....Q....|
 [2016-02-12 22:36:13] 00000020   e5 43 cd ec dc c6 53 86  6f e2 7c e3 b1 a9 7a 6b |.C....S.o.|...zk|
 [2016-02-12 22:36:13] 00000030   a6 a1 44 5c 17 f2 bb 9c  50 e6 22 45 6f 1d 48 b1 |..D\....P."Eo.H.|
 [2016-02-12 22:36:13] 00000040   16 ca e7 25 ea 4b 14 cf  82 4e 99 48 5e be 56 3e |...%.K...N.H^.V>|
 [2016-02-12 22:36:13] 00000050   38 01 1b 6f b2 86 cd 00  00 00 00                |8..o.......     |
 [2016-02-12 22:36:13] 00000000   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000020   00 1a 7a 52 b3 00 00 00  00 00 00 00 00 00 00 00 |..zR............|
 [2016-02-12 22:36:13] 00000030   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000040   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000050   00 00 00 00 00 00 00 81  00 ff ff                |...........     |
 [2016-02-12 22:36:13] KNC 0[1:0:65535]: Core busy (0)

Basically flipflops between that and:

 [2016-02-12 22:36:13] KnC spi:
 [2016-02-12 22:36:13] 00000000   82 a7 80 00 00 00 cc 1d  69 27 00 00 00 00 00 00 |........i'......|
 [2016-02-12 22:36:13] 00000010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000020   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000030   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000040   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000050   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000060   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000070   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000080   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000090   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 000000a0   00 00 00 00 00 00 00 00  00                      |.........       |

Yep, it's stupid. Interesting.

[lightfoot]

Edit: Damn, I am wrong. They are not ADP20's as I thought. But they are something. Let's describe what they are here.

First, they have power input from pins 4 and 6 of the SPI board. Interesting. Power comes in on pin 5, I know this because they put a stupid cap to ground and they always seem to do that. Likewise Pin 2 seems to be ground.

Pin 1 and 3 are unusual, they tend to go to pin 1 of the 10pin on u18, pin 9 on U17, etc.

Pin 4 and 6 go somewhere except on U19, where pin 4 goes to pin 3 on the 10 pin and 6 goes to gnd maybe.

The chips are 6 pin TSOT, the labeling is Z17R on Titans, Z17c on Neptunes except for one which is labelled THR.

So what are they? Jury is out, but something has to be powering the hotel on the chips.

More damn, trying to search on those parts is giving me crap like this:



Ok, fine, cute,but not what I need right now.

I really wish someone at KNC would just drop me a damn hijavascript:void(0);nt....

[GenTarkin]

Question: From the command line is it possible to screen over to see the cgminer process running? I'd like to find out the error rates.

Yes, you need to login with username 'pi' instead of 'admin', then screen -r will bring it up.

Bingo! Thanks for that, with screen up and running I don't have to lobotomize bfgminer (though it's nice to do if you want logs).

Talking to SPI directly and playing with WAAS tells me all sorts of things. Like the fact that on this Titan with two working engines and two non-working (voltage shows up, small current draws)

Die 0 is insane. Powers up, can query, but can't set the nonces. Mumbles sometimes on the SPI, corrupting results from engine 2.

Die 1 is dead as a doornail. 4 amps of nothing.

Die 2 and 3 are great. Leaving die 0 off gives a solid 40mh.

With die 0 on, 1,2,3 off:

 [2016-02-12 22:36:00] KnC 1-0: Found TITAN die with 571 cores
 [2016-02-12 22:36:00] KnC 1-1: No KnC chip found
 [2016-02-12 22:36:00] KnC 1-2: No KnC chip found
 [2016-02-12 22:36:00] KnC 1-3: No KnC chip found

...

[2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x01, get 0x00
)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x01, get 0x00
)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x02, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x02, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x03, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x03, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x04, get 0x00)
 [2016-02-12 22:36:06] KnC 1-0: Failed to set nonce range (wanted 0x05, get 0x00)

...

[2016-02-12 22:36:13] KnC spi:
 [2016-02-12 22:36:13] 00000000   82 59 81 00 ff ff f2 04  00 00 00 5e f2 33 e3 a6 |.Y.........^.3..|
 [2016-02-12 22:36:13] 00000010   3e 2d 18 36 1c b2 4d 90  06 9d 90 51 d8 85 92 0e |>-.6..M....Q....|
 [2016-02-12 22:36:13] 00000020   e5 43 cd ec dc c6 53 86  6f e2 7c e3 b1 a9 7a 6b |.C....S.o.|...zk|
 [2016-02-12 22:36:13] 00000030   a6 a1 44 5c 17 f2 bb 9c  50 e6 22 45 6f 1d 48 b1 |..D\....P."Eo.H.|
 [2016-02-12 22:36:13] 00000040   16 ca e7 25 ea 4b 14 cf  82 4e 99 48 5e be 56 3e |...%.K...N.H^.V>|
 [2016-02-12 22:36:13] 00000050   38 01 1b 6f b2 86 cd 00  00 00 00                |8..o.......     |
 [2016-02-12 22:36:13] 00000000   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000020   00 1a 7a 52 b3 00 00 00  00 00 00 00 00 00 00 00 |..zR............|
 [2016-02-12 22:36:13] 00000030   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000040   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000050   00 00 00 00 00 00 00 81  00 ff ff                |...........     |
 [2016-02-12 22:36:13] KNC 0[1:0:65535]: Core busy (0)

Basically flipflops between that and:

 [2016-02-12 22:36:13] KnC spi:
 [2016-02-12 22:36:13] 00000000   82 a7 80 00 00 00 cc 1d  69 27 00 00 00 00 00 00 |........i'......|
 [2016-02-12 22:36:13] 00000010   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000020   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000030   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000040   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000050   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000060   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000070   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000080   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 00000090   00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
 [2016-02-12 22:36:13] 000000a0   00 00 00 00 00 00 00 00  00                      |.........       |

Yep, it's stupid. Interesting.

It would be interesting to see if theres a way to complete the functionality of the Titan bfgminer driver for disabling and enabling a die.
If you look at the driver source code for titan, you an see where it says "not implemented" for disabling / enabling dies while bfgminer is running. I think if that code could be finished, I could issue hard resets of dies without having to start / stop bfgminer.
Because, just telling the DCDC's to power down for a die while bfgminer is running wont actually fix the "sleeping die" problem. Its like the "data lines" are still opened by bfgminer and a full die powerdown cant be performed. Its my theory that if we could get the "Enable / Disable" die functionality to work in bfgminer ... that may be the trick to allowing the full reset of a die to take place without having to restart bfgminer.
I just have no idea how to even begin doing something like that. Im not familiar enough with this lower level code =P

That being said, how did u query the spi information from the Titan die???
I wonder if this can be used to configure the die outside the limits set in the controller board(such as clock speed).

[lightfoot]

These dingbats have a picture of it, but no description.

http://www.aliexpress.com/item/Transistor-IC-chip-SMD-6-pin-IC-THR-SOT23-6/32240467148.html

Bedtime.

[sobe-it]

Well I just bought 10 Cubes with 3 controllers, hopefully they work as he stated. I have been following this thread but now I may end up playing along. First thing I will try just powering up the controllers alone and try each cube one by one.

[lightfoot]

Neat! I highly recommend taking things one slow step at a time in setting them up. For example I would:

Start up each controller without anything attached. Look for the boot, the IP address, and of course the light show. Upgrade each controller to the latest software version and make sure it's doing what you would expect.

Fire up each Titan on a power supply. Smell it. Smell for burned stuff, see if it blows up the power supply, etc. Then (if you know how to check resistance look for shorts on the 10 pin plug to ground. Only one pin should do this). After doing that I would plug in a Titan to one controller, make sure it works, hashes, whatever then move it to the "production controller" along with peers.

Good luck and mine like hell!

[lightfoot]

So I got another controller working here. This was the third of three fried controllers, this one was the worst with lifted PCB traces on the TPS power supply side. To be honest it took *days* to troubleshoot and clear the three shorts and one open connection. But it's all cleaned up, new solder to bridge the connections.

I did have to use a .5mm solder ball to bridge one of the connections, but that's life, it's a sense lead not a power lead (thus resistance is negligible) and it's securely holding without any shorts anymore.

Put down the FPGA (getting very good at this), fired it up, nice white then green light and it's hashing on the test Neptunes. I'll box it up and send it out tomorrow to another hopefully happy customer.

Time for lunch.

[sobe-it]

Neat! I highly recommend taking things one slow step at a time in setting them up. For example I would:

Start up each controller without anything attached. Look for the boot, the IP address, and of course the light show. Upgrade each controller to the latest software version and make sure it's doing what you would expect.

Fire up each Titan on a power supply. Smell it. Smell for burned stuff, see if it blows up the power supply, etc. Then (if you know how to check resistance look for shorts on the 10 pin plug to ground. Only one pin should do this). After doing that I would plug in a Titan to one controller, make sure it works, hashes, whatever then move it to the "production controller" along with peers.

Good luck and mine like hell!

Well I should say they are Neptunes, I wish they were Titans. I am just getting bare boards and cables only to save on shipping. But I could eventually buy a Titan cube here and there. By far the best $/Ghs deal I have ever gotten. Hopefully if all is well I will throw on some heat sinks and solder another 6 pin on them, hopefully underclock/undervolt to replace my noisy s5s in my house.

[helipotte]

Looking to buy a Neptune in hopes of underclocking it for maximum power saving.  What is the lowest voltage you have used and had all the dies come

up and hash.  Do these things to .5w/GH or better?

[lightfoot]

Well if you run them more slowly then one connector should be fine.

In the meantime I unboxed a shorted Neptune that came in today. Unlike many of the shorted units this one had no capacitor damage, everything looked fine. Hm. So I checked continuity on the 10 pin connector, all good there, then checked the resistance of the cube power lines.

One pair of supplies had 0 ohms, which usually points to a short in the high side FETs. The reason is that those FETs kick power from the 12 volt rail to the .7 volt rail. The trick is they are on for a very short time, so the voltage rises a bit then goes down.

In this case one of the FETs had shorted (probably overheat) which put the full 12v on the chip die. Down goes the unit, but more interestingly bye bye goes the die. Removing the FETs and restarting shows a die that will suck power, but will not come up to hash. Still this is another avenue for fixing a shorted Neptune or Titan and can probably get at least 2-3 dies back online.

Never dull. Got another controller to fix, and am packing up the latest one to go back to the client. I still haven't figured out the Titan issue, if I can't by tomorrow I will send it back and back-burner this for a bit.

[boomin]

I have to say - Lightfoot is the MAN!  Not only did he fix a few of my controllers for me - he also advised me about the tools I should buy to tinker!  I just successfully fixed one of my own controllers!  

Lightfoot is a different kind of cool.  The kind that are in rare form these days!

I just told him, this is so cool!  I only had to spend a little over $500 in tools, tons of hours learning and practicing and now I fixed my own controller! HAHA

Las Vegas now has a little more hash! LOL

Thanks again Lightfoot!

Boomin