Yep. Interesting. Is the FPGA hot?

Did it not work with just the one cube by chance?

So pulled one of the dies off this evening. Not easy, I had to get under it with a knife after cutting around it, that ribbon/mask is of course glued to the side of the chip....

The dies themselves are a micro-sized BGA, never seen anything quite like it. So small you can only see the grid with a loupe. But yes, that solder in there can melt and when it does it smears. Most times it just shorts out the chip, sometimes it hits the one signal to spi (mumbles), and sometimes it hits the supply for the die (which shorts the die). Reballing would be interesting, less than a .2mm pitch.

In further notes, heating just that corner of a chip is not enough to clear a short. Damn.

Or maybe not. I just pulled a die and under it is what can be best described as an IGA (insane grid array). Let's just say it looks like a ball pitch of zero, but still bga. Maybe .1mm. But yes, the failure mode seems to be die overheats, melts solder under it, and either just shorts the +.6v lines or if very unlucky shorts the 3.3v supply to power the die's housekeeping.

Fascinating.

Scraping it off is not the issue: What we need here is a way to break that bond without putting up-pressure on the carrier since this thing is glued to what is basically a flexible interconnect soldered to the pins. Put up pressure on it and you will rip the interconnect.

Oh and you need to do it while it is on the board so we can have a running part at the end of the day. :-)

Can't use heat or you will cause the dies to explode their solder because *they* are glued to and when the solder heats up it has nowhere to go till it blows out (which is why the dies short).

Cheap nail remover (acetone+water) isn't doing anything. Will get some full strength acetone, question is what will that do to the board.

However if I can get the lids off pulling the bad die should be do-able. In fact re-seating the bad die should be do-able as well....

No clue. However those solder balls around the bottom two dies are interesting. It's possible that one problem is an overheat which extrudes solder, either shorting the die's +.8v supply or (way worse) shorting the SPI communication bus (which is common to all 4 dies).

Hm.

And the most interesting thing under a loupe: See those little balls around the dies? Those are solder balls. Probably from under the dies, extruded either due to overheating while pulling the top or overheating while running. That could short out a die very quickly, and could be the source of a control line short to ground.

Hm.

Meantime I tore open a chip to see what's inside. Needed stupid amounts of heat.



Now to see what can dissolve that glue. Theoretically I could open a chip, desolder and pull the bad die, and get things working again....

And in the meantime I've been working hard on a way to isolate chips. To that end I finally took the top off a chip to see what's up.

Here it is.



Note removing it was a bitch on wheels, however it told me a few things:

1) There are only four small dies on a chip, location of each is where the heat is. Focus your heat sinking on that.
2) If the board tacos and the dies lose contact with the lid you're fucked (what I just fixed by de-tacoing)
3) Whatever that glue crap is required almost 500c of top heat after soaking the chip at 360c. Even then was not enough to keep the fabric mat from delaminating.
4) The mat is interesting. I can't see the possibility of shorts inside the chip but outside die so shorts must be hard inside the die. Maybe I could blow them clear with enough low voltage current.
5) Cheap way to do it, the power bus underneath must have been a bitch. Maybe the chips needed the subframe support.
6) Need to find a solvent to dissolve the glue without damaging the dies. Then I could just pull the top, rip off the bad die, and go to town. Any thoughts?

Put another cube in that slot, set it to off, unplug it, then plug in the crazy cube in the same slot. If the other cube is not there it should come up as off. Needless to say don't have any other cubes plugged in with the crazy one in the event it decides to go batshit crazy.

New problem though, interesting.

Ah.... You have a cube that is going crazy. That's always kind of fun.

Any idea what it's doing? If you connect it and set all four dies to off is it still crazy?

C

admin/admin gets you into the web page the first time. Then you set a password. For ssh you use root at the username and the password you set as the password.

For a Titan controller you can also log in as user pi, same password as root

C

Does the green light go out when the ASICs are connected?

The white light coming on followed by green usually means the FPGA has booted and all is well. That's really weird.

Hm. How good are you at taking things apart? It's possible you have a heat sink compound that is no longer connecting the unit to the sink. When you take the sink and bracket off, clear all the old stuff off with alcohol, all of it, then put a thin layer of compond on the ship and sink, and screw down the screws tight enough that the heat sink no longer can torque around. Doesn't need to be crushing tight just snug.

C

There is a way to re-install the software on an SD card up on the KNC web site. For a neptune it's simply putting a bunch of files on a blank sd card, then plugging it in and letting it rip. That should restore it. For a Titan you use something like w32diskimager to re-install the software, then go from there.

If that doesn't work does the white light come on then off on the controller board or does it just flash?

C

This is interesting. So I fixed a bridge board for someone and since the source of the failure was their Pi was shorted hard (oh well) I got a new Rpi 2 model B.

Note: The one on the Titan is a Rpi MOdel B+ version 1.2 which is *NOT* source compatible with the Rpi2 in the boot loader code. Thus the default Titan code from KNC will not boot on a Model 2B+.

Instead you need the Rpi Model B+ version 1.2.

I could probably hack it to work but I feel a bit lazy today.

Well the taco board is working:

--------------------------------------------------------------------------------
 KNC 0:       | 43.62/40.47/40.58Mh/s | A:2478 R:3+0(.12%) HW: 331/.41%
 KNC 1:       | 74.94/74.33/74.54Mh/s | A:4749 R:5+0(.11%) HW: 863/.59%
--------------------------------------------------------------------------------

KNC0 runs with two dies at 300mhz each, the other two cause the weird mumbling error with lost nonces all over the place. Turn them off and it runs more happily. KNC1 is the control running at 275mhz.

Fixing this sag required me to put the board in the jig, snug it up to add some tension, then pre-heat to 300c followed by 390c hot air on the chip for 2 minutes. That seems to have melted the solder (yes, lots of 951 flux) balls enough to re-seat part of the chip. Further heat probably will not fix the other two dies, and I am advising the owner that 2 dies is a lot better than 0. :-)

On to the next problem.

Boy that's interesting! Nice job, what were the symptoms before re-pasting the dies?

I just took out about 50% of the hog on a board that was kind of shaped like a taco chip with all dies dead. After a reflow and gentle torquing of the board to remove the sag, I have two dies are running at full speed, and the other two put out a lot of noise and fail (so they have to be disabled). The source of the hog was probably having the heat sink screwed down way too tight (don't do this).

The remaining two do the "mumble" error when powered on, in BFGMiner you see tons of shares without requests coming in on all four dies, making the unit hash slowly. Still pulls full power on the dies, but garbage output. Turning off the dies makes the remaining two stable at 300mhz/40mh.

Edit: This is an example of the gibberish message:

[2016-03-14 21:33:38] KNC 1b[5:2:76]: Got nonce for unknown work in slot 5


As to water cooling I tried a Corsair H100 on a KNC Neptune. Worked great in terms of keeping the CPU cool, right up to when the unit leaked and vented the coolant with a foom. Oh well.

And since lithium fires bring their own oxygen to the party it will be.... funny.

Here's a new one: A board that is shaped kind of like a Taco:





And the back is burned as well:



Note: Don't screw down the heat sink too hard. Adding washers should be done with a lot of care as this can cause the board to flex, get hot, then delaminate under the chip.

Hm......

So a busy weekend. On this second board that came in with the totally burned +12v lines I decided to try and hook every supply into the +12v bypass bus like so:



For a total of seven connections to the board (the last supply is too close to the power bus to be able to connect it directly). Once again the board works fine at 200mhz on all dies but when I brought supplies 1,2,3,4 (the four supplies up by the front of the board, not the back near the supply) to 250mhz I would get running fine for hours, then one supply would drop off and shut down that die. Odd.

So adding supply points doesn't seem to fix that. Still, it's a 55-60mh cube now instead of a 0mh cube so that's nice. But it seems that wiring every supply does not improve the performance to full.

Back to figuring out the shorts in the chip dies, but at this point I seem to be able to fix totally cremated plugs as long as the chip is not shorted (can be seen by checking resistance of pins 4,6,8 to ground)