Close. I can see how to clear the lines for the pin 6 connection to the chip. There's a reason why I can't blow this open with voltage, it's eight connections into the chip off the transfer line. Given the chance it will destroy the board transfer lines before the chip core opens.

Unfortunately it's a one-way trip to clear the lines so I need to be able to find the *right* die. Back to work.

Interesting development on the pin 6 short thing this morning.

Got another board, this time it's a Neptune that had a destroyed power plug that someone repaired... poorly. The cube does not run. So as a part of my inspection I checked the resistance of the pins.

Pin 6 is VERY unusual: It is supposed to read 10k or so ohms, Titans that have had power plug issues can read 0 ohms (kills the controller board). This one doesn't read 0 ohms.

It reads 17 ohms. *VERY* *VERY* interesting. Every time I have seen this, it's been on a unit with a burned power connector. Not all burns do this, but I haven't found a unit that wasn't in a burned situation with a short on those lines.

So I take a chance, clean up the plug solder connections (cold solder joints do not conduct properly, you NEED board heat to do this...) and plugged it in. Get this at 100mhz clock:

KNC 0a:       |   0.0/  0.0/  0.0 h/s | A:  0 R:0+0(none) HW:  0/none
KNC 0b:       | 34.42/36.08/36.08Gh/s | A:  4 R:0+0(none) HW:  1/.02%
KNC 0c:       | OFF  /  0.0/  0.0 h/s | A:  0 R:0+0(none) HW:  0/none
KNC 0d:       |  7.70/14.28/128.5Mh/s | A:  0 R:0+0(none) HW:128/ 88%

0b is running, 0d is trying to run but wrecked, but look at 0c. It's "off".

I think the short here is in cube die 0c. I think in this case the short isn't to ground, it's to the chip engine's +vcc so chip control is shorting to chip engine power. And I think this is happening on die "C"....

Now to rip apart anything around die C to see if I can clear this. Finally I have a location on the chip, I can't get this with Titans because they short the controller. But in this case I have a chance to find this.

Food time.

(Edit: Look at this insanity at 200mhz)
 KNC 0a:       |   0.0/  0.0/  0.0 h/s | A:0 R:0+0(none) HW:   0/none
 KNC 0b:       | 114.2/173.7/346.7Gh/s | A:0 R:0+0(none) HW:4750/ 59%
 KNC 0c:       | OFF  /  0.0/  0.0 h/s | A:0 R:0+0(none) HW:   0/none
 KNC 0d:       | OFF  /  0.0/  0.0 h/s | A:0 R:0+0(none) HW:   0/none

It was at 400+gh for a second there. All through one die? Nope, short.....

Aw damn, you're going to beat me to it. The key for me was realizing that the fpga-titan code can be loaded with the neptune fpga loader and the io-pwr command from the neptunes.

Couple that with compiling up the driver support code with a few hacks into bfgminer and you're up and running.

Nice job, thanks for working on this. Now I have more time time fix hardware fails.

If you can get a Titan processing board, sure. The controller baseboard for the Neptune, Titan, and Jupiters are identical, the Titan uses a Raspberry pi plus a bridgeboard but I'm whacking away at getting a modern beaglebone running. The trick is figuring out how to remap the gpio ports, but I'm working on it. Unix lets you pull all sorts of funny crap.

Myself, I am not mining so much as fixing other people's stuff. The titans are worth fixing at this point, good systems.

So in the meantime I figured out two other things:

1) How to fix a completely wedged Beaglebone: The normal recovery image doesn't work, the thing does nothing but light the one blue LED and you feel sad. Solution is to build a beaglebone/debian image, set it to flash to the emmc. It will pound for 30 mins with the cylon light pattern, then go solid. Reboot, then install the recovery image and restart. Presto, emmc is back, beaglebone is up, miller time!

2) Fix a display. If you have a display that's crapped out but lights check the little connectors on the bottom that go to the board. On this one it turned out that the connectors had literally not been soldered down properly to the board's pads. A little work with flux first then a soldering iron at 700f (something C) and a quick touch to each pad reflows the solder and makes for a happy display.

On to other stuff.

Hardware failure is one of those really hilarious terms in Titan-land. I'll have to grab a copy over the weekend, hopefully you take... bitcoin? Or maybe that litestuff? :-)

On an un-related note I figured out how to fix wedged Neptune/beaglebones, fix dead display devices (hint the KNC soldering job sucks), and now that I have a spare am fiddling with the concept of how to run the titan firmware. FPGAs are up and running, just need to tweak the titan driver in bfgminer....

Never dull.

The die pulls power, but throws errors from time to time on the die that shuts down either the die or the die next to it as well. Fixed by cycling the miner, but then it comes back. Sometimes you see the current on the supplies drop to zero.

I don't have a unit now that does this, but did awhile back.

Other annoyances include dies that hash fine for awhile, then the supplies drop to zero. Rebooting the cube fixes it, sometimes dropping the supplies to zero does so as well. Also an odd one, I had a unit that once had a power supply simply disappear. Because I was running it at 200mhz the other supply handled the load fine but it was odd.

These are the days of our lives. :-)

That is so cool! I wonder if it can fix a titan with a mumbling die.

Sent out the unit with 3/4 dies restored, that's nice. It's actually not that bad, you can run the 3 dies to 60mh total without drawing 200w from the supply so you still won't blow up anything as long as the DC-DC temps are reasonable.

And doing it for bitcoin works, it's fast, reliable, works across borders, etc. Maybe I should also accept litecoin...

Now to spend the weekend tackling these two other Titans. I'm going to try to clear the short on the first one after seeing if there is any way I can disconnect the line from the bottom of the board. The onto replacing all the signal components on this second board to see if I can get more dies working (it's one die and minimal quality at that. We'll see).

C

Sure. I can take a look at it. No guarantees I can get it working, but the fact that pressure on the chip makes differences is something worth taking a deeper look into. I'll PM you information, working on getting a few things out today.

What do you get with the waas command? And how does bfgminer look when cranked down (don't do this too hard).

Based on the condition of the back of the board my guess is that either it was run like super hell or the heat sink was bad/loose and the top dies burned the board. That takes more heat than I can generate with pre-heat and my air tools, so it's a lot of heat. Enough to melt the solder and make a mess under the board.

A reflow *might* help, but those are dicey. If you want me to try let me know and send it in. I'll give it a shot, but my guess is that's the root of the problem.

Anyone found a source for huge .6mm pitch reballing stencils?

Hate to say it but most of my success is on boards that are blown up. Either incinerated, blown supplies, and the like. The titan chips are finicky, have you tried Tarkin's firmware?

Anyway when you log into the board using ssh as user pi and password whatever your password is and type screen -r what do you see?

C

Thank you! At the moment it looks like this repaired unit hashes pretty well at 60 or so, will be contacting the user to discuss. I think it's a pretty neat repair, all things considered :-)

And on another Titan note, got one in that was dead shorted on the 12v side, turns out the person took off the heat sink and didn't re-apply paste when putting it back on. The paste KNC uses is a one time thing that hardens like a ROCK. The result is this:



And a DC-DC converter that blows its' FETs (you can see the worst heat damage is over in that corner of the board), which causes +12 to hit the hashing die which if the die wasn't already fried, ends the die. Fortunately the other three were fix-able and the unit is back to hashing at a solid 60mh.

On to the next thing. Still working on trying to fix a unit that hashes on only one engine (I think this was from the factory, it already has two bypassed) and thinking about how to clear these shorts on line 6.

Damaged die most likely. The power supplies are actually pretty reliable and the connections on the .6v side to the chip seem to be pretty good. So that's not it. These dies are flakey, I think if they go odd or down it's due to manufacturing failures. Hm.
 
In the meantime I spent the evening letting this unit hash while I looked at my sacrifice board. It has had a short on pin 6 since the day I got it, and nothing seems to clear the problem (the guy had three boards that all did this after a major power supply failure, one board was merely burned). This board was my sacrificial lamb, I used it to trace the shorts on pin 4 (to the chip but could be bypassed with jumpers), pin 8 (same) and pin 6 (which I could not clear).

So tonight I decided to clear the short. Pre-heated the board to full, fluxed everything, Pulled every component around the chip, every chip, capacitor, jumper, plug, supply, you name it. Nothing cleared the short. Then I pulled the chip, somewhat messy on the lift. Still shorted, but some solder smears. Cleaned up the smears and when I was on the last corner the meter stopped beeping.

Ran the solder ball back, BEEP! Sure enough I had cleared the final fault. Checked the chip at those pads and sure enough, short in the chip itself.

Fuck.

So the problem is in the chip. What is happening is that when the chip fails or gets voltage spikes or surges it takes out the housekeeping circuits on a die. This is a common line shared by all four dies on the chip and is not possible to isolate via the board. Without it, the board will not hash.

You're basically fucked.

Need to think for awhile about this: Without new hashing chips this really can't be fixed. I mean it's theoretically possible that one could apply a very high current 3.3 volt spike to try and blow the line open on the affected chip without taking out the other three chips, but that's really iffy. I might try this on board 2, but it is not what we call an optimal solution.

Hm. Bedtime.

Ok these are good: When your miner burns it's power connectors so badly it destroys the board connections:



Don't junk it, just build a new bypass circuit!



Yep, it works. Yep, I'm pleased with myself. More here: https://bitcointalk.org/index.php?topic=1283859.new#new

And how do you fix the above so it runs at a nice 60mh for 24 hours at a nice and smooth <50c power supply temps and <40c chip temps?

Why build a bypass of course!



This will go down as one of the cooler things I have done in a day. As discussed the via lines for the +12v supply to the board were destroyed. +12 is on the inside layers so you can't just connect. However what I figured might work is to build a bypass bus that starts on one side of the board, goes *through* the molex connector, then winds its' way around the board dropping power at several points to both minimize the load being dumped into a particular plane and to reduce potential noise coming from supplies near the drop site destabilizing ones further away.

First step was to pick the wire. After considering 10 gauge (way too big) and 12 gauge (stiff) I went with 14 gauge UL rated teflon coated copper stranded wire. Copper stranded because I could interleave the strands into the molex plug pins for maximum contact points, UL rated because typical automotive wire insulation is only rated to 80c whereas UL rated wire is 100c rated. We don't want the wire to smoke since it will be near the power supply inductors, which get hot. And 14 gauge because that is double the gauge of 16g wire which is considered "really good" in a power supply. 14 gauge has twice the capacity of 16g, so if I go in two directions I can equal 16g wire with 4 times the load.

Second step was figuring out a connection point. I thought about scraping the board by the power supplies (bad, hard to solder, and prone to lifting), the supply standoffs (close to electronic components under the power supplies, limited connection) then finally realized I could just make the wire flat, solder-tin it, cut it to shape, then tin and solder to the capacitor banks next to a power supply. That would give me 8 connection points into the board, with physical reinforcement (between supply and capacitors) and by using rosin core solder I could melt it onto the capacitors without having to worry about melting the connection between capacitor and board (which is ROHS solder that needs +100c more heat to melt).

Third step was forming the wire. This took awhile, with a lot of bends and thoughts. Ultimately I went for three connection points, at supply 2, 5, and 7. This leaves a slight imbalance on supply 4, but it's the best I can do right now. Other supplies will pick up power through the existing via and +12 ground planes. The power distribution would be odd, but it should hold as long as you don't pull max power on every supply.




Here is the wire being formed. The top one was the first attempt, I broke a strand while forming it, and since every strand is critical I gave it up and tried again. The middle one has the molex connector soldered on, once again I went with strands on both sides of the pins, yet high enough to keep it away from the board. The bottom (red) one is automotive wire. Thicker insulation for physical protection, but a lower rating and not UL listed.

Soldered the wire onto the caps making sure there was no tension/compression against any of the components (difficult) and soldered the Molex to the board. Tried it out for resistance, saw the usual 1k or so on the 12v lines. Then put the heat sink on and mounted it in the box:



Bent the wire a little bit with two pliers (remember *NO STRESS ON ANYTHING*) to ensure it doesn't touch the side (even insulated you never want things to rub) and tried my smallest supply.

Board came up. 50mhz got me 20mh total. 100mh got me 40, and 200mhz got me the magic *60*mh. So far the supplies are stable at .78v or so, with temps at 50-53c.

Titans were built to run at 60mh, running them over this starts increasing power draw and moves the supplies closer to their limits. On a normal board this is annoying, on this kind of board you will probably start to see instabilities on the 12v planes as power goes places it was not designed to go. At this point though the connector is cool, the voltage drop to the supplies from the 12v line is not bad (11.8v at connector input, 11.78v at furthest end) showing a .02v drop at 180 watts measured. Or a .3w drop in the wires, not too bad.

Very difficult to fix, took more time and effort than one might thing. Need to let it run for some more time, but so far so good. Moral: Don't burn boards. :-)

There is a way to do it with less heat; alloy the solder. Specifically put flux on the pins and melt normal solder onto it using a normal iron at normal temps. The lead solder will mix with the ROHS stuff lowering the melting temperature by a *huge* amount (basically to the level of pb/tin solder). Then the pins can come right out without any fuss or worry about overheating the board.

However if the pins look like this.



It's going to be a bit more... complicated. (Note how the pins are pointing in different directions, the actual anchors into the board have been burned clear through by the heat).

Bad.


Hm. What kind of lights come on and do they hash with Neptunes?

Night!

By the way, when you melt the connector to the point where the board burns it's a bitch on wheels to fix.

Melting is caused by a couple of things. When you plug a miner in while the power is on, there is a little spark that happens because the miner's 12 volt caps are charging up from zero. Each spark leaves a nice pock mark on the plug or socket. Eventually that causes resistance which causes heat which causes more resistance.

The other issue is voltage. The cube is going to take a certain number of watts. When you start lowering th voltage below 12v due to cheap cabling, the current draw has to go up. Current is what generates heat in voltage drop situations (P=IE), so when you voltage drops to 10v the thing is now dropping 2v*20a=40w of heat along that line. Then it gets hotter resistance goes up, hilarity ensues.

Third thing is the interconnections between the PCIe connector and the Titan board, if you look at the ground lines on layer 1 and 6 you see that the PCIe pin goes to a via which connects to a very small set of four spider connectors to the ground plane. They do the same thing with +12, it's only a few mm, but as you heat up those pins you will start to burn out those spiders. Once one pin's spiders go the other spiders have to carry more of the load which burns those pins and you know where this leads.

Solution? You could run a supply with a massive amount of power, but remember those cubes don't have fuses in them so if something like a FET shorts inside a power supply you go from a simple *pop* to what could nicely be described as a small plasma fireball (see Amp Interrupt rating). This can then cause the phenolic on the board itself to conduct which will turn your unit into a 3,000 watt electric dryer in a 1 foot cube. This could then be described as "hilarious".

Or try dropping down the power. These units hash nicely at 60mh, do you really need 80+?

Pounding away, busy weekend bending wire/ More later, moral please please do not blow out your connectors. Clean them, drop power usage below 200w/cube, use really good supplies, whatever. Bitch to fix.

Perhaps, in which case the only Titans that can be fixed are the ones that smoke power connectors or blow out power supplies, both resulting in shorting on power-up. This is possible, although sad.

However other things just seem odd. For example: I have another dead titan board in for a review. It hashes on one die. Checking in bfgminer I see that only two dies ever respond, the other two do *nothing* no matter what. It has the jumper missing.

I wonder what happens if I put the jumper in. Will all four dies respond like insane children? Why would KNC do this?

Then I look at the supplies. Two are missing, so I check resistances on the chip side. Normally I see around 7.5 ohms if chip is good, 0 if shorted (to ground). On this one I see four power supplies reading 7.5,7.5,7.5,7.5. The other four read 7.5,600 and 7.5,140 ohms.

THAT IS IMPOSSIBLE! The supply outputs are ganged together in parallel. There is no way on EARTH that the outputs can read higher resistance. What the heck is going on?

I'll pull a supply and see. But I'll also pop in a temporary jumper and see what happens.