My BFL SC Single 60 went out in a puff of smoke - what next?

[peterepeat]

Hi All,
I had my SC Single running nicely at 64 GH until a few days ago when it stopped hashing. I thought it may have been the power supply, and tried a brand new Corsair GS800 PSU. Anyhow, once connected, I heard a "crack" sound, no fans spun and a fair amount of smoke exited from near the middle of the PCB from underneath the smaller black heatsinks (not the ASICS). I doubt BFL will be any help, and with shipping to Australia taking so long+expensive - anyhow, has anyone else experienced this, and any suggestions on what I can try to fix it?
Cheers

[1714976175]

1714976175

[1714976175]

1714976175

[1714976175]

1714976175

[1714976175]

1714976175

[1714976175]

1714976175

[Matumaru]

I am not a electronic expert... but "crack" sounds like some bug exploded. Talking about the plausible cause... I would bet for static electricity while manipulating the machine.

I would remove (CAREFULLY) the heatsink and review the PCB, even with a magnifier in order to locate the exact place and component.

Or you can post a picture of the PCB both front and back and wait till someone take a look.

[demonmaestro]

puff of smoke. was it white smoke? and how loud was the pop?

[DrG]

Been there, done that.  I was sitting next to mine fortunately when it blew.  Heard a pop (probably a cheap cap) and white smoke out the back as the fans spun down.

Good luck with RMA, took me 2 weeks just to get attention of somebody on the BFL forums.

There are a few people who are doing mods like lightfoot - maybe he would want the dead device for a fee.

[peterepeat]

thanks - it wasnt a loud crack, and it was white smoke that had a really acrid odour. I thought I saw something bubbling as well (maybe it was electrolyte from a cap?) - anyhow, I emailed BFL when it happened, but havent heard a peep from them, so I guess I will try and get the heatsinks off and have a closer look tomorrow. Ill try and get some pics

[cowandtea]

thanks - it wasnt a loud crack, and it was white smoke that had a really acrid odour. I thought I saw something bubbling as well (maybe it was electrolyte from a cap?) - anyhow, I emailed BFL when it happened, but havent heard a peep from them, so I guess I will try and get the heatsinks off and have a closer look tomorrow. Ill try and get some pics

good luck, you can try sell it if you want.. im sure there are people who buy broken asic...

[runlinux]

Have you tried giving them a call during normal business hours? They have answered my calls in the past. With this possibly being a warranty issue, I can see them trying to take care of it.

[peterepeat]

Have you tried giving them a call during normal business hours? They have answered my calls in the past. With this possibly being a warranty issue, I can see them trying to take care of it.

Thanks. I'll try to get hold of them

[peterepeat]

So I took the covers off to try and get a closer look. It appears what ever is underneath the small black heatsink failed.

[lightfoot]

Hm, neat! Let me take a look at that...

Question: Can you take a picture of the whole half of the board? And the whole board front and back?

Edit: Fast thought: Based on where that is, I'd say it's one of the FET clusters (push/pull) for the 1 volt power rail for the board. The little sc/jalapeno has two of them, I wonder if the 60 gh SC has four or eight...
 
C

[peterepeat]

Hm, neat! Let me take a look at that...

Question: Can you take a picture of the whole half of the board? And the whole board front and back?

Edit: Fast thought: Based on where that is, I'd say it's one of the FET clusters (push/pull) for the 1 volt power rail for the board. The little sc/jalapeno has two of them, I wonder if the 60 gh SC has four or eight...
 
C

Thanks for having a look

heres the other pics

[lightfoot]

Thank you for posting those. I hadn't seen the newer boards fully loaded, that explains a lot.

In a nutshell, that section is where the 12 volt to 1 volt FETs are. There are two sets per side, for a total of four sets per big board (the little_single and jalapeno have two sets). Each set consists of three FETs which gate the "high" half of the 1 volt supply, and three FETs which gate the "low" half. At any moment, half the FETs are on. The capacitors then smooth out the flow and the voltage sensor on the 1 volt side feeds the computer back information on how long to keep the FETs on/off to maintain a smooth 1 volt.

When a FET fails due to overload or whatever it usually shorts. This puts the high and low lines on the 1 volt bus at the same time which shorts out one of the low FETs and the fuses blow in the power supply. Putting a new supply on it just fed straight into the dead short and let out the smoke in a "poof".

I've worked on things like this except they were 400a 300 volt IGBTs that would short on electric car transmissions. Same results with a *LOT* more smoke before the 350 amp main fuse would blow with a BOOM. :-)

Chris

[peterepeat]

Thank you for posting those. I hadn't seen the newer boards fully loaded, that explains a lot.

In a nutshell, that section is where the 12 volt to 1 volt FETs are. There are two sets per side, for a total of four sets per big board (the little_single and jalapeno have two sets). Each set consists of three FETs which gate the "high" half of the 1 volt supply, and three FETs which gate the "low" half. At any moment, half the FETs are on. The capacitors then smooth out the flow and the voltage sensor on the 1 volt side feeds the computer back information on how long to keep the FETs on/off to maintain a smooth 1 volt.

When a FET fails due to overload or whatever it usually shorts. This puts the high and low lines on the 1 volt bus at the same time which shorts out one of the low FETs and the fuses blow in the power supply. Putting a new supply on it just fed straight into the dead short and let out the smoke in a "poof".

I've worked on things like this except they were 400a 300 volt IGBTs that would short on electric car transmissions. Same results with a *LOT* more smoke before the 350 amp main fuse would blow with a BOOM. :-)

Chris

Thanks for your input lightfoot - I will see if a local place can rework it next week. Maybe still some life left in it yet! 

[peterepeat]

I took the heatsink off which reveals 2 failed components I'm guessing Q2 and Q4 have failed. I couldnt see the identifier on these components, however one in the next row is 014NE2LI which I think is a power MOSFET.
The image of the heatsink shows the magic smoke escape locations.....

[jojo69]

yup, VRM

question now is, did they fail because the ASIC failed and presented them with a bad load?  or did they take anything else out when they went?

[Cranky4u]

Hi All,
I had my SC Single running nicely at 64 GH until a few days ago when it stopped hashing. I thought it may have been the power supply, and tried a brand new Corsair GS800 PSU. Anyhow, once connected, I heard a "crack" sound, no fans spun and a fair amount of smoke exited from near the middle of the PCB from underneath the smaller black heatsinks (not the ASICS). I doubt BFL will be any help, and with shipping to Australia taking so long+expensive - anyhow, has anyone else experienced this, and any suggestions on what I can try to fix it?
Cheers

I am an Electrical Engineer located in Bendigo, Victoria, Australia. I would be happy to take a look at your SC single to see if it can be bought back from the dead...PM me if ur interested

[DrG]

Build date 9-11.... no conspiracy theorists jumping on that one yet? 

[Turbor]

Build date 9-11.... no conspiracy theorists jumping on that one yet? 

Saw that too. Suicide single of death

[vm1990]

Build date 9-11.... no conspiracy theorists jumping on that one yet? 

3rd image bottom right "fun pass" fun for who? BFL because of all the money?

anyhow it looks f***d XD and a bit of my ability XD that said it dosnt look like its dmaged anything else other than the 2 chips so it dosnt look like the voltage got any further meaning they did there jobs and failed (probably failed a bit to well and made a nice mess)

[lightfoot]

yup, VRM

question now is, did they fail because the ASIC failed and presented them with a bad load?  or did they take anything else out when they went?

Sharing my experience with a shorted chip, I can say it was probably not that.

Specifically as I add chips to my jally, the first chip I added I accidentally shorted some pins so +1 was connected to gnd. Happens when you move the chip while placing it hot. Anyway, the symptom (documented in the forum) was that the unit would "fast flash" on startup and I thought I was fucked. Removing the chip allowed the unit to start normally.

What seems to happen is that if +1 gets hard shorted, the board detects it (probably in hardware for speed) and the oscillator that gates the FETs shuts down. This immediately will collapse the voltage on the +1 rail to zero, and the board does it's "fast flash" dance. No damage. I think BTW this is why some people who take the heat sink off their jally to reprogram it get the fast flash; they torque down the damn heat sink too much and crush the chips into the board. Short develops, jally don't work no more. Solution would be to pull the chips and replace them, board is probably ok. But that's jally 101, we're in single 60 land.

Anyway, in this case it looks like running temps on the FETs have been high for some time. FETs don't share loads equally as temps go up, that's why on the better/bigger/more expensive electric car controllers you use a big 300-600amp IGBT instead of a bank of 20 30 amp FETs in parallel. but IGBTs cost large money and need more expensive drivers around their 2708's, but I digress. Moral is when a Curtis golf cart controller is used to drive a small car the FETs heat unevenly and the hot one is the first to short. When you short 600 amps at 150 volts, hilarity ensues as the other FETs blow up around it. :-)

I would guess that Q4 and Q(burned to a crisp) were heating up together, with Qcrisp finally failing shorted. That would probably also fail Q12 and provide a dead short from +12 to ground and blow the fuse in the power supply. End of story, board down.

What caused the total warping though on Qtoast is hooking up a bigger 12 volt supply. Since it's shorted, all current from the supply would shoot through Q4/Q12, and that would basically be a resistive load. It would heat up until either the silicon melted (which it did, letting out a lot of smoke) or the power supply fuse blew. The board couldn't stop it because the FETs were shorted, so the gates could not be opened under programatic control. Foom.

I've seen this happen on a 300 amp IGBT; the main pack fuse opened, but in the brief meantime there was about 90,000 watts of heat being generated in that IGBT. This is why they have big big big heat sinks. And why you have DC rated fuses, if someone was stupid enough to put an AC rated fuse in or bypass the fuse you would have a 90kw air heater in your aluminum box. Hilarity would... ensue...

What to do? I think there's a way forward, I'll post that next. Peter took the time to post this intel, I'll post a possible fix. Note I take no responsibility for *ANYTHING*, this is all pure speculation that I would never expect anyone to try in any way, shape, or form.

C