That sucks. In this game downtime is doom, which is one of the reasons I'm giving all my thoughts out for free: I could never warranty my work in a reasonable manner given the difficulty increases, and the best fix could be "lobotomize half the thing" which would upset the owner but would you rather have had 2 weeks of 30gh hashing (and 7.5btc) or 2 weeks of zilch and a 60gh system running with 2 week higher difficulty.

That's math over emotion. Bitcoin cares nothing about feelings, it's all about math and time.

If the unit is stock, then BFL should replace it. They can detect the code change; they signed their code and our screwings are as obvious as a coal pile in a ballroom. If I were BFL I'd offer an immediate ship for a full charge deposit, returned when we get your unit and verify it's not custom. However in that case people here would game the shit out of BFL by using this to jump the line so that's why they can't do it. And I would be out of business because I am too much of a nice guy...

Which makes people bitch about BFL. But economically they would have no choice, see "bitcoin cares nothing about feelings..." :-)

And why I am not going to get stupid eBay-sentimental and pay anything more than what I think dead units are worth. They're dead for a reason, other people can fix them at a loss.

Honest thought here: I'm worried (why should I care? I don't know) that people have seen us jally hoppers getting 8gh out of our 5gh units and think "hey, I can do the same thing with my single 60, going from 60gh to 80-90-MORE!". But I don't think it works that way; our jallies are a unique special place and we can boost them without too much effort because they were built underpowered and overbuilt due to a unique event. BFL was totally fair to limit them to the amount we paid for, but we can take chances because they left all the FETs on the boards (probably because they were already built and such).

I believe the singles were DIFFERENT. They built them to run at 60gh, and there is not 120gh of overcapacity there. If you clock them higher there is a bigger chance that things are going to go foom. I will bet a dollar to a donut that if I put all 8 chips on my jally I will have a 50% shot of blowing the FETs within 2 weeks. I'm going to stop at 7, and heat sink the FETs and I'm watching them and not running the chips all-out.

Jallies cost us $169 which is peanuts in the grand scheme of things. People that bought a single/60 spent what $2000 for the thing? Playing mod with one of those takes balls of GOLD in my opinion. And although I have made $1000 in profit on my Sept delivery jally, I know people who bricked their units (and sent them to me to unbrick but they lost out on lower difficulty time) and would have done better to not have screwed with it without knowing what they are doing. I know what I'm doing sort of (I still bitch and whine when it doesn't WOOOOORK), I got lucky, and I have been pushing my luck to earn that money. I wouldn't do it with higher stakes.

Thus my thought of "don't dick with your single and think it's a big jally". It's not.

Ah well. This is a fun thought-experiment.

So.... What to do? Note all of this is from a complete random stranger on the Internet dreaming about stuff, so disclaimer, disclaimer, don't do any of this in real life, you're responsible for your own universe, this is just me.

If *I* had a board like this....

Bit of knowledge about the BFL design: If the 60 single boards are like the little_single, then the power supplies for the FETs are connected together and connected to the 8 chips that run on the board. What I don't know is if the left side 8 chips are powered by the left two power converters (the FETs that went foom) independently of the right side 8 chips. They may all be on one bus, or two.

Update: Based on reviews of the SC30, it looks like the two sides run separate chip power converters. That is good and bad from a design stage, but it can explain how one can lose half their hashing power.

Anyway, what's the damage? Well, it could be limited to the FETs, I would say scrape off the wreckage, hope the Pads behind them are still good (test for shorts and opens to the other FET baks) and if there are no shorts, try firing it with a regulated 12 volt supply (without any mining software, just see if the chips come up with the remaining 3 supplies). If they do then either lobotomize 4 chips on that side to be safe by turning them off in software (making a 45gh unit tops) or if this thing has all the power supplied in parallel lobotomize six chips to keep the remaining three supplies from overloading (assuming they also have some damage). And make sure your power supply's 12 volt source has a *FUSE*, wire in a (12 watt*12=144 watt/12 volts=12 amp fuse *TOPS* since you blew things start with a much smaller fuse TOPS). If you blow the fuse without hashing, you have a board short, start digging.

That's the fast way. Sure it's 35-45gh instead of 60, but half a loaf is better than none, all that. Peter if you have SMD soldering skills and you were me, you might want to try taking off all six FETs, check for shorts on the 1 volt to ground and 12 volt to ground lines, put in the power supply fuse on your lines,  and see if anything works (you need to remove both sides because they might be back-feeding the oscillator)

What's the worst that could happen?

If you want to fix this whole power supply it's going to be harder. My experience is that this will blow the FETs and possibly the gate driver chips. I don't know if they have 1 gate driver powering all twelve chips on a side, or two sets of gate drivers. If the driver is blown and they have two drivers per side, the other side should still work and you can power 12 chips. If the sides are isolated and they have one set per side you can run 8. If the sides are not isolated and they use one honking driver to power everything then you're screwed till you replace the driver (a 2708 chip on the bigger controllers, look for something like that in the chip mainfest).

What's the worst that could happen?

If the FET pads are warped, you might not be able to get it going without bypassing the 1 volt lines or re-routing power from the 12 volt source supply. That's way beyond the source of my ability to help from the other side of the earth.

So anyway, a fast potential solution that I would use to get me something would be:

Remove the wreckage with a good pair of soldering irons. Don't go with insane amounts of heat; the carmelized board will conduct heat faster and melt more than normal.

Check for shorts

Try powering it up without hashing with a fuse in the supply.

Reprogram it to only run 8 chips instead of 16.

Try hashing.

Add a few more chips, do not exceed 11 of them if you want it to last.

Good luck. Time is of the essence since difficulty keeps going up and time waits for no man in Bitcoin world. Having 30gh running now is better than 60 in a month's time. Good luck and keep us posted.

And thank you for sharing this: It's given me a chance to think through a problem, which to me is always fun. If anyone else finds this useful and it makes them money do me a favor: Donate some of that to your local soup kitchen in the name of "Bob Dobbs". That way my rantings here make the world a bit better or something like that.

C

Sharing my experience with a shorted chip, I can say it was probably not that.

Specifically as I add chips to my jally, the first chip I added I accidentally shorted some pins so +1 was connected to gnd. Happens when you move the chip while placing it hot. Anyway, the symptom (documented in the forum) was that the unit would "fast flash" on startup and I thought I was fucked. Removing the chip allowed the unit to start normally.

What seems to happen is that if +1 gets hard shorted, the board detects it (probably in hardware for speed) and the oscillator that gates the FETs shuts down. This immediately will collapse the voltage on the +1 rail to zero, and the board does it's "fast flash" dance. No damage. I think BTW this is why some people who take the heat sink off their jally to reprogram it get the fast flash; they torque down the damn heat sink too much and crush the chips into the board. Short develops, jally don't work no more. Solution would be to pull the chips and replace them, board is probably ok. But that's jally 101, we're in single 60 land.

Anyway, in this case it looks like running temps on the FETs have been high for some time. FETs don't share loads equally as temps go up, that's why on the better/bigger/more expensive electric car controllers you use a big 300-600amp IGBT instead of a bank of 20 30 amp FETs in parallel. but IGBTs cost large money and need more expensive drivers around their 2708's, but I digress. Moral is when a Curtis golf cart controller is used to drive a small car the FETs heat unevenly and the hot one is the first to short. When you short 600 amps at 150 volts, hilarity ensues as the other FETs blow up around it. :-)

I would guess that Q4 and Q(burned to a crisp) were heating up together, with Qcrisp finally failing shorted. That would probably also fail Q12 and provide a dead short from +12 to ground and blow the fuse in the power supply. End of story, board down.

What caused the total warping though on Qtoast is hooking up a bigger 12 volt supply. Since it's shorted, all current from the supply would shoot through Q4/Q12, and that would basically be a resistive load. It would heat up until either the silicon melted (which it did, letting out a lot of smoke) or the power supply fuse blew. The board couldn't stop it because the FETs were shorted, so the gates could not be opened under programatic control. Foom.

I've seen this happen on a 300 amp IGBT; the main pack fuse opened, but in the brief meantime there was about 90,000 watts of heat being generated in that IGBT. This is why they have big big big heat sinks. And why you have DC rated fuses, if someone was stupid enough to put an AC rated fuse in or bypass the fuse you would have a 90kw air heater in your aluminum box. Hilarity would... ensue...

What to do? I think there's a way forward, I'll post that next. Peter took the time to post this intel, I'll post a possible fix. Note I take no responsibility for *ANYTHING*, this is all pure speculation that I would never expect anyone to try in any way, shape, or form.

C

Why not: A picture of my BFL Jalapeno, currently hashing at 20gh, to be 28gh by this weekend.



Details at my write-up page here: https://bitcointalk.org/index.php?topic=336782.0

C

Hey everyone!

I'm Lightfoot, and I've spent the last few weeks boosting my Jalapeno from 5gh to 7.3gh to 20gh and hopefully by this weekend to 28gh. A thread on this is over here, and I'm having fun with it: https://bitcointalk.org/index.php?topic=336782.0

So I'm starting to get a good understanding of these boards. And I have a concern forming in my head for people with the Singles, specifically the 60gh ones.

Basically people have been hacking the Jallies to boost performance by overclocking. Which is fine; the Jallies were built on little_single boards and have a lot of room to play in. You can run them at higher clocks, leave dead engines online (they use full power and heat for nothing), and so forth in the quest to wring out the max hashes.

You can even be nuts like me and stick more chips on your Jally. I'm up to 5 chips now, going to 7, and I have learned a lot. You can tweak a Jally. It's fun and all that. But note this carefully, it is my opinion that:

YOU CAN'T DO THIS WITH A 30GH SINGLE OR A 60GH UNIT.

Here's my thoughts and reasons why:

The Singles appear to be similar to the little_single boards with updated FETs, more heat sinks, and of course 16 chips capacity. They both use the same 1 volt power supply to power the main part of the hashing chips, and each chip pulls 12-13 watts, which at 1 volt is 12-13 amps. Normally. When you overclock, they pull a lot more power, and I have seen this by greatly increased temps, which means more heat. Heat comes from only two places: Volcanoes under your mining rig, and the chips burning more power. It's usually the latter.

Anyway, the more power you pull, the more you stress those FETs. BFL uses 4 banks of them in a 60, each bank composing six FETs in a 3 way push/pull configuration. Overload these FETs and they will fail.

When they fail, they will fail shorted and crash your unit. I think they also blow the fuse in the power supply. Fixing this is a pain, but if you hook up a power supply without a fuse, your unit will burn the board and make smoke-like smells and boom like sounds. Board will be toast.

This failure mode is not unusual: Curtis controllers for electric cars are known for shorting the FETs and blowing up the controllers on 150 volt/600 amp systems when overloaded. Lot of boom and smoke. The fact that it's only 1 volt is irrelevant, these things are pulling 160 amps at 1 volt and that will make smokies.

This is why BFL has the code in the 1.2.8 and 1.2.9 releases that includes a DO NOT EXCEED 60gh thing. I thought it was them being a bit dick-headed, but I think the real reason is that this keeps the chips from driving the FETs into overload.

I'm seeing this on my jalapeno; every chip I add boosts the temps on my FETs. I've already got stick-on heat sinks; and I think the reason that BFL switched the board design was because a full set of 8 chips would cause the FETs to go foom too often. So they built the bigger board, balanced the load, put heat sinks and fans on those chips (explains the fans in the ends of the boxes; I have a vornado pointing at mine) and keeps the unit at 60gh.

If you go beyond, I think you are risking your unit.

So what it my unit is blown?

If you have already blown up your unit and you flashed it with hot code, I'd be willing to buy it. Now, I'm not going to start a home for wayward mining rigs, and I'm not irrational like ebay people. So I'll offer the following bitcoin formula:

Price is based on delivery 1 month hence (time for me to get it and repair) * .33% of it's total residual mining value as measured by mining.thegenesisblock.com using default values, 15c for power, 60gh for the hash rate, and 240 watts for the power use. As an example, my price right now for a 60gh, December mining date, unit would be:

.984 BTC *.33=.3276 BTC.

http://mining.thegenesisblock.com/a/17a5fd6db2

Note, I boosted the number from 20% to 33% because I believe a single/60 has a better than even chance of firing up as a 30. It is however better than nothing, and as time goes on the residual value keeps going down on these things. Ah well.

Anyway, I hope this helps people. My personal motto is "mine like hell" and I'd rather see people mining than nursing dead equipment.

C

Nicely done. Try bringing the speed down to 7, you will probably get rid of most of the errors. For 2 cpus the errors aren't that big of a deal, but it's wasted heat.

Or mine for awhile, I was able to run 7.3-7.4gh on mine before boosting it to 20.

C

Easy is relative, if you can use air tools you can add more chips on them. They're little_single boards, can take 8 but I WOULD NOT RECOMMEND GOING PAST 7.

Anyway, my thread is over here. https://bitcointalk.org/index.php?topic=336782.0. Worth checking out.

C

Hm. That might be the crappiest jalapeno in the universe :-) I guess it is possible that it's really that sad, I thought my chips were in need of love.

Did you scan it again after you flashed it?

I've found that lowering the error checking loops might help, as will boosting to speed 7. However don't disable error checking all the way, it just results in a lot of power wasted.

Perhaps you could put some more chips on it?

Chris

Thank you for posting those. I hadn't seen the newer boards fully loaded, that explains a lot.

In a nutshell, that section is where the 12 volt to 1 volt FETs are. There are two sets per side, for a total of four sets per big board (the little_single and jalapeno have two sets). Each set consists of three FETs which gate the "high" half of the 1 volt supply, and three FETs which gate the "low" half. At any moment, half the FETs are on. The capacitors then smooth out the flow and the voltage sensor on the 1 volt side feeds the computer back information on how long to keep the FETs on/off to maintain a smooth 1 volt.

When a FET fails due to overload or whatever it usually shorts. This puts the high and low lines on the 1 volt bus at the same time which shorts out one of the low FETs and the fuses blow in the power supply. Putting a new supply on it just fed straight into the dead short and let out the smoke in a "poof".

I've worked on things like this except they were 400a 300 volt IGBTs that would short on electric car transmissions. Same results with a *LOT* more smoke before the 350 amp main fuse would blow with a BOOM. :-)

Chris

Hm, neat! Let me take a look at that...

Question: Can you take a picture of the whole half of the board? And the whole board front and back?

Edit: Fast thought: Based on where that is, I'd say it's one of the FET clusters (push/pull) for the 1 volt power rail for the board. The little sc/jalapeno has two of them, I wonder if the 60 gh SC has four or eight...
 
C

*rolls eyes* Imagine getting a list of lottery winning numbers five months back.. you would be rich now...

:-)

I'm boosting the performance of my Jalapeno at this point. Turns out it's profitable to buy chips and load them on the board. Should be at 30gh by the weekend, at 20 now.

Aside from that I'm taking a wait-and-see approach. Mining contracts sound like a losing scam.

C

Update: Looks like the latest ebay seller is sold out. Apparently I'm good advertising. :-)

Also to note: Going to speed 9 actually slowed the unit down: The weakest chips went down to 2.2gh/s and temps went up. So I moved things back to speed 7 and speeds are back at 19gh.

Sounds good. I think it's worth it, and I'll be interested in hearing how you do with an IR unit. Post what kind of tool you're using and the temps that work out.

Still waiting for my chips. Looked at it again last night, still hashing solid at 19gh. Oddly enough though most of the chips are at 3.5-3.8gh, with only one at 4.2+. So there might either be a residual heat issue, or I have the testing up a bit too high. I should also try speed=9 to see if I can bring out more speed at the expense of more heat...

C

Bad news: My offer from the previous Ebay vendor for the two chips was declined. Drat.

However I placed an order with a different vendor and got the two chips no problem. So now I can go to at least seven, should have them by Saturday for an install. I'm just going to put them in the last two spaces (one corner, one middle of row) that I have not soldered chips and take a good long look at the last spot. If it looks ok then I'll try getting one more chip and going for a full 8.

Here's the vendor link: http://cgi.ebay.com/ws/eBayISAPI.dll?ViewItem&item=151168046486. Looks good, good reputation, I'd say order 4 chips at a time and see how it goes.

The power supply arrived, and temps on my rig with the heat sink have been a nice reasonable 60c. So I have room to expand both in terms of power and in terms of cooling capacity.

Next stop: 28gh!

Well, in the "fail" category I think I damaged this first chip beyond use.

This evening I took it outside and removed the existing solder balls with the Radio Shack monster temp controlled iron, set at 725F with a lot of rosin flux and traditional solder removal wick (copper braid). That did the trick far better than hot air; and I got the balls off in quick order.

Rinsed it with 91% isopropyl and took a look at the pads under a 6x loupe. Several of them are missing, along the edge, and I'm guessing that might be where the communications lines were. Which is worse than losing a pad or two in the power side. Anyway this picture shows it, the dot is on the other side of the chip, upper right.



Worst case I'll get one more chip. But I don't think I will risk this chip on the board now.

C

By the way, I wrote up a pretty good summary of how to do it on my other thread. Got the temps, times, and tools needed to make this happen.

Note that I'm going for 8 chips now with some more serious heat sinking. Apparently it's possible. And what's the worst that can happen?

C

Done. I'd be interested in trying out a better heat sink solution on either/both sides.

In the meantime placed the order for two more chips, I'll put chip 6 on in the next day or two.

C

I have no idea how I could accept payment in bitcoin if the price goes up and down by $200 in a few minutes :-)

Ah, ok, haven't seen one of those personally. Is there a part number or something I can order that sink from somewhere? Or something similar aftermarket?

I will say that checking it this morning that the copper heat sink I have on the bottom is contributing a *lot* to keeping the board cool-ish. Heat can go either way, and a good sink on the bottom seems to work well at 5.

As for the warrantly, yeah this is way beyond "I just opened it to peek". If it blows up it blows up. I (and anyone else trying this) can deal with that. :-)