Experimenting with Jalapeno firmware...

[lightfoot]

I dont think you could just add more chips to a jally. Can the board support that extra load?

More like are all the components there to support more chips correctly or have they been left off?

One way to find out....

C

[lightfoot]

*crackle*

Houston, we have a Jally with 15.67gh-16.04gh, error ratio of .76 or so, temp with unit open fan pointing down of between 54 and 61c. Power use is a pig; went from 20 watts with nothing to 40 idle to 95 hashing. So the extra two chips bump the jally from 40-75 watts draw.

Running un-modified 1.2.9 release, speed 7, little single.

Chips themselves: New chips are running at 4.2-4.3gh, older chips are at 3.5-3.76gh. New chips good, old chips suck.

No errors at all on the new chips, all errors on old chips. Moral: Old chips suck cock.

More later.

C

[minternj]

Good work. Let us know.

[lightfoot]

Houston we have explosion....

Crap. So after fiddling with it a bit I decided to put it in the case. Unplugged it, put it in the case, powered up.

Fast flash.

Crap. So I pulled it out and checked it. Remembered that when I put the fan plug in I heard a small crack, thought the fan plug had become delaminated from the board. No, looks good, but tried reflowing solder anyway.

Now it seems to work at random. For awhile I thought the issue was the fan cable, wiggling it then powering up would sometimes get all four chips up. Now I seem to have a dud.

Power supply is reading 13.6 volts, it seems good. However this thing is not happy. Question: What kind of tools can I use to query this? If it's on a com port can I use putty, and if so what kinds of things can I do to get to the diagnostic messages?

Crud.

C

[lightfoot]

Well, now it's consistiently dead. I've tried flashing ck's firmware (came up with 4 processors but flashed fast when rebooted), 1.2.9 with fan stuff commented out, 1.2.9 stock with little_single only, 1.2.9 with the changes recommended by Dana. Once it just came up by itself after a reprogram but silly me I powered it down to put it together and dud again.

At this point it can be one of three things:

1) I've broken the fan sense wire somehow and the Jally flash fails on startup.
2) One of the chips is shorting or something and flash failing
3) I have broken something else on the board.

What is bugging me is why it went bad; I must have power cycled it a dozen times with no problems while testing, then foom dead. Now the only things I can think of were:

1) I did snug down the bolts a bit more before closing it up the first time when it went bad; that might have caused a solder ball to go bad on one of the new chips.
2) I did have that fan thing, but I think that was before it started going totally bad.

It did run without errors for about 20-30 minutes there, it was running well.

So it's either try to re-heat the chips, pull them off one at a time, or pull diags somehow. I don't like the latter option; that really sucks as I will have to reball the things.

Any pointers on how I can use the USB port to pull diags? I tried plugging putty in at com6, but it didn't seem to listen.

Ah well, candle that burns brightly and all that.

On the info side, I got the chips to solder with a pre-heat of 120c on the board, followed by 400c for about 30 seconds per chip with the heat tool. Running at 280c didn't even melt the balls, even with 2 minutes of solid heat. Both look square on the board, I can see the edges with a 4x jewelers loupe, and one is a bit closer to the board than the other.

Any thoughts appreciated. I'll work on it more tomorrow after work.

C

Bedtime.

[lightfoot]

I really. Really. Wish I was a normal person and could just go to sleep with a problem.

Naah.

So I finally got up at 1:30 and took apart the Jally again. Looked at the chips and one was higher on balls. Not good. Tried re-melting it, unit fired up when hot, not when cold.

Bullseye.

Pulled the chip. Turns out it moved a bit and had some solder traces shorted. Would work sometimes, failed when I cranked down the heat sink. Now I'll have to learn to re-ball it.

Put things together, fired up Jally. Fired up with 3 chips, even without fan. Note to all: Fan speed is not monitored, total red herring. Cycled it 5 times, then put some heat sink grease on what's left of the pads, put together, fired up.

Running with some errors because I disabled error checking pretty much everywhere. Crap. But it is hashing at 12gh, which is damn slow for a jally. Temps are 75c, super high. So apart it is, lid off and flipped it and it's down to 67c.

Need a lower temp set here, those dead engines are pulling heat.

Tried reflashing with error checking on, got flashing light after test. Removed, back to normal. 12gh hashing at 65c with the case open; I am declaring victory for tonight and going to bed. Because I have to like... work tomorrow.

I'll put on two more chips when they come in this weekend.

C

[FalconFour]

Damn, you're as persistent as I am. Can't sleep with a problem. >.<

Yeah, with BGA as fucking insanely sensitive as it is, it's a goddamn miracle you got the new chips working. I was just looking at my Jally a few minutes ago as I was putting it back together (getting it back in the case, may be a bad idea) I noticed my (all-aluminum) heatsink had room to completely cover exactly 2 more chips. The heat sink design seems very "not fully thought-out" though, allowing WAY too much room for error in over-cranking the screws and cracking the fuck out of the board. I see a *lot* of people messing that up (and not assuming you have, either). Mechanically, over-tightening the screws wouldn't feel like over-tightening at all, but just barely going beyond the point where the screw head meets the board, and the board will begin to bow, causing the chips to bend away from the heatsink instead of pulling them tighter. Be really careful about that.

The fan control and monitoring in the firmware is a complete, unprecedented joke. There is no control at all - from what I've seen, there's a flag set in there with a code comment that almost literally says "this thing works better without control, so fuck it". The fan-control commands literally have no hook in the code at all, and no way to trigger it from the interface. Basically an initial control is ran to start the fan, doesn't even check to see if it runs, then it forgets it even has one. Really wish I could control mine. I have to feed it external 5V power (when the case is removed, of course) to keep my sanity.

With temps in the 60s and 70s though, you've got issues for sure. Mine maxes out around 55-60 right now with the case on and running the 8.5gh firmware, with freshly applied Ceramique II and screws lightly tightened just to remove the rotating wobble from the board/heatsink, and a thin (credit-card-thickness) slice of cardboard between the aluminum plate and PCB for insulation and padding (WTF was BFL thinking, metal-on-PCB?!).

Error checking shouldn't affect startup though. Maybe the code you're turning on is set to check all (non-existent) chips and craps out trying to test the ones that aren't there. /shrug... random shotgun guess without sifting through the code.

[FalconFour]

God damn it... well, I had 8.5-8.6gh/s, but now after fucking with the grease again thinking I had air pockets in the last application, this "last time I can screw with it before buying more paste" attempt resulted in a nearly-800mh/s reduction in performance - now running a solid 7.9gh/s average. Thermal problems are insanely understated because hardware usually compensates for most of the stupid crap people do, and it goes unnoticed. With the Jally, I think many hardware tweakers are gonna get a crash course in thermodynamics.

And this thing has painfully tight tolerances inside. Even adding that slice of cardboard spaced out the bottom plate such that the board now would flex a little if I tightened down the four corner mounting screws. So now it sits a little loose (and back to open-air again). Sigh... but this time, sleep takes priority.

[erk]

God damn it... well, I had 8.5-8.6gh/s, but now after fucking with the grease again thinking I had air pockets in the last application, this "last time I can screw with it before buying more paste" attempt resulted in a nearly-800mh/s reduction in performance - now running a solid 7.9gh/s average. Thermal problems are insanely understated because hardware usually compensates for most of the stupid crap people do, and it goes unnoticed. With the Jally, I think many hardware tweakers are gonna get a crash course in thermodynamics.

And this thing has painfully tight tolerances inside. Even adding that slice of cardboard spaced out the bottom plate such that the board now would flex a little if I tightened down the four corner mounting screws. So now it sits a little loose (and back to open-air again). Sigh... but this time, sleep takes priority.

I don't think you should use paste with multiple chips, they will not all be at the same height, best to use those thermal pads like the ones that come with the Jally in the first place.

[lightfoot]

God damn it... well, I had 8.5-8.6gh/s, but now after fucking with the grease again thinking I had air pockets in the last application, this "last time I can screw with it before buying more paste" attempt resulted in a nearly-800mh/s reduction in performance - now running a solid 7.9gh/s average. Thermal problems are insanely understated because hardware usually compensates for most of the stupid crap people do, and it goes unnoticed. With the Jally, I think many hardware tweakers are gonna get a crash course in thermodynamics.

And this thing has painfully tight tolerances inside. Even adding that slice of cardboard spaced out the bottom plate such that the board now would flex a little if I tightened down the four corner mounting screws. So now it sits a little loose (and back to open-air again). Sigh... but this time, sleep takes priority.

By the way, that bottom plate helps transfer board heat to the bottom of the case. Don't put cardboard between that and board.

Ran all night, some errors on one of the new chips, bit odd. Eligus shows me at 11.93gh/hr over last 3 hours. That about matches the performance on the unit from BFG.

Still an annoying amount of errors on that one core, there has to be some mistake I am making in the firmware that is causing the unit to not properly clock the chips; CK's code runs the most nicely; maybe I need to try his 1.2.6 code run and see if it works.

Meantime, time to go to workies!

[lightfoot]

When I was running with a friend's Jally I fixed till a few days ago, my 8gh normal speed, and my current speed at 12. Expect faster speeds "soon" once I get a new pad of that heat sink stuff and my other two chips.

C

[lightfoot]

Damn, you're as persistent as I am. Can't sleep with a problem. >.<

Yeah, with BGA as fucking insanely sensitive as it is, it's a goddamn miracle you got the new chips working. I was just looking at my Jally a few minutes ago as I was putting it back together (getting it back in the case, may be a bad idea) I noticed my (all-aluminum) heatsink had room to completely cover exactly 2 more chips. The heat sink design seems very "not fully thought-out" though, allowing WAY too much room for error in over-cranking the screws and cracking the fuck out of the board. I see a *lot* of people messing that up (and not assuming you have, either). Mechanically, over-tightening the screws wouldn't feel like over-tightening at all, but just barely going beyond the point where the screw head meets the board, and the board will begin to bow, causing the chips to bend away from the heatsink instead of pulling them tighter. Be really careful about that.

Yep. My method here is to start the screws, then flip the jally over and press on the bottom plate to take up slack evenly on the chips. Then hand tighten the screws down with a torx bit, stopping when they are *just* snug. That way I am not cranking into the board and I know the chips are getting finger pressure.

With temps in the 60s and 70s though, you've got issues for sure. Mine maxes out around 55-60 right now with the case on and running the 8.5gh firmware, with freshly applied Ceramique II and screws lightly tightened just to remove the rotating wobble from the board/heatsink, and a thin (credit-card-thickness) slice of cardboard between the aluminum plate and PCB for insulation and padding (WTF was BFL thinking, metal-on-PCB?!).

Remember I am now running three chips (was 4, soon to be six) so my temps are going to be high. Yes this will lower the life expectency of the unit, but after Feb it's not going to be able to mine enough coin to matter, so it doesn't matter. Might as well go for broke :-)

Error checking shouldn't affect startup though. Maybe the code you're turning on is set to check all (non-existent) chips and craps out trying to test the ones that aren't there. /shrug... random shotgun guess without sifting through the code.

It's possible that is what RUN HEAVY DIAGNOSTICS is. It's also possible that last night I was flashing it without the fan which caused the board to heat up enough to trip the "I'm too hot" sensor. Because it started working again after I went to hit the can with it unplugged. I'll try more tonight, but I really should just hard-wire the damn jtag port so I'm not always pulling the sink.

Running it with bad cores disabled cuts power usage (I think bad cores run 100% *on* and suck full power for zero benefit and take work thus slowing the overall chip down) so I'll re-flash again tonight. I think 1.2.6 might work with the newer chips, and ck has a compiled elf for that so I might give it a try. Or try Tarkin's load. Or just go back to a stock 1.2.9 with speed 7 and little_single. That worked well.

Anyway back to work.

[FalconFour]

Well, I definitely understand how chips can be different heights, but in theory, the mounting process should be uniform enough to make the variances stay within a level of tolerance that'd be filled in by design of thermal paste. Pads are extremely poor conductors compared to grease, and my Jalapeno had visibly poor cooling using its stock pads - there were discolorations on the chips from the hot spots! That's why I went with the grease.

The use of an (unlaminated) metal plate on the bottom seems like amateur design suicide... yes, it'll conduct a tiny amount of heat, but at the extreme risk of scraping off insulation of the PCB and shorting out contacts over time. There's no thermal conductive pad there (which itself would be a good idea as well), so it seems entirely unlikely the plate serves any real purpose for cooling.

I like your method of tightening the screws, lightfoot... that sounds like a good plan. Huge, huge kudos for getting 4 chips temporarily running as well. Sounds like something I might try doing myself as well, if I can get the equipment together; I've got some chip credits to burn.

edit:
Disassembled it again and recycled the grease that slopped over the sides - something I'm sure a lot of people would argue with. Then, lightly reassembled using the light press and finger-tightening with the torx bit, and also replaced the cardboard with a thinner thermal conductive pad covering both chips' positions on the board. Sure enough, the plate does get plenty warm too. Back to normal temps again and rolling at 8.6GH/s again, after hashing all day at 7.8GH/s from last night's attempt. Cool deal! I very much believe thermal transfer has a lot more to do with speed than just crapping out at 70C or 80C.

[erk]

Well, I definitely understand how chips can be different heights, but in theory, the mounting process should be uniform enough to make the variances stay within a level of tolerance that'd be filled in by design of thermal paste. Pads are extremely poor conductors compared to grease, and my Jalapeno had visibly poor cooling using its stock pads - there were discolorations on the chips from the hot spots! That's why I went with the grease.

The use of an (unlaminated) metal plate on the bottom seems like amateur design suicide... yes, it'll conduct a tiny amount of heat, but at the extreme risk of scraping off insulation of the PCB and shorting out contacts over time. There's no thermal conductive pad there (which itself would be a good idea as well), so it seems entirely unlikely the plate serves any real purpose for cooling.

I like your method of tightening the screws, lightfoot... that sounds like a good plan. Huge, huge kudos for getting 4 chips temporarily running as well. Sounds like something I might try doing myself as well, if I can get the equipment together; I've got some chip credits to burn.

If you have ever read the instructions on good thermal paste like Arctic Silver it typically says apply with using something like a razor blade as a squeegee so the layer you end up with is like .003".  Totally inappropriate for filling gaps. Thermal pads are like .01" or thicker.

 

[FalconFour]

The point of grease is to replace air pockets with conductive material, so there are always air gaps to fill by the very fact that two pieces of metal are put together. The difference in height between the two chips is within the tolerance of melting solder balls, as the two chips are identical (not the case with most other cooling scenarios). After years of applying grease and seeing its effects, I can definitely say the application directions for AS are a bit flawed - I never recommend manually flattening the layer, always leaving a blob to be evenly distributed by the heat sink as it comes down. Flattening it allows air pockets to form when the heat sink comes down (and I've seen that cause huge problems).

The problem is that pads add that fixed gap to the thermal equation, and pads aren't that great at conducting heat. I've played with replacing grease with pads in other applications due to the difference in height, and the results were amazing - pads absolutely suck at conducting heat, and I use them only as a last measure. Usually I find myself replacing pads with appropriate-thickness copper shims which, while it has its own set of naysayers and objections, almost always makes the cooling system work much more reliably.

I can say my Jally seems pretty happy with its new arrangement

[lightfoot]

Well, I definitely understand how chips can be different heights, but in theory, the mounting process should be uniform enough to make the variances stay within a level of tolerance that'd be filled in by design of thermal paste. Pads are extremely poor conductors compared to grease, and my Jalapeno had visibly poor cooling using its stock pads - there were discolorations on the chips from the hot spots! That's why I went with the grease.

There are two versions of chips. Version 1 is taller, and has engine 0 disabled by default. Version 2 (the ones I have seen on ebay) is a bit thinner, and has engine 0 enabled. This is why Jallys with version 2 chips need 1.2.7 or better; they probably put code in to turn engine 0 off explicitly if you're running an older style board (the little_single model) and later boards can handle engine 0.

The use of an (unlaminated) metal plate on the bottom seems like amateur design suicide... yes, it'll conduct a tiny amount of heat, but at the extreme risk of scraping off insulation of the PCB and shorting out contacts over time. There's no thermal conductive pad there (which itself would be a good idea as well), so it seems entirely unlikely the plate serves any real purpose for cooling.

Ok. However in their defense it does wick away a *lot* of heat through the bottom of the unit, and the PCB conformal coating is both thick and has minimal vias and other stuff around there. I see no problem with it, the real danger is tightening those screws too tight.

I like your method of tightening the screws, lightfoot... that sounds like a good plan. Huge, huge kudos for getting 4 chips temporarily running as well. Sounds like something I might try doing myself as well, if I can get the equipment together; I've got some chip credits to burn.

The morons at BFL never really sold the chips like they should. Those credits were worthless which is even more annoying. But I got my JP in end-August, at least I was able to make the money back. A friend got a single 25 last week; I might help him boost his code a good bit just to do it.

With grease and crap pads my JP is still hashing at 11.5 gh/s, I ordered this from frozencpu.com as a pad for my chips; I lost the first one I bought so I got another one coming today.

thr-181--Fujipoly Ultra Extreme System Builder Thermal Pad - 60 x 50 x 0.5 - Thermal Conductivity 17.0 W/mK   $15.99

Best. BEST money you can spend IMO. Without it I have to run my JP open with the fan upside down to maintain 65C. With it I should be able to run 4-5 chips instead of 3 and maybe even with the case on.

edit:
Disassembled it again and recycled the grease that slopped over the sides - something I'm sure a lot of people would argue with. Then, lightly reassembled using the light press and finger-tightening with the torx bit, and also replaced the cardboard with a thinner thermal conductive pad covering both chips' positions on the board. Sure enough, the plate does get plenty warm too. Back to normal temps again and rolling at 8.6GH/s again, after hashing all day at 7.8GH/s from last night's attempt. Cool deal! I very much believe thermal transfer has a lot more to do with speed than just crapping out at 70C or 80C.

It does. Also I have found that turning off errant cores is worth it both in terms of power use (dead cores run at full power and do zilch) and heat (see that "full power" bit). Running heavy diagnostics is how you turn the crap cores off, which allows the remaining ones to run faster.

Today's goal after going to "church" for family pictures is going to be to install one more chip, test for 16gh, then install my last new chip and go for 20gh. I have the chip still that needs to be re-balled, will think about that later. But if 20gh causes the power supply to crap the bed I can disable one of the chips in software (I think it will be the crappiest one of my original chips) and run at a better 16gh till the new power supply comes in.

I also figured out the software. Running 1.2.9 totally stock, LITTLE_SINGLE identified, and speed 7. This produces the best run config for this oddball.

This oddly enough is "not dull"

C

[EzCheese]

Sounds promising!

[lightfoot]

I'm getting better at this...

Now running four chips,
PROCESSOR 0: 15 engines @ 274 MHz -- MAP: FFDF
PROCESSOR 3: 13 engines @ 271 MHz -- MAP: EFFC
PROCESSOR 6: 16 engines @ 265 MHz -- MAP: FFFF
PROCESSOR 7: 15 engines @ 248 MHz -- MAP: FFFE
THEORETICAL MAX: 15593 MH/s

Odd, one of them is running 16 engines? Whatever, it looks like things are running. Now to let it hash for awhile and see how the temps are on the power supply and body.

And man, she is now running *hot* at the plate. Power usage is about 100 watts total, temps are holding now at 75 degrees C. Power usage breakdown is as follows:

Laptop (x61) 20 watts normal.
Two crock-erupters 5 watts
Everything else: Jalapeno.

Oddly enough this is about 10 watts less than when I was running two 8gh jallies, so the jally hotel load is probably about 10 watts.

New chips don't seem to be pulling any errors, overall error rate <1%, probably means the clocks are good.

Note, this is programmed at speed 7, I wonder if it would hash faster at 8. To be honest it might be smarter to back down speed to something like 5 and put on the last engine. Hm....

See. I still have another engine. And another one after that if I want to run in suicide mode. Have not checked the temps on the power FETs yet, they may be the weak point here.

Oh by the way, I figured out why version 1 chips are taller: It's because they are not really BGA chips. See, BFL did these as edge connector chips but my guess is they failed due to heat. So they built BGA adapter boards, and used up the version 1 chips. V2 were already BGA, and thus are a bit smaller.

C
C

[lightfoot]

Now I am thinking: If mounds of heat are coming off the back of this thing I should just "Don't beat them, join them", get another heat sink, and mount it on the bottom of the board with a second fan on the top of that. Put in super-long studs in place of the screws and run this thing on it's side with two fans.

Even better, does anyone know the plate size we have to cover here? Maybe what I *REALLY* should do is get a heat pipe equipped heat sink, put that on the top, and run this thing with the aluminum sink on the bottom....

Next up, small heat sinks for the FETs? How much can this little thing take? Time to go to Home Depot and drop a few bucks on an IR thermometer, the power FETs seem to be getting warmer. Not sure yet if 16gh is a tipping point...

C

[lightfoot]

Note: At 16gh, forget running with the case closed. Temps passed 90 and the unit throttled responses. So much for that.

Back to open, I'm going to leave the fan top open for now and get some screws from the shed. 75C. This is going to be a problem.

I think the heat sink area would be 40*48*60 mm high, does anyone know of a good super cooler for this? I'm going to need more cooling power, I can see that.

C