Hacking KNC Titan / Jupiter / Neptune miners back to life. Why not?

[frenchy]

Very good work is what i was expecting to write lol Sorry

[hawkfish007]

@hawkfish: this is exactly the case i was living last summer:I'm sorry for you but you seems to have a dead controller....
The only solution for me was to RMA the controller....
But you're very lucky because now there is a king of the knc controller who can repair it (he's doing very goog work) with fair prices.

Crap, I have 2 controllers like that at the moment and 7 cubes sitting idle  Sent a pm to lightfoot.

[_Miracle]

Like your other threads, I'll have fun watching this.

[jstew]

the neptune runs perfect with the other 4 cubes but if i try to plug in that one specific cube it stops hashing on all cubes and the controller goes wacky

Yup. Question 0 is "what does it do with nothing else plugged into it" (even the display). What should happen is the unit should power up, pick up an IP address, then the light on the side of the board should light up super bright for a second, then turn on a little green LED there. If the bright LED flashes very briefly 3-4 times in a row, you have a short and the DC-DC can't turn on the FPGA or the SPI busses.

If one unit causes the whole string to crash, then it has the odd short between pins 4,6, and ground. I'm still trying to figure that one out.

controller starts up just fine with nothing plugged in


edit

i have 2 controllers and the same thing happens when i plug that cube into either controller

[adaseb]

I got a KNC Controller that fails to start on cold. When it warms up, after a few power cycles it works normally.

Basically when it first starts, there is a whistle, and then the lights flash and then it finally starts.

If there is no whistle, then I need to keep restarting it.


Anyone had this issue? Could be the BB instead of the controller.

[lightfoot]

Ada: That's something which is on the Beaglebone and *not* the controller board. I think the problem is the BB SD card pins are touching the frame from being pressed on. Unit can't turn on its' DC-DC, doesn't boot, power cycling sometimes gets it up.

What does work for me is to put a sd micro card in the slot and turn on the unit, then pop it out once the power is on. That seems to allow it to boot every time, then I just leave it running. Weird, but a $40 BB problem and not a board problem.

[lightfoot]

So latest from last night's work:

1) Still can't find the destination for those 10 pins. Thought it was the LM75's but nope. Drat, still working. Will get batteries for fuzzer.

2) One of the two bad power supplies on this Titan is fixed, waiting for the other to come in from China.

0   0.7848   43.2500   33.943   68.000
1   0.7811   43.8750   34.271   79.000
2   0.7881   44.1250   34.775   79.200
3   0.7894   43.8125   34.586   75.800
4   0.7885   41.4375   32.673   56.500
5   0.7914   41.0000   32.447   50.400
7   0.7909   41.8125   33.070   52.200

Note that PS 4 is back and powered (300mhz), Power supply 6 is still offline but supply 7 is hashing the die at 150mhz (thus the 41a, if I go any higher it will blow). Interesting thing is stable as a rock, and hashing at 60-70mh. Not bad for a wreck.

Now if I can just find out what that SPI short is... Getting there bit by bit...

[qberty]

lightfoot,

What did I mess up the RPi or the controller, please see the picture below? Only 1 LED is on instead of three and red LED on the RPi is off. Screen lights up but without any text or bright light at start up. I was using 2 PSUs but 1 to power both controllers, probably turned on one before the other or something to create a short. I can access the web GUI, but it won't show any cubes or hash.

Hm. When you power up the controller without any cubes attached does the super-bright light on the side come on bright, then go out or just do 3-4 little flashes? That's usually the sign the TPS chip or the FPGA has shorted, both are fix-able.

It seems to be when the webgui keeps restarting and power cycling, it's the RPi. Just happened to me and replacing the RPi with a fresh one fixed the rebooting issue (but the original bridge blew when I did that). Seemed to have fixed it though.

[hawkfish007]

lightfoot,

What did I mess up the RPi or the controller, please see the picture below? Only 1 LED is on instead of three and red LED on the RPi is off. Screen lights up but without any text or bright light at start up. I was using 2 PSUs but 1 to power both controllers, probably turned on one before the other or something to create a short. I can access the web GUI, but it won't show any cubes or hash.

Hm. When you power up the controller without any cubes attached does the super-bright light on the side come on bright, then go out or just do 3-4 little flashes? That's usually the sign the TPS chip or the FPGA has shorted, both are fix-able.

It seems to be when the webgui keeps restarting and power cycling, it's the RPi. Just happened to me and replacing the RPi with a fresh one fixed the rebooting issue (but the original bridge blew when I did that). Seemed to have fixed it though.

Should I order a couple of RPi just in case it's the RPi causing all the trouble?

[qberty]

lightfoot,

What did I mess up the RPi or the controller, please see the picture below? Only 1 LED is on instead of three and red LED on the RPi is off. Screen lights up but without any text or bright light at start up. I was using 2 PSUs but 1 to power both controllers, probably turned on one before the other or something to create a short. I can access the web GUI, but it won't show any cubes or hash.

Hm. When you power up the controller without any cubes attached does the super-bright light on the side come on bright, then go out or just do 3-4 little flashes? That's usually the sign the TPS chip or the FPGA has shorted, both are fix-able.

It seems to be when the webgui keeps restarting and power cycling, it's the RPi. Just happened to me and replacing the RPi with a fresh one fixed the rebooting issue (but the original bridge blew when I did that). Seemed to have fixed it though.

Should I order a couple of RPi just in case it's the RPi causing all the trouble?

I know for sure the restarting problem IS your RPi. I literally had to deal with that twice. So replacing the RPi is definitely solving part of your problem right there. Be sure to get the exact same model (the different available Pi models have different IO pins.

[lightfoot]

Kinda. I tried putting the Rpi code on an old first generation Rpi to trace it out. Damn thing kept rebooting every two minutes, finally found it has a daemon running that looks for the FPGA to power up. If it doesn't see that in 2 minutes it assumes a problem and reboots. Clearing out /etc/init.d directory stopped it from doing this, enough that I was able to move the files to a Beaglebone for further testing and review.

But it could be a blown interconnect board (still don't know why that would fail), would need one in here to review further.

[qberty]

Hawk is already getting two bridge boards from me, so the last things that could fail software wise would be rpi foolery. the actual knc board hasn't blown or failed for any of my titan sets (i have over 6) yet, it's always been the Pi that needs replacing or the bridge. Wish the titan boards used the beaglebone version.

[lightfoot]

Hawk is already getting two bridge boards from me, so the last things that could fail software wise would be rpi foolery. the actual knc board hasn't blown or failed for any of my titan sets (i have over 6) yet, it's always been the Pi that needs replacing or the bridge. Wish the titan boards used the beaglebone version.

I think they can, once I get this next shipment I'll spend some quality time seeing if I can hack something together.

[lightfoot]

Well, here's my first less than perfect success: Blown up Titan, fireballed power supplies, replaced two. One of the new supplies refuses to come up regardless of reflow and doesn't appear to have any shorts. My guess is that the Titan board took some damage and knocked out the SPI connection to that supply.

Result is a Titan that hashes at about 70mh, keeping the last die at a speed of 150mhz, the others at 300. For the moment I'll classify this as a lot "better than dead" and make an offer on the repairs. Still, not bad overall for a dead unit.

On to the next problem.

C

[lightfoot]

Snowing like hell this evening, so I unpacked my next job: 5 Neptunes, 3 controllers, three beaglebones.

Of the five Neptunes one has a shorted supply, and the rest don't have shorts on the 10 pin SPI connector. Plugged the first one in, power supplies appear but refuse to power anything. Possibly a reflow, will work on tomorrow.

Second one is *very* interesting. Comes up and runs, but is a super hot runner on the power supplies. No real reason why, but even at 350mhz it's running high 90's. So after letting it run a bit I picked up the cube from the floor.

The bottom of the cube was *HOT*. Odd, since the reference neptune (450mhz, all supplies around 80c) is not really warm on the bottom. Weird. I wonder if the heat sink has lost contact, and as a result the chip is getting super-hot and polluting the surrounding supplies with heat. One way to find out....

[adaseb]

Ada: That's something which is on the Beaglebone and *not* the controller board. I think the problem is the BB SD card pins are touching the frame from being pressed on. Unit can't turn on its' DC-DC, doesn't boot, power cycling sometimes gets it up.

What does work for me is to put a sd micro card in the slot and turn on the unit, then pop it out once the power is on. That seems to allow it to boot every time, then I just leave it running. Weird, but a $40 BB problem and not a board problem.

It seems to work with the SD card in but I need to take it out or it will not boot, anyway to leave the SD card inside and have it start?

[lightfoot]

It seems to work with the SD card in but I need to take it out or it will not boot, anyway to leave the SD card inside and have it start?

Weird, huh. How about power it up with the SD card inserted, then immediately pop it out. Or put it in far enough that it almost engages, the whine should stop. Once it's going, it seems to go.

Once again, not quite sure why.

[adaseb]

It seems to work with the SD card in but I need to take it out or it will not boot, anyway to leave the SD card inside and have it start?

Weird, huh. How about power it up with the SD card inserted, then immediately pop it out. Or put it in far enough that it almost engages, the whine should stop. Once it's going, it seems to go.

Once again, not quite sure why.

Yes that's what i've been doing. But if I leave it in, it tries to read the empty SD Card and fails to boot. If I unclick it and leave it in there, it doesn't work and goes back to that whine error.

[lightfoot]

That shows you this is some sort of a short in the SD card connector. Are any of the pins touching the frame? How about sliding a small bit of cardboard cut like a SD card in there?

Will check out mine further once I get these queued jobs done.

[lightfoot]

Ok, so an update on that super-hot Neptune (Titans will do this too). Thing was running with PS temps at >100c, chip at 60c+ at a mere 350mhz, bottom was extremely hot to the touch (these things must be laughs on carpet).

Took unit apart, found this:


Yep. Filth and crummy hardened heat sink compound. Cleaned up the chip, the box, everything, and now:

44C

0   0.7838   34.5625   27.090   75.500
1   0.7834   35.0000   27.419   79.625
2   0.7859   35.0000   27.507   67.625
3   0.7823   34.3750   26.892   67.750
4   0.7838   35.3125   27.678   60.000
5   0.7870   35.7500   28.135   65.625
6   0.7848   34.2500   26.879   63.688
7   0.7828   35.0000   27.398   65.125

500gh, 218 watts, bottom normal temp to the touch.

Moral: If your unit is running hot, try opening it up and cleaning the thing. Replace the heat sink compound, clean the old crap off the chip (if there is hardened crud on there you will break the chip when you torque down the sink) and torque the screws down "finger snug". Likewise snug them in sequence, to even the pressure.

Never dull, on to the next unit.