Hacking KNC Titan / Jupiter / Neptune miners back to life. Why not?

[lightfoot]

Maybe. I have one incinerated Y adapter, showing that even that can't make up for a bad power supply or overloading or a connector that has been plugged and unplugged with the power on (thus causing sparks which cause little bits of resistance which warm up which.....)

Still, it is also true that there seem to be these islands of stability on Titans, and if that island is at 300-325mhz then you're kind of stuck with it.

Back to work here. Have two more boards to keep me busy over the next few days.

[GenTarkin]

Maybe. I have one incinerated Y adapter, showing that even that can't make up for a bad power supply or overloading or a connector that has been plugged and unplugged with the power on (thus causing sparks which cause little bits of resistance which warm up which.....)

Still, it is also true that there seem to be these islands of stability on Titans, and if that island is at 300-325mhz then you're kind of stuck with it.

Back to work here. Have two more boards to keep me busy over the next few days.

If ya ask me, based on my single Titan up in hosted up in Alaska .... I believe heat is the number 1 killer of these things.... that combined w/ the fact how close these things are pushed to the edge by users.

But, my Titan has ran for less than 75-80C on all DCDC's since the day it arrived in 2014 ... clocks 325mhz, volts set to -0.0366v or lower.
It did survive one PSU dieing - modular connector sockets on the PSU actually burned up. Titan connectors were fine.
Other than that, it has hashed along happily the whole time, just like it did since day 1.

I could be totally wrong tho =P

But yeah, its a hunch based off of seeing the settings and temps some users are pushing these things through. I mean YOWCH!...

[lightfoot]

Perhaps, in which case the only Titans that can be fixed are the ones that smoke power connectors or blow out power supplies, both resulting in shorting on power-up. This is possible, although sad.

However other things just seem odd. For example: I have another dead titan board in for a review. It hashes on one die. Checking in bfgminer I see that only two dies ever respond, the other two do *nothing* no matter what. It has the jumper missing.

I wonder what happens if I put the jumper in. Will all four dies respond like insane children? Why would KNC do this?

Then I look at the supplies. Two are missing, so I check resistances on the chip side. Normally I see around 7.5 ohms if chip is good, 0 if shorted (to ground). On this one I see four power supplies reading 7.5,7.5,7.5,7.5. The other four read 7.5,600 and 7.5,140 ohms.

THAT IS IMPOSSIBLE! The supply outputs are ganged together in parallel. There is no way on EARTH that the outputs can read higher resistance. What the heck is going on?

I'll pull a supply and see. But I'll also pop in a temporary jumper and see what happens.

[Searing]

Maybe. I have one incinerated Y adapter, showing that even that can't make up for a bad power supply or overloading or a connector that has been plugged and unplugged with the power on (thus causing sparks which cause little bits of resistance which warm up which.....)

Still, it is also true that there seem to be these islands of stability on Titans, and if that island is at 300-325mhz then you're kind of stuck with it.

Back to work here. Have two more boards to keep me busy over the next few days.

If ya ask me, based on my single Titan up in hosted up in Alaska .... I believe heat is the number 1 killer of these things.... that combined w/ the fact how close these things are pushed to the edge by users.

But, my Titan has ran for less than 75-80C on all DCDC's since the day it arrived in 2014 ... clocks 325mhz, volts set to -0.0366v or lower.
It did survive one PSU dieing - modular connector sockets on the PSU actually burned up. Titan connectors were fine.
Other than that, it has hashed along happily the whole time, just like it did since day 1.

I could be totally wrong tho =P

But yeah, its a hunch based off of seeing the settings and temps some users are pushing these things through. I mean YOWCH!...

yeah i essentially agree...BUT again....was told back in the day my temps/speed with overclock etc was just dandy (silly me) and that it has run more or less as a champ at 15 months i lose a bit of hash on a few dies (not counting the 2 dead dies on 2 dead cubes that never really worked) well hell.....so yeah I likely will pay the piper ..due to the only lately (last couple months) seeings how close to the edge I am

but to be fair...to the folk over the last year running these things ..esp us 1st batch titans with the bad dies which I and others were told 'specifically' to run full out at 325 0.0366 etc...well ..clueless as I was I ran mine full out all summer of 2015 with die rates in the low 90's some at 95c 96c

the guy I got the used Titan from Australia (way in the outback I kid you not) I think was away on work and eventually could NOT run them remote fire hazard...also had burned out 2 sets of psu's I think..thus how I got them in the panic sale of march 2015 on the expectation of all that NEW scrypt equip that never came to pass

I should post a pic of those cubes...run hard ..really hard...the cubes are faded with heat spots ...looks like the hood of a hot rod in Arizonna after racing all summer...kinda cool....

and these 4 cubes are MY MOST STABLE OF THEM ALL ...FULL OUT at 325 and 0.0366 since March w/o missing a beat

go figure

being electronic clueless is there some kind 'burn in' in that they ran too hot ...adapted and then now 'like' it that way?

anyway ..take advice here of everyone on how to tweak this stuff...the recent cube I got is set to 0.06xx volts at 300mh
and stable...so I left it....no idea how he got voltage that low at 300mh I've never had much luck

anyway my setup is probably an 'freak' setup and due to decent high end psu's as stated in other posts etc

but again .....I was 'assured' when I did over clock these to 325mh by I think it was Kurt at KNC? All was fine. They
could run easy and safe between 90c and 95c no problem and up to 100c...I kid thee not..what I was and sure others
were told....there is a reason that knc forums was 'yanked' imho ......sheesh...I saved a lot of stuff I should see if I
can dig out that email exchange (hope it was email ...coulda been knc forum but i don't thinks so) I'll look

I'm sure others on here were told the same back in the day as first batch titan owners.

I should have known better ..but in my case NO real issues of note...thus never had to delve into tweaking much (till lately a bit)

anyway got on a side note got 3 cubes 1 cube only 1 die works...1 cube the 300mh at low 0.06xx low voltage..go figure stable and another one in the mail supposedly fully working...so figure about 190mh to 200mh for 260 ltc (I hoard so on paper I sold LTC at it was almost exactly 750 usd)

my way around it 2 cubes and an extra die as over kill ...when stuff starts for 'fade' as i should 'mentally prepare for"  a way to compensate for the likely 'dribble down in hash' coming for the as of march 8th 16months on both Titans the new and used one...both are Nov 2014 units

so hell 'overcompensating' with equipment....better then fretting about it ...like most of my life issues if it bugs me enough

toss money at it..seems to work


But listen to folk here...very likely ...very soon I will be scrambling like everyone else with different voltage and hash settings
due to the 'heat debt wear and tear I have put these units thru.....but as yet ..still good (knock wood)

later

Perhaps, in which case the only Titans that can be fixed are the ones that smoke power connectors or blow out power supplies, both resulting in shorting on power-up. This is possible, although sad.

However other things just seem odd. For example: I have another dead titan board in for a review. It hashes on one die. Checking in bfgminer I see that only two dies ever respond, the other two do *nothing* no matter what. It has the jumper missing.

I wonder what happens if I put the jumper in. Will all four dies respond like insane children? Why would KNC do this?

Then I look at the supplies. Two are missing, so I check resistances on the chip side. Normally I see around 7.5 ohms if chip is good, 0 if shorted (to ground). On this one I see four power supplies reading 7.5,7.5,7.5,7.5. The other four read 7.5,600 and 7.5,140 ohms.

THAT IS IMPOSSIBLE! The supply outputs are ganged together in parallel. There is no way on EARTH that the outputs can read higher resistance. What the heck is going on?

I'll pull a supply and see. But I'll also pop in a temporary jumper and see what happens.

Just saying KNC being 'evil' and all ...could they have PULLED these resistors when they sent out those 1/2 hashing titan cubes for replacements for 1 dead cube trick..thus making sure ..as they 'claimed' at the time these were UNFIXABLE?

I don't do electronics...but could it be...something like this to make sure we could not hash with them 'after the fact'?

just a thought ..but again I can barely follow your electronics sometimes

[boomin]

My theory is the engineers thought it to be a GREAT IDEA having all of the Dies on the bus so they could constantly analyze data.  Then they figured out its like trying to hear a conversation in the middle of a goal at a great hockey game.  God forbid one of the dies die -(that almost sounded funny) then the entire thing is a wreck. 

just my theory. 

Boomin

[lightfoot]

Pounding away, busy weekend bending wire/ More later, moral please please do not blow out your connectors. Clean them, drop power usage below 200w/cube, use really good supplies, whatever. Bitch to fix.

[boomin]

I will share my trick to the molex.  I actually break the plastic all away - both cutting and prying.  Once that is all done, I preheat to 300 on bottom and use 425 hot air to each individual pin.  Remove them one by one with controlled MASS heat.  Be VERY careful with your aim with the hot air that hot....

Now if I can figure out how to resurrect the bad titans -and the bad neptunes for that matter.  

I also came across something really weird.  I have a couple of controllers that wont go in to advanced tab from web software.  However, if you flash the board to rc6 it will work.  The second you update to latest software, no more advanced.  2 separate controllers.  

I wish that the details in Screen -r were as nice in Neptune as it is in Titan.  It would really help to see the dies broken down like the Titan software does.  If anyone knows how to see individual dies and cubes in command form, that would be very helpful.

Thanks!  Happy Hackin!

Boomin


 

[lightfoot]

I will share my trick to the molex.  I actually break the plastic all away - both cutting and prying.  Once that is all done, I preheat to 300 on bottom and use 425 hot air to each individual pin.  Remove them one by one with controlled MASS heat.  Be VERY careful with your aim with the hot air that hot....

There is a way to do it with less heat; alloy the solder. Specifically put flux on the pins and melt normal solder onto it using a normal iron at normal temps. The lead solder will mix with the ROHS stuff lowering the melting temperature by a *huge* amount (basically to the level of pb/tin solder). Then the pins can come right out without any fuss or worry about overheating the board.

However if the pins look like this.



It's going to be a bit more... complicated. (Note how the pins are pointing in different directions, the actual anchors into the board have been burned clear through by the heat).

Bad.

I also came across something really weird.  I have a couple of controllers that wont go in to advanced.  However, if you flash the board to rc6 it will work.  The second you update to latest software, no more advanced.  2 separate controllers. 

Hm. What kind of lights come on and do they hash with Neptunes?

Night!

[mmfiore]

It might be a good idea for people to just step down the frequency or voltage just a dab in order to prevent the over current and extreme heating that leads to molex meltdown. That really looks like a mess. Its a good thing you guys are able to repair these for people.

[lightfoot]

And how do you fix the above so it runs at a nice 60mh for 24 hours at a nice and smooth <50c power supply temps and <40c chip temps?

Why build a bypass of course!



This will go down as one of the cooler things I have done in a day. As discussed the via lines for the +12v supply to the board were destroyed. +12 is on the inside layers so you can't just connect. However what I figured might work is to build a bypass bus that starts on one side of the board, goes *through* the molex connector, then winds its' way around the board dropping power at several points to both minimize the load being dumped into a particular plane and to reduce potential noise coming from supplies near the drop site destabilizing ones further away.

First step was to pick the wire. After considering 10 gauge (way too big) and 12 gauge (stiff) I went with 14 gauge UL rated teflon coated copper stranded wire. Copper stranded because I could interleave the strands into the molex plug pins for maximum contact points, UL rated because typical automotive wire insulation is only rated to 80c whereas UL rated wire is 100c rated. We don't want the wire to smoke since it will be near the power supply inductors, which get hot. And 14 gauge because that is double the gauge of 16g wire which is considered "really good" in a power supply. 14 gauge has twice the capacity of 16g, so if I go in two directions I can equal 16g wire with 4 times the load.

Second step was figuring out a connection point. I thought about scraping the board by the power supplies (bad, hard to solder, and prone to lifting), the supply standoffs (close to electronic components under the power supplies, limited connection) then finally realized I could just make the wire flat, solder-tin it, cut it to shape, then tin and solder to the capacitor banks next to a power supply. That would give me 8 connection points into the board, with physical reinforcement (between supply and capacitors) and by using rosin core solder I could melt it onto the capacitors without having to worry about melting the connection between capacitor and board (which is ROHS solder that needs +100c more heat to melt).

Third step was forming the wire. This took awhile, with a lot of bends and thoughts. Ultimately I went for three connection points, at supply 2, 5, and 7. This leaves a slight imbalance on supply 4, but it's the best I can do right now. Other supplies will pick up power through the existing via and +12 ground planes. The power distribution would be odd, but it should hold as long as you don't pull max power on every supply.




Here is the wire being formed. The top one was the first attempt, I broke a strand while forming it, and since every strand is critical I gave it up and tried again. The middle one has the molex connector soldered on, once again I went with strands on both sides of the pins, yet high enough to keep it away from the board. The bottom (red) one is automotive wire. Thicker insulation for physical protection, but a lower rating and not UL listed.

Soldered the wire onto the caps making sure there was no tension/compression against any of the components (difficult) and soldered the Molex to the board. Tried it out for resistance, saw the usual 1k or so on the 12v lines. Then put the heat sink on and mounted it in the box:



Bent the wire a little bit with two pliers (remember *NO STRESS ON ANYTHING*) to ensure it doesn't touch the side (even insulated you never want things to rub) and tried my smallest supply.

Board came up. 50mhz got me 20mh total. 100mh got me 40, and 200mhz got me the magic *60*mh. So far the supplies are stable at .78v or so, with temps at 50-53c.

Titans were built to run at 60mh, running them over this starts increasing power draw and moves the supplies closer to their limits. On a normal board this is annoying, on this kind of board you will probably start to see instabilities on the 12v planes as power goes places it was not designed to go. At this point though the connector is cool, the voltage drop to the supplies from the 12v line is not bad (11.8v at connector input, 11.78v at furthest end) showing a .02v drop at 180 watts measured. Or a .3w drop in the wires, not too bad.

Very difficult to fix, took more time and effort than one might thing. Need to let it run for some more time, but so far so good. Moral: Don't burn boards. :-)

[padrino]

Very impressive! I dig what you did running the wire external to the board.

[hawkfish007]

And how do you fix the above so it runs at a nice 60mh for 24 hours at a nice and smooth <50c power supply temps and <40c chip temps?

Why build a bypass of course!

https://i.imgur.com/Sc7VlA6.jpg

This will go down as one of the cooler things I have done in a day. As discussed the via lines for the +12v supply to the board were destroyed. +12 is on the inside layers so you can't just connect. However what I figured might work is to build a bypass bus that starts on one side of the board, goes *through* the molex connector, then winds its' way around the board dropping power at several points to both minimize the load being dumped into a particular plane and to reduce potential noise coming from supplies near the drop site destabilizing ones further away.

First step was to pick the wire. After considering 10 gauge (way too big) and 12 gauge (stiff) I went with 14 gauge UL rated teflon coated copper stranded wire. Copper stranded because I could interleave the strands into the molex plug pins for maximum contact points, UL rated because typical automotive wire insulation is only rated to 80c whereas UL rated wire is 100c rated. We don't want the wire to smoke since it will be near the power supply inductors, which get hot. And 14 gauge because that is double the gauge of 16g wire which is considered "really good" in a power supply. 14 gauge has twice the capacity of 16g, so if I go in two directions I can equal 16g wire with 4 times the load.

Second step was figuring out a connection point. I thought about scraping the board by the power supplies (bad, hard to solder, and prone to lifting), the supply standoffs (close to electronic components under the power supplies, limited connection) then finally realized I could just make the wire flat, solder-tin it, cut it to shape, then tin and solder to the capacitor banks next to a power supply. That would give me 8 connection points into the board, with physical reinforcement (between supply and capacitors) and by using rosin core solder I could melt it onto the capacitors without having to worry about melting the connection between capacitor and board (which is ROHS solder that needs +100c more heat to melt).

Third step was forming the wire. This took awhile, with a lot of bends and thoughts. Ultimately I went for three connection points, at supply 2, 5, and 7. This leaves a slight imbalance on supply 4, but it's the best I can do right now. Other supplies will pick up power through the existing via and +12 ground planes. The power distribution would be odd, but it should hold as long as you don't pull max power on every supply.

https://i.imgur.com/4BdvS2Q.jpg


Here is the wire being formed. The top one was the first attempt, I broke a strand while forming it, and since every strand is critical I gave it up and tried again. The middle one has the molex connector soldered on, once again I went with strands on both sides of the pins, yet high enough to keep it away from the board. The bottom (red) one is automotive wire. Thicker insulation for physical protection, but a lower rating and not UL listed.

Soldered the wire onto the caps making sure there was no tension/compression against any of the components (difficult) and soldered the Molex to the board. Tried it out for resistance, saw the usual 1k or so on the 12v lines. Then put the heat sink on and mounted it in the box:



Bent the wire a little bit with two pliers (remember *NO STRESS ON ANYTHING*) to ensure it doesn't touch the side (even insulated you never want things to rub) and tried my smallest supply.

Board came up. 50mhz got me 20mh total. 100mh got me 40, and 200mhz got me the magic *60*mh. So far the supplies are stable at .78v or so, with temps at 50-53c.

Titans were built to run at 60mh, running them over this starts increasing power draw and moves the supplies closer to their limits. On a normal board this is annoying, on this kind of board you will probably start to see instabilities on the 12v planes as power goes places it was not designed to go. At this point though the connector is cool, the voltage drop to the supplies from the 12v line is not bad (11.8v at connector input, 11.78v at furthest end) showing a .02v drop at 180 watts measured. Or a .3w drop in the wires, not too bad.

Very difficult to fix, took more time and effort than one might thing. Need to let it run for some more time, but so far so good. Moral: Don't burn boards. :-)

Wonderful lightfoot, so do you think unstable die (showing low current, amp, voltage and temp) situation is a result of bad or burned power line? Or, damaged die?

[lightfoot]

Damaged die most likely. The power supplies are actually pretty reliable and the connections on the .6v side to the chip seem to be pretty good. So that's not it. These dies are flakey, I think if they go odd or down it's due to manufacturing failures. Hm.
 
In the meantime I spent the evening letting this unit hash while I looked at my sacrifice board. It has had a short on pin 6 since the day I got it, and nothing seems to clear the problem (the guy had three boards that all did this after a major power supply failure, one board was merely burned). This board was my sacrificial lamb, I used it to trace the shorts on pin 4 (to the chip but could be bypassed with jumpers), pin 8 (same) and pin 6 (which I could not clear).

So tonight I decided to clear the short. Pre-heated the board to full, fluxed everything, Pulled every component around the chip, every chip, capacitor, jumper, plug, supply, you name it. Nothing cleared the short. Then I pulled the chip, somewhat messy on the lift. Still shorted, but some solder smears. Cleaned up the smears and when I was on the last corner the meter stopped beeping.

Ran the solder ball back, BEEP! Sure enough I had cleared the final fault. Checked the chip at those pads and sure enough, short in the chip itself.

Fuck.

So the problem is in the chip. What is happening is that when the chip fails or gets voltage spikes or surges it takes out the housekeeping circuits on a die. This is a common line shared by all four dies on the chip and is not possible to isolate via the board. Without it, the board will not hash.

You're basically fucked.

Need to think for awhile about this: Without new hashing chips this really can't be fixed. I mean it's theoretically possible that one could apply a very high current 3.3 volt spike to try and blow the line open on the affected chip without taking out the other three chips, but that's really iffy. I might try this on board 2, but it is not what we call an optimal solution.

Hm. Bedtime.

[mmfiore]

Amazing job Lightfoot. Nice fix... Keep up the good work.

[lightfoot]

Thank you! At the moment it looks like this repaired unit hashes pretty well at 60 or so, will be contacting the user to discuss. I think it's a pretty neat repair, all things considered :-)

And on another Titan note, got one in that was dead shorted on the 12v side, turns out the person took off the heat sink and didn't re-apply paste when putting it back on. The paste KNC uses is a one time thing that hardens like a ROCK. The result is this:



And a DC-DC converter that blows its' FETs (you can see the worst heat damage is over in that corner of the board), which causes +12 to hit the hashing die which if the die wasn't already fried, ends the die. Fortunately the other three were fix-able and the unit is back to hashing at a solid 60mh.

On to the next thing. Still working on trying to fix a unit that hashes on only one engine (I think this was from the factory, it already has two bypassed) and thinking about how to clear these shorts on line 6.

[FineHash]

I have a cube (board) that looks identical to this pic.  Sent PM, could greatly use your assistance if and when you might have time.

It is awake, but lazy and unmotivated to do any work 
https://i.imgur.com/iHxOgVR.png

Can compensate you in electronic currency and compliments, and/or beer if you're close enough.

[lightfoot]

I have a cube (board) that looks identical to this pic.  Sent PM, could greatly use your assistance if and when you might have time.

It is awake, but lazy and unmotivated to do any work 


Can compensate you in electronic currency and compliments, and/or beer if you're close enough.

Hate to say it but most of my success is on boards that are blown up. Either incinerated, blown supplies, and the like. The titan chips are finicky, have you tried Tarkin's firmware?

Anyway when you log into the board using ssh as user pi and password whatever your password is and type screen -r what do you see?

C

[FineHash]

I have a cube (board) that looks identical to this pic.  Sent PM, could greatly use your assistance if and when you might have time.

It is awake, but lazy and unmotivated to do any work 
https://i.imgur.com/iHxOgVR.png

Can compensate you in electronic currency and compliments, and/or beer if you're close enough.

Hate to say it but most of my success is on boards that are blown up. Either incinerated, blown supplies, and the like. The titan chips are finicky, have you tried Tarkin's firmware?

Anyway when you log into the board using ssh as user pi and password whatever your password is and type screen -r what do you see?

C

Too bad about this "newbie" label I have...    I am very familiar with bfgminer, linux, etc.

Here is a bfgminer screenshot:
https://i.imgur.com/e17mQIf.png

Here is a pic of the bottom of the board:
https://i.imgur.com/bsqMHSX.jpg

Now here is a crazy detail... depending on how tight/loose the heat sink is cinched, the entire cube will/won't appear on the (gentarkin) firmware advanced settings page.  So in other words, if I screw it on normally, no cube or dies appear at all.  If I crank it down a bit more, I see dies but no hashing.  Possible short?

[lightfoot]

https://i.imgur.com/e17mQIf.png[/img]

Here is a pic of the bottom of the board:


Now here is a crazy detail... depending on how tight/loose the heat sink is cinched, the entire cube will/won't appear on the (gentarkin) firmware advanced settings page.  So in other words, if I screw it on normally, no cube or dies appear at all.  If I crank it down a bit more, I see dies but no hashing.  Possible short?

What do you get with the waas command? And how does bfgminer look when cranked down (don't do this too hard).

Based on the condition of the back of the board my guess is that either it was run like super hell or the heat sink was bad/loose and the top dies burned the board. That takes more heat than I can generate with pre-heat and my air tools, so it's a lot of heat. Enough to melt the solder and make a mess under the board.

A reflow *might* help, but those are dicey. If you want me to try let me know and send it in. I'll give it a shot, but my guess is that's the root of the problem.

Anyone found a source for huge .6mm pitch reballing stencils?

[FineHash]

https://i.imgur.com/e17mQIf.png[/img]

Here is a pic of the bottom of the board:
https://i.imgur.com/bsqMHSX.jpg

Now here is a crazy detail... depending on how tight/loose the heat sink is cinched, the entire cube will/won't appear on the (gentarkin) firmware advanced settings page.  So in other words, if I screw it on normally, no cube or dies appear at all.  If I crank it down a bit more, I see dies but no hashing.  Possible short?

What do you get with the waas command? And how does bfgminer look when cranked down (don't do this too hard).

Based on the condition of the back of the board my guess is that either it was run like super hell or the heat sink was bad/loose and the top dies burned the board. That takes more heat than I can generate with pre-heat and my air tools, so it's a lot of heat. Enough to melt the solder and make a mess under the board.

A reflow *might* help, but those are dicey. If you want me to try let me know and send it in. I'll give it a shot, but my guess is that's the root of the problem.

Anyone found a source for huge .6mm pitch reballing stencils?

Here is the waas result, running as root below, heat sink cranked down good (but not excessive) so that dies appear.  bfg shows the same as prior, "NO DEVICES FOUND" message, keeping in mind that I put the single cube on a controller all by itself for this particular test.  Interesting thing to note is that for a very brief time last night the cube hashed on 3 dies.  hmm.

Would you be OK with taking a look at it?  I'd be happy to cover shipping there/back.

Code:

Error reading /sys/bus/i2c/devices/2-0050/eeprom
ASIC board #0 is non-functional: Bad EEPROM data
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
KnC: Frequency change FAILED!
KnC 1-0: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 1-1: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 1-2: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 1-3: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 2-0: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 2-1: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 2-2: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 2-3: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 3-0: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 3-1: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 3-2: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 3-3: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 4-0: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 4-1: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 4-2: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 4-3: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 5-0: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 5-1: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 5-2: Frequency change FAILED!KnC: Frequency change FAILED!
KnC 5-3: Frequency change FAILED!i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
i2c_smbus_read_word_data failed: addr 0xD0
i2c_smbus_read_byte_data failed: addr 0x98
{
"asic_1_voltage": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_1_frequency": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_2_voltage": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_2_frequency": {
        "die1": "325",
        "die2": "300",
        "die3": "300",
        "die4": "325"
},
"asic_3_voltage": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_3_frequency": {
        "die1": "325",
        "die2": "325",
        "die3": "325",
        "die4": "325"
},
"asic_4_voltage": {
        "die1": "-0.0366",
        "die2": "-0.0366",
        "die3": "-0.0366",
        "die4": "-0.0366"
},
"asic_4_frequency": {
        "die1": "100",
        "die2": "100",
        "die3": "100",
        "die4": "100"
},
"asic_5_voltage": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_5_frequency": {
        "die1": "300",
        "die2": "300",
        "die3": "300",
        "die4": ""
},
"asic_6_voltage": {
        "die1": "",
        "die2": "",
        "die3": "",
        "die4": ""
},
"asic_6_frequency": {
        "die1": "325",
        "die2": "",
        "die3": "",
        "die4": "325"
}
}