Hacking The KNC Firmware: Overclocking

[sickpig]

from my limited experience I think  that both points apply to everyone. For what is worth I've been albe to verify only the second one: every time I overclock the miner a good number of cores will be disabled during the first two minutes of the new cgminer session. Usually they're concentrated in one or two die.

In my case dies are being disabled (1 die = 48 cores). If just cores are disabled then that is easy to fix - just apply more voltage to stabilize them.

ok got it

On the other hand I've never tried
to verify the first point, mainly because I don't know how to do it. How do you know for sure that the clock has been reset, do you look at the Amps or is there anyway to read the value of a PLL registry? 

Yes, voltage drops. For example you apply the overclock and you see a VRM working at 54A. You change the voltage setting just 1-2 values lower and after you hit apply the Amps drop from 54 to 44, which means the higher overclock frequency is no longer applied.

ok.

next time I've to do it I will pay attention to Amps values just after changing the voltage. in the meantime I'll try also to find a way to read the PLL register that contain clock settings.

Instead of increasing the voltage to the maximum value, I just set it a little bit higher take into accounts on how many
cores are disabled in a particular die. the I restart cgminer and wait 1 minute to see if changes make any difference.

I use this approch because I don't want to cook my asics/vrms.

I've tried that, but it doesn't work as effective as applying max voltage. With max voltage the sleepy dies usually kick in immediately.
Also I have a 2nd theory, which I haven't tested a lot, but if you supply sufficient voltage to a sleepy die it might awake after 2-3-4-5-6 hours. But I don't think there is any consistency in results with this method and I prefer to wake them up immediately with stress/shock voltage that waiting hours for them to wake up naturally, which is not guaranteed to happen.

since any other methods you have tried have failed just give this theory of yours a try, no? 

I've also increased the SPI frequency because of what 'orama said in one of hist last post said:

How much? I played with this and I usually stick to 256000Mhz. I tried even more, but I can't see any correlation between this and any results.

actually I'm OC quite slightly, using 1F1, and I'm using as SPI freq 299707 and prev I 've used 256000, I haven't seen now difference, though.

Another think I do is taking not of all the changes I apply along the way (a goodthing is coping /config/adavanced.conf at differnet moment in time)   

To check the distribution of disabled cores I use a modified version of a pl script included in bertmod. It is an ASCII version, it only outputs temps and disabled core per die, e.g.

How exactly did you do that?

bertmode 0.2.X bin file contained a perl script, asic_status.pl, used to generate a modified version of the status page. A btctalk user (don't rember the name sorry) have changed it to just puts on standard output a few info.
 

You created an additional page within the lighttpd server?

no I've just placed the modifed script into /config (just to make persistent across reboost), to use it I just log in using ssh and execute from the command line:

Code:

perl /config/asice_status_ascii.pl 

if you find it useful I could upload the file somewhere...

[r1senfa17h]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

[CYPER]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

[r1senfa17h]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

[CYPER]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

Wow, that is a crazy overclock then 

Do you mind posting a screenshot of your Status and Advanced Tab + SSH Scgminer Tab + your settings.

Also are you not worried about drawing more than 200A per board?

Thank you.

[r1senfa17h]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

Wow, that is a crazy overclock then 

Do you mind posting a screenshot of your Status and Advanced Tab + SSH Scgminer Tab + your settings.

Also are you not worried about drawing more than 200A per board?

Thank you.

Yes, I am worried about the 200A per board, but I'm willing to risk it. Here's the info you requested:

FW version: 0.99-tuning (October)
cgminer version: 3.9.0
Status Page
Advanced Tab
modified cgminer.sh file

[CYPER]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

Wow, that is a crazy overclock then 

Do you mind posting a screenshot of your Status and Advanced Tab + SSH Scgminer Tab + your settings.

Also are you not worried about drawing more than 200A per board?

Thank you.

Yes, I am worried about the 200A per board, but I'm willing to risk it. Here's the info you requested:

FW version: 0.99-tuning (October)
cgminer version: 3.9.0
Status Page
Advanced Tab
modified cgminer.sh file

Thank you. It looks like your last board is not a team player Is the last die so bad that you even underclocked it below factory frequency?

Also I assume you installed CGminer manually, because 0.99-tuning comes with 3.8.1 and 0.99.1-Tune comes with 3.9.0.

[r1senfa17h]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

Wow, that is a crazy overclock then 

Do you mind posting a screenshot of your Status and Advanced Tab + SSH Scgminer Tab + your settings.

Also are you not worried about drawing more than 200A per board?

Thank you.

Yes, I am worried about the 200A per board, but I'm willing to risk it. Here's the info you requested:

FW version: 0.99-tuning (October)
cgminer version: 3.9.0
Status Page
Advanced Tab
modified cgminer.sh file

Thank you. It looks like your last board is not a team player Is the last die so bad that you even underclocked it below factory frequency?

Also I assume you installed CGminer manually, because 0.99-tuning comes with 3.8.1 and 0.99.1-Tune comes with 3.9.0.

Yep, that last board hates me. And yes, I did install cgminer manually.

[CYPER]

nothing.

your overclock settings will stay in place, but your change to voltage won't get applied :/

So the voltages I have set would not get back to default when I apply the overclocking?

[sickpig]

nothing.

your overclock settings will stay in place, but your change to voltage won't get applied :/

So the voltages I have set would not get back to default when I apply the overclocking?

right, they would not get back.

restarting cgminer through cgminer.sh does not overwrite the voltage settings you apply via "Advanced" tab.

[sickpig]

I just wanted to report that my Oct Jupiter has been running at ~950GH/s (6-modules) and pulling an between 57-60 amps per VRM for about a week now. Temps range between 57-78.5 degrees Celsius. So far so good!

You are a brave man for pulling much more than the recommended max safe current 

You use 211 I assume?

I use different settings for each die, but many of them are using 231

Wow, that is a crazy overclock then 

Do you mind posting a screenshot of your Status and Advanced Tab + SSH Scgminer Tab + your settings.

Also are you not worried about drawing more than 200A per board?

Thank you.

Yes, I am worried about the 200A per board, but I'm willing to risk it. Here's the info you requested:

FW version: 0.99-tuning (October)
cgminer version: 3.9.0
Status Page
Advanced Tab
modified cgminer.sh file

I want to congrat for your setup and for your achievement

having said that, the temps showed in the status page is very high for the two 4 VRMs asic slots (68 and 78). Have you bought this two from KNC's expansion batch?

One last thing I saw is that, always for this 2 4 VRMs slots, you have setted negative voltage for each die, what does it mean?

Anyway after seeing this it seems that  8 VRMs are better suited for OC.

Many thanks for sharing those info, really appreciated.

[Cablez]

Wonderful thread gentlemen, really shows the pioneering spirit of the mining core. Thank you.

Has there been a conglomeration yet of the tuning suite's best practices.  Kind of like 'these ranges of voltages seem to keep up performance for less power' or 'certain SPI frequencies yield better stability results' etc.?

I would like to start adjusting values on the Adv. page if there is cause but without a manual or even best practices it is daunting. 

[sickpig]

Wonderful thread gentlemen, really shows the pioneering spirit of the mining core. Thank you.

Has there been a conglomeration yet of the tuning suite's best practices.  Kind of like 'these ranges of voltages seem to keep up performance for less power' or 'certain SPI frequencies yield better stability results' etc.?

I would like to start adjusting values on the Adv. page if there is cause but without a manual or even best practices it is daunting. 

no. unluckily there's no a "easy" howto so far. anyway the two things I've learnt about "Advanced" settings are:

- once you've overclocked probably you'll notice more disabled cores than usual. to make those cores working again you need to set a higher voltage for the die to which they belong to.

- The higher the hashrate the higher the SPI freq if you want to keep the HW error rate in shape, and that means that probably you need to higher also SPI voltage (I still haven't tried this, though)

[sickpig]

other two things I've just discovered:

- voltage settings are persistant across reboot. it makes sense since they're stored into /confg/advanced.conf

- the rear asics board run hotter than the front one (with or without case, and with or without overclocking). in my case the hotter is the rear-left one. I have a spare case fan and I've decided to use it to cool down the aformentioned asic board. The result was quite impressive on a 1F1 OC machine the temps lower from 65C to 42C and over HW error rate lower from 4.5% to 3.1%.

I'm planning to the same for the rear-right asic as soon the next time I'll go to the colo facility. Once I've done that I'll try to run the miner to higher freq/voltage.

[Cablez]

I have my jupiter now at '201' with the case off and an extra 120mm fan right in the middle pushing the cooler intake air toward the rear boards.  Seems to give steady 621GH/s (12+hrs) with temps from 40-58C across the boards.  With proper cooling I have kept the wattage per asic around 40W so as not to overload the vrms.  Not a bad result for a small text editor change and a fan.

Next up will be to see if I can lower the voltage needed to keep this performance and maybe go to '211'.  I had missed tinkering in the asic era,  no longer. 

[sickpig]

I have my jupiter now at '201' with the case off and an extra 120mm fan right in the middle pushing the cooler intake air toward the rear boards.  Seems to give steady 621GH/s (12+hrs) with temps from 40-58C across the boards.  With proper cooling I have kept the wattage per asic around 40W so as not to overload the vrms.  Not a bad result for a small text editor change and a fan.

Next up will be to see if I can lower the voltage needed to keep this performance and maybe go to '211'.  I had missed tinkering in the asic era,  no longer.  

what's your HW errors rate?

I'm going to apply 201/211 to the rear-left asic just to see how it beahave. if everything is ok I'll set 201 also for the front boards.

[jbah01]

Just to share my settings/performance:
- October unit (Asic 1 = 4 VRM, other 3 are 8 VRM but only 4 active)
- Firmware 0.99-tune
- Clocksetting on 231, no other firmware changes.
- Harware mod: Lowered the Asicboard-fans (more airflow over the board), added 2 fans at the outlet (so now push + pull) and slightly lowered fan speed to reduce the noise (80%).

Images
Status page --> 676 GH/s / 9131 WU
Advanced page --> 2.5V SPI and 256kHz frequency, slightly modded volts (higher, as required for the increased clock), not completely fine-tuned.
CGminer --> Error rate used to be 3%@560GH/s, since I lowered the volts a bit after the initial increase this increased to just below 4%. Fine with me as it lowered the temps 5-10 degrees an decreased power to 'around' 40 W per DC instead of closer to 50 W.
Pool stat  --> 650ghs average
Power consumption in bertmod used to be 453 W, now it is 651 W (43% increase in consumption for 18/19% performance increase). With voltage finetuning the power consumption should be able to come down a bit, now it is all pretty much on the safe side (safe as in: higher voltage as required to avoid that cores get shut down).

[Cablez]

I have my jupiter now at '201' with the case off and an extra 120mm fan right in the middle pushing the cooler intake air toward the rear boards.  Seems to give steady 621GH/s (12+hrs) with temps from 40-58C across the boards.  With proper cooling I have kept the wattage per asic around 40W so as not to overload the vrms.  Not a bad result for a small text editor change and a fan.

Next up will be to see if I can lower the voltage needed to keep this performance and maybe go to '211'.  I had missed tinkering in the asic era,  no longer.  

what's your HW errors rate?

I'm going to apply 201/211 to the rear-left asic just to see how it beahave. if everything is ok I'll set 201 also for the front boards.

Right now its at 1.9% hardware errors.  I may go up to '211' if I can keep the wattage close to 40W per vrm.

[wizkid057]

On the November unit I have for testing, disabling work flushes in the cgminer source and running cgminer with -q -T gives me the following stats:

(5s):731.0G (avg):681.2Gh/s | A:88420915  R:1590264  HW:133439  WU:9414.2/m

Comes to ~0.15% HW errors.  Pool side 3-hour rate is 670Gh.  Disabling flushes causes an increase in rejected shares, but this seems to be far less of a loss than the false HW errors.

Basically, flushing is broken in all current firmwares it seems.  There is also a false HW error issue in the cgminer driver that was partially fixed in this commit to their cgminer fork.

I'm currently working on a fresh rewrite of the driver.

Just figured I'd throw this out there somewhere.

-wk

[padrino]

On the November unit I have for testing, disabling work flushes in the cgminer source and running cgminer with -q -T gives me the following stats:

(5s):731.0G (avg):681.2Gh/s | A:88420915  R:1590264  HW:133439  WU:9414.2/m

Comes to ~0.15% HW errors.  Pool side 3-hour rate is 670Gh.  Disabling flushes causes an increase in rejected shares, but this seems to be far less of a loss than the false HW errors.

Basically, flushing is broken in all current firmwares it seems.  There is also a false HW error issue in the cgminer driver that was partially fixed in this commit to their cgminer fork.

I'm currently working on a fresh rewrite of the driver.

Just figured I'd throw this out there somewhere.

-wk

What was this Jupiter reporting at the pool for a 3-hour before this change? Any chance you would be willing to share the build? Would like to shake it out some on one of my Octobers..