To me there is nothing obviously wrong in your kernel log.  I've checked it against one of my miners with what I believe is the same hw/fw config as yours and the log is identical "line for line" including "bmminer not found, restart bmminer" and the chain temps etc - so I don't think either of those previous suggestions are your problem.   You've obviously done the logical troubleshooting - i.e. Miner-Bad runs fine and doesn't reset with original fans attached, but still resets even with a known working fan delete mod swapped from Miner-Good.   Frustrating.  It's possibly crashing when it gets to the "checking fans......" stage e.g. this is from one of my auto-tune S9is which occurs around the same time in the kernel log as when yours crashes:


As you have already trouble-shot your hardware mod, it's possible that there is some subtle difference with your Miner-Bad's compared to the Miner-Good's - but the issue only shows up when you attach the fan delete mod. You have probably done this already - but I'd start looking for what is different with these crashing miners - if anything.  E.g. you mention that you have S9's &  S9i's in 13, 13.5 and 14 variants - so is there any obvious pattern with the 7 in your tank that are not working e.g. it would give us more clues if, for example, they were all S9i's etc?   Also as per the quote below, you use an example of 2 x 13.5 S9i Miners, but your kernel log appears to be from a plain S9 - which is fine obviously as you only mentioned S9i's "as an example".    So as the "devil is in the detail" can you just confirm that the kernel log was from an S9 i.e. not from an S9i but with the "wrong" S9 asci-boost firmware applied?    


Until recently, there was only the fixed-freq and the auto-freq firmware for the S9 available.  But since Bitmain launched the S9i and S9j and then also released the asic-boost firmware's (very confusingly in different ways for the 3 S9 models  - i.e. loaded "on-top" of the previous firmware for the S9s, but a total new replacement firmware for the S9i and S9j)  - there are now 11 current "latest" firmware's for the S9 models available on the Bitmain website.   Combined with a lack of Bitmain documentation this has led to the people flashing the wrong firmware to their miners - e.g. S9 FW on an S9i and lots of confusion about the asic-boost update process. The miners may appear to work with wrong firmware in some cases, but it could cause "strange" issues or sub-optimal performance.

So assuming that you are using the standard auto-tune / auto-freq, to completely rule out any FW incompatibility issues, your miners should be as follows if they are up to date:

S9 - the asic-boost patch Antminer-S9-LPM-20181102.tar.gz  flashed on top of the latest auto-tune FW - Antminer-S9-all-201711171757-autofreq-user-Update2UBI-NF.tar.gz which was probably what your miners already had if they were previously up-to-date.  Unfortunately due to the inconsistent way Bitmain released the asic-boost for the S9s as a small patch file, once it is applied "on-top" there is no way of knowing what the "main" firmware version is - the miner's overview page just displays the asic boost version FW details i.e. - "File System Version   Sun Nov 2 11:55:42 UTC 2018"

S9i - just the asic-boost new FW - Antminer-S9i-all-201811071119-autofreq-user-Update2UBI-NF.tar.gz   (the very latest S9i version released a few days ago drops both the power and hashrate by around 40% - which Bitmain either didn't intend or haven't documented - so I guess you don't want to use that).

Apologies if the above is obvious to you and you are certain that you have the correct up-to-date FW on all your different S9 models.  I only mention it as the Bitmain documentation is not clear and some experienced miners have therefore flashed the wrong firmware as can be seen in other sections of this forum.

Therefore if you are 100% certain of your fan delete mods, all the FW is up-to-date and the correct version for the S9 variant and there are no obvious patterns with the "bad miners", then I guess it is probably going to have to be trial and error to get them working without the fans.   A couple of suggestions which are quick to try are:

a. Try flashing a "bad" S9 with the appropriate fixed-freq firmware: e.g. Antminer-S9-all-201705031838-650M-user-Update2UBI-NF.tar.gz  which is for a 14 Th/s S9 but should be fine for all of them.   The fixed-freq FW does a much quicker "boot" process and skips all of the auto-tune and auto-freq processes - so that may help to get round the "fan speed checking" issue.  Some of my miners are using fixed-freq FW and unlike my normal auto-tune miners the are no "fan checking" lines in the log - so it might work for your set-up - but no guarantee.   If it works then you can apply the LPM asic-boost FW on-top which will leave you with a boosted fixed-freq miner still running with less power, but now also faster than before.  Your 13 and 13.5 Th/s S9's may be fine running with the 14 TH/s fixed-freq FW - but if not (i.e. unstable or too many HW errors) just use  http://192.168.1.xx/cgi-bin/minerAdvanced.cgi to get to the hidden config page and set the freq manually to whatever is stable (or back to what the miner originally was e.g. freq=631.25 for 13.5Th/s etc).   Obviously you can't do this for your S9i's as Bitmain doesn't offer any fixed-freq firmware for this model.

b. Another suggestion - it may not work - but it only takes a few seconds to try - is to manually fix the fan speed - or notional fan speed in your case.  I don't know if a fixed "dummy" fan speed is compatible with your "fan delete hw mod".   This should just end up with an entry as follows as the very last lines of the kernel log and hopefully will mean that the fan speed does not get checked during restart - and hence hopefully stop your miners from crashing.  
  

As you are doing immersion cooling, you are probably far more expert than me with tweaking fans etc, however below is the method I use to manually over-ride the auto-tune fan speed - only takes a few seconds + a restart (it's only temporarily revealing the hidden options in the standard Bitmain miner web-server - so it's not changing any code etc):

Great - thanks - so that confirms you are actually running the previous S9i 7th November firmware - which I think everyone agrees works fine as per your experience.   It is the newest 27th November Antminer-S9i-xilinx-201811271554-autofreq-user-Update2UBI-NF.tar.gz firmware that is causing huge hash-rate reductions.   So unless you are happy to reduce your S9i power draw and hashrate by around 30 to 40%, don't load the new S9i firmware.

That's an important question.   There are a number of earlier posts that have inadvertently caused confusion by stating that "everything was fine on the latest firmware" etc with the posters not realizing that Bitmain had released a newer version or just simply they were using different firmware to the poster asking the question.  Also some of the posts responding to overclocking and FW update procedure questions are again giving inaccurate or generalized  information e.g. suggesting procedures for the S9i or j that are only applicable to the S9.

To easily avoid this confusion, I suggest that we explicitly post either the full FW file name or at least the miner type and FW release date e.g.  "S9j 20181127" rather than using "it" or "this firmware" which could potentially refer to any of the 11 current valid firmware versions for the 3 models in the S9 family which are still available for download on the Bitmain Support website.


The recent posts about significantly reduced power and hash-rate are referring to the new firmware Bitmain released a few days ago (27/11/2018) only for the S9i and S9j, so by mentioning you downloaded this to S9s are you actually referring to the earlier asicboost firmware?   No firmware for the S9 variant has been released since the 2nd November 2018.  So did you load the new 27112018 S9i/j firmware onto an S9?

After yesterday's two failures, I've just tried the new 27/11/2018 S9i FW on a 3rd miner - a S9i 13.5Th/s.   Same result - certainly it reduced power consumption (roughly from the PDU meter by around 1 to 2 amps at 230v) but it also reduced the hashrate by 5 Th/s.  

It's hopeless - it's as if Bitmain didn't even test it on a single miner, let alone many.  I've tested it with LPM ticked and unticked - no difference.

It sets the voltage to 8v on all boards rather the higher different voltages depending on each board's quality.  It also takes longer to boot because it goes through the game of setting the voltages starting at 9.40v and decrements 0.1v at a time as below - it does this sequence twice at reboot (before and after it sets the chain freq).  Complete waste of time - as it is always going to be the same answer i.e. 8.00v.  This new firmware is terrible code.


It then sets the board freq much lower than it's base.  This is the best board - the other two got even lower 4nn speeds.


You can see the low hashrate compared to the other miners on the earlier asicboost FW in the image below.  Also as you can see every 5 or 10 minutes all the 189 chips disappear.   The HW error rate is also much higher than the 4 identical S9i's underneath it in the table.

There is one advantage to this FW - temps are 10 degrees less and the fans should last a long time - they're barely moving.  

https://imgur.com/mrewNnm

Finally now the HW errors shown next to each board no longer add up to the total.  According to this FW 8+1=111    See image below.

https://imgur.com/8c9TbMS

I just tried this new Antminer-S9i-xilinx-201811271554-autofreq-user-Update2UBI-NF.tar.gz on a S9i 14 and a S9i 13.5.    Very bad results for both of them - it certainly reduced the power considerably  on both of them and as a result also hugely reduced the hash rate from 14 to 9.5 and 13.5 to 7.5.   Plus because of the low chain voltage it was clocking up lots of HW errors.   So I obviously flashed them both back to the 201811071119 FW.

The only difference in the firmware that I can see is that it introduces a new chain backup voltage of 8.00v for all boards that (at least on the 2 miners I tested) is overriding the voltages stored on each boards PIC controller e.g. the PICs have 8.9v, 8.9v & 8.7v on one of my S9i 13.5s - hence the significantly lower power, rubbish hash-rate and increased HW errors.  e.g. the only frequency my S9i 14.0 could manage to get up to was 493 instead of it's normal 650.  Even worse because it was clocking errors and didn't seem stable at 8v, the real RT hash-rate was a lot less than the ideal (which is based on the freq) e.g. on a S9i 13.5, RT = 7500, Ideal = 9500  - RT was around 13650 before this FW - so it's only hashing at 55% of it's normal rate.

I'm not sure why Bitmain released a second asicboost FW for the S9i's or what the difference is between the two versions - nothing on Twitter etc - anyone seen a Bitmain explanation anywhere?  The description next to the firmware has changed from

"1. Reduce power consumption."  

to

 "1. Add support for low power mode, reduce power consumption".    

I think a few of us of assumed that the LPM option didn't do anything on the initial asciboost FWs.  Maybe Bitmain have enabled it now.   Maybe unticking LPM on the General Settings screen will stop it selecting the very low (and unstable) backup voltage - it's late here - so I'll try one tomorrow. Stryfe - you said yours are hashing normally - did you have LPM ticked?  

If this is meant to be a more efficient Low Power Mode, it's not working for me as the % drop in hashrate is more then the drop in power so it's less efficient, more unstable and I'd need to install more miners to get my total hash-rate back to where it was before.
 
I didn't unplug any miners to check the precise power drop, but the PDU amps dropped by around 1.5a (350w) - which won't be precise because there are other miners on that PDU.

---

You can't adjust the frequencies on the S9i/j's with Bitmain firmware - there are no fixed freq FWs listed for these miners.  There are other ways of doing it with 3rd party firmware.  Also there is no dependency on the prior firmware as you mention with this LPM update.

The S9i and S9j asicboost FWs are complete replacement firmware's - so they will just totally overwrite the previous firmware.  If you look at the various FW sizes for the S9 and S9i/j it is obvious.  My posts earlier in this thread explaining about applying the LPM FW on top of a fixed FW only applied to the S9 - which only had a small "patch" LPM FW upgrade which was applied "on top".   It doesn't work that way for the S9i/j - unless someone wants to risk trying the S9 firmware's on the S9i/j's.

---

Almost certainly (99.9%) you an on a slightly older version of Antminer-S9-all-201711171757-autofreq-user-Update2UBI-NF.tar.gz.  

This is the autotune / autofreq FW.  You don't have to load any other firmware first unless you want to try over-clocking your miner - so just go ahead and apply the S9 LPM firmware i.e.  Antminer-S9-LPM-20181102.tar.gz, although it would be better to get your miner up-to-date first by flashing it with Antminer-S9-all-201711171757-autofreq-user-Update2UBI-NF.tar.gz if you want to stay on autotune before applying the LPM update.

If you want to be absolutely sure what FW you are you can post your full kernel log and I'll be able to confirm 100%.

Hi - From your varying fan speed results and your partial kernel log with PWM value of 50, your miner is clearly still on auto-tune fan speeds - so it is ignoring your manual fan settings - which should never happen.   That's strange as the "fixed-fan-speed" is standard Bitmain functionality that is available on all of the S9 firmware - all my "hack" did was to to make the manual option temporarily visible on the Miner Config page.   So I suspect something else has changed on your miner - e.g. the line in the kernel log about "no temp offset" value found stored in the pic controller doesn't look right for a normal S9, although that alone isn't the obvious cause as the software then did a fix and obtained a reasonable looking offset.   But it is suspicious nonetheless.   Did you "back out" the previous edits that you did to bmminer.conf or start from a clean firmware install before setting the manual speeds?   Although your bmminer.conf edit didn't appear to work, it may still have made some unpredictable changes.  

So, if you haven't already tried it, perhaps re-flash with a clean fixed-650 firmware (and LPM of course) and retry the fixed fan settings.  Or for even more "belt & braces" you could try a factory reset and re-flash firmware etc.   As mentioned manual fan control is a standard built-in (but hidden) Bitmain function so it should always work on any S9.

Sorry - I forgot to mention about reconfirming the freq when you change the fan setting.   On the Fixed freq 650 FW version, if the inputted freq field is left blank, it sets a safe 600 default value - so you have to input the value again like you did.

I don't understand why the fan started at 6,120 and then went to 3,600 - as you have input a PWM of 85% it shouldn't change at all.

For thoroughness - as we are on a tweaking-fest  , it would be worth checking what chain voltages got set in the kernel log for your high hash rate.  Hopefully it is nice and low like 8.7 or 8.8v (it will show in the log as 880).   It doesn't matter much if it is higher like 9.4v etc, but higher voltages will obviously draw more power, although they may be needed for stability.    The voltages can also be tweaked to find the ideal compromise between stability and power consumption, but there usually isn't much need unless the miner starts misbehaving or you start getting missing hash boards.

The very simplest method - it takes at most 30 seconds per machine and doesn't involve customizing Bitmain's code at all - is as follows:

Note: I've used Chrome - but it is similar for other browsers.

1.  Open your miner in your browser and go to the "Miner Configuration" tab.
2.  Right-click the area around the new "Low Power Mode" and click "Inspect" - the  Chrome DevTools Elements panel will appear on the right.
3.  A few lines down from the Low Power Mode, you'll notice a line with "fan-ctrl" in it.  
4.  On the fan-ctrl line, highlight the  text :none from the style="display :none", press delete and then press enter.
5.  You will now temporarily have a "Customize the fan speed percentage" check-box and input box showing in the miner GUI.  
6.  Enter your new fixed fan speeds (85 to 90% should be good for most overclocking) and do the normal "Save & Apply".
7.  Once the miner restarts - check the kernel log - at the very bottom, it will now show the following instead of the usual auto-tune lines:


In summary - it's just right-click to inspect the web-page, delete the :none text on the fan control line and you are done.  

Before edit screenshot:
https://imgur.com/cDHmAlF

After edit screenshot:
https://imgur.com/cIvktyx

S9i and S9j Asic Boost firmware has just released on Bitmain site.  No T9+ release yet.   Will try them out now.

Update - the S9i version is the full firmware - i.e. over 60mb and not like the S9 LPM which just patched a few files and was < 1mb.   Makes sense because there are no fixed freq firmware's for the S9i - so they can just fully replace the last version.   For the S9 - they would have to replace the previous 3 fixed and the 1 auto-tune versions if it was a complete firmware upgrade.

You were quick with the firmwares and reboots!  Glad to see it is working.  

Check your fan speeds - at those temps they would normally be nearing max - i.e. 5,600 to 6,000 on the intake fan.   Yours are probably lower than this because for some strange reason the fixed freq firmware sets them lower - hence the boards get hot for no good reason.

If so, you basically have spare "fan speed" capacity to speed them up and still run a high hash rate at reasonable temps.   If you want to increase the fan speeds back to the "usual" speed let me know.    It only takes around 30 seconds to do, but it takes a lot longer to explain how to do it than to actually do it  

Edit: your update 2 crossed in the post with my message.  Yep as expected the fans are far too slow - but your temps are not too bad even with these slow fans.     If you set them to around 90% , you'll get speeds of around 4080 / 5640 on the 2 fans.  This should bring your temps down to the 70s even on your 15TH/s.  The fans will also then be fixed - which shouldn't be a problem unless the weather / room temp changes a lot.  It only takes 30s to tweak the fan speeds if you need to occasionally.   Or you can be lazy and set them to a nice noisy, fan bearing ruining 100%!

Hi - the LPM release didn't really change the previous firmware - it just added the Asic Boost function "on-top" - so whatever firmware you were on - i.e. fixed or autofreq - will still essentially stay the same.   Assuming that you were / are  on the standard autofreq firmware, if you have input a higher speed using http://192.168.1.xx/cgi-bin/minerAdvanced.cgi, it will just get ignored - you can see that in the kernel log - it will still do the auto-tune function.

So you need to flash your miner with a fixed freq firmware i.e. Antminer-S9-all-201705031838-650M-user-Update2UBI-NF.tar.gz  and then reapply the LPM firmware i.e.  Antminer-S9-LPM-20181102.tar.gz  "on-top" of the fixed freq firmware.    This will give you the Asic Boost function and the ability to increase your hash-rate.   As standard on the fixed firmware your freq will initially be fixed at 650 (i.e. 14 TH/s), so then you need to over-ride that using http://192.168.1.xx/cgi-bin/minerAdvanced.cgi  which I guess you are already trying .  That new frequency will get processed this time -  and easy to check in the kernel log.

You are correct - they have used the beeper checkbox.   The html just shows that the display label has been changed (and obviously the style= "display :none" has been removed) - everything else rather sloppily still says beep etc.   My guess at the moment is that it doesn't do anything, but that they might use it with a future "major" firmware update.   I think the LPM firmware was a small scale rushed patch, that they haven't yet  integrated with the previous autofreq and fixed firmwares.


A couple of my 14s were set as 13.5s when they arrived from Bitmain, so I manually changed the settings as it wasn't worth the expense and time to send them back - hence on these two I have all of the possible fields available on the miner config screen.  As an aside, they run fine at 14 at nice low voltage - so they were probably good quality 14s and just set-up with the wrong pic freq by Bitmain.    When I applied the LPM update to these two machines, the "Beeper Ringing" checkbox label changed to "Low Power Mode" - but everything else stayed the same - see image below:

https://imgur.com/a/gCVww5V

---

The message is ambiguous / misleading.   The "3 hour" message says about searching - even on the fixed freq firmware, which doesn't make sense.  As you will be using the fixed frequency firmware, there is no frequency searching, so the miner will reboot much quicker than on the autofreq firmware.  You can compare the kernel logs - you can see that it just sets the same frequency on all 3 boards to whatever you input and skips all the time consuming searching and setting each board individually part.  You'll probably notice the fans don't go up and down like screaming banshees so much when the miner starts either.   

Keep an eye on your temps - the fixed freq firmware runs the fans slower than the autofreq, so with two identical S9s on different firmware but set for the same hash rate, the fixed freq one will run hotter.  If they do run too hot, you can manually set the fans to a similar speed as they were on the autofreq firmware.

It looked like mine were hashing a little higher after flashing the latest firmware - but only around 0.15, so couldn't be sure.   Now that they have been running for a few days, the "boosted" S9s are definitely around 0.15TH/s faster on average.  See image below from Slush Pool worker page.  Filtering on 13.5 machines (labelled as 13) and sorting by the average daily hash rate the boosted S9s are all above 13.5, whilst the S9is without Asic Boost are all usually on or below 13.5 TH/s.   Previously the S9is and S9s appeared all mixed up on the list  - i.e. similar speeds.

The cropped image just shows a few miners above and below 13.5 - there are more S9s higher and S9is lower on the list obviously with slightly bigger differences in hash rate.   The daily hash rate is the middle column in the image.  The 9's have been consistently above the 9i's for the past few days so it is not a random occurrence.   It will be interesting to see how the worker order changes when Bitmain get round to releasing the S9i/j firmware update.

https://imgur.com/aGTwrp6

Been running the latest version for around 40hours – looks OK.
Applied the asic boost patch to 13.5 and 14 TH/s S9s - mostly on top of the autofreq firmware and a few on the fixed freq firmware – previous firmware features / settings all retained – including a few custom ones I’d tweaked on the fixed freq devices.  Haven’t tried any S9i’s – will wait for Bitmain to release this.
Haven’t scientifically logged the individual miner hash-rates or amps, but from the circuit PDU power meters etc I am seeing the following:

1.   No gradual speed decline with this version – stable for over 40 hours.
2.   Greater than 10% power saving.
3.   Good temps – no obvious change – a few degrees less than the “unboosted” S9i’s so might be running a fraction lower?
4.   Noise – subjectively appears to be less – didn’t check before / after with a decibel meter – fan speeds appear to be a few 100s rpm lower.
5.   HW error all showing 0.0000% so far – so no obvious change.  All devices and boards are very similar i.e. 55 to 65 errors per S9 after 40 hours so around 1 error per board every 2 hours at 70 degrees c.
6.   Hash rate is consistently at least equal to what the device should be and possibly 0.1 to 0.2 more i.e. around 13.65 and 14.15 TH/s.
7.   No damage during the numerous firmware flashes over the past week but did get the following 2 “funnies” – resolved by reboots.
a.   One hash board showing 0 hash rate with all chips showing good.
b.   One board dropped to around 50% again with all chips allegedly good.
8.     The scoring hash-rate stability is reflected on Slushpool stats - i.e. overall HR is no longer slightly fading over 48 hours and appears to be around 0.8% up (overall including some unboosted S9is).