S19 Pro Randomly Stops working with no Kernel Errors

[CryptoLLC]

I have 250 units. About 8 of these randomly turn off for no reason at all. There are no errors, temps are fine, I tried different psu, different control board, fans are at 97%. If you know what the reason is please let me know. Thank you.

Code:

CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
Machine model: Xilinx Zynq
cma: Reserved 16 MiB at 0x0e000000
Memory policy: Data cache writealloc
On node 0 totalpages: 61440
free_area_init_node: node 0, pgdat c0b3c040, node_mem_map cde10000
  Normal zone: 480 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 61440 pages, LIFO batch:15
percpu: Embedded 12 pages/cpu @cddf0000 s19916 r8192 d21044 u49152
pcpu-alloc: s19916 r8192 d21044 u49152 alloc=12*4096
pcpu-alloc: [0] 0 [0] 1 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 60960
Kernel command line: mem=240M console=ttyPS0,115200 ramdisk_size=33554432 root=/dev/ram rw earlyprintk
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 209572K/245760K available (6317K kernel code, 243K rwdata, 1932K rodata, 1024K init, 232K bss, 19804K reserved, 16384K cma-reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xcf800000 - 0xff800000   ( 768 MB)
    lowmem  : 0xc0000000 - 0xcf000000   ( 240 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .text : 0xc0008000 - 0xc090e410   (9242 kB)
      .init : 0xc0a00000 - 0xc0b00000   (1024 kB)
      .data : 0xc0b00000 - 0xc0b3cda0   ( 244 kB)
       .bss : 0xc0b3cda0 - 0xc0b77024   ( 233 kB)
Preemptible hierarchical RCU implementation.
	Build-time adjustment of leaf fanout to 32.
	RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
efuse mapped to cf800000
ps7-slcr mapped to cf802000
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at cf802100
Zynq clock init
sched_clock: 64 bits at 333MHz, resolution 3ns, wraps every 4398046511103ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x4ce07af025, max_idle_ns: 440795209040 ns
Switching to timer-based delay loop, resolution 3ns
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 537538477 ns
ps7-ttc #0 at cf80a000, irq=18
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 666.66 BogoMIPS (lpj=3333333)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100058
CPU1: failed to boot: -1
Brought up 1 CPUs
SMP: Total of 1 processors activated (666.66 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor menu
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
zynq-ocm f800c000.ps7-ocmc: ZYNQ OCM pool: 256 KiB @ 0xcf880000
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
media: Linux media interface: v0.10
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
EDAC MC: Ver: 3.0.0
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm_global_timer
NET: Registered protocol family 2
TCP established hash table entries: 2048 (order: 1, 8192 bytes)
TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
PCI: CLS 0 bytes, default 64
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (no cpio magic); looks like an initrd
Freeing initrd memory: 6732K (cd466000 - cdaf9000)
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
futex hash table entries: 512 (order: 3, 32768 bytes)
workingset: timestamp_bits=28 max_order=16 bucket_order=0
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
dma-pl330 f8003000.ps7-dma: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.ps7-dma: 	DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
e0000000.serial: ttyPS0 at MMIO 0xe0000000 (irq = 159, base_baud = 6249999) is a xuartps
console [ttyPS0] enabled
xdevcfg f8007000.ps7-dev-cfg: ioremap 0xf8007000 to cf86e000
[drm] Initialized drm 1.1.0 20060810
brd: module loaded
loop: module loaded
CAN device driver interface
gpiod_set_value: invalid GPIO
libphy: MACB_mii_bus: probed
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 31 (00:0a:35:00:00:00)
Generic PHY e000b000.etherne:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=e000b000.etherne:00, irq=-1)
e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
i2c /dev entries driver
cdns-i2c e0005000.ps7_i2c: 100 kHz mmio e0005000 irq 154
Xilinx Zynq CpuIdle Driver started
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
mmc0: SDHCI controller on e0100000.ps7-sdio [e0100000.ps7-sdio] using ADMA
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nand: disable subpage write
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xda
nand: Micron MT29F2G08ABAEAWP
nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
nand: NAND_ECC_HW
nand: NAND_ECC_HW_SYNDROME
mtd->writesize = 2048
ecc->strength = 1
ecc->size = 2048
mtd->writesize = 2048
chip->ecc_strength_ds = 4
chip->ecc_step_ds = 512
nand: WARNING: pl35x-nand: the ECC used on your system is too weak compared to the one required by the NAND chip
Bad block table found at page 131008, version 0x01
Bad block table found at page 130944, version 0x01
8 ofpart partitions found on MTD device pl35x-nand
Creating 8 MTD partitions on "pl35x-nand":
0x000000000000-0x000002800000 : "BOOT.bin-dts-marker-kernel"
0x000002800000-0x000004800000 : "ramfs"
0x000004800000-0x000005000000 : "configs"
0x000005000000-0x000005200000 : "sig"
0x000005200000-0x000006000000 : "reserve1"
0x000006000000-0x000007000000 : "upgrade-ramfs"
0x000007000000-0x00000a800000 : "upgrade-file"
0x00000a800000-0x000010000000 : "reserve2"
nf_conntrack version 0.5.0 (3635 buckets, 14540 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 10
ip6_tables: (C) 2000-2006 Netfilter Core Team
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
can: controller area network core (rev 20120528 abi 9)
NET: Registered protocol family 29
can: raw protocol (rev 20120528)
can: broadcast manager protocol (rev 20120528 t)
can: netlink gateway (rev 20130117) max_hops=1
zynq_pm_ioremap: no compatible node found for 'xlnx,zynq-ddrc-a05'
zynq_pm_late_init: Unable to map DDRC IO memory.
Registering SWP/SWPB emulation handler
hctosys: unable to open rtc device (rtc0)
ALSA device list:
  No soundcards found.
RAMDISK: gzip image found at block 0
EXT4-fs (ram0): couldn't mount as ext3 due to feature incompatibilities
EXT4-fs warning (device ram0): ext4_update_dynamic_rev:746: updating to rev 1 because of new feature flag, running e2fsck is recommended
EXT4-fs (ram0): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 1024K (c0a00000 - c0b00000)
EXT4-fs (ram0): re-mounted. Opts: block_validity,delalloc,barrier,user_xattr,errors=remount-ro
devpts: called with bogus options
ubi0: attaching mtd2
ubi0: scanning is finished
ubi0: attached mtd2 (name "configs", size 8 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 64, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 6/2, WL threshold: 4096, image sequence number: 262741923
ubi0: available PEBs: 36, total reserved PEBs: 28, PEBs reserved for bad PEB handling: 4
ubi0: background thread "ubi_bgt0d" started, PID 729
UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 733
UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "configs"
UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi0:0): FS size: 1396736 bytes (1 MiB, 11 LEBs), journal size 888833 bytes (0 MiB, 5 LEBs)
UBIFS (ubi0:0): reserved for root: 65970 bytes (64 KiB)
UBIFS (ubi0:0): media format: w4/r0 (latest is w4/r0), UUID A9E8BDA4-70DE-45D9-83BE-2DD9F129C0C6, small LPT model
ubi2: attaching mtd4
ubi2: scanning is finished
ubi2: attached mtd4 (name "reserve1", size 14 MiB)
ubi2: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi2: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi2: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi2: good PEBs: 112, bad PEBs: 0, corrupted PEBs: 0
ubi2: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi2: max/mean erase counter: 6/2, WL threshold: 4096, image sequence number: 2255068344
ubi2: available PEBs: 0, total reserved PEBs: 112, PEBs reserved for bad PEB handling: 4
ubi2: background thread "ubi_bgt2d" started, PID 740
UBIFS (ubi2:0): background thread "ubifs_bgt2_0" started, PID 744
UBIFS (ubi2:0): UBIFS: mounted UBI device 2, volume 0, name "misc"
UBIFS (ubi2:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi2:0): FS size: 11935744 bytes (11 MiB, 94 LEBs), journal size 1015809 bytes (0 MiB, 6 LEBs)
UBIFS (ubi2:0): reserved for root: 563754 bytes (550 KiB)
UBIFS (ubi2:0): media format: w4/r0 (latest is w4/r0), UUID 0EC0A221-F712-4D31-9EF9-BCF170A12764, small LPT model
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
random: avahi-daemon urandom read with 2 bits of entropy available
macb e000b000.ethernet eth0: unable to generate target frequency: 25000000 Hz
macb e000b000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
In axi fpga driver!
request_mem_region OK!
AXI fpga dev virtual address is 0xcfbc8000
*base_vir_addr = 0xb031
In fpga mem driver!
request_mem_region OK!
fpga mem virtual address is 0xd2000000
random: nonblocking pool is initialized


===========================================Miner log===========================================
1970-01-01 00:00:10 Miner compile time: Mon Oct 26 17:55:33 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:11 This is fix-freq version
1970-01-01 00:00:11 Miner compile time: Mon Oct 26 17:55:33 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:11 commit version: 62e4736 2020-09-21 11:17:56, build by: jenkins 2020-10-26 18:06:03
1970-01-01 00:00:11 opt_multi_version     = 1
1970-01-01 00:00:11 opt_bitmain_ab        = 1
1970-01-01 00:00:11 mid_auto_gen          = 0
1970-01-01 00:00:11 opt_bitmain_work_mode = 0
1970-01-01 00:00:11 mmap fpga_mem_addr_hal = 0xb5900000
1970-01-01 00:00:11 HASH_ON_PLUG V9 = 0x7
1970-01-01 00:00:11 Note: front fan is power on!
1970-01-01 00:00:11 Note: rear fan is power on!
1970-01-01 00:00:11 start the http log.
1970-01-01 00:00:11 httpListenThread start ret=0
1970-01-01 00:00:11 start listen on 6060 ...
1970-01-01 00:00:11 load machine NBP1901 conf
1970-01-01 00:00:11 machine : NBP1901 
1970-01-01 00:00:11 chain_num 4, chain_domain_num 38, chain_asic_num 114, domain_asic_num 3
2021-01-19 20:55:40 miner ID : 814254205710481c
2021-01-19 20:55:40 FPGA Version = 0xB031
2021-01-19 20:55:40 HASH_ON_PLUG V9 = 0x7
2021-01-19 20:55:40 ==========================capability start==========================
2021-01-19 20:55:40 board num = 3
2021-01-19 20:55:40 board id = 0, chain num = 1
2021-01-19 20:55:40 	chain id = 0
2021-01-19 20:55:40 board id = 1, chain num = 1
2021-01-19 20:55:40 	chain id = 1
2021-01-19 20:55:40 board id = 2, chain num = 1
2021-01-19 20:55:40 	chain id = 2
2021-01-19 20:55:40 ==========================capability end============================
2021-01-19 20:55:40 chain num = 3
2021-01-19 20:55:42 [chain 0]
2021-01-19 20:55:42 0x0000 11 42 3D C8 D9 C8 26 0A   9E 12 7A E9 BB EA D3 A4 
2021-01-19 20:55:42 0x0010 97 77 F0 AE C9 85 B0 EA   CE 56 A3 EE 0C 9A 28 78 
2021-01-19 20:55:42 0x0020 94 F8 8C 41 A5 BA F9 91   FB 7A 25 AE AF 3E 70 81 
2021-01-19 20:55:42 0x0030 E1 FA 69 61 96 60 E9 A8   E7 8B 9D 38 6A 50 59 B8 
2021-01-19 20:55:42 0x0040 31 F1 FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:42 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-01-19 20:55:42 
2021-01-19 20:55:44 [chain 1]
2021-01-19 20:55:44 0x0000 11 42 1B 38 9B 6C 44 C6   A6 7C F2 03 3B C3 4F 49 
2021-01-19 20:55:44 0x0010 99 A1 D0 04 7C F4 C8 F2   C5 04 B7 79 D6 1F 74 CA 
2021-01-19 20:55:44 0x0020 20 3E 65 D4 E3 ED A9 1D   84 79 D8 60 00 86 E2 9B 
2021-01-19 20:55:44 0x0030 7F B5 52 4C 5C 80 41 A8   E4 72 E6 6A 66 99 F3 70 
2021-01-19 20:55:44 0x0040 02 3B FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:44 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-01-19 20:55:44 
2021-01-19 20:55:45 [chain 2]
2021-01-19 20:55:45 0x0000 11 42 A4 20 E1 58 B0 FB   AB AD E8 B6 F7 23 3A A0 
2021-01-19 20:55:45 0x0010 4B C0 CC 1F 88 C7 F0 B5   45 AB 6E EE E0 55 02 DA 
2021-01-19 20:55:45 0x0020 88 D0 CE 01 95 E2 4D F4   78 5D 68 A8 54 4D F4 61 
2021-01-19 20:55:45 0x0030 E4 87 B8 25 FB E0 2E 34   ED B5 3A A5 F9 A2 3D BC 
2021-01-19 20:55:45 0x0040 E4 DD FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-01-19 20:55:45 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-01-19 20:55:45 
2021-01-19 20:55:45 fan_eft : 1  fan_pwm : 97
2021-01-19 20:55:45 create thread
2021-01-19 20:55:45 fixed working voltage = 1260
2021-01-19 20:55:45 Chain [0] PCB Version: 0x0100
2021-01-19 20:55:45 Chain [0] BOM Version: 0x0100
2021-01-19 20:55:45 Chain [1] PCB Version: 0x0100
2021-01-19 20:55:45 Chain [1] BOM Version: 0x0100
2021-01-19 20:55:45 Chain [2] PCB Version: 0x0100
2021-01-19 20:55:45 Chain [2] BOM Version: 0x0100
2021-01-19 20:55:49 Fan check passed.
2021-01-19 20:55:50 chain[0] PIC jump to app
2021-01-19 20:55:52 Check chain[0] PIC fw version=0x89
2021-01-19 20:55:53 chain[1] PIC jump to app
2021-01-19 20:55:54 Check chain[1] PIC fw version=0x89
2021-01-19 20:55:56 chain[2] PIC jump to app
2021-01-19 20:55:57 Check chain[2] PIC fw version=0x89
2021-01-19 20:55:57 create thread
2021-01-19 20:55:57 max sensor num = 4
2021-01-19 20:55:57 temperature_monitor_thread start...
2021-01-19 20:56:02 power type version: 0x0071
2021-01-19 20:56:02 Enter sleep to make sure power release finish.
2021-01-19 20:56:45 Slept 42 seconds, diff = 5.
2021-01-19 20:56:45 set_voltage_by_steps to 1500.
2021-01-19 20:56:48 start up min temp by 75a = 25
2021-01-19 20:56:48 set UART baud to 115200
2021-01-19 20:56:56 Chain[0]: find 114 asic, times 0
2021-01-19 20:57:04 Chain[1]: find 114 asic, times 0
2021-01-19 20:57:12 Chain[2]: find 114 asic, times 0
2021-01-19 20:57:12 pulse_mode = 1, ccdly_sel = 1, pwth_sel = 1
2021-01-19 20:57:16 min freq in eeprom = 525
2021-01-19 20:57:16 fixed frequency is 525
2021-01-19 20:57:16 Bring up temperature is 25
2021-01-19 20:57:16 set UART baud to 12000000
2021-01-19 20:57:17 set_voltage_by_steps to 1260.
2021-01-19 20:57:24 STATUS_INITED: soc init done!
2021-01-19 20:57:24 create thread
2021-01-19 20:57:24 create thread
2021-01-19 20:57:25 fan_etf: Set fixed fan speed=97
2021-01-19 20:57:26 start to init...
2021-01-19 20:57:35 Init done!
2021-01-19 20:57:35 STATUS_OKAY
2021-01-19 20:57:35 start the cached job
2021-01-19 20:57:35 Version num 8
2021-01-19 20:57:35 Mask num 0xe000
2021-01-19 20:57:35 freq = 525, percent = 90, hcn = 12480, timeout = 449
2021-01-19 20:57:35 set_start_time_point total_tv_start_sys=133 total_tv_end_sys=134

[philipma1957]

so when fans are 97% what rpm do they show?

and what temps do they show?

my lone s19pro is set to 70% fan

fans read

4080
3980
4440
4440

temps read

51.45------72.71
51.45------73.70
51.48------75.72

[mikeywith]

I have 250 units. About 8 of these randomly turn off for no reason at all. T

What do these 8 miners have in common? are they on the same breaker? switch? rack? when more than 1 miner acts the same it's usually an outside cause.

here are no errors, temps are fine, I tried different psu, different control board, fans are at 97%. If you know what the reason is please let me know. Thank you.

Miners turn-off or reboot for many reasons, heat is the most common, if the miner's temps are fine it could be PSU, if the PSU is dirty or isn't getting enough air flow it i will stop, also if the voltage goes out of the 200-240v range it can trigger protection mode.

The good news is here

Code:

2021-01-19 20:56:56 Chain[0]: find 114 asic, times 0
2021-01-19 20:57:04 Chain[1]: find 114 asic, times 0
2021-01-19 20:57:12 Chain[2]: find 114 asic, times 0

All three hash boards are in good shape, finding the complete asic count in the first trial is a good sign.

Since you have 250 of these, I don't think I should be telling you this, but you sure your router/switch can handle all the traffic? that could also be the reason why.

[philipma1957]

Good point he may not have a good enough set of switches and or router.

[CryptoLLC]

Miners are very cold.
43 - 64
41 - 58
40 - 59

PCB Temps Are from low 35 to high 59
Chip Temps Are from 38 to high 64
These are the min and max for all chips on all 3 boards. Average is way below the high I listed above.

Fan Speeds do show running at 97%
Speeds Are: 5040 5040 6000 6000


My router is set to be able to handle 750+ ip address. So thats not the issue. But it could be the voltage drop or voltage increase from the panel. So what happens is that these miners randomly turn off and yes they are all located in one area. Even though some miners in that same area do not have these issues. It might be the panels voltage changing for some reason. I will look into that. As for the other possibilities. All these miners are brand new and have brand new PSU. I even tried replacing 2 psu but still same problem. I tried new control board but still same problem. The really weird part is when the heat outside get to 50 degrees then this happens. Even though the miners are super cold. It might be something with the panel. The Electricians could have made a mistake somewhere.


It cant be the router. On top of these s19 pro I also have 220 units of S17 running on the same network. So there are over 500 units on the network. I increase the network to be able to handle 750+ ip address. It goes from 10.1.10.1 - 10.1.12.255. Unless I did something wrong but I can connect to these bad miners no problem. Always works. Also why would they shut down if there is a network problem? They would show network error. But I dont see any errors at all.

[mikeywith]

Just because you changed your network settings to class A, it doesn't mean the infrastructure can handle all the traffic, this why some routers/switches cost 10x the price of another router/switch, but well since the miners completely shut down, I would rule out the network thing altogether.

I would like to hear more explanation on the "shut-down / turn off f" part, does the PSU itself shut down and its fan stop spinning or does the PSU shut down its main rail and keep the 12v running so the miner's fan and control board work but the hashboards don't?

Since you mentioned that you changed the control board and PSU, what I would do is move one of the bad miners to another rack, just to ensure that this isn't an electricity-related issue, if it is not, then chances are one or some of these boards have a loose heatsink which loses contact with the chip when the mine gets warm, this is very common with the 17 series and could be with the 19 series too since they gears are new, we don't have much info about them, to troubleshoot that you will need to run the miners with 1 board at the time to figure out which board/boards are causing the problem.

[CryptoLLC]

We have a business modem / router from comcast. It should handle all these miners just fine. Also we have good upload and download speed. So what happens --> at random times the miners turn off but to be more exact they look like they restart and dont start mining after restarting. Or they just stop mining. The kernel is showing that they are most likely restarting. Here is some kernel info from a miner that has been running for 7 hours:

2021-01-19 23:48:58 freq = 525, percent = 90, hcn = 12480, timeout = 449
2021-01-19 23:48:58 set_start_time_point total_tv_start_sys=121 total_tv_end_sys=122
2021-01-19 23:48:58 set_voltage_by_steps to 1307.
2021-01-19 23:54:31 set_voltage_by_steps to 1287

You can see that its doing a == set_voltage_by_steps function -- it auto regulates the voltage as time goes on. The miners that have the restart issue are not showing multiple entrys for set_voltage_by_steps after they stop mining. That gives me the impression that the miners restarted since there is only one entry from the bad miner and once it restarted its not mining anymore. The problem could be the heat sinks but it seems that bitmain acknowledged this problem with the s17 series and supposedly fixed the issue on the S19. So i would not put much thought into the heat sink problem. I will try to diagnose the panel for these miners. I can also try moving a bad miner to a different location and see if the problem goes away. If anyone has any other thoughts let me know. This problem is really hard to figure out since there is ZERO kernel info.

[mikeywith]

The problem could be the heat sinks but it seems that bitmain acknowledged this problem with the s17 series and supposedly fixed the issue on the S19.

I do agree, after a second thought, I just remembered that the S19 have large heatsinks that cover more than one chip and it's tightened with some screws, so it's unlikely. but still, one bad chip can cause a mess in these hash boards, so I would still do the board by board troubleshooting if they miner still acts weird even after you move it to a new location.

[CryptoLLC]

Here is another miner. It just now stopped mining. Kernel Log is similar to all the other miners with the same problem.

Code:

2021-01-19 23:52:22 freq = 525, percent = 90, hcn = 12480, timeout = 449
2021-01-19 23:52:22 set_start_time_point total_tv_start_sys=270 total_tv_end_sys=271
2021-01-19 23:52:22 set_voltage_by_steps to 1292.
2021-01-19 23:53:47 set_voltage_by_steps to 1272.
2021-01-20 20:18:42 set_voltage_by_steps to 1260.
2021-01-20 21:45:40 set_voltage_by_steps to 1282.
2021-01-20 21:53:27 avg rate is lower than ideal rate, 72720.92 in 30 mins
2021-01-20 21:53:27 chain 0 hash rate 11401.00 low in 15 mins
2021-01-20 21:53:27 chain 1 hash rate 10699.00 low in 15 mins
2021-01-20 21:53:27 chain 2 hash rate 11628.00 low in 15 mins
2021-01-20 22:08:28 chain 0 hash rate 0.00 low in 15 mins
2021-01-20 22:08:28 chain 1 hash rate 0.00 low in 15 mins
2021-01-20 22:08:28 chain 2 hash rate 0.00 low in 15 mins
2021-01-20 22:23:29 avg rate is lower than ideal rate, 0.00 in 30 mins

You can see that the hash rate slowly came down to zero with no reason. No errors, temps are good. Worked fine for 22 hours. Only happens when outside temps go to 50f degrees.  Any other tips from anyone please leave them. I will inspect the copper wires for any loose connection in the outlet and the breaker.

[mikeywith]

You can see that the hash rate slowly came down to zero with no reason. No errors, temps are good. Worked fine for 22 hours. Only happens when outside temps go to 50f degrees.  Any other tips from anyone please leave them. I will inspect the copper wires for any loose connection in the outlet and the breaker.

Some miners don't like to be run in cold weather, I have seen many issues like this where people had to point the hot exhausted air of some miner to another miner that doesn't like "too low temps", your miner's fans are at 97% and seems like you have that on purpose, could you drop it to 50% or just auto and see how it goes? I mean you would know better how to troubleshoot the low temps but just know it's not uncommon for some gears to stop hashing when they get too cold.

[CryptoLLC]

I dont think they have a cold problem. They run just fine at night when its even colder. They only have a problem at daytime and when temps outside go above 50f.

[philipma1957]

[...]

So as I asked you here. what do your temps read at 97%

my room-is over 90f my fans are slower then yours .

please look at temps of the rigs.

[CryptoLLC]

I posted my temp above. But here they are again:

Miners are very cold inlet and outlet:

43 - 64
41 - 58
40 - 59

PCB Temps Are from low 35 to high 59. Chip Temps Are from 38 to high 64. These are the min and max for all chips on all 3 boards. Average is way below the high I listed above.

Fan Speeds do show running at 97%. Speeds Are: 5040 5040 6000 6000.

[philipma1957]

Set fans to 80% and give back temps.

those cold chips could be the issue.

[CryptoLLC]

I dont think it is cold temp. I have 240 miners right next to these ones that are as cold or even colder. None of those have these issues. Its about 8 miners out of 257 miners that have the issue and all 8 are located in the same area and on the same panel.

Update:

I have tightened all the bolts in the panel. The bolts holding the breaker to the panel, the cooper wire going into the breaker and the cooper wire going into the outlet. All have been tightened but the problem is still present. Today the outside temps have reached 50f and inside the container its also 50f. Inlet and out let are below 75 for all these miners. But they still stop mining. They are breaking after mining for 20-30 minutes.

[mikeywith]

I have tightened all the bolts in the panel. The bolts holding the breaker to the panel, the cooper wire going into the breaker and the cooper wire going into the outlet. All have been tightened but the problem is still present. Today the outside temps have reached 50f and inside the container its also 50f. Inlet and out let are below 75 for all these miners. But they still stop mining. They are breaking after mining for 20-30 minutes.

The reason why I asked if these "bad" miners were located in the same area is simply to help you troubleshoot further, now that you answered, it's actually good news, at least we know there is a good chance that the problem isn't the miners themselves, to confirm this you have to take 1-2 two miners and put them elsewhere, see if they act any differently.

You should also try to reduce the fan speed on some of these miners to see if they would run any longer/better, if so happen that this rack directly faces the intake fans or has a small difference in reference to the airflow they can still can weird,  I would also check the PSU screws and make sure all hash boards are tightened, I would connect the bad miners to a different switch using different cables, I would also use different power cords.

Also in regards to checking the breaker and the wires, you need to check the voltage right before the miners stop hashing, if the wire is damaged it can cause a voltage drop, and these gears don't do well with low voltage, so what could be happening is that after loading the wire for 20-30 mins it gets warm, voltage drops, the PSU goes into protection mode and stops feeding the hash boards.

[CryptoLLC]

I was thinking that maybe the psu is overheating? It feel warm but not hot. The very back of the psu next to the miners exhaust fans is probably hot? The idea I have is that maybe they are getting to warm and shutting down? I have a half inch space between the miners to allow some warm air to come back into the conex. Its only half and inch and seems like nothing. Could this be causing the problems? Maybe these psu's are sensitive to heat and shut down? I have set up a test scenario with some of my miners. I moved some to a different location and I made some have zero space between them. Next time temps are 50+ I will know if these miners have the same problem. If anyone know anything about the S19 psu shutting down because of heat. Please leave some info. Thanks.

[mikeywith]

If anyone know anything about the S19 psu shutting down because of heat. Please leave some info. Thanks.

Of course!. Nearly all PSUs have temp protection and they will shut-down to protect themselves, a very common issue that miners face on a daily bases is their PSUs shutting down due to heat mostly caused by dust and/or not enough space between the PSU's and the wall/other miners,

The very back of the psu next to the miners exhaust fans is probably hot

What's behind the PSUs exhaust is what matters most, but still a bit of spacing between the miners will help cool down the PSUs a bit.

[CryptoLLC]

I have some more info. So I have replaced the psu and i have made no space between all miners. So there is absolutely zero chance that the psu are overheating or that the boards / control boards are over heating. I even made a video in real time for one of my miners breaking / stops mining. The issue has to be either bad hash board(s) or maybe it could be volt fluctuation. Im thinking its the bad boards. Or maybe it could be one bad board that causes the whole miner to shut down. Can you guys watch the video please and let me know what you think?

https://www.youtube.com/watch?v=Ip-y7uKqFgc

[mikeywith]

I saw the video, basically nothing new to what you mentioned here, you perfectly explained the problem in this topic; however, you were supposed to report back the results of moving the miners to another location, which should be fairly easy to do and it isn't just a random thing I asked you to try, it's the backbone of this troubleshooting process, unplug one of the miners, take it somewhere far away, put it outside of the farm if you would, and see if behaves any differently.

The result of this test will eliminate many doubts and it will narrow down the problem to only a few possibilities.