S15s Keep Shutting Down - 20 out of 50

[ELECTRONEUM_TRADER]

We have 50 S15s mining and while a lot of them run without issue and achieve 28TH/s, we have about 20 that keep shutting down. Often times about 10-15 will turn off at the exact same time or within minutes of each other (or just 2-5 at a time). Sometimes they run fine for 18+ hours before dropping offline, other times they will get their TH up briefly and then crash. I've posted the kernel log from one of them below - anyone have any ideas or advice on what the cause may be? It is odd that many of them go offline at the same time - we are leaning towards a facility issue rather than a miner issue, as they were functioning without issue before we moved them to a new place.

Code:

 Booting Linux on physical CPU 0x0 
Linux version 4.6.0-xilinx-gff8137b-dirty (lzq@armdev2) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-23) ) #25 SMP PREEMPT Fri Nov 23 15:30:52 CST 2018 
CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d 
CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache 
Machine model: Xilinx Zynq 
cma: Reserved 16 MiB at 0x0e000000 
Memory policy: Data cache writealloc 
On node 0 totalpages: 61440 
free_area_init_node: node 0, pgdat c0b39280, node_mem_map cde10000 
  Normal zone: 480 pages used for memmap 
  Normal zone: 0 pages reserved 
  Normal zone: 61440 pages, LIFO batch:15 
percpu: Embedded 12 pages/cpu @cddf1000 s19776 r8192 d21184 u49152 
pcpu-alloc: s19776 r8192 d21184 u49152 alloc=12*4096 
pcpu-alloc: [0] 0 [0] 1  
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 60960 
Kernel command line: mem=240M console=ttyPS0,115200 ramdisk_size=33554432 root=/dev/ram rw earlyprintk 
PID hash table entries: 1024 (order: 0, 4096 bytes) 
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes) 
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes) 
Memory: 196092K/245760K available (6345K kernel code, 231K rwdata, 1896K rodata, 1024K init, 223K bss, 33284K reserved, 16384K cma-reserved, 0K highmem) 
Virtual kernel memory layout: 
    vector  : 0xffff0000 - 0xffff1000   (   4 kB) 
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB) 
    vmalloc : 0xcf800000 - 0xff800000   ( 768 MB) 
    lowmem  : 0xc0000000 - 0xcf000000   ( 240 MB) 
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB) 
    modules : 0xbf000000 - 0xbfe00000   (  14 MB) 
      .text : 0xc0008000 - 0xc090c424   (9234 kB) 
      .init : 0xc0a00000 - 0xc0b00000   (1024 kB) 
      .data : 0xc0b00000 - 0xc0b39fe0   ( 232 kB) 
       .bss : 0xc0b39fe0 - 0xc0b71c28   ( 224 kB) 
Preemptible hierarchical RCU implementation. 
 Build-time adjustment of leaf fanout to 32. 
 RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2. 
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2 
NR_IRQS:16 nr_irqs:16 16 
efuse mapped to cf800000 
ps7-slcr mapped to cf802000 
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000 
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000 
L2C-310 erratum 769419 enabled 
L2C-310 enabling early BRESP for Cortex-A9 
L2C-310 full line of zeros enabled for Cortex-A9 
L2C-310 ID prefetch enabled, offset 1 lines 
L2C-310 dynamic clock gating enabled, standby mode enabled 
L2C-310 cache controller enabled, 8 ways, 512 kB 
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001 
zynq_clock_init: clkc starts at cf802100 
Zynq clock init 
sched_clock: 64 bits at 333MHz, resolution 3ns, wraps every 4398046511103ns 
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x4ce07af025, max_idle_ns: 440795209040 ns 
Switching to timer-based delay loop, resolution 3ns 
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 537538477 ns 
ps7-ttc #0 at cf80a000, irq=18 
Console: colour dummy device 80x30 
Calibrating delay loop (skipped), value calculated using timer frequency.. 666.66 BogoMIPS (lpj=3333333) 
pid_max: default: 32768 minimum: 301 
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes) 
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes) 
CPU: Testing write buffer coherency: ok 
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 
Setting up static identity map for 0x100000 - 0x100058 
CPU1: failed to boot: -1 
Brought up 1 CPUs 
SMP: Total of 1 processors activated (666.66 BogoMIPS). 
CPU: All CPU(s) started in SVC mode. 
devtmpfs: initialized 
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4 
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns 
pinctrl core: initialized pinctrl subsystem 
NET: Registered protocol family 16 
DMA: preallocated 256 KiB pool for atomic coherent allocations 
cpuidle: using governor menu 
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint r
 
egisters. 
hw-breakpoint: maximum watchpoint size is 4 bytes. 
zynq-ocm f800c000.ps7-ocmc: ZYNQ OCM pool: 256 KiB @ 0xcf880000 
vgaarb: loaded 
SCSI subsystem initialized 
usbcore: registered new interface driver usbfs 
usbcore: registered new interface driver hub 
usbcore: registered new device driver usb 
media: Linux media interface: v0.10 
Linux video capture interface: v2.00 
pps_core: LinuxPPS API ver. 1 registered 
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> 
PTP clock support registered 
EDAC MC: Ver: 3.0.0 
Advanced Linux Sound Architecture Driver Initialized. 
clocksource: Switched to clocksource arm_global_timer 
NET: Registered protocol family 2 
TCP established hash table entries: 2048 (order: 1, 8192 bytes) 
TCP bind hash table entries: 2048 (order: 2, 16384 bytes) 
TCP: Hash tables configured (established 2048 bind 2048) 
UDP hash table entries: 256 (order: 1, 8192 bytes) 
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes) 
NET: Registered protocol family 1 
RPC: Registered named UNIX socket transport module. 
RPC: Registered udp transport module. 
RPC: Registered tcp transport module. 
RPC: Registered tcp NFSv4.1 backchannel transport module. 
PCI: CLS 0 bytes, default 64 
Trying to unpack rootfs image as initramfs... 
rootfs image is not initramfs (no cpio magic); looks like an initrd 
Freeing initrd memory: 20244K (cc73c000 - cdb01000) 
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available 
futex hash table entries: 512 (order: 3, 32768 bytes) 
workingset: timestamp_bits=28 max_order=16 bucket_order=0 
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc. 
io scheduler noop registered 
io scheduler deadline registered 
io scheduler cfq registered (default) 
dma-pl330 f8003000.ps7-dma: Loaded driver for PL330 DMAC-241330 
dma-pl330 f8003000.ps7-dma:  DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16 
e0000000.serial: ttyPS0 at MMIO 0xe0000000 (irq = 158, base_baud = 6249999) is a xuartps 
console [ttyPS0] enabled 
xdevcfg f8007000.ps7-dev-cfg: ioremap 0xf8007000 to cf86e000 
[drm] Initialized drm 1.1.0 20060810 
brd: module loaded 
loop: module loaded 
CAN device driver interface 
gpiod_set_value: invalid GPIO 
libphy: MACB_mii_bus: probed 
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 31 (00:0a:35:00:00:00) 
Generic PHY e000b000.etherne:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=e000b000.etherne:00, irq=-1) 
e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k 
e1000e: Copyright(c) 1999 - 2015 Intel Corporation. 
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver 
ehci-pci: EHCI PCI platform driver 
usbcore: registered new interface driver usb-storage 
mousedev: PS/2 mouse device common for all mice 
i2c /dev entries driver 
Xilinx Zynq CpuIdle Driver started 
sdhci: Secure Digital Host Controller Interface driver 
sdhci: Copyright(c) Pierre Ossman 
sdhci-pltfm: SDHCI platform and OF driver helper 
mmc0: SDHCI controller on e0100000.ps7-sdio [e0100000.ps7-sdio] using ADMA 
ledtrig-cpu: registered to indicate activity on CPUs 
usbcore: registered new interface driver usbhid 
usbhid: USB HID core driver 
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xda 
nand: Micron MT29F2G08ABAEAWP 
nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64 
nand: WARNING: pl35x-nand: the ECC used on your system is too weak compared to the one required by the NAND chip 
Bad block table found at page 131008, version 0x01 
Bad block table found at page 130944, version 0x01 
nand_read_bbt: bad block at 0x000002420000 
nand_read_bbt: bad block at 0x000002440000 
nand_read_bbt: bad block at 0x000002460000 
6 ofpart partitions found on MTD device pl35x-nand 
Creating 6 MTD partitions on "pl35x-nand": 
0x000000000000-0x000002800000 : "BOOT.bin-env-dts-kernel" 
0x000002800000-0x000004800000 : "ramfs" 
0x000004800000-0x000005000000 : "configs" 
0x000005000000-0x000006000000 : "reserve" 
0x000006000000-0x000008000000 : "ramfs-bak" 
0x000008000000-0x000010000000 : "reserve1"
 
NET:
 
Registered protocol family 10 
sit: IPv6 over IPv4 tunneling driver 
NET: Registered protocol family 17 
can: controller area network core (rev 20120528 abi 9) 
NET: Registered protocol family 29 
can: raw protocol (rev 20120528) 
can: broadcast manager protocol (rev 20120528 t) 
can: netlink gateway (rev 20130117) max_hops=1 
zynq_pm_ioremap: no compatible node found for 'xlnx,zynq-ddrc-a05' 
zynq_pm_late_init: Unable to map DDRC IO memory. 
Registering SWP/SWPB emulation handler 
hctosys: unable to open rtc device (rtc0) 
ALSA device list: 
  No soundcards found. 
RAMDISK: gzip image found at block 0 
EXT4-fs (ram0): couldn't mount as ext3 due to feature incompatibilities 
EXT4-fs (ram0): mounted filesystem without journal. Opts: (null) 
VFS: Mounted root (ext4 filesystem) on device 1:0. 
devtmpfs: mounted 
Freeing unused kernel memory: 1024K (c0a00000 - c0b00000) 
EXT4-fs (ram0): re-mounted. Opts: block_validity,delalloc,barrier,user_xattr 
random: dd urandom read with 0 bits of entropy available 
ubi0: attaching mtd2 
ubi0: scanning is finished 
ubi0: attached mtd2 (name "configs", size 8 MiB) 
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes 
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 
ubi0: good PEBs: 64, bad PEBs: 0, corrupted PEBs: 0 
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128 
ubi0: max/mean erase counter: 29/11, WL threshold: 4096, image sequence number: 1987444180 
ubi0: available PEBs: 0, total reserved PEBs: 64, PEBs reserved for bad PEB handling: 40 
ubi0: background thread "ubi_bgt0d" started, PID 708 
UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 711 
UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "configs" 
UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes 
UBIFS (ubi0:0): FS size: 1396736 bytes (1 MiB, 11 LEBs), journal size 888833 bytes (0 MiB, 5 LEBs) 
UBIFS (ubi0:0): reserved for root: 65970 bytes (64 KiB) 
UBIFS (ubi0:0): media format: w4/r0 (latest is w4/r0), UUID 84D384B4-1101-4684-B2B6-E00856C325A1, small LPT model 
ubi1: attaching mtd5 
ubi1: scanning is finished 
ubi1: attached mtd5 (name "reserve1", size 128 MiB) 
ubi1: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes 
ubi1: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 
ubi1: VID header offset: 2048 (aligned 2048), data offset: 4096 
ubi1: good PEBs: 1020, bad PEBs: 4, corrupted PEBs: 0 
ubi1: user volume: 1, internal volumes: 1, max. volumes count: 128 
ubi1: max/mean erase counter: 1534/284, WL threshold: 4096, image sequence number: 1727257993 
ubi1: available PEBs: 0, total reserved PEBs: 1020, PEBs reserved for bad PEB handling: 36 
ubi1: background thread "ubi_bgt1d" started, PID 720 
UBIFS (ubi1:0): background thread "ubifs_bgt1_0" started, PID 723 
UBIFS (ubi1:0): UBIFS: mounted UBI device 1, volume 0, name "reserve1" 
UBIFS (ubi1:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes 
UBIFS (ubi1:0): FS size: 123039744 bytes (117 MiB, 969 LEBs), journal size 6221824 bytes (5 MiB, 49 LEBs) 
UBIFS (ubi1:0): reserved for root: 4952683 bytes (4836 KiB) 
UBIFS (ubi1:0): media format: w4/r0 (latest is w4/r0), UUID CDB6D62F-8A73-4CB2-B1A2-C11F56004B34, small LPT model 
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready 
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready 
macb e000b000.ethernet eth0: unable to generate target frequency: 25000000 Hz 
macb e000b000.ethernet eth0: link up (100/Full) 
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready 
In axi fpga driver! 
request_mem_region OK! 
AXI fpga dev virtual address is 0xcfb38000 
*base_vir_addr = 0xab01b 
In fpga mem driver! 
request_mem_region OK! 
fpga mem virtual address is 0xd2000000 
random: nonblocking pool is initialized 
 [2021-01-29 00:20:58.115] start listen on 6060 ... 
 [2021-01-29 00:26:31.264] set_start_time_point total_tv_start_sys=428 total_tv_end_sys=429
 
2021-01-29 00:20:28 driver-btm-soc.c:7273:init_freq_mode: This
 
is scan-user version 
2021-01-29 00:20:28 driver-btm-soc.c:8125:bitmain_soc_init: S15 
2021-01-29 00:20:28 driver-btm-soc.c:8126:bitmain_soc_init: opt_multi_version=1 
2021-01-29 00:20:28 driver-btm-soc.c:8127:bitmain_soc_init: opt_bitmain_ab=1 
2021-01-29 00:20:28 driver-btm-soc.c:8128:bitmain_soc_init: opt_bitmain_economic_mode=false 
2021-01-29 00:20:28 driver-btm-soc.c:8129:bitmain_soc_init: opt_bitmain_work_mode_test = 0 
2021-01-29 00:20:28 driver-btm-soc.c:7682:_show_sn: no SN got, please write SN to /nvdata/sn 
2021-01-29 00:20:28 driver-btm-soc.c:7304:init_phy_mem_nonce2_jobid_address: Detect 256MB control board of XILINX 
2021-01-29 00:20:28 driver-btm-soc.c:8149:bitmain_soc_init: Miner compile time: Fri Dec 13 15:35:07 CST 2019 type: Antminer S15 
2021-01-29 00:20:28 driver-btm-soc.c:8151:bitmain_soc_init: last commit version: 53156cb commit time: 2019-11-06 10:21:42 build: 2019-12-13 15:40:49 
2021-01-29 00:20:28 driver-btm-soc.c:7336:check_fan_multi: check fan for 20 times... 
2021-01-29 00:20:30 driver-btm-soc.c:8214:bitmain_soc_init: miner ID : 80446d0618c08854 
2021-01-29 00:20:30 driver-btm-soc.c:7410:init_miner_version: FPGA Version = 0xB01B 
2021-01-29 00:20:30 driver-btm-soc.c:1092:check_chain: get_hash_on_plug is 0xf 
2021-01-29 00:20:30 driver-btm-soc.c:1104:check_chain: !! chain[0]is exist 
2021-01-29 00:20:30 driver-btm-soc.c:1104:check_chain: !! chain[1]is exist 
2021-01-29 00:20:30 driver-btm-soc.c:1104:check_chain: !! chain[2]is exist 
2021-01-29 00:20:30 driver-btm-soc.c:1104:check_chain: !! chain[3]is exist 
2021-01-29 00:20:40 eeprom.c:1418:eeprom_load: done 
2021-01-29 00:20:40 driver-btm-soc.c:7431:init_pcb_version: chain[0] PCB Version : 0x0100 
2021-01-29 00:20:40 driver-btm-soc.c:7449:init_pcb_version: chain[0] BOM Version : 0x0103 
2021-01-29 00:20:40 driver-btm-soc.c:7431:init_pcb_version: chain[1] PCB Version : 0x0100 
2021-01-29 00:20:40 driver-btm-soc.c:7449:init_pcb_version: chain[1] BOM Version : 0x0103 
2021-01-29 00:20:40 driver-btm-soc.c:7431:init_pcb_version: chain[2] PCB Version : 0x0100 
2021-01-29 00:20:40 driver-btm-soc.c:7449:init_pcb_version: chain[2] BOM Version : 0x0103 
2021-01-29 00:20:40 driver-btm-soc.c:7431:init_pcb_version: chain[3] PCB Version : 0x0100 
2021-01-29 00:20:40 driver-btm-soc.c:7449:init_pcb_version: chain[3] BOM Version : 0x0103 
2021-01-29 00:20:41 dspic33ep16gs202.c:538:reset_pic: chain[0] ok 
2021-01-29 00:20:45 dspic33ep16gs202.c:504:jump_to_app_from_loader: chain[0] ok 
2021-01-29 00:20:46 dspic33ep16gs202.c:538:reset_pic: chain[1] ok 
2021-01-29 00:20:50 dspic33ep16gs202.c:504:jump_to_app_from_loader: chain[1] ok 
2021-01-29 00:20:51 dspic33ep16gs202.c:538:reset_pic: chain[2] ok 
2021-01-29 00:20:55 dspic33ep16gs202.c:504:jump_to_app_from_loader: chain[2] ok 
2021-01-29 00:20:56 dspic33ep16gs202.c:538:reset_pic: chain[3] ok 
2021-01-29 00:21:00 dspic33ep16gs202.c:504:jump_to_app_from_loader: chain[3] ok 
2021-01-29 00:21:00 power.c:137:power_init: power init ...  
2021-01-29 00:21:00 driver-btm-soc.c:3130:heart_beat_func_one_chain: for chain 3 
2021-01-29 00:21:00 driver-btm-soc.c:3130:heart_beat_func_one_chain: for chain 2 
2021-01-29 00:21:00 driver-btm-soc.c:3130:heart_beat_func_one_chain: for chain 1 
2021-01-29 00:21:00 driver-btm-soc.c:3130:heart_beat_func_one_chain: for chain 0 
2021-01-29 00:21:00 power.c:114:power_on: init gpio907 
2021-01-29 00:21:03 dspic33ep16gs202.c:409:hashboard_power_cntl: [0] ok, i2c=0! 
2021-01-29 00:21:04 dspic33ep16gs202.c:409:hashboard_power_cntl: [0] ok, i2c=1! 
2021-01-29 00:21:04 dspic33ep16gs202.c:409:hashboard_power_cntl: [0] ok, i2c=2! 
2021-01-29 00:21:05 dspic33ep16gs202.c:409:hashboard_power_cntl: [0] ok, i2c=3! 
2021-01-29 00:21:08 driver-btm-soc.c:8241:bitmain_soc_init: Enter 30s sleep to make sure power release finish. 
2021-01-29 00:21:38 driver-btm-soc.c:7626:set_working_voltage_by_eeprom: eeprom voltage[0] = 18.000000 
2021-01-29 00:21:38 driver-btm-soc.c:7626:set_working_voltage_by_eeprom: eeprom voltage[1] = 18.000000
 
2021-01-29 00:21:38 driver-btm-soc.c:7626:set_working_voltage_by_eeprom
 
040]:385 IC[041]:385  
IC[042]:385 IC[043]:385 IC[044]:400 IC[045]:400 IC[046]:400 IC[047]:400  
IC[048]:400 IC[049]:400 IC[050]:400 IC[051]:400 IC[052]:385 IC[053]:385  
IC[054]:400 IC[055]:385 IC[056]:400 IC[057]:400 IC[058]:400 IC[059]:385  
IC[060]:400 IC[061]:400 IC[062]:400 IC[063]:400 IC[064]:385 IC[065]:385  
IC[066]:400 IC[067]:400 IC[068]:400 IC[069]:400 IC[070]:385 IC[071]:385  
voltage in mode2: 18.00 V 
hash rate in mode2: 07241 GH/s 
 
2021-01-29 00:21:39 dspic33ep16gs202.c:409:hashboard_power_cntl: [1] ok, i2c=0! 
2021-01-29 00:21:39 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000ffff 
2021-01-29 00:21:42 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffe 
2021-01-29 00:21:45 driver-btm-soc.c:5456:check_asic_num_by_pre_open_core: Try to find asic by pre core: times 0 
2021-01-29 00:21:45 driver-btm-soc.c:5412:check_asic_num_without_power_off: Open 16 core for chain 0 
2021-01-29 00:21:45 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:21:46 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:21:46 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:21:46 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:21:46 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:21:47 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:21:47 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:21:48 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:21:48 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000ffff 
2021-01-29 00:21:51 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffe 
2021-01-29 00:21:54 driver-btm-soc.c:5426:check_asic_num_without_power_off: chain 0, find asic num = 72 
2021-01-29 00:21:55 dspic33ep16gs202.c:409:hashboard_power_cntl: [1] ok, i2c=1! 
2021-01-29 00:21:55 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffe 
2021-01-29 00:21:58 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffc 
2021-01-29 00:22:01 driver-btm-soc.c:5456:check_asic_num_by_pre_open_core: Try to find asic by pre core: times 0 
2021-01-29 00:22:01 driver-btm-soc.c:5412:check_asic_num_without_power_off: Open 16 core for chain 1 
2021-01-29 00:22:01 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:02 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:22:02 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:03 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:22:03 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:04 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:22:04 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:04 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:22:04 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffe 
2021-01-29 00:22:07 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffc 
2021-01-29 00:22:11 driver-btm-soc.c:5426:check_asic_num_without_power_off: chain 1, find asic num = 72 
2021-01-29 00:22:11 dspic33ep16gs202.c:409:hashboard_power_cntl: [1] ok, i2c=2! 
2021-01-29 00:22:11 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffc 
2021-01-29 00:22:14 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:22:17 driver-btm-soc.c:5456:check_asic_num_by_pre_open_core: Try to find asic by pre core: times 0 
2021-01-29 00:22:17 driver-btm-soc.c:5412:check_asic_num_without_power_off: Open 16 core for chain 2 
2021-01-29 00:22:17 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:18 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2
 
2021-01-29 00:22:18 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:19 driver-btm-soc.c:3967:pre_open_core_one_chain: for
 
IC[038]:375 IC[039]:390 IC[040]:390 IC[041]:390  
IC[042]:390 IC[043]:375 IC[044]:390 IC[045]:390 IC[046]:390 IC[047]:390  
IC[048]:390 IC[049]:390 IC[050]:390 IC[051]:390 IC[052]:375 IC[053]:390  
IC[054]:375 IC[055]:390 IC[056]:390 IC[057]:390 IC[058]:390 IC[059]:390  
IC[060]:390 IC[061]:390 IC[062]:390 IC[063]:390 IC[064]:390 IC[065]:375  
IC[066]:390 IC[067]:390 IC[068]:390 IC[069]:390 IC[070]:390 IC[071]:375  
voltage in mode2: 18.00 V 
hash rate in mode2: 07090 GH/s 
 
2021-01-29 00:21:38 eeprom.c:840:travel_eeprom_store: chain[2] data in eeprom is: 
magic number: 0x7f 
freq data per ASIC: 
IC[000]:245 IC[001]:245 IC[002]:245 IC[003]:245 IC[004]:235 IC[005]:235  
IC[006]:235 IC[007]:245 IC[008]:245 IC[009]:245 IC[010]:235 IC[011]:245  
IC[012]:245 IC[013]:245 IC[014]:245 IC[015]:245 IC[016]:235 IC[017]:235  
IC[018]:235 IC[019]:235 IC[020]:245 IC[021]:245 IC[022]:245 IC[023]:245  
IC[024]:245 IC[025]:245 IC[026]:245 IC[027]:245 IC[028]:235 IC[029]:235  
IC[030]:235 IC[031]:235 IC[032]:245 IC[033]:245 IC[034]:245 IC[035]:245  
IC[036]:245 IC[037]:245 IC[038]:245 IC[039]:245 IC[040]:235 IC[041]:235  
IC[042]:235 IC[043]:245 IC[044]:235 IC[045]:245 IC[046]:245 IC[047]:245  
IC[048]:245 IC[049]:245 IC[050]:245 IC[051]:235 IC[052]:245 IC[053]:235  
IC[054]:235 IC[055]:235 IC[056]:245 IC[057]:245 IC[058]:245 IC[059]:245  
IC[060]:245 IC[061]:245 IC[062]:235 IC[063]:245 IC[064]:245 IC[065]:235  
IC[066]:235 IC[067]:235 IC[068]:245 IC[069]:245 IC[070]:245 IC[071]:245  
voltage: 16.30 V 
hash rate: 04415 GH/s 
 
freq data per ASIC in mode2: 
IC[000]:360 IC[001]:360 IC[002]:370 IC[003]:370 IC[004]:380 IC[005]:380  
IC[006]:380 IC[007]:380 IC[008]:370 IC[009]:370 IC[010]:360 IC[011]:360  
IC[012]:360 IC[013]:360 IC[014]:370 IC[015]:370 IC[016]:380 IC[017]:380  
IC[018]:380 IC[019]:380 IC[020]:370 IC[021]:370 IC[022]:360 IC[023]:360  
IC[024]:360 IC[025]:360 IC[026]:370 IC[027]:370 IC[028]:380 IC[029]:380  
IC[030]:380 IC[031]:380 IC[032]:370 IC[033]:370 IC[034]:360 IC[035]:360  
IC[036]:360 IC[037]:360 IC[038]:370 IC[039]:370 IC[040]:380 IC[041]:380  
IC[042]:380 IC[043]:380 IC[044]:370 IC[045]:370 IC[046]:360 IC[047]:360  
IC[048]:360 IC[049]:360 IC[050]:370 IC[051]:370 IC[052]:380 IC[053]:380  
IC[054]:380 IC[055]:380 IC[056]:370 IC[057]:370 IC[058]:360 IC[059]:360  
IC[060]:360 IC[061]:360 IC[062]:370 IC[063]:370 IC[064]:380 IC[065]:380  
IC[066]:380 IC[067]:380 IC[068]:370 IC[069]:370 IC[070]:360 IC[071]:360  
voltage in mode2: 18.00 V 
hash rate in mode2: 06790 GH/s 
 
2021-01-29 00:21:38 eeprom.c:840:travel_eeprom_store: chain[3] data in eeprom is: 
magic number: 0x7f 
freq data per ASIC: 
IC[000]:235 IC[001]:235 IC[002]:235 IC[003]:235 IC[004]:235 IC[005]:225  
IC[006]:235 IC[007]:235 IC[008]:235 IC[009]:235 IC[010]:235 IC[011]:225  
IC[012]:235 IC[013]:235 IC[014]:235 IC[015]:235 IC[016]:225 IC[017]:235  
IC[018]:235 IC[019]:235 IC[020]:235 IC[021]:225 IC[022]:235 IC[023]:235  
IC[024]:235 IC[025]:235 IC[026]:235 IC[027]:235 IC[028]:225 IC[029]:235  
IC[030]:235 IC[031]:225 IC[032]:235 IC[033]:235 IC[034]:235 IC[035]:235  
IC[036]:235 IC[037]:235 IC[038]:235 IC[039]:235 IC[040]:235 IC[041]:225  
IC[042]:235 IC[043]:225 IC[044]:235 IC[045]:235 IC[046]:235 IC[047]:235  
IC[048]:235 IC[049]:235 IC[050]:235 IC[051]:235 IC[052]:225 IC[053]:235  
IC[054]:235 IC[055]:235 IC[056]:235 IC[057]:235 IC[058]:235 IC[059]:225  
IC[060]:235 IC[061]:235 IC[062]:235 IC[063]:225 IC[064]:235 IC[065]:235  
IC[066]:235 IC[067]:235 IC[068]:235 IC[069]:225 IC[070]:235 IC[071]:235  
voltage: 16.30 V 
hash rate: 04279 GH/s 
 
freq data per ASIC in mode2: 
IC[000]:400 IC[001]:400 IC[002]:400 IC[003]:400 IC[004]:385 IC[005]:385  
IC[006]:385 IC[007]:400 IC[008]:400 IC[009]:400 IC[010]:400 IC[011]:385  
IC[012]:400 IC[013]:400 IC[014]:400 IC[015]:400 IC[016]:385 IC[017]:385  
IC[018]:385 IC[019]:400 IC[020]:400 IC[021]:385 IC[022]:400 IC[023]:400
 
IC[024]:400 IC[025]:400 IC[026]:400 IC[027]:400 IC[028]:385 IC[029]:385  
IC[030]:385 IC[031]:385 IC[032]:400 IC[033]:400 IC[034]:400 IC[035]:400  
IC[036]:400 IC[037]:400 IC[038]:400 IC[039]:400 IC[
 
: eeprom voltage[2] = 18.000000 
2021-01-29 00:21:38 driver-btm-soc.c:7626:set_working_voltage_by_eeprom: eeprom voltage[3] = 18.000000 
2021-01-29 00:21:38 driver-btm-soc.c:7630:set_working_voltage_by_eeprom: max = 18.00, min = 18.00 
2021-01-29 00:21:38 power.c:149:set_working_voltage: working_voltage = 18.000000 
2021-01-29 00:21:38 power.c:274:set_iic_power_to_highest_voltage: setting to voltage: 20.00 ... 
2021-01-29 00:21:38 power.c:227:set_iic_power_by_voltage: now setting voltage to : 20.000000  
2021-01-29 00:21:38 eeprom.c:840:travel_eeprom_store: chain[0] data in eeprom is: 
magic number: 0x7f 
freq data per ASIC: 
IC[000]:235 IC[001]:235 IC[002]:235 IC[003]:235 IC[004]:225 IC[005]:225  
IC[006]:225 IC[007]:235 IC[008]:225 IC[009]:235 IC[010]:235 IC[011]:235  
IC[012]:235 IC[013]:235 IC[014]:235 IC[015]:235 IC[016]:225 IC[017]:225  
IC[018]:225 IC[019]:225 IC[020]:235 IC[021]:235 IC[022]:235 IC[023]:235  
IC[024]:235 IC[025]:235 IC[026]:235 IC[027]:225 IC[028]:235 IC[029]:225  
IC[030]:225 IC[031]:225 IC[032]:235 IC[033]:235 IC[034]:235 IC[035]:235  
IC[036]:235 IC[037]:235 IC[038]:235 IC[039]:225 IC[040]:225 IC[041]:235  
IC[042]:235 IC[043]:225 IC[044]:235 IC[045]:235 IC[046]:225 IC[047]:235  
IC[048]:235 IC[049]:235 IC[050]:225 IC[051]:235 IC[052]:225 IC[053]:235  
IC[054]:225 IC[055]:225 IC[056]:235 IC[057]:235 IC[058]:235 IC[059]:235  
IC[060]:235 IC[061]:235 IC[062]:235 IC[063]:235 IC[064]:225 IC[065]:225  
IC[066]:225 IC[067]:225 IC[068]:235 IC[069]:235 IC[070]:235 IC[071]:235  
voltage: 16.30 V 
hash rate: 04247 GH/s 
 
freq data per ASIC in mode2: 
IC[000]:395 IC[001]:395 IC[002]:395 IC[003]:395 IC[004]:385 IC[005]:375  
IC[006]:385 IC[007]:395 IC[008]:375 IC[009]:395 IC[010]:395 IC[011]:395  
IC[012]:395 IC[013]:395 IC[014]:395 IC[015]:395 IC[016]:385 IC[017]:375  
IC[018]:375 IC[019]:385 IC[020]:395 IC[021]:395 IC[022]:395 IC[023]:395  
IC[024]:395 IC[025]:395 IC[026]:395 IC[027]:385 IC[028]:395 IC[029]:375  
IC[030]:375 IC[031]:385 IC[032]:395 IC[033]:395 IC[034]:395 IC[035]:395  
IC[036]:395 IC[037]:395 IC[038]:395 IC[039]:395 IC[040]:375 IC[041]:385  
IC[042]:395 IC[043]:375 IC[044]:395 IC[045]:395 IC[046]:385 IC[047]:395  
IC[048]:395 IC[049]:395 IC[050]:375 IC[051]:385 IC[052]:395 IC[053]:395  
IC[054]:375 IC[055]:385 IC[056]:395 IC[057]:395 IC[058]:395 IC[059]:395  
IC[060]:395 IC[061]:395 IC[062]:395 IC[063]:395 IC[064]:385 IC[065]:375  
IC[066]:375 IC[067]:395 IC[068]:395 IC[069]:395 IC[070]:395 IC[071]:385  
voltage in mode2: 18.00 V 
hash rate in mode2: 07154 GH/s 
 
2021-01-29 00:21:38 eeprom.c:840:travel_eeprom_store: chain[1] data in eeprom is: 
magic number: 0x7f 
freq data per ASIC: 
IC[000]:235 IC[001]:235 IC[002]:235 IC[003]:225 IC[004]:235 IC[005]:225  
IC[006]:225 IC[007]:225 IC[008]:235 IC[009]:235 IC[010]:235 IC[011]:235  
IC[012]:235 IC[013]:235 IC[014]:235 IC[015]:225 IC[016]:235 IC[017]:225  
IC[018]:235 IC[019]:225 IC[020]:235 IC[021]:235 IC[022]:235 IC[023]:225  
IC[024]:235 IC[025]:235 IC[026]:225 IC[027]:225 IC[028]:235 IC[029]:235  
IC[030]:225 IC[031]:235 IC[032]:235 IC[033]:225 IC[034]:235 IC[035]:235  
IC[036]:235 IC[037]:235 IC[038]:225 IC[039]:235 IC[040]:235 IC[041]:225  
IC[042]:235 IC[043]:225 IC[044]:235 IC[045]:225 IC[046]:235 IC[047]:235  
IC[048]:235 IC[049]:235 IC[050]:235 IC[051]:235 IC[052]:225 IC[053]:225  
IC[054]:225 IC[055]:235 IC[056]:225 IC[057]:235 IC[058]:235 IC[059]:235  
IC[060]:235 IC[061]:225 IC[062]:235 IC[063]:235 IC[064]:225 IC[065]:235  
IC[066]:235 IC[067]:235 IC[068]:235 IC[069]:225 IC[070]:235 IC[071]:225  
voltage: 16.30 V 
hash rate: 04248 GH/s 
 
freq data per ASIC in mode2: 
IC[000]:390 IC[001]:390 IC[002]:390 IC[003]:390 IC[004]:390 IC[005]:375  
IC[006]:375 IC[007]:390 IC[008]:390 IC[009]:390 IC[010]:390 IC[011]:390  
IC[012]:390 IC[013]:390 IC[014]:390 IC[015]:390 IC[016]:390 IC[017]:375
 
IC[018]:390 IC[019]:375 IC[020]:390 IC[021]:390 IC[022]:390 IC[023]:390  
IC[024]:390 IC[025]:390 IC[026]:375 IC[027]:390 IC[028]:390 IC[029]:390  
IC[030]:390 IC[031]:390 IC[032]:390 IC[033]:375 IC[034]:390 IC[035]:390  
IC[036]:390 IC[037]:390
 
chain 2 
2021-01-29 00:22:19 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:19 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:22:19 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:20 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:22:20 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fffc 
2021-01-29 00:22:23 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:22:26 driver-btm-soc.c:5426:check_asic_num_without_power_off: chain 2, find asic num = 72 
2021-01-29 00:22:27 dspic33ep16gs202.c:409:hashboard_power_cntl: [1] ok, i2c=3! 
2021-01-29 00:22:27 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:22:30 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:22:33 driver-btm-soc.c:5456:check_asic_num_by_pre_open_core: Try to find asic by pre core: times 0 
2021-01-29 00:22:33 driver-btm-soc.c:5412:check_asic_num_without_power_off: Open 16 core for chain 3 
2021-01-29 00:22:33 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:34 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:22:34 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:35 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:22:35 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:35 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:22:35 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:36 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:22:36 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:22:39 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:22:42 driver-btm-soc.c:5426:check_asic_num_without_power_off: chain 3, find asic num = 72 
2021-01-29 00:22:42 driver-btm-soc.c:7238:balance_domain_voltage: To balance voltage domain, open all core and reset hash board after find asic num. 
2021-01-29 00:22:42 driver-btm-soc.c:7249:balance_domain_voltage: Open core 16 for chain 0 
2021-01-29 00:22:42 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:43 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:22:43 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff1 
2021-01-29 00:22:46 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:22:49 driver-btm-soc.c:7249:balance_domain_voltage: Open core 16 for chain 1 
2021-01-29 00:22:49 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:50 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:22:50 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff2 
2021-01-29 00:22:53 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:22:56 driver-btm-soc.c:7249:balance_domain_voltage: Open core 16 for chain 2 
2021-01-29 00:22:56 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:22:57 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:22:57 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff4 
2021-01-29 00:23:00 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:03 driver-btm-soc.c:7249:balance_domain_voltage: Open core 16 for chain 3 
2021-01-29 00:23:03 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:23:03 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:23:03 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8
 
2021-01-29 00:23:06 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:09 driver-btm-soc.c:7249:balance_domain_voltage: Open core 32 for chain 0 
2021-01-29 00:23:09 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre ope
 
:25:35 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 275.00M 
2021-01-29 00:25:36 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 300.00M 
2021-01-29 00:25:37 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 325.00M 
2021-01-29 00:25:38 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 350.00M 
2021-01-29 00:25:39 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 375.00M 
2021-01-29 00:25:44 driver-btm-soc.c:6651:increase_freq_by_eeprom_slowly: Increase eeprom frequency slowly for chain 1. to 375 
2021-01-29 00:25:44 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 125.00M 
2021-01-29 00:25:45 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 150.00M 
2021-01-29 00:25:46 driver-btm-soc.c:496
 
t_clock_delay_control: core_data = 0x04 
2021-01-29 00:24:34 driver-btm-soc.c:8293:bitmain_soc_init: Open 1/4 core for reading temperature. 
2021-01-29 00:24:34 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 16 
2021-01-29 00:24:35 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:24:35 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:24:36 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:24:37 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:24:37 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:37 temperature.c:422:calibration_sensor_offset: chain[0] temp info,Bad. Reload from eeprom.. 
sensor type = 0,sensor num = 4 
sensor pos: 18  13  54  49  0  0  0  0  
sensor offset: 0  0  0  0  0  0  0  0  
2021-01-29 00:24:37 temperature.c:448:calibration_sensor_offset: chain[0] temp chip I2C addr=0x98 
2021-01-29 00:24:37 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:37 temperature.c:531:calibration_sensor_offset: Warning: Chain[0] has no temp info ! we will fix it 
2021-01-29 00:24:37 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41093318, cmd_buf[1]=0x6131, cmd_buf[2]=0x13000000, misc=0x6131 
2021-01-29 00:24:37 temperature.c:553:calibration_sensor_offset: Chain[0] use default temp chip[17]  
2021-01-29 00:24:38 temperature.c:562:calibration_sensor_offset: Chain[0] Chip[17] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:38 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41092418, cmd_buf[1]=0x6131, cmd_buf[2]=0x9000000, misc=0x6131 
2021-01-29 00:24:38 temperature.c:553:calibration_sensor_offset: Chain[0] use default temp chip[12]  
2021-01-29 00:24:39 temperature.c:562:calibration_sensor_offset: Chain[0] Chip[12] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:39 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099f18, cmd_buf[1]=0x6131, cmd_buf[2]=0x5000000, misc=0x6131 
2021-01-29 00:24:39 temperature.c:553:calibration_sensor_offset: Chain[0] use default temp chip[53]  
2021-01-29 00:24:40 temperature.c:562:calibration_sensor_offset: Chain[0] Chip[53] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:40 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099018, cmd_buf[1]=0x6131, cmd_buf[2]=0xf000000, misc=0x6131 
2021-01-29 00:24:40 temperature.c:553:calibration_sensor_offset: Chain[0] use default temp chip[48]  
2021-01-29 00:24:41 temperature.c:562:calibration_sensor_offset: Chain[0] Chip[48] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:41 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:41 temperature.c:422:calibration_sensor_offset: chain[1] temp info,Bad. Reload from eeprom.. 
sensor type = 0,sensor num = 4 
sensor pos: 18  13  54  49  0  0  0  0  
sensor offset: 0  0  0  0  0  0  0  0
 
2021-01-29 00:24:41 temperature.c:448:calibration_sensor_offset: chain[1] temp chip I2C addr=0x98 
2021-01-29 00:24:41 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:41 temperature.c:531:calibration_sensor_offset: Warning: Chain[1] has no temp info ! we will fix it 
2021-01-29 00:24:41 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41093318, cmd_buf[1]=0x6131, cmd_buf[2]=0x13000000, misc=0x6131 
2021-01-29 00:24:41 temperature.c:553:calibration_sensor_offset: Chain[1] use default temp chip[17]  
2021-01-29 00:24:42 temperature.c:562:calibration_sensor_offset: Chain[1] Chip[17] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:42 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41092418, cmd_buf[1]=0x6131, cmd_buf[2]=0x9000000, misc=0x6131 
2021-01-29 00:24:42 temperature.c:553:calibration_sensor_offset: Chain[1] use default temp chip[12]  
2021-01-29 00:24:43 temperature.c:562:calibration_sensor_offset: Chain[1] Chip[12] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:43 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099f18, cmd_buf[1]=0x6131, cmd_buf[2]=0x5000000, misc=0x6131 
2021-01-29 00:24:43 temperature.c:553:calibration_sensor_off
 
n core = 32 
2021-01-29 00:23:11 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:23:11 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff1 
2021-01-29 00:23:14 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:17 driver-btm-soc.c:7249:balance_domain_voltage: Open core 32 for chain 1 
2021-01-29 00:23:17 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 32 
2021-01-29 00:23:18 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:23:18 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff2 
2021-01-29 00:23:21 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:24 driver-btm-soc.c:7249:balance_domain_voltage: Open core 32 for chain 2 
2021-01-29 00:23:24 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 32 
2021-01-29 00:23:26 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:23:26 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff4 
2021-01-29 00:23:29 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:32 driver-btm-soc.c:7249:balance_domain_voltage: Open core 32 for chain 3 
2021-01-29 00:23:32 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 32 
2021-01-29 00:23:33 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:23:33 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:23:36 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:39 driver-btm-soc.c:7249:balance_domain_voltage: Open core 64 for chain 0 
2021-01-29 00:23:39 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 64 
2021-01-29 00:23:42 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 0 
2021-01-29 00:23:42 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff1 
2021-01-29 00:23:45 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:48 driver-btm-soc.c:7249:balance_domain_voltage: Open core 64 for chain 1 
2021-01-29 00:23:48 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 64 
2021-01-29 00:23:51 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 1 
2021-01-29 00:23:51 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff2 
2021-01-29 00:23:54 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:23:57 driver-btm-soc.c:7249:balance_domain_voltage: Open core 64 for chain 2 
2021-01-29 00:23:57 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 64 
2021-01-29 00:24:00 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 2 
2021-01-29 00:24:00 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff4 
2021-01-29 00:24:03 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0
 
2021-01-29 00:24:06 driver-btm-soc.c:7249:balance_domain_voltage: Open core 64 for chain 3 
2021-01-29 00:24:06 driver-btm-soc.c:3992:open_core_BM1391_pre_open: pre open core = 64 
2021-01-29 00:24:09 driver-btm-soc.c:3967:pre_open_core_one_chain: for chain 3 
2021-01-29 00:24:09 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff8 
2021-01-29 00:24:12 znyq.c:193:set_reset_hashboard: set_reset_hashboard = 0x0000fff0 
2021-01-29 00:24:34 driver-btm-soc.c:7475:calculate_timeout: dev->timeout = 642 
2021-01-29 00:24:34 driver-btm-soc.c:3012:set_baud: set baud=1 
2021-01-29 00:24:34 chip1391.c:336:set_baud_one_chain: cmd_buf[0]=0x51090018, cmd_buf[1]=0x2101, cmd_buf[2]=0x11000000, misc=0x2101 
2021-01-29 00:24:34 chip1391.c:336:set_baud_one_chain: cmd_buf[0]=0x51090018, cmd_buf[1]=0x2101, cmd_buf[2]=0x11000000, misc=0x2101 
2021-01-29 00:24:34 chip1391.c:336:set_baud_one_chain: cmd_buf[0]=0x51090018, cmd_buf[1]=0x2101, cmd_buf[2]=0x11000000, misc=0x2101 
2021-01-29 00:24:34 chip1391.c:336:set_baud_one_chain: cmd_buf[0]=0x51090018, cmd_buf[1]=0x2101, cmd_buf[2]=0x11000000, misc=0x2101 
2021-01-29 00:24:34 driver-btm-soc.c:3023:set_baud: read bc write is 0x83001a 
2021-01-29 00:24:34 driver-btm-soc.c:4076:se
 
53]  
2021-01-29 00:24:51 temperature.c:562:calibration_sensor_offset: Chain[3] Chip[53] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:51 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099018, cmd_buf[1]=0x6131, cmd_buf[2]=0xf000000, misc=0x6131 
2021-01-29 00:24:52 temperature.c:553:calibration_sensor_offset: Chain[3] use default temp chip[48]  
2021-01-29 00:24:52 temperature.c:562:calibration_sensor_offset: Chain[3] Chip[48] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:56 temperature.c:589:do_calibration_temperature_sensor: bring up lowest temperature is 16 
ENV 18 FAN 5220 
2021-01-29 00:25:06 driver-btm-soc.c:8298:bitmain_soc_init: Boot environment temperature is 18 
2021-01-29 00:25:06 driver-btm-soc.c:8308:bitmain_soc_init: set TICKET_MASK 
2021-01-29 00:25:06 driver-btm-soc.c:4130:open_core_bm1391: Start Open Core!! 
2021-01-29 00:25:17 driver-btm-soc.c:4202:open_core_bm1391: End Open Core!! 
2021-01-29 00:25:17 freq_tuning.c:312:freq_tuning_get_max_freq: Max freq of tuning is 425 
ENV 18 FAN 5220 
2021-01-29 00:25:17 power.c:379:slowly_set_iic_power_to_custom_voltage: slowly setting to voltage: 19.00 ... 
2021-01-29 00:25:17 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[27] = 19.837553 
2021-01-29 00:25:17 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 20.000000  
2021-01-29 00:25:18 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[37] = 19.670696 
2021-01-29 00:25:18 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.837553  
2021-01-29 00:25:19 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[47] = 19.503839 
2021-01-29 00:25:19 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.670696  
2021-01-29 00:25:20 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[57] = 19.336982 
2021-01-29 00:25:20 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.503839  
2021-01-29 00:25:21 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[67] = 19.170124 
2021-01-29 00:25:21 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.336982  
2021-01-29 00:25:22 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[69] = 19.136753 
2021-01-29 00:25:22 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.170124  
2021-01-29 00:25:23 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[71] = 19.103382 
2021-01-29 00:25:23 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.136753  
2021-01-29 00:25:24 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[73] = 19.070010 
2021-01-29 00:25:24 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.103382
 
2021-01-29 00:25:25 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[75] = 19.036639 
2021-01-29 00:25:25 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.070010  
2021-01-29 00:25:26 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[76] = 19.019953 
2021-01-29 00:25:27 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.036639  
2021-01-29 00:25:27 power.c:210:get_power_voltage_from_iic_value: iic_index for voltage[77] = 19.003267 
2021-01-29 00:25:28 power.c:244:set_iic_power_by_iic_data: now setting voltage to : 19.019953  
2021-01-29 00:25:28 driver-btm-soc.c:6651:increase_freq_by_eeprom_slowly: Increase eeprom frequency slowly for chain 0. to 375 
2021-01-29 00:25:28 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 125.00M 
2021-01-29 00:25:29 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 150.00M 
2021-01-29 00:25:30 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 175.00M 
2021-01-29 00:25:31 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 200.00M 
2021-01-29 00:25:33 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 225.00M 
2021-01-29 00:25:34 driver-btm-soc.c:4963:increase_freq_slowly:  
 
Increase frequency to 250.00M 
2021-01-29 00
 
set: Chain[1] use default temp chip[53]  
2021-01-29 00:24:44 temperature.c:562:calibration_sensor_offset: Chain[1] Chip[53] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:44 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099018, cmd_buf[1]=0x6131, cmd_buf[2]=0xf000000, misc=0x6131 
2021-01-29 00:24:44 temperature.c:553:calibration_sensor_offset: Chain[1] use default temp chip[48]  
2021-01-29 00:24:45 temperature.c:562:calibration_sensor_offset: Chain[1] Chip[48] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:45 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:45 temperature.c:422:calibration_sensor_offset: chain[2] temp info,Bad. Reload from eeprom.. 
sensor type = 0,sensor num = 4 
sensor pos: 18  13  54  49  0  0  0  0  
sensor offset: 0  0  0  0  0  0  0  0  
2021-01-29 00:24:45 temperature.c:448:calibration_sensor_offset: chain[2] temp chip I2C addr=0x98 
2021-01-29 00:24:45 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:45 temperature.c:531:calibration_sensor_offset: Warning: Chain[2] has no temp info ! we will fix it 
2021-01-29 00:24:45 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41093318, cmd_buf[1]=0x6131, cmd_buf[2]=0x13000000, misc=0x6131 
2021-01-29 00:24:45 temperature.c:553:calibration_sensor_offset: Chain[2] use default temp chip[17]  
2021-01-29 00:24:46 temperature.c:562:calibration_sensor_offset: Chain[2] Chip[17] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:46 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41092418, cmd_buf[1]=0x6131, cmd_buf[2]=0x9000000, misc=0x6131 
2021-01-29 00:24:46 temperature.c:553:calibration_sensor_offset: Chain[2] use default temp chip[12]  
2021-01-29 00:24:47 temperature.c:562:calibration_sensor_offset: Chain[2] Chip[12] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:47 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099f18, cmd_buf[1]=0x6131, cmd_buf[2]=0x5000000, misc=0x6131 
2021-01-29 00:24:47 temperature.c:553:calibration_sensor_offset: Chain[2] use default temp chip[53]  
2021-01-29 00:24:48 temperature.c:562:calibration_sensor_offset: Chain[2] Chip[53] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:48 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099018, cmd_buf[1]=0x6131, cmd_buf[2]=0xf000000, misc=0x6131 
2021-01-29 00:24:48 temperature.c:553:calibration_sensor_offset: Chain[2] use default temp chip[48]  
2021-01-29 00:24:49 temperature.c:562:calibration_sensor_offset: Chain[2] Chip[48] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:49 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:49 temperature.c:422:calibration_sensor_offset: chain[3] temp info,Bad. Reload from eeprom..
 
sensor type = 0,sensor num = 4 
sensor pos: 18  13  54  49  0  0  0  0  
sensor offset: 0  0  0  0  0  0  0  0  
2021-01-29 00:24:49 temperature.c:448:calibration_sensor_offset: chain[3] temp chip I2C addr=0x98 
2021-01-29 00:24:49 temperature.c:306:is_tempsensor_data_correct: Bad,temp_info.type = 0 
2021-01-29 00:24:49 temperature.c:531:calibration_sensor_offset: Warning: Chain[3] has no temp info ! we will fix it 
2021-01-29 00:24:49 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41093318, cmd_buf[1]=0x6131, cmd_buf[2]=0x13000000, misc=0x6131 
2021-01-29 00:24:49 temperature.c:553:calibration_sensor_offset: Chain[3] use default temp chip[17]  
2021-01-29 00:24:50 temperature.c:562:calibration_sensor_offset: Chain[3] Chip[17] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:50 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41092418, cmd_buf[1]=0x6131, cmd_buf[2]=0x9000000, misc=0x6131 
2021-01-29 00:24:50 temperature.c:553:calibration_sensor_offset: Chain[3] use default temp chip[12]  
2021-01-29 00:24:50 temperature.c:562:calibration_sensor_offset: Chain[3] Chip[12] get middle temp offset=-5 typeID=55 
2021-01-29 00:24:50 chip1391.c:729:set_misc_control: cmd_buf[0]=0x41099f18, cmd_buf[1]=0x6131, cmd_buf[2]=0x5000000, misc=0x6131

[mikeywith]

Please edit your post, use the code instead of the quote function to post the kernel log, it's so messy this way.

The kernel log doesn't seem complete, you want to copy it when the miner is hashing within those 10-15 mins you talked about, but the fact that many of them go down together the first suspect would be the network infrastructure.

The easiest way to troubleshoot this would by powering off another good working 20 miners and allowing these miners to run alone, this will let you know if your router/switch and everything in between is actually able to run all 50 miners.

The second problem will be the voltage, if your wires aren't well-sized, the voltage can drop, and these PSUs want stable voltage over 200v, you will also troubleshoot it in the same manner but after the network troubleshooting.

The last would be heat issues, which is the least likely since you mentioned they all go down at the same time but will still worth troubleshooting.

And of course, the complete kernel log will be very handy.

[ELECTRONEUM_TRADER]

Post has been edited from quote to code.

I will circle back with the facility and ask them for a complete log. Will post it once I have it.

Thanks for the other tips for testing as well.


Below is an example of 3 that went out at the same time today - the graph hasn't fully caught up yet, but they all drop right offline/to 0 on the far right side of each graph. They also all have the same drop. Yesterday you can also see when they all went offline at the same time around 14:00 - 17:00. Still waiting on confirmation if the kernel logs just randomly end like that, or if the full log was not provided.

https://imgur.com/ReZoyGY
https://imgur.com/QHuC4JW
https://imgur.com/jNMUoPC

Apparently the kernel log just ends like that/cuts off. Any idea what that means? Power issue? Connection issue? Other?

[philipma1957]

Not enough info.

Do you have a low power option?

Some s15 have that.

If you have low power option try that.

[mikeywith]

In my initial reply to your issue, I asked you to troubleshoot

1-Network
2-Voltage
3-Heat

In that exact order, now after seeing your pool graphs I would say troubleshoot the network 5 times before moving on to the other two.

[ELECTRONEUM_TRADER]

Thanks. We have a call setup to discuss these tomorrow. Is the driver behind you suggesting the network related to the graphs being identical/some external factor affecting all of them at once, opposed to it likely being an individual miner issue (i.e. we'd expect relatively unique graphs if it was an issue on a miner by miner basis)?

[mikeywith]

That is pretty much it, 2 of the 3 miners you posted that status for went offline at 8:42 and the third one went offline just a minute later (actually a maximum of 60 seconds, it could be 0.1 seconds), so this can't be a hardware problem, it is very unlikely to be a heat-related problem, so what's left out is network and electricity.

This problem takes 15 minutes to troubleshoot if your guys at the new place can't do it - not sure if they are capable of hosting miners.