S19 Pro Randomly Stops working with no Kernel Errors

[baro0k]

We have a business modem / router from comcast. It should handle all these miners just fine. Also we have good upload and download speed. So what happens --> at random times the miners turn off but to be more exact they look like they restart and dont start mining after restarting. Or they just stop mining. The kernel is showing that they are most likely restarting. Here is some kernel info from a miner that has been running for 7 hours:

2021-01-19 23:48:58 freq = 525, percent = 90, hcn = 12480, timeout = 449
2021-01-19 23:48:58 set_start_time_point total_tv_start_sys=121 total_tv_end_sys=122
2021-01-19 23:48:58 set_voltage_by_steps to 1307.
2021-01-19 23:54:31 set_voltage_by_steps to 1287

You can see that its doing a == set_voltage_by_steps function -- it auto regulates the voltage as time goes on. The miners that have the restart issue are not showing multiple entrys for set_voltage_by_steps after they stop mining. That gives me the impression that the miners restarted since there is only one entry from the bad miner and once it restarted its not mining anymore. The problem could be the heat sinks but it seems that bitmain acknowledged this problem with the s17 series and supposedly fixed the issue on the S19. So i would not put much thought into the heat sink problem. I will try to diagnose the panel for these miners. I can also try moving a bad miner to a different location and see if the problem goes away. If anyone has any other thoughts let me know. This problem is really hard to figure out since there is ZERO kernel info.

I have a similar case

Code:

2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:13
2021-01-22 13:00:17
2021-01-22 13:00:17
2021-01-22 13:00:19
2021-01-22 13:00:20 chain[1] PIC jump to app 2021-01-22 13:00:22 Check chain[1] PIC fw version=0x89 2021-01-22 13:00:23 chain[2] PIC jump to app 2021-01-22 13:00:24 Check chain[2] PIC fw version=0x89 2021-01-22 13:00:25 create thread
2021-01-22 13:00:25 max sensor num = 4
2021-01-22 13:00:25 temperature_monitor_thread start... 2021-01-22 13:00:29 power type version: 0x0071
0x0060FFFFFFFFFFFF 0x0070FFFFFFFFFFFF 0x0080FFFFFFFFFFFF 0x0090FFFFFFFFFFFF 0x00A0FFFFFFFFFFFF 0x00B0FFFFFFFFFFFF 0x00C0FFFFFFFFFFFF 0x00D0FFFFFFFFFFFF 0x00E0FFFFFFFFFFFF 0x00F0FFFFFFFFFFFF
FF FF FF FF FF FF
FF FF FF FF FF FF FF FF FF FF FF FF
FF FF
FF FF FF FF FF FF FF FF FF FF FF FF FF FF FFFFFFFFFFFFFF
FFFFFFFFFFFFFF FFFFFFFFFFFFFF FFFFFFFFFFFFFF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF
FF FF FF FF FF FF FF
fan_eft : 0 fan_pwm : 100 create thread
fixed working voltage = 1260 Chain [0] PCB Version: 0x0100 Chain [0] BOM Version: 0x0100 Chain [1] PCB Version: 0x0100 Chain [1] BOM Version: 0x0100 Chain [2] PCB Version: 0x0100 Chain [2] BOM Version: 0x0100 Fan check passed.
chain[0] PIC jump to app
Check chain[0] PIC fw version=0x89
2021-01-22 13:00:29 Enter sleep to make sure power release finish. 2021-01-22 13:01:00 Slept 30 seconds, diff = 1.
2021-01-22 13:01:00 set_voltage_by_steps to 1500.
2021-01-22 13:01:05 start up min temp by 75a = 24
2021-01-22 13:01:05 set UART baud to 115200
2021-01-22 13:01:10 chain avg vol rise to 15.17
2021-01-22 13:01:13 Chain[0]: find 114 asic, times 0
2021-01-22 13:01:21 Chain[1]: find 114 asic, times 0
2021-01-22 13:01:29 Chain[2]: find 114 asic, times 0
2021-01-22 13:01:29 pulse_mode = 1, ccdly_sel = 1, pwth_sel = 1 2021-01-22 13:01:33 min freq in eeprom = 525
2021-01-22 13:01:33 fixed frequency is 525 2021-01-22 13:01:33 Bring up temperature is 24 2021-01-22 13:01:34 set UART baud to 12000000 2021-01-22 13:01:34 set_voltage_by_steps to 1260. 2021-01-22 13:01:41 STATUS_INITED: soc init done! 2021-01-22 13:01:41 create thread
2021-01-22 13:01:41 create thread
2021-01-22 13:01:43 start to init...
2021-01-22 13:01:52 Init done!
2021-01-22 13:01:52 STATUS_OKAY
2021-01-22 13:01:52 start the cached job
2021-01-22 13:01:52 Version num 8
2021-01-22 13:01:52 Mask num 0xe000
2021-01-22 13:01:52 freq = 525, percent = 90, hcn = 12480, timeout = 449 2021-01-22 13:01:52 set_start_time_point total_tv_start_sys=122 total_tv_end_sys=123
2021-01-22 13:57:32 set_voltage_by_steps to 1240. 2021-01-22 13:57:38 set_voltage_by_steps to 1260. 2021-01-22 13:58:54 set_voltage_by_steps to 1240. 2021-01-22 13:59:34 set_voltage_by_steps to 1260. 2021-01-22 13:59:38 set_voltage_by_steps to 1240. 2021-01-22 13:59:42 set_voltage_by_steps to 1260. 2021-01-22 14:00:54 set_voltage_by_steps to 1240. 2021-01-22 14:01:12 set_voltage_by_steps to 1

can you please explain more?

[CryptoLLC]

Try tightening the bolts on the boards and the psu. Tighten all the bolts that transfer power. Also update the firmware. There is new firmware that came out 2 days ago. Let us know if that fixes the issue for you.

[CryptoLLC]

I moved one of the bad miners to a different outlet and know it breaks even faster. Im getting a little more info in the kernel. Before I would get  zero info but now im getting: (chain avg vol rise) and (chain avg vol drop) and (read asic reg error).

Something is wrong with these miners. Here is the full log for the miner I move to a different outlet. Before these errors would not be in the kernel. It would restart with no error. Now you can see Im getting more info. Any ideas what it could be? Keep in mind that if its the psu that would mean that there are 20+ bad psu out of the 257 miners. Also these miners are brand new and only been mining for 1 month. Psu's are brand new.

Code:

free_area_init_node: node 0, pgdat c0b3c040, node_mem_map cde10000
  Normal zone: 480 pages used for memmap
  Normal zone: 0 pages reserved
  Normal zone: 61440 pages, LIFO batch:15
percpu: Embedded 12 pages/cpu @cddf0000 s19916 r8192 d21044 u49152
pcpu-alloc: s19916 r8192 d21044 u49152 alloc=12*4096
pcpu-alloc: [0] 0 [0] 1 
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 60960
Kernel command line: mem=240M console=ttyPS0,115200 ramdisk_size=33554432 root=/dev/ram rw earlyprintk
PID hash table entries: 1024 (order: 0, 4096 bytes)
Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Memory: 209672K/245760K available (6317K kernel code, 243K rwdata, 1932K rodata, 1024K init, 232K bss, 19704K reserved, 16384K cma-reserved, 0K highmem)
Virtual kernel memory layout:
    vector  : 0xffff0000 - 0xffff1000   (   4 kB)
    fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
    vmalloc : 0xcf800000 - 0xff800000   ( 768 MB)
    lowmem  : 0xc0000000 - 0xcf000000   ( 240 MB)
    pkmap   : 0xbfe00000 - 0xc0000000   (   2 MB)
    modules : 0xbf000000 - 0xbfe00000   (  14 MB)
      .text : 0xc0008000 - 0xc090e410   (9242 kB)
      .init : 0xc0a00000 - 0xc0b00000   (1024 kB)
      .data : 0xc0b00000 - 0xc0b3cda0   ( 244 kB)
       .bss : 0xc0b3cda0 - 0xc0b77024   ( 233 kB)
Preemptible hierarchical RCU implementation.
	Build-time adjustment of leaf fanout to 32.
	RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2.
RCU: Adjusting geometry for rcu_fanout_leaf=32, nr_cpu_ids=2
NR_IRQS:16 nr_irqs:16 16
efuse mapped to cf800000
ps7-slcr mapped to cf802000
L2C: platform modifies aux control register: 0x72360000 -> 0x72760000
L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000
L2C-310 erratum 769419 enabled
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 ID prefetch enabled, offset 1 lines
L2C-310 dynamic clock gating enabled, standby mode enabled
L2C-310 cache controller enabled, 8 ways, 512 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001
zynq_clock_init: clkc starts at cf802100
Zynq clock init
sched_clock: 64 bits at 333MHz, resolution 3ns, wraps every 4398046511103ns
clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x4ce07af025, max_idle_ns: 440795209040 ns
Switching to timer-based delay loop, resolution 3ns
clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 537538477 ns
ps7-ttc #0 at cf80a000, irq=18
Console: colour dummy device 80x30
Calibrating delay loop (skipped), value calculated using timer frequency.. 666.66 BogoMIPS (lpj=3333333)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes)
CPU: Testing write buffer coherency: ok
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x100000 - 0x100058
CPU1: failed to boot: -1
Brought up 1 CPUs
SMP: Total of 1 processors activated (666.66 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
pinctrl core: initialized pinctrl subsystem
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor menu
hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers.
hw-breakpoint: maximum watchpoint size is 4 bytes.
zynq-ocm f800c000.ps7-ocmc: ZYNQ OCM pool: 256 KiB @ 0xcf880000
vgaarb: loaded
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
media: Linux media interface: v0.10
Linux video capture interface: v2.00
pps_core: LinuxPPS API ver. 1 registered
pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
PTP clock support registered
EDAC MC: Ver: 3.0.0
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm_global_timer
NET: Registered protocol family 2
TCP established hash table entries: 2048 (order: 1, 8192 bytes)
TCP bind hash table entries: 2048 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 2048 bind 2048)
UDP hash table entries: 256 (order: 1, 8192 bytes)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
PCI: CLS 0 bytes, default 64
Trying to unpack rootfs image as initramfs...
rootfs image is not initramfs (no cpio magic); looks like an initrd
Freeing initrd memory: 6632K (cd480000 - cdafa000)
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available
futex hash table entries: 512 (order: 3, 32768 bytes)
workingset: timestamp_bits=28 max_order=16 bucket_order=0
jffs2: version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
io scheduler noop registered
io scheduler deadline registered
io scheduler cfq registered (default)
dma-pl330 f8003000.ps7-dma: Loaded driver for PL330 DMAC-241330
dma-pl330 f8003000.ps7-dma: 	DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16
e0000000.serial: ttyPS0 at MMIO 0xe0000000 (irq = 159, base_baud = 6249999) is a xuartps
console [ttyPS0] enabled
xdevcfg f8007000.ps7-dev-cfg: ioremap 0xf8007000 to cf86e000
[drm] Initialized drm 1.1.0 20060810
brd: module loaded
loop: module loaded
CAN device driver interface
gpiod_set_value: invalid GPIO
libphy: MACB_mii_bus: probed
macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 31 (00:0a:35:00:00:00)
Generic PHY e000b000.etherne:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=e000b000.etherne:00, irq=-1)
e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
ehci-pci: EHCI PCI platform driver
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
i2c /dev entries driver
cdns-i2c e0005000.ps7_i2c: 100 kHz mmio e0005000 irq 154
Xilinx Zynq CpuIdle Driver started
sdhci: Secure Digital Host Controller Interface driver
sdhci: Copyright(c) Pierre Ossman
sdhci-pltfm: SDHCI platform and OF driver helper
mmc0: SDHCI controller on e0100000.ps7-sdio [e0100000.ps7-sdio] using ADMA
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
nand: disable subpage write
nand: device found, Manufacturer ID: 0x2c, Chip ID: 0xda
nand: Micron MT29F2G08ABAEAWP
nand: 256 MiB, SLC, erase size: 128 KiB, page size: 2048, OOB size: 64
nand: NAND_ECC_HW
nand: NAND_ECC_HW_SYNDROME
mtd->writesize = 2048
ecc->strength = 1
ecc->size = 2048
mtd->writesize = 2048
chip->ecc_strength_ds = 4
chip->ecc_step_ds = 512
nand: WARNING: pl35x-nand: the ECC used on your system is too weak compared to the one required by the NAND chip
Bad block table found at page 131008, version 0x01
Bad block table found at page 130944, version 0x01
8 ofpart partitions found on MTD device pl35x-nand
Creating 8 MTD partitions on "pl35x-nand":
0x000000000000-0x000002800000 : "BOOT.bin-dts-marker-kernel"
0x000002800000-0x000004800000 : "ramfs"
0x000004800000-0x000005000000 : "configs"
0x000005000000-0x000005200000 : "sig"
0x000005200000-0x000006000000 : "reserve1"
0x000006000000-0x000007000000 : "upgrade-ramfs"
0x000007000000-0x00000a800000 : "upgrade-file"
0x00000a800000-0x000010000000 : "reserve2"
nf_conntrack version 0.5.0 (3635 buckets, 14540 max)
ip_tables: (C) 2000-2006 Netfilter Core Team
NET: Registered protocol family 10
ip6_tables: (C) 2000-2006 Netfilter Core Team
sit: IPv6 over IPv4 tunneling driver
NET: Registered protocol family 17
can: controller area network core (rev 20120528 abi 9)
NET: Registered protocol family 29
can: raw protocol (rev 20120528)
can: broadcast manager protocol (rev 20120528 t)
can: netlink gateway (rev 20130117) max_hops=1
zynq_pm_ioremap: no compatible node found for 'xlnx,zynq-ddrc-a05'
zynq_pm_late_init: Unable to map DDRC IO memory.
Registering SWP/SWPB emulation handler
hctosys: unable to open rtc device (rtc0)
ALSA device list:
  No soundcards found.
RAMDISK: gzip image found at block 0
EXT4-fs (ram0): couldn't mount as ext3 due to feature incompatibilities
EXT4-fs warning (device ram0): ext4_update_dynamic_rev:746: updating to rev 1 because of new feature flag, running e2fsck is recommended
EXT4-fs (ram0): mounted filesystem without journal. Opts: (null)
VFS: Mounted root (ext4 filesystem) on device 1:0.
devtmpfs: mounted
Freeing unused kernel memory: 1024K (c0a00000 - c0b00000)
EXT4-fs (ram0): re-mounted. Opts: block_validity,delalloc,barrier,user_xattr,errors=remount-ro
devpts: called with bogus options
ubi0: attaching mtd2
ubi0: scanning is finished
ubi0: attached mtd2 (name "configs", size 8 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 64, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 4/1, WL threshold: 4096, image sequence number: 262741923
ubi0: available PEBs: 36, total reserved PEBs: 28, PEBs reserved for bad PEB handling: 4
ubi0: background thread "ubi_bgt0d" started, PID 729
UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 733
UBIFS (ubi0:0): recovery needed
UBIFS (ubi0:0): recovery completed
UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "configs"
UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi0:0): FS size: 1396736 bytes (1 MiB, 11 LEBs), journal size 888833 bytes (0 MiB, 5 LEBs)
UBIFS (ubi0:0): reserved for root: 65970 bytes (64 KiB)
UBIFS (ubi0:0): media format: w4/r0 (latest is w4/r0), UUID A9E8BDA4-70DE-45D9-83BE-2DD9F129C0C6, small LPT model
ubi2: attaching mtd4
ubi2: scanning is finished
ubi2: attached mtd4 (name "reserve1", size 14 MiB)
ubi2: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi2: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi2: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi2: good PEBs: 112, bad PEBs: 0, corrupted PEBs: 0
ubi2: user volume: 1, internal volumes: 1, max. volumes count: 128
ubi2: max/mean erase counter: 5/2, WL threshold: 4096, image sequence number: 443719576
ubi2: available PEBs: 0, total reserved PEBs: 112, PEBs reserved for bad PEB handling: 4
ubi2: background thread "ubi_bgt2d" started, PID 740
UBIFS (ubi2:0): background thread "ubifs_bgt2_0" started, PID 744
UBIFS (ubi2:0): recovery needed
UBIFS (ubi2:0): recovery completed
UBIFS (ubi2:0): UBIFS: mounted UBI device 2, volume 0, name "misc"
UBIFS (ubi2:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
UBIFS (ubi2:0): FS size: 11935744 bytes (11 MiB, 94 LEBs), journal size 1015809 bytes (0 MiB, 6 LEBs)
UBIFS (ubi2:0): reserved for root: 563754 bytes (550 KiB)
UBIFS (ubi2:0): media format: w4/r0 (latest is w4/r0), UUID 83280604-31C3-4236-B043-08F4084F4891, small LPT model
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
random: avahi-daemon urandom read with 2 bits of entropy available
macb e000b000.ethernet eth0: unable to generate target frequency: 25000000 Hz
macb e000b000.ethernet eth0: link up (100/Full)
IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
In axi fpga driver!
request_mem_region OK!
AXI fpga dev virtual address is 0xcfbc8000
*base_vir_addr = 0xb031
In fpga mem driver!
request_mem_region OK!
fpga mem virtual address is 0xd2000000
random: nonblocking pool is initialized


===========================================Miner log===========================================
1970-01-01 00:00:09 Miner compile time: Fri Dec 11 11:23:44 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:10 This is fix-freq version
1970-01-01 00:00:10 Miner compile time: Fri Dec 11 11:23:44 CST 2020 type: Antminer S19 Pro
1970-01-01 00:00:10 commit version: 1821c90 2020-11-16 16:05:37, build by: jenkins 2020-12-11 11:35:43
1970-01-01 00:00:10 opt_multi_version     = 1
1970-01-01 00:00:10 opt_bitmain_ab        = 1
1970-01-01 00:00:10 mid_auto_gen          = 0
1970-01-01 00:00:10 opt_bitmain_work_mode = 0
1970-01-01 00:00:10 mmap fpga_mem_addr_hal = 0xb5900000
1970-01-01 00:00:10 HASH_ON_PLUG V9 = 0x7
1970-01-01 00:00:10 Note: front fan is power on!
1970-01-01 00:00:10 Note: rear fan is power on!
1970-01-01 00:00:10 start the http log.
1970-01-01 00:00:10 httpListenThread start ret=0
1970-01-01 00:00:10 start listen on 6060 ...
1970-01-01 00:00:10 load machine NBP1901 conf
1970-01-01 00:00:10 machine : NBP1901 
1970-01-01 00:00:10 chain_num 4, chain_domain_num 38, chain_asic_num 114, domain_asic_num 3
2021-02-06 01:23:05 miner ID : 801265c85710481c
2021-02-06 01:23:05 FPGA Version = 0xB031
2021-02-06 01:23:05 HASH_ON_PLUG V9 = 0x7
2021-02-06 01:23:05 ==========================capability start==========================
2021-02-06 01:23:05 board num = 3
2021-02-06 01:23:05 board id = 0, chain num = 1
2021-02-06 01:23:05 	chain id = 0
2021-02-06 01:23:05 board id = 1, chain num = 1
2021-02-06 01:23:05 	chain id = 1
2021-02-06 01:23:05 board id = 2, chain num = 1
2021-02-06 01:23:05 	chain id = 2
2021-02-06 01:23:05 ==========================capability end============================
2021-02-06 01:23:05 chain num = 3
2021-02-06 01:23:07 [chain 0]
2021-02-06 01:23:07 0x0000 11 42 72 E7 87 CF FD 7F   93 C8 CF A1 C9 26 17 EA 
2021-02-06 01:23:07 0x0010 3E FB 3F 8C 61 DC B4 77   C9 04 15 70 A9 F6 E7 A4 
2021-02-06 01:23:07 0x0020 C0 F2 A1 97 33 5E FD 7F   2E 57 E6 7D 9E 2B FE 39 
2021-02-06 01:23:07 0x0030 D3 0E BE FE 70 E7 7B BD   10 35 D2 05 82 8C 8C 63 
2021-02-06 01:23:07 0x0040 F5 CC FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:07 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-02-06 01:23:07 
2021-02-06 01:23:09 [chain 1]
2021-02-06 01:23:09 0x0000 11 42 D2 04 7D 08 1D 10   6F 71 FE CA A1 80 14 0D 
2021-02-06 01:23:09 0x0010 C8 4A 73 F1 78 EC 4F 1B   8D 8B CF D8 C4 19 E8 1F 
2021-02-06 01:23:09 0x0020 73 A4 F0 AA 52 48 E8 91   36 B8 FA C5 FD DB CC C9 
2021-02-06 01:23:09 0x0030 AC 3A 05 B3 E0 2E 74 4E   E0 FB C7 F0 18 10 4C BE 
2021-02-06 01:23:09 0x0040 69 90 FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:09 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-02-06 01:23:09 
2021-02-06 01:23:10 [chain 2]
2021-02-06 01:23:10 0x0000 11 42 FE 43 A1 7F ED E1   4B F6 00 7C 82 B4 59 14 
2021-02-06 01:23:10 0x0010 36 50 5D 70 F6 5B 9C 33   88 B2 BF E0 2D 85 79 57 
2021-02-06 01:23:10 0x0020 BA 6A 14 67 BC 45 C7 01   6A EB AC 0E B4 39 7E 2F 
2021-02-06 01:23:10 0x0030 75 48 96 9D 3C 74 50 D3   EB 6E DE C1 41 5B 54 9F 
2021-02-06 01:23:10 0x0040 04 B4 FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x0050 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x0060 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x0070 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x0080 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x0090 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00A0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00B0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00C0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00D0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00E0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF FF 
2021-02-06 01:23:10 0x00F0 FF FF FF FF FF FF FF FF   FF FF FF FF FF FF FF 5A 
2021-02-06 01:23:10 
2021-02-06 01:23:10 fan_eft : 1  fan_pwm : 90
2021-02-06 01:23:10 create thread
2021-02-06 01:23:10 fixed working voltage = 1260
2021-02-06 01:23:10 Chain [0] PCB Version: 0x0100
2021-02-06 01:23:10 Chain [0] BOM Version: 0x0100
2021-02-06 01:23:10 Chain [1] PCB Version: 0x0100
2021-02-06 01:23:10 Chain [1] BOM Version: 0x0100
2021-02-06 01:23:10 Chain [2] PCB Version: 0x0100
2021-02-06 01:23:10 Chain [2] BOM Version: 0x0100
2021-02-06 01:23:14 Fan check passed.
2021-02-06 01:23:15 chain[0] PIC jump to app
2021-02-06 01:23:17 Check chain[0] PIC fw version=0x89
2021-02-06 01:23:18 chain[1] PIC jump to app
2021-02-06 01:23:19 Check chain[1] PIC fw version=0x89
2021-02-06 01:23:21 chain[2] PIC jump to app
2021-02-06 01:23:22 Check chain[2] PIC fw version=0x89
2021-02-06 01:23:22 create thread
2021-02-06 01:23:22 max sensor num = 4
2021-02-06 01:23:22 temperature_monitor_thread start...
2021-02-06 01:23:27 power type version: 0x0071
2021-02-06 01:23:27 Enter sleep to make sure power release finish.
2021-02-06 01:26:28 Slept 180 seconds, diff = 8.
2021-02-06 01:26:28 set_voltage_by_steps to 1500.
2021-02-06 01:26:33 start up min temp by 75a = 17
2021-02-06 01:26:33 set UART baud to 115200
2021-02-06 01:26:41 Chain[0]: find 114 asic, times 0
2021-02-06 01:26:43 chain avg vol rise to 15.46
2021-02-06 01:26:48 Chain[1]: find 114 asic, times 0
2021-02-06 01:26:56 Chain[2]: find 114 asic, times 0
2021-02-06 01:26:56 pulse_mode = 1, ccdly_sel = 1, pwth_sel = 1
2021-02-06 01:27:01 min freq in eeprom = 525
2021-02-06 01:27:01 fixed frequency is 525
2021-02-06 01:27:01 Bring up temperature is 17
2021-02-06 01:27:01 set UART baud to 12000000
2021-02-06 01:27:01 set_voltage_by_steps to 1267.
2021-02-06 01:27:08 STATUS_INITED: soc init done!
2021-02-06 01:27:08 create thread
2021-02-06 01:27:08 create thread
2021-02-06 01:27:09 fan_etf: Set fixed fan speed=90
2021-02-06 01:27:10 start to init...
2021-02-06 01:27:19 Init done!
2021-02-06 01:27:19 STATUS_OKAY
2021-02-06 01:27:19 start the cached job
2021-02-06 01:27:19 Version num 8
2021-02-06 01:27:19 Mask num 0xe000
2021-02-06 01:27:19 freq = 525, percent = 90, hcn = 12480, timeout = 449
2021-02-06 01:27:19 set_start_time_point total_tv_start_sys=271 total_tv_end_sys=272
2021-02-06 01:27:19 set_voltage_by_steps to 1290.
2021-02-06 01:28:42 set_voltage_by_steps to 1270.
2021-02-06 01:32:48 set_voltage_by_steps to 1260.
2021-02-06 02:32:10 chain avg vol drop to 0.99
2021-02-06 02:34:47 set_voltage_by_steps to 1280.
2021-02-06 02:35:27 read asic reg error: expect chain = 0, chip = 0, reg = 176, got chain = 1, chip = 48, reg = 128
2021-02-06 02:35:35 chain avg vol rise to 13.25
2021-02-06 02:42:11 chain 0 hash rate 12253.00 low in 15 mins
2021-02-06 02:42:11 chain 1 hash rate 12031.00 low in 15 mins
2021-02-06 02:42:11 chain 2 hash rate 12252.00 low in 15 mins
2021-02-06 02:57:12 avg rate is lower than ideal rate, 18268.86 in 30 mins
2021-02-06 02:57:12 chain 0 hash rate 0.00 low in 15 mins
2021-02-06 02:57:12 chain 1 hash rate 0.00 low in 15 mins
2021-02-06 02:57:12 chain 2 hash rate 0.00 low in 15 mins

[irrWN]

Hey, i have same problem. But i found one nuance. You was try to start only one of this boards?

When i start only one of 3 boards it work. If more than one board - get error.

[mikeywith]

Please use the code tag to paste kernel logs, my take on this issue is that you damaged all those 20 PSUs by feeding them improper voltage or any other electrical related issue, the PSU acts differently when plugged elsewhere gives more strength to the theory, to confirm/deny this theory use a PSU which has not been plugged in that "bad" area of the farm on one of the "bad" miners, obviously, the test should be done in a different outlet/area, chances are it will work perfectly fine, if not -- then there is even worse news, which I doubt for now.

[CryptoLLC]

Please use the code tag to paste kernel logs, my take on this issue is that you damaged all those 20 PSUs by feeding them improper voltage or any other electrical related issue, the PSU acts differently when plugged elsewhere gives more strength to the theory, to confirm/deny this theory use a PSU which has not been plugged in that "bad" area of the farm on one of the "bad" miners, obviously, the test should be done in a different outlet/area, chances are it will work perfectly fine, if not -- then there is even worse news, which I doubt for now.

Looks like I solved the issue. The back of the psu is heating up. Even though the chips are cool and the pcb is cool and the psu is cool. The very back of the psu is not. I think the fans on the psu are not fast enough and cause this issue. Make sure you remove all the heat. This problem does not exsist on the S17 series and its only on the S19.. Watch the video.
https://www.youtube.com/watch?v=qg7HFFAkB7A

[mikeywith]

I said in the very first post in this "long" thread:

... if the PSU is dirty or isn't getting enough air flow it i will stop.

You have even suspected that the back of the PSU was getting hot, looks like your initial troubleshooting went wrong and led you (and us for that matter) into thinking it was something else.

And FYI, this isn't an S19 only a problem, it's common with the 17 series as well, the S17 Pro is kind of an expectation due to its high efficiency and relatively low overall power consumtion, of course, weaker fans on the S19 PSU could make this problem even worse, but bad PSU airflow is a very common problem.

Thanks for reporting back (most people don't), it was quite a ride, but we all learned some stuff along the way. Mine on.

[CryptoLLC]

Yeah. The reason I did not suspect it to be the psu is because the chips on the boards are below 50 and the psu is cool to the touch. Also its 40f degrees inside the container. So I assumed it cant be the psu. (The PSU's are brand new and have zero dust). Also I have an even worse heat situation with the s17+ and none of them have problems. These S19 Pro have very sensitive PSU's when compared to the S17+.

[PT-Mining]

i think i have a similar issue.
What was shown on the youtube video?!
this sadly is offline now...

i wonder how units work, that use intake and exhaust adapters for ducted colder air?! usually the PSU has no separat duct on these, so it still gets only room temperatue. But no extra or cooler airflow.

 Or does such ducted solutions work bad anyhow?