Bitcoin Forum

Bitcoin => Mining support => Topic started by: cableiso on November 11, 2017, 04:39:11 AM



Title: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: cableiso on November 11, 2017, 04:39:11 AM
Hi Folks,

I got lucky in fixing a corrupted control board, and was able to bring it back to life using the serial console and uboot.  Wanted to mention my story to open a discussion on debricking S9/T9.  Also, if anyone wants to sell me a bricked S9 I'm interested lol :)

The symptoms were that upon power on, the green lights lit and the ethernet port showed connection (yellow) and data (green).  Data was occasionally blinking.  Really, the device looked like it was alive but didn't mine.  It never did its earsplitting singleboardtest, never even requested an IP from my router, so I could not SSH in or access the web server.  5 second reset, and IP reporter reset didn't help either.

I decided to check out the terminal console.  I disconnected the ribbon cables and pulled the control board out of the miner, then used a $2 USB to serial converter board to connect to the RX, TX, and GND headers on the board. [NOTE: Use 3.3v mode]  I powered the board up using the PCIE-6 plug from a spare PC power supply.  I got text!

Code:
blah blah lots of miscellaneous messages blah blah hex characters
 
Copying Linux from NAND flash to RAM...

NAND read: device 0 offset 0x1100000, size 0x800000
 8388608 bytes read: OK

NAND read: device 0 offset 0x1020000, size 0x20000
 131072 bytes read: OK
## Booting kernel from Legacy Image at 02000000 ...
   Image Name:   Linux-3.14.0-xilinx-gb190cb0-dir
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3779264 Bytes = 3.6 MiB
   Load Address: 00008000
   Entry Point:  00008000

Then a complaint about invalid kernel, and it dropped into the uboot console.  Boo!  So the thing wasn't even starting to boot, but since I had terminal access from the serial port I was hopeful.  It seems that in a borked upgrade (at least this case) the kernel image is what goes bad.  We will learn more over time, I'm sure.

Here's what I did to address it
1) Studied the shit out of all the different bootup types that can be seen using the "printenv" and "?" commands in uboot.  Bitmain must have 8-10 types of boot scripts in the environment variables, accessed by the run command.  "run qspiboot", "run nandboot", etc.  Lots of interesting reading, but they all do similar things - load stuff to ram and execute.  Hey, I'm capable of doing that!  Let me jump in the mix!

2) Formatted an SD card to FAT32, and unpacked the "xilinx" folder from a valid firmware image onto it.  This gives me BOOT.bin, devicetree.tdb, rootfs.jffs2, uImage, and upgrade-marker.bin.  Everything needed for a zero-state boot except uramdisk.image.gz.  SO IF ANYONE HAS EXTRACTED THIS, PLEASE LET ME KNOW.  I'd like to make an unbricking toolkit.
NOTE: for S9 without an SD card, you're going to have to TFTP your files.  I don't think this is a big issue though, just download a TFTP server and drop the files in it's root dir.  Should work the same.

3) Stumbled my way through the SD card commands: mmc rescan, mmc list, mmc part to find out that windows format tool had called the FAT32 partition 5, so thats where my "xilinx" files were.  Why 5?  No clue, but you need to know where they are so you can tell uboot where to load from.

4) Hacked an mmc load of the kernel from SD partition 5 into the memory location uboot was expecting:  Note that I am using 0:5 to denote mmc 0, partition 5.  ${kernel_image} is a variable defined in the environment that stores the filename uImage.  You can simply type uImage into the command below, it will work fine.

Code:
fatload mmc 0:5 0x2000000 ${kernel_image}

5) Booted the little bastard using bootp.  Here I got lucky that the ramdisk and other parts of linux were not corrupt, just the kernel image.  Worse corruptions may need more surgery, hence the need for a valid ramdisk image.  But in my case, injecting a new kernel was all it needed.

6) Once it was up, connected via the web interface and flashed a new firmware image as fast as I could.  Woo!  The miner is back!  After that, it was cured and the board went back into the miner.

All in all, very simple.  But having a $2 USB to serial adapter on hand saved me the cost of a new control board, as well as the hassle and delay of shipping, waiting, dealing with bitmain lol.  Well worth it's weight in gold this evening, for sure.  Amazon or ebay, mine was called "FT232 Mini USB to Serial".  About two inches long, mini USB on one side, 6 pins on the other side for RX, TX, GND and other signals I never use.

Anyway - if anyone else has a bricked control board you want me to hack on, just PM me.  I'd like to see some other examples of how boards brick, because developing a generic "unbricking tool" would be really nice to have.


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: Biffa on November 11, 2017, 10:59:45 AM
That's a very informative and useful post. Thank you


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: NotFuzzyWarm on November 11, 2017, 08:43:14 PM
Excellent post and bookmarked!
Now if we can just find out what the s9 hash board failure modes & causes are....


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: HagssFIN on November 11, 2017, 08:47:32 PM
This is very nice little guide, thank you!

Will come back to look at this, if my S9 control board bricks  :)


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: bgasolder on September 29, 2018, 09:14:42 AM
I did as shown in the video and my antminer s9 came to life :D

https://preview.ibb.co/iMWgE9/photo_2018_09_14_19_37_16.jpg (https://preview.ibb.co/iMWgE9/photo_2018_09_14_19_37_16.jpg)

Recovery NAND antminer s9 t9 via micro SD card
https://youtu.be/dzArOe8KuHI


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: technak on July 05, 2019, 12:02:19 PM
Hi,
I am just locked my self with new secure firmware :-( and when i try boot from sd card nothing happen immediatelly as i switch last jumper to boot from sd and power on both leds green and red are constantly on :-( I am thinking about using your method to downgrade  firmware to not secure one. You mentioned tftp server could you please give us more info how to replace kernel image in nand with tftp ? I think this will be very helpfull to community as bitmain is not playing fair ... May be easiest way will be just enable ssh throught serial console but i am not sure if this is possible. Many thanks for your help. MK


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: Artemis3 on July 05, 2019, 02:05:58 PM
Hmm this guide would be useful to recover a controller without the sd slot soldered in. I think an ftdi cable plugged to the rj45 jack would have done the same, but yeah, same thing if you plug to that 3 pin header.

Also if the nand becomes damaged, move jp4 and do everything directly from the sd card (ie load from sd into ram, then boot).

But then if you have the sd slot you might as well just use the sd card recovery method.


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: technak on July 05, 2019, 04:15:10 PM
currently waiting for cable but I think the steps will be:
1. setup tftp server / load uImage from not secure kernel to root
2. issue tftp $0x2000000 ${kernel_image} - kernel_image = uImage
3. bootp to start kernel
4. upgrade firmware via web interface
will test it when cable arrive not sure if it will work as kernel will be from different version of firmware


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: MigaoMiner on July 24, 2019, 02:57:28 AM
Dear All,

I have connected the USB to serial converter to the S9 controller board. Connected via serial connection and got bunch of messages.
Weird is i cannot login, and the system just continue to reboot n ended at Fan detection.

What ever i type seems not received by the Controller board.

Appreciate kind advise.

regards,

MM


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: Mihaiinvader on December 26, 2019, 03:48:37 PM
Hello,

I have 2 control boards that are bricked now, my father did a firmware update from sd and during the update he reseted them and probably corrupted the boot. I want to try the steps that you presented but i dont really understand linux. All comands are done in on a linux platform ? Can i do it also on windows ? Thank you!


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: Matce on September 03, 2021, 05:44:22 PM
Hi Folks,

I got lucky in fixing a corrupted control board, and was able to bring it back to life using the serial console and uboot.  Wanted to mention my story to open a discussion on debricking S9/T9.  Also, if anyone wants to sell me a bricked S9 I'm interested lol :)

The symptoms were that upon power on, the green lights lit and the ethernet port showed connection (yellow) and data (green).  Data was occasionally blinking.  Really, the device looked like it was alive but didn't mine.  It never did its earsplitting singleboardtest, never even requested an IP from my router, so I could not SSH in or access the web server.  5 second reset, and IP reporter reset didn't help either.

I decided to check out the terminal console.  I disconnected the ribbon cables and pulled the control board out of the miner, then used a $2 USB to serial converter board to connect to the RX, TX, and GND headers on the board. [NOTE: Use 3.3v mode]  I powered the board up using the PCIE-6 plug from a spare PC power supply.  I got text!

Code:
blah blah lots of miscellaneous messages blah blah hex characters
 
Copying Linux from NAND flash to RAM...

NAND read: device 0 offset 0x1100000, size 0x800000
 8388608 bytes read: OK

NAND read: device 0 offset 0x1020000, size 0x20000
 131072 bytes read: OK
## Booting kernel from Legacy Image at 02000000 ...
   Image Name:   Linux-3.14.0-xilinx-gb190cb0-dir
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3779264 Bytes = 3.6 MiB
   Load Address: 00008000
   Entry Point:  00008000

Then a complaint about invalid kernel, and it dropped into the uboot console.  Boo!  So the thing wasn't even starting to boot, but since I had terminal access from the serial port I was hopeful.  It seems that in a borked upgrade (at least this case) the kernel image is what goes bad.  We will learn more over time, I'm sure.

Here's what I did to address it
1) Studied the shit out of all the different bootup types that can be seen using the "printenv" and "?" commands in uboot.  Bitmain must have 8-10 types of boot scripts in the environment variables, accessed by the run command.  "run qspiboot", "run nandboot", etc.  Lots of interesting reading, but they all do similar things - load stuff to ram and execute.  Hey, I'm capable of doing that!  Let me jump in the mix!

2) Formatted an SD card to FAT32, and unpacked the "xilinx" folder from a valid firmware image onto it.  This gives me BOOT.bin, devicetree.tdb, rootfs.jffs2, uImage, and upgrade-marker.bin.  Everything needed for a zero-state boot except uramdisk.image.gz.  SO IF ANYONE HAS EXTRACTED THIS, PLEASE LET ME KNOW.  I'd like to make an unbricking toolkit.
NOTE: for S9 without an SD card, you're going to have to TFTP your files.  I don't think this is a big issue though, just download a TFTP server and drop the files in it's root dir.  Should work the same.

3) Stumbled my way through the SD card commands: mmc rescan, mmc list, mmc part to find out that windows format tool had called the FAT32 partition 5, so thats where my "xilinx" files were.  Why 5?  No clue, but you need to know where they are so you can tell uboot where to load from.

4) Hacked an mmc load of the kernel from SD partition 5 into the memory location uboot was expecting:  Note that I am using 0:5 to denote mmc 0, partition 5.  ${kernel_image} is a variable defined in the environment that stores the filename uImage.  You can simply type uImage into the command below, it will work fine.

Code:
fatload mmc 0:5 0x2000000 ${kernel_image}

5) Booted the little bastard using bootp.  Here I got lucky that the ramdisk and other parts of linux were not corrupt, just the kernel image.  Worse corruptions may need more surgery, hence the need for a valid ramdisk image.  But in my case, injecting a new kernel was all it needed.

6) Once it was up, connected via the web interface and flashed a new firmware image as fast as I could.  Woo!  The miner is back!  After that, it was cured and the board went back into the miner.

All in all, very simple.  But having a $2 USB to serial adapter on hand saved me the cost of a new control board, as well as the hassle and delay of shipping, waiting, dealing with bitmain lol.  Well worth it's weight in gold this evening, for sure.  Amazon or ebay, mine was called "FT232 Mini USB to Serial".  About two inches long, mini USB on one side, 6 pins on the other side for RX, TX, GND and other signals I never use.

Anyway - if anyone else has a bricked control board you want me to hack on, just PM me.  I'd like to see some other examples of how boards brick, because developing a generic "unbricking tool" would be really nice to have.

Is there any update on this?

I have a bricked S9 board, got the RX/TX/GND connected via USB to windows / putty console ... but I'm getting nothing.

thanks


Title: Re: Rescuing a bricked T9 (and S9?) Control Board - HOWTO
Post by: BitMaxz on September 03, 2021, 10:54:17 PM
Is there any update on this?

I have a bricked S9 board, got the RX/TX/GND connected via USB to windows / putty console ... but I'm getting nothing.

thanks

Take note of this

Code:
[Use 3.3v mode]  I powered the board up using the PCIE-6 plug from a spare PC power supply. 

I think he uses a Linux OS to detect the control board but try the commands above under #3 read that part.