[Solved] Avalon 921 Blue Light but no Hashing

[s_falahati]

Hey

I just bought 12 Avalon 921 and out of these 12, I can't get 3 of them to work. Incidentally, I have 3 AUC3s and there is one faulty miner per each AUC3 in my initial setup; however, I checked one of the miners alone and the issue was still there, so controller and the AUC3s should be alright.

All AUC3s are green and sometimes I get a very short blink of red; however, all miners are blue even when I can feel they are cold and the fan is slow. This is also apparent with the CGMiner API Log. Controllers also detect all miners and there is no problem there. The only thing I noticed with faulty miners comparing to healthy ones is the fact that they have high "DH" value which I don't know what it really means considering the fact that all other error values are zero (MH, etc etc).

These devices make no hash what so ever (MW = 0 for all; DH > 99%). The fan is working, temp is alright and CGMiner shows ~10,000 GH activity (which is a lie considering the power consumption being like 0.5 to 1 amp). All PSUs are new and providing around ~12.3v.

Attached is the CGMiner API Log (With Debug Switch) while all 12 miners are connected. 3 faulty ones are easy to spot tho.

3 out of 12 is a big number for all new devices, and with all 4 boards not working and everything seems normal otherwise (MM, Internal Power Board) I think maybe I missed something; They can't all be really faulty; can they?

Sending them for warranty back probably going to cost me the same as buying 3 new miners.

Status: https://i.imgur.com/iYeTYW8.png
Log: https://justpaste.it/4pyc8

[1714501139]

1714501139

[1714501139]

1714501139

[1714501139]

1714501139

[1714501139]

1714501139

[NotFuzzyWarm]

Ref the troubleshooting guide for A721/741/821/941 at the top of this section.

All the Avalons from at least 7 on up all work the same and have pretty substantial diagnostics built in to the API reports.

[Steamtyme]

I would be looking to try a couple of things, as all 3 may not have the same issue, if any.

First I would just start working with the 3 miners giving you issues, so you don't have to unnecessarily start and stop the others.

My first thought on this is that maybe you have some bad/ 4 pin cables running to those miners, or they could be not seated properly. I had to once that despite appearing and feeling correct they needed to be reseated.

You can try swapping around cables with others that you know work for other miners.

Also just to note for further troubleshooting, are either of these miners on the same chain/auc line? If so the second one may have no problem if it is a cable issue.

What are you using to power your controller?

Edit: It's weird I took some time to go through everything and nothing jumps out at me as something obvious. I would still try them just on their own swapping some of the 4 pin cables around just to see if that helps. Definitely start a repair ticket with Canaan, and let them know what you have done so far.

In my experience they are really good at working through things with you, and from time to time depending on your comfort and knowledge can send parts out as opposed to asking for everything to be shipped back. Good luck and keep us up to date.

[s_falahati]

Thanks for replying, I checked the PDF version of this topic and couldn't troubleshoot it myself, that's why I am asking here for help.

"ECHU" is zero for all boards, "ECMM" is zero for all miners, "PVT_T"s are all within range, "CRC"s are all zero.
The fan is working at 10% (with rpm reading; and I can see and hear it); power is above 12v (multimeter and log reading), AUC is working and the controller detects the miner, so the MM should be alright. Also, controllers work with other miners so no problem there. PMU board reports ~12.3v input and ~4.1v output which is in line with other miners working correctly. So PMU should be fine. "PVT_T"s are also within reason. "MV" values are also similar.

However, I don't know what some values mean, for example, there are some >90% values for "ERATIO" and there is no description in the documentation. In the summary following is the difference between a working miner and a faulty miner for me:

"DH" > 90%
"TMax" is low comparing to a healthy miner due to the fact that there is no mining going on
"Fan" is at 1650 (10%) but this is expected, again, as there is no mining going on
"PLLX" for all healthy miners start with "0 0 0 0" but this isn't the case with faulty miners; but guessing this is the performance state of chips, this should be expected as the frequency is lower
"GHSmm" for all faulty miners are around 10TH, but this isn't true as I can clearly see power usage is too low for real 10TH mining.
"WU" is zero for faulty miners
"Freq" is around ~380 for faulty miners comparing to ~740 for healthy ones
"MW0", "MW1", "MW2" and "MW3" values are all zero in faulty miners
"PVT_V" values in a healthy miner are all around 4v, but this isn't the case for faulty ones, there are values ranging from 1.8v to 7.3v
"ERATIOX" values are between 2% to 6% for healthy miners, but faulty ones have a lot of 0% and some >90%
"C_XX_XX" values for faulty miners have a lot of 0 and low values in general (<60, <1000)
"GHSmmXX" lot of zeros and low values for faulty miners here comparing to healthy ones

The thing is, I have no Idea how to interpret these values and the troubleshooting thread or PDF don't provide an answer.

EDIT: I also just noticed that all three faulty miners have 9211901-faf6d80 MM version.

[NotFuzzyWarm]

... I also just noticed that all three faulty miners have 9211901-faf6d80 MM version.

That could be part of the problem -- a while back someone mentioned that the 921 is very picky about which controller firmware version and which miner firmware versions are used. They sort of had to be matched up as I recall.

As said before, put all 3 on 1 controller to make it easier to troubleshoot without losing hash rate from the ones that are working.

[s_falahati]

Thanks for replying.

I also suspected a problem with cables and even maybe AUCs having a problem with not supporting more than 3 miners (I know it might be silly, but at first I got one faulty miner per each AUC, so that was a valid guess) However cables should be alright as miners further the line get the signal and work alright; I also tried changing cables, trying with one miner directly connected to and AUC again directly connected to the controller and in summery, every combination I thought might be useful to isolate the problem. No success tho.

I am using the same adapter provided by the controller in my case; don't know the specifications tho; I might try to find a 3A adapter I know I have lying around somewhere that I used for my RP3 plex setup to see if it changes anything. But I doubt it is an undervoltage problem with USB ports as all three AUC are powered and each has 3 miners connected to them working alright.

At this stage, considering the fact that all faulty miners have the same MM version, I think it might be a firmware or a hardware problem. I am now going to open a ticket with Canaan; but does anyone here used a miner with MM Version 9211901-faf6d80 before? Couldn't find anything about it by googling it.

[Steamtyme]

I think you nailed it. I was trying to track the same thing but kept getting lost in the log, that's one of the reasons I was gonna get you to group things together, it would be easier to compare them.

I don't have any past the 841's, and I haven't updated anything in a while. Best to confirm with Canaan/or someone here. I think you'll need to just set up the 3 miners having problems and then upgrade the other version that is working across the board. This way everything will be the same. Not sure if there's benefits to one over the other, but once everything is the same, you can make changes in one broad stroke moving forward.

[s_falahati]

Well, based on this page: https://github.com/Canaan-Creative/avalon9-docs/wiki/AvalonMiner-921-firmware-downloads

There is no 9211901-faf6d80 firmware version at all (latest being the one already installed on the other 9 miners) and if number here means anything, the version installed on these 3 is newer.

The thing is, this is my first batch of Avalons and I have no prior experience with them. I only had S9 to this point. Do I need to use the "MM Upgrade" option in the control panel? Does it matter that this controller is not even fully compatible with these devices? Should I worry about this fact that the faulty MM version seems higher? and do I need to disconnect other miners before updating the firmware?

Avalons are both good (less Lan cable, no software needed to change settings globally, easier reporting and settings when working with one WebUI) and a pain in the a**; mixed feelings already. xD


Edit:

Just as I expected, the firmware installed on these 3 miners are in fact newer. Github page is outdated, but the firmware is available to download from their download directory: https://canaan.io/downloads/software/avalon921/mm/2019-01-03/

If this is, in fact, the case, I need to update my controller firmware to this version which is released on the same day as the MM firmware: https://canaan.io/downloads/software/avalon921/openwrt/20190103/

I might need to upgrade my other 9 miners to the newer firmware before moving on with the controller as I have only one controller at the time.

[frodocooper]

It's most likely the case that your Controller's older OpenWrt firmware is not communicating properly with the newer MM firmware of your three non-functioning A921s. Canaan made some changes and added some new parameters post-20180929 that may cause communication issues between the older Controller firmware and the newer MM firmware. I do not see any other apparent issues in your logs.

I therefore suggest a clean install of version 20190103 of the Controller firmware and then a reflash of all of your A921s to version 2019-01-03 of the MM firmware.

[s_falahati]

It's most likely the case that your Controller's older OpenWrt firmware is not communicating properly with the newer MM firmware of your three non-functioning A921s. Canaan made some changes and added some new parameters post-20180929 that may cause communication issues between the older Controller firmware and the newer MM firmware. I do not see any other apparent issues in your logs.

I therefore suggest a clean install of version 20190103 of the Controller firmware and then a reflash of all of your A921s to version 2019-01-03 of the MM firmware.

The thing is that the controller I got with the miners is a Vidtoo H2 and there is no older firmware for it; only the latest version. And I can't check the firmware on the controller as I don't know where to look in to see the version inside the OpenWRT control panel (there is only one version for OpenWRT itself); therefore I have a feeling maybe controller is the latest version and in that case I might need to downgrade those 3 miners instead of upgrading the other 9. Canaan is yet to respond to my ticket and I am little scared to change anything before getting an ok from them; however, I am going to try some of your suggestions this evening guys anyway to see if I can get them all working. Have my RP3 ready with the latest firmware to see if I can get those 3 to work with the new controller before trying to upgrade the main controller and then all miners. A lot of confusions here.

If the firmware is, in fact, the number I get at the top of CGMiner API Log, then it is 20180929 which is, two versions old; but this version is not even available in their download repository (nearest being 20180926).

[HagssFIN]

I would setup it with a RPi and do as frodocooper said in his post.

[Steamtyme]

It's convenient you already have a rpi3 ready to go. I hadn't heard of anyone using other controllers, where did you order  your gear from

You should try at the moment just swapping your controller for the rpi and see if you still have the issue with the 3.


As a side note do they run any risk at the moment if they were to load the 3 machines with the same MM version as the others.

I only ask because if they did not have a pi available, that would be the fastest option for trying to get all machines hashing.

[NotFuzzyWarm]

Ja. Be aware that when you flash MM firmware ALL miners on that controller get the update.
Any miners you do not want to change must first be disconnected from the controller.

[s_falahati]

Got them all working.

First I tried my RP3 with the latest firmware (20190103: https://canaan.io/downloads/software/avalon921/openwrt/20190103/rpi3-modelb/) and this managed to get those 3 miners with newer firmware working, but failed to get other 9 with an older version of firmware to work. So the statement about Avalon miners and controllers being picky about the firmware version is completely right and it happens both ways.

Then I tried updating the Vidtoo H2 to the latest controller firmware and used the 20190103 ext4 SysUpgrade (https://canaan.io/downloads/software/avalon921/openwrt/20190103/h3/) file system but this resulted in controller failing to boot up and stuck in an infinite restart. So my Vidtoo controller is dead at this point. Brought it home to see if I can fix it.

Meanwhile, I used my RP3 to update the other 9 miners to the 9211901-faf6d80 (https://canaan.io/downloads/software/avalon921/mm/2019-01-03/) firmware after disconnecting the 3 miners that are already up to date and each time with 3 of them connected to minimize the risk in case I mess something up. And it went well. RP3 now successfully runs all 12 miners all being on the latest firmware.

One thing to keep in mind is that Vidtoo H2 probably uses SquashFS filesystem so if anyone got a Vidtoo H2 controller you should be extra cautious with upgrading the firmware especially as it seems to have internal memory (along with an SDCard slot) and the default firmware is probably on the internal memory because in my case the SDCard Slot is empty. I need to add that I am not totally sure if it is in fact formatted with SquashFS; what I know is that the 20190103 ext4 SysUpgrade, when applied without removing old settings, will fail. It might be also because I missed something.

... I hadn't heard of anyone using other controllers, where did you order  your gear from...

I got my devices from a local distributor; I also failed to find any information about this board. In fact, even Canaan only provides roms for this device in their latest firmware and there is no mention of this device with their older firmwares. My distributor mentioned that these controllers are new so my guess is either Canaan is sending this controller instead of RP with their 2019 batches or that my distributor just jumped on the opportunity to buy a cheaper board as soon as Canaan added support. However, with the controller having a private firmware (a version not available anywhere publicly) there is more chance that this was what Canaan sent to them.

Thank you all guys, specially Steamtyme and NotFuzzyWarm; your suggestions were priceless. Also everyone else for taking the time reading the log file; I know it wasn't easy to follow things and it probably took a lot of time. I am yet to receive a response from Canaan btw.

[NotFuzzyWarm]

Glad to hear things are now working. Ja, not sure why the A9's are so picky about versions as I never had the problem with the A7 and A8 series.
Merit to ya for keeping us up-to-date on the results and changing the Topic title to show problem solved. Way too many folks fail to do that...

Surprised Canaan has not responded to you yet! In the past they have always been very fast responding to the few issues I and others have ran into.

[frodocooper]

I too did not receive any reply from Canaan when I recently contacted them to request for a replacement for a failed A921 hash board. Their dedicated email address for RMA requests — rma@canaan.io — apparently no longer exists; my email to that address was returned.

Only when Blokforge contacted Canaan on my behalf — Blokforge took three weeks before responding to my support request — did Lily Han from Canaan inform me that the person in charge of after-sales support had left the company. There is therefore currently no one handling support tickets at Canaan, according to Lily.

Anyone seeking support from Canaan should therefore contact Lily directly at lily@canaan.io.

[s_falahati]

Just for the sake of completion and as probably the first one here getting a VidToo controller; following are some pictures of the board and some of my thoughts regarding it. First I should inform you that I am yet to be able to boot it up again due to the fact that I don't have a MicroSD card in hand at the moment and my experiment with Ethernet and the USB stick failed so far (it was a long shot anyway).

A very cheap and simple box is used with this board, but it does the job I suppose:
https://i.imgur.com/sa8N9mn.jpg

This board supposedly has a hardware version 1.3 and software version 1.0 and is created for Avalon A9 series:
https://i.imgur.com/vCjdsYV.jpg

Nothing special going on inside, the board has 4 USB ports and one Ethernet Port along with a MicroUSB port for power and a MicroSD card slot. There is a place for a power adapter connector but it seems they ditched it out in favour of the MicroUSB connection. If you look closely at the left side thereof the board there is a 3pin TTL UART connector operating at 3.3v. I failed to find my TTL UART 2 USB converter to check if anything is going on with this connector. However, I don't expect much. There is also a button next to the MicroUSB port that I don't know what it supposedly does. Pressing it once doesn't change anything but holding it while the board is booting enters in some sort of firmware update or bootloader update I think because the board no longer restarts. However, without an HDMI connector, it is hard to guess what really happens there.
Image: https://i.imgur.com/7fuCBYW.jpg

Nothing at all is going on at the back side of the PCB:
Image: https://i.imgur.com/Tew8r2i.jpg

This board has an Id of ZG834 but I found nothing online:
Image: https://i.imgur.com/HoHbuu7.jpg

There is a Nanya NT5CC64M16GP 128MB (1024Mbit) DDR3 DRAM here:
Image: https://i.imgur.com/KQnfF0G.jpg
Info: https://www.arrow.com/en/products/nt5cc64m16gp-di/nanya-technology

And a GigaDevice GD25Q127CSIG 16MB (128Mbit) flash memory. This is the internal memory of the board:
Image: https://i.imgur.com/P6cP7Ak.jpg
Info: https://www.digikey.com/product-detail/en/gigadevice-semiconductor-hk-limited/GD25Q127CSIGR/1970-1027-1-ND/9484777
AliExpress: https://www.aliexpress.com/item/50pcs-GD25Q127CSIG-GD25Q127CSIGR-25Q127CSIG-25Q128-25Q127-SOP-8-new-original/32895669867.html
Datasheet: http://www.elm-tech.com/en/products/spi-flash-memory/gd25q127/gd25q127.pdf

No wonder I messed this up. The firmware was like 50mb.

And a TNK BT16B03 ethernet transformer chip for isolation:
Image: https://i.imgur.com/b7j5W3V.jpg
Info: https://lcsc.com/product-detail/RJ45-Transformer_TNK-BT16B03_C216354.html

The main chipset is an AllWinner H2+ ARM Cortex 7 and is hidden behind a heatsink. I didn't try to take the heatsink off as I knew the type of the chipset already from the firmware and the OpenWRT stat page. This chipset is very similar to the AllWinner H3 that is widely used in different devices including the OrangePi boards. I wonder if this means we can use the H3-H2+ ROM provided by Canaan to create a controller with OrangePi boards, they are widely available and a lot cheaper compared to a RasberryPi (1/3 of the price to be exact).
This chipset does have an HDMI output; it's just that with this PCB they didn't expose the pins.
Datasheet: http://wiki.friendlyarm.com/wiki/images/0/08/Allwinner_H2%2B_Datasheet_V1.2.pdf
Info: https://docs.armbian.com/Hardware_Allwinner-H3/

Couldn't find much about the VidToo company that designed and created this board except a piece of old news:
https://www.design-reuse.com/news/45322/vidtoo-codasip-bk3-risc-v-processor.html

And the fact that they are close to Canaan:
https://canaan-creative.com/en/ai-blockchain-diedairenleiwenmingawalongmugongsijiananyunzhiabcdzhiyeji-2018-yingban.html

So I can say with a quite high certainty that this board was in fact sent by the Canaan and not by my distributor.

If I fail to boot this with an SDCard I am going to take the flash memory out and try my luck by programming it with a serial connection using the firmware provided by Cannan. If I succeed; you hear about me; otherwise you can consider that I failed to revive the controller. Which means I lost more than 50$ using my RP instead of this. xD Had a lot of plans for that RP.

[NotFuzzyWarm]

Weird. Wonder why they went with that controller vs a good old RasPi-3B?
Along that line, if you do pick up another Pi, be aware that (at least in the past) the 3B+ version does NOT work.

[s_falahati]

Weird. Wonder why they went with that controller vs a good old RasPi-3B?

Obviously the price. This board is way cheaper than an RP3. Smaller RAM, a cheap small internal memory instead of an 8GB SDCard, a slow and cheap chipset and no unnecessary interface like HDMI, GIO Pins, Wifi, etc. I bet this cost them 1/10 of what a RP cost especially as these seems custom made for Canaan.

To be honest; RP3 is way too overpowered for OpenWRT; my PR3 was always under 0.01 load in the past two days. That's what, around 1% of one core?

Along that line, if you do pick up another Pi, be aware that (at least in the past) the 3B+ version does NOT work.

Didn't expect this. Thanks for informing me.

[HagssFIN]

Cool information, thank you.

If anyone has a Orange Pi, it would be interesting to test the H3-H2+ firmware with it and see if it works.