Have my S9 miners been hacked?

[Ameador1]

I use Awesome Miner software to manage my miners. About 2 months ago I started using the Awesome Miner Antminer Firmware. I tuned them myself - lowered voltages to 8.5V and set frequencies to 762, then, based on number of chips on a board that were not running in their happy zone, I upped voltage by 0.1 and lowered the frequency. After getting the boards so that they were at a point were most of the chips were in their happy zone, I then reduced to individual chip frequencies until I got them very consistently into their happy green zone. For the last month, my 8 S9s (S9s: S9, S9i, S9j - 13.5 to 14.5TH/s), have been running fine. Then today when I came home from work they were all only running for a few minutes each, so I check them and they are running very badly - MANY chips at 0 and red, missing whole boards, all of them only getting up to around a third or less of their regular hash rate. The firmware is set to restart them when there are more than 5 X's on a chain - so they were just setting there restarting themselves every few minutes.

Something appears to have changed around noon or so yesterday. I mine with Nicehash - and they are paying out ever 4 hours. I am generally, as of late, at around 0.00042000 to 0.00046000BTC per 4 hours cycle. But yesterday the payout at 4PM was 0.00054000BTC, then at 8PM it was 0.00055000BTC, at midnight is was 0.00051000BTC, at 4AM this morning it was 0.00064000BTC, at 8AM it was 0.00056000BTC, at noon it was 0.00058000BTC, and at 4PM it was at 0.00059000BTC. Now, I got home and saw them looking all screwed up at around 5PM and Nicehash was showing a significant drop-off after the 4PM payout - so I assume they were still running strong at 4PM as these payout are like 25% higher than normal. I restarted them all via Awesome Miner and that made no difference. I shut off the power to them via the power panel breakers and restarted them. No change. I did it again, but left them off for about 45 minutes, but still not difference. I exported the Awesome Miner history and reviewed it, for the last 30 days, and there is not change in temps or hashrate recorded for these miners before or during this 25% uptick in mining payouts from Nicehash. I NEVER get these kind of jumps in payouts from Niceshash - here and there it may get 0.00048000BTC or something, but nothing like these numbers.

I also checked the power - they are all running on Antminer APW3++ PSUs at 240V and the outlets are testing good. I thought maybe a 120V leg had dropped and they may not have been getting enough power to run properly, but this does not seem to be the case.

When I look at the web interface of the miners with the Awesome Miner Firmware running on them and check the chip frequency page where it also shows the status of the chips - MANY of them are now red 0's and almost all of the rest are in the orange/yellow zone where they are not happy (the green zone). When I was tuning them, when the were like this, I would lower the frequency on those chips by a few selections lower in the drop-down list (like from 750M to 743M or lower) and after doing this a few times which a batch of chips at a time - I got them all into the green zones. This is what is look like now. It's like the frequency is set too high for the voltage and now they are out of the happy green zone again. But, the board frequencies and voltages are the same as I had them, with the same per chip frequencies as well - yet now all 8 miners are acting like I really jacked up the frequencies on the boards. Which could make sense with the much larger payouts on Nicehash. But the web interfaces just don't show like there has been any changes to the voltages or frequencies of anything.

But to have them all running fine for well over a month with no issues and then they all go crazy at the same time - it feels like a virus or hack since everything else seems to check out. But why would they hack me, up the hashing power, and still direct it to my pool? Unless just to cause damage?

So, what do you think? What can I do, check, or whatever at this point to get them back to normal? If I re-install the firmware via using the web interface to flash/update the firmware - would this overwrite the hack/virus? All of my S9's have SD card slots, so I can boot from them directly. I was thinking of trying to boot one from SD with like Braiins OS to see if it works normal like that. But I am worried at this point if whatever is going on here has damaged my miners. Any ideas, suggestions, etc... would be great!

Thanks in advance!

[1714803691]

1714803691

[1714803691]

1714803691

[1714803691]

1714803691

[1714803691]

1714803691

[1714803691]

1714803691

[philipma1957]

don't mine at nicehash.

try viabtc  and see if it works better.

[Ameador1]

don't mine at nicehash.

try viabtc  and see if it works better.

The choice of pool can cause chips to be marked as X's in the web interface of the miner, chains to drop, and chain status (in the miner web interface tab "Miner Status") to go from Healthy down to Slowed and cause the miner to restart? I have never had this problem with any pool including Nicehash and I've been mining with them for 2 years.

[philipma1957]

Nicehash is red hot right now and they may be switching you around a lot.

Which can screw up a miner.

you could try to set a level in the password so you dont switch as much.

but going to viabtc and setting to pps+ is a better way to see if you have an issue.

as nicehash could have a problem due to a hot bs coin.

you may have a voltage sag issue if you have a lot of xxx and oooo.

the voltage needs to be high enough for an apw3+ like 205 to 240 is better then 190 to 210

[thierry4wd]

[...]

+1

So, i have same problem, the problem come by Nicehash ! arround 12 - 14h nicehash have big problem with SHA256ASICBOOST pool ! if you want buy hash, showing only arroun 1 - 2PH ready ... is big problem ! i thinks server is down...

Now, all is return to normal... is only temporary problem

[Ameador1]

[...]

Voltage was tested at just a little over 240V. My power has thus far been very steady. I have a commercial power run just for my miners.

I have a LOT of xxx, I think sss too (I'm not home now where I can look, but pretty sure about the sss). Even hash boards not showing up. All of my miners were running fine - all hash boards were active - no lost chips on any of them. Until bang - now they all look like they got hit by a low level EMP!

I can give it a try on viabtc - but to me this looks like hardware at this point. If a virus/hack did this - then there's not much to do, but if Nicehash did this somehow - that's another story.

These units act like they had their frequencies on the chains/chips set so high that it burned them out - or is still set so high that they are just not able to run properly. Again, making me wonder about a virus that is running them at higher frequencies the the miner's web interface is showing - like the virus is hiding the true frequencies from the web interface.

I have additional fans on these units (to help cool them - to them point that if their own fans were not running - they might still be fine running. There's no indications that power dropped. Our house and the miners are coming off of the same transformer - they just have a feed from the transformer to a meter for the house, and another feed to a commercial meter for my miners. Nothing in the house gives any indication that power dropped. Sagged - possibly, but wouldn't that just cause the miners to drop some chips or chains temporarily? Once power cycled - especially with 45 minutes down before powering back up, that they would come back on normal?

What do you mean about setting a level in the password to keep from switching so much - I did not know there were things to be done with the password to effect anything?

I will try the viaBTC when I get home. That's one of the things that makes me question getting hacked or a virus though. My network is very inaccessible from the outside. I mean, I would HAVE to run a VPN to gain access. My ISP, oddly, issues private IPs to the WAN connection. So like 192.168.x.x, 172.16.x.x, or 10.x.x.x type addresses. So I can't use remote access software other than ones like TeamView or LogMeIn that have a host side application that links to their online systems to allow for making connections via their intermediary websites. And currently - I don't have a VPN or any of those remote access systems setup. I am careful about my computers as well - I am an IT guy, and I have (to my knowledge) not had a virus on any of my systems for better than 30 years - at least that wasn't a controlled scenario. I have no indications of any viruses on my computers in the network - at least nothing that virus scanning in finding.

The firmware has a tab in it that will scan the miner for it looks like 3 known viruses - and that is showing that none is found. But, I don't really know the virus world for miners. I would think there are more than 3 and like with PC's would be being changed and new ones being made that might evade this firmware's detection.

I'll let you know what I see with viaBTC - but I'm not optimistic - the chips and chains don't look happy before they even start trying to get and process packets - so, I don't honestly think that it makes any difference what pool I connect to at this point.

[...]

I think my issue is with the hardware at this point. It's not the amount of the payouts or hashing power seen by Nicehash that I'm concerned with. My miners chips and chains (hash boards) look like they are damaged or trying to run at too high of a frequency for the voltage they are configure for. Currently I have all the hash boards running between 8.5v to 8.8v with frequencies of 725M to 762M - and then the individual chip frequencies adjusted for ones that could not keep up with the board level frequency setting. Now they all act like the voltages are still where I set them, but they are trying to use frequencies well over 762M - causing the chips to not be able to run properly or at all and/or hash boards dropping out completely.

Again, my concern here that something has gotten on them and is actually running the frequencies higher than the firmware is configure for and showing in their web interfaces. That could explain the 25% increase in BTC payouts from Nicehash and why the hardware is having the problems it is having - I just don't know how to verify it or fix it - or if like Philipma1957 said - somehow Nicehash switching - thrashed them somehow.

Thanks.

[BitMaxz]

If you suspected that it was a virus trying to steal your hash rate or you doubt that you've been hacked then start to investigate.

If it was a virus or you've been hack can you try to flash it with SD card to clean up the NAND program.

The procedure can be found here "How to flash antminer s9 through SD card"

After the flashing is done. Flash it again with the latest firmware(A firmware with SSH disabled or SSH port closed) through WebGUI.

Now, after the flashing is done try to run it again and monitor the difference but use the ViaBTC pool first to test. Update here and let's see if you've been hacked or not.

[Artemis3]

[...]

As far as the chips are concerned, the pool being PPS+ or PPLNS has NO DIFFERENCE.

You may be referring to the downtime when switching, as typically done by Nicehash. This should not be an issue with the #xnsub option enabled.

To OP: The reasoning behind it is that the Antminers get hotter when they disconnect from a pool. But needing to use PPS+ is false. ViaBTC in both modes use the miners identically (and PPS+ is the default anyway). You can do the same test on any pool, such as ckpool and its perfectly valid, if not better simply because the pool operator might help you diagnose things. Kanopool is out of the question, because the operator will ban you.

Yes there is another advantage to testing in ckpool: You don't need an user account to use it, you can just use your bitcoin address.

[Ameador1]

As far as the chips are concerned, the pool being PPS+ or PPLNS has NO DIFFERENCE.

You may be referring to the downtime when switching, as typically done by Nicehash. This should not be an issue with the #xnsub option enabled.

To OP: The reasoning behind it is that the Antminers get hotter when they disconnect from a pool. But needing to use PPS+ is false. ViaBTC in both modes use the miners identically (and PPS+ is the default anyway). You can do the same test on any pool, such as ckpool and its perfectly valid, if not better simply because the pool operator might help you diagnose things. Kanopool is out of the question, because the operator will ban you.

Yes there is another advantage to testing in ckpool: You don't need an user account to use it, you can just use your bitcoin address.

I have the #xnsub option enabled with Nicehash. My miner fans are all running well plus I have additional large fans pulling 3200CFM across 12 miners (4 are L3+/++ units). Also, miner's firmware is set to target a temp of 76 and if the logs from Awesome Miners are correct (and I believe they are) the temps never went above this.

I will try ckpool too when I can. I'll probably turn on one breaker - which runs 2 miners - I can point one at viaBTC and one at ckpool - maybe, depends on my testing, I only have one SD card to try booting from at the moment, but I guess I could try them before booting up from the SD card.

Thanks!

[philipma1957]

Currently I have all the hash boards running between 8.5v to 8.8v with frequencies of 725M to 762M - and then the individual chip frequencies adjusted for ones that could not keep up with the board level frequency setting...

some times settings don't take on awesome miner.  are you posting on his thread and asking him and others about it?

https://bitcointalk.org/index.php?topic=676942.0  this is the link to the awesome miner thread.

[Ameador1]

some times settings don't take on awesome miner.  are you posting on his thread and asking him and others about it?

https://bitcointalk.org/index.php?topic=676942.0  this is the link to the awesome miner thread.

I haven't specifically posted there, but I did not configure these setting through Awesome Miner. I used Awesome Miner to push the firmware to the machines, then I manually tuned them myself through the miners web interface. And then they have been running - with corresponding has rate since (around 2 months). So I wouldn't think this would be an issue with Awesome Miners. I still look at the miners web interface as much as I do through Awesome Miner - since installing their (VNISH) firmware and I have found them to show corresponding values. Every setting I have configures and saved has stuck. I did not like the template patterns that Awesome Miner used to configure the miners for voltages and clock speeds - so I manually configured them directly. Reduced power and increased hash rates over their templates and have been running stably since. If I started having problems with them her and there on an individual basis I might would consider this to be related to the issue - but for them to all crap out within 30 minutes of each other after running with no issue for a couple of months, I don't think so. But, I will check with Patrik to see if he is aware of anything like viruses or other issues.

[mikeywith]

You have done so many things now it's nearly impossible to really diagnose the exact problem, Antminer viruses do in some cases increase the frequency and voltage of the infected miner when possible, after all, they know it's a hit and run and they want to squeeze as much hashrate as they can before you become aware, but from the way you describe the issue, it does not seem like a virus.

There is a little possibility that the firmware itself is faulty, or AM itself got a glitch somewhere that caused all this mess, maybe you saved an auto-tune profile and it only got activated now after your miners restarted for whatever reason/s, you need to consult with Patrick regarding this issue, however, one thing I am sure about is that Nicehash does cause a lot of damage especially chips showing as xxxx, I have personally had that experience, it doesn't matter if I use #xnsub or not, boards will drop very often, some boards got toasted and never came back to life, I wish I have the techinal proof to back this accusation, but sadly I don't, you just got to take my and other people's words for it.

[Ameador1]

Not sure what you mean - again, I manually configured them - one at a time directly on their web interface and not via Awesome Miner. I used Awesome Miner once 2+ months ago to push template settings to them, and when I saw the results - I configure them manually after that. I haven't touched them for close to a month now - only monitoring, during which time the have had restarts due to 1 or 2 power drops we had during that time - but again, a couple of weeks ago. The only recent activity was the restarts I did when I found them like this yesterday.

Hopefully it's not some BS from NH, but with the miners configured to maintain temps at 76 and plenty of fan power - I don't understand how NH could cause their temps to go high enough to cause them problems - and doubly so when AM logs don't show any temp spikes. But maybe the fans in this scenario kept the temps ok, even though the chips may have been had frequencies set crazy high? But how could NH have modified chip frequencies? And again for a virus or hackers - the hashing gain seems to have went to my NH account - not elsewhere. This is weird and that is why I am here.

[philipma1957]

so are they all set to viabtc or to ckpool as i ask?

do that first lets say one half to ckpool and one half to viabtc.


once you did that.

boot each one three times.

then  run the gear and monitor it.

see what happens.


if it sucks let us know.

if it works let us know.

so far every thing we suggest you simply disagree with us.

stop that.  just follow these ideas .

there are actual reasons we suggest it.

if i was at you place i could sit down and explain to you each and every reason.

but on the internet it is simply too time consuming.

I have yet to see a screen shot of any of this gear set to via btc or to ckpool.

also three boots tends to clear a lot of bullshit out of the ram in the controllers.

good luck 🍀

[Ameador1]

I am just now home and starting to work on it. So far many of the suggestions come with assumptions or comments that I clarified in my OP - so I am responding to those and also answering other ideas. Don't get so worked up. I said I would try these ideas when I got home - and that is what I am now doing. But, I am taking down settings, etc... before so I don't have to start from scratch. I had everything shut down until I could get to it, so bringing stuff back up piecemeal to do it.... relax!

Workin' on it...

[philipma1957]

just took three deep breaths.

i am calm.

thanks for suggestion to chill.

[mikeywith]

Not sure what you mean - again, I manually configured them - one at a time directly on their web interface and not via Awesome Miner.

This does not stop AM from actually changing the setting assuming there is a glitch in AW program itself (not the firmware).

Hopefully it's not some BS from NH, but with the miners configured to maintain temps at 76 and plenty of fan power - I don't understand how NH could cause their temps to go high enough to cause them problems...

If you don't have a second pool setting in place, and assuming for whatever reason the connection to NH is lost due to whatever reason, be it your side or theirs, the miners will run pretty hot and might restart themselves, that is a known problem in the default firmware I am not sure if Vnish works differently.

... and doubly so when AM logs don't show any temp spikes.

Assuming the above scenario is true, AM won't have access to the miner, it will show as offline as long as no pool is showing "alive" on the miner.


All the above is rather speculations with a pretty low probability of being the real cause, the most likely cause of all this mess is NH, we know that NH causes some chips to disappear or show xxxx, all those chips are connected to each other, so if a few fail the rest will too, that might also have to do with why the setting changed, maybe Vnish is set up to auto-tune when some chips fail and thus all of the previous setting you saved were overwritten.

As I mentioned before, there is really no technical explanation on how or why does NH cause the issue, the only semi-proper explanation we have so far is the non-stop changing for difficulty, I had the same problem using the default firmware, switched to a proper pool, some of chips never showed up again, some did and worked without an issue ever since.

Long story short: start by changing the pool.

Good luck.

[Ameador1]

Thanks @mikeywith!

I am still muddling through everything. I only have one miner back on at the moment and looking at the pools to figure out accounts, settings, how to deal with payouts, etc... BUT, the one miner that I brought up is obviously still connected to NH and it is running like nothing ever happened. I HOPE the others will follow suite, but I want to prepare for these other pools first, before bringing the others up that is.




Ok everyone - long story short - these miners seem to be back to normal with no changes! 

After finding them all out of kilter yesterday and restarting and power cycling a couple times (including one of those times for at least 45 minutes) with the external large fans still running - and still all messed up like described - I just shut it all down until I could see what everyone thought. I didn't want to damage them by leaving them on that way and it was getting late with work coming in the morning.

So, maybe a hiccup with Nicehash? Or they needed to be left off longer to "reset"? I don't know.

But, they came back up running on Nicehash - as usual with all the hash boards and chips back in operation at normal hash rates. I still setup a ViaBTC account and moved my 4 L3+/++ units as well as 4 of my S9's there and currently have the other 4 S9's on ckpool.

I appreciate the effort you guys offered and wish I knew better what happened and why it cleared up. Nicehash seems like a likely candidate from the comments so far with this ending as well - maybe they just fixed their bug?

Crazy!

[philipma1957]

Part of what is happening is nicehash  is paying really high for sha-256 hash.  Some hot shit coin is why that happens.

0.0152 is about what it should pay per ph

last few days it is paying  0.0181 and up.

this causes lots of switching as bots and renters will raise the price they are willing to pay.  So your s9  switches like mad  that is the #xnsnb code purpose it allows easy switching.

Right now nicehash is paying about 0.0160  which is not too much over the 'true'  price of 0.0151  so the switching is less.

if you see a lot of xxx mixed with ooo  in general  the gear still works.

If you don't see a board at all in general the board is dead.  but that is a generality maybe 70-30 or 75-25 true number.

[Ameador1]

I guess I was just surprised that with all the air flow I have and the target temps - this would have kept the good - but I suppose still in the deeper core of the chips they would be getting hotter. Which I suspect is what is happening to cause the X's as their performance drops.

Currently they are completely normal again - no X's, S's, missing boards, anything - all fine!

Thanks again!