Mining disaster with 34 L3+ ASICs, need some serious help!!!!

[Stubby5000]

I had a horrible thing happen today and I need guidance from someone who understands the technical aspects of an Antminer L3+. I have 9 of these running in my attic and recently I added 25 in the garage. Most of the miners are running a moderate overclock to ~600 MH/sec using about 1000W. Today I came home to find every single miner displaying a flashing red "fault" light and a solid red light on the front of the control board. I have absolutely no idea what happened. It wasn't a particularly hot day. There were no thunderstorms. My wife was home most of the day and reported nothing out of the ordinary and said around noon all the miners in the garage "had a green light." None of the circuit breakers in my house were tripped. I have the miners hooked to various 230v and 120v circuits throughout the house via heavy duty extension cords. I've done the math and none of the circuits should be overloaded. I have a 100 amp sub panel in the attic that usually draws around 85-90 amps. I also have a "whole house" surge protector for the attic service. In fact since I had just brought all the garage miners online the night before, I had my electrician go by this afternoon and measure my whole house and attic-only load: it was about 186 amps whole house (I have 200 amp service) and 92 amps in the attic. When I got home all the other electrical components in the house were working, with one exception: a fan I was using in the garage had stopped working, even though it was plugged into a seemingly working outlet. I switched outlets and it started working again. All the lighted ends of my extension cords looked normal. When I got home today I quickly unplugged all the miners and tripped then reset every breaker in the house. Now only a handful in my row of 25 miners in the garage power up. The ones that don't work have a weak flicker of lights on the back when initially plugged in, then the fault and control board lights come on. The rear fan spins (slowly) but not the exhaust fan. Interestingly, the miners in the attic fared better. Only 4 of them don't power up.

Can someone shed light on just what the FUCK has happened here??? I don't understand it. Is it from the overclocking? But why did everything fail at once? My wife was charging her electric car earlier in the day, did that screw something up? And what was up with that fan in the garage? Finally, most importantly, what do I do now??? I just took delivery of the garage batch a few days ago, but I gather the warranty is void because of the overclocking... Is it the control board that has gone bad? If so BITMAIN sells them for $60 each, which wouldn't be too painful of a fix.

I really need some help with this. Thanks in advance if you have any information that might help.

[1714199002]

1714199002

[1714199002]

1714199002

[Marvell2]

It could be a heat problem i heard l3 plus is sensitive to temps over 75

I still cant belive you ran 30 l3 in a house lol on 200 amp no less , just get a warehouse or host em bro

[smoolae]

Maybe You can get some help from here https://bitmainhelp.zendesk.com/hc/en-us/articles/220872007-Possible-problems-for-Antminer-Troubleshooting-for-S7-S9-L3-

Perhaps something happened with some PSUs (don't know how)? You could try taking a PSU from a working L3 and connect it to the one(s) that is(are) not working. If You have a multimeter in Your household, You could test the voltages from the PSU (the output should be between 11.6-13V).

Hope everything works out for You!

[gigidinizza]

Clearly can't just be coming from the machines, and the only thing in common they have is the electric supply, so before you look forward into a hardware problem might be a good idea to control your house first.

[smoolae]

Also, how many amps does the electric car being charged pull?
I saw some articles that gave me numbers 220-240V and 16-or 32-Amps for home charging points. I do not know how the specs of Your charging point, but if it is rated up to 240V 32Amps for charging, then it was some mad wattage being used with all the L3+'s.

If You plug the garage fan in the old outlet (where it was originally) does it work normally?

[Meltingin364]

So you have 200 amp service to the house. You also have 100 amp service to the attic. First, if that 100 amps to the attic is on a sub panel you probably only have 200 amps. If it is on a separate meter then you probably have 300 amps. Most circuits in a house are either 15 or 20 amp circuits. Each circuit generally has multiple outlets. Regardless of the number of outlets each 15amp circuit will supply a max output of 1800 watts. so on a 15amp circuit you could operate 2 L3+ at 800watts each. You would really be pushing it here as that is about 90% of the rated power output of that circuit. On a 20 amp circuit you could put 3 in theory but that would a 100% capacity use of that circuit so 2 is probably the limit. Remember, this is per Circuit, Not Outlet! So no matter how many outlets you have you must abide by this limit per circuit. of course, putting this many devices on each circuit assumes you will not turn on a light, run the fridge or dishwasher, you know, use any of the stuff we know you (and more particularly your wife who is a normal and rational human being, she did not sign up for your sudden mining fetish) are using throughout the day. If you go out to your panel and simply count the breakers you will soon see that running 35 (requires 18 separate 15-20 amp breakers )of these machines on your household electric would be an unlikely proposition in theory and nearly impossible in practice. I think plugging in the electric car ( not your wifes fault, she lives there remember) was the 50Lbs bag of cement that broke the Camels back. Also, without seeing your electric service in person I have a sneaking suspicion that either your electrician does not understand the draw of these units or he/she is not competent for the task. For 35 units you need a dedicated electric service panel and a lot juice. 28000 watts is about 60% of the max capacity of 200 amp service. but you must have the circuits to support the load.
I hope I am wrong but I think you only option is to return the non-working units for service and pay bitmain to replace the boards.
Post a pic of your meter and electrical panels and I am sure we could get you a definitive answer.

Sorry this happened to you, I hope I am wrong.

[Stubby5000]

Thanks for the answers. The miners in the garage were a temporary proposition, for two weeks at the most. I'm aware of what you said about the way electrical circuits work. Here are some details of my setup:
The attic does not have a separate meter (I'm told this is not legal where I live). Thus the 100 amps going up there are a part of the overall 200 amp service. My permanent attic setup consists of six 230v circuits on which I have the capacity to run four machines. Currently I have only 15 up there. There are a few 120v 20 amp circuits up there for fans and whatnot. I also routed the circuit to my sup pump (which I don't have) to the attic, which is also a 20 amp line. To power the temporary miners in the garage, I ran four heavy duty 230v extension cords to the garage, plus 120v extension cords rated at 20 amp from some of the attic 120v circuits, plus the former sump pump circuit. I never power more than two miners (about 2000 watts total) on a 20 amp 120v circuit, and never more than four on the 230v 20 amp lines. Then I used a number of unused (or lightly used) 20 amp circuits from the rest of the house to power a number of the garage miners. Finally I had my electrician make a custom cord that plugged into my dryer outlet, which is a 30 amp 230v line. His cord split into three NEMA 5-20 230v outlets, which I further split to power two miners each, 6 miners total. All the extension cords and splitters I used were heavy duty and were rated to handle the amps/watts being asked of them.
Even if the wife and her EV pushed the house past the brink, it should have simply tripped the main 200 amp breaker. Why would that have done damage to the miners? That's the point of a breaker.
With regards to heat, I doubt this was the problem. I had all the miners lined up next to each other (which they were made to do, since they have tracks on the side that fit into each other) and they were exhausting directly out the garage door, which was open just a crack. There were also a couple fans blowing across the miners themselves. In the past, before my attic setup was complete, some of those miners/PSUs would get extremely hot. When this happened, either the miner or the PSU would shut down, as it is designed to do. This happened many times. I don't understand why this wouldn't have happened again if heat was the issue.
And by the way, I did an experiment where I took the power cord from a working machine and plugged it into a non working machine. No luck. Then I took the PSU from the working machine and hooked it to the non working machine. No luck still. So whatever the problem, it does seem to be the miner itself. Unfortunately right after this happened I had to leave town for the weekend, so I haven't had a chance to play around any further than that. One thing that gives me hope: when I came home, I unplugged everything and then immediately plugged them back in, and several machines came back to life. I'm hoping when I come home Sunday afternoon, even more will will work again.
I'll keep you all updated. Again, thanks for the input.

[Stubby5000]

I wonder if the electric company is throttling my service or something? That would explain why all of them were affected (and the fan too). Maybe some machines are more sensitive to decreased wattage/voltage than others, which explains why now some work and some don't? And that would explain why the lighted ends of the extension cords still work, yet they won't power a miner. It's clear whatever happened was a result of some house electrical problem (again, the fan) but the question is, did it do lasting damage to the miners.
Also, an electrical surge doesn't seem likely since miners in both the attic (behind a whole-house surge protector at the sub-panel) and miners in the garage (no surge protection) were affected. Other than a surge, what sort of electrical derangement can permanently damage a miner?

[GaryH]

you're gonna have to figure out what is wrong with the miner.  It is power supply, control board or something else?  Once you know that, you can probably backtrack and figure out the issue.

[Marvell2]

I wonder if the electric company is throttling my service or something? That would explain why all of them were affected (and the fan too). Maybe some machines are more sensitive to decreased wattage/voltage than others, which explains why now some work and some don't? And that would explain why the lighted ends of the extension cords still work, yet they won't power a miner. It's clear whatever happened was a result of some house electrical problem (again, the fan) but the question is, did it do lasting damage to the miners.
Also, an electrical surge doesn't seem likely since miners in both the attic (behind a whole-house surge protector at the sub-panel) and miners in the garage (no surge protection) were affected. Other than a surge, what sort of electrical derangement can permanently damage a miner?

yeah take one of the bad miners to a working outlet and see if it works , do this one by one to trouble shoot all of them and separate the bad from the good.

I have a 50 amp surge protection on my panels as well and it has saved my equpiment from brownouts several times

btw you would have got a letter from the power company , if there was an issue and they cant throttle your power , its either on or off unless they come out and fuck with the meter

[xhomerx10]

I wonder if the electric company is throttling my service or something? That would explain why all of them were affected (and the fan too). Maybe some machines are more sensitive to decreased wattage/voltage than others, which explains why now some work and some don't? And that would explain why the lighted ends of the extension cords still work, yet they won't power a miner. It's clear whatever happened was a result of some house electrical problem (again, the fan) but the question is, did it do lasting damage to the miners.
Also, an electrical surge doesn't seem likely since miners in both the attic (behind a whole-house surge protector at the sub-panel) and miners in the garage (no surge protection) were affected. Other than a surge, what sort of electrical derangement can permanently damage a miner?

yeah take one of the bad miners to a working outlet and see if it works , do this one by one to trouble shoot all of them and separate the bad from the good.

I have a 50 amp surge protection on my panels as well and it has saved my equpiment from brownouts several times

btw you would have got a letter from the power company , if there was an issue and they cant throttle your power , its either on or off unless they come out and fuck with the meter

 Surge protection doesn't normally protect equipment from brownouts.  What product have you installed?

[yobigd20]

Also, how many amps does the electric car being charged pull?
I saw some articles that gave me numbers 220-240V and 16-or 32-Amps for home charging points. I do not know how the specs of Your charging point, but if it is rated up to 240V 32Amps for charging, then it was some mad wattage being used with all the L3+'s.

If You plug the garage fan in the old outlet (where it was originally) does it work normally?

tesla pulls 80A 240V continuous on their home HPWC charger.

[Stubby5000]

Our car charging setup is nothing like that. It's a Volvo XC90 (plug-in hybrid) and we don't even have a 230v charging station. We use the 120v cord that comes with the car and plugs into a regular outlet. It was using a 20 amp circuit that had no miners on it. But again, even if the car charging pushed things past what the house can handle, I don't see why that would damage the miners. It should cause the main breaker to trip and that's it, right? Am I missing something?

[jstefanop]

Why would you overclock these the day you get them? If you had them at 600 MH thats a massive overclock, not a "moderate" one. Sounds like your garage fan going off cause them to overheat...you probably blew the fets on your boards or worse. What temps were inside your garage, and what temps were the boards reporting before this happened?

[bitcoin9999]

I think your internet is down.  Have you checked your internet connections?

[sil2222]

 I had a similar situation with a D3 blinking light would not initiate connection would not hash and it turned out to be a issue with power. The miner was not receiving enough power to start up so it would sit in stand by mode. Once it was moved all was working perfectly.

[qctechno]

I think your internet is down.  Have you checked your internet connections?

Red light on every miners ... Same answer, reboot your router, ethernet switches, mostly you lost internet connection or pool is down.

[Stubby5000]

It wasn't hot in the garage at all. The garage door was open just enough for the exhaust fans of the miners to stick out. I had a side door open with multiple fans arranged such that cool air would be pulled from outside and blown across the miners. Miner temps were in the low to mid 70s. I overclocked them the day I got them because I'd had such success overclocking my attic miners (where it's much hotter BTW). For weeks I was running even more aggressive overclocks with no problems at all. I can't imagine overheating as a result of overclocking is the problem. They wouldn't all fail simultaneously like that, plus I know for a fact the miners will shut off before they let themselves overheat. The one fan that went out wouldn't have had much of an effect on airflow.

[qctechno]

What P/S are you using ?  600MHs is 20% OC from stock speed.
I first burn-in my miners for 30 days for stability check before any OC.
But ATM I run mine at stock speed.

[Stubby5000]

I'm using BITMAIN's PSU, the single one (A3) for each miner.