Electrical box failure? Some circuits now read 71V while others remain 125V

[DrG]

So today at 5:30PM over 1/2 my rigs went offline.  I have them mining at my wife's work and most of them have been going on almost 2 years now without problem. Only recently (2 weeks ago) did I start noticing one of the breakers tripper a couple times when an employee of hers used a microwave I'm guessing.

Some event happened in the evening and 1/2 my rigs powered off.  I got to the office 2 hours later.  I figured maybe the power went out quickly (brownout) and some of the rigs managed to stay online without reseting.  The I notice some of the box fans I use on my 4 card rigs aren't spinning fast and then I knew I had a power problem.  I go to her server closet and the power backups is just on it's very last leg of juice (the server had already powered down and only the switch and router were drawing power).  I found it odd that the powerbackups remained on though.  I cycled through the little LCD menu on the APC 1500 (older model) and it showed the input voltage as 71v, output 125v.  So I unplugged it and moved it to a different socket.  Same thing, 71v.

I shut off all the breakers (there's no main breaker that controls everything so I did each one).  Then turned them all on slowly.  Still voltage input showed 71v.  Some miners were still mining when I first got there so I moved the UPS around and found that roughly 1/2 the circuits showed 125V input to the UPS, the other 1/2 showed 71 or 72v.

So I'm guessing something went bad in the panel or the breakers. It has been getting hotter here in SoCal and I had been placing a higher load with the last set of 7950s I got last month. Perhaps the heat and load were too much.  My dad is an EE, but hasn't done any real electrical work in 20 years (he went to defense contractor programming).  I'll ask him later tonight about this.

The panel appears to be a 3 phase commericial panel.  It has a couple dedicated circuits for 240V like an X-Ray machine, a vacuum pump and air compressor.  The rest are standard 110 or 120V lines.

Any ideas on what went bad?  My friend who knows more than me (that's not saying much) said either several breakers went bad, or the bars in the panel might have been burned or even the lines leading to the bars.

The office is in a 20k sq commercial 2 story building.  My wife renovated the office in 2005 for 150k, but the first contractor was a scammer.  Don't know the quality of the second but he did get the job done.

Any ideas? Suggestions? Repair estimates?

[jimmydorry]

Wow bad luck.

Nothing like stress testing to find out how close things actually come to their rating. 

[AlexBBB]

Check your "common" wire. I am sure, it is burned/disconnected from main cable.
In this case, all your equipment, connected between 2 different phases become to connect serial, and divide voltage between 2 different phase wires. So voltage will be float in case of power consumption between this two set of equipments, connected serially.

I am sure 99% this is your case.
It's frequently happening for old old buildings.

P.S. Be careful, because voltage between 2 phases higher then normal 1 phase voltage (by ~1,7 coefficient), so you can easy damage your equipment.

[centove]

Actually it's more then likely the breaker that feeds that panel has had a phase burn out. I.E. The 'main' breaker feeding that sub panel needs to be changed. I bet if you measure the circuits that have the full voltage vs the ones that don't you'll see a predictable pattern.
If the netural was burned out then nothing would work. Also if you measure the 240 double pole circuits more then likely they will read around 185 or so (125 + 72).

[camaro69327]

The panel appears to be a 3 phase commericial panel.  It has a couple dedicated circuits for 240V like an X-Ray machine, a vacuum pump and air compressor.  The rest are standard 110 or 120V lines.

More then Likely ONE of the 3 fuses blew....consult an Electrician....

[DrG]

Looks like the whole building went down.  Waiting on Edison to swap lines to the transformer.  First time in 2 years that I've been cold in that office 

[centove]

Well a broken transformer would do it as well

[DrG]

You ought to have an electrician look at what you are doing.

If you are putting an unbalanced load on a 3 phase system you could easily burn out the transformer.

It's a commercial building with an elevator and was going to have a dialysis center remodel (that got stuck 1/2 way).  Even with 20-30GH in GPU mining rigs my pull is not what brought down the transformer.  It's 35-40 years old, they might be changing it out today, but will have to dig up 500 feet of pavement to get to the main line (44k I think coming from the street lines).  Either way the city's crew is on it (about 15 guys and 6 trucks).  I think my sub-panel is fine, just need to wait for them to restore power and see if everything starts up.  They shut the whole building down and my wife is doing braces by the window  )

So my rigs are all offline 

[Littleshop]

It is the transformer for the power company or building not you. 

I had this happen to me once where one side of my panel was like that and the other fine.  It was happening to my neighbors too.

[Soros Shorts]

You ought to have an electrician look at what you are doing.

If you are putting an unbalanced load on a 3 phase system you could easily burn out the transformer.

It's a commercial building with an elevator and was going to have a dialysis center remodel (that got stuck 1/2 way).  Even with 20-30GH in GPU mining rigs my pull is not what brought down the transformer.  It's 35-40 years old, they might be changing it out today, but will have to dig up 500 feet of pavement to get to the main line (44k I think coming from the street lines).  Either way the city's crew is on it (about 15 guys and 6 trucks).  I think my sub-panel is fine, just need to wait for them to restore power and see if everything starts up.  They shut the whole building down and my wife is doing braces by the window  )

So my rigs are all offline 

More than 5% imbalance on a phase can damage a transformer.  So you're little 30GH of GPUs could easily be the root cause of the failure.  I'd make sure you don't power that gear on until long after the crews have left.  If they are on the ball and measure the neutral line current you are dumping, you could be the happy owner of a 100k bill for transformer replacement.

How could a sub-unit of a commercial building be responsible for the balance of a 3-phase transformer feeding the entire building? It doesn't even sound like he has access to all 3 phases.

[DrG]

You ought to have an electrician look at what you are doing.

If you are putting an unbalanced load on a 3 phase system you could easily burn out the transformer.

It's a commercial building with an elevator and was going to have a dialysis center remodel (that got stuck 1/2 way).  Even with 20-30GH in GPU mining rigs my pull is not what brought down the transformer.  It's 35-40 years old, they might be changing it out today, but will have to dig up 500 feet of pavement to get to the main line (44k I think coming from the street lines).  Either way the city's crew is on it (about 15 guys and 6 trucks).  I think my sub-panel is fine, just need to wait for them to restore power and see if everything starts up.  They shut the whole building down and my wife is doing braces by the window  )

So my rigs are all offline 

More than 5% imbalance on a phase can damage a transformer.  So you're little 30GH of GPUs could easily be the root cause of the failure.  I'd make sure you don't power that gear on until long after the crews have left.  If they are on the ball and measure the neutral line current you are dumping, you could be the happy owner of a 100k bill for transformer replacement.

I'm pretty sure the Botox/Laser clinic on the other side of the building uses more power.  They have a glass waterfall/fountain/pump that uses 1KW alone.  I doubt the 10KW pull my rigs had do anything.  The suite next door was set up to be a dialysis unit - those machines run on 240V and suck huge amounts of power, as does the hydroelectric elevator.

Anywho, they dug up the parking lot and replaced the cables leading to the transformer - 50 year old cable was going bad.  Transformer was fine and wasn't changed, just replaced some risers on the main pole.

Only thing is the thermostat reset and now I have no AC - waiting for engineering to come down and turn it on otherwise it gets toasty in the office.

I was hoping BFL would have shipped my ASICs in October, November, December, January, February, March, April (see, I left that last little "l" out for the glimmer of hope but I don't have any jalapenos ordered  ).