After a recovery, everything is back in order. Looks like a PSU problem of the paper clip method not secure enough to keep the PSU stable.
That's interesting. Was the unit resetting? The PS_ON signal is just a TTL level, it's weird that if it worked a little but not at high power levels.
I was using 2 HX850 psus. I think one of the wires on the 24pin adapter is loose. I even tried one of the antminer switches and that's where it really showed up as the led would come on and then go off with just the slightest movement. I put it on a different psu and it's been rock solid ever since.
Edit: Yes the unit was probably resetting since it wasn't hashing above 500gh/s. But by the time I would notice, it wasn't hashing at all.