Mining rig slowly decreases hash-rate over time

[mccminer]

I've got a dedicated open-air mining rig with 3 Gigabyte 7970's in it, that has been running just fine until about a week ago.  I've noticed that the hashrates per card will start to decrease after some time.  Right now, they will run at the rate I've run them at for the last month with no problems for about 12 hours, and then I start to see small bits drop off of all three cards at the same rate until the rig crashes requiring a manual powercycle.  GPU temps are not an issue.  I keep track of them with a cron job and all of them have been staying between 60C - 74C.  I've used extensive logging with cgminer and can't identify anything going wrong in it.  When it stops, it just stops. I have a similar setup sitting right next to it with two of the same 7970's in it, and it's still running rock-solid.  Both units have dedicated 1300W Rosewill PSU's.  Both units are running Ubuntu 12.10 and cgminer 3.0.1.  Any ideas?

[1714618500]

1714618500

[1714618500]

1714618500

[taenaive]

looks like you have the cards gone bad or something is wrong with your system. have you try swapping them around to see what happens?

[mccminer]

I dropped the intensity from 13 to 10 last night, and it mined all night with no problems.  And then, out of nowhere this morning, one of the GPU's went SICK then DEAD, and I had to reboot.  I just switched some cards around trying to see if I can isolate an issue.  Temps all looked fine over night.  For some reason once it reaches about the 12 hour mark something goes wrong.  This miner was running fine for a long time. 

[BBQKorv]

How is your mobo temps?

[mccminer]

Mobo temps look fine too.  I think it may all be related to one of the cards that I had tweaked a little bit higher.  I dropped my core/mem settings on all three cards and left the Intensity at 10.  Going to let it run for a while and see what it does.  That's the hard part, is that it'll run for 12 hours or so before locking up, so it's a lot of 'wait-n-see'! LOL!

Thanks for the suggestions guys.  Keep them coming.

[mccminer]

18+ hours on new lower settings!     Hash rate is rock steady on all three cards now.

[Hippie Tech]

Do you have a psu tester ?