I think one of my GPUs has the the dreaded electromigration.

[DeathAndTaxes]

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.

[Mousepotato]

Hmm that's certainly something I've been worrying about as time goes on.  What I'm more curious though is how the hardware vendors will handle warranty claims against cards that are nigh irreplaceable at this point.

[rjk]

Hmm that's certainly something I've been worrying about as time goes on.  What I'm more curious though is how the hardware vendors will handle warranty claims against cards that are nigh irreplaceable at this point.

From what I hear, 5970's are being replaced by 6990s when there is a warranty claim.

[DeathAndTaxes]

From what I hear, 5970's are being replaced by 6990s when there is a warranty claim.

Oh god. 

At this point I am not sure I can even warranty it.  It runs perfectly stable at stock clock and I doubt any warranty guarantees stability at > than stock clock.

[racerguy]

Can it play games at stock clocks?  Mining is easier on the card than playing games.

[malevolent]

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

[bulanula]

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

That is the important bit here : Linux does not properly support overvolting ( discard effects of EM aftewards ) / undervolting ( prevent EM in first place ) and reading VRM temps for 5870 / 5970.

That sucks and needs to be fixed.

[Buckwheet]

I mentioned to you in my PM that recently the quality of the 5970s I have been getting on flea-bay have been sub par. I am seeing ~40% "failure rate" of the 5970s in the past ~30 days in the ones I got. Most of the failures have to do with the card starting up just fine and running without problems for about two to three minutes, and then the temp gauges suddenly go from 60-65C to 120ishC and the card stops responding and crashes out the whole system.

When I set the clocks down to normal levels the card will run without issue, but it runs 10-15C hotter then the 5970s next to it that are overclocked to 800. They are all spaced out with enough room between the cards to have two dual slot cards between them. I also have box fans blowing in cool air into the rack they are mounted on so I know its not a heat problem.

I also have AX1200s for all my rigs and I still experienced this problem when I did the following:

1. Brand new motherboard of the same make/model.
2. Replaced riser.
3. Replaced PSU with brand new AX1200
4. Replaced outlet.
5. Replaced breaker.
6. Clean install.

It "feels" like people run into this issue and since they are out of warranty they throw the card on Ebay and sell it off. It could also be pure luck that I have gotten bad cards lately. 5870s all seem to the fan or fan controller that are burning out on me a couple weeks after getting them.

Edit: It happens to me any any mem clock. I don't have a good number on specific temps.

[The-Real-Link]

  One of my cards is seeming to have a dead core quite often but IDK if it's due to EM or simply a dud card.  Diamond's looking into it though so we'll see.  I wonder if EM is the issue of the second card in that system though as it randomly sees and doesn't see the cores.  Either way, sending both back. 

  Can't vouch for standard clock instability really so far.  I'll get back to you in a year

[DeathAndTaxes]

That is the important bit here : Linux does not properly support overvolting ( discard effects of EM aftewards ) / undervolting ( prevent EM in first place ) and reading VRM temps for 5870 / 5970.

On correction.  Linux has no problem undervolting.  I have on rig running @ 730/240 1.00V (vs 1.05V stock) as a test.  cgminer has no problem setting values allowed by voltage and 5870/5970 allow core voltage to be lowered significantly (I tested down to 0.6V and it ran fine). However it relies on ADL library and as such only will run values allowed by bios so no overvolting without bios flash to increase the allowable range. 

Would be nice if AMD exposed all functionality via drivers and allowed larger range of values.  It would make 3rd party hack utilities obsolete.

[DeathAndTaxes]

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

[malevolent]

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

You had same cards/same settings/same environment when you used windows?
As for the memclock I had a card where it started to lock up at 280MHz after some time, but it did not occur when I increased the mem clock to 350.

[DeathAndTaxes]

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

You had same cards/same settings/same environment when you used windows?
As for the memclock I had a card where it started to lock up at 280MHz after some time, but it did not occur when I increased the mem clock to 350.

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

[malevolent]

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

48 cards, that's quite a lot and an increased chance you stumble upon one that is of lower quality (as we know every chip is different, same as with CPUs)

[DeathAndTaxes]

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

48 cards, that's quite a lot and an increased chance you stumble upon one that is of lower quality (as we know every chip is different, same as with CPUs)

Well 48 chips, 24 cards.

Yeah it may be the "weak link" just thought it was interesting as this card has been running 24/7 for over a year now.  I figure these topics will become more common place as the oldest GPUs hit 18 months old, 24 months old, etc.

[coretechs]

One of my 5970s has a similar problem.  It wont mine at 725mhz/150mhz mem/1.05v without crashing, but it runs fine at 690mhz/150mhz mem/0.95v.  I always assumed it was a fried VRM because the GPU temps are fine.

[cablepair]

5970s are notorious for that

I have one with the same problem, I actually found a (discontinued) Accelero Xtreme 5970 some place and am waiting for it to be delivered, I hope it gives me the cooling I need to keep it mining for a few more months

I know the accelero xtreme I used on one of my 5870s that the fan died was an amazing difference

this thing can mine 24/7 clocked at 950/180 - give out 430 mhash at 42 degrees (c) with 50% fans! - amazing!

[DeathAndTaxes]

One of my 5970s has a similar problem.  It wont mine at 725mhz/150mhz mem/1.05v without crashing, but it runs fine at 690mhz/150mhz mem/0.95v.  I always assumed it was a fried VRM because the GPU temps are fine.

Hmm it could be a bad VRM.  I can't check VRM temps because it is in Linux.  I may drop the card in my Windows workstation to see what VRM is reading.  If one of 3 VRMs is missing in GPU-Z that is a good (well bad but clear) sign.

[coretechs]

If one of 3 VRMs is missing in GPU-Z that is a good (well bad but clear) sign.

It's better than electromigration because you can still use the card and go for max hash/watt on air by underclocking/undervolting as long as the functional VRMs don't get overloaded.  It's unfortunate for your water-cooled setup though.  Good luck in either case.

[AniceInovation]

Not trying to be obvious, but i did get some of these problems when i was messing with some cards.
Turned out to be bad thermal paste, or i somewhat moved the coolers, destroying the timms.

If you didn't, remove the heat-sink and re-paste it, and check if it gets better.