Bitcoin Forum
December 03, 2016, 08:02:10 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: I think one of my GPUs has the the dreaded electromigration.  (Read 2920 times)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 06:22:56 PM
 #1

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
"Governments are good at cutting off the heads of a centrally controlled networks like Napster, but pure P2P networks like Gnutella and Tor seem to be holding their own." -- Satoshi
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
1480795330
Hero Member
*
Offline Offline

Posts: 1480795330

View Profile Personal Message (Offline)

Ignore
1480795330
Reply with quote  #2

1480795330
Report to moderator
Mousepotato
Hero Member
*****
Offline Offline

Activity: 896


Seal Cub Clubbing Club


View Profile
March 06, 2012, 06:26:44 PM
 #2

Hmm that's certainly something I've been worrying about as time goes on.  What I'm more curious though is how the hardware vendors will handle warranty claims against cards that are nigh irreplaceable at this point.

Mousepotato
rjk
Sr. Member
****
Offline Offline

Activity: 420


1ngldh


View Profile
March 06, 2012, 06:29:01 PM
 #3

Hmm that's certainly something I've been worrying about as time goes on.  What I'm more curious though is how the hardware vendors will handle warranty claims against cards that are nigh irreplaceable at this point.
From what I hear, 5970's are being replaced by 6990s when there is a warranty claim.

Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 06:30:20 PM
 #4

From what I hear, 5970's are being replaced by 6990s when there is a warranty claim.

Oh god.  Sad

At this point I am not sure I can even warranty it.  It runs perfectly stable at stock clock and I doubt any warranty guarantees stability at > than stock clock.
racerguy
Sr. Member
****
Offline Offline

Activity: 271


View Profile
March 06, 2012, 07:11:48 PM
 #5

Can it play games at stock clocks?  Mining is easier on the card than playing games.
malevolent
can into space
Staff
Legendary
*
Offline Offline

Activity: 1624



View Profile
March 06, 2012, 07:24:50 PM
 #6

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?
bulanula
Hero Member
*****
Offline Offline

Activity: 518



View Profile
March 06, 2012, 07:28:09 PM
 #7

On one of my rigs I kept noticing the same GPU (#4 grr) going DEAD.  I lowered the clock, dead, lowered clock, dead. Finally put it at only 750 Mhz it was fine for a couple days and then I noticed it dead again.

I verified it now is no longer stable @ even 750.   Three "dead"s in 24 hours.  I dropped it to 725 (stock) and it has been stable for almost a day now (will update post for longer periods of testing).  So far just that one core on one rig (1 of 48 gpus) dislikes even moderate overclock.  The other GPU on the same card is fine @ 800 MHz.  At higher overclocks the "bad core" dies almost instantly.  Those two facts and a 1200W single rail PSU design make me think it isn't a power issue.

This is a very old card (over a year old need to lookup sales date).  It was one of the first 5970s I bought, and it is what convinced me to replace all my other GPUs with 5970s to consolidate my growing farm.  It has never been overvolted but last summer it did run "hot" for about a month before I bought a mini-split system to supplement the house main AC.  It has run overclocked but nothing crazy.  I ran it (guestimates) 835 early on.  I dropped all the clocks to 820 and then 800 once I got up to 6 rigs because micro managing them became a hassle.   I can't try overvolting at this point (which can "restore" performance on circuits suffer electromigration) because it is in a Linux rig.

Other than 3 fan failures (and me dropping a motherboard down the stairs) this is the first casualty of mining I have had.   As our GPUs get older I wonder if anyone else has noticed something similar.

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

That is the important bit here : Linux does not properly support overvolting ( discard effects of EM aftewards ) / undervolting ( prevent EM in first place ) and reading VRM temps for 5870 / 5970.

That sucks and needs to be fixed. Sad
Buckwheet
Member
**
Offline Offline

Activity: 108


View Profile
March 06, 2012, 08:18:24 PM
 #8

I mentioned to you in my PM that recently the quality of the 5970s I have been getting on flea-bay have been sub par. I am seeing ~40% "failure rate" of the 5970s in the past ~30 days in the ones I got. Most of the failures have to do with the card starting up just fine and running without problems for about two to three minutes, and then the temp gauges suddenly go from 60-65C to 120ishC and the card stops responding and crashes out the whole system.

When I set the clocks down to normal levels the card will run without issue, but it runs 10-15C hotter then the 5970s next to it that are overclocked to 800. They are all spaced out with enough room between the cards to have two dual slot cards between them. I also have box fans blowing in cool air into the rack they are mounted on so I know its not a heat problem.

I also have AX1200s for all my rigs and I still experienced this problem when I did the following:

1. Brand new motherboard of the same make/model.
2. Replaced riser.
3. Replaced PSU with brand new AX1200
4. Replaced outlet.
5. Replaced breaker.
6. Clean install.

It "feels" like people run into this issue and since they are out of warranty they throw the card on Ebay and sell it off. It could also be pure luck that I have gotten bad cards lately. 5870s all seem to the fan or fan controller that are burning out on me a couple weeks after getting them.

Edit: It happens to me any any mem clock. I don't have a good number on specific temps.
The-Real-Link
Hero Member
*****
Offline Offline

Activity: 530



View Profile
March 06, 2012, 08:22:49 PM
 #9


  One of my cards is seeming to have a dead core quite often but IDK if it's due to EM or simply a dud card.  Diamond's looking into it though so we'll see.  I wonder if EM is the issue of the second card in that system though as it randomly sees and doesn't see the cores.  Either way, sending both back. 

  Can't vouch for standard clock instability really so far.  I'll get back to you in a year Smiley

Oh Loaded, who art up in Mt. Gox, hallowed be thy name!  Thy dollars rain, thy will be done, on BTCUSD.  Give us this day our daily 10% 30%, and forgive the bears, as we have bought their bitcoins.  And lead us into quadruple digits
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 08:30:35 PM
 #10

That is the important bit here : Linux does not properly support overvolting ( discard effects of EM aftewards ) / undervolting ( prevent EM in first place ) and reading VRM temps for 5870 / 5970.

On correction.  Linux has no problem undervolting.  I have on rig running @ 730/240 1.00V (vs 1.05V stock) as a test.  cgminer has no problem setting values allowed by voltage and 5870/5970 allow core voltage to be lowered significantly (I tested down to 0.6V and it ran fine). However it relies on ADL library and as such only will run values allowed by bios so no overvolting without bios flash to increase the allowable range. 

Would be nice if AMD exposed all functionality via drivers and allowed larger range of values.  It would make 3rd party hack utilities obsolete.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 08:32:30 PM
 #11

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.
malevolent
can into space
Staff
Legendary
*
Offline Offline

Activity: 1624



View Profile
March 06, 2012, 08:42:31 PM
 #12

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

You had same cards/same settings/same environment when you used windows?
As for the memclock I had a card where it started to lock up at 280MHz after some time, but it did not occur when I increased the mem clock to 350.
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 08:44:04 PM
 #13

Does it happen at different mem clocks too?
How high were the temperatures (vrms too)?

Haven't change memclock to see if that help/hurts I doubt it will make a difference but will look into it.

temps were mid 60C.  Linux has no way to see VRM temps but based on Windows experience of similar clocks that likely puts VRMs in the mid 80C range.

You had same cards/same settings/same environment when you used windows?
As for the memclock I had a card where it started to lock up at 280MHz after some time, but it did not occur when I increased the mem clock to 350.

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.
malevolent
can into space
Staff
Legendary
*
Offline Offline

Activity: 1624



View Profile
March 06, 2012, 08:47:45 PM
 #14

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

48 cards, that's quite a lot and an increased chance you stumble upon one that is of lower quality (as we know every chip is different, same as with CPUs)
DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 08:50:40 PM
 #15

Yeah.  I will try but given 47 or 48 cards work at sub 300 Mhz and one card dies instantly at clocks it ran fine before I doubt it is memclock is too low.

48 cards, that's quite a lot and an increased chance you stumble upon one that is of lower quality (as we know every chip is different, same as with CPUs)

Well 48 chips, 24 cards. Smiley

Yeah it may be the "weak link" just thought it was interesting as this card has been running 24/7 for over a year now.  I figure these topics will become more common place as the oldest GPUs hit 18 months old, 24 months old, etc.
coretechs
Donator
Sr. Member
*
Offline Offline

Activity: 362



View Profile
March 06, 2012, 10:04:15 PM
 #16

One of my 5970s has a similar problem.  It wont mine at 725mhz/150mhz mem/1.05v without crashing, but it runs fine at 690mhz/150mhz mem/0.95v.  I always assumed it was a fried VRM because the GPU temps are fine.

http://bitcoindoc.com - The Rise and Rise of Bitcoin | http://nxtportal.org - Nxt blockchain explorer
cablepair
Hero Member
*****
Offline Offline

Activity: 854


https://btc-republic.com/index.php?ref=cablepair


View Profile WWW
March 06, 2012, 10:25:45 PM
 #17

5970s are notorious for that

I have one with the same problem, I actually found a (discontinued) Accelero Xtreme 5970 some place and am waiting for it to be delivered, I hope it gives me the cooling I need to keep it mining for a few more months

I know the accelero xtreme I used on one of my 5870s that the fan died was an amazing difference

this thing can mine 24/7 clocked at 950/180 - give out 430 mhash at 42 degrees (c) with 50% fans! - amazing!

DeathAndTaxes
Donator
Legendary
*
Offline Offline

Activity: 1218


Gerald Davis


View Profile
March 06, 2012, 10:25:56 PM
 #18

One of my 5970s has a similar problem.  It wont mine at 725mhz/150mhz mem/1.05v without crashing, but it runs fine at 690mhz/150mhz mem/0.95v.  I always assumed it was a fried VRM because the GPU temps are fine.

Hmm it could be a bad VRM.  I can't check VRM temps because it is in Linux.  I may drop the card in my Windows workstation to see what VRM is reading.  If one of 3 VRMs is missing in GPU-Z that is a good (well bad but clear) sign.
coretechs
Donator
Sr. Member
*
Offline Offline

Activity: 362



View Profile
March 06, 2012, 11:31:55 PM
 #19

If one of 3 VRMs is missing in GPU-Z that is a good (well bad but clear) sign.

It's better than electromigration because you can still use the card and go for max hash/watt on air by underclocking/undervolting as long as the functional VRMs don't get overloaded.  It's unfortunate for your water-cooled setup though.  Good luck in either case.

http://bitcoindoc.com - The Rise and Rise of Bitcoin | http://nxtportal.org - Nxt blockchain explorer
AniceInovation
Donator
Sr. Member
*
Offline Offline

Activity: 314


Interesting.


View Profile
March 07, 2012, 03:50:56 AM
 #20

Not trying to be obvious, but i did get some of these problems when i was messing with some cards.
Turned out to be bad thermal paste, or i somewhat moved the coolers, destroying the timms.

If you didn't, remove the heat-sink and re-paste it, and check if it gets better.

Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!