Mining rig keeps freezing, need some outside opinions

[nick1803]

Rig:

5x Zotac Amp Extreme 1070 ti
1x PNY Dual Fan 1070 ti
2x EVGA G2 Gold 750 using the Add2PSU adapter.  One PSU powers 3 cards and all the risers.  The other PSU powers 3 cards and the mobo/SSD
4gb Ballistix DDR4
Asus Z270-A
Intel Celeron CPU
120GB Adata SSD
Windows 10 64bit
Nicehash Miner



So I initially got this rig without the other 5 Zotac cards so I just set the motherboard on a box and used the single PNY card for a couple of weeks while I waited for the backordered Zotacs.  Once I got a stable OC on the card, I ran the system without issue with the card plugged straight into the motherboard, no riser.  

Last Thursday I got my other 5 cards so I fully assembled the rig.  I got all the components in the frame and all the cards on the risers.  Booted up fine and I got all 6 cards detected by doing the bios mods for this mobo that are on youtube.  Everything looking good I thought.

I put a mild OC on the cards and started mining.  About 12 hours later the machine was frozen, locked up.  I rebooted and halved the mild OC to just +100 core and +200 mem.  Started mining again, froze.  Ok...so I did no overclocks at all, stock speeds.  80% power in MSI afterburner this whole time, FYI.  Started mining again, froze up in just a few hours.  What the heck?  Then I tried one more time, stock speeds but this time 90% power instead of 80...although 80 should be just fine especially with no OC.  Froze after 10 hours.  

I don't get it.  I'm pretty sure my motherboard and RAM are fine, considering the machine ran for a couple weeks with no issue when I just had the single PNY card.  I have 6 extra risers for a new 1050 ti build I have coming this week, should I try swapping out the risers and see?  Anything I'm overlooking here?  

Its so frustrating to see your rig frozen up on your dashboard and you are helpless at work. Sigh

[1715087844]

1715087844

[Bigpiggy01]

What miner are you using and how much windows bloat have you killed off from background processes and is virtual mem set to 16Gb or more?

[nick1803]

What miner are you using and how much windows bloat have you killed off from background processes and is virtual mem set to 16Gb or more?

Using Nicehash miner, sorry I should have included that.  I haven't done much to background processes and I haven't tweaked virtual memory.  I have a 6x 1060 rig that's stable as a rock, I never did anything to the virtual memory on that one either.  Is it worth a shot? 

[MarkAz]

So keep in mind when tweaking the values and applying them to all your cards, it's only going to be as good as your worst card - so it's quite possible you have one bad card in the mix (or picky card), and it doesn't like to run restricted power or OC'd.  I would try applying the settings one card at a time and let them run for 24+ hours - more than likely this will help you identify the offending card.

You could also try one of the Linux systems and see if you have the same issue, to see if it might be driver related or not - if you see the same thing on the Linux side, then you can be pretty confident it's hardware related.

[treanski]

What miner are you using and how much windows bloat have you killed off from background processes and is virtual mem set to 16Gb or more?

Using Nicehash miner, sorry I should have included that.  I haven't done much to background processes and I haven't tweaked virtual memory.  I have a 6x 1060 rig that's stable as a rock, I never did anything to the virtual memory on that one either.  Is it worth a shot? 

how did you connected your risers? use 4pin or 6pin never with sata and you need to increase virtual memory thats one of the most importand things

[SCSI2]

Have you used those risers before? If not - pull them out and wipe the small PCIe 1x gold fingers with isopropyl alcohol or some such. Some batches of risers come unwashed from the "factory" after manual USB3 socket soldering. Flux (rosin) gets everywhere and in my case is the primary cause of all mysterious problems.

[nick1803]

What miner are you using and how much windows bloat have you killed off from background processes and is virtual mem set to 16Gb or more?

Using Nicehash miner, sorry I should have included that.  I haven't done much to background processes and I haven't tweaked virtual memory.  I have a 6x 1060 rig that's stable as a rock, I never did anything to the virtual memory on that one either.  Is it worth a shot? 

how did you connected your risers? use 4pin or 6pin never with sata and you need to increase virtual memory thats one of the most importand things

I have these risers:

https://www.amazon.com/gp/product/B077JYBDYT/ref=oh_aui_detailpage_o06_s01?ie=UTF8&psc=1

I use the 6pin to sata adapter.  I power only 2 risers with each sata cable from my PSU, as per the norm.  My 1060 rig uses molex adapters and I never tweaked virtual memory on that PC and its fine.  Do you think these 6 pin adapters are requiring it though?

[X299]

Rig:

5x Zotac Amp Extreme 1070 ti
1x PNY Dual Fan 1070 ti
2x EVGA G2 Gold 750 using the Add2PSU adapter.  One PSU powers 3 cards and all the risers.  The other PSU powers 3 cards and the mobo/SSD
4gb Ballistix DDR4
Asus Z270-A
Intel Celeron CPU
120GB Adata SSD
Windows 10 64bit
Nicehash Miner



So I initially got this rig without the other 5 Zotac cards so I just set the motherboard on a box and used the single PNY card for a couple of weeks while I waited for the backordered Zotacs.  Once I got a stable OC on the card, I ran the system without issue with the card plugged straight into the motherboard, no riser.  

Last Thursday I got my other 5 cards so I fully assembled the rig.  I got all the components in the frame and all the cards on the risers.  Booted up fine and I got all 6 cards detected by doing the bios mods for this mobo that are on youtube.  Everything looking good I thought.

I put a mild OC on the cards and started mining.  About 12 hours later the machine was frozen, locked up.  I rebooted and halved the mild OC to just +100 core and +200 mem.  Started mining again, froze.  Ok...so I did no overclocks at all, stock speeds.  80% power in MSI afterburner this whole time, FYI.  Started mining again, froze up in just a few hours.  What the heck?  Then I tried one more time, stock speeds but this time 90% power instead of 80...although 80 should be just fine especially with no OC.  Froze after 10 hours.  

I don't get it.  I'm pretty sure my motherboard and RAM are fine, considering the machine ran for a couple weeks with no issue when I just had the single PNY card.  I have 6 extra risers for a new 1050 ti build I have coming this week, should I try swapping out the risers and see?  Anything I'm overlooking here?  

Its so frustrating to see your rig frozen up on your dashboard and you are helpless at work. Sigh

Power limit decreases to 75%, then it will go well.

[treanski]

never use fucking sata i already said it...not with adapters not any other way, only 4pin or 6pin molex
since i stopped using sata for powering risers 99% of my problems are gone

[fanatic26]

Windows 10 64bit
Nicehash Miner

Found your problems. You are using an unstable OS and mining to a company that takes a cut for no good reason because people are too lazy to manage their own wallets.

[nick1803]

About to head home and tinker, any last thoughts?

I think I will swap out all my risers with this other set I have and increase the virtual memory size.  Also I will check every connection to make sure its seated properly.

[gt_addict]

About to head home and tinker, any last thoughts?

I think I will swap out all my risers with this other set I have and increase the virtual memory size.  Also I will check every connection to make sure its seated properly.

Thats probably best. Personally i would run a gpu and its riser on the same psu. Then you at keast know that the both i puts are receiving the exact same voltage.

[bigjee]

About to head home and tinker, any last thoughts?

I think I will swap out all my risers with this other set I have and increase the virtual memory size.  Also I will check every connection to make sure its seated properly.

Yup. As others have pointed out find the bad card/riser/psu.
Before you test that please remove any overclocks on your gpus as that can add to system instability.
Install one card at a time directly to the mobo. Check all cards this way.
Then start adding risers to each card and test.

On the software side starting with a DDU and a driver reinstall would help eliminate software related issues.

FYI, what celeron are you using. Is it a 3900 (just keep in mind the system can be laggy since its not the most powerful).
 

[tangonaboat]

Use simplemining os, it is well worth $2/rig just the time you’ll save from not having to install drivers for everything

[philipma1957]

did you use msi afterburner to tweak the tdp on the cards?


it polls every 1 second  to show the info

the setting is in microseconds

1000

change to

15000

a lots less work on the cpu

cheap easy free under 1 minute to do it.

[tinfoilMining]

Always power GPU and its riser from same PSU.

[klintistwood]

i don't know if this is available under windows but in the nvidia suite there is something called nvidia-smi, it gives a detailed overview of what's happening with your cards and you can see how much they use (in terms of power), how much they are used by the system, temps, fans, ... With this you can see if one of the card is not responding and try to swap cables, cards, ... If not, then the usual debugging of mining for a couple of hours with card, then another one, ...until it starts crashing again.

[flipflop.0]

I had problems with random freezes and it turned out to be related to the power draw. You can try using the same PSU for the GPUs and their associated risers to it. If it does not help, I'd go with checking all the cables and using treanski's advice.

never use fucking sata i already said it...not with adapters not any other way, only 4pin or 6pin molex
since i stopped using sata for powering risers 99% of my problems are gone

Getting rid of random freezes is a real pain, took me two days going step by step to rule out possible problems.

[nick1803]

I had problems with random freezes and it turned out to be related to the power draw. You can try using the same PSU for the GPUs and their associated risers to it. If it does not help, I'd go with checking all the cables and using treanski's advice.

never use fucking sata i already said it...not with adapters not any other way, only 4pin or 6pin molex
since i stopped using sata for powering risers 99% of my problems are gone

Getting rid of random freezes is a real pain, took me two days going step by step to rule out possible problems.

I think you may be on to something with the power aspect.

So last night I swapped out all my risers and changed virtual memory to 16gb.  I started mining and went to sleep.  I woke up and the machine was frozen up again.  The 1070 ti Zotac AMP EXTREME cards require two 8 pin connections, so I was forced to buy these adapters:

https://www.amazon.com/gp/product/B07611QXG4/ref=oh_aui_detailpage_o05_s00?ie=UTF8&psc=1

I ran those adapters for 4 cards.  I unplugged all four of those cards and just ran the two cards that I am running without adapters, nearly 9 hours of up time so far.  It has not locked up today.  I think 1 or more of those splitters might be bad.  

[SCSI2]

Could be bad/flaky RAM chips. Swap in a known good memory or find a way to test it in place.

[nick1803]

Could be bad/flaky RAM chips. Swap in a known good memory or find a way to test it in place.

I think the RAM stick is fine.  Like I said, this system ran fine for a couple of weeks with just the single PNY card before the other Zotac's showed up.  I would think if the RAM was bad I would have seen the problems before I installed the rest of my GPU's

[nick1803]

Update -

So I have been running 3 cards just fine for coming up on 48 hours, so I'm pretty sure the rig is stable and my issue revolves around the power somehow.


As I explained before, the Zotac 1070 ti Amp Extreme requires 2x 8 pin power, and my EVGA 750 watt G2 Gold's only have 4 VGA ports each, so there isnt enough to run cables directly to each card.  I have the three running now powered directly to the PSU's, but the other 3 I had VGA splitters. 

I had a cable from my PSU connected to one of these:

https://www.amazon.com/gp/product/B07611QXG4/ref=oh_aui_detailpage_o06_s00?ie=UTF8&psc=1

Then that would power a card.  I am new to this so bear with me, is this a no-no?  Or did I just possibly get a bad splitter?  Im fairly certain the system freezes I was having was due to these splitters in some fashion. 

Thanks for any tips! 

[flipflop.0]

One 8-pin power connector and cable of the PSU should be able to handle 2 connections to the GPU. however, running 3 cards with 2x8 pin connectors, you should check the power draw of the cards. could be too much for one 750w psu if you also connect the mb, cpu, hdd and risers to it. you can also check the splitters by adding them one by one and let it run for some time. another point: do you start your 2nd psu manually or do you have a psu adapter for it? add2psu adapter could be one cause, too

[nick1803]

One 8-pin power connector and cable of the PSU should be able to handle 2 connections to the GPU. however, running 3 cards with 2x8 pin connectors, you should check the power draw of the cards. could be too much for one 750w psu if you also connect the mb, cpu, hdd and risers to it. you can also check the splitters by adding them one by one and let it run for some time. another point: do you start your 2nd psu manually or do you have a psu adapter for it? add2psu adapter could be one cause, too

Thanks for the reply.  Do you think I should try running a single 8 pin connector to the cards instead of trying to use the splitters?


Also as far as the add2psu adapter, the 3 cards I have running stable right now are split up between the two PSU's.  One PSU is currently powering the HDD, mobo, cpu, and 2 GPU's and 2 risers.  The other is powering 1 riser and 1 GPU.  So they are both powering things right now just fine, so I would THINK that the psu adapter is ok, but I cant be 100%

[nick1803]

Update -

So since its been running fine now for over a day and a half with the three cards without the splitters, I thought I would try to narrow down what card/splitter is causing me the issue.  I plugged in a 4th card and fired it up and started mining.  While I was updating NiceHash I was looking at the two remaining splitters, I noticed one of them had one out of the eight pins only coming halfway up, probably not even making contact with the cable out of my PSU, effectively running on 7 pins...or 7.5 pins at best lol.  

That was probably my issue all along, I hope at least.  It's a cheap fix, and in the future I will more carefully examine each cable.  It was a little difficult to notice, in my defense.  

For now I am just still running the four cards to play it safe and ensure stability.  I did email the vendor to try and get a replacement though.  

Thanks everyone for tips and advice along the way, I greatly appreciate it.  I will continue to update this thread though in case anyone else out there runs into a similar issue

[nick1803]

Update - issue persists

So the issue is still not fixed, although I feel like I am getting close.  I would love some advice from this forum.

So I ran 1 GPU on each PSU for 48 hours, no issue.  Then I ran 2 GPU on each PSU for 48 hours, no problem.  Then I plugged in the 5th GPU on the PSU that also powers the motherboard.  I wanted to test the PSU that shoulders a bit more of the load.  I ran this for 48 hours, no issue.

I then plugged in the 6th and final GPU, which is powered by the 2nd PSU.  Crashed after 3 hours.  I changed out my Add2PSU for a different unit and started mining again.  Crashed after a few hours or so.  I changed out the VGA cable from the PSU to this GPU, crashed after while too.

I checked my event viewer and see:  Kernel-Power Event 43 (63).  I googled and a lot of people found this error ties back to the memory or PSU.  I ran a memtest, came back clean.  I have a new PSU on order, a slightly beefier EVGA G3 850 that I got a good price on.  

Anyone ever seen this error?  Am I headed in the right direction with my troubleshooting?

Thank you


*edit*  for the record I have tried 3 different risers as well

[nick1803]

This might be the most frustrated I have been in a long time.  I'm about ready to just resell the cards on ebay and make a profit and walk away, I can't for the life of me figure this thing out.  Feeling pretty down

[SCSI2]

Don't worry and keep digging. Memtest doesn't run at the same clock as it may in Windows. Try other stress tests and ensure system is stable under Prime95 for hours or some such.

Does it hang or bugcheck (BSOD) with any miner or only some? Try something stable like ewbf with no OC.

Next I would do chkdsk to make sure your SSD is good.

In general, your troubleshooting should aim at trying to reproduce the problem as often as possible and go from there. Better PSU is not helping with this, but may rather mask the issue if power stability is part of the problem. Try adding more GPUs to see if hangs happen more frequently. If they do, your problem has to do with CPU/RAM/Storage/PCIe.

BTW, did you clean those PCIe connectors already? Turned off spread spectrum clocking for PCIe bus?

Also, try to go back to the previous OS build like 1703 if your current is 1709.

Your HW combo with Asus Z270-A/Celeron/Win10 x64 is rather solid. I have had around ten of them running with no issues I can't understand.

Not sure if you did it already, but be certain that your GPUs and respective risers are powered by same PSU. Don't power riser from one PSU and GPU itself from another. Especially if your PSUs themselves are powered from different AC circuits (or outlets).

[SCSI2]

A few more things to check.

Do you have *any* other power hungry equipment powered from same AC circuit as your rig? If yes, move it elsewhere.

Any EMI sources located nearby? I once had issues with microwave oven interfering with a PC and causing crashes and hangs.

Fresh motherboard BIOS, nVidia driver and recent Windows updates in place? Windows Defender disabled and no other AV in play?

Lastly, to simplify the situation, try to get rid of NiceHash miner and run plain vanilla ewbf targeting NiceHash pool to eliminate possible complications with other miners NiceHash may engage at will. Some miners may not just cause hangs but even kill GPUs like ccminer did with my Gigabyte 1070 ITX by running skunk with no power cap in place.

Finally, uninstall Afterburner and run all GPUs bone stock while troubleshooting this. Phil is correct saying to reduce polling frequency because it caused huge strain on weak CPUs when you have many GPUs to monitor.

Disable all "digital audio" devices in device manager. Use onboard video to connect your monitor and don't use RDP or team viewer while troubleshooting this crap.

[leonix007]

just want to add this, might worth a shot

- try to increase your RAM, if you have an extra stick then try to add it.

- try other miners like Claymore miner and observe, it should have logs on what happen to your specific GPU's before crash, it may give you a hint in some way

Good luck