Dual PSU Woes - 3GH/s w/ 3x5970 + 2x5870

[JWU42]

Been trying to achieve a 3 GH/s box.  Specs below:

AsRock 970 Extreme 4MB
1 x 2GB ram
Sempron 145
2 x 750 Seasonic Gold (usign Add2PSU adapter)
Running BAMT (USB Key)

I have had it running 2x5970 and 2x5870 and 1x5770 for a day or two.  Added a 3rd 5970 with just one 5870 and it booted and mined OK.  Then went for the final 5870 and it all fell apart.  Now I am unable to get even a single 5970 working off the secondary PSU.

My plan was:

Main PSU
 - MB
 - 1 x 5970
 - 1 x 5870

Second PSU
 - 2 x 5970
 - 2x10W Delta screamer fans (someone mentioned putting some 5v load on a secondary PSU).

The last 5870 was being powered via dual 4pin molex to 6pin PCI-e with each PSU pushing one of the 6pin plugs (2x4pin molex).

Any words of wisdom here?  Maybe use a 950 as the secondary PSU (gives it more headroom with 2x5970).  Put the 2x5970's on the main PSU?

Yes, I understand that a 1500W PSU would be simpler  

So all you folks running 1200W+ rigs what have you learned with respect to what cards where on multi-psu setups.

I am planning to mess with it some more this evening but would love some pointers if I am missing something simple.

Thanks in Advance!

EDIT -- Yes, the 5970's are in the x16 slots and the 5870's are in x1

 

[1713886181]

1713886181

[1713886181]

1713886181

[BkkCoins]

The only thing that comes to mind here is DON'T split  a card between two PSU. I've never tried that but I'm pretty sure that would cause problems. Each PSU has potentially slightly different voltage outputs and this would cause some cross currents between the supplies as they try to "merge". Depending on how the board power connectors actually hook together and what components run off each one there could be various other effects.

I'm not a 5970 user so I don't know what they draw. I would think that 2x5970+MB on one 950W PSU should work and 1x5970+2x5870 on the other one. Definitely don't split a GPU between 2 PSU.

[rjk]

The only thing that comes to mind here is DON'T split  a card between two PSU. I've never tried that but I'm pretty sure that would cause problems. Each PSU has potentially slightly different voltage outputs and this would cause some cross currents between the supplies as they try to "merge". Depending on how the board power connectors actually hook together and what components run off each one there could be various other effects.

I'm not a 5970 user so I don't know what they draw. I would think that 2x5970+MB on one 950W PSU should work and 1x5970+2x5870 on the other one. Definitely don't split a GPU between 2 PSU.

From a strictly engineering perspective, this would usually be true, but according to the PCIe spec it should work fine. ArtForz has posted on this issue several times with links to the technical documents explaining why.

However, I do think you could be underpowered still, OP. I never run my PSUs beyond 50-60% load, and with that hardware combo you might be pushing it over 80-90% load which might have stability issues.

[JWU42]

Thanks guys - 2 things I should have made clear earlier (hastily posting earlier).

1.  Total watts should be 1250 at the wall.  Assuming 88% efficiency the PSU is providing ~1100 watts or roughly 73.33%.

2.  I am struggling to get it to properly load all 3 5970's even without the last 5870 thrown in.  It seems that the 2 x 5970 on one 750 is causing problems but believe I have seen others using this combination of PSUs w/ 4x5970.

[JWU42]

UPDATE...

Have moved the main PSU to a Corsair 950 and the secondary as Seasonic Gold 750.

Good news - all 5 cards show up - with a minor caveat...

The 5970 on the secondary PSU only has 1 core showing active -- so 7 GPUs total.

Any bright ideas?

Code:

root@kong:~# aticonfig --lsa
* 0. 03:00.0 ATI Radeon HD 5900 Series
  1. 04:00.0 ATI Radeon HD 5900 Series
  2. 07:00.0 ATI Radeon HD 5900 Series
  3. 0b:00.0 ATI Radeon HD 5900 Series
  4. 0c:00.0 ATI Radeon HD 5900 Series
  5. 10:00.0 ATI Radeon HD 5800 Series
  6. 11:00.0 ATI Radeon HD 5800 Series

* - Default adapter

[AniceInovation]

BAMT only takes 7 GPUs, lodcrappo thought on moving BAMT to x64 and so support 8, but it's showing as problematic for some beta testers, and he is giving up on the idea.

[JWU42]

I am running the x64 version - should have mentioned that...

Code:

root@kong:/opt/bamt# uname -a
Linux kong 2.6.32-5-amd64 #1 SMP Mon Jan 16 16:22:28 UTC 2012 x86_64 GNU/Linux

Didn't realize the X64 didn't fix the 7 GPU max limit though...

[check_status]

Are you using any PCIe extender cables?

[JWU42]

It is an issue with BAMT...

Nope on the powered extenders...

Have gone to 2x5970 and 3x5870 for now = 7 GPUs.  Working fine at 2.8 GH/s 

[check_status]

I wonder if the GPU restriction is a northbridge or southbridge issue?
There are manufacturers making custom 10 pcie boards for the brute force community. I believe, might be wrong, that the creator of the Social Engineering Toolkit had one built for hash cracking with CUDA.

Edit: I was correct.
 David Kennedy (ReL1K)
http://www.social-engineer.org/framework/Computer_Based_Social_Engineering_Tools:_Social_Engineer_Toolkit_(SET)

[jake262144]

Don't forget there is also a driver-introduced limit of 8 GPUs... you can thank ATI engineers for that.

[bulanula]

I wonder if the GPU restriction is a northbridge or southbridge issue?
There are manufacturers making custom 10 pcie boards for the brute force community. I believe, might be wrong, that the creator of the Social Engineering Toolkit had one built for hash cracking with CUDA.

Please show us some links or proof because drivers restrict you to 8 GPUs per rig AFAIK.

Thanks !

[check_status]

It’s using the Intel X5677 4core processor and a custom Renderstream VDACTr8 motherboard for the x8 – 2 wide PCI-E slots. It’s got 12 gigs of 1333mhz DDR3. An Intel X25-E 64GB solid state drive, and a Seagate 500GB 7200RPM hard-drive. For now it’s running Ubuntu 10.10 x64 Maverick.

We still have some work to do with adding two Magma expansion slots and adding up to 10 new cards (coming soon). On to the machines specifications, it’s a 4U enclosure with x3 1200W power supplies (208V) with 8 Nvidia GTX 580′s with 512 cores (total of 4096 cores) and 1.5GB DDR5.

http://www.secmaniac.com/blog/2011/02/06/building-the-ultimate-bad-arse-cuda-cracking-server/

18 gtx580 cards, 3200 watts, 220v.

Update:
Renderstream VDACTr8 Motherboard
http://www.youtube.com/watch?v=QFUpPe3o1rE
$15,500 (probably with 8 GTX580's)

[jake262144]

Once again, ATI drivers limit you to 8 AMD GPUs.
Bulanula, check_status never said it was an AMD-based machine.

NVidia-based machines are totally orthogonal to mining unless Kepler kicks some serious ass when it finally launches (April?).

Although the machine you linked to is a beauty, it's not the way to go due to hardware cost of the non-standard HPC platform.
While a VDACTr8-based rig would further decrease the system power usage overhead, any decent rig pulling 1100W has only 10% overhead at 110W for the fans, drives, board, and cpu.

[JWU42]

Thanks for taking this way OT 

[jake262144]

Thanks for taking this way OT

Sorry.
I recommend you base your rig on any standard Linux distro. Follow the installer, then just toss in the driver, miner, and sdk and you're ready to go.
It's not that hard and there are tutorials you could follow.
If that secondary PSU shuts down with lower load than before, it is clear that it took some damage when you overloaded it. Judging from how the X-750 is built, I'm guessing one of the 031N06L power transistors responsible for +12V rail died. You should have that PSU looked at.

[JWU42]

Agreed - might do that but like the bells and whistles of BAMT so it stays as is for now...

I was using Xubuntu on HDD for the last year so know how to do it 

[cuz0882]

It's best have 50-60% load on your psu. Even if you get it running your power supplys will burn out. I would run a 1200 or 1500 with a 1000w. In the end you will pay more if you don't. In extra power costs or replacing them.  You could always go with third power supply. I run all my systems with 3x 5970's 1500w psu. I tried a 1000w rosewell originally with just 2 5970's, after 2 weeks it caught on fire.

[jjiimm_64]

It's best have 50-60% load on your psu. Even if you get it running your power supplys will burn out. I would run a 1200 or 1500 with a 1000w. In the end you will pay more if you don't. In extra power costs or replacing them.  You could always go with third power supply. I run all my systems with 3x 5970's 1500w psu. I tried a 1000w rosewell originally with just 2 5970's, after 2 weeks it caught on fire.

NOT tru. maybe you shouldn't buy rosewells.

 the seasonics are rated at full capacity.   ( a 1250 will peak at about 1400)

[check_status]

60% of the peak wattage rating or 60% of the continuous wattage rating?