Finally got 7x 7970 and 7x 7950 cards running on one motherboard!

[Prelude]

Hey guys,

I know a lot of you have given up on running 7x 79xx series cards because it's a huge pain in the ass. Well, I've found the right hardware combo to make it happen!



All cards are normally pegged at 740KH/s, but they're a little slower on this pool for some reason.

Rig hardware:

Ivy Bridge i3 or Pentium
MSI Z77 MPower
7x Sapphire Dual-X 7970
Seasonic 1000w Platinum + Seasonic X760w

Software:

Windows 8 x64
Techpowerup's 13.1 drivers


I have another rig running 7x GB WF3 7950s with an MSI Big Bang Xpower X58. I also have an MSI Z77-GD65 running another 6x 7970s which I'm going to try a 7th card on when I get a chance, but I don't think that board is going to work with 7 cards.

TLDR: 7 cards is possible! 

[1715123411]

1715123411

[1715123411]

1715123411

[1715123411]

1715123411

[Bitweasil]

I could have told you that. :p

I sell systems loaded with 8x 7970 or 4x 7990 for password cracking use, but you wouldn't want to pay their prices - they're enterprise grade rackable systems with a warranty and typically onsite setup of the cluster as we're doing infiniband interconnects for Virtual OpenCL (VCL) clustering.

It's difficult getting 7 or 8 cards reliably working in systems.  It's even more difficult when you have to make them rack mountable.



That's one of our development systems.

[CryptoCluster]

@Bitweasil: Which GPU's are you using in your racks, that they are able to keep acceptable temps while spaced so tight?

[Bitweasil]

@Bitweasil: Which GPU's are you using in your racks, that they are able to keep acceptable temps while spaced so tight?

We're mostly using reference design 7970s/7990s at this point.  That development box was interesting in that we gained significant cooling by removing the shrouds from the GPUs and cooling them with case airflow alone - the shrouds/fans were hurting more than helping.

The airflow in the case is absurd - the newer ones have two full rows of very high power fans shoving air through the PCI section.

The bulk of our builds are reference 7990s at this point, which have no problems as there are only 4 in the system at a time (this gives us space to put an infiniband card in without losing any GPU space as we had to do with 7970s).

[Beaflag VonRathburg]

Why do you think the Mpower will do it while the gd-65 won't? They have the same PCIe layout.

EDIT: Could you also post your GPU settings and if you know it power consumption numbers? I have some of those same cards and love them. I run mine on low voltage, roughly .969-.99x and they put out 630 kh/s each at 950/1500.

[ronaldinho_07]

I could have told you that. :p

I sell systems loaded with 8x 7970 or 4x 7990 for password cracking use, but you wouldn't want to pay their prices - they're enterprise grade rackable systems with a warranty and typically onsite setup of the cluster as we're doing infiniband interconnects for Virtual OpenCL (VCL) clustering.

It's difficult getting 7 or 8 cards reliably working in systems.  It's even more difficult when you have to make them rack mountable.



That's one of our development systems.

i saw this picture a year ago

[Bitweasil]

i saw this picture a year ago

That's about right, given that we've had this system for around a year...

[ronaldinho_07]

Why do you think the Mpower will do it while the gd-65 won't? They have the same PCIe layout.

EDIT: Could you also post your GPU settings and if you know it power consumption numbers? I have some of those same cards and love them. I run mine on low voltage, roughly .969-.99x and they put out 630 kh/s each at 950/1500.

7970 ??
i'm running at 950 core / 1250 mem
they reach 660Khash easily 

[Beaflag VonRathburg]

Why do you think the Mpower will do it while the gd-65 won't? They have the same PCIe layout.

EDIT: Could you also post your GPU settings and if you know it power consumption numbers? I have some of those same cards and love them. I run mine on low voltage, roughly .969-.99x and they put out 630 kh/s each at 950/1500.

7970 ??
i'm running at 950 core / 1250 mem
they reach 660Khash easily  

Yes, 7970s. I have three of the newer ones with the blue PCB and one original one with the black PCB. They behave completely differently in regards to how they react to thread concurrency and settings. I have a weird issue that as I add GPUs to each machine even at the same settings the hashrate of the individual GPU will decrease. This is an older post, but it applies the same to LTC mining: https://bitcointalk.org/index.php?topic=220635.msg2336788#msg2336788 . I haven't done too much testing on the blue cards as I'm really happy with their performance, but I've had the black card for over a year and done a lot with it.

Sapphire: Dual X 7970 (Black PCB) - In a system with 3 other 7970s; one Blue dual x and two reference visiontek cards.

GPU2 is the black PCB and GPU3 is the blue PCB

Sapphire: Dual X 7970 (Black PCB) - In a system by itself getting 120 kh/s more at almost the same settings

[iGotSpots]

Hey guys,

I know a lot of you have given up on running 7x 79xx series cards because it's a huge pain in the ass. Well, I've found the right hardware combo to make it happen!



All cards are normally pegged at 740KH/s, but they're a little slower on this pool for some reason.

Rig hardware:

Ivy Bridge i3 or Pentium
MSI Z77 MPower
7x Sapphire Dual-X 7970
Seasonic 1000w Platinum + Seasonic X760w

Software:

Windows 8 x64
Techpowerup's 13.1 drivers


I have another rig running 7x GB WF3 7950s with an MSI Big Bang Xpower X58. I also have an MSI Z77-GD65 running another 6x 7970s which I'm going to try a 7th card on when I get a chance, but I don't think that board is going to work with 7 cards.

TLDR: 7 cards is possible! 

Let's see the whole rig, please. I'm jealous and want to see what it looks like

[spoid]

who wants to see camera pictures of your screen! if you post stuff like that we want to see the monster. :p

also, for both your systems, what's the wattage pulled from the wall when mining?

I have sapphire OC 7950s (only 3 in a rig) and they could be undervolted quite a bit, from 930 W stock to 570 W without losing performance. I'm curious about the hash/watt ratio your rigs offer.

3x Sapphire OC will get you 1650 MH/s at 570 W for scrypt. I could push them harder, but heat, noise and energy usage started increasing too quickly after 550 MH/s per card.

[Ethera]

I could have told you that. :p

I sell systems loaded with 8x 7970 or 4x 7990 for password cracking use, but you wouldn't want to pay their prices - they're enterprise grade rackable systems with a warranty and typically onsite setup of the cluster as we're doing infiniband interconnects for Virtual OpenCL (VCL) clustering.

It's difficult getting 7 or 8 cards reliably working in systems.  It's even more difficult when you have to make them rack mountable.



That's one of our development systems.

yeah looks uber for the coin mining...
NOT

heat mate, heat! this one has no chance even with 1.5kw direct cooling system blowing directly at it. This is noncence.. This hardware can be used only for old ladies to check emails (thats counting the cryptoworld heat requirements).

[fluffypony]

i saw this picture a year ago

Do your research. This system was featured about a year ago (http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/), but Bitweasil is well-known as being part of Stricture, so he's not kidding.

[hendo420]

I could have told you that. :p

I sell systems loaded with 8x 7970 or 4x 7990 for password cracking use, but you wouldn't want to pay their prices - they're enterprise grade rackable systems with a warranty and typically onsite setup of the cluster as we're doing infiniband interconnects for Virtual OpenCL (VCL) clustering.

It's difficult getting 7 or 8 cards reliably working in systems.  It's even more difficult when you have to make them rack mountable.



That's one of our development systems.

yeah looks uber for the coin mining...
NOT

heat mate, heat! this one has no chance even with 1.5kw direct cooling system blowing directly at it. This is noncence.. This hardware can be used only for old ladies to check emails (thats counting the cryptoworld heat requirements).

Why keep the plastic shrouds on them at that point? They are so close together that the fans on the cards are not doing any good. You might as well take the shoud and fans off, this would open up more surface area for the heatsinks to radiate the heat. Also, thoes dinky fans(even tho they are server fans) are not going to cut it. Use a squirlcage fan, they are AWESOME!. Check out my Old Trunk Litecoin Mining Rig. I should be able to fit 20-25 cards inside of a "wind tunnel" and keep them all below 70c even at 100f outside.

[Bitweasil]

yeah looks uber for the coin mining...
NOT

heat mate, heat! this one has no chance even with 1.5kw direct cooling system blowing directly at it. This is noncence.. This hardware can be used only for old ladies to check emails (thats counting the cryptoworld heat requirements).

It's /not sold as a bitcoin mining system/ - though we certainly have run it under mining loads (both bitcoin and litecoin), just to verify that it can handle the heat/power.  Turns out, it can.  Also, oclHashcat-lite with the register-only algorithms (MD5, MD4, SHA1, etc) is just as hard on the GPUs as bitcoin is, and it handles that great - we use it during the system burnin tests to check all the cards and ensure they're all running properly and cooling properly.  In production, the system runs a blend of oclHashcat-plus, oclHashcat-lite, JTR, Cryptohaze, and some other tools we have.  It's a 24/7/365 production system in the data center, and other than one bad GPU we had a few months ago that was overheating no matter what we did (bad card - we RMA'd it), we have no trouble with it.

To be fair, that's an old photo.  We've since removed the shrouds/fans, as they were not needed in that system and it runs cooler without them, and we've pulled one card so we had room for the infiniband controller card that we're using to cluster that with a number of other systems for our development cluster.  But I assure you, it ran just fine in that configuration under sustained heavy loads.

As a matter of fact, right now, I just grabbed temps off it during an oclHashcat-plus run that's on it.  It's not as heavily loaded right now as mining, but you can see, we don't have any temperature issues (and, apparently, only 6 cards in it - I think we pulled another one for some testing in a different system).

Adapter 0 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 48.00 C

Adapter 1 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 49.00 C

Adapter 2 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 52.00 C

Adapter 3 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 48.00 C

Adapter 4 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 46.00 C

Adapter 5 - AMD Radeon HD 7900 Series
            Sensor 0: Temperature - 52.00 C

But, hey, continue informing me that the system we're using on a regular basis isn't capable of functioning how we are running it (and advertising it).

Do your research. This system was featured about a year ago (http://arstechnica.com/security/2012/12/25-gpu-cluster-cracks-every-standard-windows-password-in-6-hours/), but Bitweasil is well-known as being part of Stricture, so he's not kidding.

Glad you put the pieces together.  Yes, I am part of Stricture, and yes, the systems we sell work as advertised.  I'm actually the owner of about half the systems in our cluster.

If anyone is interested in some quotes, let me know.  If anyone is going to ask for a quote expecting a dirt cheap milk crate system price, don't bother - our stuff isn't cheap, but it is solid, tested hardware designed for sustained use in data centers.  We only sell things we know work.  And we offer a warranty.

[Bitweasil]

Why keep the plastic shrouds on them at that point? They are so close together that the fans on the cards are not doing any good. You might as well take the shoud and fans off, this would open up more surface area for the heatsinks to radiate the heat. Also, thoes dinky fans(even tho they are server fans) are not going to cut it. Use a squirlcage fan, they are AWESOME!. Check out my Old Trunk Litecoin Mining Rig. I should be able to fit 20-25 cards inside of a "wind tunnel" and keep them all below 70c even at 100f outside.

As I noted above, we've since removed the shrouds - that was an early photo.  They run cooler without the shrouds.

Further, those "dinky fans" do, in fact, "cut it" - they move an incredible amount of air, and keep the cards quite cool.  That's also an old system - we've moved to a newer chassis that moves even more air with "dinky fans" as you call them.

Please realize we're not selling systems for bitcoin or litecoin mining.  We're selling large OpenCL clusters for password auditing and other compute needs that scale beyond the capability of one system.  All our boxes are rack mountable, and designed for use in data centers.

I'm sorry, you will not find a data center willing to host your "Old Trunk" mining rig.  Most data centers won't even allow any sort of open air rig in their racks.

[CartmanSPC]

Prelude,

Can you please post your power consumption on your (7) 7970s rig?

Watts and amps if you have them.

I am at (4) 7970s and am hitting over 1000w. and 9.4 amps with each GPU at 1.125 volts.


Watts          Amps     # of GPUs    Notes
291            2.45     1
539            4.55     2
804            6.906    3             verified at the wall with Kill-a-watt
1072           9.422    4             Started using power-in for watts in CorsairLink as efficiency in/out started to go down. Have not verified with Kill-a-watt yet.

Trying to see if I can fit 7 GPUs inside of 16amps. Looks like I'll be just over :/
Note: worst case watts/amps

[FiiNALiZE]

Prelude,

Can you please post your power consumption on your (7) 7970s rig?

Watts and amps if you have them.

I am at (4) 7970s and am hitting over 1000w. and 9.4 amps with each GPU at 1.125 volts.


Watts          Amps     # of GPUs    Notes
291            2.45     1
539            4.55     2
804            6.906    3             verified at the wall with Kill-a-watt
1072           9.422    4             Started using power-in for watts in CorsairLink as efficiency in/out started to go down. Have not verified with Kill-a-watt yet.

Trying to see if I can fit 7 GPUs inside of 16amps. Looks like I'll be just over :/
Note: worst case watts/amps

There's no way you can run 7 7970's on a standard 16amp circuit.

Even if you could undervolt enough to make it fit, it's probably not safe to run that 24/7 for months anyway.

Sometimes I wish I lived in the UK :\

[CartmanSPC]

There's no way you can run 7 7970's on a standard 16amp circuit.
Even if you could undervolt enough to make it fit, it's probably not safe to run that 24/7 for months anyway.

It's a 20 amp circuit. Don't want to go over 16 amps (80%) on it.

[FiiNALiZE]

There's no way you can run 7 7970's on a standard 16amp circuit.
Even if you could undervolt enough to make it fit, it's probably not safe to run that 24/7 for months anyway.

It's a 20 amp circuit. Don't want to go over 16 amps (80%) on it.

Sorry got a bit mixed up on my wording there. I meant standard as in "the safety limit."

My 7970's run at 290w each, so 7 of them will go a bit over.

If you undervolt a few, it could fit. I would just run 6 to be safe.