No point...he's using an outdated GPU-Z, hence wrong information there--and he's getting only ~200 kH/s per the two cards he's running in cudaminer. Very deceptive...

Windows will see a PLX PCIe bridge and then see the four cards connected to that bridge. We'll find out soon how the operating systems affect the qty of cards to have drivers loaded. I'm too lazy to try and look it up because there are too many people that are on these forums that are full of crap when they cite limitations of this or that. The extender is in the mail and I've got 9 cards at my disposal so we shall see.

I assume I'll run into a BIOS issue before anything else. Likely run out of BIOS PCIe address space. I'm not too familiar with this stuff.

I'm paying about $190 for the sku that allows me to directly connect 4 GPUs and use an ATX PSU to power the adapter boards.

My math says that for $450 + cost of GPUs, I can get 6 cards running just by running 6 cards per system with x1 risers.
My math says that for $820 + cost of GPUs, I can get 12 cards running on one system by using two of the x4 splitters with a single system.

Tells me I'm saving $80 per 12 GPU's I want to scale.

Probably close enough to not matter, but I just think it's cool to get 12 cards working on one system. Plus, the supplier said that volume discounts are applicable.

To be honest...I could probably design a clone of their host and adapter boards pretty easily...I wouldn't have a problem having PLX sending me their silicon. If I was serious about mining, that's probably what I'd do. If I didn't have a nice office job, it's probably something that would be fun to try. It's just an over-glorified high speed inter-poser board. I'm sure PLX has a reference design that takes the guesswork out of the schematic and layout. It may be interesting to do a BOM scrub of the host/adapter boards, contact the suppliers, and see how much it costs in materials. If only I had the time. If you were semi-serious about GPU mining and wanted to fill a small warehouse, designing your own PCIe extender assembly with PLX silicon would probably be a good move given then triviality of designing the host/adapter boards.

Christian, that other x1 splitter you're linking to splits the bandwidth of a single x1 PCIe link between four cards. Given that cudaminer chokes an x1 slot with one card already, splitting that four ways sounds like a horrible idea. From what I can tell running one instance of cudaminer for a 750 Ti with 5x24 config uses about 68% of a PCIe 2.0 x1 interface bandwidth. Of course, if you find a way to severely reduce the BW required by Cudaminer, then yes, that cheaper splitter could work just fine.

The PLX based splitter that I linked two is actually an x4 to four x1 bridge. Big difference.

I'm doing all of this more just for fun than to try and make a buck. I consider all money spent on anything related to mining basically lost.

ATX means you can use a 12V/5V FDD connector from an ATX PSU...external version means that you only need to provide 12V with the same connector (I think). I believe the details are in the datasheet on the product page.

I'm running:
2x ASUS OC 750 Ti
2x MSI TF 750 Ti
2x EVGA FTW 750 Ti

EVGA are the best for me.

Running two windows windows concurrently with these settings: -H 2 -i 0 -C 1 -T5x24

Gave me 40-60 kH/s boost across six cards to push me consistently over 1800 kH/s. I still need to re-tweak the ASUS cards as their OC is low and not optimized.


The main point of showing this is that I have been able to recoup some of the "lost kH/s" from the "cudaminer riser bottleneck," but not all of it. Some of these cards will do 340 kH/s when plugged into an x16 slot directly and using the -H2 flag.


I have bought one of these babies: http://amfeltec.com/products/flexible-x4-pci-express-4-way-splitter-gpu-oriented/

It is an active x4 to 4x x1 bridge. They use PLX silicon, so it should be legit. I am getting the sku that allows me to power the card on the "adapter" boards. I have purchased four SATA male to FDD adapters from monoprice and plan to use cards that have a six-pin PCIe power header to make sure I'm not pushing too much current through the adapter board (better safe than sorry). The ribbon cable between the host and adapter board is 3.3VAUX and signals ONLY, so we don't have to worry about melting ribbon cable riser cables. smile.gif

The host to adapter board pinout is GND, PRSNT, TX+,TX-,RX+,RX-,CLK+,CLK-, RST,AUX3v3

If any of you guys who have 4+ rigs under your control and have found a cheaper solutions towards a PCIe bridge/extender, please let me know. I think that this may be one of the best ways to do it.

I confirmed that on my Z87-PRO MB that the first two x16 slots will operate at up to x8 link width while still working with four additional risers. Adding this PCIe extender will allow me to run 9 cards on one motherboard. Theoretically, I could purchase an additional extender and run 13 cards. I will be interested to know what limits I hit with Windows 7x64 + Nvidia driver. I figure after I hit 8 GPU's, I'll need to switch to Linux. It will be interesting to see how things scale with Cudaminer today. I believe that this is how we make the 750 Ti scalable to compete with the higher hash-rate density cards out there on the AMD/ATI side.

C

(Assuming Windows environment) I use VNC to start/re-start cudaminer. For all other purposes, I use RDP for other managerial tasks. RDP does not allow for CUDA to run because of the way it works.

For me, I did not see much stability issues until I pushed it beyond four cards. But there was definitely overhead noticed on the aggregate hash rate I was getting with four cards. I'm on Z87-Pro w/ Intel Core-i5 2670k. Latest drivers, etc.

I suspect that the Nvidia driver for W8.1x64 has a bug because I have 6x750 Ti's and I ran into the exact same thing.

Swap over to W7x64 and use settings: -l T5x24 -H2 -i 0 -C 1 -m 1
...I haven't really seen much difference between -m1 and no flag at all.

Use the x86 version of cudaminer when doing this on W7x64.

Finally, for whatever reason, Chrome running in the background with some Flash or HTML5 app (e.g. YouTube) gives a small boost.

Now you're good to go.

I really wanted this to work on W8.1x64 as well, but I think between MSFT/Nvidia/Cudaminer status quo, we're stuck with W7x64 for more than 2 cards running simultaneously.

I think the Zotac cards were one of the best ones to use...so you may be right.

No offense to those desiring screen to work with cudaminer, but I have to say, it seems like there are more pressing "bugs" and features for Christian to work out first. But then again, Christian is the one who is ranking the request priorities here.

BTW, Christian, if you need any additional testing against riser related items, let us know.

LOL Nice. I guess I should screen cap mine as well. I don't have a cool desktop background though. Just a sea of blue....

Although, do you mind sharing your individual overclock settings?

I'm doing this:

MFG   Device   GPU-Z   GPU+   MEM+
Zotac   1   1   135   125
ASUS   5   2   135   100
Zotac   0   3   100   -100
ASUS   2   4   135   50
ASUS   3   5   90   25
ASUS   4   6   135   100

BUT, I am also using custom BIOS that have the ASUS cards already at MEMCLK+200 and ZOTAC cards at MEMCLK+500. So the MEM+ numbers in the table above are additive to that. e.g. Zotac1 is actually +625.

Point in asking is that my power at the wall is ~625. I suspect I can down clock quite a bit on the GPU CLKs and probably retain within 5 kH/s or 1750 while saving some watts.

What risers did you scope out? Powered/Unpowered? USB3/Ribbon Cable?

I ended up getting mine off Ebay from some folks in Oregon. Mine look similar to this (http://www.sintech.cn/riser%20card/ST8019C%20PCI-E%20express%20X1%20to%20x16%20riser%20+USB%20cable.html)

Tried x86 cudaminer with my six cards in W7x64. Peaked a little higher than on x64. I was looking at 1740-1750, with x86, it will peak from time to time at 1770

Will update that post now.

This is great news, thank you, Christian.

6x 750 Ti's (OC'd to stable + BIOS Power Mod)
-2x Zotac Reference
-4x ASUS OC'd w/ 6-pin PCIe header
-1x ASUS OC'd w/ 6-pin PCIe header (not currently in use...someone please help me get 7 cards working on a Z87-Pro!)
ASUS Z87-Pro
Intel G3220
4GB DDR1333
1kW PSU
320GB HDD
6x USB3.0 x1-x16 Powered Riser Cable Assemblies

W/ Windows 8.1 , scaling past two cards was futile--even placed in x16 slots and running at PCIe 3.0 x8 for each, there was a small performance hit. With six cards in the riser cables, I could not get > 230-240kH/s from the cards...some had a hard time hitting 200Kh/s. I was not getting above 1400 kH/s. Machine was not stable enough to allow for overclocking via software tool (MSI AB).

Same HW setup w/ Ubuntu 12.04 yielded about 1600 kH/s (BIOS power mod+modest MEM CLK increase).

Same HW setup w/ WINDOWS 7, was able to individually OC with MSI AB, each card, and get ~ 1750 kH/s out of six riser'd cards. This is the closest I could get to the 1800 kH/s holy grail.

Someone mentioned about that disabling iGPU would allow for the 7th PCie slot to be utilized? That sounds interesting.

I also have witnessed the performance degradtion from using anything but a PCIe 3.0 x16 slot in that native mode. Going down to x8 makes a small hit. Forcing a PCIe 3.0 x16 slot to operate in x1 mode does cause about the same performance hit as a x1 riser.


SO there were two issues going on here...the x1 / riser performance hit....and then a WIndows 8.1 scalability issue. No idea WTF was going on with W8.1, but I'm sticking iwth W7x64 for now for sure.

Just having six cards connected + W8.1 = unstable feeling system.

Here is screen capture of W7x64 with six riser'd GTX 750 Ti's.
http://1drv.ms/1fBX72c

My feeling is that I'm leaving about 10-30 kH/s per card on the table because of x1 slots / risers.

You using lowercase or uppercase k? scrypt needs uppercase K.

You do misunderstand. Each line is a different configuration to track the scalability of performance--I tried 1 card, 2 cards, 4 cards, and then 6 cards. The performance I listed above is what I got.

The degradation of performance and overclockability seems to be linear with the # of cards plugged in at once...

I've now got 6 750 Ti's running on Win8 x64. I'm pretty disappointed.

1 Cards, OC'd, in x16 slot (running at x16): 320-330 kH/s [Total of 320-330 kH/s]
2 Cards, OC'd, in 2 x16 slots (each running at x8): 300 kH/s each  [Total of ~600KH/s]
4 Cards, OC'd, running on x1-x16 risers (each running at x1): ~270-280 each [Total of ~1100 kH/s]
6 Cards, stock, running on x1-x16 risers (each running at x1): 230-240 each (one runs at ~210 and is operating at PCIe 1.1 x1...whereas all the other cards are running at PCIe 2.0 x1) [Total of ~1430kH/s]

I haven't been able to get 7 Cards to show up in Windows...figure that it could be a MoBo/BIOS address space limitation.

I can't really apply any overclock at all when I have 6 cards loaded. MSI AB, GPU-Z, etc., all feel a little unstable when using. Applying even just +100MHz to MEMCLK will cause cudaminer to crash.

I have also noticed that my little G3220 is way too puny to run -H 1 flag.

Here are the flags I'm running -H 2 -i 0 -l T25x16

Will I see improved scaling in Ubuntu with cudaminer?

I'm thinking of returning all of these cards because 230 kH/s per card is not so great. If I was even hitting 280, I'd maybe be able to swallow this.

What is the primary factor here causing a scalability issue? Is it the PCIe operating mode? Is it Windows 8? Is it the Nvidia Driver? Is cudaminer not written to scale well past 4 GPUs?

Yes, I did try running 2 or 4 cards each in a separate cudaminer instance. While the system stability improved, the results yielded the same aggregate hash rate, and any attempts to boost any clocks cause a quick cudaminer crash.

If I can't get more out of these cards by Monday, I'll have to send them back to Newegg.

This chrome trick is only for linux, yes?