Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]

[Garr255]

I prefer printers as peripherals

[bulanula]

Any updates ?

I see the nice setup in the magazine thread

[rjk]

Well I was trying to get a new post written up for another thread, but it is a lot harder than I thought. The main problem I have now is that with 12 cards installed, only 11 are shown in lspci

Anyways, I finally took a deep breath and dove into the wiring of the PSU, and I can say I am pretty proud of the result:

[Garr255]

Looking good!

[PolymorphicAssasin]

I hope that window doesn't leak!!   

[gyverlb]

Am I mistaken or are you chaining PCI-e extenders ? I was wondering if I could do that...

[rjk]

Am I mistaken or are you chaining PCI-e extenders ? I was wondering if I could do that...

Yep. For mining with its minimal bandwidth, it should be fine. Has worked in my other rigs as well, you can chain 1x as well as 16x or combine them however you like.

[gyverlb]

Yep. For mining with its minimal bandwidth, it should be fine. Has worked in my other rigs as well, you can chain 1x as well as 16x or combine them however you like.

Sweet... Thanks for the information, one problem less to think about for my growing rig.

[bulanula]

This die for some reason ?

BFL ASIC convinced you to sell off

[rjk]

Been shelved for a bit. Current hurdles are:

Set up a bus - currently have 2x 2360 watt PSUs waiting to juice everything up, but no way to connect them properly.
Figure out how to load a lot of cards - currently with 12 5870s plugged in, only 11 are detected.
Figure out how to properly implement virtualization. As a Linux noob, it's really difficult and time consuming.

I am in the process of consolidating lots of the info from this thread into one big new mega post, but that is also time consuming and difficult.

[EnergyVampire]

Hello rjk,

Your rig is looking really nice!

Quick question on your setup. Which driver(s) are you using? Is the driver causing the 11 GPU limitation? I believe I read somewhere that the AMD drivers could only handle 8 GPUs but your rig has proven otherwise, kudos!

[rjk]

Hello rjk,

Your rig is looking really nice!

Quick question on your setup. Which driver(s) are you using? Is the driver causing the 11 GPU limitation? I believe I read somewhere that the AMD drivers could only handle 8 GPUs but your rig has proven otherwise, kudos!

I don't think it's a driver limit because I don't have any drivers loaded. I am just listing the PCI devices, which as far as I know does not require a driver. That's why I'm pretty sure it's a BIOS limit. There will of course be a driver limit, likely still of 8 devices, which is why I was planning on using VT-d and PCIe passthough with some virtual machines, but I didn't get very far on that.

[imsaguy]

Try adding more RAM.  The 12th might not be detected because of being out of memory.

[rjk]

Try adding more RAM.  The 12th might not be detected because of being out of memory.

I've got 8GB and the board doesn't support any more.

[gyverlb]

@rjk : I tried chaining pci-extenders and surprisingly it worked, but with a reduced hashrate
I lost ~6% on the card I connected in a 1x -> 1x -> 16x configuration with cablesaurus' extenders.

I checked several times that I didn't make a mistake (only inserting or removing the 1x -> 1x cable). Did you see anything like it (it was with a Saphire 5850 extreme) ?

[rjk]

@rjk : I tried chaining pci-extenders and surprisingly it worked, but with a reduced hashrate
I lost ~6% on the card I connected in a 1x -> 1x -> 16x configuration with cablesaurus' extenders.

I checked several times that I didn't make a mistake (only inserting or removing the 1x -> 1x cable). Did you see anything like it (it was with a Saphire 5850 extreme) ?

I don't recall any issues like that, but that was a while ago. Have you tried other cards and other boards? Wonder if it is related to a PCIe timing issue.

[DiabloD3]

@rjk : I tried chaining pci-extenders and surprisingly it worked, but with a reduced hashrate
I lost ~6% on the card I connected in a 1x -> 1x -> 16x configuration with cablesaurus' extenders.

I checked several times that I didn't make a mistake (only inserting or removing the 1x -> 1x cable). Did you see anything like it (it was with a Saphire 5850 extreme) ?

6% is very high. It sounds like your -f/-I is too high/low.

[gyverlb]

@rjk : I tried chaining pci-extenders and surprisingly it worked, but with a reduced hashrate
I lost ~6% on the card I connected in a 1x -> 1x -> 16x configuration with cablesaurus' extenders.

I checked several times that I didn't make a mistake (only inserting or removing the 1x -> 1x cable). Did you see anything like it (it was with a Saphire 5850 extreme) ?

6% is very high. It sounds like your -f/-I is too high/low.

I don't think so I use cgminer with "-I d -g 1 --gpu-dyninterval 250", on my least powerful cards, intensity is then set to 10.

I'm planning to test this on another system with several reference 5870s next time I'll have to stop it for maintenance, I'll report then.

I've compared outputs of lspci -vvv for the two Sapphire 5850 (only one of which was on a double extender), there were some rare differences but not meaningful to me. If you want me to collect some data, I'd be happy to do so on my next maintenance work.

[crosby]

So, you are selling the video cards ?  Sad, I was really hoping to see this completed one day. Backplane up for grabs ??

[Cablez]

So so sad....would have been epic.