Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]

[rjk]

Don't anyone ever accuse me of under-sizing my wiring. Each PSU puts out about 200 amps of 12vdc, across 2 negative and 2 positive terminals. I couldn't find a consistent reference across the internet as to what size wire to use - some said 10 gauge was fine, and others said no you must use 2 AWG, with varying opinions in between. So I eventually went with 6 AWG battery cable, and split it across both terminals, like so:


Lots of solder and a bit of flame to make this work.


Special connectors to fit the PSUs I have.


Plugged into the PSU.

The next thing to do is get some copper bus bar and shrink tubing or similar and start hooking it all up as soon as I have the new frame in my grubby hands.

[PatrickHarnett]

Guys, check out the pure awesome radiating from this thing. What a beast.

Quoted so the pic gets more air time.  Drool

[Spotswood]

That thing is a beast!!!!   What are the outer dimensions if I might ask?

Its only 17 by 41 inches.  One of the key features of this tray is its flexibility.  The placement of all the floor "beams" are adjustable (horizontally), along with the vertical placement of the PCI and extra GPU support beams.  Plus all manor of hardware can be attached to all of those t-slots.   

[Inspector 2211]

Its only 17 by 41 inches.  One of the key features of this tray is its flexibility.  The placement of all the floor "beams" are adjustable (horizontally), along with the vertical placement of the PCI and extra GPU support beams.  Plus all manor of hardware can be attached to all of those t-slots.   

Why not simply bolt all ground connections to this aluminum frame? It probably sports lower resistance than traditional power cables. Only relatively short stub ground cables would go to the graphics cards, and, of course, the conventional +12V cables.

Thoughts?

[rjk]

Its only 17 by 41 inches.  One of the key features of this tray is its flexibility.  The placement of all the floor "beams" are adjustable (horizontally), along with the vertical placement of the PCI and extra GPU support beams.  Plus all manor of hardware can be attached to all of those t-slots.  

Why not simply bolt all ground connections to this aluminum frame? It probably sports lower resistance than traditional power cables. Only relatively short stub ground cables would go to the graphics cards, and, of course, the conventional +12V cables.

Thoughts?

I've been considering that. Not exactly sure how well the angle connectors will conduct current though, some seem to be plastic, and others are bolted with a metal connector. It would suck to have the slightest loose connection, which is why I was considering copper busbar.

BTW, I could use some help in determining the correct sizing of the bus bar, as well as the best way to attach things to it. Each PSU is good for 200 amps, and there probably won't be more than 4 of them (or 5 at the most), so 800 to 1000 amps at 12 vdc. I can use crimp connectors on the 6 AWG wire, and then bolt the connectors to the bus with copper bolts, but how should I connect the dozens of little 16-18 AWG wires that go to the 6/8-pin connectors? Perhaps each connector could have its of pair of bolts, and each bolt gets 3 conductors using ring terminals? Like so:

        ===X==*=*=*=*==X==*=*=*=*==X==*=*=*=*==X==*=*=*=*==X===

= is Busbar
X is large power terminal
* is small load terminal

Also, I have been in contact with Advantech about getting a VT-d capable host board, but it will run me about a grand, including the Xeon E3-1225 and 8 GB of RAM . has anyone been able to test the "multiple x sessions" theory to see if it would work for many video cards (instead of VT-d)? I think an earlier poster mentioned that they were going to test it, but I'm not sure who and whether it ever happened.

[Spotswood]

...
I've been considering that. Not exactly sure how well the angle connectors will conduct current though, some seem to be plastic, and others are bolted with a metal connector. It would suck to have the slightest loose connection, which is why I was considering copper busbar.
...

The t-slot extrusions are anodized.  All of the angle brackets are cast aluminum.  

[ARapalo]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

[rjk]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

I am the only one (that I know of) that has tried this. Although, Bigpiggy01 has a different board that works similarly, that he has gotten to work with more than 8 GPUs using IOMMU.

...
I've been considering that. Not exactly sure how well the angle connectors will conduct current though, some seem to be plastic, and others are bolted with a metal connector. It would suck to have the slightest loose connection, which is why I was considering copper busbar.
...

The t-slot extrusions are anodized.  All of the angle brackets are cast aluminum. 

Cool. Maybe it will be sufficient then to use as my ground plane.

[tnkflx]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

I am the only one (that I know of) that has tried this. Although, Bigpiggy01 has a different board that works similarly, that he has gotten to work with more than 8 GPUs using IOMMU.

8 GPUs is not a problem. Going above that is

[rjk]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

I am the only one (that I know of) that has tried this. Although, Bigpiggy01 has a different board that works similarly, that he has gotten to work with more than 8 GPUs using IOMMU.

8 GPUs is not a problem. Going above that is

I know, he has it working with 11 GPUs.

[tnkflx]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

I am the only one (that I know of) that has tried this. Although, Bigpiggy01 has a different board that works similarly, that he has gotten to work with more than 8 GPUs using IOMMU.

8 GPUs is not a problem. Going above that is

I know, he has it working with 11 GPUs.

Do you have a link to his setup somewhere?

[rjk]

This thread/concept is quite awesome. I skimmed some of it, I'll have to go back and try to read every page.

Did majority of people that tried this backplane actually get at least 8 GPUs working?

I am the only one (that I know of) that has tried this. Although, Bigpiggy01 has a different board that works similarly, that he has gotten to work with more than 8 GPUs using IOMMU.

8 GPUs is not a problem. Going above that is

I know, he has it working with 11 GPUs.

Do you have a link to his setup somewhere?

https://bitcointalk.org/index.php?topic=64133.msg834841#msg834841

[discordian666]

I have much respect for the OP and this project, when he gets this fully operational and stable I'd love to do the same thing at my place

Clearly one of my more favourite threads on this forum

[rjk]

If there is a way to use virtualization but still access the GPU hardware, that could allow for some really interesting ways of doing things.

There is, and a lot of commercial virtualization software offers it. It's usually called "GPU pass-through", or something along those lines.

Yes, but I have yet to hear from anyone that has tested it much. Luke-jr is the only one that I know of that has actually mined with a video card in a virtualized OS, using KVM passthough virtualization. But that was only one card, and the system was somewhat unstable. Others have mentioned that since the drivers run in usermode with X, that I might be able to run multiple X instances. But I don't really know enough about how that works to know how to set it up, or even whether it would work at all.

I'm fairly swamped at the moment, but if I can find some time I'll give it a try and see how it works, then report my results.

Did you get a chance to try this? I really would like to know before dropping another grand on this box.

[rjk]

I just got the custom frame from Spotswood, and it is solid. It was flat packed, and I will have to wait until later this evening after work to put it together and take pics. But +100 for Spotswood for an awesome custom-built open air frame.

Also got fan grilles and screws, and have shipped off a couple fans to Garr255 to develop a custom PWM controller for them. Still wondering whether I should get a VT-d capable host board, especially considering that this rig could be somewhat obsolete soon.  Unless Vladimir develops a PCIe based ASIC card that I can shove in.

[Sherkel]

Wow. I just don't know what to say. For 18 GPU's, it would probably be a better deal in the end if you have a good PSU, right?

[rjk]

Wow. I just don't know what to say. For 18 GPU's, it would probably be a better deal in the end if you have a good PSU, right?

Done and done. But if I want more than 8 GPUs, I need VT-d, unless anyone has been able to test and confirm otherwise.

[rjk]

OK I got the custom frame in today, and have partially assembled the system in it. Here are some pics, and I'm going to upload a short video soon to demonstrate how loud the fans are.

[DiabloD3]

Huh, now that I see that coming together thats almost exactly like the rig I always wanted to build but never could afford.

[Spotswood]

To properly mount those fans, you could remove one of the "beams" in the top, placing it in the front in between the posts.